README.md 12.6 KB
Newer Older
1
# EMAM2Middleware
Alexander David Hellwig's avatar
Alexander David Hellwig committed
2 3
![pipeline](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/badges/master/build.svg)
![coverage](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/badges/master/coverage.svg)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
4 5 6 7 8 9
## Purpose
This generator takes an EMAM or EMADL model and connects it to a middleware library. If all Ports of two connected Components are marked as middleware Ports, the generator will create 2 executables that can be deployed on different machines.
All communication of these 2 Components will then be tunneled trough the specified middleware:
![MiddlewareAdapter](/uploads/6e9c69e6b56554579551769174df3697/MiddlewareAdapter.png)


10 11 12
## Writing your own Middleware Generator
see [TUTORIAL_ADD_MIDDLEWARE.md](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/TUTORIAL_ADD_MIDDLEWARE.md)

Alexander David Hellwig's avatar
Alexander David Hellwig committed
13
## Dependencies needed to compile the generated projects
Alexander David Hellwig's avatar
Alexander David Hellwig committed
14
### All generated projects
Alexander David Hellwig's avatar
Alexander David Hellwig committed
15
CMake, Make,a C++ compiler, and Armadillo are required to compile the generated projects.
Alexander David Hellwig's avatar
Alexander David Hellwig committed
16 17 18 19 20 21
#### Linux
Gcc is recommended as the C++ compiler.
Example install of all needed packages for ubuntu:
```bash
sudo apt install gcc cmake make
```
22
Then download Armadillo from here: [Linux](https://rwth-aachen.sciebo.de/s/igDWzLpdO5zYHBj/download?path=%2Fubuntu%2F18.10.24-armadillo-linux&files=armadillo-8.500.1-linux.zip) and set the environment variable `Armadillo_HOME` to the base dir of your installation.
Alexander David Hellwig's avatar
Alexander David Hellwig committed
23

Alexander David Hellwig's avatar
Alexander David Hellwig committed
24
To check everything is installed correctly use whereis/ls:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
25 26 27 28 29 30 31
```bash
$ whereis g++
g++: /usr/bin/g++
$ whereis cmake
cmake: /usr/bin/cmake
$ whereis make
make: /usr/bin/make
Alexander David Hellwig's avatar
Alexander David Hellwig committed
32
$ ls "$Armadillo_HOME/include"
Alexander David Hellwig's avatar
Alexander David Hellwig committed
33 34 35
armadillo_bits armadillo.h
```

Alexander David Hellwig's avatar
Alexander David Hellwig committed
36
Compiling a generated project:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
37
```bash
Alexander David Hellwig's avatar
Alexander David Hellwig committed
38
cd /path/to/build/directory
Alexander David Hellwig's avatar
Alexander David Hellwig committed
39 40 41
cmake /path/to/generated/project/source
make
```
Alexander David Hellwig's avatar
Alexander David Hellwig committed
42

Alexander David Hellwig's avatar
Alexander David Hellwig committed
43
#### Windows
44
Mingw64 gcc is recommended as the C++ compiler. Download a version with all needed tools from [here](https://rwth-aachen.sciebo.de/s/igDWzLpdO5zYHBj/download?path=%2Fwin64&files=mingw64.zip).
Alexander David Hellwig's avatar
Alexander David Hellwig committed
45

Alexander David Hellwig's avatar
Alexander David Hellwig committed
46
CMake for Windows: https://cmake.org/download/
Alexander David Hellwig's avatar
Alexander David Hellwig committed
47

Alexander David Hellwig's avatar
Alexander David Hellwig committed
48 49
Make for Windows: http://gnuwin32.sourceforge.net/packages/make.htm

50 51
Add the bin directories of all 3 to your PATH variable.

Alexander David Hellwig's avatar
Alexander David Hellwig committed
52 53
Then download Armadillo from here: [Windows](https://rwth-aachen.sciebo.de/s/igDWzLpdO5zYHBj/download?path=%2Fwin64&files=armadillo-8.200.2.zip) and set the environment variable `Armadillo_HOME` to the base dir of your installation.
To check everything is installed correctly use where/dir:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
54
```batch
Alexander David Hellwig's avatar
Alexander David Hellwig committed
55 56 57 58 59 60
> where g++
C:\mingw64\bin\g++.exe
> where cmake
C:\Program Files\CMake\bin\cmake.exe
> where make
C:\Program Files\make-3.81-bin\bin\make.exe
61
> dir /b "%Armadillo_HOME%\include"
Alexander David Hellwig's avatar
Alexander David Hellwig committed
62 63 64 65 66 67 68 69
armadillo
armadillo.h
...
```

Compiling a generated project:
```batch
cd C:\path\to\build\directory
Alexander David Hellwig's avatar
Alexander David Hellwig committed
70
cmake C:\path\to\generated\project\source -G "MinGW Makefiles"
Alexander David Hellwig's avatar
Alexander David Hellwig committed
71 72
make
```
Alexander David Hellwig's avatar
Alexander David Hellwig committed
73

Michael Günther Beyer's avatar
Michael Günther Beyer committed
74 75
Please note: It is highly recommended, you stick to the exact versions as stated above. Otherwise you might run into trouble regarding the interplay between cmake/make and the Armadillo library. In particular problems have been reported using Cygwin.

Alexander David Hellwig's avatar
Alexander David Hellwig committed
76
### Projects with roscpp generator
Alexander David Hellwig's avatar
Alexander David Hellwig committed
77 78 79
Only for generated projects that contain a ROS adapter(e.g. -g=cpp,roscpp).
ROS Kinetic currently only supports Linux and the installation is described [here](http://wiki.ros.org/kinetic/Installation/Ubuntu).

ahellwig's avatar
ahellwig committed
80
## Usage
81 82
### CLI
Maven generates the jar `embedded-montiarc-math-middleware-generator-{Version}-jar-with-dependencies.jar`
83
and the cli is located in `de.monticore.lang.monticar.generator.middleware.cli.DistributedTargetGeneratorCli`.
84

85
Parameters: `${file path to config json}` OR `-r ${raw json config string}`
86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145

Example: [CliUsage.sh](src/test/resources/CliUsage.sh)

An example config file with all clustering algorithms: [config](src/test/resources/config/parameterTest/clusterParamsAllAlgos.json)

| Name                 | Type   | Required | Description                                                                               |
|----------------------|--------|----------|-------------------------------------------------------------------------------------------|
| modelsDir            | String |     ✅    | path to directory with EMAM models                                                        |
| outputDir            | String |     ✅    | path to output directory for generated files                                              |
| rootModel            | String |     ✅    | fully qualified name of the root model                                                    |
| generators           | List   |     ✅    | List of generator identfiers<br> 'cpp', 'emadlcpp', 'roscpp', 'rclcpp'                    |
| emadlBackend         | String |     ❓    | deep-learning-framework backend<br> 'MXNET'(Default), 'CAFFE2'                                     |
| writeTagFile         | Bool   |     ❓    | Writes a .tag file with all Middleware tags into the generated code<br> Defaults to false |
| clusteringParameters | Object |     ❓    | Options to cluster the component before generating<br> See below                          |

Clustering Parameters:

| Name                | Type         | Required | Description                                                                                                                                                                                                                                       |
|---------------------|--------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| numberOfClusters    | int          | ❓        | Number of clusters the subcomponents should be divided into<br> Overrides numberOfClusters in algorithmParameters                                                                                                                                 |
| flatten             | bool         | ❓        | Replace all components with their subcomponents execpt when it is atomic or the flatten level is reached                                                                                                                                          |
| flattenLevel        | int          | ❓        | Maximal level of component flattening                                                                                                                                                                                                             |
| chooseBy            | String       | ❓        | Strategie to choose from the resulting clusterings<br> bestWithFittingN(Default): if numberOfClusters is set, all results with a different number of clusters are ignored<br> bestOverall: ignore numberOfClusters, choose result with best score |
| algorithmParameters | List<Object> | ❓        | Used to specify which algorithms(and their parameters) are used for clustering                                                                                                                                                                    |

There are 4 different Clustering Algorithms with distinct parameters

Spectral Clustering:

| Name             | Type   | Required | Description                                                                     |
|------------------|--------|----------|---------------------------------------------------------------------------------|
| name             | String | ✅️        | must equal "SpectralClustering"                                                 |
| numberOfClusters | int    | ✅️        | Number of clusters that are created<br> Overwritten by global numberOfClusters  |
| l                | int    | ❓        |                                                                                 |
| sigma            | double | ❓        |                                                                                 |

DBScan:

| Name    | Type   | Required | Description         |
|---------|--------|----------|---------------------|
| name    | String | ✔️        | must equal "DBScan" |
| min_pts | int    | ✔️        |                     |
| radius  | double | ✔️        |                     |

Markov:

| Name         | Type   | Required | Description         |
|--------------|--------|----------|---------------------|
| name         | String | ✔️     | must equal "Markov" |
| max_residual | double | ❓        |                     |
| gamma_exp    | double | ❓        |                     |
| loop_gain    | double | ❓        |                     |
| zero_max     | double | ❓        |                     |

Affinity Propagation:

| Name | Type   | Required | Description                      |
|------|--------|----------|----------------------------------|
| name | String | ✔️        | must equal "AffinityPropagation" |

146 147
    

ahellwig's avatar
ahellwig committed
148
### Defining the connection between a component and the middleware
Alexander David Hellwig's avatar
Alexander David Hellwig committed
149
The connection between middleware and the component is defined as tags on Ports in .tag files.
150
### Example with ROS Middleware:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
151 152
Tags of the type RosConnection can either be simple tags(see Example 3) or define a topic(http://wiki.ros.org/Topics) with name, type and optional msgField(http://wiki.ros.org/msg , 2.)
Examples:
153 154 155
1. [src/test/resources/tests/a/Add.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/Add.tag)
1. [src/test/resources/tests/a/Echo.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/Echo.tag)
1. [src/test/resources/tests/dist/SimpleDist.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/dist/SimpleDist.tag)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
156

157 158
#### Use-case 1: Creating 1 executable
Look at [GenerationTest::testMiddlewareGenerator](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/java/de/monticore/lang/monticar/generator/middleware/GenerationTest.java). The component is defined in [src/test/resources/tests/a/AddComp.emam](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/AddComp.emam) and the tags for the connection to ros are defined in [src/test/resources/tests/a/Add.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/Add.tag)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
159

160 161
#### Use-case 2: Creating multiple executables for distributed systems
Look at [GenerationTest::testDistributedTargetGenerator](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/java/de/monticore/lang/monticar/generator/middleware/GenerationTest.java). The component is defined in [src/test/resources/dist/DistComp.emam](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/dist/DistComp.emam) and the tags for the connection to ros are defined in [src/test/resources/tests/dist/SimpleDist.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/dist/SimpleDist.tag)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
162

163
#### Compile and run the generated ROS Projects
Alexander David Hellwig's avatar
Alexander David Hellwig committed
164 165 166
1. install needed software:
    * ROS Kinetic(http://wiki.ros.org/kinetic/Installation)
    * CMake(https://cmake.org/)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
167 168
    * Armadillo 8 or higher( www.arma.sourceforge.net)
        * creating a copy of the library named armadillo.h might be necessary.
Alexander David Hellwig's avatar
Alexander David Hellwig committed
169 170 171 172 173 174 175 176 177
1. source your ros environment(http://wiki.ros.org/ROS/Tutorials/InstallingandConfiguringROSEnvironment , 2.)
1. a) run src/test/resources/TargetCompilation.sh from **this project's** root
1. or b) compile a single project by
    * changing to the **generated project's** root
    * create a folder build/ and change into it
    * run: cmake ../src
    * run: make
1. Start ros and the other nodes
    * minimal working example: run: roscore
ahellwig's avatar
ahellwig committed
178
1. If the project was created by a MiddlewareGenerator, run the executable(s) at build/coordinator(/<subcomp.name>)/Coordinator_<(sub)component.name>
179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197
2.

## Automatic Clustering
To simplify the creation of distributed systems, the generator can automatically split the model into a given number of clusters.

Supported:
* Spectral Clustering
* ...

Procedure:
* Convert the Symbol Table of a Component into a adjacency matrix
* (Add costs given by type to matrix. E.g. float is cheaper than Matrix of floats)
* Feed into ml library(e.g. [smile ml](https://github.com/haifengl/smile)) with the selected clustering algorithm
* (Compare the result of different algorithms)
* Generate Middleware tags seperating the clusters
* Feed into existing manual clustering architecture
* (generate)


198