README.md 14.4 KB
Newer Older
1
# EMAM2Middleware
Alexander David Hellwig's avatar
Alexander David Hellwig committed
2 3
![pipeline](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/badges/master/build.svg)
![coverage](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/badges/master/coverage.svg)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
4 5 6 7
## Purpose
This generator takes an EMAM or EMADL model and connects it to a middleware library. If all Ports of two connected Components are marked as middleware Ports, the generator will create 2 executables that can be deployed on different machines.
All communication of these 2 Components will then be tunneled trough the specified middleware:
![MiddlewareAdapter](/uploads/6e9c69e6b56554579551769174df3697/MiddlewareAdapter.png)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
8
It also supports automatic clustering of the subcomponents to deploy on different machines.
Alexander David Hellwig's avatar
Alexander David Hellwig committed
9

10 11 12
## Writing your own Middleware Generator
see [TUTORIAL_ADD_MIDDLEWARE.md](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/TUTORIAL_ADD_MIDDLEWARE.md)

Alexander David Hellwig's avatar
Alexander David Hellwig committed
13
## Dependencies needed to compile the generated projects
Alexander David Hellwig's avatar
Alexander David Hellwig committed
14
### All generated projects
Alexander David Hellwig's avatar
Alexander David Hellwig committed
15
CMake, Make,a C++ compiler, and Armadillo are required to compile the generated projects.
Alexander David Hellwig's avatar
Alexander David Hellwig committed
16 17 18 19 20 21
#### Linux
Gcc is recommended as the C++ compiler.
Example install of all needed packages for ubuntu:
```bash
sudo apt install gcc cmake make
```
22
Then download Armadillo from here: [Linux](https://rwth-aachen.sciebo.de/s/igDWzLpdO5zYHBj/download?path=%2Fubuntu%2F18.10.24-armadillo-linux&files=armadillo-8.500.1-linux.zip) and set the environment variable `Armadillo_HOME` to the base dir of your installation.
Alexander David Hellwig's avatar
Alexander David Hellwig committed
23

Alexander David Hellwig's avatar
Alexander David Hellwig committed
24
To check everything is installed correctly use whereis/ls:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
25 26 27 28 29 30 31
```bash
$ whereis g++
g++: /usr/bin/g++
$ whereis cmake
cmake: /usr/bin/cmake
$ whereis make
make: /usr/bin/make
Alexander David Hellwig's avatar
Alexander David Hellwig committed
32
$ ls "$Armadillo_HOME/include"
Alexander David Hellwig's avatar
Alexander David Hellwig committed
33 34 35
armadillo_bits armadillo.h
```

Alexander David Hellwig's avatar
Alexander David Hellwig committed
36
Compiling a generated project:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
37
```bash
Alexander David Hellwig's avatar
Alexander David Hellwig committed
38
cd /path/to/build/directory
Alexander David Hellwig's avatar
Alexander David Hellwig committed
39 40 41
cmake /path/to/generated/project/source
make
```
Alexander David Hellwig's avatar
Alexander David Hellwig committed
42

Alexander David Hellwig's avatar
Alexander David Hellwig committed
43
#### Windows
44
Mingw64 gcc is recommended as the C++ compiler. Download a version with all needed tools from [here](https://rwth-aachen.sciebo.de/s/igDWzLpdO5zYHBj/download?path=%2Fwin64&files=mingw64.zip).
Alexander David Hellwig's avatar
Alexander David Hellwig committed
45

Alexander David Hellwig's avatar
Alexander David Hellwig committed
46
CMake for Windows: https://cmake.org/download/
Alexander David Hellwig's avatar
Alexander David Hellwig committed
47

Alexander David Hellwig's avatar
Alexander David Hellwig committed
48 49
Make for Windows: http://gnuwin32.sourceforge.net/packages/make.htm

50 51
Add the bin directories of all 3 to your PATH variable.

Alexander David Hellwig's avatar
Alexander David Hellwig committed
52 53
Then download Armadillo from here: [Windows](https://rwth-aachen.sciebo.de/s/igDWzLpdO5zYHBj/download?path=%2Fwin64&files=armadillo-8.200.2.zip) and set the environment variable `Armadillo_HOME` to the base dir of your installation.
To check everything is installed correctly use where/dir:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
54
```batch
Alexander David Hellwig's avatar
Alexander David Hellwig committed
55 56 57 58 59 60
> where g++
C:\mingw64\bin\g++.exe
> where cmake
C:\Program Files\CMake\bin\cmake.exe
> where make
C:\Program Files\make-3.81-bin\bin\make.exe
61
> dir /b "%Armadillo_HOME%\include"
Alexander David Hellwig's avatar
Alexander David Hellwig committed
62 63 64 65 66 67 68 69
armadillo
armadillo.h
...
```

Compiling a generated project:
```batch
cd C:\path\to\build\directory
Alexander David Hellwig's avatar
Alexander David Hellwig committed
70
cmake C:\path\to\generated\project\source -G "MinGW Makefiles"
Alexander David Hellwig's avatar
Alexander David Hellwig committed
71 72
make
```
Alexander David Hellwig's avatar
Alexander David Hellwig committed
73

Michael Günther Beyer's avatar
Michael Günther Beyer committed
74 75
Please note: It is highly recommended, you stick to the exact versions as stated above. Otherwise you might run into trouble regarding the interplay between cmake/make and the Armadillo library. In particular problems have been reported using Cygwin.

Alexander David Hellwig's avatar
Alexander David Hellwig committed
76
### Projects with roscpp generator
Alexander David Hellwig's avatar
Alexander David Hellwig committed
77 78 79
Only for generated projects that contain a ROS adapter(e.g. -g=cpp,roscpp).
ROS Kinetic currently only supports Linux and the installation is described [here](http://wiki.ros.org/kinetic/Installation/Ubuntu).

ahellwig's avatar
ahellwig committed
80
## Usage
81 82
### CLI
Maven generates the jar `embedded-montiarc-math-middleware-generator-{Version}-jar-with-dependencies.jar`
83
and the cli is located in `de.monticore.lang.monticar.generator.middleware.cli.DistributedTargetGeneratorCli`.
84

85
Parameters: `${file path to config json}` OR `-r ${raw json config string}`
86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107

Example: [CliUsage.sh](src/test/resources/CliUsage.sh)

An example config file with all clustering algorithms: [config](src/test/resources/config/parameterTest/clusterParamsAllAlgos.json)

| Name                 | Type   | Required | Description                                                                               |
|----------------------|--------|----------|-------------------------------------------------------------------------------------------|
| modelsDir            | String |     ✅    | path to directory with EMAM models                                                        |
| outputDir            | String |     ✅    | path to output directory for generated files                                              |
| rootModel            | String |     ✅    | fully qualified name of the root model                                                    |
| generators           | List   |     ✅    | List of generator identfiers<br> 'cpp', 'emadlcpp', 'roscpp', 'rclcpp'                    |
| emadlBackend         | String |     ❓    | deep-learning-framework backend<br> 'MXNET'(Default), 'CAFFE2'                                     |
| writeTagFile         | Bool   |     ❓    | Writes a .tag file with all Middleware tags into the generated code<br> Defaults to false |
| clusteringParameters | Object |     ❓    | Options to cluster the component before generating<br> See below                          |

Clustering Parameters:

| Name                | Type         | Required | Description                                                                                                                                                                                                                                       |
|---------------------|--------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| numberOfClusters    | int          | ❓        | Number of clusters the subcomponents should be divided into<br> Overrides numberOfClusters in algorithmParameters                                                                                                                                 |
| flatten             | bool         | ❓        | Replace all components with their subcomponents execpt when it is atomic or the flatten level is reached                                                                                                                                          |
| flattenLevel        | int          | ❓        | Maximal level of component flattening                                                                                                                                                                                                             |
108 109
| metric            | String       | ❓        | Metric to evaluate the quality of the resulting clusters. Available: "CommunicationCost"(Default), "Silhouette"|
| chooseBy            | String       | ❓        | Strategy to choose from the resulting clusterings<br> bestWithFittingN(Default): if numberOfClusters is set, all results with a different number of clusters are ignored<br> bestOverall: ignore numberOfClusters, choose result with best score |
110 111 112 113
| algorithmParameters | List<Object> | ❓        | Used to specify which algorithms(and their parameters) are used for clustering                                                                                                                                                                    |

There are 4 different Clustering Algorithms with distinct parameters

114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129
Every parameter of the clustering algorithms can be dynamic, enabling automatic search for the best values. Available are lists and generators as seen in the example below:
```json
"sigma":[1,2,3]
"sigma":{
  "min":1,
  "max":3,
  "step":1
}
"sigma":{
  "min":1,
  "max":3,
  "count":3
}
```
Also see [clusterDynamic.json](src/test/resources/config/parameterTest/clusterDynamic.json) and [clusterDynamicList.json](src/test/resources/config/parameterTest/clusterDynamicList.json)

130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162
Spectral Clustering:

| Name             | Type   | Required | Description                                                                     |
|------------------|--------|----------|---------------------------------------------------------------------------------|
| name             | String | ✅️        | must equal "SpectralClustering"                                                 |
| numberOfClusters | int    | ✅️        | Number of clusters that are created<br> Overwritten by global numberOfClusters  |
| l                | int    | ❓        |                                                                                 |
| sigma            | double | ❓        |                                                                                 |

DBScan:

| Name    | Type   | Required | Description         |
|---------|--------|----------|---------------------|
| name    | String | ✔️        | must equal "DBScan" |
| min_pts | int    | ✔️        |                     |
| radius  | double | ✔️        |                     |

Markov:

| Name         | Type   | Required | Description         |
|--------------|--------|----------|---------------------|
| name         | String | ✔️     | must equal "Markov" |
| max_residual | double | ❓        |                     |
| gamma_exp    | double | ❓        |                     |
| loop_gain    | double | ❓        |                     |
| zero_max     | double | ❓        |                     |

Affinity Propagation:

| Name | Type   | Required | Description                      |
|------|--------|----------|----------------------------------|
| name | String | ✔️        | must equal "AffinityPropagation" |

163

Alexander David Hellwig's avatar
Alexander David Hellwig committed
164 165 166 167 168 169
### Visulization of clustering results
There are 3 scripts available to visualise the results of the clustering process. They all create graphs for each of the 4 evaluation models:
1. [evaluationVisualisation.py](src/test/resources/evaluationVisualisation.py): bar graphs that compare the size of clusters, distance score, and time taken in ms
2. [montecarlovisualisation.py](src/test/resources/montecarlovisualisation.py): line graph visualising the average distance cost for random clustering(with Monte Carlo)
3. [silhouetteVisualisation.py](src/test/resources/silhouetteVisualisation.py): point graph visualising the silhouette score of different clusterings sorted by cluster size

Alexander David Hellwig's avatar
Alexander David Hellwig committed
170 171 172
Before using them install Python 3+ and the packages `matplotlib` and `numpy`.

After running `EvaluationTest`(Warning: very long runtime) you can visualise the results by calling(from the project root):
Alexander David Hellwig's avatar
Alexander David Hellwig committed
173 174 175 176 177
```bash
python3 src/test/resources/evaluationVisualisation.py target/evaluation/autopilot/emam/clusteringResults.json target/evaluation/pacman/emam/clusteringResults.json target/evaluation/supermario/emam/clusteringResults.json target/evaluation/daimler/emam/clusteringResults.json
```
or
```bash
Alexander David Hellwig's avatar
Alexander David Hellwig committed
178 179 180 181
python3 src/test/resources/montecarlovisualisation.py target/evaluation/autopilotMC/monteCarloResults.json target/evaluation/pacmanMC/monteCarloResults.json target/evaluation/supermarioMC/monteCarloResults.json target/evaluation/daimlerMC/monteCarloResults.json
```
or
```bash
Alexander David Hellwig's avatar
Alexander David Hellwig committed
182 183 184 185
python3 src/test/resources/silhouetteVisualisation.py target/evaluation/autopilotSilhouette/emam/clusteringResults.json target/evaluation/pacmanSilhouette/emam/clusteringResults.json target/evaluation/supermarioSilhouette/emam/clusteringResults.json target/evaluation/daimlerSilhouette/emam/clusteringResults.json
```

## Defining the connection between a component and the middleware
Alexander David Hellwig's avatar
Alexander David Hellwig committed
186
The connection between middleware and the component is defined as tags on Ports in .tag files.
187
### Example with ROS Middleware:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
188 189
Tags of the type RosConnection can either be simple tags(see Example 3) or define a topic(http://wiki.ros.org/Topics) with name, type and optional msgField(http://wiki.ros.org/msg , 2.)
Examples:
190 191 192
1. [src/test/resources/tests/a/Add.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/Add.tag)
1. [src/test/resources/tests/a/Echo.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/Echo.tag)
1. [src/test/resources/tests/dist/SimpleDist.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/dist/SimpleDist.tag)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
193

194 195
#### Use-case 1: Creating 1 executable
Look at [GenerationTest::testMiddlewareGenerator](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/java/de/monticore/lang/monticar/generator/middleware/GenerationTest.java). The component is defined in [src/test/resources/tests/a/AddComp.emam](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/AddComp.emam) and the tags for the connection to ros are defined in [src/test/resources/tests/a/Add.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/Add.tag)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
196

197 198
#### Use-case 2: Creating multiple executables for distributed systems
Look at [GenerationTest::testDistributedTargetGenerator](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/java/de/monticore/lang/monticar/generator/middleware/GenerationTest.java). The component is defined in [src/test/resources/dist/DistComp.emam](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/dist/DistComp.emam) and the tags for the connection to ros are defined in [src/test/resources/tests/dist/SimpleDist.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/dist/SimpleDist.tag)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
199

200
#### Compile and run the generated ROS Projects
Alexander David Hellwig's avatar
Alexander David Hellwig committed
201 202 203
1. install needed software:
    * ROS Kinetic(http://wiki.ros.org/kinetic/Installation)
    * CMake(https://cmake.org/)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
204 205
    * Armadillo 8 or higher( www.arma.sourceforge.net)
        * creating a copy of the library named armadillo.h might be necessary.
Alexander David Hellwig's avatar
Alexander David Hellwig committed
206 207 208 209 210 211 212 213 214
1. source your ros environment(http://wiki.ros.org/ROS/Tutorials/InstallingandConfiguringROSEnvironment , 2.)
1. a) run src/test/resources/TargetCompilation.sh from **this project's** root
1. or b) compile a single project by
    * changing to the **generated project's** root
    * create a folder build/ and change into it
    * run: cmake ../src
    * run: make
1. Start ros and the other nodes
    * minimal working example: run: roscore
ahellwig's avatar
ahellwig committed
215
1. If the project was created by a MiddlewareGenerator, run the executable(s) at build/coordinator(/<subcomp.name>)/Coordinator_<(sub)component.name>
216 217
2.

218