Aufgrund einer Störung des s3 Storage, könnten in nächster Zeit folgende GitLab Funktionen nicht zur Verfügung stehen: LFS, Container Registry, Job Artifacs, Uploads (Wiki, Bilder, Projekt-Exporte). Wir bitten um Verständnis. Es wird mit Hochdruck an der Behebung des Problems gearbeitet. Weitere Informationen zur Störung des Object Storage finden Sie hier: https://maintenance.itc.rwth-aachen.de/ticket/status/messages/59-object-storage-pilot

README.md 14.6 KB
Newer Older
1
# EMAM2Middleware
Alexander David Hellwig's avatar
Alexander David Hellwig committed
2
3
![pipeline](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/badges/master/build.svg)
![coverage](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/badges/master/coverage.svg)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
4
5
6
7
8
9
## Purpose
This generator takes an EMAM or EMADL model and connects it to a middleware library. If all Ports of two connected Components are marked as middleware Ports, the generator will create 2 executables that can be deployed on different machines.
All communication of these 2 Components will then be tunneled trough the specified middleware:
![MiddlewareAdapter](/uploads/6e9c69e6b56554579551769174df3697/MiddlewareAdapter.png)


10
11
12
## Writing your own Middleware Generator
see [TUTORIAL_ADD_MIDDLEWARE.md](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/TUTORIAL_ADD_MIDDLEWARE.md)

Alexander David Hellwig's avatar
Alexander David Hellwig committed
13
## Dependencies needed to compile the generated projects
Alexander David Hellwig's avatar
Alexander David Hellwig committed
14
### All generated projects
Alexander David Hellwig's avatar
Alexander David Hellwig committed
15
CMake, Make,a C++ compiler, and Armadillo are required to compile the generated projects.
Alexander David Hellwig's avatar
Alexander David Hellwig committed
16
17
18
19
20
21
#### Linux
Gcc is recommended as the C++ compiler.
Example install of all needed packages for ubuntu:
```bash
sudo apt install gcc cmake make
```
22
Then download Armadillo from here: [Linux](https://rwth-aachen.sciebo.de/s/igDWzLpdO5zYHBj/download?path=%2Fubuntu%2F18.10.24-armadillo-linux&files=armadillo-8.500.1-linux.zip) and set the environment variable `Armadillo_HOME` to the base dir of your installation.
Alexander David Hellwig's avatar
Alexander David Hellwig committed
23

Alexander David Hellwig's avatar
Alexander David Hellwig committed
24
To check everything is installed correctly use whereis/ls:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
25
26
27
28
29
30
31
```bash
$ whereis g++
g++: /usr/bin/g++
$ whereis cmake
cmake: /usr/bin/cmake
$ whereis make
make: /usr/bin/make
Alexander David Hellwig's avatar
Alexander David Hellwig committed
32
$ ls "$Armadillo_HOME/include"
Alexander David Hellwig's avatar
Alexander David Hellwig committed
33
34
35
armadillo_bits armadillo.h
```

Alexander David Hellwig's avatar
Alexander David Hellwig committed
36
Compiling a generated project:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
37
```bash
Alexander David Hellwig's avatar
Alexander David Hellwig committed
38
cd /path/to/build/directory
Alexander David Hellwig's avatar
Alexander David Hellwig committed
39
40
41
cmake /path/to/generated/project/source
make
```
Alexander David Hellwig's avatar
Alexander David Hellwig committed
42

Alexander David Hellwig's avatar
Alexander David Hellwig committed
43
#### Windows
44
Mingw64 gcc is recommended as the C++ compiler. Download a version with all needed tools from [here](https://rwth-aachen.sciebo.de/s/igDWzLpdO5zYHBj/download?path=%2Fwin64&files=mingw64.zip).
Alexander David Hellwig's avatar
Alexander David Hellwig committed
45

Alexander David Hellwig's avatar
Alexander David Hellwig committed
46
CMake for Windows: https://cmake.org/download/
Alexander David Hellwig's avatar
Alexander David Hellwig committed
47

Alexander David Hellwig's avatar
Alexander David Hellwig committed
48
49
Make for Windows: http://gnuwin32.sourceforge.net/packages/make.htm

50
51
Add the bin directories of all 3 to your PATH variable.

Alexander David Hellwig's avatar
Alexander David Hellwig committed
52
53
Then download Armadillo from here: [Windows](https://rwth-aachen.sciebo.de/s/igDWzLpdO5zYHBj/download?path=%2Fwin64&files=armadillo-8.200.2.zip) and set the environment variable `Armadillo_HOME` to the base dir of your installation.
To check everything is installed correctly use where/dir:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
54
```batch
Alexander David Hellwig's avatar
Alexander David Hellwig committed
55
56
57
58
59
60
> where g++
C:\mingw64\bin\g++.exe
> where cmake
C:\Program Files\CMake\bin\cmake.exe
> where make
C:\Program Files\make-3.81-bin\bin\make.exe
61
> dir /b "%Armadillo_HOME%\include"
Alexander David Hellwig's avatar
Alexander David Hellwig committed
62
63
64
65
66
67
68
69
armadillo
armadillo.h
...
```

Compiling a generated project:
```batch
cd C:\path\to\build\directory
Alexander David Hellwig's avatar
Alexander David Hellwig committed
70
cmake C:\path\to\generated\project\source -G "MinGW Makefiles"
Alexander David Hellwig's avatar
Alexander David Hellwig committed
71
72
make
```
Alexander David Hellwig's avatar
Alexander David Hellwig committed
73

Michael Günther Beyer's avatar
Michael Günther Beyer committed
74
75
Please note: It is highly recommended, you stick to the exact versions as stated above. Otherwise you might run into trouble regarding the interplay between cmake/make and the Armadillo library. In particular problems have been reported using Cygwin.

Alexander David Hellwig's avatar
Alexander David Hellwig committed
76
### Projects with roscpp generator
Alexander David Hellwig's avatar
Alexander David Hellwig committed
77
78
79
Only for generated projects that contain a ROS adapter(e.g. -g=cpp,roscpp).
ROS Kinetic currently only supports Linux and the installation is described [here](http://wiki.ros.org/kinetic/Installation/Ubuntu).

ahellwig's avatar
ahellwig committed
80
## Usage
81
82
### CLI
Maven generates the jar `embedded-montiarc-math-middleware-generator-{Version}-jar-with-dependencies.jar`
83
and the cli is located in `de.monticore.lang.monticar.generator.middleware.cli.DistributedTargetGeneratorCli`.
84

85
Parameters: `${file path to config json}` OR `-r ${raw json config string}`
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107

Example: [CliUsage.sh](src/test/resources/CliUsage.sh)

An example config file with all clustering algorithms: [config](src/test/resources/config/parameterTest/clusterParamsAllAlgos.json)

| Name                 | Type   | Required | Description                                                                               |
|----------------------|--------|----------|-------------------------------------------------------------------------------------------|
| modelsDir            | String |     ✅    | path to directory with EMAM models                                                        |
| outputDir            | String |     ✅    | path to output directory for generated files                                              |
| rootModel            | String |     ✅    | fully qualified name of the root model                                                    |
| generators           | List   |     ✅    | List of generator identfiers<br> 'cpp', 'emadlcpp', 'roscpp', 'rclcpp'                    |
| emadlBackend         | String |     ❓    | deep-learning-framework backend<br> 'MXNET'(Default), 'CAFFE2'                                     |
| writeTagFile         | Bool   |     ❓    | Writes a .tag file with all Middleware tags into the generated code<br> Defaults to false |
| clusteringParameters | Object |     ❓    | Options to cluster the component before generating<br> See below                          |

Clustering Parameters:

| Name                | Type         | Required | Description                                                                                                                                                                                                                                       |
|---------------------|--------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| numberOfClusters    | int          | ❓        | Number of clusters the subcomponents should be divided into<br> Overrides numberOfClusters in algorithmParameters                                                                                                                                 |
| flatten             | bool         | ❓        | Replace all components with their subcomponents execpt when it is atomic or the flatten level is reached                                                                                                                                          |
| flattenLevel        | int          | ❓        | Maximal level of component flattening                                                                                                                                                                                                             |
108
109
| metric            | String       | ❓        | Metric to evaluate the quality of the resulting clusters. Available: "CommunicationCost"(Default), "Silhouette"|
| chooseBy            | String       | ❓        | Strategy to choose from the resulting clusterings<br> bestWithFittingN(Default): if numberOfClusters is set, all results with a different number of clusters are ignored<br> bestOverall: ignore numberOfClusters, choose result with best score |
110
111
112
113
| algorithmParameters | List<Object> | ❓        | Used to specify which algorithms(and their parameters) are used for clustering                                                                                                                                                                    |

There are 4 different Clustering Algorithms with distinct parameters

114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
Every parameter of the clustering algorithms can be dynamic, enabling automatic search for the best values. Available are lists and generators as seen in the example below:
```json
"sigma":[1,2,3]
"sigma":{
  "min":1,
  "max":3,
  "step":1
}
"sigma":{
  "min":1,
  "max":3,
  "count":3
}
```
Also see [clusterDynamic.json](src/test/resources/config/parameterTest/clusterDynamic.json) and [clusterDynamicList.json](src/test/resources/config/parameterTest/clusterDynamicList.json)

130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
Spectral Clustering:

| Name             | Type   | Required | Description                                                                     |
|------------------|--------|----------|---------------------------------------------------------------------------------|
| name             | String | ✅️        | must equal "SpectralClustering"                                                 |
| numberOfClusters | int    | ✅️        | Number of clusters that are created<br> Overwritten by global numberOfClusters  |
| l                | int    | ❓        |                                                                                 |
| sigma            | double | ❓        |                                                                                 |

DBScan:

| Name    | Type   | Required | Description         |
|---------|--------|----------|---------------------|
| name    | String | ✔️        | must equal "DBScan" |
| min_pts | int    | ✔️        |                     |
| radius  | double | ✔️        |                     |

Markov:

| Name         | Type   | Required | Description         |
|--------------|--------|----------|---------------------|
| name         | String | ✔️     | must equal "Markov" |
| max_residual | double | ❓        |                     |
| gamma_exp    | double | ❓        |                     |
| loop_gain    | double | ❓        |                     |
| zero_max     | double | ❓        |                     |

Affinity Propagation:

| Name | Type   | Required | Description                      |
|------|--------|----------|----------------------------------|
| name | String | ✔️        | must equal "AffinityPropagation" |

163

Alexander David Hellwig's avatar
Alexander David Hellwig committed
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
### Visulization of clustering results
There are 3 scripts available to visualise the results of the clustering process. They all create graphs for each of the 4 evaluation models:
1. [evaluationVisualisation.py](src/test/resources/evaluationVisualisation.py): bar graphs that compare the size of clusters, distance score, and time taken in ms
2. [montecarlovisualisation.py](src/test/resources/montecarlovisualisation.py): line graph visualising the average distance cost for random clustering(with Monte Carlo)
3. [silhouetteVisualisation.py](src/test/resources/silhouetteVisualisation.py): point graph visualising the silhouette score of different clusterings sorted by cluster size

Execute them by running from the root of the project after executing all test in `EvaluationTest` or `MonteCarloIntegrationTest`:
```bash
python3 src/test/resources/evaluationVisualisation.py target/evaluation/autopilot/emam/clusteringResults.json target/evaluation/pacman/emam/clusteringResults.json target/evaluation/supermario/emam/clusteringResults.json target/evaluation/daimler/emam/clusteringResults.json
```
or
```bash
python3 src/test/resources/silhouetteVisualisation.py target/evaluation/autopilotSilhouette/emam/clusteringResults.json target/evaluation/pacmanSilhouette/emam/clusteringResults.json target/evaluation/supermarioSilhouette/emam/clusteringResults.json target/evaluation/daimlerSilhouette/emam/clusteringResults.json
```

## Defining the connection between a component and the middleware
Alexander David Hellwig's avatar
Alexander David Hellwig committed
180
The connection between middleware and the component is defined as tags on Ports in .tag files.
181
### Example with ROS Middleware:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
182
183
Tags of the type RosConnection can either be simple tags(see Example 3) or define a topic(http://wiki.ros.org/Topics) with name, type and optional msgField(http://wiki.ros.org/msg , 2.)
Examples:
184
185
186
1. [src/test/resources/tests/a/Add.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/Add.tag)
1. [src/test/resources/tests/a/Echo.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/Echo.tag)
1. [src/test/resources/tests/dist/SimpleDist.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/dist/SimpleDist.tag)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
187

188
189
#### Use-case 1: Creating 1 executable
Look at [GenerationTest::testMiddlewareGenerator](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/java/de/monticore/lang/monticar/generator/middleware/GenerationTest.java). The component is defined in [src/test/resources/tests/a/AddComp.emam](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/AddComp.emam) and the tags for the connection to ros are defined in [src/test/resources/tests/a/Add.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/Add.tag)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
190

191
192
#### Use-case 2: Creating multiple executables for distributed systems
Look at [GenerationTest::testDistributedTargetGenerator](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/java/de/monticore/lang/monticar/generator/middleware/GenerationTest.java). The component is defined in [src/test/resources/dist/DistComp.emam](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/dist/DistComp.emam) and the tags for the connection to ros are defined in [src/test/resources/tests/dist/SimpleDist.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/dist/SimpleDist.tag)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
193

194
#### Compile and run the generated ROS Projects
Alexander David Hellwig's avatar
Alexander David Hellwig committed
195
196
197
1. install needed software:
    * ROS Kinetic(http://wiki.ros.org/kinetic/Installation)
    * CMake(https://cmake.org/)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
198
199
    * Armadillo 8 or higher( www.arma.sourceforge.net)
        * creating a copy of the library named armadillo.h might be necessary.
Alexander David Hellwig's avatar
Alexander David Hellwig committed
200
201
202
203
204
205
206
207
208
1. source your ros environment(http://wiki.ros.org/ROS/Tutorials/InstallingandConfiguringROSEnvironment , 2.)
1. a) run src/test/resources/TargetCompilation.sh from **this project's** root
1. or b) compile a single project by
    * changing to the **generated project's** root
    * create a folder build/ and change into it
    * run: cmake ../src
    * run: make
1. Start ros and the other nodes
    * minimal working example: run: roscore
ahellwig's avatar
ahellwig committed
209
1. If the project was created by a MiddlewareGenerator, run the executable(s) at build/coordinator(/<subcomp.name>)/Coordinator_<(sub)component.name>
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
2.

## Automatic Clustering
To simplify the creation of distributed systems, the generator can automatically split the model into a given number of clusters.

Supported:
* Spectral Clustering
* ...

Procedure:
* Convert the Symbol Table of a Component into a adjacency matrix
* (Add costs given by type to matrix. E.g. float is cheaper than Matrix of floats)
* Feed into ml library(e.g. [smile ml](https://github.com/haifengl/smile)) with the selected clustering algorithm
* (Compare the result of different algorithms)
* Generate Middleware tags seperating the clusters
* Feed into existing manual clustering architecture
* (generate)


229