README.md 13.2 KB
Newer Older
1
# EMAM2Middleware
Alexander David Hellwig's avatar
Alexander David Hellwig committed
2 3
![pipeline](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/badges/master/build.svg)
![coverage](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/badges/master/coverage.svg)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
4 5 6 7 8 9
## Purpose
This generator takes an EMAM or EMADL model and connects it to a middleware library. If all Ports of two connected Components are marked as middleware Ports, the generator will create 2 executables that can be deployed on different machines.
All communication of these 2 Components will then be tunneled trough the specified middleware:
![MiddlewareAdapter](/uploads/6e9c69e6b56554579551769174df3697/MiddlewareAdapter.png)


10 11 12
## Writing your own Middleware Generator
see [TUTORIAL_ADD_MIDDLEWARE.md](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/TUTORIAL_ADD_MIDDLEWARE.md)

Alexander David Hellwig's avatar
Alexander David Hellwig committed
13
## Dependencies needed to compile the generated projects
Alexander David Hellwig's avatar
Alexander David Hellwig committed
14
### All generated projects
Alexander David Hellwig's avatar
Alexander David Hellwig committed
15
CMake, Make,a C++ compiler, and Armadillo are required to compile the generated projects.
Alexander David Hellwig's avatar
Alexander David Hellwig committed
16 17 18 19 20 21
#### Linux
Gcc is recommended as the C++ compiler.
Example install of all needed packages for ubuntu:
```bash
sudo apt install gcc cmake make
```
22
Then download Armadillo from here: [Linux](https://rwth-aachen.sciebo.de/s/igDWzLpdO5zYHBj/download?path=%2Fubuntu%2F18.10.24-armadillo-linux&files=armadillo-8.500.1-linux.zip) and set the environment variable `Armadillo_HOME` to the base dir of your installation.
Alexander David Hellwig's avatar
Alexander David Hellwig committed
23

Alexander David Hellwig's avatar
Alexander David Hellwig committed
24
To check everything is installed correctly use whereis/ls:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
25 26 27 28 29 30 31
```bash
$ whereis g++
g++: /usr/bin/g++
$ whereis cmake
cmake: /usr/bin/cmake
$ whereis make
make: /usr/bin/make
Alexander David Hellwig's avatar
Alexander David Hellwig committed
32
$ ls "$Armadillo_HOME/include"
Alexander David Hellwig's avatar
Alexander David Hellwig committed
33 34 35
armadillo_bits armadillo.h
```

Alexander David Hellwig's avatar
Alexander David Hellwig committed
36
Compiling a generated project:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
37
```bash
Alexander David Hellwig's avatar
Alexander David Hellwig committed
38
cd /path/to/build/directory
Alexander David Hellwig's avatar
Alexander David Hellwig committed
39 40 41
cmake /path/to/generated/project/source
make
```
Alexander David Hellwig's avatar
Alexander David Hellwig committed
42

Alexander David Hellwig's avatar
Alexander David Hellwig committed
43
#### Windows
44
Mingw64 gcc is recommended as the C++ compiler. Download a version with all needed tools from [here](https://rwth-aachen.sciebo.de/s/igDWzLpdO5zYHBj/download?path=%2Fwin64&files=mingw64.zip).
Alexander David Hellwig's avatar
Alexander David Hellwig committed
45

Alexander David Hellwig's avatar
Alexander David Hellwig committed
46
CMake for Windows: https://cmake.org/download/
Alexander David Hellwig's avatar
Alexander David Hellwig committed
47

Alexander David Hellwig's avatar
Alexander David Hellwig committed
48 49
Make for Windows: http://gnuwin32.sourceforge.net/packages/make.htm

50 51
Add the bin directories of all 3 to your PATH variable.

Alexander David Hellwig's avatar
Alexander David Hellwig committed
52 53
Then download Armadillo from here: [Windows](https://rwth-aachen.sciebo.de/s/igDWzLpdO5zYHBj/download?path=%2Fwin64&files=armadillo-8.200.2.zip) and set the environment variable `Armadillo_HOME` to the base dir of your installation.
To check everything is installed correctly use where/dir:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
54
```batch
Alexander David Hellwig's avatar
Alexander David Hellwig committed
55 56 57 58 59 60
> where g++
C:\mingw64\bin\g++.exe
> where cmake
C:\Program Files\CMake\bin\cmake.exe
> where make
C:\Program Files\make-3.81-bin\bin\make.exe
61
> dir /b "%Armadillo_HOME%\include"
Alexander David Hellwig's avatar
Alexander David Hellwig committed
62 63 64 65 66 67 68 69
armadillo
armadillo.h
...
```

Compiling a generated project:
```batch
cd C:\path\to\build\directory
Alexander David Hellwig's avatar
Alexander David Hellwig committed
70
cmake C:\path\to\generated\project\source -G "MinGW Makefiles"
Alexander David Hellwig's avatar
Alexander David Hellwig committed
71 72
make
```
Alexander David Hellwig's avatar
Alexander David Hellwig committed
73

Michael Günther Beyer's avatar
Michael Günther Beyer committed
74 75
Please note: It is highly recommended, you stick to the exact versions as stated above. Otherwise you might run into trouble regarding the interplay between cmake/make and the Armadillo library. In particular problems have been reported using Cygwin.

Alexander David Hellwig's avatar
Alexander David Hellwig committed
76
### Projects with roscpp generator
Alexander David Hellwig's avatar
Alexander David Hellwig committed
77 78 79
Only for generated projects that contain a ROS adapter(e.g. -g=cpp,roscpp).
ROS Kinetic currently only supports Linux and the installation is described [here](http://wiki.ros.org/kinetic/Installation/Ubuntu).

ahellwig's avatar
ahellwig committed
80
## Usage
81 82
### CLI
Maven generates the jar `embedded-montiarc-math-middleware-generator-{Version}-jar-with-dependencies.jar`
83
and the cli is located in `de.monticore.lang.monticar.generator.middleware.cli.DistributedTargetGeneratorCli`.
84

85
Parameters: `${file path to config json}` OR `-r ${raw json config string}`
86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107

Example: [CliUsage.sh](src/test/resources/CliUsage.sh)

An example config file with all clustering algorithms: [config](src/test/resources/config/parameterTest/clusterParamsAllAlgos.json)

| Name                 | Type   | Required | Description                                                                               |
|----------------------|--------|----------|-------------------------------------------------------------------------------------------|
| modelsDir            | String |     ✅    | path to directory with EMAM models                                                        |
| outputDir            | String |     ✅    | path to output directory for generated files                                              |
| rootModel            | String |     ✅    | fully qualified name of the root model                                                    |
| generators           | List   |     ✅    | List of generator identfiers<br> 'cpp', 'emadlcpp', 'roscpp', 'rclcpp'                    |
| emadlBackend         | String |     ❓    | deep-learning-framework backend<br> 'MXNET'(Default), 'CAFFE2'                                     |
| writeTagFile         | Bool   |     ❓    | Writes a .tag file with all Middleware tags into the generated code<br> Defaults to false |
| clusteringParameters | Object |     ❓    | Options to cluster the component before generating<br> See below                          |

Clustering Parameters:

| Name                | Type         | Required | Description                                                                                                                                                                                                                                       |
|---------------------|--------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| numberOfClusters    | int          | ❓        | Number of clusters the subcomponents should be divided into<br> Overrides numberOfClusters in algorithmParameters                                                                                                                                 |
| flatten             | bool         | ❓        | Replace all components with their subcomponents execpt when it is atomic or the flatten level is reached                                                                                                                                          |
| flattenLevel        | int          | ❓        | Maximal level of component flattening                                                                                                                                                                                                             |
108 109
| metric            | String       | ❓        | Metric to evaluate the quality of the resulting clusters. Available: "CommunicationCost"(Default), "Silhouette"|
| chooseBy            | String       | ❓        | Strategy to choose from the resulting clusterings<br> bestWithFittingN(Default): if numberOfClusters is set, all results with a different number of clusters are ignored<br> bestOverall: ignore numberOfClusters, choose result with best score |
110 111 112 113
| algorithmParameters | List<Object> | ❓        | Used to specify which algorithms(and their parameters) are used for clustering                                                                                                                                                                    |

There are 4 different Clustering Algorithms with distinct parameters

114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129
Every parameter of the clustering algorithms can be dynamic, enabling automatic search for the best values. Available are lists and generators as seen in the example below:
```json
"sigma":[1,2,3]
"sigma":{
  "min":1,
  "max":3,
  "step":1
}
"sigma":{
  "min":1,
  "max":3,
  "count":3
}
```
Also see [clusterDynamic.json](src/test/resources/config/parameterTest/clusterDynamic.json) and [clusterDynamicList.json](src/test/resources/config/parameterTest/clusterDynamicList.json)

130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162
Spectral Clustering:

| Name             | Type   | Required | Description                                                                     |
|------------------|--------|----------|---------------------------------------------------------------------------------|
| name             | String | ✅️        | must equal "SpectralClustering"                                                 |
| numberOfClusters | int    | ✅️        | Number of clusters that are created<br> Overwritten by global numberOfClusters  |
| l                | int    | ❓        |                                                                                 |
| sigma            | double | ❓        |                                                                                 |

DBScan:

| Name    | Type   | Required | Description         |
|---------|--------|----------|---------------------|
| name    | String | ✔️        | must equal "DBScan" |
| min_pts | int    | ✔️        |                     |
| radius  | double | ✔️        |                     |

Markov:

| Name         | Type   | Required | Description         |
|--------------|--------|----------|---------------------|
| name         | String | ✔️     | must equal "Markov" |
| max_residual | double | ❓        |                     |
| gamma_exp    | double | ❓        |                     |
| loop_gain    | double | ❓        |                     |
| zero_max     | double | ❓        |                     |

Affinity Propagation:

| Name | Type   | Required | Description                      |
|------|--------|----------|----------------------------------|
| name | String | ✔️        | must equal "AffinityPropagation" |

163 164
    

ahellwig's avatar
ahellwig committed
165
### Defining the connection between a component and the middleware
Alexander David Hellwig's avatar
Alexander David Hellwig committed
166
The connection between middleware and the component is defined as tags on Ports in .tag files.
167
### Example with ROS Middleware:
Alexander David Hellwig's avatar
Alexander David Hellwig committed
168 169
Tags of the type RosConnection can either be simple tags(see Example 3) or define a topic(http://wiki.ros.org/Topics) with name, type and optional msgField(http://wiki.ros.org/msg , 2.)
Examples:
170 171 172
1. [src/test/resources/tests/a/Add.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/Add.tag)
1. [src/test/resources/tests/a/Echo.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/Echo.tag)
1. [src/test/resources/tests/dist/SimpleDist.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/dist/SimpleDist.tag)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
173

174 175
#### Use-case 1: Creating 1 executable
Look at [GenerationTest::testMiddlewareGenerator](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/java/de/monticore/lang/monticar/generator/middleware/GenerationTest.java). The component is defined in [src/test/resources/tests/a/AddComp.emam](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/AddComp.emam) and the tags for the connection to ros are defined in [src/test/resources/tests/a/Add.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/a/Add.tag)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
176

177 178
#### Use-case 2: Creating multiple executables for distributed systems
Look at [GenerationTest::testDistributedTargetGenerator](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/java/de/monticore/lang/monticar/generator/middleware/GenerationTest.java). The component is defined in [src/test/resources/dist/DistComp.emam](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/dist/DistComp.emam) and the tags for the connection to ros are defined in [src/test/resources/tests/dist/SimpleDist.tag](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/tests/dist/SimpleDist.tag)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
179

180
#### Compile and run the generated ROS Projects
Alexander David Hellwig's avatar
Alexander David Hellwig committed
181 182 183
1. install needed software:
    * ROS Kinetic(http://wiki.ros.org/kinetic/Installation)
    * CMake(https://cmake.org/)
Alexander David Hellwig's avatar
Alexander David Hellwig committed
184 185
    * Armadillo 8 or higher( www.arma.sourceforge.net)
        * creating a copy of the library named armadillo.h might be necessary.
Alexander David Hellwig's avatar
Alexander David Hellwig committed
186 187 188 189 190 191 192 193 194
1. source your ros environment(http://wiki.ros.org/ROS/Tutorials/InstallingandConfiguringROSEnvironment , 2.)
1. a) run src/test/resources/TargetCompilation.sh from **this project's** root
1. or b) compile a single project by
    * changing to the **generated project's** root
    * create a folder build/ and change into it
    * run: cmake ../src
    * run: make
1. Start ros and the other nodes
    * minimal working example: run: roscore
ahellwig's avatar
ahellwig committed
195
1. If the project was created by a MiddlewareGenerator, run the executable(s) at build/coordinator(/<subcomp.name>)/Coordinator_<(sub)component.name>
196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214
2.

## Automatic Clustering
To simplify the creation of distributed systems, the generator can automatically split the model into a given number of clusters.

Supported:
* Spectral Clustering
* ...

Procedure:
* Convert the Symbol Table of a Component into a adjacency matrix
* (Add costs given by type to matrix. E.g. float is cheaper than Matrix of floats)
* Feed into ml library(e.g. [smile ml](https://github.com/haifengl/smile)) with the selected clustering algorithm
* (Compare the result of different algorithms)
* Generate Middleware tags seperating the clusters
* Feed into existing manual clustering architecture
* (generate)


215