Commit 40609548 authored by Evgeny Kusmenko's avatar Evgeny Kusmenko

Merge branch 'ML_clustering' into 'master'

Ml clustering

See merge request !34
parents 17d1c533 399ec45c
Pipeline #117559 passed with stages

Too many changes to show.

To preserve performance only 1000 of 1000+ files are displayed.

# Java Maven CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-java/ for more details
#
version: 2
general:
branches:
ignore:
- gh-pages
jobs:
build:
docker:
# specify the version you desire here
- image: circleci/openjdk:8-jdk
cmd: ["/bin/bash"]
# Specify service dependencies here if necessary
# CircleCI maintains a library of pre-built images
# documented at https://circleci.com/docs/2.0/circleci-images/
# - image: circleci/postgres:9.4
working_directory: ~/repo
environment:
# Customize the JVM maximum heap limit
- MAVEN_OPTS: -Xmx3200m
- ES_JAVA_OPTS=-Xms3200m -Xmx3200m
steps:
- checkout
# run tests!
- run: mvn -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -B clean install --settings "settings.xml"
workflows:
version: 2
commit-workflow:
jobs:
- build
scheduled-workflow:
triggers:
- schedule:
cron: "30 1 * * *"
filters:
branches:
only: master
jobs:
- build
......@@ -7,7 +7,8 @@ stages:
masterJobWindows:
stage: windows
script:
- mvn -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -B clean install --settings settings.xml
- call mvn -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -B dependency:purge-local-repository -DactTransitively=false --settings settings.xml
- call mvn -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -B clean install --settings settings.xml
tags:
- Windows10
......
script:
- mvn -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -B clean install cobertura:cobertura org.eluder.coveralls:coveralls-maven-plugin:report --settings "settings.xml"
after_success:
- if [ "${TRAVIS_BRANCH}" == "master" ]; then mvn -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -B deploy --debug --settings "./settings.xml"; fi
......@@ -5,6 +5,7 @@
This generator takes an EMAM or EMADL model and connects it to a middleware library. If all Ports of two connected Components are marked as middleware Ports, the generator will create 2 executables that can be deployed on different machines.
All communication of these 2 Components will then be tunneled trough the specified middleware:
![MiddlewareAdapter](/uploads/6e9c69e6b56554579551769174df3697/MiddlewareAdapter.png)
It also supports automatic clustering of the subcomponents to deploy on different machines.
## Other important documents
### Quickstart
......@@ -21,29 +22,109 @@ See [INSTALL_DEPENDENCIES.md](INSTALL_DEPENDENCIES.md)
## Usage
### CLI
Maven generates the jar `embedded-montiarc-math-middleware-generator-{Version}-jar-with-dependencies.jar`
and the cli is located in `de.monticore.lang.monticar.generator.middleware.DistributedTargetGeneratorCli`.
and the cli is located in `de.monticore.lang.monticar.generator.middleware.cli.DistributedTargetGeneratorCli`.
Parameters: `${file path to config json}` OR `-r ${raw json config string}`
```
Schema of config json:
{
'modelsDir':'<path to directory with EMAM models>',
'outputDir':'<path to output directory for generated files>',
'rootModel':'<fully qualified name of the root model>',
'generators':['<identifier for first generator>', '<identifier for second generator>',...],
'emadlBackend':'<deep-learning-framework backend. Options: MXNET, CAFFE2>'
Example: [CliUsage.sh](src/test/resources/CliUsage.sh)
An example config file with all clustering algorithms: [config](src/test/resources/config/parameterTest/clusterParamsAllAlgos.json)
| Name | Type | Required | Description |
|----------------------|--------|----------|-------------------------------------------------------------------------------------------|
| modelsDir | String | ✅ | path to directory with EMAM models |
| outputDir | String | ✅ | path to output directory for generated files |
| rootModel | String | ✅ | fully qualified name of the root model |
| generators | List | ✅ | List of generator identfiers<br> 'cpp', 'emadlcpp', 'roscpp', 'rclcpp' |
| emadlBackend | String | ❓ | deep-learning-framework backend<br> 'MXNET'(Default), 'CAFFE2' |
| writeTagFile | Bool | ❓ | Writes a .tag file with all Middleware tags into the generated code<br> Defaults to false |
| clusteringParameters | Object | ❓ | Options to cluster the component before generating<br> See below |
Clustering Parameters:
| Name | Type | Required | Description |
|---------------------|--------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| numberOfClusters | int | ❓ | Number of clusters the subcomponents should be divided into<br> Overrides numberOfClusters in algorithmParameters |
| flatten | bool | ❓ | Replace all components with their subcomponents execpt when it is atomic or the flatten level is reached |
| flattenLevel | int | ❓ | Maximal level of component flattening |
| metric | String | ❓ | Metric to evaluate the quality of the resulting clusters. Available: "CommunicationCost"(Default), "Silhouette"|
| chooseBy | String | ❓ | Strategy to choose from the resulting clusterings<br> bestWithFittingN(Default): if numberOfClusters is set, all results with a different number of clusters are ignored<br> bestOverall: ignore numberOfClusters, choose result with best score |
| algorithmParameters | List<Object> | ❓ | Used to specify which algorithms(and their parameters) are used for clustering |
There are 4 different Clustering Algorithms with distinct parameters
Every parameter of the clustering algorithms can be dynamic, enabling automatic search for the best values. Available are lists and generators as seen in the example below:
```json
"sigma":[1,2,3]
"sigma":{
"min":1,
"max":3,
"step":1
}
"sigma":{
"min":1,
"max":3,
"count":3
}
```
Generator Options:
- Behaviour generators:
- 'cpp': EMAM2CPP
- 'emadlcpp': EMADL2CPP
- Middleware generators:
- 'roscpp': EMAM2Roscpp
Example: [CliUsage.sh](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/src/test/resources/CliUsage.sh)
### Defining the connection between a component and the middleware
Also see [clusterDynamic.json](src/test/resources/config/parameterTest/clusterDynamic.json) and [clusterDynamicList.json](src/test/resources/config/parameterTest/clusterDynamicList.json)
Spectral Clustering:
| Name | Type | Required | Description |
|------------------|--------|----------|---------------------------------------------------------------------------------|
| name | String | ✅️ | must equal "SpectralClustering" |
| numberOfClusters | int | ✅️ | Number of clusters that are created<br> Overwritten by global numberOfClusters |
| l | int | ❓ | |