Skip to content

GitLab

E
EMAM2Middleware
Commit 6fa1b264 authored by Alexander David Hellwig's avatar Alexander David Hellwig
Browse files
Options

Merge branch 'ML_clustering' of... 

Merge branch 'ML_clustering' of https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware into ML_clustering
parents aebebd66 7280b84a
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 52 additions and 33612 deletions
.circleci/config.yml deleted 100644 → 0
View file @ aebebd66
# Java Maven CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-java/ for more details
#
version: 2
general:
branches:
ignore:
- gh-pages
jobs:
build:
docker:
# specify the version you desire here
- image: circleci/openjdk:8-jdk
cmd: ["/bin/bash"]
# Specify service dependencies here if necessary
# CircleCI maintains a library of pre-built images
# documented at https://circleci.com/docs/2.0/circleci-images/
# - image: circleci/postgres:9.4
working_directory: ~/repo
environment:
# Customize the JVM maximum heap limit
- MAVEN_OPTS: -Xmx3200m
- ES_JAVA_OPTS=-Xms3200m -Xmx3200m
steps:
- checkout
# run tests!
- run: mvn -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -B clean install --settings "settings.xml"
workflows:
version: 2
commit-workflow:
jobs:
- build
scheduled-workflow:
triggers:
- schedule:
cron: "30 1 * * *"
filters:
branches:
only: master
jobs:
- build
.travis.yml deleted 100644 → 0
View file @ aebebd66
script:
- mvn -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -B clean install cobertura:cobertura org.eluder.coveralls:coveralls-maven-plugin:report --settings "settings.xml"
after_success:
- if [ "${TRAVIS_BRANCH}" == "master" ]; then mvn -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -B deploy --debug --settings "./settings.xml"; fi
README.md
View file @ 6fa1b264
......@@ -5,7 +5,7 @@
This generator takes an EMAM or EMADL model and connects it to a middleware library. If all Ports of two connected Components are marked as middleware Ports, the generator will create 2 executables that can be deployed on different machines.
All communication of these 2 Components will then be tunneled trough the specified middleware:
![MiddlewareAdapter](/uploads/6e9c69e6b56554579551769174df3697/MiddlewareAdapter.png)
It also supports automatic clustering of the subcomponents to deploy on different machines.
## Writing your own Middleware Generator
see [TUTORIAL_ADD_MIDDLEWARE.md](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMAM2Middleware/blob/master/TUTORIAL_ADD_MIDDLEWARE.md)
......@@ -215,21 +215,4 @@ Look at [GenerationTest::testDistributedTargetGenerator](https://git.rwth-aachen
1. If the project was created by a MiddlewareGenerator, run the executable(s) at build/coordinator(/<subcomp.name>)/Coordinator_<(sub)component.name>
2.
## Automatic Clustering
To simplify the creation of distributed systems, the generator can automatically split the model into a given number of clusters.
Supported:
* Spectral Clustering
* ...
Procedure:
* Convert the Symbol Table of a Component into a adjacency matrix
* (Add costs given by type to matrix. E.g. float is cheaper than Matrix of floats)
* Feed into ml library(e.g. [smile ml](https://github.com/haifengl/smile)) with the selected clustering algorithm
* (Compare the result of different algorithms)
* Generate Middleware tags seperating the clusters
* Feed into existing manual clustering architecture
* (generate)
pom.xml
View file @ 6fa1b264
......@@ -24,6 +24,7 @@
<Embedded-montiarc-math-roscpp-generator.version>0.1.1-SNAPSHOT</Embedded-montiarc-math-roscpp-generator.version>
<EMADL.version>0.2.4</EMADL.version>
<EMADL2CPP.version>0.2.6-SNAPSHOT</EMADL2CPP.version>
<embedded-montiarc-component-clustering.version>0.0.1-SNAPSHOT</embedded-montiarc-component-clustering.version>
<!-- .. Libraries .................................................. -->
<guava.version>18.0</guava.version>
......@@ -64,36 +65,12 @@
<version>${gson.version}</version>
</dependency>
<dependency>
<groupId>net.sourceforge</groupId>
<artifactId>javaml</artifactId>
<version>0.1.5</version>
</dependency>
<!--dependency>
<groupId>be.abeel</groupId>
<artifactId>ajt</artifactId>
<version>2.9</version>
</dependency-->
<dependency>
<groupId>com.github.haifengl</groupId>
<artifactId>smile-core</artifactId>
<version>1.5.2</version>
</dependency>
<dependency>
<groupId>com.github.haifengl</groupId>
<artifactId>smile-plot</artifactId>
<version>1.5.2</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<!--<dependency>
<groupId>com.clust4j</groupId>
<artifactId>clust4j</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>-->
<dependency>
<groupId>org.jgrapht</groupId>
<artifactId>jgrapht-core</artifactId>
......@@ -128,13 +105,6 @@
<version>${Embedded-montiarc-math-roscpp-generator.version}</version>
</dependency>
<dependency>
<groupId>org.graphstream</groupId>
<artifactId>gs-core</artifactId>
<version>1.3</version>
<optional>false</optional>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-math</artifactId>
......@@ -154,6 +124,12 @@
<version>${EMADL2CPP.version}</version>
</dependency>
<dependency>
<groupId>de.monticore.lang.monticar</groupId>
<artifactId>embedded-montiarc-component-clustering</artifactId>
<version>${embedded-montiarc-component-clustering.version}</version>
</dependency>
<!-- MontiCore Dependencies -->
<dependency>
<groupId>de.se_rwth.commons</groupId>
......@@ -183,16 +159,6 @@
</dependencies>
<repositories>
<repository>
<id>java-ds</id>
<!--
<url>https://raw.githubusercontent.com/alexeygrigorev/maven-repo/master/</url>
-->
<url>https://raw.githubusercontent.com/mgbeyer/maven-repo/master/</url>
</repository>
</repositories>
<!-- == PROJECT BUILD SETTINGS =========================================== -->
<build>
......
src/main/java/com/clust4j/Clust4j.java deleted 100644 → 0
View file @ aebebd66
/*******************************************************************************
* Copyright 2015, 2016 Taylor G Smith
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*******************************************************************************/
package com.clust4j;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
/**
* The absolute super type for all clust4j objects (models and datasets)
* that should be able to commonly serialize their data.
* @author Taylor G Smith
*/
public abstract class Clust4j implements java.io.Serializable {
private static final long serialVersionUID = -4522135376738501625L;
/**
* Load a model from a FileInputStream
* @param fos
* @return
* @throws IOException
* @throws ClassNotFoundException
*/
public static Clust4j loadObject(final FileInputStream fis) throws IOException, ClassNotFoundException {
ObjectInputStream in = null;
Clust4j bm = null;
try {
in = new ObjectInputStream(fis);
bm = (Clust4j) in.readObject();
} finally {
if(null != in)
in.close();
fis.close();
}
return bm;
}
/**
* Save a model to FileOutputStream
* @param fos
* @throws IOException
*/
public void saveObject(final FileOutputStream fos) throws IOException {
ObjectOutputStream out = null;
try {
out = new ObjectOutputStream(fos);
out.writeObject(this);
} finally {
if(null != out)
out.close();
fos.close();
}
}
}
src/main/java/com/clust4j/GlobalState.java deleted 100644 → 0
View file @ aebebd66
/*******************************************************************************
* Copyright 2015, 2016 Taylor G Smith
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*******************************************************************************/
package com.clust4j;
import java.util.Random;
import java.util.concurrent.ForkJoinPool;
import org.apache.commons.math3.util.FastMath;
/**
* A set of global config values used in multiple classes. Some values may
* be set to the user's preference, while others are final.
*
* @author Taylor G Smith
*/
public abstract class GlobalState {
/** The default random state */
public final static Random DEFAULT_RANDOM_STATE = new Random(999);
public final static int MAX_ARRAY_SIZE = 25_000_000;
/**
* Holds static mathematical values
* @author Taylor G Smith
*/
public static abstract class Mathematics {
/** Double.MIN_VALUE is not negative; this is */
public final static double SIGNED_MIN = Double.NEGATIVE_INFINITY;
public final static double MAX = Double.POSITIVE_INFINITY;
public final static double TINY = 2.2250738585072014e-308;
public final static double EPS = 2.2204460492503131e-16;
/*===== Gamma function assistants =====*/
public final static double LOG_PI = FastMath.log(Math.PI);
public final static double LOG_2PI = FastMath.log(2 * Math.PI);
public final static double ROOT_2PI= FastMath.sqrt(2 * Math.PI);
/** Euler's Gamma constant */
public final static double GAMMA = 0.577215664901532860606512090;
public final static double HALF_LOG2_PI = 0.91893853320467274178032973640562;
final static double[] GAMMA_BOUNDS = new double[]{0.001, 12.0};
final static double HIGH_BOUND = 171.624;
/** numerator coefficients for approximation over the interval (1,2) */
private final static double[] p = new double[]{
-1.71618513886549492533811E+0,
2.47656508055759199108314E+1,
-3.79804256470945635097577E+2,
6.29331155312818442661052E+2,
8.66966202790413211295064E+2,
-3.14512729688483675254357E+4,
-3.61444134186911729807069E+4,
6.64561438202405440627855E+4
};
/** denominator coefficients for approximation over the interval (1,2) */
private final static double[] q = new double[]{
-3.08402300119738975254353E+1,
3.15350626979604161529144E+2,
-1.01515636749021914166146E+3,
-3.10777167157231109440444E+3,
2.25381184209801510330112E+4,
4.75584627752788110767815E+3,
-1.34659959864969306392456E+5,
-1.15132259675553483497211E+5
};
/**
* Abramowitz and Stegun 6.1.41
* Asymptotic series should be good to at least 11 or 12 figures
* For error analysis, see Whittiker and Watson
* A Course in Modern Analysis (1927), page 252
*/
private final static double[] c = new double[]{
1.0/12.0,
-1.0/360.0,
1.0/1260.0,
-1.0/1680.0,
1.0/1188.0,
-691.0/360360.0,
1.0/156.0,
-3617.0/122400.0
};
// Any assertion failures will cause exception to be thrown right away
static {
// These should never change
assert GAMMA_BOUNDS.length == 2;
assert p.length == 8;
assert p.length == q.length;
assert c.length == p.length;
}
/**
* Adapted from sklearn_gamma, which was in turn adapted from
* John D. Cook's public domain version of lgamma, from
* http://www.johndcook.com/stand_alone_code.html
* @param x
* @return
*/
public static double gamma(double x) {
if(x <= 0)
throw new IllegalArgumentException("x must exceed 0");
// Check if in first boundary
int boundaryIdx = 0;
if(x < GAMMA_BOUNDS[boundaryIdx++])
return 1.0 / (x * (1.0 + GAMMA * x));
if(x < GAMMA_BOUNDS[boundaryIdx++]) {
double den = 1.0, num = 0.0, res, z, y = x;
int i, n = 0;
boolean lt1 = y < 1.0;
if(lt1)
y += 1.0;
else {
n = ((int)y) - 1;
y -= n;
}
z = y - 1;
for(i = 0; i < p.length; i++) {
num = (num + p[i]) * z;
den = den * z + q[i];
}
res = num/den + 1.0;
// Correction if arg was not initially in (1,2)
if(lt1)
res /= (y - 1.0);
else {
for(i = 0; i < n; i++, y++)
res *= y;
}
return res;
}
if(x > HIGH_BOUND)
return Double.POSITIVE_INFINITY;
return FastMath.exp(lgamma(x));
}
public static double lgamma(double x) {
if(x <= 0)
throw new IllegalArgumentException("x must exceed 0");
double z, sum;
int i;
if(x < GAMMA_BOUNDS[1])
return FastMath.log(FastMath.abs(gamma(x)));
z = 1.0/ (x * x);
sum = c[7];
for(i = 6; i >= 0; i--) {
sum *= z;
sum += c[i];
}
return (x - 0.5) * FastMath.log(x) - x + HALF_LOG2_PI + sum / x;
}
}
/**
* A class to hold configurations for parallelism
* @author Taylor G Smith
*/
public abstract static class ParallelismConf {
/**
* Matrices with number of elements exceeding this number
* will automatically trigger parallel events as supported
* in clustering methods.
*/
public static final int MIN_ELEMENTS = 15000;
/**
* The minimum number of cores to efficiently
* allow parallel operations.
*/
public static final int MIN_PARALLEL_CORES_RECOMMENDED = 8;
/**
* The minimum number of required cores to allow any
* parallelism at all.
*/
public static final int MIN_CORES_REQUIRED = 4;
/**
* The number of available cores on the machine. Used for determining
* whether or not to use parallelism & how large parallel chunks should be. */
public static final int NUM_CORES = Runtime.getRuntime().availableProcessors();
/**
* Whether to allow parallelism at all or quietly force serial jobs where necessary
*/
public static boolean PARALLELISM_ALLOWED = NUM_CORES >= MIN_CORES_REQUIRED;
/**
* Whether parallelization is recommended for this machine.
* Default value is true if availableProcessors is at least 8.
*/
public static final boolean PARALLELISM_RECOMMENDED = NUM_CORES >= MIN_PARALLEL_CORES_RECOMMENDED;
/** If true and the size of the vector exceeds {@value #MAX_SERIAL_VECTOR_LEN},
* auto schedules parallel job for applicable operations. This can slow
* things down on machines with a lower core count, but speed them up
* on machines with a higher core count. More heap space may be required.
* Defaults to {@link #PARALLELISM_RECOMMENDED}
*/
public static boolean ALLOW_AUTO_PARALLELISM = PARALLELISM_RECOMMENDED;
/**
* The global ForkJoin thread pool for parallel recursive tasks. */
final static public ForkJoinPool FJ_THREADPOOL = new ForkJoinPool();
/**
* The max length a vector may be before defaulting to a parallel process, if applicable */
static public int MAX_SERIAL_VECTOR_LEN = 10_000_000;
/**
* The max length a parallel-processed chunk may be */
public static int MAX_PARALLEL_CHUNK_SIZE = MAX_SERIAL_VECTOR_LEN / NUM_CORES; //2_500_000;
}
}
src/main/java/com/clust4j/LICENSE.txt deleted 100644 → 0
View file @ aebebd66
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
2. Grant of Copyright License.
Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
3. Grant of Patent License.
Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
4. Redistribution.
You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: