Added installation instructions to readme, and scripts. Also a dockerfile for tensorflow.

parent eab5ae71
Pipeline #187516 failed with stages
in 16 seconds
<!-- (c) https://github.com/MontiCore/monticore -->
![pipeline](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMADL2CPP/badges/master/build.svg)
![coverage](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/generators/EMADL2CPP/badges/master/coverage.svg)
# EMADL2CPP
Generates CPP/Python code for EmbeddedMontiArcDL.
See example project [EMADL-Demo](https://git.rwth-aachen.de/thomas.timmermanns/EMADL-Demo) for more information on how the generated code can be used.
......@@ -20,7 +21,23 @@ See example project [EMADL-Demo](https://git.rwth-aachen.de/thomas.timmermanns/E
* Caffe2
* training - generated is Python code. Follow [ official instructions on Caffe2 site ](https://caffe2.ai/docs/getting-started.html?platform=ubuntu&configuration=prebuilt)
* Gluon
* See the scripts under Installation for better instructions, as an old caffe vversion is used that needs special considerations.
* Gluon
* Tensorflow
* training - generated is Python code.
* prediction - generated code is C++.
## Installation
The two bash scripts found under ... should build and install all prerequisits for all backends as of 26.09.2019.
Note that the installation might take some time (hours) and you will need some disk space (> 60GB) for all backends. Also enough RAM or a big
enough swapspace is advisable (>10GB) for the installation of the cpp part of tensorflow. This scripts were tested with a completly clean Ubuntu 16.04,
without system updates installed. Using another Ubuntu version or installing other stuff, system updates included might/ have caused problems.
If you want to install the backends with CUDA GPU support(only MXNet/Gluon and Tensorflow, the used caffe2 version does not work with GPU support anymore),
you have to install CUDA 10.0(!!), CUDNN and NCCL (Obtainable from the nvidai webpage. You can follow their instructions.) inbetween the two scripts.
Furthermore you will have to change the pip commands for mxnet and tensorflow to the respective commented out parts.
Also docker images for the cpu version of each backend are provided at ..., though some of these might be outdated.
### HowTo
1. Define a EMADL component containing architecture of a neural network and save it in a `.emadl` file. For more information on architecture language please refer to [CNNArchLang project](https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/languages/CNNArchLang). An example of NN architecture:
......
#!/bin/bash
############################ Part 2/2 ######################################
# This script should help you to get ready to do machine learning #
# at SE at RWTH, installing all backends (mxnet/gluon, tensorflow, caffe2. #
# - Jonas Eckhardt & Julian Steinsberger-Dührßen - #
############################################################################
# It is based both on instrcutions and docker files from the git
# repos as well as our own work
############################################################################
# Requirements: #
############################################################################
# -Git set up to work with RWTH Gitlab
# -Virgin Ubuntu 16.04.(No touching, no other version) and not isntalling the system updates
# -Time: Installation may need more than one hour. It may prompt you
# for your passowrd multiple times
# -Disable Screensaver: It just saves you some time
# - > 10GB RAM or a swapfile so you have ram+swap > 10GB
# - > 60GB disk space
###########################################################################
# Usage: #
###########################################################################
# chmod u+x installer_bomb.sh
# ./installer_bomb.sh
# Dont use twice. It will fill your ~/.bashrc with garbage#!/bin/bash
sudo apt install -y --no-install-recommends cuda-command-line-tools-10-0
#Install MXNet
pip install mxnet --user
#for cuda support: use mxnet-cu100 --user instead
git clone https://github.com/apache/incubator-mxnet.git mxnet-source
cd mxnet-source
git checkout tags/1.4.0
cd ..
sudo cp -r mxnet-source/include/mxnet /usr/include/mxnet
#Test mxnet
git clone git@git.rwth-aachen.de:monticore/EmbeddedMontiArc/applications/mnistcalculator.git
cd mnistcalculator
cd mxnet
./build.sh
cd ..
cd ..
#Install OpenCV
git clone https://github.com/opencv/opencv.git
cd opencv
git checkout 3.3.1
cd ..
git clone https://github.com/opencv/opencv_contrib.git
cd opencv_contrib
git checkout 3.3.1
cd ..
cd opencv
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local \
-D INSTALL_C_EXAMPLES=ON \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D WITH_TBB=ON \
-D WITH_V4L=ON \
-D WITH_QT=ON \
-D WITH_OPENGL=ON \
-D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules \
-D BUILD_EXAMPLES=ON ..
make -j$(nproc)
sudo make install
sudo sh -c 'echo "/usr/local/lib" >> /etc/ld.so.conf.d/opencv.conf'
sudo ldconfig
#Install Tensorflow
pip install --upgrade tensorflow --user
#for cuda support: pip install --upgrade tensorflow-gpu --user
git clone https://github.com/FloopCZ/tensorflow_cc.git
#Install bazel
sudo wget -P /tmp https://github.com/bazelbuild/bazel/releases/download/0.21.0/bazel-0.21.0-installer-linux-x86_64.sh
sudo chmod +x /tmp/bazel-0.21.0-installer-linux-x86_64.sh
sudo /tmp/bazel-0.21.0-installer-linux-x86_64.sh
sudo rm /tmp/bazel-0.21.0-installer-linux-x86_64.sh
sudo apt-get -y clean
sudo updatedb
cd tensorflow_cc/tensorflow_cc/
mkdir build
cd build
cmake -DTENSORFLOW_STATIC=OFF -DTENSORFLOW_SHARED=ON ..
make
sudo rm -rf /home/tensorflow_cc/.cache
make install
cd ..
sudo rm -rf build
\ No newline at end of file
#!/bin/bash
############################ Part 1/2 ######################################
# This script should help you to get ready to do machine learning #
# at SE at RWTH, installing all backends (mxnet/gluon, tensorflow, caffe2. #
# - Jonas Eckhardt & Julian Steinsberger-Dührßen - #
############################################################################
# It is based both on instrcutions and docker files from the git
# repos as well as our own work
############################################################################
# Requirements: #
############################################################################
# -Git set up to work with RWTH Gitlab
# -Virgin Ubuntu 16.04.(No touching, no other version) and not isntalling the system updates
# -Time: Installation may need more than one hour. It may prompt you
# for your passowrd multiple times
# -Disable Screensaver: It just saves you some time
# - > 10GB RAM or a swapfile so you have ram+swap > 10GB
# - > 60GB disk space
###########################################################################
# Usage: #
###########################################################################
# chmod u+x installer_bomb.sh
# ./installer_bomb.sh
# Dont use twice. It will fill your ~/.bashrc with garbage
sudo apt update
sudo apt upgrade -y
sudo apt install -y --no-install-recommends maven libopencv-dev\
python-pip openjdk-8-jdk git wget python python3-numpy gcc build-essential cmake \
liblapack-dev libblas-dev libopenblas-dev libboost-dev libarmadillo-dev build-essential \
libgoogle-glog-dev libgtest-dev libiomp-dev libleveldb-dev liblmdb-dev \
libopenmpi-dev libsnappy-dev libprotobuf-dev openmpi-bin openmpi-doc python-numpy swig
sudo apt install -y --no-install-recommends protobuf-compiler python-dev python-setuptools \
libgflags-dev checkinstall pkg-config yasm gfortran libjpeg8-dev libjasper-dev libpng12-dev \
libtiff5-dev libavcodec-dev libavformat-dev libswscale-dev libdc1394-22-dev libxine2-dev \
libgstreamer-plugins-base0.10-dev qt5-default libgtk2.0-dev libtbb-dev libatlas-base-dev \
libfaac-dev libmp3lame-dev libtheora-dev libvorbis-dev libxvidcore-dev \
libopencore-amrnb-dev libopencore-amrwb-dev x264 v4l-utils libgphoto2-dev libeigen3-dev \
libhdf5-dev doxygen curl unzip autoconf autogen libtool mlocate zlib1g-dev g++ sudo zip automake
#Fix JDK version
sudo update-alternatives --config java
pip install --upgrade pip --user
pip install numpy opencv-python six lmdb graphviz pyyaml future typing --user
pip install --no-deps h5py --user
#prepare path variables
echo "export PATH=\$PATH:\"/usr/local/lib\"" >> ~/.bashrc
echo "export PYTHON_PATH=\"/usr/local:$(pwd)/pytorch/build:/usr/bin/python:/usr/local/lib\"">> ~/.bashrc
echo "export LD_LIBRARY_PATH=\"/usr/local/lib\"">> ~/.bashrc
source ~/.bashrc
#Install caffee2
git clone --recursive https://github.com/pytorch/pytorch.git
cd pytorch
git checkout v0.4.0
git -c submodule.'third_party/nervanagpu'.update=none submodule update --init
sudo rm -r build
mkdir build
cd build
cmake -DUSE_MPI=OFF ..
sudo make clean install
cd ..
sudo FULL_CAFFE2=1 python setup.py clean install
sudo cp -r /usr/local/lib/libcaffe2.so /usr/lib/
cd ..
#Fix protocbuf
pip2 install protobuf==3.5.1 --user
sudo snap install --classic notepadqq
sudo snap install pycharm-community --classic
\ No newline at end of file
FROM floopcz/tensorflow_cc:ubuntu-shared
#Install everything needed with apt
RUN apt-get update
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get install -y --no-install-recommends \
tzdata \
git \
wget python gcc \
build-essential cmake \
maven unzip\
python3 python3-pip python-dev python-setuptools python python-pip\
libopencv-dev libarmadillo-dev
#Fix Java version
RUN apt remove -y *openjdk*
RUN apt-get install -y --no-install-recommends openjdk-8-jdk
#Install stuff for python and python3
RUN pip install tensorflow==1.13.1
RUN pip3 install opencv-python h5py numpy matplotlib scipy Pillow
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment