MeLOn
Loading...
Searching...
No Matches
Feed forward neural network

Training

MeLOn supports two options for training feed forward neural networks:

  • Using python with the keras interface for training neural networks with tensorflow
  • Using Matlab and its included Machine Learning Toolbox

It is recommended that you rely on the provided example scripts for training your model. The reason for this is that MeLOn doesn't support all function included in keras or the Matlab Machine Learning Toolbox and by using the examples you can assure to generate models which are compatible with the MeLOn functionalities.

Python

Requirements

  • Python (>=3.6)
  • tensorflow (>=1.14.0)
  • tensorflow-model-optimization (>=0.50)
  • scikit-learn (>=0.23.2)
  • numpy (>=1.19.4)
  • matplotlib (>=3.3.3)

Using the example scripts

The two training script included provide you with the following options:

  • Train a simple dense network
  • Train a network and perform pruning on its layers to create sparser ones. This can boost the performance of your optimization when you are using the model in your MAiNGO problem formulation later.

To use the scripts on your data you have to provide it in a .csv file where each row contains a data point with the first columns containing the values of the input and the following columns containing the ones of the output.

Example:

#input1,input2,ouput1
-3,-3,6.6713e-05
-2.8776,-3,0.0001249
-2.7551,-3,0.00022533
-2.6327,-3,0.00039126
-2.5102,-3,0.0006529
...

Further, you have to set the file path to your csv and the variables containing your in- and output dimensions in the script file.

Example:

...
filename_data = "./data.csv"
input_dim = 2
output_dim = 1
...

The variables containing the values of training parameters and basic network structure can be found under the section SET PARAMETERS within the training script. You are free to set the according to your requirements. Please notice the restriction for activation functions in the next section.

The scripts will generate an xml file containing the network data which you have to provide to MeLOn in your C++ program.

Writing your own script

You are also free to write your own script. As the C++ part of MeLOn only is compatible with a subset of the features contained in keras there are a few restrictions you have to keep in mind when writing your training script:

  • Model: The network has to be a keras sequential model.
  • Scaling: For scaling the in- and output values you have to use the following functions from utils.py:

    You can also choose not to use scaling on either of these.

  • Layers: The C++ part of MeLOn only supports dense layers. But this doesn't mean that you can only use keras dense layers in your training. You are free to use layers that only influence the training (e.g. dropout layers) or to use layer modifiers like pruning layers that can be removed after training.
  • Activation functions: You can use the following activation functions with their default parameters:
    • linear
    • tanh
    • relu
    • relu6
  • Output: To generate the xml file used by the C++ part of MeLOn you have to use the save_model_to_xml function from utils.py

Matlab

The training script gives you an example of how to train a feedforward neural network in Matlab using the feedforwardnet implementation from the Deep Learning Toolbox. In the example, the training data is generated in the script, but you can also load the data using your preferred method and then use it for training by calling the configuration function for your network. To save the trained network you have to use the sNetwork2CSV function which is located in feedforward neural network\training\matlab\Write ANN to files\sNetwork2CSV.m.

Restrictions for training with Matlab:

  • Activation-/Transferfunction: You can use the following activation functions:
    • purelin
    • tansig

Usage

To include the trained network in your MAiNGO problem formulation you can use the MeLOn C++ interface. An example of how to include a feed forward neural network in your optimization can be found in the MAiNGO folder.

Constructors

  1. melon::FeedForwardNet<T>::FeedForwardNet()
  2. melon::FeedForwardNet<T>::FeedForwardNet(std::string modelName, MODEL_FILE_TYPE fileType)
  3. melon::FeedForwardNet<T>::FeedForwardNet(std::string modelPath, std::string modelName, MODEL_FILE_TYPE fileType)
  4. melon::FeedForwardNet<T>::FeedForwardNet(std::shared_ptr<AnnData> modelData)

Prediction

  1. melon::FeedForwardNet<T>::calculate_prediction_reduced_space(const std::vector<T> input)
  2. melon::FeedForwardNet<T>::calculate_prediction_full_space(const std::vector<T> input, const std::vector<T> internalVariables, std::vector<T>& constraints)

Internal variables

  1. melon::FeedForwardNet<T>::get_number_of_full_space_variables()
  2. melon::FeedForwardNet<T>::get_full_space_variables(unsigned int& variableNumber, std::vector<std::string>& variableNames, std::vector<std::pair<double, double>>& variableBounds)

Settings

  1. melon::FeedForwardNet<T>::set_tanh_formulation(const TANH_REFORMULATION& reformulation)