!!This project is obsolete !! Use ConfLang instead !!

CNNTrain

CNNTrain is a domain specific language for describing training parameters of a feedforward neural network. CNNTrain files must have .cnnt extension. Training configuration starts with a configuration word, followed by the configuration name and a list of parameters. The available parameters are batch size, number of epochs, loading previous checkpoint as well as an optimizer with its parameters. All these parameters are optional.

An example of a config:

configuration FullConfig{
     num_epoch : 5
     batch_size : 100
     load_checkpoint: true
     optimizer:rmsprop{
         learning_rate:0.001
         weight_decay:0.01
         learning_rate_decay:0.9
         learning_rate_policy:step
         step_size:1000
         rescale_grad:1.1
         clip_gradient:10
         gamma1:0.9
         gamma2:0.9
         epsilon:0.000001
         centered:true
         clip_weights:10
     }
}

See CNNTrain.mc4 for full grammar definition.

Using CNNTrainGenerator class, a Python file can be generated, which looks as following (for an example above):

batch_size = 100,
num_epoch = 5,
load_checkpoint = True,
optimizer = 'rmsprop',
optimizer_params = {
   'epsilon': 1.0E-6,
   'weight_decay': 0.01,
   'rescale_grad': 1.1,
   'centered': True,
   'clip_gradient': 10.0,
   'gamma2': 0.9,
   'gamma1': 0.9,
   'learning_rate_policy': 'step',
   'clip_weights': 10.0,
   'learning_rate': 0.001,
   'learning_rate_decay': 0.9,
   'step_size': 1000}

Reinforcement Learning

CNNTrain can be used to describe training parameters for supervised learning methods as well as for reinforcement learning methods. If reinforcement learning is selected, the network is trained with the Deep-Q-Network algorithm (Mnih et. al. in Playing Atari with Deep Reinforcement Learning).

An example of a supervised learning configuration can be seen above. The following is an example configuration for reinforcement learning:

configuration ReinforcementConfig {
    learning_method : reinforcement

    agent_name : "reinforcement-agent"

    environment : gym { name:"CartPole-v1" }

    context : cpu

    num_episodes : 300
    num_max_steps : 9999
    discount_factor : 0.998
    target_score : 1000
    training_interval : 10

    loss : huber

    use_fix_target_network : true
    target_network_update_interval : 100

    use_double_dqn : true

    replay_memory : buffer{
        memory_size : 1000000
        sample_size : 64
    }

    action_selection : epsgreedy{
        epsilon : 1.0
        min_epsilon : 0.01
        epsilon_decay_method: linear
        epsilon_decay : 0.0001
    }

    optimizer : rmsprop{
        learning_rate : 0.001
        learning_rate_minimum : 0.00001
        weight_decay : 0.01
        learning_rate_decay : 0.9
        learning_rate_policy : step
        step_size : 1000
        rescale_grad : 1.1
        clip_gradient : 10
        gamma1 : 0.9
        gamma2 : 0.9
        epsilon : 0.000001
        centered : true
        clip_weights : 10
    }
}

Available Parameters for Reinforcement Learning

Environment

Option: ros_interface

If selected, the communication between the environment and the agent is done via ROS. Additional parameters:

• state_topic: Topic on which the state is published
• action_topic: Topic on which the action is published
• reset_topic: Topic on which the reset command is published
• terminal_state_topic: Topic on which the terminal flag is published

Option: gym

The gym environment comes with a set of environments which are listed here. Additional parameters:

• name: Name (see https://gym.openai.com/) of the environment

Replay Buffer

Different buffer behaviour can be selected for the training. For more information about the buffer behaviour see "A deeper look at Experience Replay" by Zhang, Sutton

Option: buffer

A simple buffer in which stores the SARS (State, Action, Reward, next State) tuples. Additional parameters:

• memory_size: Determines the size of the buffer
• sample_size: Number of samples that are used for each training step

Option: online

No buffer is used. Only the current SARS tuple is used for taining.

Option: combined

Combination of online and buffer. Both the current SARS tuple as well as a sample from the buffer are used for each training step. Parameters are the same as buffer.

Strategy

Determines the behaviour when selecting an action based on the values.

Option: epsgreedy

This strategy is only available for discrete problems. It selects an action based on Epsilon-Greedy-Policy. This means, based on epsilon, either a random action is choosen or an action with the highest Q-value. Additional parameters:

• epsilon: Probability of choosing an action randomly
• epsilon_decay_method: Method which determines how epsilon decreases after each step. Can be linear for linear decrease or no for no decrease.
• epsilon_decay_start: Number of Episodes after the decay of epsilon starts
• epsilon_decay: The actual decay of epsilon after each step.
• min_epsilon: After min_epsilon is reached, epsilon is not decreased further.
• epsilon_decay_per_step:Expects either true or false. If true, the decay will be performed for each step the agent executes instead of performing the decay after each episode. The default value is false

Option: ornstein_uhlenbeck

This strategy is only available for continuous problems. The action is selected based on the actor network. Based on the current epsilon, noise is added based on the Ornstein-Uhlenbeck process. Additional parameters:

All epsilon parameters from epsgreedy strategy can be used. Additionally, mu, theta, and sigma needs to be specified. For each action output you can specify the corresponding value with a tuple-style notation: (x,y,z)

Example: Given an actor network with action output of shape (3,), we can write

    strategy: ornstein_uhlenbeck{
        ...
        mu: (0.0, 0.1, 0.3)
        theta: (0.5, 0.0, 0.8)
        sigma: (0.3, 0.6, -0.9)
    }

to specify the parameters for each place.

Option: gaussian

This strategy is also only available for continuous problems. If this strat- egy is selected, uncorrelated Gaussian noise with zero mean is added to the current policy action selection. This strategy provides the same parameters as the epsgreedy option and the parameter noise_variance that determines the variance of the noise.

Generation

To execute generation in your project, use the following code to generate a separate Config file:

import de.monticore.lang.monticar.cnntrain.generator.CNNTrainGenerator;
...
CNNTrainGenerator cnnTrainGenerator =  new CNNTrainGenerator();
Path modelPath = Paths.get("path/to/cnnt/file");
cnnTrainGenerator.generate(modelPath, cnnt_filename);

Use the following code to get file contents as a map ( fileContents.getValue() contains the generated code):

import de.monticore.lang.monticar.cnntrain.generator.CNNTrainGenerator;
import de.monticore.lang.monticar.cnntrain._symboltable.CNNTrainLanguage;
...
CNNTrainGenerator cnnTrainGenerator =  new CNNTrainGenerator();
ModelPath mp = new ModelPath(Paths.get("path/to/cnnt/file"));
GlobalScope trainScope = new GlobalScope(mp, new CNNTrainLanguage());
Map.Entry<String, String> fileContents = cnnTrainGenerator.generateFileContent( trainScope, cnnt_filename );

CNNTrain can be used together with CNNArch language, which describes architecture of a NN. EmbeddedMontiArcDL uses both languages.