Before starting the the exercise, we import the necessary package
%% Cell type:code id: tags:
```
import numpy as np
import random
random.seed()
%% Cell type:markdown id: tags:
## Exercise 1 (Single neuron)
Given input of values $[1,2,3]$. Program a code in Python which prints out the output of a single neuron with certain given (or random) values of weights and bias of this neuron respectively with:
Given input of values $[1,2,3]$. Program a code in Python which prints out the output of two neurons with certain given (random) values of weights and bias of this neuron respectively with:
The input of the neural network is dimension 3. Assume we have a mini-batch of size 3 of a certain given (or random) values. Program a code in Python which prints out the output of two neurons with certain given (random) values of weights and bias of this neuron respectively with:
1. You can calculate for each value in the mini-batch sequentially by a for-loop for example. However, it is highly inefficient. What solution can you propose?
1. With the solution, what is the shape of the input data, the weights, and biases?
1. Implement the solution in Python with **Numpy array**. (using **np.dot** or **np.matmul** (matrix multiplication)).
Any values of input and weights are fine, the students can use a numpy random generator for creating the input and weights (e.g. **np.random.randn**).
%% Cell type:markdown id: tags:
1. Use matrix multiplication.
1. The input data needs to be 3x3 (size of mini-batch times the input dimension). The weights is 3x2 (input size times number of neurons) and bias is 1x2.
%% Cell type:code id: tags:
```
inputs = # to complete
weights = # to complete
bias = # to complete
outputs = # to complete
print(outputs)
%% Output
[[ 2.92800142 -3.55475399]
[ 2.293519 -0.84206431]
[-0.03909697 -1.18018202]]
%% Cell type:markdown id: tags:
## Exercise 4 (Multiple layers)
We consider the input of dimension 3 of mini-batches of size 5. We now consider a two-layer neural network with the first layer having 3 neurons and second layer having 2 neurons. Program to prints out the outputs.
%% Cell type:code id: tags:
```
inputs = np.random.randn(5,3)
# to complete
layer1output = # to complete
print("Output layer1", layer1output)
layer2output = # to complete
print("Output layer2", layer2output)
%% Cell type:markdown id: tags:
## Exercise 5 (Activation function)
The activation function is applied on the output of a layer of neurons.
It defines the output values. For instance, softmax activation function is usually used for classification tasks, and its value represents the confidence scores for each class.
with $z_{i,j}$ the output from the previous layer, with $i$ the size of mini-batches and $j$ along the output dimension.
Program the values after a softmax activation function for a given input.
%% Cell type:code id: tags:
```
layer_outputs = np.array([[4.8, 1.21, 2.385],
[8.9, -1.81, 0.2]])
activated_values = # to complete
print(activated_values)
%% Cell type:markdown id: tags:
## Exercise 6 (Loss)
Training a given neural network model means to improve the model's accuracy. The error of a model is referred to as **loss function**.
We ideally want it to be 0.
A cross-entropy loss is used to compare a "ground-truth" probability distribution with predicted distribution (in our case, output after a softmax activation function). Its expression is: