Skip to content
Snippets Groups Projects
Commit a9cc9287 authored by Dennis Noll's avatar Dennis Noll
Browse files

[exercise] 3/all: init exercises

parent 724609bb
Branches
No related tags found
No related merge requests found
%% Cell type:markdown id:484cae12 tags:
# Regression of a very complex function
In this task you will learn how to perform a regression of a very complex function.
%% Cell type:markdown id:26b13114 tags:
## Imports and Seeding
First we will do the necessary imports:
* `numpy` for general data handling and array manipulation
* `tensorflow` to build and train the regression model
* `matplotlib.pyplot` for plotting
%% Cell type:code id:1833559e tags:
``` python
import numpy as np
import tensorflow as tf
from matplotlib import pyplot as plt
```
%% Cell type:markdown id:48e0eee1 tags:
Then we set a random seed for the `np.random` module. This makes our code reproducible as the random operations will yield the same results in every run through the notebook.
%% Cell type:code id:b3ab0336 tags:
``` python
np.random.seed(42)
```
%% Cell type:markdown id:9d66dc3c tags:
## Data Creation
First we define the parameters of the data.
* `n_data`: number of data points
* `uncertainty`: the uncertainty that is used
%% Cell type:code id:cb0f589d tags:
``` python
N = 10 ** 4
uncertainty = 0.0
```
%% Cell type:markdown id:b89b8f85 tags:
We define `some_complicated_function` that we want to regress and model with our neural network.
%% Cell type:code id:f7de78f4 tags:
``` python
def some_complicated_function(x):
return (
(np.abs(x)) ** 0.5
+ 0.1 * x
+ 0.01 * x ** 2
+ 1
- np.sin(x)
+ 0.5 * np.exp(x / 10.0)
) / (0.5 + np.abs(np.cos(x)))
```
%% Cell type:markdown id:b6370006 tags:
And now we create the training data according to `some_complicated_function`.
%% Cell type:code id:7a21cca8 tags:
``` python
x_lin = np.linspace(-10, 10, 1000)[:, None]
y_lin = some_complicated_function(x_lin)
x = np.random.uniform(-10, 10, size=(N,1))
y = some_complicated_function(x)
y += np.random.normal(0, uncertainty, size=y.shape[0])[..., None]
```
%% Cell type:markdown id:5745f7cf tags:
## Data Visualization
Visualize the used function (`some_complicated_function`) andd the created data `(x, y)` in an appropriate way.
%% Cell type:code id:c2066a2e tags:
``` python
"""
TODO: Visualize, x, y, x_lin, y_lin
"""
```
%% Output
'\nTODO: Visualize, x, y, x_lin, y_lin\n'
%% Cell type:markdown id:576b2822 tags:
## Model Setup
Now create the model:
- What is a suitable size?
- How many inputs and outputs are needed?
- What are suitable activations?
%% Cell type:code id:5b91c84a tags:
``` python
"""
TODO: Create the model
"""
```
%% Output
'\nTODO: Create the model\n'
%% Cell type:markdown id:29a3ba47 tags:
Now compile the model:
- Which loss function should be used? ([Documentation](https://www.tensorflow.org/api_docs/python/tf/keras/losses))
- Which optimizer should be used?
%% Cell type:code id:3419b16a tags:
``` python
"""
TODO: Compile the model
"""
```
%% Output
'\nTODO: Compile the model\n'
%% Cell type:markdown id:1916abda tags:
## Model Training
Now train the model:
* What is a suitable number of epochs?
* What is a suitable size for the batches?
%% Cell type:code id:e862ab76 tags:
``` python
"""
TODO: Train the model
"""
```
%% Output
'\nTODO: Train the model\n'
%% Cell type:markdown id:3a0d56fa tags:
## Model Evaluation
Visualize the model prediction alogn with the original function and the training data. What do you observe?
%% Cell type:code id:ad7a746c tags:
``` python
"""
TODO: what does the model predict for each x value?
"""
```
%% Output
'\nTODO: what does the model predict for each x value?\n'
%% Cell type:code id:d8bbf7cc tags:
``` python
"""
TODO: Plot model prediction along with the original function and the training data
"""
```
%% Output
'\nTODO: Plot model prediction along with the original function and the training data\n'
%% Cell type:markdown id:85071641 tags:
## Model Improvements
Try to improve your model and its training. You may try the following configurations.
- Different activation functions (ReLU, Sigmoid, Thanh)
- Different learning rates (0.0001, 0.001, 0.01 ,0.1, 1, 10)
Describe your observations.
%% Cell type:markdown id:c4c1382a tags:
## Further Tasks
Go back to the beginning of the notebook and increase the uncertainty of the data.
Describe your observations.
%% Cell type:markdown id:c069e5c2 tags:
## Summary
This concludes our tutorial on the regression of a very complicated function.
In this tutorial you have learned:
* How to perform a regression with a neural network
* The limits of very simple neural networks
* The limits of very simple optimizers
* How to improve:
* the network
* the optimization of the network
* The influence of uncertain data on the model training
%% Cell type:code id:14ca73a2 tags:
``` python
```
%% Cell type:markdown id:484cae12 tags:
# Regression of a very complex function
In this task you will learn how to perform a regression of a very complex function.
%% Cell type:markdown id:26b13114 tags:
## Imports and Seeding
First we will do the necessary imports:
* `numpy` for general data handling and array manipulation
* `tensorflow` to build and train the regression model
* `matplotlib.pyplot` for plotting
%% Cell type:code id:1833559e tags:
``` python
import numpy as np
import tensorflow as tf
from matplotlib import pyplot as plt
```
%% Cell type:markdown id:48e0eee1 tags:
Then we set a random seed for the `np.random` module. This makes our code reproducible as the random operations will yield the same results in every run through the notebook.
%% Cell type:code id:b3ab0336 tags:
``` python
np.random.seed(42)
```
%% Cell type:markdown id:9d66dc3c tags:
## Data Creation
First we define the parameters of the data.
* `n_data`: number of data points
* `uncertainty`: the uncertainty that is used
%% Cell type:code id:cb0f589d tags:
``` python
N = 10 ** 4
uncertainty = 0.0
```
%% Cell type:markdown id:b89b8f85 tags:
We define `some_complicated_function` that we want to regress and model with our neural network.
%% Cell type:code id:f7de78f4 tags:
``` python
def some_complicated_function(x):
return (
(np.abs(x)) ** 0.5
+ 0.1 * x
+ 0.01 * x ** 2
+ 1
- np.sin(x)
+ 0.5 * np.exp(x / 10.0)
) / (0.5 + np.abs(np.cos(x)))
```
%% Cell type:markdown id:b6370006 tags:
And now we create the training data according to `some_complicated_function`.
%% Cell type:code id:7a21cca8 tags:
``` python
x_lin = np.linspace(-10, 10, 1000)[:, None]
y_lin = some_complicated_function(x_lin)
x = np.random.uniform(-10, 10, size=(N,1))
y = some_complicated_function(x)
y += np.random.normal(0, uncertainty, size=y.shape[0])[..., None]
```
%% Cell type:markdown id:5745f7cf tags:
## Data Visualization
Visualize the used function (`some_complicated_function`) andd the created data `(x, y)` in an appropriate way.
%% Cell type:code id:c2066a2e tags:
``` python
"""
TODO: Visualize, x, y, x_lin, y_lin
"""
plt.scatter(x, y, label="data", alpha=0.1)
plt.plot(x_lin, y_lin, label="Function f(x)", color="red")
plt.xlabel("x")
plt.ylabel("y")
plt.legend()
```
%% Output
<matplotlib.legend.Legend at 0x7fd3503afee0>
%% Cell type:markdown id:576b2822 tags:
## Model Setup
Now create the model:
- What is a suitable size?
- How many inputs and outputs are needed?
- What are suitable activations?
%% Cell type:code id:5b91c84a tags:
``` python
"""
TODO: Create the model
"""
model = tf.keras.Sequential(
layers=[
tf.keras.Input(shape=(1,)),
tf.keras.layers.Dense(16, activation="relu"),
tf.keras.layers.Dense(16, activation="relu"),
tf.keras.layers.Dense(16, activation="relu"),
tf.keras.layers.Dense(1),
]
)
```
%% Output
2022-08-05 12:03:39.786451: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
%% Cell type:markdown id:29a3ba47 tags:
Now compile the model:
- Which loss function should be used? ([Documentation](https://www.tensorflow.org/api_docs/python/tf/keras/losses))
- Which optimizer should be used?
%% Cell type:code id:3419b16a tags:
``` python
"""
TODO: Compile the model
"""
model.compile(loss="mse", optimizer="sgd")
```
%% Cell type:markdown id:1916abda tags:
## Model Training
Now train the model:
* What is a suitable number of epochs?
* What is a suitable size for the batches?
%% Cell type:code id:e862ab76 tags:
``` python
"""
TODO: Train the model
"""
model.fit(x, y, epochs=100)
```
%% Output
2022-08-05 12:03:40.145120: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
Epoch 1/100
313/313 [==============================] - 3s 4ms/step - loss: 3.3984
Epoch 2/100
313/313 [==============================] - 1s 3ms/step - loss: 2.5578
Epoch 3/100
313/313 [==============================] - 1s 3ms/step - loss: 2.5399
Epoch 4/100
313/313 [==============================] - 1s 2ms/step - loss: 2.4957
Epoch 5/100
313/313 [==============================] - 1s 3ms/step - loss: 2.3274
Epoch 6/100
313/313 [==============================] - 1s 3ms/step - loss: 2.1456
Epoch 7/100
313/313 [==============================] - 1s 3ms/step - loss: 2.1278
Epoch 8/100
313/313 [==============================] - 1s 3ms/step - loss: 1.9232
Epoch 9/100
313/313 [==============================] - 1s 2ms/step - loss: 1.9760
Epoch 10/100
313/313 [==============================] - 1s 2ms/step - loss: 1.8729
Epoch 11/100
313/313 [==============================] - 1s 4ms/step - loss: 1.7648
Epoch 12/100
313/313 [==============================] - 0s 2ms/step - loss: 1.7690
Epoch 13/100
313/313 [==============================] - 0s 2ms/step - loss: 1.7519
Epoch 14/100
313/313 [==============================] - 0s 2ms/step - loss: 1.6320
Epoch 15/100
313/313 [==============================] - 1s 2ms/step - loss: 1.7010
Epoch 16/100
313/313 [==============================] - 0s 1ms/step - loss: 1.6759
Epoch 17/100
313/313 [==============================] - 1s 2ms/step - loss: 1.6220
Epoch 18/100
313/313 [==============================] - 1s 4ms/step - loss: 1.5958
Epoch 19/100
313/313 [==============================] - 1s 2ms/step - loss: 1.4919
Epoch 20/100
313/313 [==============================] - 1s 2ms/step - loss: 1.5811
Epoch 21/100
313/313 [==============================] - 1s 2ms/step - loss: 1.5218
Epoch 22/100
313/313 [==============================] - 1s 2ms/step - loss: 1.5173
Epoch 23/100
313/313 [==============================] - 0s 1ms/step - loss: 1.4912
Epoch 24/100
313/313 [==============================] - 1s 2ms/step - loss: 1.4987
Epoch 25/100
313/313 [==============================] - 1s 2ms/step - loss: 1.6173
Epoch 26/100
313/313 [==============================] - 0s 1ms/step - loss: 1.4515
Epoch 27/100
313/313 [==============================] - 0s 942us/step - loss: 1.6338
Epoch 28/100
313/313 [==============================] - 0s 986us/step - loss: 1.6017
Epoch 29/100
313/313 [==============================] - 0s 942us/step - loss: 1.6380
Epoch 30/100
313/313 [==============================] - 0s 965us/step - loss: 1.6105
Epoch 31/100
313/313 [==============================] - 0s 1ms/step - loss: 1.5330
Epoch 32/100
313/313 [==============================] - 0s 1ms/step - loss: 1.4850
Epoch 33/100
313/313 [==============================] - 0s 1ms/step - loss: 1.6237
Epoch 34/100
313/313 [==============================] - 1s 4ms/step - loss: 1.5624
Epoch 35/100
313/313 [==============================] - 1s 4ms/step - loss: 1.6068
Epoch 36/100
313/313 [==============================] - 0s 1ms/step - loss: 1.5679
Epoch 37/100
313/313 [==============================] - 0s 1ms/step - loss: 1.5057
Epoch 38/100
313/313 [==============================] - 0s 1ms/step - loss: 1.5352
Epoch 39/100
313/313 [==============================] - 0s 1ms/step - loss: 1.4374
Epoch 40/100
313/313 [==============================] - 0s 963us/step - loss: 1.4601
Epoch 41/100
313/313 [==============================] - 0s 1ms/step - loss: 1.4545
Epoch 42/100
313/313 [==============================] - 0s 1ms/step - loss: 1.3235
Epoch 43/100
313/313 [==============================] - 0s 1ms/step - loss: 1.3738
Epoch 44/100
313/313 [==============================] - 0s 988us/step - loss: 1.2638
Epoch 45/100
313/313 [==============================] - 0s 953us/step - loss: 1.3944
Epoch 46/100
313/313 [==============================] - 0s 1ms/step - loss: 1.4198
Epoch 47/100
313/313 [==============================] - 0s 1ms/step - loss: 1.3505
Epoch 48/100
313/313 [==============================] - 0s 1ms/step - loss: 1.2997
Epoch 49/100
313/313 [==============================] - 0s 1ms/step - loss: 1.3908
Epoch 50/100
313/313 [==============================] - 0s 1ms/step - loss: 1.2984
Epoch 51/100
313/313 [==============================] - 0s 1ms/step - loss: 1.3460
Epoch 52/100
313/313 [==============================] - 0s 1ms/step - loss: 1.2285A: 0s - loss: 1.2
Epoch 53/100
313/313 [==============================] - 0s 1ms/step - loss: 1.2504
Epoch 54/100
313/313 [==============================] - 0s 1ms/step - loss: 1.2345
Epoch 55/100
313/313 [==============================] - 1s 2ms/step - loss: 1.3697
Epoch 56/100
313/313 [==============================] - 0s 1ms/step - loss: 1.2797
Epoch 57/100
313/313 [==============================] - 0s 1ms/step - loss: 1.2825
Epoch 58/100
313/313 [==============================] - 0s 1ms/step - loss: 1.4122
Epoch 59/100
313/313 [==============================] - 0s 1ms/step - loss: 1.2927
Epoch 60/100
313/313 [==============================] - 0s 1ms/step - loss: 1.2715
Epoch 61/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1922
Epoch 62/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1647
Epoch 63/100
313/313 [==============================] - 1s 2ms/step - loss: 1.0985
Epoch 64/100
313/313 [==============================] - 0s 1ms/step - loss: 1.0921
Epoch 65/100
313/313 [==============================] - 1s 4ms/step - loss: 1.3994A: 0s
Epoch 66/100
313/313 [==============================] - 1s 3ms/step - loss: 1.2860
Epoch 67/100
313/313 [==============================] - 1s 2ms/step - loss: 1.1705
Epoch 68/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1978
Epoch 69/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1500
Epoch 70/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1244
Epoch 71/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1054
Epoch 72/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1750
Epoch 73/100
313/313 [==============================] - 0s 1ms/step - loss: 1.0776
Epoch 74/100
313/313 [==============================] - 0s 1ms/step - loss: 1.0974
Epoch 75/100
313/313 [==============================] - 0s 1ms/step - loss: 1.0780
Epoch 76/100
313/313 [==============================] - 0s 993us/step - loss: 1.1288
Epoch 77/100
313/313 [==============================] - 0s 944us/step - loss: 1.0981
Epoch 78/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1474
Epoch 79/100
313/313 [==============================] - 0s 976us/step - loss: 1.1374
Epoch 80/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1090
Epoch 81/100
313/313 [==============================] - 0s 1ms/step - loss: 1.0766
Epoch 82/100
313/313 [==============================] - 0s 946us/step - loss: 1.0638
Epoch 83/100
313/313 [==============================] - 0s 1ms/step - loss: 1.0908
Epoch 84/100
313/313 [==============================] - 0s 1ms/step - loss: 1.0933
Epoch 85/100
313/313 [==============================] - 0s 1ms/step - loss: 1.0628
Epoch 86/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1284
Epoch 87/100
313/313 [==============================] - 0s 1ms/step - loss: 0.9667
Epoch 88/100
313/313 [==============================] - 0s 1ms/step - loss: 0.9885
Epoch 89/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1622
Epoch 90/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1594
Epoch 91/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1145
Epoch 92/100
313/313 [==============================] - 0s 1ms/step - loss: 1.0588
Epoch 93/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1608
Epoch 94/100
313/313 [==============================] - 0s 1ms/step - loss: 0.9633
Epoch 95/100
313/313 [==============================] - 0s 1ms/step - loss: 0.9691
Epoch 96/100
313/313 [==============================] - 0s 1ms/step - loss: 0.9562
Epoch 97/100
313/313 [==============================] - 0s 1ms/step - loss: 1.0622
Epoch 98/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1901
Epoch 99/100
313/313 [==============================] - 0s 1ms/step - loss: 1.0907
Epoch 100/100
313/313 [==============================] - 0s 1ms/step - loss: 1.1041
<tensorflow.python.keras.callbacks.History at 0x7fd3506951f0>
%% Cell type:markdown id:3a0d56fa tags:
## Model Evaluation
Visualize the model prediction alogn with the original function and the training data. What do you observe?
%% Cell type:code id:ad7a746c tags:
``` python
"""
TODO: what does the model predict for each x value?
"""
y_pred = model.predict(x_lin)
```
%% Cell type:code id:d8bbf7cc tags:
``` python
"""
TODO: Plot model prediction along with the original function and the training data
"""
plt.scatter(x, y, label="data", alpha=0.1)
plt.plot(x_lin, y_lin, label="Function f(x)", color="red")
plt.plot(x_lin, y_pred, linestyle="--", label="Fit DNN(x)", color="orange")
plt.xlabel("x")
plt.ylabel("y")
plt.legend()
plt.savefig("fit.pdf")
```
%% Output
%% Cell type:markdown id:85071641 tags:
## Model Improvements
Try to improve your model and its training. You may try the following configurations.
- Different activation functions (ReLU, Sigmoid, Thanh)
- Different learning rates (0.0001, 0.001, 0.01 ,0.1, 1, 10)
Describe your observations.
%% Cell type:markdown id:c4c1382a tags:
## Further Tasks
Go back to the beginning of the notebook and increase the uncertainty of the data.
Describe your observations.
%% Cell type:markdown id:c069e5c2 tags:
## Summary
This concludes our tutorial on the regression of a very complicated function.
In this tutorial you have learned:
* How to perform a regression with a neural network
* The limits of very simple neural networks
* The limits of very simple optimizers
* How to improve:
* the network
* the optimization of the network
* The influence of uncertain data on the model training
%% Cell type:code id:14ca73a2 tags:
``` python
```
%% Cell type:markdown id:e703cbc8 tags:
# Classification between Two Classes
In this exercise, we will learn how to perform a binary classification.
## Introduction
You are given data from two classes. In each class the data follows a distribution out of one or many gaussian distributions with class dependent parameters. Your task is to build a model which can classify between the two classes.
%% Cell type:markdown id:07066edc tags:
## Imports and Seeding
First we will do the necessary imports:
* `numpy` for general data handling and array manipulation
* `tensorflow` to build and train the regression model
* `matplotlib.pyplot` for plotting
* `sklearn.utils.shuffle` to randomly shuffle our training dataset
* `cycler.cycler` helps with plotting multiple distributions
%% Cell type:code id:33bfb028 tags:
``` python
import numpy as np
from matplotlib import pyplot as plt
import tensorflow as tf
from sklearn.utils import shuffle
from cycler import cycler
```
%% Cell type:markdown id:b17ea7df tags:
Then we set a random seed for the `np.random` module. This makes our code reproducible as the random operations will yield the same results in every run through the notebook.
%% Cell type:code id:d7118dd4 tags:
``` python
np.random.seed(42)
```
%% Cell type:markdown id:43930b0d tags:
## Data Creation
First we will create the data.
To make things a little bit more interesting we have written a small piece of code, which creates `N_DIM` dimensional data following distributions consiting of one or more (`N_PEAK`) different Gaussian functions. Increasing the number of `N_PEAK` will in general make the distributions and thus the task more complex.
%% Cell type:code id:cf21127e tags:
``` python
N_DIMS = 1
N_PEAK = 1
SCALE = 0.1
centers_class_1 = np.random.uniform(0, 1, size=(N_PEAK, N_DIMS))
centers_class_2 = np.random.uniform(0, 1, size=(N_PEAK, N_DIMS))
def make_samples(centers, n_samples=1_000):
output = []
for i, center in enumerate(centers):
output.append(np.random.normal(
loc=center,
scale=SCALE,
size=(n_samples // N_PEAK, N_DIMS)
))
return np.concatenate(output)
class_1 = make_samples(centers_class_1, 100_000)
class_2 = make_samples(centers_class_2, 100_000)
```
%% Cell type:markdown id:7789856e tags:
## Data Visualization
Visualize the data. When looking at one dimension (`N_DIMS=1`) a single histogram will solve the task. If plotting many dimensions (`N_DIMS>1`) you may want to plot 1-dimensional projections onto each of the axes.
%% Cell type:code id:33880fbe tags:
``` python
"""
TODO: Visualize the data of the two classes.
Wou may want to plot all 1-dimensional projections of the `N_DIM` dimensions of our data.
"""
```
%% Output
'\nTODO: Visualize the data of the two classes.\nWou may want to plot all 1-dimensional projections of the `N_DIM` dimensions of our data.\n'
%% Cell type:markdown id:42822770 tags:
## Data Preparation
Next we prepare the training data.
We built one dataset made out of both classes.
The `x` values of the training data are given by the distributions themselves
For the `y` values we use ones for `class_1` and zeros for`class_2`.
%% Cell type:code id:0f9b848e tags:
``` python
x = np.concatenate((class_1, class_2))
y = np.concatenate((np.ones(len(class_1)), np.zeros(len(class_2))))[..., None]
```
%% Cell type:markdown id:130d9122 tags:
Next we suffle our dataset. This prevents the case that during training the network only sees one type of events in a particular or even many subsequent batches.
%% Cell type:code id:060541ec tags:
``` python
x, y = shuffle(x, y, random_state=0)
```
%% Cell type:markdown id:b6a322e2 tags:
## Model Creation
Next we will create the model.
- What is a suitable size?
- How many inputs and outputs does the model need?
- What are suitable activations?
- Hint: Think about the activation of the last layer.
%% Cell type:code id:dd7278fd tags:
``` python
"""
TODO: Create the model
"""
```
%% Output
'\nTODO: Create the model\n'
%% Cell type:markdown id:17270e29 tags:
Now compile the model:
- Which loss function should be used? ([Documentation](https://www.tensorflow.org/api_docs/python/tf/keras/losses))
- Which optimizer should be used?
%% Cell type:code id:77b344b7 tags:
``` python
"""
TODO: Compile the model
"""
```
%% Output
'\nTODO: Compile the model\n'
%% Cell type:markdown id:baa6a06c tags:
Next we inspect our model. How many parameteres does it have?
%% Cell type:code id:4c364d71 tags:
``` python
"""
TODO: Use model.summary() to look at number of parameters
"""
```
%% Output
'\nTODO: Use model.summary() to look at number of parameters\n'
%% Cell type:markdown id:b6da7aab tags:
## Model Training
Now train the model:
* What is a suitable number of epochs?
* What is a suitable size for the batches?
%% Cell type:code id:10f26b8a tags:
``` python
"""
TODO: Train the model
"""
```
%% Output
'\nTODO: Train the model\n'
%% Cell type:markdown id:7501eec2 tags:
## Model Evaluation
Visualize the model prediction. Describe your observation.
%% Cell type:code id:2bcc84f6 tags:
``` python
"""
TODO: Prepare data for the model evaluation/prediction
"""
```
%% Output
'\nTODO: Prepare data for the model evaluation/prediction\n'
%% Cell type:code id:819684ee tags:
``` python
"""
TODO: Perform the model evaluation/prediction
"""
```
%% Output
'\nTODO: Perform the model evaluation/prediction\n'
%% Cell type:code id:cb0ccba0 tags:
``` python
"""
TODO: Visualize the model evaluation/prediciton
"""
```
%% Output
'\nTODO: Visualize the model evaluation/prediciton\n'
%% Cell type:markdown id:42bf1cc0 tags:
## Futher Tasks
Now we will make our exercise more difficult:
* Make the functions more complex (N_PEAK) and train the classifier again. Describe your observations.
* Raise the number of dimensions (N_DIM) to 2 (and 10) and train the classifier again. Describe your observations.
%% Cell type:markdown id:05fcec7d tags:
## Summary
This concludes our tutorial on the Classification between Two Classes.
In this tutorial you have learned:
* How to visualize n-dimensional data distributions from two classes
* How to prepare the data for a classification
* How to create a neural network for a classification
* Which loss to use for a classification
* How to interpret the output of a classification network
* The strenghts and limits of a simple network and a simple optimizer according to:
* Number of dimensions
* Complexity of functions
%% Cell type:code id:f5d7c7bf tags:
``` python
```
%% Cell type:markdown id:e703cbc8 tags:
# Classification between Two Classes
In this exercise, we will learn how to perform a binary classification.
## Introduction
You are given data from two classes. In each class the data follows a distribution out of one or many gaussian distributions with class dependent parameters. Your task is to build a model which can classify between the two classes.
%% Cell type:markdown id:07066edc tags:
## Imports and Seeding
First we will do the necessary imports:
* `numpy` for general data handling and array manipulation
* `tensorflow` to build and train the regression model
* `matplotlib.pyplot` for plotting
* `sklearn.utils.shuffle` to randomly shuffle our training dataset
* `cycler.cycler` helps with plotting multiple distributions
%% Cell type:code id:33bfb028 tags:
``` python
import numpy as np
from matplotlib import pyplot as plt
import tensorflow as tf
from sklearn.utils import shuffle
from cycler import cycler
```
%% Cell type:markdown id:b17ea7df tags:
Then we set a random seed for the `np.random` module. This makes our code reproducible as the random operations will yield the same results in every run through the notebook.
%% Cell type:code id:d7118dd4 tags:
``` python
np.random.seed(42)
```
%% Cell type:markdown id:43930b0d tags:
## Data Creation
First we will create the data.
To make things a little bit more interesting we have written a small piece of code, which creates `N_DIM` dimensional data following distributions consiting of one or more (`N_PEAK`) different Gaussian functions. Increasing the number of `N_PEAK` will in general make the distributions and thus the task more complex.
%% Cell type:code id:cf21127e tags:
``` python
N_DIMS = 1
N_PEAK = 1
SCALE = 0.1
centers_class_1 = np.random.uniform(0, 1, size=(N_PEAK, N_DIMS))
centers_class_2 = np.random.uniform(0, 1, size=(N_PEAK, N_DIMS))
def make_samples(centers, n_samples=1_000):
output = []
for i, center in enumerate(centers):
output.append(np.random.normal(
loc=center,
scale=SCALE,
size=(n_samples // N_PEAK, N_DIMS)
))
return np.concatenate(output)
class_1 = make_samples(centers_class_1, 100_000)
class_2 = make_samples(centers_class_2, 100_000)
```
%% Cell type:markdown id:7789856e tags:
## Data Visualization
Visualize the data. When looking at one dimension (`N_DIMS=1`) a single histogram will solve the task. If plotting many dimensions (`N_DIMS>1`) you may want to plot 1-dimensional projections onto each of the axes.
%% Cell type:code id:33880fbe tags:
``` python
"""
TODO: Visualize the data of the two classes.
Wou may want to plot all 1-dimensional projections of the `N_DIM` dimensions of our data.
"""
for d0 in range(N_DIMS):
fig, axs = plt.subplots(figsize=(10,4))
for opts in cycler(
x=[class_1, class_2],
label=["Class 1","Class 2"],
linestyle=["-", "--"],
):
axs.hist(
opts.pop("x")[:, d0],
**opts,
bins=20,
range=[0, 1],
density=True,
histtype="step",
lw=3,
)
axs.legend(loc=0)
axs.set_title(f"Dimension {d0}")
axs.set_xlabel(rf"$x_{{{d0}}}$")
axs.set_ylabel(f"Counts (normalized)")
fig.show()
plt.savefig("plot.pdf")
```
%% Output
/var/folders/dw/z8g0cw_s0nzcv0z1jn7vfgsm0000gn/T/ipykernel_2041/288129882.py:26: UserWarning: Matplotlib is currently using module://matplotlib_inline.backend_inline, which is a non-GUI backend, so cannot show the figure.
fig.show()
%% Cell type:markdown id:42822770 tags:
## Data Preparation
Next we prepare the training data.
We built one dataset made out of both classes.
The `x` values of the training data are given by the distributions themselves
For the `y` values we use ones for `class_1` and zeros for`class_2`.
%% Cell type:code id:0f9b848e tags:
``` python
x = np.concatenate((class_1, class_2))
y = np.concatenate((np.ones(len(class_1)), np.zeros(len(class_2))))[..., None]
```
%% Cell type:markdown id:130d9122 tags:
Next we suffle our dataset. This prevents the case that during training the network only sees one type of events in a particular or even many subsequent batches.
%% Cell type:code id:060541ec tags:
``` python
x, y = shuffle(x, y, random_state=0)
```
%% Cell type:markdown id:b6a322e2 tags:
## Model Creation
Next we will create the model.
- What is a suitable size?
- How many inputs and outputs does the model need?
- What are suitable activations?
- Hint: Think about the activation of the last layer.
%% Cell type:code id:dd7278fd tags:
``` python
"""
TODO: Create the model
"""
model = tf.keras.Sequential(
layers=[
tf.keras.Input(shape=(N_DIMS,)),
tf.keras.layers.Dense(256, activation="relu"),
tf.keras.layers.Dense(256, activation="relu"),
tf.keras.layers.Dense(256, activation="relu"),
tf.keras.layers.Dense(1, activation="sigmoid"),
]
)
```
%% Output
2022-08-05 12:03:31.241346: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
%% Cell type:markdown id:17270e29 tags:
Now compile the model:
- Which loss function should be used? ([Documentation](https://www.tensorflow.org/api_docs/python/tf/keras/losses))
- Which optimizer should be used?
%% Cell type:code id:77b344b7 tags:
``` python
"""
TODO: Compile the model
"""
model.compile(optimizer="SGD", loss="bce", metrics=["accuracy"])
```
%% Cell type:markdown id:baa6a06c tags:
Next we inspect our model. How many parameteres does it have?
%% Cell type:code id:4c364d71 tags:
``` python
"""
TODO: Use model.summary() to look at number of parameters
"""
model.summary()
```
%% Output
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 256) 512
_________________________________________________________________
dense_1 (Dense) (None, 256) 65792
_________________________________________________________________
dense_2 (Dense) (None, 256) 65792
_________________________________________________________________
dense_3 (Dense) (None, 1) 257
=================================================================
Total params: 132,353
Trainable params: 132,353
Non-trainable params: 0
_________________________________________________________________
%% Cell type:markdown id:b6da7aab tags:
## Model Training
Now train the model:
* What is a suitable number of epochs?
* What is a suitable size for the batches?
%% Cell type:code id:10f26b8a tags:
``` python
"""
TODO: Train the model
"""
model.fit(x, y)
```
%% Output
2022-08-05 12:03:31.387168: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
6250/6250 [==============================] - 20s 3ms/step - loss: 0.2305 - accuracy: 0.9128
<tensorflow.python.keras.callbacks.History at 0x7fe4b1173b50>
%% Cell type:markdown id:7501eec2 tags:
## Model Evaluation
Visualize the model prediction. Describe your observation.
%% Cell type:code id:2bcc84f6 tags:
``` python
"""
TODO: Prepare data for the model evaluation/prediction
"""
n = 100
s = np.linspace(0, 1, n)
xy = np.meshgrid(*(N_DIMS * [s]), indexing="ij")
xy = np.concatenate(tuple(x[..., None] for x in xy), axis=-1)
```
%% Cell type:code id:819684ee tags:
``` python
"""
TODO: Perform the model evaluation/prediction
"""
xy = xy.reshape(-1, N_DIMS)
y_pred = model.predict(xy)
y_pred = y_pred.reshape(*(n for i in range (N_DIMS)), 1)
```
%% Cell type:code id:cb0ccba0 tags:
``` python
"""
TODO: Visualize the model evaluation/prediciton
"""
for i in range(N_DIMS):
_y_pred = y_pred.mean(axis=tuple(set(range(N_DIMS))-{i}))
plt.plot(s, _y_pred)
plt.title(f"Dimension {i}")
plt.show()
```
%% Output
%% Cell type:markdown id:42bf1cc0 tags:
## Futher Tasks
Now we will make our exercise more difficult:
* Make the functions more complex (N_PEAK) and train the classifier again. Describe your observations.
* Raise the number of dimensions (N_DIM) to 2 (and 10) and train the classifier again. Describe your observations.
%% Cell type:markdown id:05fcec7d tags:
## Summary
This concludes our tutorial on the Classification between Two Classes.
In this tutorial you have learned:
* How to visualize n-dimensional data distributions from two classes
* How to prepare the data for a classification
* How to create a neural network for a classification
* Which loss to use for a classification
* How to interpret the output of a classification network
* The strenghts and limits of a simple network and a simple optimizer according to:
* Number of dimensions
* Complexity of functions
%% Cell type:code id:f5d7c7bf tags:
``` python
```
%% Cell type:markdown id:93c2f0ba tags:
# Classification with DNN and Pen&Paper
In this task we will explore the analytic solution of DNN classifications.
## Introduction
We will create data from two Gaussian distributions and train a classification between them.
In parallel we will take pen and paper to calculate the solution of the classification.
Then we will compare the results of our network training and our pen&paper calculation.
%% Cell type:code id:20329dfd tags:
``` python
import numpy as np
from matplotlib import pyplot as plt
import tensorflow as tf
```
%% Cell type:markdown id:df4f97e5 tags:
## Creating and plotting the data
First we fix the parametrisation of our Gaussian distributions (A and B) and create the data.
%% Cell type:code id:6c92b864 tags:
``` python
# parametrisation of the underlying probability distributions
loc_a, scale_a = 0, 1.5
loc_b, scale_b = 1, 1.2
# creating the data
a = np.random.normal(loc=loc_a, scale=scale_a, size=(100000,))
b = np.random.normal(loc=loc_b, scale=scale_b, size=(100000,))
```
%% Cell type:markdown id:2890ec5b tags:
We bin the data in histograms with equidistant bins, plot the histograms and plot (parts of) the raw data.
%% Cell type:code id:f687f21b tags:
``` python
# creating the figure
fig, (ax1, ax2) = plt.subplots(nrows=2, ncols=1, sharex='col', gridspec_kw={'height_ratios': [2, 1], "hspace": 0})
# plot histograms
hist_a, bins_a, _ = ax1.hist(a, bins=np.arange(-4, 4.4, 0.4), alpha=0.5, label="Distribution A")
hist_b, bins_b, _ = ax1.hist(b, bins=bins_a, alpha=0.5, label="Distribution B")
# plot 1000 example points
ax2.plot(a[:1000], np.zeros_like(a)[:1000], linestyle="None", marker="|", alpha=0.1)
ax2.plot(b[:1000], np.zeros_like(b)[:1000], linestyle="None", marker="|", alpha=0.1)
# styling plot
ax2.axes.get_yaxis().set_visible(False)
ax2.set_xlabel("x")
ax1.set_ylabel("Counts [#]")
ax2.set_xlim([-4, 4])
ax1.legend()
```
%% Output
<matplotlib.legend.Legend at 0x7f9211e57c70>
%% Cell type:markdown id:77dfd868 tags:
## Task
Now it is your turn. The two data distributions is everything you will get to solve this task. Train a classification between the two distributions and compare the output of your network with the pen&paper calculation you performed before. Describe your observation.
%% Cell type:code id:a7e53779 tags:
``` python
```
%% Cell type:markdown id:93c2f0ba tags:
# Classification with DNN and Pen&Paper
In this task we will explore the analytic solution of DNN classifications.
## Introduction
We will create data from two Gaussian distributions and train a classification between them.
In parallel we will take pen and paper to calculate the solution of the classification.
Then we will compare the results of our network training and our pen and paper calculation.
%% Cell type:code id:20329dfd tags:
``` python
import numpy as np
from matplotlib import pyplot as plt
import tensorflow as tf
```
%% Cell type:markdown id:df4f97e5 tags:
## Creating and plotting the data
First we fix the parametrisation of our Gaussian distributions (A and B) and create the data.
%% Cell type:code id:6c92b864 tags:
``` python
# parametrisation of the underlying probability distributions
loc_a, scale_a = 0, 1.5
loc_b, scale_b = 1, 1.2
# creating the data
a = np.random.normal(loc=loc_a, scale=scale_a, size=(100000,))
b = np.random.normal(loc=loc_b, scale=scale_b, size=(100000,))
```
%% Cell type:markdown id:2890ec5b tags:
We bin the data in histograms with equidistant bins, plot the histograms and plot (parts of) the raw data.
%% Cell type:code id:f687f21b tags:
``` python
# creating the figure
fig, (ax1, ax2) = plt.subplots(nrows=2, ncols=1, sharex='col', gridspec_kw={'height_ratios': [2, 1], "hspace": 0})
# plot histograms
hist_a, bins_a, _ = ax1.hist(a, bins=np.arange(-4, 4.4, 0.4), alpha=0.5, label="Distribution A")
hist_b, bins_b, _ = ax1.hist(b, bins=bins_a, alpha=0.5, label="Distribution B")
# plot 1000 example points
ax2.plot(a[:1000], np.zeros_like(a)[:1000], linestyle="None", marker="|", alpha=0.1)
ax2.plot(b[:1000], np.zeros_like(b)[:1000], linestyle="None", marker="|", alpha=0.1)
# styling plot
ax2.axes.get_yaxis().set_visible(False)
ax2.set_xlabel("x")
ax1.set_ylabel("Counts [#]")
ax2.set_xlim([-4, 4])
ax1.legend()
```
%% Output
<matplotlib.legend.Legend at 0x7f9211e57c70>
%% Cell type:markdown id:d78ecea0 tags:
## DNN based Classification
Now create a DNN model for the classification between Distribution A and Distribution B.
- How many inputs do we have?
- How many outputs do we have?
- How many layers with which activation funcitons do we need?
- Does the network output probabilities or logits?
%% Cell type:code id:7dbbc75c tags:
``` python
"""
TODO: Create the model
"""
model = tf.keras.Sequential(
layers=[
tf.keras.Input(shape=(1,)),
tf.keras.layers.Dense(30, activation="relu"),
tf.keras.layers.Dense(30, activation="relu"),
tf.keras.layers.Dense(1, activation="sigmoid"),
]
)
```
%% Cell type:markdown id:3131b396 tags:
Now compile the model.
- Which loss function do we want?
- Which optimizer do we want?
%% Cell type:code id:52114d30 tags:
``` python
"""
TODO: Compile the model
optimizer = "sgd"
loss = tf.keras.losses.BinaryCrossentropy(from_logits=False)
model.compile(optimizer=optimizer, loss=loss, metrics=["accuracy"])
```
%% Cell type:markdown id:51d8dee9 tags:
Look at the model summary to see how many trainable parameters or model has.
%% Cell type:code id:0e8cd8a5 tags:
``` python
model.summary()
```
%% Output
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_3 (Dense) (None, 30) 60
_________________________________________________________________
dense_4 (Dense) (None, 30) 930
_________________________________________________________________
dense_5 (Dense) (None, 1) 31
=================================================================
Total params: 1,021
Trainable params: 1,021
Non-trainable params: 0
_________________________________________________________________
%% Cell type:markdown id:664744b9 tags:
Now we prepare the data for the training by interleaving datasets A and B and shuffling the data.
%% Cell type:code id:47253403 tags:
``` python
# prepare the data for training (interleave+shuffle)
x = np.concatenate([a, b])
y = np.concatenate([np.ones_like(a), np.zeros_like(b)])
p = np.random.permutation(len(x))
x, y = x[p], y[p]
```
%% Cell type:markdown id:85757895 tags:
Now we fit the model to the training data:
%% Cell type:code id:7013e12d tags:
``` python
# fit the model
model.fit(x=x, y=y, epochs=5, validation_split=0.2)
```
%% Output
Epoch 1/5
5000/5000 [==============================] - 7s 1ms/step - loss: 0.6348 - accuracy: 0.6359 - val_loss: 0.6210 - val_accuracy: 0.6504
Epoch 2/5
5000/5000 [==============================] - 5s 1ms/step - loss: 0.6214 - accuracy: 0.6501 - val_loss: 0.6208 - val_accuracy: 0.6507
Epoch 3/5
5000/5000 [==============================] - 6s 1ms/step - loss: 0.6217 - accuracy: 0.6486 - val_loss: 0.6205 - val_accuracy: 0.6509
Epoch 4/5
5000/5000 [==============================] - 6s 1ms/step - loss: 0.6208 - accuracy: 0.6494 - val_loss: 0.6205 - val_accuracy: 0.6510
Epoch 5/5
5000/5000 [==============================] - 6s 1ms/step - loss: 0.6217 - accuracy: 0.6503 - val_loss: 0.6204 - val_accuracy: 0.6508
<tensorflow.python.keras.callbacks.History at 0x7f9211fa5850>
%% Cell type:markdown id:31e4e01b tags:
## Plotting and Results
In the following cells we will plot the analytic solution, our training solution and the analytic solution using the actual data distributions which were used for the training.
%% Cell type:code id:3056147d tags:
``` python
# define x_values for inference and plotting
x_values = np.arange(-4, 4.01, 0.01)[:, None]
```
%% Cell type:code id:160928b9 tags:
``` python
# analytic solution
def gaussian(x, loc, scale):
return 1 / np.sqrt(2 * np.pi * scale ** 2) * np.exp(-((x - loc) ** 2) / (2 * scale ** 2))
gauss_a = gaussian(x_values, loc=loc_a, scale=scale_a)
gauss_b = gaussian(x_values, loc=loc_b, scale=scale_b)
analytic = gauss_a / (gauss_a + gauss_b)
```
%% Cell type:code id:7d2b85be tags:
``` python
# classic solution by histogram division
hist = hist_a / (hist_a + hist_b)
```
%% Cell type:code id:eb42db1a tags:
``` python
# DNN based classification solution
pred = model.predict(x_values)
```
%% Cell type:code id:86373833 tags:
``` python
# plot all the infered ratios
plt.plot(x_values, analytic, label="Analytic", color="black", linestyle="--")
plt.plot((bins_a[1:]+bins_a[:-1])/2, hist, label="Hist")
plt.plot(x_values, pred, label="DNN", color="red")
plt.xlabel("x")
plt.ylabel("Ratio A/(A+B)")
plt.xlim([-4, 4])
plt.legend()
```
%% Output
<matplotlib.legend.Legend at 0x7f9211204e80>
%% Cell type:markdown id:0742090c tags:
As you can see, the all three distributions match quite well!
%% Cell type:markdown id:ef7e916c tags:
## Summary
This concludes our tutorial for today.
In this tutorial you learned:
- How to train a neural network based classification task
- How to predict the output of the trained network using pen and paper
%% Cell type:code id:b42625de tags:
``` python
```
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment