{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ZZ1WcKdg4zLm"
},
"source": [
"# co@work: Robust ML Training with Conditional Gradients\n",
"\n",
"In this tutorial we will go through two examples and implement versions of both (stochastic) Gradient Descent and (stochastic) Conditional Gradient Descent. Most of the code is already implemented, but you will be asked to fill in the important aspects along the way. The frameworks that we will use for these examples are [Numpy](https://www.numpy.org) and [TensorFlow](https://www.tensorflow.org). Let's get started with the first example."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "LwifbWJA47v7"
},
"source": [
"# Example 1 – Predicting Apartment Cost in Numpy\n",
"\n",
"In this example we will implement a simple linear regression. Before explaining the setup in more detail, we need to load some libraries."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "fYGp6ZLl5Xbh"
},
"outputs": [],
"source": [
"import time\n",
"import numpy as np\n",
"import math\n",
"\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "8VK8y26CerQP"
},
"source": [
"The data we will consider comes from apartments in Berlin. A vector $\\bf x$ lists the size of each apartment in square-meters and a vector $\\bf y$ the monthly rent in Euros. The entry $i$ of each vector always corresponds to the same apartment. Let us load the data and have a look at it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 285
},
"colab_type": "code",
"id": "azM3DSW7gPNN",
"outputId": "1e173e4b-3344-4fc3-bb88-ed64d092f1f0"
},
"outputs": [],
"source": [
"xvec = np.array([45, 59, 60, 74, 40, 63, 66, 63, 53, 49, 40, 54, 53, 59, 50, 35, 69, 53, 39, 72, 56, 66, 45, 60, 41, 74, 33, 56, 78, 37, 59, 63, 54, 47, 64, 39, 51, 81, 79, 52, 58, 72, 94, 71, 62, 63, 95, 84, 40, 79, 81, 85, 88, 54, 67, 72, 47, 67, 81, 65, 59, 53, 76, 52, 42, 83, 62, 67, 45, 67, 72, 61, 78, 65, 50, 62, 45, 67, 61, 53, 85, 40, 53, 56, 81, 61, 27, 33, 73, 52, 63, 61, 65, 83, 59, 31, 67, 64, 50, 89])\n",
"yvec = np.array([699, 808, 778, 888, 640, 764, 863, 818, 643, 790, 479, 844, 733, 907, 766, 690, 908, 750, 677, 1014, 771, 791, 663, 812, 709, 1091, 625, 807, 998, 764, 712, 808, 757, 638, 825, 783, 568, 1016, 990, 733, 915, 919, 1047, 931, 809, 948, 1131, 945, 605, 883, 1086, 938, 973, 763, 775, 932, 730, 887, 990, 754, 824, 699, 953, 775, 786, 1039, 729, 944, 786, 829, 919, 832, 916, 893, 846, 863, 750, 808, 739, 796, 871, 668, 665, 695, 1011, 863, 621, 622, 828, 757, 840, 800, 805, 1007, 883, 598, 834, 881, 792, 979])\n",
"\n",
"fig = plt.figure(); plt.scatter(xvec, yvec);\n",
"plt.xlabel(\"apartment size in sqm\"); plt.ylabel(\"monthly rent in €\");"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "-6VyChDphaAf"
},
"source": [
"As expected, there seems to be a clear correlation between the size of an apartment and its monthly rent. We want to determine the precise nature of that relation by approximating the data using a linear function $f_{a,b}(x) = a \\cdot x + b$. We want to find a combination of parameters $a,b \\in \\mathbb{R}$ that reduces the loss $C(a,b)$ as given by the *Mean Squared Error*\n",
"\n",
"$$C_{{\\bf x}, {\\bf y}}(a,b) = C(a,b) = \\frac{1}{n} \\sum_{i = 1}^n \\big( f_{a,b}(x_i) - y_i \\big)^2.$$\n",
"\n",
"Here $x_i$ and $y_i$ correspond to the $i$-th entries of $\\bf x$ and $\\bf y$ and $n$ is the length of both vectors.\n",
"\n",
"### Task 1: Implement the functions $f_{a,b}$ and $C(a,b)$\n",
"\n",
"Let us implement both the function $f_{a,b}$ and the loss function $C(a,b)$. Make sure that `f(x, a, b)` accepts both scalars and numpy arrays for `x` and that it returns an array of equal length. In the function `C(a, b)` you may use both `xvec` and `yvec` as we have loaded them in the previous cell as well as the function `f(x, a, b)` that you just implemented. If you are unfamiliar with Numpy arrays, you can find all the basic operations you will need summarized [here](https://scipy-lectures.org/intro/numpy/operations.html)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "4HXsl5zcsMcP"
},
"outputs": [],
"source": [
"def f(x, a, b):\n",
" # FILL #####################################\n",
" return a * x + b\n",
" ############################################\n",
"\n",
"def C(a, b):\n",
" # FILL #####################################\n",
" return np.mean( (f(xvec, a, b) - yvec)**2 )\n",
" ############################################"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Run the following cell to verify that your implementation is correct:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"assert f(1, 2, -1) == 1\n",
"assert (f(xvec, 1, 0) == xvec).all()\n",
"assert C(1, 2) == 586722.18"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "tLuBWgV9UnTb"
},
"source": [
"Try some different parameters for $a$ and $b$ and see how well they fit the data!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 285
},
"colab_type": "code",
"id": "5mySZbsZUmCu",
"outputId": "d34b09e8-7715-4ab5-857b-c31e3809a0cb"
},
"outputs": [],
"source": [
"# TRY SOME PARAMETERS ######################\n",
"a, b = -5, 1200\n",
"############################################\n",
"\n",
"fig, ax = plt.subplots(); ax.scatter(xvec, yvec); ax.plot(xvec, f(xvec, a, b), color='red');\n",
"ax.set_xlabel(\"apartment size in sqm\"); ax.set_ylabel(\"monthly rent in €\");\n",
"ax.text(0.05, 0.95, f\"C({a},{b}) = {C(a, b)}\", transform=ax.transAxes, fontsize=14, verticalalignment='top');"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "CNV8P9civrRa"
},
"source": [
"### Task 2: Determine the partial derivatives\n",
"\n",
"In order to take a Gradient Descent step, we need to determine the partial derivatives of the loss function $C(a,b)$ with respect to the parameters $a$ and $b$. Determine formulas for $\\partial C / \\partial a$ and $\\partial C / \\partial b$ and implement them below. Note that, since we will later both implement batched updates and normalize the input data, your functions should be able to determine the derivatives for arbitrary given arrays `x` and `y` and not just for the particular arrays `xvec` and `yvec`. You can again use the function `f(x, a, b)` that you just implemented.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "XxCD2MqjvZWU"
},
"outputs": [],
"source": [
"def dCda(x, y, a, b):\n",
" # FILL #####################################\n",
" return np.mean(2 * x * (f(x, a, b) - y))\n",
" ############################################\n",
"\n",
"def dCdb(x, y, a, b):\n",
" # FILL #####################################\n",
" return np.mean(2 * (f(x, a, b) - y))\n",
" ############################################"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Run the following cell to verify that your implementation is correct:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"assert dCda(np.array([0]), ([0]), 0, 0) == 0.0\n",
"assert dCda(xvec, yvec, 1, 0) == -94936.46\n",
"assert dCdb(np.array([0]), ([0]), 0, 0) == 0.0\n",
"assert dCdb(xvec, yvec, 2, 10) == -1377.78"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "nsONEh5SlPAw"
},
"source": [
"### Task 3: Normalize the input data\n",
"We now have almost all the ingredients we need in order to apply Gradient Descent to our problem. As one last piece of preparation, we will need to normalize our input vector $\\bf x$, that is we will subtract the mean of $\\bf x$ from each entry and divide the resulting vector by its standard deviation. Written formally, we want to determine\n",
"\n",
"$${\\bf x}_{\\text{normalized}} = \\frac{{\\bf x} - \\sum_{i = 1}^n x_i / n}{\\sqrt{ \\frac{\\sum_{i=1}^n(x_i - \\sum_{i = 1}^n x_i / n)^2}{n} }}.$$\n",
"\n",
"Normalization is a technique that helps us use a common scale for both parameters $a$ and $b$ so that we can use a common learning rate for both. The resulting vector `xvec_normalized` should have mean value $0$ and standard deviation $1$. You can use some built-in functionality from Numpy to make your life easier."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "Qbj65ri0eRnQ"
},
"outputs": [],
"source": [
"# FILL #####################################\n",
"xvec_normalized = ( xvec - np.mean(xvec) ) / np.std(xvec)\n",
"############################################"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Run the following cell to verify that your implementation is correct:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"assert np.abs( np.mean(xvec_normalized) ) < 1e-10\n",
"assert np.std(xvec_normalized) == 1"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "dGohadWTc1QK"
},
"source": [
"Before proceeding to the next task, let just need to define a simple logger object to store information during intermediate steps."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "8-L2HtTbmSDZ"
},
"outputs": [],
"source": [
"class Logger:\n",
" def __init__(self):\n",
" self.loss_history = []\n",
" self.a_history = []\n",
" self.b_history = []\n",
" \n",
" def log(self, a, b):\n",
" a_real, b_real = a / 14.90147, b - a * 60.69 / 14.90147\n",
" self.loss_history.append(C(a_real, b_real))\n",
" self.a_history.append(a_real)\n",
" self.b_history.append(b_real)\n",
"\n",
" def plot(self):\n",
" fig, ax = plt.subplots(1, 4); fig.set_size_inches(25, 4);\n",
" ax[0].scatter(xvec, yvec); ax[0].plot(xvec, f(xvec, self.a_history[-1], self.b_history[-1]), color='red')\n",
" ax[0].set_xlabel(\"apartment size in sqm\"); ax[0].set_ylabel(\"monthly rent in €\");\n",
" ax[1].plot(range(len(self.loss_history)), self.loss_history, color='green'); \n",
" ax[1].set_title(\"loss\", fontsize='large'); ax[1].set_xlabel(\"steps\"); ax[1].set_yscale(\"log\")\n",
" ax[2].plot(range(len(self.a_history)), self.a_history, color='orange')\n",
" ax[2].set_title(\"parameter a\", fontsize='large'); ax[2].set_xlabel(\"steps\")\n",
" ax[3].plot(range(len(self.b_history)), self.b_history, color='orange')\n",
" ax[3].set_title(\"parameter b\", fontsize='large'); ax[3].set_xlabel(\"steps\")\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "iEJLY2jxdDKc"
},
"source": [
"### Task 4: Implement Gradient Descent updates\n",
"\n",
"We are now ready to implement Gradient Descent, which simply takes the Gradient of the loss function, pointing in the direction of steepest incline, and takes a step in the opposite direction. Written formally, we initialize some parameters $a_0$ and $b_0$ and iteratively update them through\n",
"\n",
"$$a_{t+1} = a_t - \\gamma \\, \\partial C (a_t, b_t)/ \\partial a \\quad \\text{and} \\quad b_{t+1} = b_t - \\gamma \\, \\partial C (a_t, b_t) / \\partial b.$$\n",
"\n",
"As you can see below, we have already initialized the parameters `a` and `b` to be `0` and set up a loop going over `nr_epochs = 10` epochs, that is steps $t$. The paramter $\\gamma$ is called `learning_rate` and is set to `0.3`, but feel free to experiment with different values both for the learning rate and the number of epochs. An instance of `Logger` is created to store and plot the values at each iterate. Make sure you always use `xvec_normalized` instead of `xvec`!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 273
},
"colab_type": "code",
"id": "z6jVuYw9dL_g",
"outputId": "c5f0f692-a8de-4119-b88f-84274e35c63f"
},
"outputs": [],
"source": [
"logger = Logger()\n",
"\n",
"learning_rate = 0.3\n",
"nr_epochs = 10\n",
"a, b = 0, 0\n",
"\n",
"for epoch in range(nr_epochs):\n",
" # FILL ###################\n",
" da = dCda(xvec_normalized, yvec, a, b)\n",
" db = dCdb(xvec_normalized, yvec, a, b)\n",
" a -= learning_rate * da\n",
" b -= learning_rate * db\n",
" ##########################\n",
"\n",
" logger.log(a, b)\n",
"\n",
"logger.plot()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Kv6ch9wzPc8y"
},
"source": [
"You should see the loss decrease over time and the values for $a$ and $b$ to converge to around 7.16 and 385, that is for every additional square meter of living space, you should expect to pay an additional 7,16€ per month in rent."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "5HHvp5N2tMkd"
},
"source": [
"### Task 5: Implement batched Gradient Descent updates\n",
"\n",
"Finally, let us modify the previous loop slightly to implement batched Gradient Descent. Instead of determining the gradient of the loss function with respect to the entire dataset $(\\bf x, \\bf y)$ each time, we split our data into equally sized so-called batches $({\\bf x}_0, {\\bf y}_0), ..., ({\\bf x}_k, {\\bf y}_k)$ and update our parameters using the gradient only with respect to each individual batch. More formally, we now have an additional *inner loop* of size $k+1$ where we pass over all batches $({\\bf x}_i, {\\bf y}_i)$ and update our parameters as\n",
"\n",
"$$a_{t+1} = a_t - \\gamma \\, \\partial C_{{\\bf x}_i, {\\bf y}_i} (a_t, b_t) / \\partial a \\quad \\text{and} \\quad b_{t+1} = b_t - \\gamma \\, \\partial C_{{\\bf x}_i, {\\bf y}_i} (a_t, b_t) / \\partial b.$$\n",
"\n",
"Besides dividing the data into several batches, you should also shuffle it at the beginning of each epoch so that you do not use the same batches each epoch. Look up `np.random.permutation` for one possible way of how to achieve this in Numpy. We have again already set up some parameters, including a batch size of 20, but feel free to experiment with different combinations. When you set `batch_size = len(yvec)`, your code should run as in the previous cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 273
},
"colab_type": "code",
"id": "iI7mUUTiZuJC",
"outputId": "5d46d497-95a4-4aa0-e674-500682a5076c"
},
"outputs": [],
"source": [
"logger = Logger()\n",
"\n",
"learning_rate = 0.1\n",
"nr_epochs = 10\n",
"batch_size = 20\n",
"a, b = 0, 0\n",
"\n",
"for epoch in range(nr_epochs):\n",
" # FILL ###################\n",
" random_permutation = np.random.permutation(len(yvec))\n",
" xvec_normalized_shuffled = xvec_normalized[random_permutation]\n",
" yvec_shuffled = yvec[random_permutation]\n",
" ##########################\n",
"\n",
" # FILL ###################\n",
" for i in range(math.ceil(len(yvec)/batch_size)):\n",
" x_batch = xvec_normalized_shuffled[i*batch_size:(i+1)*batch_size]\n",
" y_batch = yvec_shuffled[i*batch_size:(i+1)*batch_size]\n",
" ##########################\n",
"\n",
" # FILL ###################\n",
" da = dCda(x_batch, y_batch, a, b)\n",
" db = dCdb(x_batch, y_batch, a, b)\n",
" a -= learning_rate * da\n",
" b -= learning_rate * db\n",
" ##########################\n",
"\n",
" logger.log(a, b)\n",
"\n",
"logger.plot()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Gn0Ltm1hS-SK"
},
"source": [
"# Example 2 – Handwriting Recognition in TensorFlow\n",
"\n",
"We are now going to implement Conditional Gradient Descent for a real Machine Learning example: training a Neural Network to recognize handwritten numbers. As before, we will first load some necessary libraries."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "H5JrMg0BUgwE"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"import tensorflow_datasets as tfds\n",
"import numpy as np\n",
"import math\n",
"\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "xQaCaXQS1fBS"
},
"source": [
"The dataset we will use is the [MNIST database of handwritten digits](http://yann.lecun.com/exdb/mnist/). It is a common easy training example where even simple classifier functions can achieve very high accuracy in a short time. Let us load the dataset and have a look at it. The second of the following two cells displays five images at random from the dataset of 60 000 images. Re-run it to get five different images."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "DMn31xM2UjQB"
},
"outputs": [],
"source": [
"dataset, info = tfds.load('mnist', data_dir='./dataset', as_supervised=True, with_info=True)\n",
"train_dataset, test_dataset = dataset['train'], dataset['test']\n",
"\n",
"mnist_raw_input = next(iter(train_dataset.batch(5000)))[0] / 255\n",
"\n",
"num_train_examples = info.splits['train'].num_examples\n",
"num_test_examples = info.splits['test'].num_examples"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 143
},
"colab_type": "code",
"id": "oZmtbrc4lk8z",
"outputId": "3f83f1f9-b2e6-4591-c928-b8e809ebe379"
},
"outputs": [],
"source": [
"fig, ax = plt.subplots(1,5); fig.set_size_inches(10, 2);\n",
"\n",
"for idx, (image, label) in enumerate(iter(train_dataset.shuffle(100))):\n",
" ax[idx].imshow(np.repeat(image, 3, axis=2)); ax[idx].set_title(label.numpy())\n",
" ax[idx].get_xaxis().set_visible(False); ax[idx].get_yaxis().set_visible(False);\n",
" if idx == 4: break\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "lZleREtqtJd2"
},
"source": [
"Previously in the housing example we had a 1-dimensional input that we ran through a linear function with a *weight term* $a$ and a *bias term* $b$. This simple linear regression module is referred to as a *Neuron* in Machine Learning. The simple idea behind Neural Networks is to construct highly sophisticated functions by chaining and stacking a whole lot of these building blocks, each followed by a non-linear activation function such as the tanh, [sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function) or [relu](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) function. \n",
"\n",
"In particular, in our handwriting example we will consider the 28x28 input matrices as vectors of length $28 \\cdot 28 = 784$, so we are going from a 1- to a 784-dimensional input. We call this the *input layer* and it is connected to 64 neurons in the *hidden layer*, that is we run 64 distinct linear regressions, followed by non-linear relu activations, over each image. The resulting vector of length 64 is then connected to 10 neurons in the *output layer*, so that we ultimately end up mapping each image to a vector of length 10. There are 10 neurons in the output layer because we are trying to classify each image as one of the ten digits: the $i$-th entry of the vector will tell us the certainty of the image representing the digit $i$ where $0 \\leq i \\leq 9$. This output will in fact be a probability vector because we are applying a [softmax](https://en.wikipedia.org/wiki/Softmax_function) function on the output layer.\n",
"\n",
"Thankfully, modern frameworks like [TensorFlow](https://www.tensorflow.org) and [Pytorch](https://pytorch.org) make constructing such networks very simple. The following short code defines a class that instantiates precisely the type of function described in the previous paragraph. Not only that, but the class also takes care of determining the partial derivatives whenever feeding in any input. The following cell also prints a simple summary of the network, which tells us that it has a total of 50 890 trainable parameters. Our previous example had 2!\n",
"\n",
"### Task 1: Understanding our function\n",
"\n",
"See if you can figure out why exactly our neural network has 50 890 trainable parameters."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 272
},
"colab_type": "code",
"id": "wQp3Ha_gYUoG",
"outputId": "db51908e-8925-4a46-a4c3-2783fe5fe485"
},
"outputs": [],
"source": [
"class Model(tf.keras.models.Sequential):\n",
" def __init__(self):\n",
" super().__init__([\n",
" # flatten the 28x28 images into vectors of length 784\n",
" tf.keras.layers.Flatten(input_shape = (28, 28, 1)),\n",
" # a hidden layer with 64 neurons and relu activations\n",
" tf.keras.layers.Dense(64, activation='relu'),\n",
" # an output layer with 10 classes and softmax activation\n",
" tf.keras.layers.Dense(10, activation='softmax')\n",
" ])\n",
"\n",
"print(Model().summary())"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "nRnRmLg-qCis"
},
"source": [
"Where previously we used the mean squared error, we will now use a [categorical cross-entropy](https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_loss_function_and_logistic_regression) loss function. TensorFlow already has all the necessary fuctionality built into the object `tf.keras.losses.SparseCategoricalCrossentropy`. The next cell will instanciate such a loss object as well as two metrics objects that we will use to store both the loss and accuracy during epoch, that is the number of images from our dataset correctly classified.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "-f1eThFxo2Nw"
},
"outputs": [],
"source": [
"# define the loss object\n",
"loss_object = tf.keras.losses.SparseCategoricalCrossentropy()\n",
"\n",
"# objects used to store the loss and accuracy metrics\n",
"train_loss = tf.keras.metrics.Mean(name='train_loss')\n",
"train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy('train_accuracy')"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "iqi5Inf8TfcT"
},
"source": [
"### Task 2: Normalize input\n",
"\n",
"We will again normalize our input to have zero mean and a standard deviation of one. As before, determine the values `mnist_mean` and `mnist_std`. You can use the input data in the form of the 60000x28x28x1 array `mnist_raw_input` and you may want to look up the functions `tf.math.reduce_mean` and `tf.math.reduce_std`. If you cannot figure out how to determine these values, don't worry! Normalizing isn't strictly necessary here, so you can also just set `mnist_mean` to zero and `mnist_std` to one and proceed to the next part."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "OSWKq0MDTOfG"
},
"outputs": [],
"source": [
"# FILL ###################\n",
"mnist_mean = tf.math.reduce_mean(mnist_raw_input)\n",
"mnist_std = tf.math.reduce_std(mnist_raw_input - mnist_mean)\n",
"#########################\n",
"\n",
"def normalize(image, label):\n",
" # cast the image to [0,1] with dtype tf.float32\n",
" image = tf.image.convert_image_dtype(image, tf.float32)\n",
" # normalize to mean 0 and standard deviation 1 \n",
" image = (image - mnist_mean) / mnist_std \n",
" return image, label"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "WpSq4lnLZfrA"
},
"source": [
"Let us see how our neural network performs before being trained. The following defines a training function that accepts a batch consisting of `x` and `y` and returns the corresponding ouput of the network, the associated loss and the gradient. This cell might take a few seconds to run as it determines the networks prediction for all 60 000 images in the dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "By7uoxc3ZdYk",
"outputId": "6ba751c0-559d-4002-d305-9ba7d6952f8f"
},
"outputs": [],
"source": [
"# function returning the output, loss and partial derivatives for a single given a batch (x,y)\n",
"@tf.function\n",
"def train_step(model, x, y):\n",
" with tf.GradientTape() as tape:\n",
" # feed the input x through the model\n",
" output = model(x, training=True)\n",
" # determine the loss by comparing the output to y\n",
" loss = loss_object(y, output)\n",
"\n",
" # determine the gradient w.r.t. the loss\n",
" grads = tape.gradient(loss, model.trainable_variables)\n",
"\n",
" return output, loss, grads\n",
"\n",
"\n",
"# determine the performance of an untrained network\n",
"model = Model()\n",
"import time\n",
"t = time.time()\n",
"for x, y in train_dataset.shuffle(num_train_examples).map(normalize).batch(64):\n",
" output, loss, _ = train_step(model, x, y)\n",
" train_loss.update_state(loss, sample_weight=len(y))\n",
" train_accuracy.update_state(y, output, sample_weight=len(y))\n",
"print(time.time() - t)\n",
"print(f\"The untrained network has a loss of {train_loss.result().numpy():.2f} and an accuracy of {train_accuracy.result().numpy()*100:.2f}%.\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ealFU5qXqAlh"
},
"source": [
"The accuracy should be around 10%, which simply means that our untrained network randomly maps each image to one of the ten digits. Let's see if we can improve that! The following defines a training loop exactly as we did previously in the housing example, though this time using some of the handy shortcuts provided by TensorFlow."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "LAhetcSjt0eC"
},
"outputs": [],
"source": [
"import time\n",
"\n",
"def train(model, optimizer, nepochs=5, batch_size=64):\n",
" for epoch in range(nepochs):\n",
" t = time.time()\n",
" # reset metrics at beginning of each epoch\n",
" train_loss.reset_states()\n",
" train_accuracy.reset_states()\n",
"\n",
" # normalize, shuffle and batch the input data\n",
" shuffled_batched_dataset = train_dataset.map(normalize).shuffle(num_train_examples).batch(batch_size)\n",
"\n",
" # train loop\n",
" for x, y in shuffled_batched_dataset:\n",
" # get the predicted output, associated loss and partial derivatives for the current batch (x,y)\n",
" output, loss, grads = train_step(model, x, y)\n",
"\n",
" # update the loss and accuracy metrics\n",
" train_loss.update_state(loss, sample_weight=len(y))\n",
" train_accuracy.update_state(y, output, sample_weight=len(y))\n",
"\n",
" # use the partial derivatives to update our network\n",
" optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
"\n",
" # print metrics of current epoch\n",
" print(time.time() - t)\n",
" print(f\"{epoch + 1}) loss {train_loss.result().numpy():.2f} and accuracy {train_accuracy.result().numpy()*100:.2f}%\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "V-At9jintTpU"
},
"source": [
"### Task 3: Train the network using Gradient Descent\n",
"\n",
"Stochastic Gradient Descent is already implemented in TensorFlow in the form of `tf.keras.optimizers.SGD`. Note though that you still have to specify your own learning rate! In the following cell, instanciate the model and the optimizer and use the just defined `train` function to train the network using the optimizer. We have already suggested some values for the batch size, learning rate and number of epochs, but feel free to experiment!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 102
},
"colab_type": "code",
"id": "am2fqhmFcpT6",
"outputId": "a4dd09d1-2f11-46e9-891c-a67c73bef899"
},
"outputs": [],
"source": [
"learning_rate = 0.1\n",
"nepochs = 5\n",
"batch_size = 64\n",
"\n",
"# FILL ###################################\n",
"sgd_trained_model = Model()\n",
"sgd_optimizer = tf.keras.optimizers.SGD(learning_rate=learning_rate)\n",
"train(sgd_trained_model, sgd_optimizer, nepochs=nepochs, batch_size=batch_size)\n",
"##########################################"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "OCV6asWvs88N"
},
"source": [
"Even withing the first epoch you can see an accuracy of over 90%!\n",
"\n",
"### Task 4: Implement stochastic Conditional Gradient Descent\n",
"\n",
"We are now ready to implement Conditional Gradient Descent. Recall that given some differentiable function $f$ that we want to minimize and a feasible region $\\mathcal{D}$ in which our trainable parameters ${\\bf w}_t$ must lie, Conditional Gradient Descent first determines the $\\bf v \\in \\mathcal{D}$ that minimizes\n",
"\n",
"$$\\langle {\\bf v}, \\nabla f ({\\bf w}_t)\\rangle.$$\n",
"\n",
"Conditional Gradient Descent then updates the parameters by taking a step towards $\\bf v$, that is \n",
"\n",
"$${\\bf w}_{t+1} = {\\bf w}_t + \\gamma ({\\bf v} - {\\bf w}_t).$$\n",
"\n",
"Here $\\gamma$ usually is set to be $2/(t+1)$, though for us it will be a constant `learning_rate` that needs to be between $0$ and $1$.\n",
"\n",
"\n",
"\n",
"The feasible region $\\mathcal{D}$ we will use here is a very simple one: the unit hypercube, that is $\\mathcal{D} = \\{{\\bf x} : ||{\\bf x}||_\\infty \\leq 1\\}$. You should first derive a closed formula for how to determine $\\text{argmin}_{{\\bf v} \\in \\mathcal{D}}\\langle {\\bf v}, \\nabla f ({\\bf w}_t)\\rangle$ when $\\mathcal{D}$ is the unit hypercube. You should then implement the update in the following code skeleton below. We are relying on the TensorFlow optimizer template `tf.keras.optimizers.Optimizer`. This might look intimidating at first, but the only thing you need to implement happens in the method `resource_apply_dense`. Determine $\\bf v$ minimizing $\\langle {\\bf v}, \\nabla f ({\\bf w}_t)\\rangle$ over $\\mathcal{D}$ using the gradient `grad` and then update the weights $\\bf w$ which are stored in the variable `var`. You may need to use the functions `tf.sign` and `tf.cast` and in order to update `var`, TensorFlow will require you to use `.assign` or `.assign_sub` instead of assigning a value through `=` as you normally would."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "HHmsFajlTNhS"
},
"outputs": [],
"source": [
"class CGD(tf.keras.optimizers.Optimizer):\n",
"\n",
" def __init__(self, learning_rate=0.03, name=\"CGD\"):\n",
" super().__init__(name)\n",
" self._set_hyper(\"learning_rate\", learning_rate)\n",
"\n",
" def _resource_apply_dense(self, grad, var, apply_state):\n",
" lr = self._get_hyper('learning_rate')\n",
"\n",
" # FILL ##############\n",
" v = -tf.cast( tf.sign(grad), grad.dtype )\n",
" return var.assign(var + lr * (v - var))\n",
" ##################### \n",
"\n",
" def get_config(self):\n",
" config = super().get_config()\n",
" config.update({ \"learning_rate\": self._serialize_hyperparameter(\"learning_rate\") })\n",
" return config"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "IhckucqPcq1P"
},
"source": [
"### Task 5: Train the network using Conditional Gradient Descent\n",
"\n",
"You can now train a model using Conditional Gradient Descent exactly the same way you did before."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 102
},
"colab_type": "code",
"id": "uLLEJcQMkmfo",
"outputId": "262d8c95-2949-400b-8434-ff62f605b9ea"
},
"outputs": [],
"source": [
"learning_rate = 0.001\n",
"nepochs = 5\n",
"batch_size = 64\n",
"\n",
"# FILL ##############\n",
"cgd_trained_model = Model()\n",
"cgd_optimizer = CGD(learning_rate=learning_rate)\n",
"train(cgd_trained_model, cgd_optimizer, nepochs=nepochs, batch_size=batch_size)\n",
"##################### "
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "XQCDM32QdWc-"
},
"source": [
"## Additional Exercises\n",
"\n",
"### Task 1: Taking more features into account in Example 1\n",
"\n",
"Imagine that you obtained additional information about the apartments in Example 1. On top of the size of the apartment, you now know the distance of each apartment to the nearest U-Bahn, S-Bahn or bus station in minutes. Can you optimize the parameters a linear function taking both features as an input? How much does it improve the accuracy? The additional data is given below, you may reuse any of the code from Example 1."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"distances_vec = np.array([1, 11, 12, 17, 14, 8, 16, 10, 19, 11, 12, 10, 8, 1, 18, 13, 6, 7, 15, 1, 15, 8, 13, 6, 19, 3, 4, 6, 9, 8, 6, 12, 10, 17, 18, 17, 1, 13, 15, 11, 1, 6, 12, 11, 9, 7, 2, 18, 4, 12, 7, 13, 18, 7, 11, 18, 4, 8, 5, 2, 16, 3, 8, 6, 10, 9, 8, 4, 13, 6, 15, 6, 5, 7, 6, 18, 16, 15, 7, 1, 14, 12, 13, 12, 11, 16, 2, 7, 14, 4, 13, 10, 10, 1, 6, 18, 19, 17, 5, 13])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Task 2: Try out different Networks in Tensorflow\n",
"\n",
"If you already have some familiarity with Neural Networks and Machine Learning, then you should try out different network designs in Example 2. Can you train a Convolutional Neural Network? You may find this overview of layers already implemented in Keras helpful: [link](https://keras.io/api/layers/)."
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "co@work.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "tensorflow",
"language": "python",
"name": "tensorflow"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
}
},
"nbformat": 4,
"nbformat_minor": 1
}