handson-ml/10_neural_nets_with_keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Chapter 10 – Introduction to Artificial Neural Networks with Keras**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains all the sample code and solutions to the exercises in chapter 10._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/ageron/handson-ml3/blob/main/10_neural_nets_with_keras.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://kaggle.com/kernels/welcome?src=https://github.com/ageron/handson-ml3/blob/main/10_neural_nets_with_keras.ipynb\"><img src=\"https://kaggle.com/static/images/open-in-kaggle.svg\" /></a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This project requires Python 3.8 or above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "assert sys.version_info >= (3, 8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It also requires Scikit-Learn ≥ 1.0.1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "\n",
    "assert sklearn.__version__ >= \"1.0.1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And TensorFlow ≥ 2.6:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "assert tf.__version__ >= \"2.6.0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we did in previous chapters, let's define the default font sizes to make the figures prettier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.rc('font', size=14)\n",
    "plt.rc('axes', labelsize=14, titlesize=14)\n",
    "plt.rc('legend', fontsize=14)\n",
    "plt.rc('xtick', labelsize=10)\n",
    "plt.rc('ytick', labelsize=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's create the `images/ann` folder (if it doesn't already exist), and define the `save_fig()` function which is used through this notebook to save the figures in high-res for the book:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "IMAGES_PATH = Path() / \"images\" / \"ann\"\n",
    "IMAGES_PATH.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = IMAGES_PATH / f\"{fig_id}.{fig_extension}\"\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# From Biological to Artificial Neurons\n",
    "## The Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.linear_model import Perceptron\n",
    "\n",
    "iris = load_iris(as_frame=True)\n",
    "X = iris.data[[\"petal length (cm)\", \"petal width (cm)\"]].values\n",
    "y = (iris.target == 0)  # Iris setosa\n",
    "\n",
    "per_clf = Perceptron(random_state=42)\n",
    "per_clf.fit(X, y)\n",
    "\n",
    "X_new = [[2, 0.5], [3, 1]]\n",
    "y_pred = per_clf.predict(X_new)  # predicts True and False for these 2 flowers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Perceptron` is equivalent to a `SGDClassifier` with `loss=\"perceptron\"`, no regularization, and a constant learning rate equal to 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – extra material\n",
    "\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "sgd_clf = SGDClassifier(loss=\"perceptron\", penalty=None,\n",
    "                        learning_rate=\"constant\", eta0=1, random_state=42)\n",
    "sgd_clf.fit(X, y)\n",
    "assert (sgd_clf.coef_ == per_clf.coef_).all()\n",
    "assert (sgd_clf.intercept_ == per_clf.intercept_).all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the Perceptron finds a decision boundary that properly separates the classes, it stops learning. This means that the decision boundary is often quite close to one class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – extra material\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.colors import ListedColormap\n",
    "\n",
    "a = -per_clf.coef_[0, 0] / per_clf.coef_[0, 1]\n",
    "b = -per_clf.intercept_ / per_clf.coef_[0, 1]\n",
    "axes = [0, 5, 0, 2]\n",
    "x0, x1 = np.meshgrid(\n",
    "    np.linspace(axes[0], axes[1], 500).reshape(-1, 1),\n",
    "    np.linspace(axes[2], axes[3], 200).reshape(-1, 1),\n",
    ")\n",
    "X_new = np.c_[x0.ravel(), x1.ravel()]\n",
    "y_predict = per_clf.predict(X_new)\n",
    "zz = y_predict.reshape(x0.shape)\n",
    "custom_cmap = ListedColormap(['#9898ff', '#fafab0'])\n",
    "\n",
    "plt.figure(figsize=(7, 3))\n",
    "plt.plot(X[y == 0, 0], X[y == 0, 1], \"bs\", label=\"Not Iris setosa\")\n",
    "plt.plot(X[y == 1, 0], X[y == 1, 1], \"yo\", label=\"Iris setosa\")\n",
    "plt.plot([axes[0], axes[1]], [a * axes[0] + b, a * axes[1] + b], \"k-\",\n",
    "         linewidth=3)\n",
    "plt.contourf(x0, x1, zz, cmap=custom_cmap)\n",
    "plt.xlabel(\"Petal length\")\n",
    "plt.ylabel(\"Petal width\")\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.axis(axes)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Activation functions**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – this code generates and saves Figure 10–8\n",
    "\n",
    "from scipy.special import expit as sigmoid\n",
    "\n",
    "def relu(z):\n",
    "    return np.maximum(0, z)\n",
    "\n",
    "def derivative(f, z, eps=0.000001):\n",
    "    return (f(z + eps) - f(z - eps))/(2 * eps)\n",
    "\n",
    "max_z = 4.5\n",
    "z = np.linspace(-max_z, max_z, 200)\n",
    "\n",
    "plt.figure(figsize=(11, 3.1))\n",
    "\n",
    "plt.subplot(121)\n",
    "plt.plot([-max_z, 0], [0, 0], \"r-\", linewidth=2, label=\"Heaviside\")\n",
    "plt.plot(z, relu(z), \"m-.\", linewidth=2, label=\"ReLU\")\n",
    "plt.plot([0, 0], [0, 1], \"r-\", linewidth=0.5)\n",
    "plt.plot([0, max_z], [1, 1], \"r-\", linewidth=2)\n",
    "plt.plot(z, sigmoid(z), \"g--\", linewidth=2, label=\"Sigmoid\")\n",
    "plt.plot(z, np.tanh(z), \"b-\", linewidth=1, label=\"Tanh\")\n",
    "plt.grid(True)\n",
    "plt.title(\"Activation functions\")\n",
    "plt.axis([-max_z, max_z, -1.65, 2.4])\n",
    "plt.gca().set_yticks([-1, 0, 1, 2])\n",
    "plt.legend(loc=\"lower right\", fontsize=13)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.plot(z, derivative(np.sign, z), \"r-\", linewidth=2, label=\"Heaviside\")\n",
    "plt.plot(0, 0, \"ro\", markersize=5)\n",
    "plt.plot(0, 0, \"rx\", markersize=10)\n",
    "plt.plot(z, derivative(sigmoid, z), \"g--\", linewidth=2, label=\"Sigmoid\")\n",
    "plt.plot(z, derivative(np.tanh, z), \"b-\", linewidth=1, label=\"Tanh\")\n",
    "plt.plot([-max_z, 0], [0, 0], \"m-.\", linewidth=2)\n",
    "plt.plot([0, max_z], [1, 1], \"m-.\", linewidth=2)\n",
    "plt.plot([0, 0], [0, 1], \"m-.\", linewidth=1.2)\n",
    "plt.plot(0, 1, \"mo\", markersize=5)\n",
    "plt.plot(0, 1, \"mx\", markersize=10)\n",
    "plt.grid(True)\n",
    "plt.title(\"Derivatives\")\n",
    "plt.axis([-max_z, max_z, -0.2, 1.2])\n",
    "\n",
    "save_fig(\"activation_functions_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regression MLPs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neural_network import MLPRegressor\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "housing = fetch_california_housing()\n",
    "X_train_full, X_test, y_train_full, y_test = train_test_split(\n",
    "    housing.data, housing.target, random_state=42)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "    X_train_full, y_train_full, random_state=42)\n",
    "\n",
    "mlp_reg = MLPRegressor(hidden_layer_sizes=[50, 50, 50], random_state=42)\n",
    "pipeline = make_pipeline(StandardScaler(), mlp_reg)\n",
    "pipeline.fit(X_train, y_train)\n",
    "y_pred = pipeline.predict(X_valid)\n",
    "rmse = mean_squared_error(y_valid, y_pred, squared=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification MLPs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – this was left as an exercise for the reader\n",
    "\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "\n",
    "iris = load_iris()\n",
    "X_train_full, X_test, y_train_full, y_test = train_test_split(\n",
    "    iris.data, iris.target, test_size=0.1, random_state=42)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "    X_train_full, y_train_full, test_size=0.1, random_state=42)\n",
    "\n",
    "mlp_clf = MLPClassifier(hidden_layer_sizes=[5], max_iter=10_000,\n",
    "                        random_state=42)\n",
    "pipeline = make_pipeline(StandardScaler(), mlp_clf)\n",
    "pipeline.fit(X_train, y_train)\n",
    "accuracy = pipeline.score(X_valid, y_valid)\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementing MLPs with Keras\n",
    "## Building an Image Classifier Using the Sequential API\n",
    "### Using Keras to load the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by loading the fashion MNIST dataset. Keras has a number of functions to load popular datasets in `tf.keras.datasets`. The dataset is already split for you between a training set (60,000 images) and a test set (10,000 images), but it can be useful to split the training set further to have a validation set. We'll use 55,000 images for training, and 5,000 for validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "fashion_mnist = tf.keras.datasets.fashion_mnist.load_data()\n",
    "(X_train_full, y_train_full), (X_test, y_test) = fashion_mnist\n",
    "X_train, y_train = X_train_full[:-5000], y_train_full[:-5000]\n",
    "X_valid, y_valid = X_train_full[-5000:], y_train_full[-5000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training set contains 60,000 grayscale images, each 28x28 pixels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each pixel intensity is represented as a byte (0 to 255):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's scale the pixel intensities down to the 0-1 range and convert them to floats, by dividing by 255:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_valid, X_test = X_train / 255., X_valid / 255., X_test / 255."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can plot an image using Matplotlib's `imshow()` function, with a `'binary'`\n",
    " color map:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book\n",
    "\n",
    "plt.imshow(X_train[0], cmap=\"binary\")\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The labels are the class IDs (represented as uint8), from 0 to 9:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the corresponding class names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_names = [\"T-shirt/top\", \"Trouser\", \"Pullover\", \"Dress\", \"Coat\",\n",
    "               \"Sandal\", \"Shirt\", \"Sneaker\", \"Bag\", \"Ankle boot\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the first image in the training set is an ankle boot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_names[y_train[0]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at a sample of the images in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – this cell generates and saves Figure 10–10\n",
    "\n",
    "n_rows = 4\n",
    "n_cols = 10\n",
    "plt.figure(figsize=(n_cols * 1.2, n_rows * 1.2))\n",
    "for row in range(n_rows):\n",
    "    for col in range(n_cols):\n",
    "        index = n_cols * row + col\n",
    "        plt.subplot(n_rows, n_cols, index + 1)\n",
    "        plt.imshow(X_train[index], cmap=\"binary\", interpolation=\"nearest\")\n",
    "        plt.axis('off')\n",
    "        plt.title(class_names[y_train[index]])\n",
    "plt.subplots_adjust(wspace=0.2, hspace=0.5)\n",
    "\n",
    "save_fig(\"fashion_mnist_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the model using the Sequential API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "model = tf.keras.Sequential()\n",
    "model.add(tf.keras.layers.InputLayer(input_shape=[28, 28]))\n",
    "model.add(tf.keras.layers.Flatten())\n",
    "model.add(tf.keras.layers.Dense(300, activation=\"relu\"))\n",
    "model.add(tf.keras.layers.Dense(100, activation=\"relu\"))\n",
    "model.add(tf.keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – clear the session to reset the name counters\n",
    "tf.keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    tf.keras.layers.Dense(300, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(100, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – another way to display the model's architecture\n",
    "tf.keras.utils.plot_model(model, \"my_fashion_mnist_model.png\", show_shapes=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "hidden1 = model.layers[1]\n",
    "hidden1.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.get_layer('dense') is hidden1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights, biases = hidden1.get_weights()\n",
    "weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "biases.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compiling the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=\"sgd\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is equivalent to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – this cell is equivalent to the previous cell\n",
    "model.compile(loss=tf.keras.losses.sparse_categorical_crossentropy,\n",
    "              optimizer=tf.keras.optimizers.SGD(),\n",
    "              metrics=[tf.keras.metrics.sparse_categorical_accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – this code shows how to convert class ids to one-hot vectors\n",
    "tf.keras.utils.to_categorical([0, 5, 1, 0], num_classes=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: it's important to set `num_classes` when the number of classes is greater than the maximum class id in the sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – this code shows how to convert one-hot vectors to class ids\n",
    "np.argmax(\n",
    "    [[1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
    "     [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],\n",
    "     [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
    "     [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.]],\n",
    "    axis=1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and evaluating the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(X_train, y_train, epochs=30,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "history.params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(history.epoch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "pd.DataFrame(history.history).plot(\n",
    "    figsize=(8, 5), xlim=[0, 29], ylim=[0, 1], grid=True, xlabel=\"Epoch\",\n",
    "    style=[\"r--\", \"r--.\", \"b-\", \"b-*\"])\n",
    "plt.legend(loc=\"lower left\")  # not in the book\n",
    "save_fig(\"keras_learning_curves_plot\")  # not in the book\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – shows how to shift the training curve by -1/2 epoch\n",
    "plt.figure(figsize=(8, 5))\n",
    "for key, style in zip(history.history, [\"r--\", \"r--.\", \"b-\", \"b-*\"]):\n",
    "    epochs = np.array(history.epoch) + (0 if key.startswith(\"val_\") else -0.5)\n",
    "    plt.plot(epochs, history.history[key], style, label=key)\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.axis([-0.5, 29, 0., 1])\n",
    "plt.legend(loc=\"lower left\")\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the model to make predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = X_test[:3]\n",
    "y_proba = model.predict(X_new)\n",
    "y_proba.round(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = y_proba.argmax(axis=-1)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.array(class_names)[y_pred]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_new = y_test[:3]\n",
    "y_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – this cell generates and saves Figure 10–12\n",
    "plt.figure(figsize=(7.2, 2.4))\n",
    "for index, image in enumerate(X_new):\n",
    "    plt.subplot(1, 3, index + 1)\n",
    "    plt.imshow(image, cmap=\"binary\", interpolation=\"nearest\")\n",
    "    plt.axis('off')\n",
    "    plt.title(class_names[y_test[index]])\n",
    "plt.subplots_adjust(wspace=0.2, hspace=0.5)\n",
    "save_fig('fashion_mnist_images_plot', tight_layout=False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a Regression MLP Using the Sequential API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load, split and scale the California housing dataset (the original one, not the modified one as in chapter 2):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – load and split the California housing dataset, like earlier\n",
    "housing = fetch_california_housing()\n",
    "X_train_full, X_test, y_train_full, y_test = train_test_split(\n",
    "    housing.data, housing.target, random_state=42)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "    X_train_full, y_train_full, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "norm_layer = tf.keras.layers.Normalization(input_shape=X_train.shape[1:])\n",
    "model = tf.keras.Sequential([\n",
    "    norm_layer,\n",
    "    tf.keras.layers.Dense(50, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(50, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(50, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(1)\n",
    "])\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)\n",
    "model.compile(loss=\"mse\", optimizer=optimizer, metrics=[\"RootMeanSquaredError\"])\n",
    "norm_layer.adapt(X_train)\n",
    "history = model.fit(X_train, y_train, epochs=20,\n",
    "                    validation_data=(X_valid, y_valid))\n",
    "mse_test, rmse_test = model.evaluate(X_test, y_test)\n",
    "X_new = X_test[:3]\n",
    "y_pred = model.predict(X_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "rmse_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building Complex Models Using the Functional API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not all neural network models are simply sequential. Some may have complex topologies. Some may have multiple inputs and/or multiple outputs. For example, a Wide & Deep neural network (see [paper](https://ai.google/research/pubs/pub45413)) connects all or part of the inputs directly to the output layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – reset the name counters and make the code reproducible\n",
    "tf.keras.backend.clear_session()\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "normalization_layer = tf.keras.layers.Normalization()\n",
    "hidden_layer1 = tf.keras.layers.Dense(30, activation=\"relu\")\n",
    "hidden_layer2 = tf.keras.layers.Dense(30, activation=\"relu\")\n",
    "concat_layer = tf.keras.layers.Concatenate()\n",
    "output_layer = tf.keras.layers.Dense(1)\n",
    "\n",
    "input_ = tf.keras.layers.Input(shape=X_train.shape[1:])\n",
    "normalized = normalization_layer(input_)\n",
    "hidden1 = hidden_layer1(normalized)\n",
    "hidden2 = hidden_layer2(hidden1)\n",
    "concat = concat_layer([input_, hidden2])\n",
    "output = output_layer(concat)\n",
    "\n",
    "model = tf.keras.Model(inputs=[input_], outputs=[output])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)\n",
    "model.compile(loss=\"mse\", optimizer=optimizer, metrics=[\"RootMeanSquaredError\"])\n",
    "normalization_layer.adapt(X_train)\n",
    "history = model.fit(X_train, y_train, epochs=20,\n",
    "                    validation_data=(X_valid, y_valid))\n",
    "mse_test = model.evaluate(X_test, y_test)\n",
    "y_pred = model.predict(X_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if you want to send different subsets of input features through the wide or deep paths? We will send 5 features (features 0 to 4), and 6 through the deep path (features 2 to 7). Note that 3 features will go through both (features 2, 3 and 4)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)  # not in the book"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_wide = tf.keras.layers.Input(shape=[5])  # features 0 to 4\n",
    "input_deep = tf.keras.layers.Input(shape=[6])  # features 2 to 7\n",
    "norm_layer_wide = tf.keras.layers.Normalization()\n",
    "norm_layer_deep = tf.keras.layers.Normalization()\n",
    "norm_wide = norm_layer_wide(input_wide)\n",
    "norm_deep = norm_layer_deep(input_deep)\n",
    "hidden1 = tf.keras.layers.Dense(30, activation=\"relu\")(norm_deep)\n",
    "hidden2 = tf.keras.layers.Dense(30, activation=\"relu\")(hidden1)\n",
    "concat = tf.keras.layers.concatenate([norm_wide, hidden2])\n",
    "output = tf.keras.layers.Dense(1)(concat)\n",
    "model = tf.keras.Model(inputs=[input_wide, input_deep], outputs=[output])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)\n",
    "model.compile(loss=\"mse\", optimizer=optimizer, metrics=[\"RootMeanSquaredError\"])\n",
    "\n",
    "X_train_wide, X_train_deep = X_train[:, :5], X_train[:, 2:]\n",
    "X_valid_wide, X_valid_deep = X_valid[:, :5], X_valid[:, 2:]\n",
    "X_test_wide, X_test_deep = X_test[:, :5], X_test[:, 2:]\n",
    "X_new_wide, X_new_deep = X_test_wide[:3], X_test_deep[:3]\n",
    "\n",
    "norm_layer_wide.adapt(X_train_wide)\n",
    "norm_layer_deep.adapt(X_train_deep)\n",
    "history = model.fit((X_train_wide, X_train_deep), y_train, epochs=20,\n",
    "                    validation_data=((X_valid_wide, X_valid_deep), y_valid))\n",
    "mse_test = model.evaluate((X_test_wide, X_test_deep), y_test)\n",
    "y_pred = model.predict((X_new_wide, X_new_deep))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding an auxiliary output for regularization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.keras.backend.clear_session()\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_wide = tf.keras.layers.Input(shape=[5])  # features 0 to 4\n",
    "input_deep = tf.keras.layers.Input(shape=[6])  # features 2 to 7\n",
    "norm_layer_wide = tf.keras.layers.Normalization()\n",
    "norm_layer_deep = tf.keras.layers.Normalization()\n",
    "norm_wide = norm_layer_wide(input_wide)\n",
    "norm_deep = norm_layer_deep(input_deep)\n",
    "hidden1 = tf.keras.layers.Dense(30, activation=\"relu\")(norm_deep)\n",
    "hidden2 = tf.keras.layers.Dense(30, activation=\"relu\")(hidden1)\n",
    "concat = tf.keras.layers.concatenate([norm_wide, hidden2])\n",
    "output = tf.keras.layers.Dense(1)(concat)\n",
    "aux_output = tf.keras.layers.Dense(1)(hidden2)\n",
    "model = tf.keras.Model(inputs=[input_wide, input_deep],\n",
    "                       outputs=[output, aux_output])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)\n",
    "model.compile(loss=(\"mse\", \"mse\"), loss_weights=(0.9, 0.1), optimizer=optimizer,\n",
    "              metrics=[\"RootMeanSquaredError\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "norm_layer_wide.adapt(X_train_wide)\n",
    "norm_layer_deep.adapt(X_train_deep)\n",
    "history = model.fit(\n",
    "    (X_train_wide, X_train_deep), (y_train, y_train), epochs=20,\n",
    "    validation_data=((X_valid_wide, X_valid_deep), (y_valid, y_valid))\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_results = model.evaluate((X_test_wide, X_test_deep), (y_test, y_test))\n",
    "weighted_sum_of_losses, main_loss, aux_loss, main_rmse, aux_rmse = eval_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_main, y_pred_aux = model.predict((X_new_wide, X_new_deep))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_tuple = model.predict((X_new_wide, X_new_deep))\n",
    "y_pred = dict(zip(model.output_names, y_pred_tuple))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the Subclassing API to Build Dynamic Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "class WideAndDeepModel(tf.keras.Model):\n",
    "    def __init__(self, units=30, activation=\"relu\", **kwargs):\n",
    "        super().__init__(**kwargs)  # needed to support naming the model\n",
    "        self.norm_layer_wide = tf.keras.layers.Normalization()\n",
    "        self.norm_layer_deep = tf.keras.layers.Normalization()\n",
    "        self.hidden1 = tf.keras.layers.Dense(units, activation=activation)\n",
    "        self.hidden2 = tf.keras.layers.Dense(units, activation=activation)\n",
    "        self.main_output = tf.keras.layers.Dense(1)\n",
    "        self.aux_output = tf.keras.layers.Dense(1)\n",
    "        \n",
    "    def call(self, inputs):\n",
    "        input_wide, input_deep = inputs\n",
    "        norm_wide = self.norm_layer_wide(input_wide)\n",
    "        norm_deep = self.norm_layer_deep(input_deep)\n",
    "        hidden1 = self.hidden1(norm_deep)\n",
    "        hidden2 = self.hidden2(hidden1)\n",
    "        concat = tf.keras.layers.concatenate([norm_wide, hidden2])\n",
    "        output = self.main_output(concat)\n",
    "        aux_output = self.aux_output(hidden2)\n",
    "        return output, aux_output\n",
    "\n",
    "tf.random.set_seed(42)  # not in the book – just for reproducibility\n",
    "model = WideAndDeepModel(30, activation=\"relu\", name=\"my_cool_model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)\n",
    "model.compile(loss=\"mse\", loss_weights=[0.9, 0.1], optimizer=optimizer,\n",
    "              metrics=[\"RootMeanSquaredError\"])\n",
    "model.norm_layer_wide.adapt(X_train_wide)\n",
    "model.norm_layer_deep.adapt(X_train_deep)\n",
    "history = model.fit(\n",
    "    (X_train_wide, X_train_deep), (y_train, y_train), epochs=10,\n",
    "    validation_data=((X_valid_wide, X_valid_deep), (y_valid, y_valid)))\n",
    "eval_results = model.evaluate((X_test_wide, X_test_deep), (y_test, y_test))\n",
    "weighted_sum_of_losses, main_loss, aux_loss, main_rmse, aux_rmse = eval_results\n",
    "y_pred_main, y_pred_aux = model.predict((X_new_wide, X_new_deep))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving and Restoring a Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – delete the directory, in case it already exists\n",
    "\n",
    "import shutil\n",
    "\n",
    "shutil.rmtree(\"my_keras_model\", ignore_errors=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_keras_model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – show the contents of the my_keras_model/ directory\n",
    "for path in sorted(Path(\"my_keras_model\").glob(\"**/*\")):\n",
    "    print(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.models.load_model(\"my_keras_model\")\n",
    "y_pred_main, y_pred_aux = model.predict((X_new_wide, X_new_deep))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save_weights(\"my_weights\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.load_weights(\"my_weights\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – show the list of my_weights.* files\n",
    "for path in sorted(Path().glob(\"my_weights.*\")):\n",
    "    print(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Callbacks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "shutil.rmtree(\"my_checkpoints\", ignore_errors=True)  # not in the book"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint_cb = tf.keras.callbacks.ModelCheckpoint(\"my_checkpoints\",\n",
    "                                                   save_weights_only=True)\n",
    "history = model.fit(\n",
    "    (X_train_wide, X_train_deep), (y_train, y_train), epochs=10,\n",
    "    validation_data=((X_valid_wide, X_valid_deep), (y_valid, y_valid)),\n",
    "    callbacks=[checkpoint_cb])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "early_stopping_cb = tf.keras.callbacks.EarlyStopping(patience=10,\n",
    "                                                     restore_best_weights=True)\n",
    "history = model.fit(\n",
    "    (X_train_wide, X_train_deep), (y_train, y_train), epochs=100,\n",
    "    validation_data=((X_valid_wide, X_valid_deep), (y_valid, y_valid)),\n",
    "    callbacks=[checkpoint_cb, early_stopping_cb])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PrintValTrainRatioCallback(tf.keras.callbacks.Callback):\n",
    "    def on_epoch_end(self, epoch, logs):\n",
    "        ratio = logs[\"val_loss\"] / logs[\"loss\"]\n",
    "        print(f\"Epoch={epoch}, val/train={ratio:.2f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "val_train_ratio_cb = PrintValTrainRatioCallback()\n",
    "history = model.fit(\n",
    "    (X_train_wide, X_train_deep), (y_train, y_train), epochs=10,\n",
    "    validation_data=((X_valid_wide, X_valid_deep), (y_valid, y_valid)),\n",
    "    callbacks=[val_train_ratio_cb], verbose=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using TensorBoard for Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorBoard is preinstalled on Colab, but not the `tensorboard-plugin-profile`, so let's install it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "if \"google.colab\" in sys.modules:  # not in the book\n",
    "    %pip install -q -U tensorboard-plugin-profile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "shutil.rmtree(\"my_logs\", ignore_errors=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from time import strftime\n",
    "\n",
    "def get_run_logdir(root_logdir=\"my_logs\"):\n",
    "    return Path(root_logdir) / strftime(\"run_%Y_%m_%d_%H_%M_%S\")\n",
    "\n",
    "run_logdir = get_run_logdir()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – builds the first regression model we used earlier\n",
    "tf.keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "norm_layer = tf.keras.layers.Normalization(input_shape=X_train.shape[1:])\n",
    "model = tf.keras.Sequential([\n",
    "    norm_layer,\n",
    "    tf.keras.layers.Dense(30, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(30, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(1)\n",
    "])\n",
    "optimizer = tf.keras.optimizers.SGD(learning_rate=1e-3)\n",
    "model.compile(loss=\"mse\", optimizer=optimizer, metrics=[\"RootMeanSquaredError\"])\n",
    "norm_layer.adapt(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensorboard_cb = tf.keras.callbacks.TensorBoard(run_logdir,\n",
    "                                                profile_batch=(100, 200))\n",
    "history = model.fit(X_train, y_train, epochs=20,\n",
    "                    validation_data=(X_valid, y_valid),\n",
    "                    callbacks=[tensorboard_cb])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"my_logs\")\n",
    "for path in sorted(Path(\"my_logs\").glob(\"**/*\")):\n",
    "    print(\"  \" * (len(path.parts) - 1) + path.parts[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the `tensorboard` Jupyter extension and start the TensorBoard server: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext tensorboard\n",
    "%tensorboard --logdir=./my_logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: if you prefer to access TensorBoard in a separate tab, click the \"localhost:6006\" link below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book\n",
    "\n",
    "if \"google.colab\" in sys.modules:\n",
    "    from google.colab import output\n",
    "\n",
    "    output.serve_kernel_port_as_window(6006)\n",
    "else:\n",
    "    from IPython.core.display import display, HTML\n",
    "\n",
    "    display(HTML('<a href=\"http://localhost:6006/\">http://localhost:6006/</a>'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use also visualize histograms, images, text, and even listen to audio using TensorBoard:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_logdir = get_run_logdir()\n",
    "writer = tf.summary.create_file_writer(str(test_logdir))\n",
    "with writer.as_default():\n",
    "    for step in range(1, 1000 + 1):\n",
    "        tf.summary.scalar(\"my_scalar\", np.sin(step / 10), step=step)\n",
    "        \n",
    "        data = (np.random.randn(100) + 2) * step / 100  # gets larger\n",
    "        tf.summary.histogram(\"my_hist\", data, buckets=50, step=step)\n",
    "        \n",
    "        images = np.random.rand(2, 32, 32, 3) * step / 1000  # gets brighter\n",
    "        tf.summary.image(\"my_images\", images, step=step)\n",
    "        \n",
    "        texts = [\"The step is \" + str(step), \"Its square is \" + str(step ** 2)]\n",
    "        tf.summary.text(\"my_text\", texts, step=step)\n",
    "        \n",
    "        sine_wave = tf.math.sin(tf.range(12000) / 48000 * 2 * np.pi * step)\n",
    "        audio = tf.reshape(tf.cast(sine_wave, tf.float32), [1, -1, 1])\n",
    "        tf.summary.audio(\"my_audio\", audio, sample_rate=48000, step=step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can share your TensorBoard logs with the world by uploading them to https://tensorboard.dev/. For this, you can run the `tensorboard dev upload` command, with the `--logdir` and `--one_shot` options, and optionally the `--name` and `--description` options. The first time, it will ask you to accept Google's Terms of Service, and to authenticate. This requires user input. Colab supports user input from shell commands, but the main other Jupyter environments do not, so for them we use a hackish workaround (alternatively, you could run the command in a terminal window, after you make sure to activate this project's conda environment and move to this notebook's directory)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book\n",
    "\n",
    "if \"google.colab\" in sys.modules:\n",
    "    !tensorboard dev upload --logdir ./my_logs --one_shot \\\n",
    "                            --name \"Quick test\" --description \"This is a test\" \n",
    "else:\n",
    "    from tensorboard.main import run_main\n",
    "\n",
    "    argv = \"tensorboard dev upload --logdir ./my_logs --one_shot\".split()\n",
    "    argv += [\"--name\", \"Quick test\", \"--description\", \"This is a test\"]\n",
    "    try:\n",
    "        original_sys_argv_and_sys_exit = sys.argv, sys.exit\n",
    "        sys.argv, sys.exit = argv, lambda status: None\n",
    "        run_main()\n",
    "    finally:\n",
    "        sys.argv, sys.exit = original_sys_argv_and_sys_exit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get list your published experiments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tensorboard dev list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To delete an experiment, use the following command:\n",
    "\n",
    "```python\n",
    "!tensorboard dev delete --experiment_id <experiment_id>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you stop this Jupyter kernel (a.k.a. Runtime), it will automatically stop the TensorBoard server as well. Another way to stop the TensorBoard server is to kill it, if you are running on Linux or MacOSX. First, you need to find its process ID:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – extra material\n",
    "\n",
    "from tensorboard import notebook\n",
    "\n",
    "notebook.list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next you can use the following command on Linux or MacOSX, replacing `<pid>` with the pid listed above:\n",
    "\n",
    "    !kill <pid>\n",
    "\n",
    "On Windows:\n",
    "\n",
    "    !taskkill /F /PID <pid>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine-Tuning Neural Network Hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we'll use the Fashion MNIST dataset again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = fashion_mnist\n",
    "X_train, y_train = X_train_full[:-5000], y_train_full[:-5000]\n",
    "X_valid, y_valid = X_train_full[-5000:], y_train_full[-5000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.keras.backend.clear_session()\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "if \"google.colab\" in sys.modules:\n",
    "    %pip install -q -U keras_tuner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras_tuner as kt\n",
    "\n",
    "def build_model(hp):\n",
    "    n_hidden = hp.Int(\"n_hidden\", min_value=0, max_value=8, default=2)\n",
    "    n_neurons = hp.Int(\"n_neurons\", min_value=16, max_value=256)\n",
    "    learning_rate = hp.Float(\"learning_rate\", min_value=1e-4, max_value=1e-2,\n",
    "                             sampling=\"log\")\n",
    "    optimizer = hp.Choice(\"optimizer\", values=[\"sgd\", \"adam\"])\n",
    "    if optimizer == \"sgd\":\n",
    "        optimizer = tf.keras.optimizers.SGD(learning_rate=learning_rate)\n",
    "    else:\n",
    "        optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)\n",
    "\n",
    "    model = tf.keras.Sequential()\n",
    "    model.add(tf.keras.layers.Flatten())\n",
    "    for _ in range(n_hidden):\n",
    "        model.add(tf.keras.layers.Dense(n_neurons, activation=\"relu\"))\n",
    "    model.add(tf.keras.layers.Dense(10, activation=\"softmax\"))\n",
    "    model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
    "                  metrics=[\"accuracy\"])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_search_tuner = kt.RandomSearch(\n",
    "    build_model, objective=\"val_accuracy\", max_trials=5, overwrite=True,\n",
    "    directory=\"my_fashion_mnist\", project_name=\"my_rnd_search\", seed=42)\n",
    "random_search_tuner.search(X_train, y_train, epochs=10,\n",
    "                           validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "top3_models = random_search_tuner.get_best_models(num_models=3)\n",
    "best_model = top3_models[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "top3_params = random_search_tuner.get_best_hyperparameters(num_trials=3)\n",
    "top3_params[0].values  # best hyperparameter values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_trial = random_search_tuner.oracle.get_best_trials(num_trials=1)[0]\n",
    "best_trial.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_trial.metrics.get_last_value(\"val_accuracy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_model.fit(X_train_full, y_train_full, epochs=10)\n",
    "test_loss, test_accuracy = best_model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyClassificationHyperModel(kt.HyperModel):\n",
    "    def build(self, hp):\n",
    "        return build_model(hp)\n",
    "\n",
    "    def fit(self, hp, model, X, y, **kwargs):\n",
    "        if hp.Boolean(\"normalize\"):\n",
    "            norm_layer = tf.keras.layers.Normalization()\n",
    "            X = norm_layer(X)\n",
    "        return model.fit(X, y, **kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperband_tuner = kt.Hyperband(\n",
    "    MyClassificationHyperModel(), objective=\"val_accuracy\", seed=42,\n",
    "    max_epochs=10, factor=3, hyperband_iterations=2,\n",
    "    overwrite=True, directory=\"my_fashion_mnist\", project_name=\"hyperband\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "root_logdir = Path(hyperband_tuner.project_dir) / \"tensorboard\"\n",
    "tensorboard_cb = tf.keras.callbacks.TensorBoard(root_logdir)\n",
    "early_stopping_cb = tf.keras.callbacks.EarlyStopping(patience=2)\n",
    "hyperband_tuner.search(X_train, y_train, epochs=10,\n",
    "                       validation_data=(X_valid, y_valid),\n",
    "                       callbacks=[early_stopping_cb, tensorboard_cb])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "bayesian_opt_tuner = kt.BayesianOptimization(\n",
    "    MyClassificationHyperModel(), objective=\"val_accuracy\", seed=42,\n",
    "    max_trials=10, alpha=1e-4, beta=2.6,\n",
    "    overwrite=True, directory=\"my_fashion_mnist\", project_name=\"bayesian_opt\")\n",
    "bayesian_opt_tuner.search(X_train, y_train, epochs=10,\n",
    "                          validation_data=(X_valid, y_valid),\n",
    "                          callbacks=[early_stopping_cb])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "%tensorboard --logdir {root_logdir}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. to 9."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Visit the [TensorFlow Playground](https://playground.tensorflow.org/) and play around with it, as described in this exercise.\n",
    "2. Here is a neural network based on the original artificial neurons that computes _A_ ⊕ _B_ (where ⊕ represents the exclusive OR), using the fact that _A_ ⊕ _B_ = (_A_ ∧ ¬ _B_) ∨ (¬ _A_ ∧ _B_). There are other solutions—for example, using the fact that _A_ ⊕ _B_ = (_A_ ∨ _B_) ∧ ¬(_A_ ∧ _B_), or the fact that _A_ ⊕ _B_ = (_A_ ∨ _B_) ∧ (¬ _A_ ∨ ¬ _B_), and so on.<br /><img width=\"70%\" src=\"images/ann/exercise2.png\" />\n",
    "3. A classical Perceptron will converge only if the dataset is linearly separable, and it won't be able to estimate class probabilities. In contrast, a Logistic Regression classifier will generally converge to a reasonably good solution even if the dataset is not linearly separable, and it will output class probabilities. If you change the Perceptron's activation function to the sigmoid activation function (or the softmax activation function if there are multiple neurons), and if you train it using Gradient Descent (or some other optimization algorithm minimizing the cost function, typically cross entropy), then it becomes equivalent to a Logistic Regression classifier.\n",
    "4. The sigmoid activation function was a key ingredient in training the first MLPs because its derivative is always nonzero, so Gradient Descent can always roll down the slope. When the activation function is a step function, Gradient Descent cannot move, as there is no slope at all.\n",
    "5. Popular activation functions include the step function, the sigmoid function, the hyperbolic tangent (tanh) function, and the Rectified Linear Unit (ReLU) function (see Figure 10-8). See Chapter 11 for other examples, such as ELU and variants of the ReLU function.\n",
    "6. Considering the MLP described in the question, composed of one input layer with 10 passthrough neurons, followed by one hidden layer with 50 artificial neurons, and finally one output layer with 3 artificial neurons, where all artificial neurons use the ReLU activation function:\n",
    "    * The shape of the input matrix **X** is _m_ × 10, where _m_ represents the training batch size.\n",
    "    * The shape of the hidden layer's weight matrix **W**<sub>_h_</sub> is 10 × 50, and the length of its bias vector **b**<sub>_h_</sub> is 50.\n",
    "    * The shape of the output layer's weight matrix **W**<sub>_o_</sub> is 50 × 3, and the length of its bias vector **b**<sub>_o_</sub> is 3.\n",
    "    * The shape of the network's output matrix **Y** is _m_ × 3.\n",
    "    * **Y** = ReLU(ReLU(**X** **W**<sub>_h_</sub> + **b**<sub>_h_</sub>) **W**<sub>_o_</sub> + **b**<sub>_o_</sub>). Recall that the ReLU function just sets every negative number in the matrix to zero. Also note that when you are adding a bias vector to a matrix, it is added to every single row in the matrix, which is called _broadcasting_.\n",
    "7. To classify email into spam or ham, you just need one neuron in the output layer of a neural network—for example, indicating the probability that the email is spam. You would typically use the sigmoid activation function in the output layer when estimating a probability. If instead you want to tackle MNIST, you need 10 neurons in the output layer, and you must replace the sigmoid function with the softmax activation function, which can handle multiple classes, outputting one probability per class. If you want your neural network to predict housing prices like in Chapter 2, then you need one output neuron, using no activation function at all in the output layer. Note: when the values to predict can vary by many orders of magnitude, you may want to predict the logarithm of the target value rather than the target value directly. Simply computing the exponential of the neural network's output will give you the estimated value (since exp(log _v_) = _v_).\n",
    "8. Backpropagation is a technique used to train artificial neural networks. It first computes the gradients of the cost function with regard to every model parameter (all the weights and biases), then it performs a Gradient Descent step using these gradients. This backpropagation step is typically performed thousands or millions of times, using many training batches, until the model parameters converge to values that (hopefully) minimize the cost function. To compute the gradients, backpropagation uses reverse-mode autodiff (although it wasn't called that when backpropagation was invented, and it has been reinvented several times). Reverse-mode autodiff performs a forward pass through a computation graph, computing every node's value for the current training batch, and then it performs a reverse pass, computing all the gradients at once (see Appendix B for more details). So what's the difference? Well, backpropagation refers to the whole process of training an artificial neural network using multiple backpropagation steps, each of which computes gradients and uses them to perform a Gradient Descent step. In contrast, reverse-mode autodiff is just a technique to compute gradients efficiently, and it happens to be used by backpropagation.\n",
    "9. Here is a list of all the hyperparameters you can tweak in a basic MLP: the number of hidden layers, the number of neurons in each hidden layer, and the activation function used in each hidden layer and in the output layer. In general, the ReLU activation function (or one of its variants; see Chapter 11) is a good default for the hidden layers. For the output layer, in general you will want the sigmoid activation function for binary classification, the softmax activation function for multiclass classification, or no activation function for regression. If the MLP overfits the training data, you can try reducing the number of hidden layers and reducing the number of neurons per hidden layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Exercise: Train a deep MLP on the MNIST dataset (you can load it using `tf.keras.datasets.mnist.load_data()`. See if you can get over 98% accuracy by manually tuning the hyperparameters. Try searching for the optimal learning rate by using the approach presented in this chapter (i.e., by growing the learning rate exponentially, plotting the loss, and finding the point where the loss shoots up). Next, try tuning the hyperparameters using Keras Tuner with all the bells and whistles—save checkpoints, use early stopping, and plot learning curves using TensorBoard.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**: update this solution to use Keras Tuner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = tf.keras.datasets.mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like for the Fashion MNIST dataset, the MNIST training set contains 60,000 grayscale images, each 28x28 pixels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_full.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each pixel intensity is also represented as a byte (0 to 255):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_full.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's split the full training set into a validation set and a (smaller) training set. We also scale the pixel intensities down to the 0-1 range and convert them to floats, by dividing by 255, just like we did for Fashion MNIST:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_valid, X_train = X_train_full[:5000] / 255., X_train_full[5000:] / 255.\n",
    "y_valid, y_train = y_train_full[:5000], y_train_full[5000:]\n",
    "X_test = X_test / 255."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot an image using Matplotlib's `imshow()` function, with a `'binary'`\n",
    " color map:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(X_train[0], cmap=\"binary\")\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The labels are the class IDs (represented as uint8), from 0 to 9. Conveniently, the class IDs correspond to the digits represented in the images, so we don't need a `class_names` array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The validation set contains 5,000 images, and the test set contains 10,000 images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_valid.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at a sample of the images in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_rows = 4\n",
    "n_cols = 10\n",
    "plt.figure(figsize=(n_cols * 1.2, n_rows * 1.2))\n",
    "for row in range(n_rows):\n",
    "    for col in range(n_cols):\n",
    "        index = n_cols * row + col\n",
    "        plt.subplot(n_rows, n_cols, index + 1)\n",
    "        plt.imshow(X_train[index], cmap=\"binary\", interpolation=\"nearest\")\n",
    "        plt.axis('off')\n",
    "        plt.title(y_train[index])\n",
    "plt.subplots_adjust(wspace=0.2, hspace=0.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build a simple dense network and find the optimal learning rate. We will need a callback to grow the learning rate at each iteration. It will also record the learning rate and the loss at each iteration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "K = tf.keras.backend\n",
    "\n",
    "class ExponentialLearningRate(tf.keras.callbacks.Callback):\n",
    "    def __init__(self, factor):\n",
    "        self.factor = factor\n",
    "        self.rates = []\n",
    "        self.losses = []\n",
    "    def on_batch_end(self, batch, logs):\n",
    "        self.rates.append(K.get_value(self.model.optimizer.learning_rate))\n",
    "        self.losses.append(logs[\"loss\"])\n",
    "        K.set_value(self.model.optimizer.learning_rate, self.model.optimizer.learning_rate * self.factor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    tf.keras.layers.Dense(300, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(100, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start with a small learning rate of 1e-3, and grow it by 0.5% at each iteration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.SGD(learning_rate=1e-3)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "expon_lr = ExponentialLearningRate(factor=1.005)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train the model for just 1 epoch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(X_train, y_train, epochs=1,\n",
    "                    validation_data=(X_valid, y_valid),\n",
    "                    callbacks=[expon_lr])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now plot the loss as a functionof the learning rate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(expon_lr.rates, expon_lr.losses)\n",
    "plt.gca().set_xscale('log')\n",
    "plt.hlines(min(expon_lr.losses), min(expon_lr.rates), max(expon_lr.rates))\n",
    "plt.axis([min(expon_lr.rates), max(expon_lr.rates), 0, expon_lr.losses[0]])\n",
    "plt.grid()\n",
    "plt.xlabel(\"Learning rate\")\n",
    "plt.ylabel(\"Loss\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss starts shooting back up violently when the learning rate goes over 6e-1, so let's try using half of that, at 3e-1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    tf.keras.layers.Dense(300, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(100, activation=\"relu\"),\n",
    "    tf.keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.SGD(learning_rate=3e-1)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_index = 1 # increment this at every run\n",
    "run_logdir = Path() / \"my_mnist_logs\" / \"run_{:03d}\".format(run_index)\n",
    "run_logdir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "early_stopping_cb = tf.keras.callbacks.EarlyStopping(patience=20)\n",
    "checkpoint_cb = tf.keras.callbacks.ModelCheckpoint(\"my_mnist_model.h5\", save_best_only=True)\n",
    "tensorboard_cb = tf.keras.callbacks.TensorBoard(run_logdir)\n",
    "\n",
    "history = model.fit(X_train, y_train, epochs=100,\n",
    "                    validation_data=(X_valid, y_valid),\n",
    "                    callbacks=[checkpoint_cb, early_stopping_cb, tensorboard_cb])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.models.load_model(\"my_mnist_model.h5\") # rollback to best model\n",
    "model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We got over 98% accuracy. Finally, let's look at the learning curves using TensorBoard:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "%tensorboard --logdir=./my_mnist_logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "nav_menu": {
   "height": "264px",
   "width": "369px"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}