handson-ml/06_decision_trees.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Chapter 6 – Decision Trees**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains all the sample code and solutions to the exercises in chapter 6._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/ageron/handson-ml3/blob/main/06_decision_trees.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://kaggle.com/kernels/welcome?src=https://github.com/ageron/handson-ml3/blob/main/06_decision_trees.ipynb\"><img src=\"https://kaggle.com/static/images/open-in-kaggle.svg\" /></a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This project requires Python 3.8 or above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "assert sys.version_info >= (3, 8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It also requires Scikit-Learn ≥ 1.0.1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "\n",
    "assert sklearn.__version__ >= \"1.0.1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we did in previous chapters, let's define the default font sizes to make the figures prettier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.rc('font', size=12)\n",
    "plt.rc('axes', labelsize=14, titlesize=14)\n",
    "plt.rc('legend', fontsize=14)\n",
    "plt.rc('xtick',labelsize=10)\n",
    "plt.rc('ytick',labelsize=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's create the `images/decision_trees` folder (if it doesn't already exist), and define the `save_fig()` function which is used through this notebook to save the figures in high-res for the book:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "IMAGES_PATH = Path() / \"images\" / \"decision_trees\"\n",
    "IMAGES_PATH.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = IMAGES_PATH / f\"{fig_id}.{fig_extension}\"\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training and Visualizing a Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "iris = load_iris(as_frame=True)\n",
    "X_iris = iris.data[[\"petal length (cm)\", \"petal width (cm)\"]].values\n",
    "y_iris = iris.target\n",
    "\n",
    "tree_clf = DecisionTreeClassifier(max_depth=2, random_state=42)\n",
    "tree_clf.fit(X_iris, y_iris)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**This code example generates Figure 6–1. Iris Decision Tree:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import export_graphviz\n",
    "\n",
    "export_graphviz(\n",
    "        tree_clf,\n",
    "        out_file=str(IMAGES_PATH / \"iris_tree.dot\"),  # path differs in the book\n",
    "        feature_names=[\"petal length (cm)\", \"petal width (cm)\"],\n",
    "        class_names=iris.target_names,\n",
    "        rounded=True,\n",
    "        filled=True\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from graphviz import Source\n",
    "\n",
    "Source.from_file(IMAGES_PATH / \"iris_tree.dot\")  # path differs in the book"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Graphviz also provides the `dot` command line tool to convert `.dot` files to a variety of formats. The following command converts the dot file to a png image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book\n",
    "!dot -Tpng {IMAGES_PATH / \"iris_tree.dot\"} -o {IMAGES_PATH / \"iris_tree.png\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# not in the book – just formatting details\n",
    "from matplotlib.colors import ListedColormap\n",
    "custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])\n",
    "plt.figure(figsize=(8, 4))\n",
    "\n",
    "lengths, widths = np.meshgrid(np.linspace(0, 7.2, 100), np.linspace(0, 3, 100))\n",
    "X_iris_all = np.c_[lengths.ravel(), widths.ravel()]\n",
    "y_pred = tree_clf.predict(X_iris_all).reshape(lengths.shape)\n",
    "plt.contourf(lengths, widths, y_pred, alpha=0.3, cmap=custom_cmap)\n",
    "for idx, (name, style) in enumerate(zip(iris.target_names, (\"yo\", \"bs\", \"g^\"))):\n",
    "    plt.plot(X_iris[:, 0][y_iris == idx], X_iris[:, 1][y_iris == idx],\n",
    "             style, label=f\"Iris {name}\")\n",
    "\n",
    "# not in the book – this section beautifies and saves Figure 6–2\n",
    "tree_clf_deeper = DecisionTreeClassifier(max_depth=3, random_state=42)\n",
    "tree_clf_deeper.fit(X_iris, y_iris)\n",
    "th0, th1, th2a, th2b = tree_clf_deeper.tree_.threshold[[0, 2, 3, 6]]\n",
    "plt.xlabel(\"Petal length (cm)\")\n",
    "plt.ylabel(\"Petal width (cm)\")\n",
    "plt.plot([th0, th0], [0, 3], \"k-\", linewidth=2)\n",
    "plt.plot([th0, 7.2], [th1, th1], \"k--\", linewidth=2)\n",
    "plt.plot([th2a, th2a], [0, th1], \"k:\", linewidth=2)\n",
    "plt.plot([th2b, th2b], [th1, 3], \"k:\", linewidth=2)\n",
    "plt.text(th0 - 0.05, 1.0, \"Depth=0\", horizontalalignment=\"right\", fontsize=15)\n",
    "plt.text(3.2, th1 + 0.02, \"Depth=1\", verticalalignment=\"bottom\", fontsize=13)\n",
    "plt.text(th2a + 0.05, 0.5, \"(Depth=2)\", fontsize=11)\n",
    "plt.axis([0, 7.2, 0, 3])\n",
    "plt.legend()\n",
    "save_fig(\"decision_tree_decision_boundaries_plot\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access the tree structure via the `tree_` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree_clf.tree_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information, check out this class's documentation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# help(sklearn.tree._tree.Tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the extra material section below for an example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Estimating Class Probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree_clf.predict_proba([[5, 1.5]]).round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree_clf.predict([[5, 1.5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "\n",
    "X_moons, y_moons = make_moons(n_samples=150, noise=0.2, random_state=42)\n",
    "\n",
    "tree_clf1 = DecisionTreeClassifier(random_state=42)\n",
    "tree_clf2 = DecisionTreeClassifier(min_samples_leaf=5, random_state=42)\n",
    "tree_clf1.fit(X_moons, y_moons)\n",
    "tree_clf2.fit(X_moons, y_moons)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – this cell generates and saves Figure 6–3\n",
    "\n",
    "def plot_decision_boundary(clf, X, y, axes, cmap):\n",
    "    x1, x2 = np.meshgrid(np.linspace(axes[0], axes[1], 100),\n",
    "                         np.linspace(axes[2], axes[3], 100))\n",
    "    X_new = np.c_[x1.ravel(), x2.ravel()]\n",
    "    y_pred = clf.predict(X_new).reshape(x1.shape)\n",
    "    \n",
    "    plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=cmap)\n",
    "    plt.contour(x1, x2, y_pred, cmap=\"Greys\", alpha=0.8)\n",
    "    colors = {\"Wistia\": [\"#78785c\", \"#c47b27\"], \"Pastel1\": [\"red\", \"blue\"]}\n",
    "    markers = (\"o\", \"^\")\n",
    "    for idx in (0, 1):\n",
    "        plt.plot(X[:, 0][y == idx], X[:, 1][y == idx],\n",
    "                 color=colors[cmap][idx], marker=markers[idx], linestyle=\"none\")\n",
    "    plt.axis(axes)\n",
    "    plt.xlabel(r\"$x_1$\")\n",
    "    plt.ylabel(r\"$x_2$\", rotation=0)\n",
    "\n",
    "fig, axes = plt.subplots(ncols=2, figsize=(10, 4), sharey=True)\n",
    "plt.sca(axes[0])\n",
    "plot_decision_boundary(tree_clf1, X_moons, y_moons,\n",
    "                       axes=[-1.5, 2.4, -1, 1.5], cmap=\"Wistia\")\n",
    "plt.title(\"No restrictions\")\n",
    "plt.sca(axes[1])\n",
    "plot_decision_boundary(tree_clf2, X_moons, y_moons,\n",
    "                       axes=[-1.5, 2.4, -1, 1.5], cmap=\"Wistia\")\n",
    "plt.title(f\"min_samples_leaf = {tree_clf2.min_samples_leaf}\")\n",
    "plt.ylabel(\"\")\n",
    "save_fig(\"min_samples_leaf_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_moons_test, y_moons_test = make_moons(n_samples=1000, noise=0.2,\n",
    "                                        random_state=43)\n",
    "tree_clf1.score(X_moons_test, y_moons_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree_clf2.score(X_moons_test, y_moons_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's prepare a simple quadratic training set:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Code example:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "np.random.seed(42)\n",
    "X_quad = np.random.rand(200, 1) - 0.5  # a single random input feature\n",
    "y_quad = X_quad ** 2 + 0.025 * np.random.randn(200, 1)\n",
    "\n",
    "tree_reg = DecisionTreeRegressor(max_depth=2, random_state=42)\n",
    "tree_reg.fit(X_quad, y_quad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – we've already seen how to use export_graphviz()\n",
    "export_graphviz(\n",
    "    tree_reg,\n",
    "    out_file=str(IMAGES_PATH / \"regression_tree.dot\"),\n",
    "    feature_names=[\"x1\"],\n",
    "    rounded=True,\n",
    "    filled=True\n",
    ")\n",
    "Source.from_file(IMAGES_PATH / \"regression_tree.dot\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree_reg2 = DecisionTreeRegressor(max_depth=3, random_state=42)\n",
    "tree_reg2.fit(X_quad, y_quad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree_reg.tree_.threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree_reg2.tree_.threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – this cell generates and saves Figure 6–5\n",
    "\n",
    "def plot_regression_predictions(tree_reg, X, y, axes=[-0.5, 0.5, -0.05, 0.25]):\n",
    "    x1 = np.linspace(axes[0], axes[1], 500).reshape(-1, 1)\n",
    "    y_pred = tree_reg.predict(x1)\n",
    "    plt.axis(axes)\n",
    "    plt.xlabel(\"$x_1$\")\n",
    "    plt.plot(X, y, \"b.\")\n",
    "    plt.plot(x1, y_pred, \"r.-\", linewidth=2, label=r\"$\\hat{y}$\")\n",
    "\n",
    "fig, axes = plt.subplots(ncols=2, figsize=(10, 4), sharey=True)\n",
    "plt.sca(axes[0])\n",
    "plot_regression_predictions(tree_reg, X_quad, y_quad)\n",
    "\n",
    "th0, th1a, th1b = tree_reg.tree_.threshold[[0, 1, 4]]\n",
    "for split, style in ((th0, \"k-\"), (th1a, \"k--\"), (th1b, \"k--\")):\n",
    "    plt.plot([split, split], [-0.05, 0.25], style, linewidth=2)\n",
    "plt.text(th0, 0.16, \"Depth=0\", fontsize=15)\n",
    "plt.text(th1a + 0.01, -0.01, \"Depth=1\", horizontalalignment=\"center\", fontsize=13)\n",
    "plt.text(th1b + 0.01, -0.01, \"Depth=1\", fontsize=13)\n",
    "plt.ylabel(\"$y$\", rotation=0)\n",
    "plt.legend(loc=\"upper center\", fontsize=16)\n",
    "plt.title(\"max_depth=2\")\n",
    "\n",
    "plt.sca(axes[1])\n",
    "th2s = tree_reg2.tree_.threshold[[2, 5, 9, 12]]\n",
    "plot_regression_predictions(tree_reg2, X_quad, y_quad)\n",
    "for split, style in ((th0, \"k-\"), (th1a, \"k--\"), (th1b, \"k--\")):\n",
    "    plt.plot([split, split], [-0.05, 0.25], style, linewidth=2)\n",
    "for split in th2s:\n",
    "    plt.plot([split, split], [-0.05, 0.25], \"k:\", linewidth=1)\n",
    "plt.text(th2s[2] + 0.01, 0.15, \"Depth=2\", fontsize=13)\n",
    "plt.title(\"max_depth=3\")\n",
    "\n",
    "save_fig(\"tree_regression_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – this cell generates and saves Figure 6–6\n",
    "\n",
    "tree_reg1 = DecisionTreeRegressor(random_state=42)\n",
    "tree_reg2 = DecisionTreeRegressor(random_state=42, min_samples_leaf=10)\n",
    "tree_reg1.fit(X_quad, y_quad)\n",
    "tree_reg2.fit(X_quad, y_quad)\n",
    "\n",
    "x1 = np.linspace(-0.5, 0.5, 500).reshape(-1, 1)\n",
    "y_pred1 = tree_reg1.predict(x1)\n",
    "y_pred2 = tree_reg2.predict(x1)\n",
    "\n",
    "fig, axes = plt.subplots(ncols=2, figsize=(10, 4), sharey=True)\n",
    "\n",
    "plt.sca(axes[0])\n",
    "plt.plot(X_quad, y_quad, \"b.\")\n",
    "plt.plot(x1, y_pred1, \"r.-\", linewidth=2, label=r\"$\\hat{y}$\")\n",
    "plt.axis([-0.5, 0.5, -0.05, 0.25])\n",
    "plt.xlabel(\"$x_1$\")\n",
    "plt.ylabel(\"$y$\", rotation=0)\n",
    "plt.legend(loc=\"upper center\")\n",
    "plt.title(\"No restrictions\")\n",
    "\n",
    "plt.sca(axes[1])\n",
    "plt.plot(X_quad, y_quad, \"b.\")\n",
    "plt.plot(x1, y_pred2, \"r.-\", linewidth=2, label=r\"$\\hat{y}$\")\n",
    "plt.axis([-0.5, 0.5, -0.05, 0.25])\n",
    "plt.xlabel(\"$x_1$\")\n",
    "plt.title(f\"min_samples_leaf={tree_reg2.min_samples_leaf}\")\n",
    "\n",
    "save_fig(\"tree_regression_regularization_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sensitivity to axis orientation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rotating the dataset also leads to completely different decision boundaries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – this cell generates and saves Figure 6–7\n",
    "\n",
    "np.random.seed(6)\n",
    "X_square = np.random.rand(100, 2) - 0.5\n",
    "y_square = (X_square[:, 0] > 0).astype(np.int64)\n",
    "\n",
    "angle = np.pi / 4  # 45 degrees\n",
    "rotation_matrix = np.array([[np.cos(angle), -np.sin(angle)],\n",
    "                            [np.sin(angle), np.cos(angle)]])\n",
    "X_rotated_square = X_square.dot(rotation_matrix)\n",
    "\n",
    "tree_clf_square = DecisionTreeClassifier(random_state=42)\n",
    "tree_clf_square.fit(X_square, y_square)\n",
    "tree_clf_rotated_square = DecisionTreeClassifier(random_state=42)\n",
    "tree_clf_rotated_square.fit(X_rotated_square, y_square)\n",
    "\n",
    "fig, axes = plt.subplots(ncols=2, figsize=(10, 4), sharey=True)\n",
    "plt.sca(axes[0])\n",
    "plot_decision_boundary(tree_clf_square, X_square, y_square,\n",
    "                       axes=[-0.7, 0.7, -0.7, 0.7], cmap=\"Pastel1\")\n",
    "plt.sca(axes[1])\n",
    "plot_decision_boundary(tree_clf_rotated_square, X_rotated_square, y_square,\n",
    "                       axes=[-0.7, 0.7, -0.7, 0.7], cmap=\"Pastel1\")\n",
    "plt.ylabel(\"\")\n",
    "\n",
    "save_fig(\"sensitivity_to_rotation_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "pca_pipeline = make_pipeline(StandardScaler(), PCA())\n",
    "X_iris_rotated = pca_pipeline.fit_transform(X_iris)\n",
    "tree_clf_pca = DecisionTreeClassifier(max_depth=2, random_state=42)\n",
    "tree_clf_pca.fit(X_iris_rotated, y_iris)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – this cell generates and saves Figure 6–8\n",
    "\n",
    "plt.figure(figsize=(8, 4))\n",
    "\n",
    "axes = [-2.2, 2.4, -0.6, 0.7]\n",
    "z0s, z1s = np.meshgrid(np.linspace(axes[0], axes[1], 100),\n",
    "                       np.linspace(axes[2], axes[3], 100))\n",
    "X_iris_pca_all = np.c_[z0s.ravel(), z1s.ravel()]\n",
    "y_pred = tree_clf_pca.predict(X_iris_pca_all).reshape(z0s.shape)\n",
    "\n",
    "plt.contourf(z0s, z1s, y_pred, alpha=0.3, cmap=custom_cmap)\n",
    "for idx, (name, style) in enumerate(zip(iris.target_names, (\"yo\", \"bs\", \"g^\"))):\n",
    "    plt.plot(X_iris_rotated[:, 0][y_iris == idx],\n",
    "             X_iris_rotated[:, 1][y_iris == idx],\n",
    "             style, label=f\"Iris {name}\")\n",
    "\n",
    "plt.xlabel(\"$z_1$\")\n",
    "plt.ylabel(\"$z_2$\", rotation=0)\n",
    "th1, th2 = tree_clf_pca.tree_.threshold[[0, 2]]\n",
    "plt.plot([th1, th1], axes[2:], \"k-\", linewidth=2)\n",
    "plt.plot([th2, th2], axes[2:], \"k--\", linewidth=2)\n",
    "plt.text(th1 - 0.01, axes[2] + 0.05, \"Depth=0\",\n",
    "         horizontalalignment=\"right\", fontsize=15)\n",
    "plt.text(th2 - 0.01, axes[2] + 0.05, \"Depth=1\",\n",
    "         horizontalalignment=\"right\", fontsize=13)\n",
    "plt.axis(axes)\n",
    "plt.legend(loc=(0.32, 0.67))\n",
    "save_fig(\"pca_preprocessing_plot\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Decision Trees Have High Variance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've seen that small changes in the dataset (such as a rotation) may produce a very different Decision Tree.\n",
    "Now let's show that training the same model on the same data may produce a very different model every time, since the CART training algorithm used by Scikit-Learn is stochastic. To show this, we will set `random_state` to a different value than earlier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree_clf_tweaked = DecisionTreeClassifier(max_depth=2, random_state=40)\n",
    "tree_clf_tweaked.fit(X_iris, y_iris)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# not in the book – this cell generates and saves Figure 6–9\n",
    "\n",
    "plt.figure(figsize=(8, 4))\n",
    "y_pred = tree_clf_tweaked.predict(X_iris_all).reshape(lengths.shape)\n",
    "plt.contourf(lengths, widths, y_pred, alpha=0.3, cmap=custom_cmap)\n",
    "\n",
    "for idx, (name, style) in enumerate(zip(iris.target_names, (\"yo\", \"bs\", \"g^\"))):\n",
    "    plt.plot(X_iris[:, 0][y_iris == idx], X_iris[:, 1][y_iris == idx],\n",
    "             style, label=f\"Iris {name}\")\n",
    "\n",
    "th0, th1 = tree_clf_tweaked.tree_.threshold[[0, 2]]\n",
    "plt.plot([0, 7.2], [th0, th0], \"k-\", linewidth=2)\n",
    "plt.plot([0, 7.2], [th1, th1], \"k--\", linewidth=2)\n",
    "plt.text(1.8, th0 + 0.05, \"Depth=0\", verticalalignment=\"bottom\", fontsize=15)\n",
    "plt.text(2.3, th1 + 0.05, \"Depth=1\", verticalalignment=\"bottom\", fontsize=13)\n",
    "plt.xlabel(\"Petal length (cm)\")\n",
    "plt.ylabel(\"Petal width (cm)\")\n",
    "plt.axis([0, 7.2, 0, 3])\n",
    "plt.legend()\n",
    "save_fig(\"decision_tree_high_variance_plot\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extra Material – Accessing the tree structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A trained `DecisionTreeClassifier` has a `tree_` attribute that stores the tree's structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree = tree_clf.tree_\n",
    "tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get the total number of nodes in the tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree.node_count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And other self-explanatory attributes are available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree.max_depth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree.max_n_classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree.n_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree.n_outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree.n_leaves"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the information about the nodes is stored in NumPy arrays. For example, the impurity of each node:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree.impurity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The root node is at index 0. The left and right children nodes of node _i_ are `tree.children_left[i]` and `tree.children_right[i]`. For example, the children of the root node are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree.children_left[0], tree.children_right[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the left and right nodes are equal, it means this is a leaf node (and the children node ids are arbitrary):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree.children_left[3], tree.children_right[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So you can get the leaf node ids like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "is_leaf = (tree.children_left == tree.children_right)\n",
    "np.arange(tree.node_count)[is_leaf]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Non-leaf nodes are called _split nodes_. The feature they split is available via the `feature` array. Values for leaf nodes should be ignored:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree.feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the corresponding thresholds are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree.threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the number of instances per class that reached each node is available too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree.value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree.n_node_samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.all(tree.value.sum(axis=(1, 2)) == tree.n_node_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how you can compute the depth of each node:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_depth(tree_clf):\n",
    "    tree = tree_clf.tree_\n",
    "    depth = np.zeros(tree.node_count)\n",
    "    stack = [(0, 0)]\n",
    "    while stack:\n",
    "        node, node_depth = stack.pop()\n",
    "        depth[node] = node_depth\n",
    "        if tree.children_left[node] != tree.children_right[node]:\n",
    "            stack.append((tree.children_left[node], node_depth + 1))\n",
    "            stack.append((tree.children_right[node], node_depth + 1))\n",
    "    return depth\n",
    "\n",
    "depth = compute_depth(tree_clf)\n",
    "depth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how to get the thresholds of all split nodes at depth 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree_clf.tree_.feature[(depth == 1) & (~is_leaf)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree_clf.tree_.threshold[(depth == 1) & (~is_leaf)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. to 6."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. The depth of a well-balanced binary tree containing _m_ leaves is equal to log₂(_m_), rounded up. log₂ is the binary log; log₂(_m_) = log(_m_) / log(2). A binary Decision Tree (one that makes only binary decisions, as is the case with all trees in Scikit-Learn) will end up more or less well balanced at the end of training, with one leaf per training instance if it is trained without restrictions. Thus, if the training set contains one million instances, the Decision Tree will have a depth of log₂(10<sup>6</sup>) ≈ 20 (actually a bit more since the tree will generally not be perfectly well balanced).\n",
    "2. A node's Gini impurity is generally lower than its parent's. This is due to the CART training algorithm's cost function, which splits each node in a way that minimizes the weighted sum of its children's Gini impurities. However, it is possible for a node to have a higher Gini impurity than its parent, as long as this increase is more than compensated for by a decrease in the other child's impurity. For example, consider a node containing four instances of class A and one of class B. Its Gini impurity is 1 – (1/5)² – (4/5)² = 0.32. Now suppose the dataset is one-dimensional and the instances are lined up in the following order: A, B, A, A, A. You can verify that the algorithm will split this node after the second instance, producing one child node with instances A, B, and the other child node with instances A, A, A. The first child node's Gini impurity is 1 – (1/2)² – (1/2)² = 0.5, which is higher than its parent's. This is compensated for by the fact that the other node is pure, so its overall weighted Gini impurity is 2/5 × 0.5 + 3/5 × 0 = 0.2, which is lower than the parent's Gini impurity.\n",
    "3. If a Decision Tree is overfitting the training set, it may be a good idea to decrease `max_depth`, since this will constrain the model, regularizing it.\n",
    "4. Decision Trees don't care whether or not the training data is scaled or centered; that's one of the nice things about them. So if a Decision Tree underfits the training set, scaling the input features will just be a waste of time.\n",
    "5. The computational complexity of training a Decision Tree is 𝓞(_n_ × _m_ log(_m_)). So if you multiply the training set size by 10, the training time will be multiplied by _K_ = (_n_ × 10 _m_ × log(10 _m_)) / (_n_ × _m_ × log(_m_)) = 10 × log(10 _m_) / log(_m_). If _m_ = 10<sup>6</sup>, then _K_ ≈ 11.7, so you can expect the training time to be roughly 11.7 hours.\n",
    "6. If the number of features doubles, then the training time will also roughly double."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: train and fine-tune a Decision Tree for the moons dataset._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. Generate a moons dataset using `make_moons(n_samples=10000, noise=0.4)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding `random_state=42` to make this notebook's output constant:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "\n",
    "X_moons, y_moons = make_moons(n_samples=10000, noise=0.4, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. Split it into a training set and a test set using `train_test_split()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_moons, y_moons,\n",
    "                                                    test_size=0.2,\n",
    "                                                    random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Use grid search with cross-validation (with the help of the `GridSearchCV` class) to find good hyperparameter values for a `DecisionTreeClassifier`. Hint: try various values for `max_leaf_nodes`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "params = {\n",
    "    'max_leaf_nodes': list(range(2, 100)),\n",
    "    'max_depth': list(range(1, 7)),\n",
    "    'min_samples_split': [2, 3, 4]\n",
    "}\n",
    "grid_search_cv = GridSearchCV(DecisionTreeClassifier(random_state=42),\n",
    "                              params,\n",
    "                              cv=3)\n",
    "\n",
    "grid_search_cv.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_search_cv.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Train it on the full training set using these hyperparameters, and measure your model's performance on the test set. You should get roughly 85% to 87% accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, `GridSearchCV` trains the best model found on the whole training set (you can change this by setting `refit=False`), so we don't need to do it again. We can simply evaluate the model's accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "y_pred = grid_search_cv.predict(X_test)\n",
    "accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: Grow a forest._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. Continuing the previous exercise, generate 1,000 subsets of the training set, each containing 100 instances selected randomly. Hint: you can use Scikit-Learn's `ShuffleSplit` class for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import ShuffleSplit\n",
    "\n",
    "n_trees = 1000\n",
    "n_instances = 100\n",
    "\n",
    "mini_sets = []\n",
    "\n",
    "rs = ShuffleSplit(n_splits=n_trees, test_size=len(X_train) - n_instances,\n",
    "                  random_state=42)\n",
    "\n",
    "for mini_train_index, mini_test_index in rs.split(X_train):\n",
    "    X_mini_train = X_train[mini_train_index]\n",
    "    y_mini_train = y_train[mini_train_index]\n",
    "    mini_sets.append((X_mini_train, y_mini_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. Train one Decision Tree on each subset, using the best hyperparameter values found above. Evaluate these 1,000 Decision Trees on the test set. Since they were trained on smaller sets, these Decision Trees will likely perform worse than the first Decision Tree, achieving only about 80% accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.base import clone\n",
    "\n",
    "forest = [clone(grid_search_cv.best_estimator_) for _ in range(n_trees)]\n",
    "\n",
    "accuracy_scores = []\n",
    "\n",
    "for tree, (X_mini_train, y_mini_train) in zip(forest, mini_sets):\n",
    "    tree.fit(X_mini_train, y_mini_train)\n",
    "    \n",
    "    y_pred = tree.predict(X_test)\n",
    "    accuracy_scores.append(accuracy_score(y_test, y_pred))\n",
    "\n",
    "np.mean(accuracy_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Now comes the magic. For each test set instance, generate the predictions of the 1,000 Decision Trees, and keep only the most frequent prediction (you can use SciPy's `mode()` function for this). This gives you _majority-vote predictions_ over the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_pred = np.empty([n_trees, len(X_test)], dtype=np.uint8)\n",
    "\n",
    "for tree_index, tree in enumerate(forest):\n",
    "    Y_pred[tree_index] = tree.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import mode\n",
    "\n",
    "y_pred_majority_votes, n_votes = mode(Y_pred, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Evaluate these predictions on the test set: you should obtain a slightly higher accuracy than your first model (about 0.5 to 1.5% higher). Congratulations, you have trained a Random Forest classifier!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_score(y_test, y_pred_majority_votes.reshape([-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "nav_menu": {
   "height": "309px",
   "width": "468px"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}