handson-ml/06_decision_trees.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Chapter 6 – Decision Trees**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains all the sample code and solutions to the exercises in chapter 6._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/ageron/handson-ml3/blob/main/06_decision_trees.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://kaggle.com/kernels/welcome?src=https://github.com/ageron/handson-ml3/blob/main/06_decision_trees.ipynb\"><img src=\"https://kaggle.com/static/images/open-in-kaggle.svg\" /></a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This project requires Python 3.7 or above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "assert sys.version_info >= (3, 7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It also requires Scikit-Learn ≥ 1.0.1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from packaging import version\n",
    "import sklearn\n",
    "\n",
    "assert version.parse(sklearn.__version__) >= version.parse(\"1.0.1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we did in previous chapters, let's define the default font sizes to make the figures prettier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.rc('font', size=14)\n",
    "plt.rc('axes', labelsize=14, titlesize=14)\n",
    "plt.rc('legend', fontsize=14)\n",
    "plt.rc('xtick', labelsize=10)\n",
    "plt.rc('ytick', labelsize=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's create the `images/decision_trees` folder (if it doesn't already exist), and define the `save_fig()` function which is used through this notebook to save the figures in high-res for the book:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "IMAGES_PATH = Path() / \"images\" / \"decision_trees\"\n",
    "IMAGES_PATH.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = IMAGES_PATH / f\"{fig_id}.{fig_extension}\"\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training and Visualizing a Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(max_depth=2, random_state=42)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "iris = load_iris(as_frame=True)\n",
    "X_iris = iris.data[[\"petal length (cm)\", \"petal width (cm)\"]].values\n",
    "y_iris = iris.target\n",
    "\n",
    "tree_clf = DecisionTreeClassifier(max_depth=2, random_state=42)\n",
    "tree_clf.fit(X_iris, y_iris)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**This code example generates Figure 6–1. Iris Decision Tree:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import export_graphviz\n",
    "\n",
    "export_graphviz(\n",
    "        tree_clf,\n",
    "        out_file=str(IMAGES_PATH / \"iris_tree.dot\"),  # path differs in the book\n",
    "        feature_names=[\"petal length (cm)\", \"petal width (cm)\"],\n",
    "        class_names=iris.target_names,\n",
    "        rounded=True,\n",
    "        filled=True\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.49.1 (0)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"360pt\" height=\"314pt\"\n",
       " viewBox=\"0.00 0.00 360.00 314.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 310)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-310 356,-310 356,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<path fill=\"#ffffff\" stroke=\"black\" d=\"M215,-306C215,-306 67,-306 67,-306 61,-306 55,-300 55,-294 55,-294 55,-235 55,-235 55,-229 61,-223 67,-223 67,-223 215,-223 215,-223 221,-223 227,-229 227,-235 227,-235 227,-294 227,-294 227,-300 221,-306 215,-306\"/>\n",
       "<text text-anchor=\"middle\" x=\"141\" y=\"-290.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">petal length (cm) &lt;= 2.45</text>\n",
       "<text text-anchor=\"middle\" x=\"141\" y=\"-275.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.667</text>\n",
       "<text text-anchor=\"middle\" x=\"141\" y=\"-260.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 150</text>\n",
       "<text text-anchor=\"middle\" x=\"141\" y=\"-245.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [50, 50, 50]</text>\n",
       "<text text-anchor=\"middle\" x=\"141\" y=\"-230.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = setosa</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<path fill=\"#e58139\" stroke=\"black\" d=\"M108,-179.5C108,-179.5 12,-179.5 12,-179.5 6,-179.5 0,-173.5 0,-167.5 0,-167.5 0,-123.5 0,-123.5 0,-117.5 6,-111.5 12,-111.5 12,-111.5 108,-111.5 108,-111.5 114,-111.5 120,-117.5 120,-123.5 120,-123.5 120,-167.5 120,-167.5 120,-173.5 114,-179.5 108,-179.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"60\" y=\"-164.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.0</text>\n",
       "<text text-anchor=\"middle\" x=\"60\" y=\"-149.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 50</text>\n",
       "<text text-anchor=\"middle\" x=\"60\" y=\"-134.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [50, 0, 0]</text>\n",
       "<text text-anchor=\"middle\" x=\"60\" y=\"-119.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = setosa</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M112.9,-222.91C105.11,-211.65 96.64,-199.42 88.8,-188.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"91.53,-185.9 82.96,-179.67 85.78,-189.88 91.53,-185.9\"/>\n",
       "<text text-anchor=\"middle\" x=\"78.49\" y=\"-200.56\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<path fill=\"#ffffff\" stroke=\"black\" d=\"M293.5,-187C293.5,-187 150.5,-187 150.5,-187 144.5,-187 138.5,-181 138.5,-175 138.5,-175 138.5,-116 138.5,-116 138.5,-110 144.5,-104 150.5,-104 150.5,-104 293.5,-104 293.5,-104 299.5,-104 305.5,-110 305.5,-116 305.5,-116 305.5,-175 305.5,-175 305.5,-181 299.5,-187 293.5,-187\"/>\n",
       "<text text-anchor=\"middle\" x=\"222\" y=\"-171.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">petal width (cm) &lt;= 1.75</text>\n",
       "<text text-anchor=\"middle\" x=\"222\" y=\"-156.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.5</text>\n",
       "<text text-anchor=\"middle\" x=\"222\" y=\"-141.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 100</text>\n",
       "<text text-anchor=\"middle\" x=\"222\" y=\"-126.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [0, 50, 50]</text>\n",
       "<text text-anchor=\"middle\" x=\"222\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = versicolor</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>0&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M169.1,-222.91C175.26,-214.01 181.84,-204.51 188.19,-195.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"191.13,-197.24 193.95,-187.02 185.38,-193.25 191.13,-197.24\"/>\n",
       "<text text-anchor=\"middle\" x=\"198.42\" y=\"-207.92\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<path fill=\"#4de88e\" stroke=\"black\" d=\"M202,-68C202,-68 102,-68 102,-68 96,-68 90,-62 90,-56 90,-56 90,-12 90,-12 90,-6 96,0 102,0 102,0 202,0 202,0 208,0 214,-6 214,-12 214,-12 214,-56 214,-56 214,-62 208,-68 202,-68\"/>\n",
       "<text text-anchor=\"middle\" x=\"152\" y=\"-52.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.168</text>\n",
       "<text text-anchor=\"middle\" x=\"152\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 54</text>\n",
       "<text text-anchor=\"middle\" x=\"152\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [0, 49, 5]</text>\n",
       "<text text-anchor=\"middle\" x=\"152\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = versicolor</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M195.93,-103.73C190.34,-94.97 184.41,-85.7 178.79,-76.91\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"181.62,-74.84 173.29,-68.3 175.72,-78.61 181.62,-74.84\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<path fill=\"#843de6\" stroke=\"black\" d=\"M340,-68C340,-68 244,-68 244,-68 238,-68 232,-62 232,-56 232,-56 232,-12 232,-12 232,-6 238,0 244,0 244,0 340,0 340,0 346,0 352,-6 352,-12 352,-12 352,-56 352,-56 352,-62 346,-68 340,-68\"/>\n",
       "<text text-anchor=\"middle\" x=\"292\" y=\"-52.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.043</text>\n",
       "<text text-anchor=\"middle\" x=\"292\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 46</text>\n",
       "<text text-anchor=\"middle\" x=\"292\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [0, 1, 45]</text>\n",
       "<text text-anchor=\"middle\" x=\"292\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = virginica</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>2&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M248.07,-103.73C253.66,-94.97 259.59,-85.7 265.21,-76.91\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"268.28,-78.61 270.71,-68.3 262.38,-74.84 268.28,-78.61\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x7fbfb8e45850>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from graphviz import Source\n",
    "\n",
    "Source.from_file(IMAGES_PATH / \"iris_tree.dot\")  # path differs in the book"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Graphviz also provides the `dot` command line tool to convert `.dot` files to a variety of formats. The following command converts the dot file to a png image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extra code\n",
    "!dot -Tpng {IMAGES_PATH / \"iris_tree.dot\"} -o {IMAGES_PATH / \"iris_tree.png\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAEQCAYAAACutU7EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABN40lEQVR4nO3deZxOZf/A8c/XYIxl7MbOjH0XsmRJiMhEhbK0yJM27T2PlKIoLbKUHkWUfoUWlK0UpUXG2MuSXfZBT3YzmLl+f9z3jFnufe59vu/X67zc97nOuc73HGq+c51rEWMMSimllFKhJF+gA1BKKaWUcpcmMEoppZQKOZrAKKWUUirkaAKjlFJKqZCjCYxSSimlQo4mMEoppZQKOX5LYESkkIgkishmEdkqIi/ZOEZE5G0R2S0iv4tIM3/Fp5RSSqnQkd+P10oBOhljzolIAeBXEfnGGJOQ6ZjuQC3r1gqYav1TKaWUUiqD31pgjMU569cC1i37LHq9gI+txyYAJUSkgr9iVEoppVRo8GsfGBGJEJFNwHHge2PMmmyHVAIOZvp+yLpPKaWUUiqDP18hYYxJBZqKSAlggYg0NMZsyXSI2Dot+w4RGQoMBShSpHDzunVr+iJcpbzCmEtculSILVs2AdCoeaPABqSUDdt/3w5Avcb1AhyJUlf9sf6Pk8aYsrbK/JrApDPGnBKRlcBNQOYE5hBQJdP3ysARG+dPA6YBtGjRxCQmLvNdsErlUlraQQ4erElcXCkAliQuCXBESikVGqpGVP3LXpk/RyGVtba8ICJRQBfgz2yHLQTuto5Gag2cNsYc9VeMSimllAoN/myBqQDMEpEILInT58aYxSLyIIAx5j1gKdAD2A1cAAb7MT6llFJKhQi/JTDGmN+Ba2zsfy/TZwM84q+YlFJKWfS4tgcAS9cuDXAkSrkmIH1glFJKBZctG7Y4P0ipIKIJjFJKKRYnLg50CEq5JewTmDNnkjl58iKXL6dicgzIVionEShQIIIyZaKIji4U6HCU8ovGzRsHOgSl3BLWCcyZM8kcP36RSpUqERVVCBFb08wolZUxhosXkzl8+DCAJjFKKRWEwno16pMnLclL4cJRmrwol4kIhQtHUalSJU6evBjocJTyiwkvTWDCSxMCHYZSLgvrBOby5VSiovS3Z+WZqKhCXL6cGugwlPKLSS9PYtLLkwIdhlIuC+tXSMagLS/KYyKi/aZUnvHEi08EOgSl3BLWCYxSSinXPDXqqUCHoJRbwvoVUri74YYuDBv2eKDDUEoppfxOW2CC0ODBQzh58m8WLfrK4XHz5n1OgQIF/BOUDfnyFeTzz+fQp8/tAYtBKeUdv6//HdDh1Cp0aAITgi5dukTBggUpVapUoENRSoWJni17AnAg9UCAI1HKNfoKyQVJSXNISKjJTz9FkpBQk6SkOX69/uDBQ4iP783rr79JlSqxVKkSC+R8hTR//gKaNGlG4cLRlC4dQ8eOnUlKSrJb7/vvT6dOnfpERRWjXLmK3HTTzVy5ciWj/MMPZ9GgQWOioopRp059Jk6cTFpaGgCxsbUA6NevP/nyFcz4nl5vrVr1iIwsQq1a9Zg+fYbL1127dh3duvWgbNkKFC9emvbtO7J6dUIun6BSypmGzRrSsFnDQIehlMu0BcaJpKQ57Nz5EGlpFwBISTnAzp0PARAT099vcfz0088ULx7NN98swtgYGnPs2DH69x/Eq6+O5fbbb+XcuXMkJCTarW/duvUMG/YYH300g3bt2nLq1Cl++GFlRvn06TMYNeol3n57Is2bN2PLlq0MHfoQBQoUYNiwh0lM/I2YmEpMm/YePXv2ICIiAoAFC77i0UcfZ8KE8XTt2oVly77nkUcepXz5GOLjezq97tmzZxk0aACTJr2FiDBlylRuvvkWdu7cRpkyZbz2PJVSWekijirUaALjxL59L2QkL+nS0i6wb98Lfk1gChUqxIwZ04mMjLRZfuTIUS5fvkyfPrdRrVo1ABo2tP/b1IEDByhSpAi33BJPsWLFqFatGk2aNMkoHzv2VV5//dWM/i2xsbEMH76XqVPfY9iwhylbtiwAJUoUp3z58hnnvfXWRAYNGsiwYQ8DULt2bTZs2MAbb4wnPr6n0+t26nRDljjfeWcS8+cv4NtvlzFo0EB3HplSSqkwpq+QnEhJOejWfl9p2LCB3eQFoEmTxnTp0plGja6hT59+TJ36PidOnLB7/I03dqFatarExdVm0KC7mTXrY86ePQvAiRMnOHjwIA8++AjFipXM2EaMeJ49e/Y6jHP79j9p2/a6LPvatm3Ltm3bnV4X4Pjx4zzwwMPUqVOfEiXKEB1diuPHj3PggH+ft1JKqeCmCYwTkZFV3NrvK0WKFHFYHhERwbJlS1m2bCmNGjVi5swPqV27Pps3b7Z5fLFixVi/PpHPPptNlSpVeO21N6hXrxFHjhzJ6OcydeoUNm5cm7H98cdGtmzZ5DRWW5MHpu9zdF2Ae+8dwrp165gwYTyrVv3Exo1rqVy5MpcuXXJ6XaWU51pUbkGLyi0CHYZSLtMExonY2DHky1c4y758+QoTGzsmQBHZJyK0adOaUaNeIDFxNRUrVuCzz76we3z+/Pnp1OkGxo17hc2bN3D+/HkWL15KTEwMlSpVYs+evdSsWTPHlq5AgQKkpmadar9evbr8+uuqLPtWrVpF/fr1nF4X4NdfVzFs2CPcfHMPGjRoQLFixTh69Kg3Ho9SyoHjR49z/OjxQIehlMu0D4wT6f1c9u17gZSUg0RGViE2doxf+7+4IiFhDcuXr6Bbt67ExJRj48ZNHDx4KEvikNnixUvYs2cvHTq0o1SpUvz440+cPXuWevXqAjBq1As89tgTlChRgh49buLy5cts2LCRw4ePMGLEcACqV6/GihU/cv31HYiMjKRkyZI888xT9OvXn+bNm9G1axe+/fY7Pv10DvPmfe7SdWvXrsWnn86mVatrOX/+AsOHj6BgwYJ+eIJK5W2JB+13+lcqGGkC44KYmP5Bl7BkV7x4NL/9tpopU/7LqVOnqFKlCiNHPme342uJEiX4+uuFjBnzChcuXKBGjTimT3+f9u3bAfCvf91HkSKFGT9+As89N5KoqCgaNKjPI488lFHH+PFv8PTT/6Fq1TgqVarEvn276N27F2+/PYm33prIk08+TbVq1Xj33XeIj+/p0nVnzJjGAw88TIsWralYsSKjRo102JdHKeUd5SuWd36QUkFEbA3JDSUtWjQxiYnLbJb9+edJ6tWr4+eIVDjZvn0Hdevmbvh2WtpBDh6sSVycZeJBnShMKaVcUzWi6npjjM3OWdoHRimlFMMfGM7wB4YHOgylXKYJjFJKKeZ8MIc5H/h3lnGlckP7wCillGLc1HGBDkEpt2gCo5RSioFDdaZrFVr0FZJSSimlQo4mMEoppfh+0fd8v+j7QIehlMv0FZJSSimG9B4C6DB/FTo0gVFKKUXnmzsHOgSl3OK3V0giUkVEfhSR7SKyVUQet3FMRxE5LSKbrNuL/opPKaXysg8XfsiHCz8MdBh+k3Q2ib6z+nL8nPfWf3JUpy+ul9f5sw/MFeBpY0w9oDXwiIjUt3HcL8aYptbtZT/GF3JuuKELw4blyAODXmxsLcaPn+C1+kL1OSilAmfyL5NJPJjI5J8n+6VOX1wvr/NbAmOMOWqM2WD9fBbYDlTy1/VDyeDBQ4iP7+30uHnzPmfcuLG+D8jLEhN/4+GHHwx0GEqpPCrpbBJfbP4CYwxfbP7CK60ijur0xfVUgEYhiUh14BpgjY3iNiKyWUS+EZEGds4fKiLrRGTdiRN/+zLUoHTp0iUASpUqRbFixQIcTVaXL192ekzZsmUpXLiwH6JxTVpaGqmpqYEOQ6mAqhpRlaoRVQMdhl9M/mUy6esAppk0r7SKOKrTF9dTAUhgRKQoMA94whhzJlvxBqCaMaYJ8A7wla06jDHTjDEtjDEtypYt7dN4K1TIT758BXJsFSr4r/9zeovM66+/SZUqsVSpEgvkfHUyf/4CmjRpRuHC0ZQuHUPHjp1JSkqyWeeAAYPo06dfln1paWlUrRr
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# extra code – just formatting details\n",
    "from matplotlib.colors import ListedColormap\n",
    "custom_cmap = ListedColormap(['#fafab0', '#9898ff', '#a0faa0'])\n",
    "plt.figure(figsize=(8, 4))\n",
    "\n",
    "lengths, widths = np.meshgrid(np.linspace(0, 7.2, 100), np.linspace(0, 3, 100))\n",
    "X_iris_all = np.c_[lengths.ravel(), widths.ravel()]\n",
    "y_pred = tree_clf.predict(X_iris_all).reshape(lengths.shape)\n",
    "plt.contourf(lengths, widths, y_pred, alpha=0.3, cmap=custom_cmap)\n",
    "for idx, (name, style) in enumerate(zip(iris.target_names, (\"yo\", \"bs\", \"g^\"))):\n",
    "    plt.plot(X_iris[:, 0][y_iris == idx], X_iris[:, 1][y_iris == idx],\n",
    "             style, label=f\"Iris {name}\")\n",
    "\n",
    "# extra code – this section beautifies and saves Figure 6–2\n",
    "tree_clf_deeper = DecisionTreeClassifier(max_depth=3, random_state=42)\n",
    "tree_clf_deeper.fit(X_iris, y_iris)\n",
    "th0, th1, th2a, th2b = tree_clf_deeper.tree_.threshold[[0, 2, 3, 6]]\n",
    "plt.xlabel(\"Petal length (cm)\")\n",
    "plt.ylabel(\"Petal width (cm)\")\n",
    "plt.plot([th0, th0], [0, 3], \"k-\", linewidth=2)\n",
    "plt.plot([th0, 7.2], [th1, th1], \"k--\", linewidth=2)\n",
    "plt.plot([th2a, th2a], [0, th1], \"k:\", linewidth=2)\n",
    "plt.plot([th2b, th2b], [th1, 3], \"k:\", linewidth=2)\n",
    "plt.text(th0 - 0.05, 1.0, \"Depth=0\", horizontalalignment=\"right\", fontsize=15)\n",
    "plt.text(3.2, th1 + 0.02, \"Depth=1\", verticalalignment=\"bottom\", fontsize=13)\n",
    "plt.text(th2a + 0.05, 0.5, \"(Depth=2)\", fontsize=11)\n",
    "plt.axis([0, 7.2, 0, 3])\n",
    "plt.legend()\n",
    "save_fig(\"decision_tree_decision_boundaries_plot\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access the tree structure via the `tree_` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.tree._tree.Tree at 0x7fbfa8b563b0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_clf.tree_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information, check out this class's documentation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# help(sklearn.tree._tree.Tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the extra material section below for an example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Estimating Class Probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.   , 0.907, 0.093]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_clf.predict_proba([[5, 1.5]]).round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_clf.predict([[5, 1.5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(min_samples_leaf=5, random_state=42)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "\n",
    "X_moons, y_moons = make_moons(n_samples=150, noise=0.2, random_state=42)\n",
    "\n",
    "tree_clf1 = DecisionTreeClassifier(random_state=42)\n",
    "tree_clf2 = DecisionTreeClassifier(min_samples_leaf=5, random_state=42)\n",
    "tree_clf1.fit(X_moons, y_moons)\n",
    "tree_clf2.fit(X_moons, y_moons)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsAAAAEQCAYAAAC++cJdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABO50lEQVR4nO3deZxcVZ3//9eneskeAiH7AoQkELYEyLBGgQlRQBHE3cHRcWGcUX8u48zPr+jg6He+MsNXXH6KCl9xnZGvS2QNS4IGjKBkbRISyEZCICELpLd0eqs6vz+qqlNdXXvdqntv1fv5ePSju6tu3XvqVtX7njr3nHPNOYeIiIiISL2I+F0AEREREZFqUgVYREREROqKKsAiIiIiUldUARYRERGRuqIKsIiIiIjUFVWARURERKSuqAIs4hEz+5CZdZa5jsvNzJnZiV6VS0SCKfFZf6ff5ag0M/uJmT0Y5m2Z2U1m9pKZxczsK16vX6pPFWCpmkQwOTP7Utrtga30mdlKM/tugYv/X2BWEeveZWafT7v5KWAK8Fqh6xGR0JoCPOB3ISQ3Mzse+B5wGzAN+N8V2s5XEsfC1J9XK7EtUQVYqq8b+Bczm1DpDZlZU6W3kbot59xR59yBctbjnOt1zr3qdIUakZqX+Kz3+F0OyeskoBF40Dm3zzlX1pm+PF4g/sUo+XN2BbdV11QBlmr7A7AL+HKuhczsjWb2FzPrNrP9ZvZNM2vOsXyyFfkaM3vGzHqBN1vcv5jZDjM7amYbzezGtMf+q5ntNrMeM3vVzH6WuP0nwGXAJ1K+jZ+cY1tDukCY2VsSz+Oomb1mZg+Y2XAzW0k8VG9LrjvteZyYso4bEuXuMbM9ZnazmVnK/bvM7Etm9kMzazezl83sn9PK8fdmtjWxPw+a2aNm1pjrNRCRwiTOFH3fzL5hZq8nPmOfNrNhZvY9M2tNnD7/QNrjBrpAJLLFmdk7zGy5mXWZ2WYzW1JgGZrM7DtmtjclK25Nuf9GM1ttZh1mdsDMfm1m01LuT2bP1Wa2NpFZfzSz6WZ2mZm1mFmnmT1oZuNTHveTxG1fSmR1p5n92MxG5ChryblcrAK3dauZvZC4f5eZ/aeZDU/c9yFgfWLRncnjQCllKVB/4otR8udgBbdV35xz+tFPVX6AnwAPAtcAvcCpidsvBxxwYuL/acAR4AfAPOCtwKvAN3KsO7mOjcCbiHdFmAD8O/Fv1FcBpwDvT6z7LYnHvQNoB94CzAQWAp9M3Hcc8S4JdwOTEz8NObb1IaAzpUxXAf3A/wTOAM4BPg+MBE4A9gD/llx3ln1xPhBNLDcX+BugE/hUynZ2Ee8y8UlgNvCpxDouTty/MFGOvyFe6Z4PfBZo9Ps9oR/91MIPsDKRI18B5gD/lPgMPgx8OvG5/BrQA0xNeZwD3pn4++TE/88D1ybW89PEZ3t0AWX4p0SmvDGRZZcAf5dy/4eJZ+8s4ALijRFPptyfzJ5ngDck8moT8CfgceDCRJa8CPx/KY/7CdAB/Bo4C3gz8ArwnbRlHkz5v+RcLmA/FLWtxDJfBi5NvAbXAC8BX0vcNyLxnBzwVySOA1m2/TDxfM76k6fsXwG6EvvvReAeYJbf7+9a/fG9APqpn5/UYEqE7z2Jv5PBm6z0/TuwHYikPPZDxA8eI7OsO7mOd6TcNgo4CrwhbdlvAcsSf38uEY5NWda7Evhuvm2llDG1Avyn5HPMsu5dwOezrDu5L/4L+H3aMl8BXk5bzy/TltkGfCnx9w1AGzDG7/eAfvRTiz+JnHg65X8DDgL3p9zWRPyL/ztTbstUAf77lPunJW5bVEAZvkO8omoFlvn0xLqnJ/5PZs+bU5b5ZOK281Ju+wqwKeX/nwCtpFTSgRsTeT0qZZlk9pedy3meV1HbyrKOjwPbU/5fmNgPJ+fZ9jTiX3ay/uR5/NXAu4l/+bgy8b56FRjv93u8Fn90ClT88i/An80s02CCecQPJrGU21YBzcRD5Nkc612T8vcZwHDgkWQXg4Qm4pVGiLdafBp40cweBR4hftAqpF/emjz3n0s8jMsxD3go7bZVwC1mNtY51564LX2f7AUmJv5eDuzm2HN8DFjqnOsos2wicszAZ9A558zsAPGzRMnb+szsMMc+l3nXQ/xzTAGPgXjWLAe2mtljwDLg4WSOmtl5wC3AAuJnoJLdqGYCL2fZ/v7E741pt6WX51k3uF/s08Tz+lSGZlOlc7nYbZHohvIZ4seX0cTP9DUUuS2cc68U+5i0xz+c+r+Z/RnYCXwQuL2cdctQ6gMsvnDOrQZ+C/xHhruN+LftjA/Ns+ojKX8n39/XEg/95M+ZxLsu4JzbA5wG/D3xU27fANaa2ag820nfVqUUui/6MtwXAUhUdM8j3rLwEvA/gOfNbKq3RRWpa5k+g1k/l4WsxyWaBQt4DM65dcRbkb+YWP6nwHIziyTy7FHip9c/QPxU/lWJh6aPrUgts0usO/22cuoOlc7lorZlZhcR72rwaGK5c4EvEa8kF8XMHk70gc76U8z6El8qniPeHUY8phZg8dMXgc0cC+KkzcC7zSyS0gq8iPjpwx1FrH8z8dNwJznnfp9tIedcN/FW1ocSg0ZeJd4f7LHENotuCUhYDywG7spyfyHr3kz8uadaRLwLRMEtuM65fuD3wO/N7BbgAPG+1XcWug4RCbZEJvwa+LXFB/H+mXir5hjgROCLzrkXIT641sNNn21mo5xzyUaBi8ie117kcqEK2dalwCvOua8lbzCzk4rYRqqPEu8z7InEQLzTiXcZFI+pAiy+cc5tN7M7iZ/qSnUH8dNRd5jZt4kP2riVeF/criLW35HoYvG/zcyAJ4mf3roIiDnn7kyM8G0E/kJ8kMJ7iLeAbEusZhdwQWLUbyfwehFP8d+BB8xsO/DfxFtz3wT8MPE8dgFvMLNfAD3OuUMZ1vENYLXFJ17/b+ItN/9E/MtDQczsrcRPRT6ZKP8VxA+IW4p4LiISYGb2OWAfsIF4hr2feOvpy8T7wvYAnzSz7xHvWvW1zGsqSSNwt5l9FZhKPK/vSqkQD/AolwtSyLaArcA0M/sb4l033gy8r9gdkNheWV0gEmV9gPiZuonEB+eNIt6aLx5TFwjx21eJz1AwIBEiVxM/FbWB+CwMv6SISl+KLxMftPF54qeSlhMfYfxi4v5W4CPAH4mPeH4HcEOylYT4hOe9xFsSDhLvL1cQ59wy4O2J57IeeIJ45TPZqv2vwAzirSQZp7pJnNZ8V6Jcm4gfWG4FCr04B8Sf4/XACuIjzD8PfNQ598ci1iEiwdYB/DPxWRzWET/Vf7VzrsvFp9L6IPEc2Ey8L/DnPNz2E8Tz9Q/A74ifbfqXHMuXm8vFyLkt59wDxC9w8S3i/ZWXEM9mP0wnfqx7AVhK/EvLRc653T6Vp6bZsS5GIiIiIoVLdLU40Tn3Vr/LIlIMtQCLiIiISF3xpQJsZncnrkSzKcv9l5tZm5ltSPz4dTpCRETEd2b2gxyzC/zA7/JVg5nNzDPLQsFd1ER86QJhZm8k3rH9Z865szLcfznxCwTolIqIiNQ9M5sIjM1yd7tz7kA1y+MHi1++/eQci+xKzHgjkpcvs0A4556s8LW0RUREakaiglvzldxcEpXb7X6XQ2pDkPsAX2xmLYmJpc/0uzAiIiIiUhuCOg/wOuITV3ea2TXAvWS5EoqZ3QTcBDBq1IjzTzv95GqVUUTK0kxPdx9tbW20tbUxasQwpk083u9CBcraTdsPOecmVGr9yk+REHNNdHf30draRnt7G8ePHcXEE7L1kqlPuTLUt2nQEl0gHszUBzjDsruAhVkuFDDg/IVnuKef+S9vCigiFWVuBju27eehBx/i4WXLOG/eDP7js+/2u1iBEpnztrXOuYXV2JbyUyRkotPYsvll7r//AZY/9ig3LD6PT9/4Zr9LFSi5MjSQXSDMbHLiqi2Y2QXEy/mav6USERERkVrgSxcIM/slcDlwopm9TPyqNE0AzrkfAO8E/sHM+oGjwHudrtgRai0btvD48lW0tXZw3LgxLF6yiPkL5vldLBGRUFCGinjLr1kgcl5
      "text/plain": [
       "<Figure size 720x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# extra code – this cell generates and saves Figure 6–3\n",
    "\n",
    "def plot_decision_boundary(clf, X, y, axes, cmap):\n",
    "    x1, x2 = np.meshgrid(np.linspace(axes[0], axes[1], 100),\n",
    "                         np.linspace(axes[2], axes[3], 100))\n",
    "    X_new = np.c_[x1.ravel(), x2.ravel()]\n",
    "    y_pred = clf.predict(X_new).reshape(x1.shape)\n",
    "    \n",
    "    plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=cmap)\n",
    "    plt.contour(x1, x2, y_pred, cmap=\"Greys\", alpha=0.8)\n",
    "    colors = {\"Wistia\": [\"#78785c\", \"#c47b27\"], \"Pastel1\": [\"red\", \"blue\"]}\n",
    "    markers = (\"o\", \"^\")\n",
    "    for idx in (0, 1):\n",
    "        plt.plot(X[:, 0][y == idx], X[:, 1][y == idx],\n",
    "                 color=colors[cmap][idx], marker=markers[idx], linestyle=\"none\")\n",
    "    plt.axis(axes)\n",
    "    plt.xlabel(r\"$x_1$\")\n",
    "    plt.ylabel(r\"$x_2$\", rotation=0)\n",
    "\n",
    "fig, axes = plt.subplots(ncols=2, figsize=(10, 4), sharey=True)\n",
    "plt.sca(axes[0])\n",
    "plot_decision_boundary(tree_clf1, X_moons, y_moons,\n",
    "                       axes=[-1.5, 2.4, -1, 1.5], cmap=\"Wistia\")\n",
    "plt.title(\"No restrictions\")\n",
    "plt.sca(axes[1])\n",
    "plot_decision_boundary(tree_clf2, X_moons, y_moons,\n",
    "                       axes=[-1.5, 2.4, -1, 1.5], cmap=\"Wistia\")\n",
    "plt.title(f\"min_samples_leaf = {tree_clf2.min_samples_leaf}\")\n",
    "plt.ylabel(\"\")\n",
    "save_fig(\"min_samples_leaf_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.898"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_moons_test, y_moons_test = make_moons(n_samples=1000, noise=0.2,\n",
    "                                        random_state=43)\n",
    "tree_clf1.score(X_moons_test, y_moons_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.92"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_clf2.score(X_moons_test, y_moons_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's prepare a simple quadratic training set:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Code example:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(max_depth=2, random_state=42)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "np.random.seed(42)\n",
    "X_quad = np.random.rand(200, 1) - 0.5  # a single random input feature\n",
    "y_quad = X_quad ** 2 + 0.025 * np.random.randn(200, 1)\n",
    "\n",
    "tree_reg = DecisionTreeRegressor(max_depth=2, random_state=42)\n",
    "tree_reg.fit(X_quad, y_quad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.49.1 (0)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"682pt\" height=\"269pt\"\n",
       " viewBox=\"0.00 0.00 682.00 269.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 265)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-265 678,-265 678,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<path fill=\"#f6d6be\" stroke=\"black\" d=\"M402,-261C402,-261 271,-261 271,-261 265,-261 259,-255 259,-249 259,-249 259,-205 259,-205 259,-199 265,-193 271,-193 271,-193 402,-193 402,-193 408,-193 414,-199 414,-205 414,-205 414,-249 414,-249 414,-255 408,-261 402,-261\"/>\n",
       "<text text-anchor=\"middle\" x=\"336.5\" y=\"-245.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x1 &lt;= &#45;0.303</text>\n",
       "<text text-anchor=\"middle\" x=\"336.5\" y=\"-230.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 0.006</text>\n",
       "<text text-anchor=\"middle\" x=\"336.5\" y=\"-215.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 200</text>\n",
       "<text text-anchor=\"middle\" x=\"336.5\" y=\"-200.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.088</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<path fill=\"#eb9d65\" stroke=\"black\" d=\"M316,-157C316,-157 185,-157 185,-157 179,-157 173,-151 173,-145 173,-145 173,-101 173,-101 173,-95 179,-89 185,-89 185,-89 316,-89 316,-89 322,-89 328,-95 328,-101 328,-101 328,-145 328,-145 328,-151 322,-157 316,-157\"/>\n",
       "<text text-anchor=\"middle\" x=\"250.5\" y=\"-141.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x1 &lt;= &#45;0.408</text>\n",
       "<text text-anchor=\"middle\" x=\"250.5\" y=\"-126.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 0.002</text>\n",
       "<text text-anchor=\"middle\" x=\"250.5\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 44</text>\n",
       "<text text-anchor=\"middle\" x=\"250.5\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.172</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M308.58,-192.88C301.07,-183.98 292.86,-174.24 285.03,-164.96\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"287.7,-162.69 278.58,-157.3 282.35,-167.2 287.7,-162.69\"/>\n",
       "<text text-anchor=\"middle\" x=\"276.46\" y=\"-178.51\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<path fill=\"#fae6d7\" stroke=\"black\" d=\"M489,-157C489,-157 358,-157 358,-157 352,-157 346,-151 346,-145 346,-145 346,-101 346,-101 346,-95 352,-89 358,-89 358,-89 489,-89 489,-89 495,-89 501,-95 501,-101 501,-101 501,-145 501,-145 501,-151 495,-157 489,-157\"/>\n",
       "<text text-anchor=\"middle\" x=\"423.5\" y=\"-141.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x1 &lt;= 0.272</text>\n",
       "<text text-anchor=\"middle\" x=\"423.5\" y=\"-126.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 0.005</text>\n",
       "<text text-anchor=\"middle\" x=\"423.5\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 156</text>\n",
       "<text text-anchor=\"middle\" x=\"423.5\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.065</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>0&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M364.75,-192.88C372.34,-183.98 380.65,-174.24 388.57,-164.96\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"391.27,-167.18 395.1,-157.3 385.95,-162.64 391.27,-167.18\"/>\n",
       "<text text-anchor=\"middle\" x=\"397.08\" y=\"-178.52\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<path fill=\"#e58139\" stroke=\"black\" d=\"M143,-53C143,-53 12,-53 12,-53 6,-53 0,-47 0,-41 0,-41 0,-12 0,-12 0,-6 6,0 12,0 12,0 143,0 143,0 149,0 155,-6 155,-12 155,-12 155,-41 155,-41 155,-47 149,-53 143,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"77.5\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 0.001</text>\n",
       "<text text-anchor=\"middle\" x=\"77.5\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 20</text>\n",
       "<text text-anchor=\"middle\" x=\"77.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.213</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M189.99,-88.95C171.66,-78.93 151.6,-67.98 133.57,-58.13\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"134.8,-54.81 124.34,-53.09 131.44,-60.95 134.8,-54.81\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<path fill=\"#f0b489\" stroke=\"black\" d=\"M316,-53C316,-53 185,-53 185,-53 179,-53 173,-47 173,-41 173,-41 173,-12 173,-12 173,-6 179,0 185,0 185,0 316,0 316,0 322,0 328,-6 328,-12 328,-12 328,-41 328,-41 328,-47 322,-53 316,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"250.5\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 0.001</text>\n",
       "<text text-anchor=\"middle\" x=\"250.5\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 24</text>\n",
       "<text text-anchor=\"middle\" x=\"250.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.138</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M250.5,-88.95C250.5,-80.72 250.5,-71.85 250.5,-63.48\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"254,-63.24 250.5,-53.24 247,-63.24 254,-63.24\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<path fill=\"#ffffff\" stroke=\"black\" d=\"M489,-53C489,-53 358,-53 358,-53 352,-53 346,-47 346,-41 346,-41 346,-12 346,-12 346,-6 352,0 358,0 358,0 489,0 489,0 495,0 501,-6 501,-12 501,-12 501,-41 501,-41 501,-47 495,-53 489,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"423.5\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 0.001</text>\n",
       "<text text-anchor=\"middle\" x=\"423.5\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 110</text>\n",
       "<text text-anchor=\"middle\" x=\"423.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.028</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>4&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M423.5,-88.95C423.5,-80.72 423.5,-71.85 423.5,-63.48\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"427,-63.24 423.5,-53.24 420,-63.24 427,-63.24\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<path fill=\"#edaa79\" stroke=\"black\" d=\"M662,-53C662,-53 531,-53 531,-53 525,-53 519,-47 519,-41 519,-41 519,-12 519,-12 519,-6 525,0 531,0 531,0 662,0 662,0 668,0 674,-6 674,-12 674,-12 674,-41 674,-41 674,-47 668,-53 662,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"596.5\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 0.002</text>\n",
       "<text text-anchor=\"middle\" x=\"596.5\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 46</text>\n",
       "<text text-anchor=\"middle\" x=\"596.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.154</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>4&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M484.01,-88.95C502.34,-78.93 522.4,-67.98 540.43,-58.13\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"542.56,-60.95 549.66,-53.09 539.2,-54.81 542.56,-60.95\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x7fbfb925c760>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# extra code – we've already seen how to use export_graphviz()\n",
    "export_graphviz(\n",
    "    tree_reg,\n",
    "    out_file=str(IMAGES_PATH / \"regression_tree.dot\"),\n",
    "    feature_names=[\"x1\"],\n",
    "    rounded=True,\n",
    "    filled=True\n",
    ")\n",
    "Source.from_file(IMAGES_PATH / \"regression_tree.dot\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(max_depth=3, random_state=42)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_reg2 = DecisionTreeRegressor(max_depth=3, random_state=42)\n",
    "tree_reg2.fit(X_quad, y_quad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.30265072, -0.40830374, -2.        , -2.        ,  0.27175756,\n",
       "       -2.        , -2.        ])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_reg.tree_.threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.30265072, -0.40830374, -0.45416115, -2.        , -2.        ,\n",
       "       -0.37022041, -2.        , -2.        ,  0.27175756, -0.21270403,\n",
       "       -2.        , -2.        ,  0.40399227, -2.        , -2.        ])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_reg2.tree_.threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsAAAAEQCAYAAAC++cJdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABoTUlEQVR4nO29eXwV1f3//zy5WYgJEESFgmwKFO0HFxKXSKuxiksLrl0QLdaqkVr9aottRWsbxYLtp/hBW6sEtWpbq21dWqi2Ll+i/r4J/RAQtVbFDREQURogbCG5Ob8/5t5kcrnLzNzZ7r3v5+Mxj+TOds6ZOec17znzPu+jtNYIgiAIgiAIQqFQFHQGBEEQBEEQBMFPxAAWBEEQBEEQCgoxgAVBEARBEISCQgxgQRAEQRAEoaAQA1gQBEEQBEEoKMQAFgRBEARBEAoKMYAFwQJKqRqllFZKjfYhrW8qpXZ4nY4gCIJXiGYKYUcMYEEIEKXUWqXUdT6kM0cptUIptV0p9YlSaolS6r+8TlcQBMFNfNTM7yilXo1p5nalVItS6stepyv4hxjAglAY1AG/Bk4Avgh0Ac8ppfYPMlOCIAghZT3wQ2ASUAP8X+BJpdQRgeZKcA0xgIXAUUo1KaXuVkotUEr9J9ZDeY1SqkwpdZdSaqtSap1S6humY25TSr2llNod6xH4uVKqX2ybUko9q5R6TimlYusqlVJvK6V+ZTFPZyil3lRK7VFKvQSMT7LPCUqpF5RSu5RSG2JlGJBQrnuUUncopdpiy38rpYri24FRwH/HPhXqhPOfopT6l1Jqp1JqmVJqjP2ra6C1Pl1r/Rut9b+01q8B3wAOBCY7PacgCMEgmumLZv5Fa/201vodrfUarfWNQDtQ6/ScQrgQA1gICxdiiMtxwG3AQuBJYA3G2/eDwL1KqWGx/XcC3wIOA64EpgM3Amhjfu+LgaOA+KeyO4G9wPczZUQpNSKW9rOxc/wS+HnCPhOBZ4C/AkcC58X2vT9JuYowRPMKoB64NrbtPIxehluAz8SWOGXAnFgZa4Eq4B5T+l9QSu3IsNyQppj9Y/lqS381BEEIKaKZPmmmUiqilJoOVALNma6HkCNorWWRJdAFaAJaTL8V8AnwV9O6Egwx/kqKc8wC3klYdw7QAcyN/T3SYn7mYTxElGndjwANjI79fgi4L+G4o2L7HGQqV7LzrDf9Xgtcl3Ceb8bO81nTugtj5S+K/S4HxmZY9k9Txj8CLwORoO+/LLLIYm8RzfRHM4GJwA4Ml7GtwJeDvveyuLcUIwjh4NX4P1prrZTaDLxmWteplGoDDgJQSn0Fo1dgLMZbeSS2YDrmSaXUwxgC+gOt9SsW83IYsFzHFDBGS8I+1cBYpdTXTetU7O+hwObY/8nOM1cpNUBrvT1NHjq01m+Zfm/EeKBVAf/RWu8G3rFUmgSUUrcDnwc+r7WOOjmHIAiBI5rZFy808y0MI70KOB94UClVp7X+l83zCCFEXCCEsNCZ8FunWFeklDoeeAT4BzANOBpDsEvMO8f8244BohiibxWVeReKgHsxxDG+HAmMA1bbSCsVXQm/4w+EuC+co895Sqn/AS4Avqi1fs+FfAqCEAyimX1xXTO11nu14QPcqrWeE8vnd13IqxACpAdYyEUmAxu01nPjK5RSo5Ls998YfmFTgH8opZ7SWv/Fwvn/DZyvlFKmnojjE/ZZBXxOa52pR+G4JOfZaOrJ2EtCL4xFWjEeIOn4j/mHUuoODL+/Oq31mw7SFAQhNxHNdKCZSSjCuD5CHiAGsJCLrAGGK6UuxPg8djpGr2YPSqkzMAZQfEFr/U+lVAPGgJB/aq03ZTj/PcBsYKFS6tcYfmCzEvb5GbBcKXUPsAhjMMoEYJrW+grTfsMSzvN94FbT9rXAF5RSv8P4hPeplQtg93OeUuoujMgP5wBtSqmhsU07tNYSQF4Q8hvRTPuaeRvwN+BDjEHDMzDCSUos4DxBXCCEnENrvQSjp2Ihhh/cFODH8e1KqQOBB4Bbtdb/jK2+DXgd+I1SKu3nOq31OozRxmcAr2B88ro+YZ9XgROB0cALsf3mAx8nnO73GL0V/wQWA/cB/2Pa/mNgBPAuxiAWr7gSQ8SfBz4yLZ4HlBcEIVhEMx0xFPgdhh/w8xiuIWdqrZ/2ME3BR1RfX3NBENwiFrPyX1rrq4LOiyAIQtgRzRT8RHqABUEQBEEQhILCVwNYGTPFvKWUekcpdX2S7RcqY+7tV5VSzUqpI03b1iqlXlNKrVZKtfqZbyG/UMZMQ6lGAd+T+QyCIAiFg2imkI/45gKhlIpgOOJPwZjJZQVwgdb636Z9TgDe0Fq3KaXOBBq01sfFtq0Faqw6vAtCKpRSBwEDUmzerrXenGKbIAhCwSGaKeQjfkaBOBZj1pn3AJRSjwBnY4RPAUBrbZ5icDlwsI/5EwqEmFiLYAuCIFhANFPIR/x0gRiOEU4kzvrYulRcCphHW2rgGaXUSqVUvQf5EwRBEARBEAoAP3uAk4VRSep/oZQ6GcMA/rxp9WSt9cbYp5hnlVJvaq1fTDiuHqgHqKioqJ4wYYI7ObfAypUrAaiurvYtzTDmQRCypRDr8c6d0N4O/ftDRUX6fVeuXPmp1vpAN9J1QzM3btzIsGHDbB3j5B47ScftPGSL22UQhFzRS7frvlXNTKuXWmtfFqAW+Ifp9xxgTpL9jsCI7zc+zbkagOvSpVddXa2bm7WeN0/r5mbtORjGvPcJhTwPgpAthVaPm5u1Li/XOhIx/mbSK6BVe6DRTjXzF7/4hfWde8tg+x47ScftPGSL22UQhFzRSzfrvh3NTKeXfvYArwDGKaXGABswpmSdYd5BKTUSeBz4htZ6jWl9BVCktW6P/X8acEu6xHbuhFNOgb17obQUnn8eamtdLpEgCEKWNDUZOhWNGn+bmoLRKqeaOXv2bO8z52M6XpIPZRAEJ7hZ993STN8MYK11l1LqKuAfGLO83K+1fl0pNSu2/R6MGV4GA7+OTTzTpbWuAYYAT8TWFQMPa63/ni699nZ/HyqXX365dycXhAKi0NpSXZ1hcMYNz7q6YPLhVDOHDRvGxo0bbaXl5B47SSds5EMZhHCRK3rpZt13SzPzdia4ww6r0R980FpQPcDx2Srz9Z4KQr7S0mIYnHV1mXVKKbUy1jHgKk410y+/VrfTCUIvxQdYKFTcrvtWNTOdXvrpAuErFRWGgFt9qAiCIARFbW3wGuVUM9esWeOLUedXOl6SD2UQBCe4Xffd0My87QGuqanRra3uTxiX6q0jDCMxpQdYyAfC0JbCjFc9wE41s66ujqamppTbk2mmk3ucKR27BKGXbpdBEHJFL4Oq++n0UgxgG7S0pB4kEgbjMwx5EIRskXqcHs8M4IoK3Tp+PHR0QFkZtLWBUlBV1fd/8/aKCrjmGqhPHpo9lWaG4R6HIQ+CkC1Sj9NTkC4QXhCW0dpCL9u3b2fz5s10dnYGnZVAKSkp4aCDDmLAgFSzlQpCBnbtgtWrbR9Wf8UVNEJSI9hNzayvr6exsdHZwSEhH8ogCE4IY90XA9gGYRmtLRhs376djz/+mOHDh1NeXt7zJlxoaK3ZvXs3GzZsABAjWPCVGoDHHktqALupmTU1rnd6+04+lEEQnBDGup/XLhC//GWr64PgUvkAh+EzRBjy4CfvvPMOw4YNY7/99gs6K6Fg165dbNy4kbFjxwadlawotHpsF89cIJTSK+Jp2D140aK0bhCJmhmGexyGPAhCtkg9Tk9BukB4NRFGGEZrCwadnZ2Ul5cHnY3QUF5eXvCuIIJzdrIfqxlPGR2MHFdG5d4MPsDvvw/bttE/EqE9hfEL7mlm//79aW9vz/5EAZIPZRAEJ4Sx7uetAez3RBhCMBSq20My5FoI2fAmhzGJViIRmHsJzJmT4YCf/hR+9CM2XnONL/nLhwkk8qEMguCEMNb9oqAz4BX9+xs9v5GI+OsKghBeWlpg/nzjb5AoZVMvS0oAaHr/fU/zFScfwoflQxkEwQlu1X039TJve4D9ngjDi5jDglCIFFJbShda0W8++1mYOdOGXsYM4MbVq5l
      "text/plain": [
       "<Figure size 720x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# extra code – this cell generates and saves Figure 6–5\n",
    "\n",
    "def plot_regression_predictions(tree_reg, X, y, axes=[-0.5, 0.5, -0.05, 0.25]):\n",
    "    x1 = np.linspace(axes[0], axes[1], 500).reshape(-1, 1)\n",
    "    y_pred = tree_reg.predict(x1)\n",
    "    plt.axis(axes)\n",
    "    plt.xlabel(\"$x_1$\")\n",
    "    plt.plot(X, y, \"b.\")\n",
    "    plt.plot(x1, y_pred, \"r.-\", linewidth=2, label=r\"$\\hat{y}$\")\n",
    "\n",
    "fig, axes = plt.subplots(ncols=2, figsize=(10, 4), sharey=True)\n",
    "plt.sca(axes[0])\n",
    "plot_regression_predictions(tree_reg, X_quad, y_quad)\n",
    "\n",
    "th0, th1a, th1b = tree_reg.tree_.threshold[[0, 1, 4]]\n",
    "for split, style in ((th0, \"k-\"), (th1a, \"k--\"), (th1b, \"k--\")):\n",
    "    plt.plot([split, split], [-0.05, 0.25], style, linewidth=2)\n",
    "plt.text(th0, 0.16, \"Depth=0\", fontsize=15)\n",
    "plt.text(th1a + 0.01, -0.01, \"Depth=1\", horizontalalignment=\"center\", fontsize=13)\n",
    "plt.text(th1b + 0.01, -0.01, \"Depth=1\", fontsize=13)\n",
    "plt.ylabel(\"$y$\", rotation=0)\n",
    "plt.legend(loc=\"upper center\", fontsize=16)\n",
    "plt.title(\"max_depth=2\")\n",
    "\n",
    "plt.sca(axes[1])\n",
    "th2s = tree_reg2.tree_.threshold[[2, 5, 9, 12]]\n",
    "plot_regression_predictions(tree_reg2, X_quad, y_quad)\n",
    "for split, style in ((th0, \"k-\"), (th1a, \"k--\"), (th1b, \"k--\")):\n",
    "    plt.plot([split, split], [-0.05, 0.25], style, linewidth=2)\n",
    "for split in th2s:\n",
    "    plt.plot([split, split], [-0.05, 0.25], \"k:\", linewidth=1)\n",
    "plt.text(th2s[2] + 0.01, 0.15, \"Depth=2\", fontsize=13)\n",
    "plt.title(\"max_depth=3\")\n",
    "\n",
    "save_fig(\"tree_regression_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsAAAAEQCAYAAAC++cJdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAACA20lEQVR4nO2dd5gb1bn/P0dli9dlXXHDBWMMhgVDTFEoXiAJJQWCU/iRYEiANS0JKZhLEggJyTU45ZJC8dKdC8lNYiDUYIplU0QxGFiwMWBjwLjb677e1Urn98eZkUbaUdvVStrV+3kePZpyZnRG0pz5zjtvUVprBEEQBEEQBKFc8BS7A4IgCIIgCIJQSEQAC4IgCIIgCGWFCGBBEARBEAShrBABLAiCIAiCIJQVIoAFQRAEQRCEskIEsCAIgiAIglBWiAAWhG5EKXW+UmpXF/dRr5TSSqkh+eqXIAjFwzqfv1bsfnQ3Sql7lFKP9uTPUko1KKU+VkpFlVLX5Xv/QvEQASwUFWvQ0kqpnyctL1nRp5QKKqX+kmXz/wP2y2Hfq5VSP0la/CIwAtiS7X4EQShpRgCPFLsTQnqUUgOBm4HfAqOA33ViHycopR5WSn1qXdPOd2mjlFLXKaXWKqVarGvMwV0+ACEtIoCFUmAvMEspNbS7P0gp5e/uz3B+lta6RWu9sSv70Vq3aa3Xa6laIwi9Aut8bi12P4SMjAV8wKNa63Va6848zesLvA38AGhJ0WYW8GPge8CRwEbgKaVUv058npAlIoCFUmAhsBq4Jl0j6076ZaXUXqXUBqXU/yilKtK0t63IpyulXlFKtQGnWHfbs5RSK6277Sal1LeTtr1WKfWRUqpVKbVeKTXPWn4PMA24zNq3VkqNS/NZHVwglFJftI6jRSm1RSn1iFKqSikVxAy4v7X3nXQcQxz7OMvqd6tS6hOl1M+UUsqxfrVS6udKqblKqR1KqTVKqSuT+jFTKfWe9X1uUko9qZTypfsNBEFIxLLW3aqU+r1Saqt1Lv1AKVWplLpZKbXNeoR+rmObmAuENX5opdR0pdRTSqk9SqllSqnPZ/n5fqXUnyzroT0e3OBY/22l1KtKqZ1KqY1KqX8qpUY51tvjy2lKqdescek5pdRopdQ0pdSbSqldSqlHlVKDHdvdYy37uTUe71JK3a2Uqk7T106PvbmS5WfdoJRaYa1frZSao5SqstadDyy1mq6yx/pc+6G1flxr/VOt9b+AqFs/gSuAG7TW87XWbwPnAf2Ac3L9PCEHtNbyklfRXsA9wKPA6UAbMMFaXg9oYIg1PwrYDdwGHAR8CVgP/D7Nvu19NAFfwLgiDAV+A6wATgXGYwaZ3cAXre2mAzuALwJjgKnA5da6ARiXhLuA4dbLm+azzgd2Ofp0KtAO/BqYDBwK/AToAwwCPgF+ae87xXfxGSBitTsA+BawC/ie43NWY1wmLgf2x1gWNBCw1k+1+vEtjOg+DPgh4Cv2f0Je8upJLyBojRfXARMxljwNPIGx+u0PXA+0AiOtbTTwNWt6nDX/LvBlax/3Wudv3yw+/8fWuHGCNV59FviOY/13MePrfsBRGIPDYsd6e3x5BTjeGpPeBl4AngGOtsaLD4E/O7a7B9gJ/BM4BDgF+BT4U1KbRx3znR57s/gecvosq801wLHWb3A68DFwvbWu2jomjbHK2mP98ZjxNt3rpyn6uAs4P2nZfvZnJC1/DLi32P/v3vwqegfkVd4v56BlDcx/t6btQdkWfb8BPgA8jm3Px1xU+qTYt72P6Y5lNZjHUMcntb0JeNya/pE1cPpT7DcI/CXTZzn66BTAL9jHmGLfq4GfpNi3/V3cBzyb1OY6YE3Sfv6W1OZ94OfW9FnAdqBfsf8D8pJXT35Z40HIMa+ATcDDjmV+zA2+LXrdBPBMR/tR1rLjsvj8P2GEqsqyvwda+x5tzdvjyymONpdby45wLLsOeNsxfw+wDYdIB75tjck1jjb2+N7lsTfDceX0WSn2cTHwgWN+qvU9jHMsq8bc1KR7DUqxfzcB/FnrM8YkLb8LeLLY/+/e/JLHnUIpMQt4SSnlFmhwEOYi43yE9DxQgRlw3kqz3yWO6clAFfAf28XAwo8RjWAsGj8APlRKPQn8B3Mxy8Znb0mG9YdjBuqucBDGOuDkeeAXSqn+Wusd1rLk72QtMMyafgr4iPgxLgAe0Frv7GLfBKEciZ1rWmutlNqIeRpkLwsrpZqJn39p94E5V8nQ3uYezPn8nlJqAfA48IQ9ViqljgB+AUzBPGWyXaXGAGtSfP4G670paVlyf97SiX6xIcyYPIGO4093j725fhaWG8oVmGtIX4yF15tux1rrFowxJt8kx3gol2VCHhEfYKFk0Fq/CswHbnRZnW4wyDRI7HZM2//5L2MuCPbrYIzrAlrrT4BJwEzM47jfA68ppWoyfE7yZ3UX2X4XYZd1HgBL6B4BfAPz2O9q4F2l1Mj8dlUQygK3cy3l+ZdpH9oyAWZob7d9HWNF/qnV/l5MAJXHGrOeBPYA52Ie5Z9qbZocP+Hsr7b2nbysK5qhu8fenD5LKXUM8HfM9/NljHHi5xiRnBKl1PGWv3O6109z6Ot663140vJhxG9EhG5ALMBCqfFTYBnxQdpmGfANpZTHYQU+DvNYcWUO+1+GeUQ3Vmv9bKpGWuu9GCvrY1ZAyXqMr9gC6zPTWgnSsBQ4Gbg9xfps9r0Mc+xOjsO4QGRtwdVatwPPAs8qpX6BiTz+EtCY7T4EQSg+1nn/T+CfygTqvoSxavYDhmB8Uj8EE0Cbx4+uU0rVaK3tG/9jSD0m52PszZZsPutY4FOt9fX2AqXU2Cz2vQQjptOxNZtOWnyIOcbPA69a/ajC+BpfmWY7oYuIABZKCq31B0qpRsxjMCe3YB5V3aKU+iMmcOAGjC/unhz2v9NysfidFX27GPPo6xggqrVutKJ/fcDLGJ+tb2KsI+9bu1kNHGVFBO8it8HuN8AjSqkPgPsx1twvAHOt41gNHK+U+l+gVWu92WUfvwdeVSYp+/0Yq86PMTcPWaGU+hLmMeViq/8nYi6Wy3M4FkEQioxS6kfAOuANzDh1DsZ6ugbjC9sKXK6UuhnjPnW9+546hQ+4Syn1K2AkZky+3SGIY+Rp7M2KbD4LeA8YpZT6FsZ14xTg/2Wx75xcIJRSfTE3I2As02OUUlOArVrrjy2XmZuAnyml3rX69XPM8d+f7ecIuSMuEEIp8itMhoIYWutPgdMwj6newAQI/I0cRJ+DazABHT8B3sH4z03H3ImDCey4AHgOEw09HTjLtqBgkqG3YawMmzC+dFmhtX4c+Kp1LEuBRRjxaVu1rwX2xVhQNqXYx+vA161+vY256NwAZFucA8wxngk8jYk+/wlwodb6uRz2IQhC8dmJsRS+AryOsU6eprXeo7XehEmpdSZmvPoFJtAsXyzCjKELgQcxT5RmpWnf1bE3F9J+ltb6EUyBi5sw/sqfx4y/+WYqZqxfigmg+6U1/StHmznAHzBFN5ZgCqV8QWIyuhcVdzUSBEEQBEHIjOVqMURr/aVi90UQOoNYgAVBEARBEISyoqACWCl1qlV15QOl1H+5rP+WUuot6/WiUuowx7rVViWXN5RSmVJNCYIgCEKPRyl1W5psA7cVu3+FQCk1JkPWhazd0ATBpmAuEEopL8a5+/MY5/xXgf+ntV7maPNZYLnWulkpdRpwndb6aGvdamBqiqAgQRAEQeh1KKWGAf1TrN6htd5YyP4UA2VKtI9L02S1ldVGELKmkFkgjsJUWFkFoJT6O3AGxjEfAK31i472LwGjC9g/QRAEQSgpLIHb60VuOixx2x3FJ4QyppAuEKMw9cpt1ljLUnEBppa6jQYWKKVeU0o1dEP/BEEQBEEQhDKgkBZg5bLM1f9CKXUiRgA7k/0fq7Veaz0Oekop9a7WenHSdg1AA0BNTc1nDjzwQFi3DtauheHDYdQo+OAD2L4dampgt5WqcPBgGDeuywcoCILQaXbuJLyxmR2+QVQPqKDPyibwW0WpwmGoq4MKU7zrtdde26y1HtrZj3IdKwVBEHoAu3fDzp3
      "text/plain": [
       "<Figure size 720x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# extra code – this cell generates and saves Figure 6–6\n",
    "\n",
    "tree_reg1 = DecisionTreeRegressor(random_state=42)\n",
    "tree_reg2 = DecisionTreeRegressor(random_state=42, min_samples_leaf=10)\n",
    "tree_reg1.fit(X_quad, y_quad)\n",
    "tree_reg2.fit(X_quad, y_quad)\n",
    "\n",
    "x1 = np.linspace(-0.5, 0.5, 500).reshape(-1, 1)\n",
    "y_pred1 = tree_reg1.predict(x1)\n",
    "y_pred2 = tree_reg2.predict(x1)\n",
    "\n",
    "fig, axes = plt.subplots(ncols=2, figsize=(10, 4), sharey=True)\n",
    "\n",
    "plt.sca(axes[0])\n",
    "plt.plot(X_quad, y_quad, \"b.\")\n",
    "plt.plot(x1, y_pred1, \"r.-\", linewidth=2, label=r\"$\\hat{y}$\")\n",
    "plt.axis([-0.5, 0.5, -0.05, 0.25])\n",
    "plt.xlabel(\"$x_1$\")\n",
    "plt.ylabel(\"$y$\", rotation=0)\n",
    "plt.legend(loc=\"upper center\")\n",
    "plt.title(\"No restrictions\")\n",
    "\n",
    "plt.sca(axes[1])\n",
    "plt.plot(X_quad, y_quad, \"b.\")\n",
    "plt.plot(x1, y_pred2, \"r.-\", linewidth=2, label=r\"$\\hat{y}$\")\n",
    "plt.axis([-0.5, 0.5, -0.05, 0.25])\n",
    "plt.xlabel(\"$x_1$\")\n",
    "plt.title(f\"min_samples_leaf={tree_reg2.min_samples_leaf}\")\n",
    "\n",
    "save_fig(\"tree_regression_regularization_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sensitivity to axis orientation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rotating the dataset also leads to completely different decision boundaries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsAAAAEQCAYAAAC++cJdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA1K0lEQVR4nO3de5Ac9XUv8O/ZXUlIJkQPBBIr2WAXtsEPgbI29jWOQaAKr0REcmQcHEglXEIRcZ26ETayDM51rKwdu1x2ZMcUwQ/hQLgE6SKVInBANthgCWvBAoRkDDgOWrEg0IM3Wmnn3D+mB2Znu2e6Z7p/r/5+qrZ2d7Z35szs7K9Pnz6/X4uqgoiIiIioLLpsB0BEREREZBITYCIiIiIqFSbARERERFQqTICJiIiIqFSYABMRERFRqTABJiIiIqJS6bEdQJ6OnDZVj50123YYFLhKVxdeefVVvPDCC3j99dcxadIkzJw503ZYVAIPPvjg86o6PY/74nhJRKF74OGHE8fMoBLgY2fNxi/+8w7bYVDgXj1sEh7YuhX/sX49tu/Ygblz5+Kaa66xHRaVwLhx4/47r/vieElEoeuecUzimMkWCCIiIiIqFSbARERERFQqTICJiIiIqFSYABMRERFRqTABJiIiIqJSYQJMRERERKXCBJiIiIiISoUJMBERERGVChNgIiIiIioVJsBEREREVCpWEmAROUtEHhORJ0TkqoRtThORrSLyqIjcYzpGIiIiIgpTj+kHFJFuAN8GMB/AIIAtIrJOVbfXbTMZwD8DOEtVnxKRo0zHSURERERhslEB/iCAJ1T1N6o6DOBmAAsatvlTAGtU9SkAUNXdhmMkIiIiokDZSIB7Aeys+34wuq3eOwFMEZG7ReQBEbnIWHRERERk1+o1kL4PQGb2Qvo+AKxeYzsiCozxFggAEnObNnzfA+D3AJwBYCKATSKyWVV/PebORC4FcCkAvHVWYx5NREQ1HC/JC6vXQJZeCXntter3g7uApVdWE4VFC21GRgGxUQEeBDC77vtZAJ6O2eYOVX1FVZ8H8FMAc+LuTFWvU9U+Ve2bPnVaIQETEYWA4yX5QPr730x+a7e99hqkv99SRBQiGwnwFgDHi8hxIjIewAUA1jVssxbAR0WkR0QmATgFwA7DcRIREZFpuxprYi1uJ2qD8RYIVT0kIksA/AhAN4DvqeqjInJZ9PNrVXWHiNwB4GEAFQDXq+o207ESERGRYb3HVNse4m4nyomNHmCo6gYAGxpuu7bh+68C+KrJuIiIiMguXbYMqO8BBqATJ1ZvJ8oJrwRHRERE7qy8sGgh9Gtfhc7qhYpUP3/tq5wAR7myUgEmIiIih7i28sKihVAmvFQgVoCJiIhKjisvUNkwASYiIiq7kFdecKW1g5zCBJiIiKjsklZY8H3lhVprx+AuiGr189IrmQQTE2AiIqKy02XLoBMnjr4tgJUX2NpBSZgAExERlV2oKy+E3NpBHWECTERERNUkeGALdGgXdGCLH8lvq/7eUFs7qGNMgImIiMg/Kfp7Q23toM4xAaawBTz7d2gImDevC888Y+f3iYhsStXfG2prB3WMCTCFK/DZvytWdOG++wQrVrT3b9zp7xMRWZW2v9fH1g4qHPd81BmHK6w+zf7NWo0dGgJWrRJUKoJVqyRzFbfT3yciso79vdQBJsDUPtcrrB7N/s1ajV2xoguVSvXrkRFkruJ2+vtFYmsGEaXB/l7qhDt7PfKO8xVWT6oDWauxte2HhwUAMDycrYrb6e8Xja0ZRB1y+MxcrtjfSx3gHoba53iF1ZfqQNZqbP32NVmquJ3+fpHYmkHUIdfPzOWN/b3UJvt7PCpeUdUA1yusHlQH2qnGbt785vY1w8OCTZsk4Tfy/f0iudyaQeQD58/Mdaos1W0qHPcuoSuwGuBFhdXx6kA71diBgREcPHhozMfAwEiqx+z094viemsGUWHyTOocPzPXkbJVt6lQTIADV2g1wIMKq+tcrsaa5nJrBlFh8k7qXD8z14Hgq9tkFPcsoSu6GuB4hdV1rlZjG5lYmYEHA1RGeSd1XpyZa1fI1W0yjglw6IqqBrAPq1RMrMzgy8EAUa7yTupsnpkrer8QcHWbzGMCHLhCqgHswyoVrsxAVKCk5K2rq/1E0saZuaL3C6vXAK+8Am24OZjqNhnHBDh0BVQD2IdVLrZXZuCFMShYSUkdABkZKbbAkHO1trD9wuo1kBPeA/nrJZB9+1FriFIAOmUK551Q25gAl0He1QD2YZWGCysz8MIYFKRaxbQxqesSNHa9515gKKJaW8R+4Y3XaN/Y1wQA3jLJ/eSX7YLOsrJHEZGzROQxEXlCRK5qst0HRGRERD5uMj5qgX1YpWF7ZQa2X1CoYiumAFBprAdHciwwFFKtLWC/EBfnKK4XXdgu6DTjCbCIdAP4NoCzAZwI4JMicmLCdl8B8COzEVIrQc8yplFsr8xgu/2CqDBZk7c8CwxJjz24q+3krJD9QqvXyPGiC9sF3WZjb/JBAE+o6m9UdRjAzQAWxGx3BYDVAHabDI5S4Pq/pWFzZQYX2i+ICpOUvE2ZUnyBIeGxBWi/QlnEfqFJgutF0YXtgk6zkQD3AthZ9/1gdNsbRKQXwB8DuNZgXJRFXF8xe50oR7bbL4iKlFgx/dLfF15giHvsmo4qlDnPN4l9jQDolMl+FF3YLug0G3uSuHOnjU1P3wDwWVVtWWYSkUtFZEBEBp7buyeP+Kgd7HWinNluvwgRx0sD0hYCmlVMi17GbNFC6OLFY3a8b8haoSyq+BH3Gn37W9Ad291PfsF2Qdf1WHjMQQCz676fBaDxv60PwM0iAgBHAjhHRA6p6m2Nd6aq1wG4DgD65sxJ/H+mYiX1OqG/H+rBQEXu4QUw8sfxsmC1QkBtLBzcBSy9sppoxo2DixbaGR9Xr4HccktsNQpAtgpl1uecla3XKA+LFlZfh/7+6kFF7zHV5NfX5xMYGxXgLQCOF5HjRGQ8gAsArKvfQFWPU9VjVfVYALcCuDwu+SWHsNeJiErOl0lPzVZXyFqh9OU5W2PjoiSUivEEWFUPAViC6uoOOwDcoqqPishlInKZ6XgoJ+x1IqKya6cQYGPuREI8CmTvre20+MG5I2SJldkkqrpBVd+pqu9Q1RXRbdeq6phJb6r656p6q/koKYvYXqdx44BXXuHARkTlkLUQYGvuRFI8s3qzVyg7KX5w7ghZxOnUlI/GyQqTJgEHD1avcsSBjYhKIOukp0ztAzlWSvOcnNXJfbF9gmxiAkz5qfU6fWsl8NprxV/Ok94wNATMm9fFNXKJbMq6Fm7a9oG8KqW1JHrJFcBhh1WXE+t0qbVO1v/l3BGyyMYqEBQ46e+HaPGX86Q3rVjRhfvuE6xY0YWVKyutf4GIipFl1YLeY6qrJsTdXieXVXYaV2vYt69aqf3Wys4nZrW7UkPK598JHTcOlZ5xud0feSQpD4kwAab8NUtyOSkud7UrplUqglWrgOXLgRkzbEdFRK3osmVAfVKKhPaBHCqlLi5Vmfr5t2l4/GF4fHAQQ0NDudwf+SVaSjcRE2DKX8JRvYpwAfAC1F8xrXalNB+rwENDwIUXduGmmypM4Kkc0q4Tm0el1MV2g4LWydXubryALmzetAnr16/Hrl0xrx0Fr7u7u+nPmQC7bPWaas+sZwtoxx7Vi0AvusiL+H1Sq/7Wrpg2POxvFZhtHFRKKdoHcqmUGmg3aEsBF7qoHDYJTz3xBAYGBvDII4+gp7sbPWyDKJ2ecc1TXCbArir66jpF4tVvjKmv/tb4WAVmGwdRgloh5LXXoN3d1X/wWb2Zx9Si2w1colCMjIygMjICVcUnLrgAl1xyie2wyIJx45IPfLgKhKO8Xx6GV78xYvPmN6u/NcPDgk2bmvc+uSaujYOo9OpXfwAgIyNALWnNOqZ2slqD51r1glI5sQLsKhf7tcg5AwMjtkPoWEhtHER5yn3iWgHtBkS+YpnFVby0MJVEszYOolJztRDCyxdTALiHcVSeV+ppCwc4MqTTNg5eBISC5WIhpOjLF3PfQ4YwAXaVzX4tXp+dDBo
      "text/plain": [
       "<Figure size 720x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# extra code – this cell generates and saves Figure 6–7\n",
    "\n",
    "np.random.seed(6)\n",
    "X_square = np.random.rand(100, 2) - 0.5\n",
    "y_square = (X_square[:, 0] > 0).astype(np.int64)\n",
    "\n",
    "angle = np.pi / 4  # 45 degrees\n",
    "rotation_matrix = np.array([[np.cos(angle), -np.sin(angle)],\n",
    "                            [np.sin(angle), np.cos(angle)]])\n",
    "X_rotated_square = X_square.dot(rotation_matrix)\n",
    "\n",
    "tree_clf_square = DecisionTreeClassifier(random_state=42)\n",
    "tree_clf_square.fit(X_square, y_square)\n",
    "tree_clf_rotated_square = DecisionTreeClassifier(random_state=42)\n",
    "tree_clf_rotated_square.fit(X_rotated_square, y_square)\n",
    "\n",
    "fig, axes = plt.subplots(ncols=2, figsize=(10, 4), sharey=True)\n",
    "plt.sca(axes[0])\n",
    "plot_decision_boundary(tree_clf_square, X_square, y_square,\n",
    "                       axes=[-0.7, 0.7, -0.7, 0.7], cmap=\"Pastel1\")\n",
    "plt.sca(axes[1])\n",
    "plot_decision_boundary(tree_clf_rotated_square, X_rotated_square, y_square,\n",
    "                       axes=[-0.7, 0.7, -0.7, 0.7], cmap=\"Pastel1\")\n",
    "plt.ylabel(\"\")\n",
    "\n",
    "save_fig(\"sensitivity_to_rotation_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(max_depth=2, random_state=42)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "pca_pipeline = make_pipeline(StandardScaler(), PCA())\n",
    "X_iris_rotated = pca_pipeline.fit_transform(X_iris)\n",
    "tree_clf_pca = DecisionTreeClassifier(max_depth=2, random_state=42)\n",
    "tree_clf_pca.fit(X_iris_rotated, y_iris)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAEQCAYAAACutU7EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABB7UlEQVR4nO3dd3xUVf7/8ddhgASpIiWhhCYgiCAQsIKgYEFUxEZdG6IogrvrT4hYVlGRFWWjsihiwRVpEpRFRAULqDRBQBEFpAZJBL8iARcCyfn9kWLKTOrM3Dsz7+fjMQ8y997c+UwC5JPP+ZxzjLUWERERkVBSwekAREREREpLCYyIiIiEHCUwIiIiEnKUwIiIiEjIUQIjIiIiIUcJjIiIiIScik4H4C916tS2TZs2djoMCRHr1m0C4KyzOjsciYifVE53OgIRv/t23bcHrbV1vZ0z4bIOTHx8B7tmzYdOhyEhwuOJBWDnzvD4+y/iidvrdAgifhfniVtnrY33dk5DSCIiYWLmtJnMnDbT6TBEgiJshpBERCJdwogEAAYPH+xwJCKBpwqMiIiIhBwlMCIiIhJyNIQk4kfp6Yc5ceIXMjNPOB2KhBCPpypVqjTCGP1OKVJSSmBE/CQ9/TAnT6bSqFFDoqKqYIxxOiQJAZmZmaSk7OP48YNER9dzOhyRkKF0X8RPTpz4hYYNGxIdfYqSFymxChUqULdufTIyfnc6FJGQogRGxE8yM08QFVXF6TAkBFWsWInMzJNOhyESUjSEJOJHqrxIWfjr782ejD1+uY9IKFAFRkREREKOEhgRKZHevXtw330jnQ5DRARwKIExxlxujPnRGLPdGDPWxzU9jDEbjDGbjTGfBztGkUgxbNgtXHtt32KvmzMnifHjJwQhIu+iow1JSe849vqhoE+XPvTp0sfpMESCIug9MMYYDzAF6A0kA2uNMQuttd/nuaYW8G/gcmvtHmOM5haKOCQ9PZ3KlStTu3Ztp0ORYny3/junQxAJGicqMF2B7dbaHdbadGA2cE2BawYBSdbaPQDW2l+CHKOIIw4cmMm6dU1ZubIC69Y15cCB4G/Ml1ORmTRpIi1aNKJFi0ZA4SGkd99NIj6+PbVqVSE2tja9el1Eamqqz/u+8srLtGvXipo1o2nUqC59+17GyZN/zryZMeN1zj67LTVrRtOuXSuef34ymZmZALRq1RSAQYNuIDra5D7PuW/btqdTvXpl2rY9nVdffaXEr/v112u58spLadiwDnXr1qBnzwtZtWplub5+IhIcTsxCagjk3fc9GTinwDWtgErGmM+A6kCitfbN4IQn4owDB2ayY8dwMjP/ACA9fTc7dgwHoG7d4G7Ot2LF59SoUZOFC5dgrS10PiUlhaFDBzB+/AT69buOo0ePsHr1Kp/3W7fua+677x6mT5/B+edfyO+/H+Kzzz7JPf/qq68wfvwjPPfcC3Ts2Jnvv/+Ou+++g0qVKjFixEi+/HItjRvX49//foU+ffri8XgAeO+9Bfz1ryN55pnJ9Op1KR9//CGjR99NTEwMV155VbGvm5aWxqBBQ3n22USMMUyd+iL9+vXhu++2UadOHT9+RUXE35xIYLzNFyz4P2RFoDNwCVAFWGmMWWWt3ZrvRsYMB4YDxMU1DECoIsGzZ8+43OQlR2bmH+zZMy7oCUx0dDTTpr1GVFSU1/P79//MiRMnuPba62nSpAkAZ57Zzuf99u7dQ9WqVenb92qqV68ONKF9+w655ydMGM+TT/6T/v2vB6BZs2bcf/9YXn7534wYMZK6desCUKtWLWJiYnI/b/LkSQwaNJQRI7IqQy1btmL9+nVMmjSRK6+8qtjX7dnz4nxxTp78Au++O5+PPlrCoEFDSv4FE5Ggc2IIKRlonOd5I+BnL9cssdYetdYeBJYDHQpcg7V2mrU23lobX7fuaQELWCQY0tO9r+Hh63ggtW3bzmfyAtC+fQcuvrgXnTu3Y8CA65g2bSoHDhzwef0ll/QmLq4JZ5zRjJtvHsx//jODtLQ0AA4cOEBy8l5GjryT006rlvt46KGx7NjxU5Fx/vjjFs4774J8x84//0J++OH7Yl8X4JdffuGee+6kXbtW1KtXkzp1qvPLL7+wd6/WU5HySU1L5YYZN/DLEXVABIoTCcxaoKUxppkxpjIwAFhY4Jr3gG7GmIrGmFPIGmLaEuQ4RYKqcuW4Uh0PpKpVqxZ53uPx8P77H7Fo0Ue0a9eeN954lXbtWrJp00av11evXp1Vq9bz1ltzadw4jmeemUCHDmfw888/5/a5vPDCS6xZsyH3sX79d3zzzeZiY/W2CFzOsaJeF2DYsJtZt24tzzwzmc8++4o1azbQsGEj0tPTi31dkaIkrkhkzd41JC5PdDqUsBX0BMZaexIYCXxIVlIy11q72RhzlzHmruxrtgBLgE3AGmC6tVbt9RLW4uKepEKFU/Idq1DhFOLinnQooqIZYzj33PN46KFH+fLLtcTGNmDevDk+r69YsSI9e17ME09M4OuvN3H06FEWL15E/fr1adiwITt2/ESLFqcXeuSoVKkSGRkZ+e7ZunUbvvrqi3zHvvrqC844o22xr5tz7YgR93LFFVfStu2ZVKtWnZSU/f748jhi4LCBDBw20OkwIl5qWirzNs7DWsu8jfNUhQkQR7YSsNYuBhYXOPZSgefPAM8EMy4RJ+X0uezZM4709D1UrhxHXNyTQe9/KYnVq1fxySdL6d37MurVq8/Gjd+QnLyXNm3aer1+8eJF7NjxExde2J1TT63N559/SlpaGmec0QaAceP+wd/+di+1atXi8sv7cOLECb75Zj0//7yPBx5IAKBJk6Z8+ukyunW7iKioKE499VT+9rf/x6BBN9CpU2d69bqUjz5awuzZM5kzJ6lEr9uyZStmzXqLrl3P4ejRozz44ANUrlw5CF/BwJj48kSnQxCyqi85ze+ZNpPE5Yk82cedv4iEMu2FJOIidesOdmXCUlDNmjVZufJLpk59gUOHDtGoUWMSEh722fhas2YtFi58l6eeepw//viD5s1bMHXqdC68sBsAt902jKpVqzJ58jM8/HACVapUoU2bM3ObcwEmTnyWBx74G6ef3pgGDRqydesurr66H5Mnv8DkyZO4//77iItrQmLiv7nyyqtK9Lovv/wa99wznPPO60xsbAMeeugfHDzou5dHpDg51Zf0jKxhyPSMdOZtnMfo7qOpV01LmvmT8TZFMhTFx3ewa9Z86HQYEiI8nlgAdu7039//tLQttGrVxm/3k8iydesWqlcv+98fT9xeNq3bBED7zu39FZaU0oOLH2Tuhrm5CQxAZU9lbjr7JlVhyiDOE7fOWhvv7ZwqMCIiYaJv16wtIbQrtXPWJ6/Pl7xAVhVmXfI6hyIKX0pgRERE/GTJ8CVOhxAxtBu1iIiIhBwlMCIiIhJylMCIiIhIyFECIyIiIiFHCYyIiIiEHM1CEhEJE4vWLHI6BJGgUQIjIhImtICdRBINIYlIifTu3YP77htZ/IUu06pVUyZPnuS3+4Xq10Ek3KgCIxLhhg27hV9/PciCBUUPP8yZk0SlSpWCFJX/fPnlWqpWrep0GEEx5s4xgDZ1DKTUtFRGJo1kynVTtLeRw1SBEZEipadnLYteu3Ztqlev7nA0+Z04caLYa+rWrcspp5wShGhKJjMzk4yMjIDce9b0WcyaPisg95YsiSsSWbN3DYnLE50OJeIpgRFxiSZNIDq68KNJk+DGMWzYLVx7bV8mTZpIixaNaNGiEVB46OTdd5OIj29PrVpViI2tTa9eF5Gamur1nkOHDmTAgOvyHcvMzKRFi8Y8//xkAKy1PPvsP2nTpgW1alWhc+ezePvtt3Kv37VrF9HRhjlzZnHZZRdTq1YVpk9/md9//51bbx1K48b1qFkzmjPOaM4LL/wr9/MKDiEdPnyYe+8dQdOmsdSsGU2HDm2YN29OvvfVufNZ1KgRRYsWjXn66ScpatPb3377jdtvv5mYmFOpVasKV1zRi++/35x7/s033+C006qxZMliOnVqR/Xqlfnhhy1FfQvEpXJ2mrbWMm/jPH458ovTIUU
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# extra code – this cell generates and saves Figure 6–8\n",
    "\n",
    "plt.figure(figsize=(8, 4))\n",
    "\n",
    "axes = [-2.2, 2.4, -0.6, 0.7]\n",
    "z0s, z1s = np.meshgrid(np.linspace(axes[0], axes[1], 100),\n",
    "                       np.linspace(axes[2], axes[3], 100))\n",
    "X_iris_pca_all = np.c_[z0s.ravel(), z1s.ravel()]\n",
    "y_pred = tree_clf_pca.predict(X_iris_pca_all).reshape(z0s.shape)\n",
    "\n",
    "plt.contourf(z0s, z1s, y_pred, alpha=0.3, cmap=custom_cmap)\n",
    "for idx, (name, style) in enumerate(zip(iris.target_names, (\"yo\", \"bs\", \"g^\"))):\n",
    "    plt.plot(X_iris_rotated[:, 0][y_iris == idx],\n",
    "             X_iris_rotated[:, 1][y_iris == idx],\n",
    "             style, label=f\"Iris {name}\")\n",
    "\n",
    "plt.xlabel(\"$z_1$\")\n",
    "plt.ylabel(\"$z_2$\", rotation=0)\n",
    "th1, th2 = tree_clf_pca.tree_.threshold[[0, 2]]\n",
    "plt.plot([th1, th1], axes[2:], \"k-\", linewidth=2)\n",
    "plt.plot([th2, th2], axes[2:], \"k--\", linewidth=2)\n",
    "plt.text(th1 - 0.01, axes[2] + 0.05, \"Depth=0\",\n",
    "         horizontalalignment=\"right\", fontsize=15)\n",
    "plt.text(th2 - 0.01, axes[2] + 0.05, \"Depth=1\",\n",
    "         horizontalalignment=\"right\", fontsize=13)\n",
    "plt.axis(axes)\n",
    "plt.legend(loc=(0.32, 0.67))\n",
    "save_fig(\"pca_preprocessing_plot\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Decision Trees Have High Variance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've seen that small changes in the dataset (such as a rotation) may produce a very different Decision Tree.\n",
    "Now let's show that training the same model on the same data may produce a very different model every time, since the CART training algorithm used by Scikit-Learn is stochastic. To show this, we will set `random_state` to a different value than earlier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(max_depth=2, random_state=40)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_clf_tweaked = DecisionTreeClassifier(max_depth=2, random_state=40)\n",
    "tree_clf_tweaked.fit(X_iris, y_iris)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAEQCAYAAACutU7EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABDuUlEQVR4nO3dd3xUVfrH8c9DgFBDDQTpIEWQXlQEVikq2BBZEMRdsaAiou66KoplbbiKIlhQbOhvFSsgCnbsSAcLIEiTHoqL9FByfn/MJCZhMiVMyUy+79frvpi55dznXtA8Ofc855pzDhEREZF4UizWAYiIiIiESgmMiIiIxB0lMCIiIhJ3lMCIiIhI3FECIyIiInFHCYyIiIjEnaglMGZWyszmmdkPZrbUzP7tYx8zs/FmtsrMfjSzttGKT0REROJH8SieKwPo5pzba2YlgG/N7EPn3Jwc+/QCGnmXU4AJ3j9FREREskWtB8Z57PV+LeFd8s6idyHwqnffOUBFM6sRrRhFREQkPkR1DIyZJZnZEmAb8Klzbm6eXWoCG3J83+hdJyIiIpItmo+QcM4dBVqbWUVgqpmd7Jz7Occu5uuwvCvMbCgwFKBM2TLtGjZtGIlwRUREJIZ+WvjTDudcqq9tUU1gsjjndpnZl8A5QM4EZiNQO8f3WsBmH8dPBCYCtGzf0s2YNyNywYqIiEhM1Emq81t+26JZhZTq7XnBzEoDPYBf8uw2HfibtxrpVOAP59yWaMUoIiIi8SGaPTA1gFfMLAlP4vSWc+4DM7sWwDn3LDAT6A2sAvYDQ6IYn4iIiMSJqCUwzrkfgTY+1j+b47MDro9WTCIiIhKfNBOviIiIxB0lMCIiIhJ3YlKFFE0ZuzPI2JFB5uFMHwXZIj4YFCtRjOSqySSnJMc6GhER8SGhE5iM3Rkc2naIWjVrUap0Kcx8TTMjkptzjoMHDrJp0yYAJTEiIoVQQj9CytiRQc2aNSldprSSFwmamVG6TGlq1qxJxo6MWIcjIiI+JHQCk3k4k1KlS8U6DIlTpUqX8jx6FBGRQiehExgc6nmRAjMzjZsSESmkEjuBERERkYSkBCaO9TyzJzcOvzHWYYiIiERdQlchxaurhlzFjh07mPb+NL/7vfnum5QoUSI6QfmQXCyZyW9Npm+/vjGLQUREiiYlMHHo0KFDlCxZksqVK8c6FBERkZjQI6QgbE+fzMI5jZj9VSkWzmnE9vTJUT3/VUOuos/5fRjznzE0qN2ABrUbAMc+Qpo2ZRrtWrWjQpkKpFVJo8cZPUhPT8+33eefe57mTZqTUjqFmtVqcu4553LkyJHs7a+8/AqtmrcipXQKzZs0Z9zYcWRmeqpyGtdvDMDA/gNJLpac/T2r3ZManUS55HKc1OgkXnz+xaDPu2D+Anqf3ZsTUk+gaoWqnNnlTOZ8P+c476CIiCQa9cAEsD19MqtXDiMzcz8AGRnrWb1yGACp1QdGLY5vvvqGChUq8P6H7+N552VuW7duZfDAwdz/0P1cdPFF7Nu7j7lz5ubb3sIFC7lx+I28OOlFOnXuxB+7/uCLWV9kb3/x+Re57577GDt+LG3atWHpz0sZNnQYJUqUYNjwYXw37ztqVa/FhIkT6H1eb5KSkgB4b+p73HTDTTz6+KP0OKsHn378KSOuH0H1tOqcd/55Ac+7Z88eLh18KY898RhmxoSnJnDhuReydOVSqlatGsY7KiIi8UwJTADr196dnbxkyczcz/q1d0c1gSlVqhQTX5xIcrLvWWG3bN7C4cOH6duvL3Xr1gWg+cnN821vw/oNlC1blvMuOI/y5ctDXWjZqmX29tEPjOah/zyUPb6lfv36rLltDc9NeI5hw4eRmpoKQMWKFUlLS8s+buxjY7l08KUMG+5J8ho3bsziRYt57JHHOO/88wKe98xuZ+aK84knn2DalGl88tEnDBo8KJRbJiIiCUyPkALIyNgQ0vpIaX5y83yTF/AkAd16dKNti7YM6DeA5yY8x/bt2/Pdv3vP7tSpW4cmDZrw98F/5/9e+T/27NkDwPbt29mwYQPXX3s9lctXzl5GjRzFmtVr/Mb5y/JfOO3003Kt63R6J5YvWx7wvADbtm1j2DXDaN6kOakVU6mSUoVt27axYX1077eIiBRuSmACSE6uHdL6SClTtozf7UlJScz8eCYzPp5BixYtmPTSJJo3bs6PP/zoc//y5cszd+FcXnvzNWrXrs0jDz9Cy5Nasnnz5uxxLk9NeIp5i+dlL4t+WsTinxcHjNXX5IFZ6/ydF+DKy69k4YKFPPr4o3z13VfMWzyPWrVqcejQoYDnFRGRokMJTAB16t9HsWK5k4dixcpQp/59MYoof2bGqaedyqh7RjF73mxqnFCDt998O9/9ixcvzpndzuSB0Q+w8IeF7Nu3j5kfzKR69erUrFmTNavXcOKJJx6zZClRogRHjx7N1WbTk5oy+9vZudbN/m42JzU7KeB5AWZ/O5thw4fR+9zeNGvejPLly7Nly5Zw3B4REUkgGgMTQNY4l/Vr7yYjYwPJybWpU/++qI5/CcbcOXOZ9dksep7dk2rVq7Fk8RI2btiYK3HIacYHM1izeg1dunahUuVKfPXFV+zZs4emJzUFYNQ9o7h5xM1UqFiBc3qfw5HDR1i8aDGbN23m1pG3AlC3Xl1mfT6LLn/pQnJyMpUqVeLmW25mUP9BtGnXhh5n9eCTjz5h8muTefPdN4M6b6PGjXj9tdfpeEpH9u3bxx233UHJkiWjcAdFRCSeKIEJQmr1gYUuYckrpUIKs2fP5pmnnmHXrl3Uql2LkaNG5jvwtWLFirz/3vs8dP9D7N+/nwYNG/Ds88/SuUtnAK646grKli3L42Me56477qJ06dI0a96M666/LruN/4z5D7f+81Ya1mlIzZo1Wbl2JRf2uZCx48cy9rGx3HLzLdSpW4fxT4/nvPPPC+q8E1+cyLBrhnFq+1OpcUIN7rrnLr9jeUREpGgyXyW58aRl+5ZuxrwZPrft+mUXTU5qEuWIJJGsWL6Cik0rxjoMEZEiqU5SnYXOufa+tmkMjIiIiMQdJTAiIiISd5TAiIiISNxRAiMiIiJxRwmMiIiIxB0lMCIiIhJ3lMCIiIhI3FECIyIiInEnagmMmdU2sy/MbLmZLTWzG33sc4aZ/WFmS7zL3dGKT0REio70Pen89ZW/sm3vtqi0GYnzFXXR7IE5AvzTOXcScCpwvZk187HfN8651t6l8L0xsRDpeWZPbhx+TB5Y6DWu35jHxzwetvbi9T6ISOyM+2Yc8zbMY9zX46LSZiTOV9RFLYFxzm1xzi3yft4DLAdqRuv88eSqIVfR5/w+Afd78903eWD0A5EPKMy+m/cd1w67NtZhiEgRlb4nnbd/eBvnHG//8HZYekX8tRmJ80mMxsCYWT2gDTDXx+bTzOwHM/vQzJrnc/xQM1tgZgt+3/57JEMtlA4dOgRA5cqVKV++fIyjye3w4cMB90lNTaVMmTJRiCY4mZmZHD16NNZhiEiUjPtmHFnvAcx0mWHpFfHXZiTOJzFIYMysHPAucJNzbneezYuAus65VsCTwDRfbTjnJjrn2jvn2ldOrRzReOvUKEFysZLHLHVqlIjoeXPK6pEZ858xNKjdgAa1GwDHPjqZNmUa7Vq1o0KZCqRVSaPHGT1IT0/32eZlgy5jQL8BudZlZmbSsE5Dxo31/MflnGPMI2NoemJTKpSpQNuWbXn9v69n779u3TqSiyXz5uQ3Obv72VQoU4Hnn3ueP/74gyF/G0Kt6rVIKZ1Ck4ZNGP/E+Ozj8j5C2r17N8OvG07dE+qSUjqFls1a8vabb+e6rrYt21K+VHka1mnIww8+jL+XkP7vf//jysuvpHrl6lQoU4Fzep7DsqXLsre/OulVKpevzIczP6RNizaUSy7HL8t/8ft3ICKJIas35NBRzy+Ch44eOu5eEX9tRuJ84lE8miczsxJ4kpfXnHNT8m7PmdA452aa2TNmVtU5tyOaceaUnm4hrY+Ub776hgoVKvD+h+/7/OG9detWBg8czP0P3c9FF1/Evr37mDvHVweXx8BLB3JJv0v
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# extra code – this cell generates and saves Figure 6–9\n",
    "\n",
    "plt.figure(figsize=(8, 4))\n",
    "y_pred = tree_clf_tweaked.predict(X_iris_all).reshape(lengths.shape)\n",
    "plt.contourf(lengths, widths, y_pred, alpha=0.3, cmap=custom_cmap)\n",
    "\n",
    "for idx, (name, style) in enumerate(zip(iris.target_names, (\"yo\", \"bs\", \"g^\"))):\n",
    "    plt.plot(X_iris[:, 0][y_iris == idx], X_iris[:, 1][y_iris == idx],\n",
    "             style, label=f\"Iris {name}\")\n",
    "\n",
    "th0, th1 = tree_clf_tweaked.tree_.threshold[[0, 2]]\n",
    "plt.plot([0, 7.2], [th0, th0], \"k-\", linewidth=2)\n",
    "plt.plot([0, 7.2], [th1, th1], \"k--\", linewidth=2)\n",
    "plt.text(1.8, th0 + 0.05, \"Depth=0\", verticalalignment=\"bottom\", fontsize=15)\n",
    "plt.text(2.3, th1 + 0.05, \"Depth=1\", verticalalignment=\"bottom\", fontsize=13)\n",
    "plt.xlabel(\"Petal length (cm)\")\n",
    "plt.ylabel(\"Petal width (cm)\")\n",
    "plt.axis([0, 7.2, 0, 3])\n",
    "plt.legend()\n",
    "save_fig(\"decision_tree_high_variance_plot\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extra Material – Accessing the tree structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A trained `DecisionTreeClassifier` has a `tree_` attribute that stores the tree's structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.tree._tree.Tree at 0x7fbfa8b563b0>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree = tree_clf.tree_\n",
    "tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get the total number of nodes in the tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.node_count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And other self-explanatory attributes are available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.max_depth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.max_n_classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.n_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.n_outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.n_leaves"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the information about the nodes is stored in NumPy arrays. For example, the impurity of each node:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.66666667, 0.        , 0.5       , 0.16803841, 0.04253308])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.impurity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The root node is at index 0. The left and right children nodes of node _i_ are `tree.children_left[i]` and `tree.children_right[i]`. For example, the children of the root node are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 2)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.children_left[0], tree.children_right[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the left and right nodes are equal, it means this is a leaf node (and the children node ids are arbitrary):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-1, -1)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.children_left[3], tree.children_right[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So you can get the leaf node ids like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 3, 4])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "is_leaf = (tree.children_left == tree.children_right)\n",
    "np.arange(tree.node_count)[is_leaf]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Non-leaf nodes are called _split nodes_. The feature they split is available via the `feature` array. Values for leaf nodes should be ignored:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0, -2,  1, -2, -2], dtype=int64)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the corresponding thresholds are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2.44999999, -2.        ,  1.75      , -2.        , -2.        ])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the number of instances per class that reached each node is available too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[50., 50., 50.]],\n",
       "\n",
       "       [[50.,  0.,  0.]],\n",
       "\n",
       "       [[ 0., 50., 50.]],\n",
       "\n",
       "       [[ 0., 49.,  5.]],\n",
       "\n",
       "       [[ 0.,  1., 45.]]])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([150,  50, 100,  54,  46], dtype=int64)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.n_node_samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.all(tree.value.sum(axis=(1, 2)) == tree.n_node_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how you can compute the depth of each node:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 1., 1., 2., 2.])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def compute_depth(tree_clf):\n",
    "    tree = tree_clf.tree_\n",
    "    depth = np.zeros(tree.node_count)\n",
    "    stack = [(0, 0)]\n",
    "    while stack:\n",
    "        node, node_depth = stack.pop()\n",
    "        depth[node] = node_depth\n",
    "        if tree.children_left[node] != tree.children_right[node]:\n",
    "            stack.append((tree.children_left[node], node_depth + 1))\n",
    "            stack.append((tree.children_right[node], node_depth + 1))\n",
    "    return depth\n",
    "\n",
    "depth = compute_depth(tree_clf)\n",
    "depth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how to get the thresholds of all split nodes at depth 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1], dtype=int64)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_clf.tree_.feature[(depth == 1) & (~is_leaf)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.75])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_clf.tree_.threshold[(depth == 1) & (~is_leaf)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. to 6."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. The depth of a well-balanced binary tree containing _m_ leaves is equal to log₂(_m_), rounded up. log₂ is the binary log; log₂(_m_) = log(_m_) / log(2). A binary Decision Tree (one that makes only binary decisions, as is the case with all trees in Scikit-Learn) will end up more or less well balanced at the end of training, with one leaf per training instance if it is trained without restrictions. Thus, if the training set contains one million instances, the Decision Tree will have a depth of log₂(10<sup>6</sup>) ≈ 20 (actually a bit more since the tree will generally not be perfectly well balanced).\n",
    "2. A node's Gini impurity is generally lower than its parent's. This is due to the CART training algorithm's cost function, which splits each node in a way that minimizes the weighted sum of its children's Gini impurities. However, it is possible for a node to have a higher Gini impurity than its parent, as long as this increase is more than compensated for by a decrease in the other child's impurity. For example, consider a node containing four instances of class A and one of class B. Its Gini impurity is 1 – (1/5)² – (4/5)² = 0.32. Now suppose the dataset is one-dimensional and the instances are lined up in the following order: A, B, A, A, A. You can verify that the algorithm will split this node after the second instance, producing one child node with instances A, B, and the other child node with instances A, A, A. The first child node's Gini impurity is 1 – (1/2)² – (1/2)² = 0.5, which is higher than its parent's. This is compensated for by the fact that the other node is pure, so its overall weighted Gini impurity is 2/5 × 0.5 + 3/5 × 0 = 0.2, which is lower than the parent's Gini impurity.\n",
    "3. If a Decision Tree is overfitting the training set, it may be a good idea to decrease `max_depth`, since this will constrain the model, regularizing it.\n",
    "4. Decision Trees don't care whether or not the training data is scaled or centered; that's one of the nice things about them. So if a Decision Tree underfits the training set, scaling the input features will just be a waste of time.\n",
    "5. The computational complexity of training a Decision Tree is _O_(_n_ × _m_ log₂(_m_)). So if you multiply the training set size by 10, the training time will be multiplied by _K_ = (_n_ × 10 _m_ × log₂(10 _m_)) / (_n_ × _m_ × log₂(_m_)) = 10 × log₂(10 _m_) / log₂(_m_). If _m_ = 10<sup>6</sup>, then _K_ ≈ 11.7, so you can expect the training time to be roughly 11.7 hours.\n",
    "6. If the number of features doubles, then the training time will also roughly double."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: train and fine-tune a Decision Tree for the moons dataset._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. Generate a moons dataset using `make_moons(n_samples=10000, noise=0.4)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding `random_state=42` to make this notebook's output constant:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "\n",
    "X_moons, y_moons = make_moons(n_samples=10000, noise=0.4, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. Split it into a training set and a test set using `train_test_split()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_moons, y_moons,\n",
    "                                                    test_size=0.2,\n",
    "                                                    random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Use grid search with cross-validation (with the help of the `GridSearchCV` class) to find good hyperparameter values for a `DecisionTreeClassifier`. Hint: try various values for `max_leaf_nodes`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, estimator=DecisionTreeClassifier(random_state=42),\n",
       "             param_grid={'max_depth': [1, 2, 3, 4, 5, 6],\n",
       "                         'max_leaf_nodes': [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,\n",
       "                                            13, 14, 15, 16, 17, 18, 19, 20, 21,\n",
       "                                            22, 23, 24, 25, 26, 27, 28, 29, 30,\n",
       "                                            31, ...],\n",
       "                         'min_samples_split': [2, 3, 4]})"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "params = {\n",
    "    'max_leaf_nodes': list(range(2, 100)),\n",
    "    'max_depth': list(range(1, 7)),\n",
    "    'min_samples_split': [2, 3, 4]\n",
    "}\n",
    "grid_search_cv = GridSearchCV(DecisionTreeClassifier(random_state=42),\n",
    "                              params,\n",
    "                              cv=3)\n",
    "\n",
    "grid_search_cv.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(max_depth=6, max_leaf_nodes=17, random_state=42)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search_cv.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Train it on the full training set using these hyperparameters, and measure your model's performance on the test set. You should get roughly 85% to 87% accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, `GridSearchCV` trains the best model found on the whole training set (you can change this by setting `refit=False`), so we don't need to do it again. We can simply evaluate the model's accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8595"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "y_pred = grid_search_cv.predict(X_test)\n",
    "accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: Grow a forest._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. Continuing the previous exercise, generate 1,000 subsets of the training set, each containing 100 instances selected randomly. Hint: you can use Scikit-Learn's `ShuffleSplit` class for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import ShuffleSplit\n",
    "\n",
    "n_trees = 1000\n",
    "n_instances = 100\n",
    "\n",
    "mini_sets = []\n",
    "\n",
    "rs = ShuffleSplit(n_splits=n_trees, test_size=len(X_train) - n_instances,\n",
    "                  random_state=42)\n",
    "\n",
    "for mini_train_index, mini_test_index in rs.split(X_train):\n",
    "    X_mini_train = X_train[mini_train_index]\n",
    "    y_mini_train = y_train[mini_train_index]\n",
    "    mini_sets.append((X_mini_train, y_mini_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. Train one Decision Tree on each subset, using the best hyperparameter values found above. Evaluate these 1,000 Decision Trees on the test set. Since they were trained on smaller sets, these Decision Trees will likely perform worse than the first Decision Tree, achieving only about 80% accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.805671"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.base import clone\n",
    "\n",
    "forest = [clone(grid_search_cv.best_estimator_) for _ in range(n_trees)]\n",
    "\n",
    "accuracy_scores = []\n",
    "\n",
    "for tree, (X_mini_train, y_mini_train) in zip(forest, mini_sets):\n",
    "    tree.fit(X_mini_train, y_mini_train)\n",
    "    \n",
    "    y_pred = tree.predict(X_test)\n",
    "    accuracy_scores.append(accuracy_score(y_test, y_pred))\n",
    "\n",
    "np.mean(accuracy_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Now comes the magic. For each test set instance, generate the predictions of the 1,000 Decision Trees, and keep only the most frequent prediction (you can use SciPy's `mode()` function for this). This gives you _majority-vote predictions_ over the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_pred = np.empty([n_trees, len(X_test)], dtype=np.uint8)\n",
    "\n",
    "for tree_index, tree in enumerate(forest):\n",
    "    Y_pred[tree_index] = tree.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import mode\n",
    "\n",
    "y_pred_majority_votes, n_votes = mode(Y_pred, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Evaluate these predictions on the test set: you should obtain a slightly higher accuracy than your first model (about 0.5 to 1.5% higher). Congratulations, you have trained a Random Forest classifier!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.873"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_test, y_pred_majority_votes.reshape([-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "nav_menu": {
   "height": "309px",
   "width": "468px"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}