Add exercise solutions for chapter 6

06_decision_trees.ipynb

@@ -104,7 +104,7 @@
    "outputs": [],
    "source": [
     "from sklearn.datasets import load_iris\n",
-    "from sklearn.tree import DecisionTreeClassifier, export_graphviz\n",
+    "from sklearn.tree import DecisionTreeClassifier\n",
     "\n",
     "iris = load_iris()\n",
     "X = iris.data[:, 2:] # petal length and width\n",
@@ -124,6 +124,8 @@
    },
    "outputs": [],
    "source": [
+    "from sklearn.tree import export_graphviz\n",
+    "\n",
     "export_graphviz(\n",
     "        tree_clf,\n",
     "        out_file=image_path(\"iris_tree.dot\"),\n",
@@ -382,6 +384,36 @@
   {
    "cell_type": "code",
    "execution_count": 13,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Quadratic training set + noise\n",
+    "rnd.seed(42)\n",
+    "m = 200\n",
+    "X = rnd.rand(m, 1)\n",
+    "y = 4 * (X - 0.5) ** 2\n",
+    "y = y + rnd.randn(m, 1) / 10"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "from sklearn.tree import DecisionTreeRegressor\n",
+    "\n",
+    "tree_reg = DecisionTreeRegressor(max_depth=2, random_state=42)\n",
+    "tree_reg.fit(X, y)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
    "metadata": {
     "collapsed": false,
     "deletable": true,
@@ -391,13 +423,6 @@
    "source": [
     "from sklearn.tree import DecisionTreeRegressor\n",
     "\n",
-    "# Quadratic training set + noise\n",
-    "rnd.seed(42)\n",
-    "m = 200\n",
-    "X = rnd.rand(m, 1)\n",
-    "y = 4 * (X - 0.5) ** 2\n",
-    "y = y + rnd.randn(m, 1) / 10\n",
-    "\n",
     "tree_reg1 = DecisionTreeRegressor(random_state=42, max_depth=2)\n",
     "tree_reg2 = DecisionTreeRegressor(random_state=42, max_depth=3)\n",
     "tree_reg1.fit(X, y)\n",
@@ -439,7 +464,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": 16,
    "metadata": {
     "collapsed": false,
     "deletable": true,
@@ -458,7 +483,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": 17,
    "metadata": {
     "collapsed": false,
     "deletable": true,
@@ -515,16 +540,273 @@
     "editable": true
    },
    "source": [
-    "**Coming soon**"
+    "## 1. to 6."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "See appendix A."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true,
+    "deletable": true,
+    "editable": true
+   },
+   "source": [
+    "## 7."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "_Exercise: train and fine-tune a Decision Tree for the moons dataset._"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "a. Generate a moons dataset using `make_moons(n_samples=10000, noise=0.4)`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Adding `random_state=42` to make this notebook's output constant:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "from sklearn.datasets import make_moons\n",
+    "\n",
+    "X, y = make_moons(n_samples=10000, noise=0.4, random_state=42)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "b. Split it into a training set and a test set using `train_test_split()`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "from sklearn.model_selection import train_test_split\n",
+    "\n",
+    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "c. Use grid search with cross-validation (with the help of the `GridSearchCV` class) to find good hyperparameter values for a `DecisionTreeClassifier`. Hint: try various values for `max_leaf_nodes`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "from sklearn.model_selection import GridSearchCV\n",
+    "\n",
+    "params = {'max_leaf_nodes': list(range(2, 100)), 'min_samples_split': [2, 3, 4]}\n",
+    "grid_search_cv = GridSearchCV(DecisionTreeClassifier(), params, n_jobs=-1, verbose=1)\n",
+    "\n",
+    "grid_search_cv.fit(X_train, y_train)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "grid_search_cv.best_estimator_"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "d. Train it on the full training set using these hyperparameters, and measure your model's performance on the test set. You should get roughly 85% to 87% accuracy."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "By default, `GridSearchCV` trains the best model found on the whole training set (you can change this by setting `refit=False`), so we don't need to do it again. We can simply evaluate the model's accuracy:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "from sklearn.metrics import accuracy_score\n",
+    "\n",
+    "y_pred = grid_search_cv.predict(X_test)\n",
+    "accuracy_score(y_test, y_pred)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 8."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "_Exercise: Grow a forest._"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "a. Continuing the previous exercise, generate 1,000 subsets of the training set, each containing 100 instances selected randomly. Hint: you can use Scikit-Learn's `ShuffleSplit` class for this."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "from sklearn.model_selection import ShuffleSplit\n",
+    "\n",
+    "n_trees = 1000\n",
+    "n_instances = 100\n",
+    "\n",
+    "mini_sets = []\n",
+    "\n",
+    "rs = ShuffleSplit(n_splits=n_trees, test_size=len(X_train) - n_instances, random_state=42)\n",
+    "for mini_train_index, mini_test_index in rs.split(X_train):\n",
+    "    X_mini_train = X_train[mini_train_index]\n",
+    "    y_mini_train = y_train[mini_train_index]\n",
+    "    mini_sets.append((X_mini_train, y_mini_train))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "b. Train one Decision Tree on each subset, using the best hyperparameter values found above. Evaluate these 1,000 Decision Trees on the test set. Since they were trained on smaller sets, these Decision Trees will likely perform worse than the first Decision Tree, achieving only about 80% accuracy."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "from sklearn.base import clone\n",
+    "\n",
+    "forest = [clone(grid_search_cv.best_estimator_) for _ in range(n_trees)]\n",
+    "\n",
+    "accuracy_scores = []\n",
+    "\n",
+    "for tree, (X_mini_train, y_mini_train) in zip(forest, mini_sets):\n",
+    "    tree.fit(X_mini_train, y_mini_train)\n",
+    "    \n",
+    "    y_pred = tree.predict(X_test)\n",
+    "    accuracy_scores.append(accuracy_score(y_test, y_pred))\n",
+    "\n",
+    "np.mean(accuracy_scores)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "c. Now comes the magic. For each test set instance, generate the predictions of the 1,000 Decision Trees, and keep only the most frequent prediction (you can use SciPy's `mode()` function for this). This gives you _majority-vote predictions_ over the test set."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "Y_pred = np.empty([n_trees, len(X_test)], dtype=np.uint8)\n",
+    "\n",
+    "for tree_index, tree in enumerate(forest):\n",
+    "    Y_pred[tree_index] = tree.predict(X_test)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "from scipy.stats import mode\n",
+    "\n",
+    "y_pred_majority_votes, n_votes = mode(Y_pred, axis=0)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "d. Evaluate these predictions on the test set: you should obtain a slightly higher accuracy than your first model (about 0.5 to 1.5% higher). Congratulations, you have trained a Random Forest classifier!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 27,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "accuracy_score(y_test, y_pred_majority_votes.reshape([-1]))"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "collapsed": true,
+    "collapsed": true
-    "deletable": true,
-    "editable": true
    },
    "outputs": [],
    "source": []
@@ -546,7 +828,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.2+"
+   "version": "3.5.3"
   },
   "nav_menu": {
    "height": "309px",