schuetoliver
/
handson-ml
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Improve alignment between notebook and book section headers

main
Aurélien Geron 2021-10-03 23:05:49 +13:00
parent 6b821335c0
commit 3f89676892
6 changed files with 560 additions and 151 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
01_the_machine_learning_landscape.ipynb
View File

@ -93,7 +93,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"The code in the book expects the data files to be located in the current directory. I just tweaked it here to fetch the files in datasets/lifesat." "The code in the book expects the data files to be located in the current directory. I just tweaked it here to fetch the files in `datasets/lifesat`."
] ]
}, },
{ {

145
05_support_vector_machines.ipynb
View File

@ -84,14 +84,16 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Large margin classification" "# Linear SVM Classification"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"The next few code cells generate the first figures in chapter 5. The first actual code sample comes after:" "The next few code cells generate the first figures in chapter 5. The first actual code sample comes after.\n",
"\n",
"**Code to generate Figure 5–1. Large margin classification**"
] ]
}, },
{ {
@ -175,7 +177,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Sensitivity to feature scales" "**Code to generate Figure 5–2. Sensitivity to feature scales**"
] ]
}, },
{ {
@ -220,7 +222,8 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Sensitivity to outliers" "## Soft Margin Classification\n",
"**Code to generate Figure 5–3. Hard margin sensitivity to outliers**"
] ]
}, },
{ {
@ -278,14 +281,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Large margin *vs* margin violations" "**This is the first code example in chapter 5:**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is the first code example in chapter 5:"
] ]
}, },
{ {
@ -325,7 +321,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"Now let's generate the graph comparing different regularization settings:" "**Code to generate Figure 5–4. Large margin versus fewer margin violations**"
] ]
}, },
{ {
@ -408,7 +404,14 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Non-linear classification" "# Nonlinear SVM Classification"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 5–5. Adding features to make a dataset linearly separable**"
] ]
}, },
{ {
@ -471,6 +474,13 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Here is second code example in the chapter:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 13, "execution_count": 13,
@ -490,6 +500,13 @@
"polynomial_svm_clf.fit(X, y)" "polynomial_svm_clf.fit(X, y)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 5–6. Linear SVM classifier using polynomial features**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 14, "execution_count": 14,
@ -513,6 +530,20 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Polynomial Kernel"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Next code example:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 15, "execution_count": 15,
@ -528,6 +559,13 @@
"poly_kernel_svm_clf.fit(X, y)" "poly_kernel_svm_clf.fit(X, y)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 5–7. SVM classifiers with a polynomial kernel**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 16, "execution_count": 16,
@ -564,6 +602,20 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Similarity Features"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 5–8. Similarity features using the Gaussian RBF**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 18, "execution_count": 18,
@ -644,6 +696,20 @@
" print(\"Phi({}, {}) = {}\".format(x1_example, landmark, k))" " print(\"Phi({}, {}) = {}\".format(x1_example, landmark, k))"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Gaussian RBF Kernel"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Next code example:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 20, "execution_count": 20,
@ -657,6 +723,13 @@
"rbf_kernel_svm_clf.fit(X, y)" "rbf_kernel_svm_clf.fit(X, y)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 5–9. SVM classifiers using an RBF kernel**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 21, "execution_count": 21,
@ -701,7 +774,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Regression\n" "# SVM Regression"
] ]
}, },
{ {
@ -716,6 +789,13 @@
"y = (4 + 3 * X + np.random.randn(m, 1)).ravel()" "y = (4 + 3 * X + np.random.randn(m, 1)).ravel()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Next code example:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 23, "execution_count": 23,
@ -728,6 +808,13 @@
"svm_reg.fit(X, y)" "svm_reg.fit(X, y)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 5–10. SVM Regression**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 24, "execution_count": 24,
@ -807,6 +894,13 @@
"**Note**: to be future-proof, we set `gamma=\"scale\"`, as this will be the default value in Scikit-Learn 0.22." "**Note**: to be future-proof, we set `gamma=\"scale\"`, as this will be the default value in Scikit-Learn 0.22."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Next code example:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 27, "execution_count": 27,
@ -819,6 +913,13 @@
"svm_poly_reg.fit(X, y)" "svm_poly_reg.fit(X, y)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 5–11. SVM Regression using a second-degree polynomial kernel**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 28, "execution_count": 28,
@ -855,7 +956,15 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Under the hood" "# Under the Hood\n",
"## Decision Function and Predictions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 5–12. Decision function for the iris dataset**"
] ]
}, },
{ {
@ -917,7 +1026,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Small weight vector results in a large margin" "**Code to generate Figure 5–13. A smaller weight vector results in a larger margin**"
] ]
}, },
{ {
@ -976,7 +1085,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Hinge loss" "**Code to generate the Hinge Loss figure:**"
] ]
}, },
{ {

130
06_decision_trees.ipynb
View File

@ -89,7 +89,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Training and visualizing" "# Training and Visualizing a Decision Tree"
] ]
}, },
{ {
@ -109,6 +109,13 @@
"tree_clf.fit(X, y)" "tree_clf.fit(X, y)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**This code example generates Figure 6–1. Iris Decision Tree:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 3, "execution_count": 3,
@ -130,6 +137,20 @@
"Source.from_file(os.path.join(IMAGES_PATH, \"iris_tree.dot\"))" "Source.from_file(os.path.join(IMAGES_PATH, \"iris_tree.dot\"))"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Making Predictions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 6–2. Decision Tree decision boundaries**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 4, "execution_count": 4,
@ -181,7 +202,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Predicting classes and class probabilities" "# Estimating Class Probabilities"
] ]
}, },
{ {
@ -206,7 +227,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# High Variance" "## Regularization Hyperparameters"
] ]
}, },
{ {
@ -227,6 +248,13 @@
"tree_clf_tweaked.fit(X, y)" "tree_clf_tweaked.fit(X, y)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 6–8. Sensitivity to training set details:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 8, "execution_count": 8,
@ -244,9 +272,16 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 6–3. Regularization using min_samples_leaf:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 10, "execution_count": 9,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -271,9 +306,16 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Rotating the dataset also leads to completely different decision boundaries:"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 11, "execution_count": 10,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -290,9 +332,16 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 6–7. Sensitivity to training set rotation**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 12, "execution_count": 11,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -324,12 +373,19 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Regression trees" "# Regression"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's prepare a simple linear dataset:"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 13, "execution_count": 12,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -341,9 +397,16 @@
"y = y + np.random.randn(m, 1) / 10" "y = y + np.random.randn(m, 1) / 10"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code example:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 14, "execution_count": 13,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -353,9 +416,16 @@
"tree_reg.fit(X, y)" "tree_reg.fit(X, y)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 6–5. Predictions of two Decision Tree regression models:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 15, "execution_count": 14,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -400,9 +470,16 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 6-4. A Decision Tree for regression:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 16, "execution_count": 15,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -417,16 +494,23 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 17, "execution_count": 16,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"Source.from_file(os.path.join(IMAGES_PATH, \"regression_tree.dot\"))" "Source.from_file(os.path.join(IMAGES_PATH, \"regression_tree.dot\"))"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 6–6. Regularizing a Decision Tree regressor:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 18, "execution_count": 17,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -512,7 +596,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 19, "execution_count": 18,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -530,7 +614,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 20, "execution_count": 19,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -548,7 +632,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 21, "execution_count": 20,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -562,7 +646,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 22, "execution_count": 21,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -585,7 +669,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 23, "execution_count": 22,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -618,7 +702,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 24, "execution_count": 23,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -645,7 +729,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 25, "execution_count": 24,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -673,7 +757,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 26, "execution_count": 25,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -685,7 +769,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 27, "execution_count": 26,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -703,7 +787,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 28, "execution_count": 27,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [

225
07_ensemble_learning_and_random_forests.ipynb
View File

@ -89,7 +89,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Voting classifiers" "# Voting Classifiers"
] ]
}, },
{ {
@ -103,6 +103,13 @@
"cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)" "cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 7–3. The law of large numbers:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 3, "execution_count": 3,
@ -121,6 +128,13 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's use the moons dataset:"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 4, "execution_count": 4,
@ -141,6 +155,13 @@
"**Note**: to be future-proof, we set `solver=\"lbfgs\"`, `n_estimators=100`, and `gamma=\"scale\"` since these will be the default values in upcoming Scikit-Learn versions." "**Note**: to be future-proof, we set `solver=\"lbfgs\"`, `n_estimators=100`, and `gamma=\"scale\"` since these will be the default values in upcoming Scikit-Learn versions."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code examples:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 5, "execution_count": 5,
@ -232,7 +253,8 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Bagging ensembles" "# Bagging and Pasting\n",
"## Bagging and Pasting in Scikit-Learn"
] ]
}, },
{ {
@ -273,6 +295,13 @@
"print(accuracy_score(y_test, y_pred_tree))" "print(accuracy_score(y_test, y_pred_tree))"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 7–5. A single Decision Tree (left) versus a bagging ensemble of 500 trees (right):**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 13, "execution_count": 13,
@ -302,7 +331,9 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 14, "execution_count": 14,
"metadata": {}, "metadata": {
"scrolled": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"fix, axes = plt.subplots(ncols=2, figsize=(10,4), sharey=True)\n", "fix, axes = plt.subplots(ncols=2, figsize=(10,4), sharey=True)\n",
@ -321,7 +352,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Random Forests" "## Out-of-Bag evaluation"
] ]
}, },
{ {
@ -331,8 +362,10 @@
"outputs": [], "outputs": [],
"source": [ "source": [
"bag_clf = BaggingClassifier(\n", "bag_clf = BaggingClassifier(\n",
" DecisionTreeClassifier(max_features=\"sqrt\", max_leaf_nodes=16),\n", " DecisionTreeClassifier(), n_estimators=500,\n",
" n_estimators=500, random_state=42)" " bootstrap=True, oob_score=True, random_state=40)\n",
"bag_clf.fit(X_train, y_train)\n",
"bag_clf.oob_score_"
] ]
}, },
{ {
@ -341,13 +374,32 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"bag_clf.fit(X_train, y_train)\n", "bag_clf.oob_decision_function_"
"y_pred = bag_clf.predict(X_test)"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 17, "execution_count": 17,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"from sklearn.metrics import accuracy_score\n",
"y_pred = bag_clf.predict(X_test)\n",
"accuracy_score(y_test, y_pred)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Random Forests"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -359,18 +411,53 @@
"y_pred_rf = rnd_clf.predict(X_test)" "y_pred_rf = rnd_clf.predict(X_test)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A Random Forest is equivalent to a bag of decision trees:"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 18, "execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"bag_clf = BaggingClassifier(\n",
" DecisionTreeClassifier(max_features=\"sqrt\", max_leaf_nodes=16),\n",
" n_estimators=500, random_state=42)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"bag_clf.fit(X_train, y_train)\n",
"y_pred = bag_clf.predict(X_test)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"np.sum(y_pred == y_pred_rf) / len(y_pred) # very similar predictions" "np.sum(y_pred == y_pred_rf) / len(y_pred) # very similar predictions"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Feature Importance"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 19, "execution_count": 22,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -384,16 +471,23 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 20, "execution_count": 23,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"rnd_clf.feature_importances_" "rnd_clf.feature_importances_"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following figure overlays the decision boundaries of 15 decision trees. As you can see, even though each decision tree is imperfect, the ensemble defines a pretty good decision boundary:"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 21, "execution_count": 24,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -412,47 +506,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Out-of-Bag evaluation" "**Code to generate Figure 7–6. MNIST pixel importance (according to a Random Forest classifier):**"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"bag_clf = BaggingClassifier(\n",
" DecisionTreeClassifier(), n_estimators=500,\n",
" bootstrap=True, oob_score=True, random_state=40)\n",
"bag_clf.fit(X_train, y_train)\n",
"bag_clf.oob_score_"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"bag_clf.oob_decision_function_"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.metrics import accuracy_score\n",
"y_pred = bag_clf.predict(X_test)\n",
"accuracy_score(y_test, y_pred)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Feature importance"
] ]
}, },
{ {
@ -516,7 +570,8 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# AdaBoost" "# Boosting\n",
"## AdaBoost"
] ]
}, },
{ {
@ -542,6 +597,13 @@
"plot_decision_boundary(ada_clf, X, y)" "plot_decision_boundary(ada_clf, X, y)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 7–8. Decision boundaries of consecutive predictors:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 31, "execution_count": 31,
@ -583,7 +645,14 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Gradient Boosting" "## Gradient Boosting"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let create a simple quadratic dataset:"
] ]
}, },
{ {
@ -597,6 +666,13 @@
"y = 3*X[:, 0]**2 + 0.05 * np.random.randn(100)" "y = 3*X[:, 0]**2 + 0.05 * np.random.randn(100)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's train a decision tree regressor on this dataset:"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 33, "execution_count": 33,
@ -658,6 +734,13 @@
"y_pred" "y_pred"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 7–9. In this depiction of Gradient Boosting, the first predictor (top left) is trained normally, then each consecutive predictor (middle left and lower left) is trained on the previous predictor’s residuals; the right column shows the resulting ensemble’s predictions:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 39, "execution_count": 39,
@ -714,6 +797,13 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's try a gradient boosting regressor:"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 41, "execution_count": 41,
@ -726,6 +816,13 @@
"gbrt.fit(X, y)" "gbrt.fit(X, y)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 7–10. GBRT ensembles with not enough predictors (left) and too many (right):**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 42, "execution_count": 42,
@ -763,7 +860,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Gradient Boosting with Early stopping" "**Gradient Boosting with Early stopping:**"
] ]
}, },
{ {
@ -789,6 +886,13 @@
"gbrt_best.fit(X_train, y_train)" "gbrt_best.fit(X_train, y_train)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 7–11. Tuning the number of trees using early stopping:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 45, "execution_count": 45,
@ -827,6 +931,13 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Early stopping with some patience (interrupts training only after there's no improvement for 5 epochs):"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 47, "execution_count": 47,
@ -873,7 +984,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Using XGBoost" "**Using XGBoost:**"
] ]
}, },
{ {

172
08_dimensionality_reduction.ipynb
View File

@ -84,8 +84,8 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Projection methods\n", "# PCA\n",
"Build 3D dataset:" "Let's build a simple 3D dataset:"
] ]
}, },
{ {
@ -110,7 +110,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## PCA using SVD decomposition" "## Principal Components"
] ]
}, },
{ {
@ -146,6 +146,13 @@
"np.allclose(X_centered, U.dot(S).dot(Vt))" "np.allclose(X_centered, U.dot(S).dot(Vt))"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Projecting Down to d Dimensions"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 6, "execution_count": 6,
@ -169,7 +176,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## PCA using Scikit-Learn" "## Using Scikit-Learn"
] ]
}, },
{ {
@ -344,6 +351,13 @@
"Notice how the axes are flipped." "Notice how the axes are flipped."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Explained Variance Ratio"
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -406,6 +420,13 @@
"Next, let's generate some nice figures! :)" "Next, let's generate some nice figures! :)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 8–2. A 3D dataset lying close to a 2D subspace:**"
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -515,6 +536,13 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 8–3. The new 2D dataset after projection:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 25, "execution_count": 25,
@ -540,8 +568,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Manifold learning\n", "**Code to generate Figure 8–4. Swiss roll dataset:**"
"Swiss roll:"
] ]
}, },
{ {
@ -551,6 +578,7 @@
"outputs": [], "outputs": [],
"source": [ "source": [
"from sklearn.datasets import make_swiss_roll\n", "from sklearn.datasets import make_swiss_roll\n",
"\n",
"X, t = make_swiss_roll(n_samples=1000, noise=0.2, random_state=42)" "X, t = make_swiss_roll(n_samples=1000, noise=0.2, random_state=42)"
] ]
}, },
@ -578,6 +606,13 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 8–5. Squashing by projecting onto a plane (left) versus unrolling the Swiss roll (right):**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 28, "execution_count": 28,
@ -603,6 +638,13 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 8–6. The decision boundary may not always be simpler with lower dimensions:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 29, "execution_count": 29,
@ -688,7 +730,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# PCA" "**Code to generate Figure 8–7. Selecting the subspace to project on:**"
] ]
}, },
{ {
@ -761,7 +803,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# MNIST compression" "## Choosing the Right Number of Dimensions"
] ]
}, },
{ {
@ -818,6 +860,13 @@
"d" "d"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 8–8. Explained variance as a function of the number of dimensions:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 35, "execution_count": 35,
@ -867,6 +916,13 @@
"np.sum(pca.explained_variance_ratio_)" "np.sum(pca.explained_variance_ratio_)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## PCA for Compression"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 39, "execution_count": 39,
@ -878,6 +934,13 @@
"X_recovered = pca.inverse_transform(X_reduced)" "X_recovered = pca.inverse_transform(X_reduced)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 8–9. MNIST compression that preserves 95% of the variance:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 40, "execution_count": 40,
@ -930,7 +993,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Incremental PCA" "## Randomized PCA"
] ]
}, },
{ {
@ -938,6 +1001,23 @@
"execution_count": 43, "execution_count": 43,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [
"rnd_pca = PCA(n_components=154, svd_solver=\"randomized\", random_state=42)\n",
"X_reduced = rnd_pca.fit_transform(X_train)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Incremental PCA"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [],
"source": [ "source": [
"from sklearn.decomposition import IncrementalPCA\n", "from sklearn.decomposition import IncrementalPCA\n",
"\n", "\n",
@ -952,16 +1032,23 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 44, "execution_count": 45,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"X_recovered_inc_pca = inc_pca.inverse_transform(X_reduced)" "X_recovered_inc_pca = inc_pca.inverse_transform(X_reduced)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's check that compression still works well:"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 45, "execution_count": 46,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -975,7 +1062,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 46, "execution_count": 47,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -991,7 +1078,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 47, "execution_count": 48,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -1007,7 +1094,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 48, "execution_count": 49,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -1018,7 +1105,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### Using `memmap()`" "**Using `memmap()`:**"
] ]
}, },
{ {
@ -1030,7 +1117,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 49, "execution_count": 50,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -1050,7 +1137,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 50, "execution_count": 51,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -1066,7 +1153,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 51, "execution_count": 52,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -1077,21 +1164,11 @@
"inc_pca.fit(X_mm)" "inc_pca.fit(X_mm)"
] ]
}, },
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [],
"source": [
"rnd_pca = PCA(n_components=154, svd_solver=\"randomized\", random_state=42)\n",
"X_reduced = rnd_pca.fit_transform(X_train)"
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Time complexity" "**Time complexity:**"
] ]
}, },
{ {
@ -1226,6 +1303,13 @@
"X_reduced = rbf_pca.fit_transform(X)" "X_reduced = rbf_pca.fit_transform(X)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 8–10. Swiss roll reduced to 2D using kPCA with various kernels:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 58, "execution_count": 58,
@ -1260,6 +1344,13 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 8–11. Kernel PCA and the reconstruction pre-image error:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 59, "execution_count": 59,
@ -1300,6 +1391,13 @@
"plt.grid(True)" "plt.grid(True)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Selecting a Kernel and Tuning Hyperparameters"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 61, "execution_count": 61,
@ -1384,6 +1482,13 @@
"X_reduced = lle.fit_transform(X)" "X_reduced = lle.fit_transform(X)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 8–12. Unrolled Swiss roll using LLE:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 67, "execution_count": 67,
@ -1405,7 +1510,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# MDS, Isomap and t-SNE" "## Other Dimensionality Reduction Techniques"
] ]
}, },
{ {
@ -1459,6 +1564,13 @@
"X_reduced_lda = lda.transform(X_mnist)" "X_reduced_lda = lda.transform(X_mnist)"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Code to generate Figure 8–13. Using various techniques to reduce the Swill roll to 2D:**"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 72, "execution_count": 72,

37
09_unsupervised_learning.ipynb
View File

@ -91,7 +91,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Introduction – Classification _vs_ Clustering" "**Introduction – Classification _vs_ Clustering**"
] ]
}, },
{ {
@ -320,7 +320,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### Fit and Predict" "**Fit and predict**"
] ]
}, },
{ {
@ -428,7 +428,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### Decision Boundaries" "**Decision Boundaries**"
] ]
}, },
{ {
@ -507,7 +507,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### Hard Clustering _vs_ Soft Clustering" "**Hard Clustering _vs_ Soft Clustering**"
] ]
}, },
{ {
@ -546,7 +546,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### K-Means Algorithm" "### The K-Means Algorithm"
] ]
}, },
{ {
@ -639,7 +639,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### K-Means Variability" "**K-Means Variability**"
] ]
}, },
{ {
@ -827,7 +827,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### K-Means++" "### Centroid initialization methods"
] ]
}, },
{ {
@ -1432,7 +1432,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### Limits of K-Means" "## Limits of K-Means"
] ]
}, },
{ {
@ -1494,7 +1494,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### Using clustering for image segmentation" "## Using Clustering for Image Segmentation"
] ]
}, },
{ {
@ -1578,7 +1578,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### Using Clustering for Preprocessing" "## Using Clustering for Preprocessing"
] ]
}, },
{ {
@ -1785,7 +1785,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### Clustering for Semi-supervised Learning" "## Using Clustering for Semi-Supervised Learning"
] ]
}, },
{ {
@ -2756,7 +2756,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Anomaly Detection using Gaussian Mixtures" "## Anomaly Detection Using Gaussian Mixtures"
] ]
}, },
{ {
@ -2797,7 +2797,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Model selection" "## Selecting the Number of Clusters"
] ]
}, },
{ {
@ -2983,7 +2983,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Variational Bayesian Gaussian Mixtures" "## Bayesian Gaussian Mixture Models"
] ]
}, },
{ {
@ -3151,7 +3151,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Likelihood Function" "**Likelihood Function**"
] ]
}, },
{ {
@ -3242,13 +3242,6 @@
"plt.show()" "plt.show()"
] ]
}, },
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},