schuetoliver
/
handson-ml
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Make the code example 1-1 easier to read, and create a better `prepare_country_stats()` function

main
Aurélien Geron 2018-01-15 17:25:17 +01:00
parent c8b7f045ee
commit 94914db82e
1 changed files with 236 additions and 219 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

453
01_the_machine_learning_landscape.ipynb
View File

@ -2,10 +2,7 @@
"cells": [ "cells": [
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {},
"deletable": true,
"editable": true
},
"source": [ "source": [
"**Chapter 1 – The Machine Learning landscape**\n", "**Chapter 1 – The Machine Learning landscape**\n",
"\n", "\n",
@ -14,20 +11,14 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {},
"deletable": true,
"editable": true
},
"source": [ "source": [
"# Setup" "# Setup"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {},
"deletable": true,
"editable": true
},
"source": [ "source": [
"First, let's make sure this notebook works well in both python 2 and 3, import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures:" "First, let's make sure this notebook works well in both python 2 and 3, import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures:"
] ]
@ -36,9 +27,6 @@
"cell_type": "code", "cell_type": "code",
"execution_count": 1, "execution_count": 1,
"metadata": { "metadata": {
"collapsed": false,
"deletable": true,
"editable": true,
"slideshow": { "slideshow": {
"slide_type": "-" "slide_type": "-"
} }
@ -50,11 +38,10 @@
"\n", "\n",
"# Common imports\n", "# Common imports\n",
"import numpy as np\n", "import numpy as np\n",
"import numpy.random as rnd\n",
"import os\n", "import os\n",
"\n", "\n",
"# to make this notebook's output stable across runs\n", "# to make this notebook's output stable across runs\n",
"rnd.seed(42)\n", "np.random.seed(42)\n",
"\n", "\n",
"# To plot pretty figures\n", "# To plot pretty figures\n",
"%matplotlib inline\n", "%matplotlib inline\n",
@ -73,34 +60,172 @@
" print(\"Saving figure\", fig_id)\n", " print(\"Saving figure\", fig_id)\n",
" if tight_layout:\n", " if tight_layout:\n",
" plt.tight_layout()\n", " plt.tight_layout()\n",
" plt.savefig(path, format='png', dpi=300)" " plt.savefig(path, format='png', dpi=300)\n",
"\n",
"# Ignore useless warnings (see SciPy issue #5998)\n",
"import warnings\n",
"warnings.filterwarnings(action=\"ignore\", module=\"scipy\", message=\"^internal gelsd\")"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {},
"deletable": true,
"editable": true
},
"source": [ "source": [
"# Load and prepare Life satisfaction data" "# Code example 1-1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This function just merges the OECD's life satisfaction data and the IMF's GDP per capita data. It's a bit too long and boring and it's not specific to Machine Learning, which is why I left it out of the book."
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 2, "execution_count": 2,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"def prepare_country_stats(oecd_bli, gdp_per_capita):\n",
" oecd_bli = oecd_bli[oecd_bli[\"INEQUALITY\"]==\"TOT\"]\n",
" oecd_bli = oecd_bli.pivot(index=\"Country\", columns=\"Indicator\", values=\"Value\")\n",
" gdp_per_capita.rename(columns={\"2015\": \"GDP per capita\"}, inplace=True)\n",
" gdp_per_capita.set_index(\"Country\", inplace=True)\n",
" full_country_stats = pd.merge(left=oecd_bli, right=gdp_per_capita,\n",
" left_index=True, right_index=True)\n",
" full_country_stats.sort_values(by=\"GDP per capita\", inplace=True)\n",
" remove_indices = [0, 1, 6, 8, 33, 34, 35]\n",
" keep_indices = list(set(range(36)) - set(remove_indices))\n",
" return full_country_stats[[\"GDP per capita\", 'Life satisfaction']].iloc[keep_indices]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The code in the book expects the data files to be located in the current directory. I just tweaked it here to fetch the files in datasets/lifesat."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"datapath = os.path.join(\"datasets\", \"lifesat\", \"\")"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# Code example\n",
"import matplotlib\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import pandas as pd\n", "import pandas as pd\n",
"import sklearn.linear_model\n",
"\n", "\n",
"# Download CSV from http://stats.oecd.org/index.aspx?DataSetCode=BLI\n", "# Load the data\n",
"datapath = \"datasets/lifesat/\"\n", "oecd_bli = pd.read_csv(datapath + \"oecd_bli_2015.csv\", thousands=',')\n",
"gdp_per_capita = pd.read_csv(datapath + \"gdp_per_capita.csv\",thousands=',',delimiter='\\t',\n",
" encoding='latin1', na_values=\"n/a\")\n",
"\n", "\n",
"# Prepare the data\n",
"country_stats = prepare_country_stats(oecd_bli, gdp_per_capita)\n",
"X = np.c_[country_stats[\"GDP per capita\"]]\n",
"y = np.c_[country_stats[\"Life satisfaction\"]]\n",
"\n",
"# Visualize the data\n",
"country_stats.plot(kind='scatter', x=\"GDP per capita\", y='Life satisfaction')\n",
"plt.show()\n",
"\n",
"# Select a linear model\n",
"model = sklearn.linear_model.LinearRegression()\n",
"\n",
"# Train the model\n",
"model.fit(X, y)\n",
"\n",
"# Make a prediction for Cyprus\n",
"X_new = [[22587]] # Cyprus' GDP per capita\n",
"print(model.predict(X_new)) # outputs [[ 5.96242338]]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Note: you can ignore the rest of this notebook, it just generates many of the figures in chapter 1."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Load and prepare Life satisfaction data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you want, you can get fresh data from the OECD's website.\n",
"Download the CSV from http://stats.oecd.org/index.aspx?DataSetCode=BLI\n",
"and save it to `datasets/lifesat/`."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"oecd_bli = pd.read_csv(datapath + \"oecd_bli_2015.csv\", thousands=',')\n", "oecd_bli = pd.read_csv(datapath + \"oecd_bli_2015.csv\", thousands=',')\n",
"oecd_bli = oecd_bli[oecd_bli[\"INEQUALITY\"]==\"TOT\"]\n", "oecd_bli = oecd_bli[oecd_bli[\"INEQUALITY\"]==\"TOT\"]\n",
"oecd_bli = oecd_bli.pivot(index=\"Country\", columns=\"Indicator\", values=\"Value\")\n", "oecd_bli = oecd_bli.pivot(index=\"Country\", columns=\"Indicator\", values=\"Value\")\n",
@ -109,12 +234,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 3, "execution_count": 6,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"oecd_bli[\"Life satisfaction\"].head()" "oecd_bli[\"Life satisfaction\"].head()"
@ -122,25 +243,24 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {},
"deletable": true,
"editable": true
},
"source": [ "source": [
"# Load and prepare GDP per capita data" "# Load and prepare GDP per capita data"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "markdown",
"execution_count": 4, "metadata": {},
"metadata": { "source": [
"collapsed": false, "Just like above, you can update the GDP per capita data if you want. Just download data from http://goo.gl/j1MSKe (=> imf.org) and save it to `datasets/lifesat/`."
"deletable": true, ]
"editable": true
}, },
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"# Download data from http://goo.gl/j1MSKe (=> imf.org)\n",
"gdp_per_capita = pd.read_csv(datapath+\"gdp_per_capita.csv\", thousands=',', delimiter='\\t',\n", "gdp_per_capita = pd.read_csv(datapath+\"gdp_per_capita.csv\", thousands=',', delimiter='\\t',\n",
" encoding='latin1', na_values=\"n/a\")\n", " encoding='latin1', na_values=\"n/a\")\n",
"gdp_per_capita.rename(columns={\"2015\": \"GDP per capita\"}, inplace=True)\n", "gdp_per_capita.rename(columns={\"2015\": \"GDP per capita\"}, inplace=True)\n",
@ -150,12 +270,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 5, "execution_count": 8,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"full_country_stats = pd.merge(left=oecd_bli, right=gdp_per_capita, left_index=True, right_index=True)\n", "full_country_stats = pd.merge(left=oecd_bli, right=gdp_per_capita, left_index=True, right_index=True)\n",
@ -165,12 +281,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 6, "execution_count": 9,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"full_country_stats[[\"GDP per capita\", 'Life satisfaction']].loc[\"United States\"]" "full_country_stats[[\"GDP per capita\", 'Life satisfaction']].loc[\"United States\"]"
@ -178,12 +290,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 7, "execution_count": 10,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"remove_indices = [0, 1, 6, 8, 33, 34, 35]\n", "remove_indices = [0, 1, 6, 8, 33, 34, 35]\n",
@ -195,12 +303,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 8, "execution_count": 11,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"sample_data.plot(kind='scatter', x=\"GDP per capita\", y='Life satisfaction', figsize=(5,3))\n", "sample_data.plot(kind='scatter', x=\"GDP per capita\", y='Life satisfaction', figsize=(5,3))\n",
@ -224,25 +328,17 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 9, "execution_count": 12,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"sample_data.to_csv(\"life_satisfaction_vs_gdp_per_capita.csv\")" "sample_data.to_csv(os.path.join(\"datasets\", \"lifesat\", \"lifesat.csv\"))"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 10, "execution_count": 13,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"sample_data.loc[list(position_text.keys())]" "sample_data.loc[list(position_text.keys())]"
@ -250,12 +346,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 11, "execution_count": 14,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"import numpy as np\n", "import numpy as np\n",
@ -278,12 +370,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 12, "execution_count": 15,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"from sklearn import linear_model\n", "from sklearn import linear_model\n",
@ -297,12 +385,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 13, "execution_count": 16,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"sample_data.plot(kind='scatter', x=\"GDP per capita\", y='Life satisfaction', figsize=(5,3))\n", "sample_data.plot(kind='scatter', x=\"GDP per capita\", y='Life satisfaction', figsize=(5,3))\n",
@ -317,12 +401,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 14, "execution_count": 17,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"cyprus_gdp_per_capita = gdp_per_capita.loc[\"Cyprus\"][\"GDP per capita\"]\n", "cyprus_gdp_per_capita = gdp_per_capita.loc[\"Cyprus\"][\"GDP per capita\"]\n",
@ -333,12 +413,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 15, "execution_count": 18,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"sample_data.plot(kind='scatter', x=\"GDP per capita\", y='Life satisfaction', figsize=(5,3), s=1)\n", "sample_data.plot(kind='scatter', x=\"GDP per capita\", y='Life satisfaction', figsize=(5,3), s=1)\n",
@ -356,12 +432,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 16, "execution_count": 19,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"sample_data[7:10]" "sample_data[7:10]"
@ -369,12 +441,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 17, "execution_count": 20,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"(5.1+5.7+6.5)/3" "(5.1+5.7+6.5)/3"
@ -382,28 +450,29 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 18, "execution_count": 21,
"metadata": { "metadata": {},
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"backup = oecd_bli, gdp_per_capita\n", "backup = oecd_bli, gdp_per_capita\n",
"\n", "\n",
"def prepare_country_stats(oecd_bli, gdp_per_capita):\n", "def prepare_country_stats(oecd_bli, gdp_per_capita):\n",
" return sample_data" " oecd_bli = oecd_bli[oecd_bli[\"INEQUALITY\"]==\"TOT\"]\n",
" oecd_bli = oecd_bli.pivot(index=\"Country\", columns=\"Indicator\", values=\"Value\")\n",
" gdp_per_capita.rename(columns={\"2015\": \"GDP per capita\"}, inplace=True)\n",
" gdp_per_capita.set_index(\"Country\", inplace=True)\n",
" full_country_stats = pd.merge(left=oecd_bli, right=gdp_per_capita,\n",
" left_index=True, right_index=True)\n",
" full_country_stats.sort_values(by=\"GDP per capita\", inplace=True)\n",
" remove_indices = [0, 1, 6, 8, 33, 34, 35]\n",
" keep_indices = list(set(range(36)) - set(remove_indices))\n",
" return full_country_stats[[\"GDP per capita\", 'Life satisfaction']].iloc[keep_indices]"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 19, "execution_count": 22,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"# Code example\n", "# Code example\n",
@ -440,12 +509,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 20, "execution_count": 23,
"metadata": { "metadata": {},
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"oecd_bli, gdp_per_capita = backup" "oecd_bli, gdp_per_capita = backup"
@ -453,12 +518,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 21, "execution_count": 24,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"missing_data" "missing_data"
@ -466,12 +527,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 22, "execution_count": 25,
"metadata": { "metadata": {},
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"position_text2 = {\n", "position_text2 = {\n",
@ -487,12 +544,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 23, "execution_count": 26,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"sample_data.plot(kind='scatter', x=\"GDP per capita\", y='Life satisfaction', figsize=(8,3))\n", "sample_data.plot(kind='scatter', x=\"GDP per capita\", y='Life satisfaction', figsize=(8,3))\n",
@ -522,12 +575,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 24, "execution_count": 27,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"full_country_stats.plot(kind='scatter', x=\"GDP per capita\", y='Life satisfaction', figsize=(8,3))\n", "full_country_stats.plot(kind='scatter', x=\"GDP per capita\", y='Life satisfaction', figsize=(8,3))\n",
@ -550,12 +599,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 25, "execution_count": 28,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"full_country_stats.loc[[c for c in full_country_stats.index if \"W\" in c.upper()]][\"Life satisfaction\"]" "full_country_stats.loc[[c for c in full_country_stats.index if \"W\" in c.upper()]][\"Life satisfaction\"]"
@ -563,12 +608,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 26, "execution_count": 29,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"gdp_per_capita.loc[[c for c in gdp_per_capita.index if \"W\" in c.upper()]].head()" "gdp_per_capita.loc[[c for c in gdp_per_capita.index if \"W\" in c.upper()]].head()"
@ -576,12 +617,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 27, "execution_count": 30,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"plt.figure(figsize=(8,3))\n", "plt.figure(figsize=(8,3))\n",
@ -611,12 +648,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 28, "execution_count": 31,
"metadata": { "metadata": {},
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"backup = oecd_bli, gdp_per_capita\n", "backup = oecd_bli, gdp_per_capita\n",
@ -627,12 +660,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 29, "execution_count": 32,
"metadata": { "metadata": {},
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"# Replace this linear model:\n", "# Replace this linear model:\n",
@ -641,12 +670,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 30, "execution_count": 33,
"metadata": { "metadata": {},
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"# with this k-neighbors regression model:\n", "# with this k-neighbors regression model:\n",
@ -655,12 +680,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 31, "execution_count": 34,
"metadata": { "metadata": {},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [ "source": [
"X = np.c_[country_stats[\"GDP per capita\"]]\n", "X = np.c_[country_stats[\"GDP per capita\"]]\n",
@ -677,11 +698,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"metadata": { "metadata": {},
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [], "outputs": [],
"source": [] "source": []
} }
@ -702,7 +719,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.5.3" "version": "3.6.3"
}, },
"nav_menu": {}, "nav_menu": {},
"toc": { "toc": {
@ -723,5 +740,5 @@
} }
}, },
"nbformat": 4, "nbformat": 4,
"nbformat_minor": 0 "nbformat_minor": 1
} }