"2. Beware of multithreading, and make TensorFlow single-threaded.\n",
"3. Set all the random seeds.\n",
"4. Eliminate any other source of variability."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Do Not Run TensorFlow on the GPU"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Some operations (like `tf.reduce_sum()`) have favor performance over precision, and their outputs may vary slightly across runs. To get reproducible results, make sure TensorFlow runs on the CPU:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Beware of Multithreading"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Because floats have limited precision, the order of execution matters:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"2. * 5. / 7."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"2. / 7. * 5."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You should make sure TensorFlow runs your ops on a single thread:"
"The thread pools for all sessions are created when you create the first session, so all sessions in the rest of this notebook will be single-threaded:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"with tf.Session() as sess:\n",
" #... also single-threaded!\n",
" pass"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set all the random seeds!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Python's built-in `hash()` function"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"print(set(\"Try restarting the kernel and running this again\"))\n",
"print(set(\"Try restarting the kernel and running this again\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since Python 3.3, the result will be different every time, unless you start Python with the `PYTHONHASHSEED` environment variable set to `0`:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```shell\n",
"PYTHONHASHSEED=0 python\n",
"```\n",
"\n",
"```pycon\n",
">>> print(set(\"Now the output is stable across runs\"))\n",
"Alternatively, you could set this environment variable system-wide, but that's probably not a good idea, because this automatic randomization was [introduced for security reasons](http://ocert.org/advisories/ocert-2011-003.html)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Unfortunately, setting the environment variable from within Python (e.g., using `os.environ[\"PYTHONHASHSEED\"]=\"0\"`) will not work, because Python reads it upon startup. For Jupyter notebooks, you have to start the Jupyter server like this:\n",
" raise Exception(\"You must set PYTHONHASHSEED=0 when starting the Jupyter server to get reproducible results.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Python Random Number Generators (RNGs)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"import random\n",
"\n",
"random.seed(42)\n",
"print(random.random())\n",
"print(random.random())\n",
"\n",
"print()\n",
"\n",
"random.seed(42)\n",
"print(random.random())\n",
"print(random.random())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### NumPy RNGs"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"np.random.seed(42)\n",
"print(np.random.rand())\n",
"print(np.random.rand())\n",
"\n",
"print()\n",
"\n",
"np.random.seed(42)\n",
"print(np.random.rand())\n",
"print(np.random.rand())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### TensorFlow RNGs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"TensorFlow's behavior is more complex because of two things:\n",
"* you create a graph, and then you execute it. The random seed must be set before you create the random operations.\n",
"* there are two seeds: one at the graph level, and one at the individual random operation level."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"\n",
"tf.set_random_seed(42)\n",
"rnd = tf.random_uniform(shape=[])\n",
"\n",
"with tf.Session() as sess:\n",
" print(rnd.eval())\n",
" print(rnd.eval())\n",
"\n",
"print()\n",
"\n",
"with tf.Session() as sess:\n",
" print(rnd.eval())\n",
" print(rnd.eval())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Every time you reset the graph, you need to set the seed again:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"tf.reset_default_graph()\n",
"\n",
"tf.set_random_seed(42)\n",
"rnd = tf.random_uniform(shape=[])\n",
"\n",
"with tf.Session() as sess:\n",
" print(rnd.eval())\n",
" print(rnd.eval())\n",
"\n",
"print()\n",
"\n",
"with tf.Session() as sess:\n",
" print(rnd.eval())\n",
" print(rnd.eval())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you create your own graph, it will ignore the default graph's seed:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"tf.reset_default_graph()\n",
"tf.set_random_seed(42)\n",
"\n",
"graph = tf.Graph()\n",
"with graph.as_default():\n",
" rnd = tf.random_uniform(shape=[])\n",
"\n",
"with tf.Session(graph=graph):\n",
" print(rnd.eval())\n",
" print(rnd.eval())\n",
"\n",
"print()\n",
"\n",
"with tf.Session(graph=graph):\n",
" print(rnd.eval())\n",
" print(rnd.eval())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You must set its own seed:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"graph = tf.Graph()\n",
"with graph.as_default():\n",
" tf.set_random_seed(42)\n",
" rnd = tf.random_uniform(shape=[])\n",
"\n",
"with tf.Session(graph=graph):\n",
" print(rnd.eval())\n",
" print(rnd.eval())\n",
"\n",
"print()\n",
"\n",
"with tf.Session(graph=graph):\n",
" print(rnd.eval())\n",
" print(rnd.eval())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you set the seed after the random operation is created, the seed has no effet:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"tf.reset_default_graph()\n",
"\n",
"rnd = tf.random_uniform(shape=[])\n",
"\n",
"tf.set_random_seed(42) # BAD, NO EFFECT!\n",
"with tf.Session() as sess:\n",
" print(rnd.eval())\n",
" print(rnd.eval())\n",
"\n",
"print()\n",
"\n",
"tf.set_random_seed(42) # BAD, NO EFFECT!\n",
"with tf.Session() as sess:\n",
" print(rnd.eval())\n",
" print(rnd.eval())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### A note about operation seeds"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can also set a seed for each individual random operation. When you do, it is combined with the graph seed into the final seed used by that op. The following table summarizes how this works:\n",
"* `op_index = graph._last_id` when there is a graph seed, different random ops without op seeds will have different outputs. However, each of them will have the same sequence of outputs at every run.\n",
"\n",
"In eager mode, there is a global seed instead of graph seed (since there is no graph in eager mode)."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"tf.reset_default_graph()\n",
"\n",
"rnd1 = tf.random_uniform(shape=[], seed=42)\n",
"rnd2 = tf.random_uniform(shape=[], seed=42)\n",
"rnd3 = tf.random_uniform(shape=[])\n",
"\n",
"with tf.Session() as sess:\n",
" print(rnd1.eval())\n",
" print(rnd2.eval())\n",
" print(rnd3.eval())\n",
" print(rnd1.eval())\n",
" print(rnd2.eval())\n",
" print(rnd3.eval())\n",
"\n",
"print()\n",
"\n",
"with tf.Session() as sess:\n",
" print(rnd1.eval())\n",
" print(rnd2.eval())\n",
" print(rnd3.eval())\n",
" print(rnd1.eval())\n",
" print(rnd2.eval())\n",
" print(rnd3.eval())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the following example, you may think that all random ops will have the same random seed, but `rnd3` will actually have a different seed:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"tf.reset_default_graph()\n",
"\n",
"tf.set_random_seed(42)\n",
"\n",
"rnd1 = tf.random_uniform(shape=[], seed=42)\n",
"rnd2 = tf.random_uniform(shape=[], seed=42)\n",
"rnd3 = tf.random_uniform(shape=[])\n",
"\n",
"with tf.Session() as sess:\n",
" print(rnd1.eval())\n",
" print(rnd2.eval())\n",
" print(rnd3.eval())\n",
" print(rnd1.eval())\n",
" print(rnd2.eval())\n",
" print(rnd3.eval())\n",
"\n",
"print()\n",
"\n",
"with tf.Session() as sess:\n",
" print(rnd1.eval())\n",
" print(rnd2.eval())\n",
" print(rnd3.eval())\n",
" print(rnd1.eval())\n",
" print(rnd2.eval())\n",
" print(rnd3.eval())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Estimators API"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Tip**: in a Jupyter notebook, you probably want to set the random seeds regularly so that you can come back and run the notebook from there (instead of from the beginning) and still get reproducible outputs."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"random.seed(42)\n",
"np.random.seed(42)\n",
"tf.set_random_seed(42)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you use the Estimators API, make sure to create a `RunConfig` and set its `tf_random_seed`, then pass it to the constructor of your estimator:"
"Unfortunately, the `numpy_input_fn` does not allow us to set the seed when `shuffle=True`, so we must shuffle the data ourself and set `shuffle=False`."
"Instead of using the `numpy_input_fn()` function (which cannot reproducibly shuffle the dataset at each epoch), you can create your own input function using the Data API and set its shuffling seed:"
"I hope you enjoyed this notebook. If you do not get reproducible results, or if they are different than mine, then please [file an issue](https://github.com/ageron/handson-ml2/issues) on github, specifying what version of Python, TensorFlow, and NumPy you are using, as well as your O.S. version. Thank you!"
"If you want to learn more about Deep Learning and TensorFlow, check out my book [Hands-On Machine Learning with Scitkit-Learn and TensorFlow](http://homl.info/amazon), O'Reilly. You can also follow me on twitter [@aureliengeron](https://twitter.com/aureliengeron) or watch my videos on YouTube at [youtube.com/c/AurelienGeron](https://www.youtube.com/c/AurelienGeron)."
