schuetoliver
/
handson-ml
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

No bias before BN layers

main
Aurélien Geron 2019-03-25 12:03:44 +08:00
parent 1774ede668
commit 3313c21723
1 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
11_deep_learning.ipynb
View File

@ -643,7 +643,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"Sometimes applying BN before the activation function works better (there's a debate on this topic):" "Sometimes applying BN before the activation function works better (there's a debate on this topic). Moreover, the layer before a `BatchNormalization` layer does not need to have bias terms, since the `BatchNormalization` layer some as well, it would be a waste of parameters, so you can set `use_bias=False` when creating those layers:"
] ]
}, },
{ {
@ -655,10 +655,10 @@
"model = keras.models.Sequential([\n", "model = keras.models.Sequential([\n",
" keras.layers.Flatten(input_shape=[28, 28]),\n", " keras.layers.Flatten(input_shape=[28, 28]),\n",
" keras.layers.BatchNormalization(),\n", " keras.layers.BatchNormalization(),\n",
" keras.layers.Dense(300),\n", " keras.layers.Dense(300, use_bias=False),\n",
" keras.layers.BatchNormalization(),\n", " keras.layers.BatchNormalization(),\n",
" keras.layers.Activation(\"relu\"),\n", " keras.layers.Activation(\"relu\"),\n",
" keras.layers.Dense(100),\n", " keras.layers.Dense(100, use_bias=False),\n",
" keras.layers.Activation(\"relu\"),\n", " keras.layers.Activation(\"relu\"),\n",
" keras.layers.BatchNormalization(),\n", " keras.layers.BatchNormalization(),\n",
" keras.layers.Dense(10, activation=\"softmax\")\n", " keras.layers.Dense(10, activation=\"softmax\")\n",