{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Chapter 11 – Training Deep Neural Networks**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"_This notebook contains all the sample code and solutions to the exercises in chapter 11._"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
" \n",
" ![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\") \n",
" | \n",
" \n",
" ![](\"https://kaggle.com/static/images/open-in-kaggle.svg\") \n",
" | \n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"# Setup"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This project requires Python 3.7 or above:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"\n",
"assert sys.version_info >= (3, 7)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It also requires Scikit-Learn ≥ 1.0.1:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import sklearn\n",
"\n",
"assert sklearn.__version__ >= \"1.0.1\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And TensorFlow ≥ 2.6:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"\n",
"assert tf.__version__ >= \"2.6.0\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we did in previous chapters, let's define the default font sizes to make the figures prettier:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"plt.rc('font', size=14)\n",
"plt.rc('axes', labelsize=14, titlesize=14)\n",
"plt.rc('legend', fontsize=14)\n",
"plt.rc('xtick', labelsize=10)\n",
"plt.rc('ytick', labelsize=10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And let's create the `images/deep` folder (if it doesn't already exist), and define the `save_fig()` function which is used through this notebook to save the figures in high-res for the book:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"\n",
"IMAGES_PATH = Path() / \"images\" / \"deep\"\n",
"IMAGES_PATH.mkdir(parents=True, exist_ok=True)\n",
"\n",
"def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
" path = IMAGES_PATH / f\"{fig_id}.{fig_extension}\"\n",
" if tight_layout:\n",
" plt.tight_layout()\n",
" plt.savefig(path, format=fig_extension, dpi=resolution)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Vanishing/Exploding Gradients Problem"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAEQCAYAAAD2/KAsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABNlUlEQVR4nO3de3yO9f/A8ddn56PzDHNYclw5HxIxpxWRYyo5ppIi9ZUvRSchfSUmyuEnEXKMnJVkkiijEWE5m83ZZrOT3ffn98c1y2xj7N6ue9v7+Xhcj+2+Pp/7+rx3ue2967o+B6W1RgghhMhrDmYHIIQQonCSBCSEEMIUkoCEEEKYQhKQEEIIU0gCEkIIYQpJQEIIIUxhkwSklJqrlLqglDqQRXkvpdT+1O03pVQdW7QrhBAi/7LVFdA8oN0dyk8AgVrr2sBYYLaN2hVCCJFPOdniIFrrX5RS/nco/+2Wl7uA8rZoVwghRP5lxjOgF4GNJrQrhBDCjtjkCii7lFKtMBLQY3eoMxAYCODu7t6gQoUKeRRd9litVhwcpO/G3ch5urszZ86gtaZixYpmh2L38vLzdMN6g9MJpwGo6F4RZwfnPGnXFuzx/114ePglrbVPZmV5loCUUrWBOUB7rfXlrOpprWeT+oyoYcOGOjQ0NI8izJ6QkBBatmxpdhh2T87T3bVs2ZLo6GjCwsLMDsXu5dXn6XzceR6Z8wjFkouxrf82Hir9UK63aUv2+P9OKXUqq7I8SZVKqYrASqCP1jo8L9oUQoh7VdKjJB2qduCH3j/ku+STH9nkCkgptRhoCZRSSkUAHwDOAFrrmcD7QEngS6UUQIrWuqEt2hZCiJy6mnCVxJREynqX5YsOX5gdTqFhq15wPe9S/hLwki3aEkIIW4pNiuXJb58kNimWsEFhODnk6aPxQk3OtBCi0Eq4kUDnJZ3ZfXY3K55ZIcknj8nZFkIUSsmWZHos70HIyRAWdF1AlxpdzA6p0JEEJIQolN7f+j7r/1nPzA4z6VW7l9nhFEr5PgFdu3aNCxcucOPGjTxpr2jRohw6dChP2srPCuN5cnZ2pnTp0hQpUsTsUEQ2DG86nId8HqJPnT5mh1Jo5esEdO3aNc6fP4+fnx/u7u6k9rDLVbGxsXh7e+d6O/ldYTtPWmsSEhI4e/YsgCQhO6W1Zv6++fR8uCelPEpJ8jGZfQ2ZvUcXLlzAz88PDw+PPEk+QmRFKYWHhwd+fn5cuHDB7HBEFt7f+j4vrH6Bb/Z9Y3YognyegG7cuIG7u7vZYQiRxt3dPc9uB4t7M3HHRMZtH8dL9V7ipfoyKsQe5OsEBMiVj7Ar8nm0T1/u/pKRP42k58M9mdlxpvw72Yl8n4CEEOJOriRc4b2t7/FUtaeY32U+jg6OZockUuXrTghCCHE3JdxLsGPADvyL+ePsmH9mti4M5ApICFEgbTq6if/9+j8AapSqgZuTm8kRidtJAhL5TkREBK+//jqPPvpoWg/IkydPmh2WsCO/nPqFrku7suTgEhJuJJgdjsiCJCCR7xw9epRly5ZRvHhxmjdvbnY4ws7sPrubjt92xL+YPz/2/hF3Z+kpa68kAYl8p0WLFpw/f54NGzbQo0cPs8MRduTAhQO0W9SOUh6l+KnPT/h4ZroQp7ATkoBEvmNvSw4L+xF2LgxPZ09+6vsTfkX8zA5H3IX0ghNC5HsWqwVHB0d61+5N1xpd8XTxNDskkQ3yp6QQIl87F3eOBrMbsOnoJgBJPvmIJKB87vXXX+epp57Kdv0pU6ZQu3ZtrFZrLkYlRN64knCFxxc8ztErRyniKhPA5jeSgPKxY8eOMWvWLD744INsv2fQoEFcuHCB+fPn52JkQuS+2KRY2i9qz5HLR1j93GqaVmhqdkjiHkkCyseCg4OpU6cODRs2zPZ73N3d6du3L5MmTcrFyITIXYkpiTy1+Cn2RO5heY/ltKncxuyQxH2QBGSHrl+/zsiRI6lSpQouLi4opdJtn332GUlJSSxcuJDnn38+3Xt37NiRof7N7dVXXwXgueee4++//+a3334z48cTIsecHZypWaomC7ouoFP1TmaHI+6T9IKzM1prunXrxo4dOxg9ejQNGzZk586djBkzBn9/f3r27MmTTz7Jrl27iI6OzjAQs0aNGuzcuTPdvk8++YSNGzfyzDPPAFC3bl2KFCnCpk2baNo089sWWmssFstd41VK4eiY95M7rlixAoA9e/YAsHHjRnx8fPDx8SEwMDDP4xF5I8WawqX4S5TxKsOMjjPMDkfklNY6xxswF7gAHMiiXAGfA0eB/UD97By3QYMG+k7+/vvvO5bnhmvXruXq8b/44gutlNI//vhjuv1du3bVpUqV0larVWut9SeffKKVUjopKemOx3vvvfe0m5ub3rBhQ7r9jz32mA4KCsryfVu3btXAXbfAwMBM35/b5+le48lL2f1cBgYG6jp16uRuMAXE1q1btcVq0f1W9dMVp1TU0QnRZodkl7Zu3Wp2CBkAoTqL3/G2ugKaB0wHslpmsD1QNXV7BJiR+lXc5uuvvyYoKIigoKB0+2vUqMGaNWvS1jGJjIykSJEiuLi4ZHmst99+m88//5y1a9fStm3bdGU+Pj6Eh4dn+d4GDRqwe/fuu8Zr1rLbxudaFBZaa4ZuHMr8ffMZ03IMRd2Kmh2SsAGbJCCt9S9KKf87VOkMfJOaDXcppYoppcpqraNs0f6tcn+dqez9wr2f34/nz58nNDSUKVOmZCiLioqiTJkyaa8TExNxdXXN8lhvvfUWs2bNYsOGDbRs2TJDubu7OwkJWU/S6OXlRd26de8asyzsJfLCnBNz+PbMtwx/dDjvtXjP7HCEjeTVMyA/4MwtryNS990xAR05ciTDL89nnnmG1157jfj4eM6dO5fJtCzVbRBuzsXFxXH27NkM+ytUqICHhwfXrl0jKir9j79//34AypYtS3R0NOfPnwfAYrGwbt06Hn/8cZKTk3FxccHDw4OrV69y5MiRdMeoXLkyw4YNY968ecyePZuyZcumq1OlShUcHR05d+4cXl5eGd5fvbpx/latWkX37t3v+nM2atSIBQsWAODo6EiVKlUAuHjxIpGRkenqOjs7U7lyZQBOnz6dIQG6urri7+8PwMmTJ0lKSkpX7u7uTsWKFQE4fvx4hqWvPT09KV++PGBMWHr7Myxvb2/KlSsHQHh4eIarqKJFi6Yl+dvPC0Dx4sUpXbo0FouFo0ePZigvWbIkpUqVwmKxZJr0X331VZ599lnOnDlDnz59CAsLIyUlJa3uW2+9xVNPPcWRI0d45ZVXMrz/3XffpW3btoSFhfHmm29mKP/4449p2rQpv/32G6NGjcpQHhwcTN26dfnpp58YN25chvJZs2ZRvXp11q5dy2effZahfMGCBVSoUIGlS5cyY0bG5y8rVqygVKlSzJs3j3nz5mUo37BhAx4eHnz55ZcsW7YsQ3lISAgAkyZNYt26dWn7o8pGEV4jnFcavMLEoImMGzeOLVu2pHtvyZIl+e677wB45513MjwHLV++PAsXLgTgzTffJCwsLF15tWrVmD17NgADBw7McHegbt26BAcHA9C7d28iIiLSlT/66KNMmDABgO7du3P58uV05W3atOG994zE2b59+wyf/Y4dOzJ8+HCATD87t/7ee/LJJzOU9+/fn/79+xMTE5Otz97tcvuzl5W8SkCZ/Zmc6TWCUmogMBCMX1jR0dHpysPDwwkJCSExMREXFxdSUlLSlZ85E0HRokWxWCyZJoBixYpRpEgRbty4kSEBAJQoUQIvLy+SkpLSEsCtihcvjre3N4mJiVy4cCFDuY+PD+7u7sTHJ2SIDYwebhaLhfj4+AzlHh4eAOzbt48WLVqklc+ePZuYmBi6d+9OXFwczs7O+Pv7c+PGDSIiItJ+aWqteeWVV1i5ciWLFi3iwQcfzNBGXFwcDg4OnDp1ioCAgAzlsbGxgJGIlixZkq7MwcGB0qVLAxATE0NCQgKenp5px9BaExsbS5EihWNA4IEDB9K9TkxMJDY2lqSkpAyfW4CDBw8SEhLChQsXiI6OJiUlBa11Wt2//voLb29vTp8+nen79+3bh5OTE0ePHs20fO/evSQnJ3PgwIFMy0NDQ4mOjmbfvn2Zlv/+++9ERUXx119/ZVq+c+dOjh07xsGDBzMt37FjB0WLFuXw4cOZlv/yyy+4ubkRHh6eafnNBHTs2LF05S7XXSiny/FMi2fYtm0bJ06cyPB+q9Wa9v7Mzp+zs3NaeURERIbyyMjItPLIyMgM5REREWnl58+fz1B++vTptPKLFy9y7dq1dOUnTpxIK79y5UqGP66OHTuWVp7Zubn1915m5YcPHyYkJIS4uLhsffZud/tnT2uwWt2wWt3R2p0ffzzD8eOhHD8exZkzNbBaXbFaXdHaDavVjU8/tVKixAnOnHHh4MGX0PpmuTOwLUN7Nylb3UtPvQW3Tmv9cCZls4AQrfXi1NdHgJZ3uwXXsGFDHRoammX5oUOHqFmzZo7ivlexsbG59txDGx0vCA8PZ+LEiTz44IOsWbOGGTNmMG3aNAYPHpxW9+TJkzzwwAN89913dOvWDYChQ4fy5ZdfMnXqVBo0aJDu2AEBAWmJITo6mhIlSjB79mxeeumlXPlZcvM82bvsfi5btmxJdHR0hr/GheHHYz/SvGJz3J3dCQkJyfQve5FeSEgILVq0JCYGrlwxtsuXja/R0XDtGsTGGtvN7zPbFxsLtpssRe3RWmc6WDGvroDWAEOUUkswOh/E5Mbzn/xOKcX333/PkCFDGDFiBFarlQYNGrB69eoM0+34+/vTuHFj1q5dS7du3dBaM3/+fCwWC0OGDMlw7CNHjqQloPXr1+Pi4kLXrl1z7Wc5e/YsX3zxBaGhoezbt4+EhAROnDiRdostv+jbt2/ardHo6GguXbpETEyMKV3PC5PlB5fz3HfP8XaztxnfZrzZ4ZjKYoFLl+DcOTh/3vh6c7t4MX2SOX++GXFxtkkebm7g6QkeHuDu/u/Xm9utrzMrc3MDV1fo1SvrNmySgJRSi4GWQCmlVATwAeAMoLWeCWwAnsTohh0PvGCLdguiihUrsmbNmmzVffXVV3njjTf44osv8PDwICYmJlvvW7hwIT169KBkyZI5CfWOjh8/zrJly2jQoAHNmzfnxx9/zLW2ctM33xgdO0+ePEnnzp2ZNWuWJJ9ctuGfDTy/8nmaVmjKqObZf56QH924AZGRcOoUnD6dfouKMpLMhQv3klCcAShSBEqUgJIlja8lSkDx4uDtbZR5e9/5e29vcHa+/59r2bJl+Pr6EhgYmPsJSGvd8y7lGhh8pzri3vXp04eJEyfy5Zdfpj3AvJuwsDC2bt2a4fmFrTVr1iztGdqcOXPuOwH5+/vTv39/PvzwQxtGZ6hfvz6nT5/OtOzPP/+kQoUKgHFLrUePHsyaNYtmzZrZPA7xr5CTIXRf1p06vnVY13NdgZjZ+soVCA+Hf/4xvh47ZiSYU6eM5JOd5FKyJJQp8+/m62t89fH5N8mULAmHDu2gY8dmOUoeOREbG8uLL77I8uXLGT58+F0HhctMCPmYo6Mjc+fOZe/evdl+z7lz5/j666/TeqvlFjMXjdNaM3fuXGbNmsWBAwcoUaIEPXr04OOPP8bd/d/lmbNz3vbs2UPfvn1ZtGhRtrqli/t3w3KDAasHULl4ZTb13pSvxvpYrUZC+esvYzty5N+kc+VK1u9TCvz8oGJFY6tUyfhaoYKx/2aSucNwv3TOnbthWvIJDQ2lU6dOXEn9ge80RvEmSUD5XJMmTWjSpEm267dr1y4Xo7EPL730EosWLWLYsGGMHz+e8PBwRo0aRWJiYqbdh7Oybds2hgwZwqpVq6hWrVouRiwAnB2dWf/8eoq6FaWURymzw8nStWvw55+wf/+/CefAAYiLy7y+lxdUrQrVqhlfq1QBf38j0fj5ZT+52Cur1cqECRMYP358uu7ldxqneJMkIGE3dBbzz1mt1nTdxe80/9w333zD3Llz0/UODAoKIjExkVGjRjF9+vRsP8N5+umncXNzS5tDz9HRkV27duFs1p+YBVT45XBWHVrFiGYjqOmTt71a7yY52Ug0f/wBu3cbXw8dynygua8v1KplbAEB/yacMmXyYoC8OaKioujevXtaR6ObHBwccHK6e3qRBCTsxrZt22jVqlWG/WPHjmXs2LFprwMDA9PGTNxu3LhxtGjRgk6dOqVLWgEBASQnJxMZGZn2bOduLl68eG8/gLhnp6JP0fabtiSmJNK/bn98vXxNjefyZfjlF2PbudO40klOTl/HxQXq1DG2WrXg4YeNrz4+5sRslrVr19K7d+9MxzQ6ODhk6w81SUDCbmQ2/1ynTp3o2LEjAwcOTNuX1fiiEydO8M8///DPP/9k+eEvWjT/PFco6KJio2jzTRtik2MJ6RdiSvK5cgV+/hlCQmDbNuNW2u1q1IDGjf/datc2uhcXVomJiQwdOpSFCxdmOZ2XXAGJfMfb2zvD4nouLi6UK1cuW4vu3Zz54uuvv+bhhzOMh8bBwaHQzNJg7y7FXyJoQRDn4s7xU9+fqFOmTp60a7EYt9I2bYIffjBuqd3aC83VFZo0gZYt4bHHoGFDKFYsT0LLF/7++286depEZGTkHeeSVEpJAhKFi5+fH2A8/LyXVWJF3tsVsYtTMadY23MtTcpnvxPN/YiNhY0b4fvvjcRz9eq/Zc7OEBgIrVoZXxs3NgZQiowOHjxIvXr1MszBmBm5BSdMl9eLxvn7+9OqVSveeOMNLly4QJ06dYiPj+fEiRNs3ryZlStXmto9XBgdTZRSdKzWkRNvnMi13m6XL8OaNbBqFfz4I9w69VrlytCunbG1amX0UhN3V6VKFd59910mTpyIxWIhMTExy7pKKUlAwlw9evRI9/q1114D7tyJICeUUixfvpwxY8YwdepUIiMjKVq0KDVq1ODpp5+W5GOypJQknlnxDC/UfYEuNbrYPPnExRlXOQsXwk8/GbfbwOiB1qwZdO0KnToZPdPEvXN1deX999/n9ddfZ/z48UyZMgVrFqNo5RacMJ0tJro9efLkPdUvWbIkn3/+OZ9//nmO2xa2k2JNoed3PVlzZA1dqnex3XFTYMsWI+msWgXXrxv7nZzg8cehWzfo3NnoCi1so3jx4jzyyCO4u7tz/eYJv41cAQkh7IJVW3lh9QusOryKqe2m8kK9nE8FeeoU/N//wVdfGfOl3dSsGfTuDc88Y0xPI2wvJSWFYcOGZUg+zs7OWK1WLBaLXAEJIcyntWbw+sEs3L+Qca3GMfSRofd9LIvF6EwwcyZs2PDvYNCqVaFPH2PW5dT1DkUu+uabb7h6a08OjMUgP/vsM9atW8eWLVtISkqSKyAhhPlcnVwZ2Wzkfc9sffUqLFpUkb594UzqusouLvD00zBokNFduqDONGBvkpKSePvttzNc/dSsWZOBAwfyyiuvEBYWxpgxY6iajYdtkoCEELkiOjGaYm7FmPLEFMB4LnAvTp2CKVNgzhy4ft24tHnwQXjlFejfv/DNPGAPZs2aRXx8fLp9Hh4eTJ06Ne3ft27duqxatSpbx5NuQUIImwveFcxDXz7EqehTKKXuKfmEhRm30x58EKZONToWNGx4hU2bjBmm//tfST5muH79Ou+//366qx+lFI0aNaJp06b3dcx8n4BstaS4ELYgn0f4au9X/OeH//Bo+UfxK+KX7ff9+Sc89RTUqwfffmvs69XL2P/pp/t54gmQnvTmCQ4OzjAI1c3NjeDg4Ps+Zr7+53R2dr7jdBBC5LWEhIRCPVv20gNLeXnty7Sr0o5F3Rbh5HD3u/x//WV0l65fH9atM5ZzfvNNY+G2hQtBlmEyX0xMDJ988km6228ODg60adMmR+tk5etnQKVLl+bs2bP4+fnh7u5+z/eYhbAVrTUJCQmcPXsWX19zZ3Q2S8jJEHqv6k3zSs357pnvcHW684ydhw/DBx/AsmXGazc3eO01GDkSSpfOg4BFtk2YMCHDjNcuLi5MmjQpR8fN1wno5sSSkZGR2ZqfyBYSExNxk8mi7qownidnZ2d8fX0L7YSnDco24LWGrzG29Vg8nD2yrHfpkpF4Zs0yula7uBgdC955B8qWzcOARbZcvHiRadOmpZt6x8nJia5du1K9evUcHTtfJyAwklBe/ocPCQmhXr16edZefiXnqfDYd24fD5Z4EG9Xb6a2n5plveRkmDYNxo6FmBjjec7AgfDee1C+fB4GLO7JmDFjMiwU6eTkxIQJE3J87Hz9DEgIYa6wc2EEzgvk1fWvZllHa2OanIAAGD7cSD5PPGGsNDprliQfexYREcHcuXNJumU2VxcXF/r27UulSpVyfPx8fwUkhDDH4UuHeXzB4xRxLcK4VuMyrXPsGLz6KmzebLyuWRM++wzat8/DQMV9GzVqVIZnP46Ojnz00Uc2Ob5cAQkh7tmJqydo+01bHJQDW/puoVKx9H8NJyfDhAnGctWbN0Px4sbtt337JPnkF0ePHmX58uXpnq+7ubnx2muv2ayjjU0SkFKqnVLqiFLqqFLq7UzKiyql1iql9imlDiqlcj4boRDCFFpreq/qTfyNeDb32UzVkumnXNmxw+hSPWoUJCYak4MePgxDhhgLwIn8YcSIERk6dzk6OjJ69GibtZHjW3BKKUfgCyAIiAB2K6XWaK3/vqXaYOBvrfVTSikf4IhSapHWOjmn7Qsh8pZSinmd5xGdGE0t31pp+69dgxEjjOc6YMxkMHMmtG1rUqDivl28eJHvv/8+3cBqd3d3RowYQfHixW3Wji2ugBoDR7XWx1MTyhKg8211NOCtjIE6XsAVIAUhRL4RkxhD8K5gtNZULVmVRn6N0sq2bYPatY3k4+QEo0cbA0wl+eRPPj4+rFy5kkqVKuHp6QkYnQ+GDRtm03ZskYD8gDO3vI5I3Xer6UBNIBL4C3hDa535UnpCCLtzPfk6Hb7twIjNIzhw4UDa/sREeOstY2nrU6eMW29//gnjxoG7u4kBixzr0qULx44dY+bMmZQrV45x48bhZeP1y1VO565SSvUAntBav5T6ug/QWGv9+i11ngaaAcOAB4HNQB2t9bVMjjcQGAjg6+vbYMmSJTmKz9bi4uJs/o9QEMl5urs333wTi8XCtGnTzA7ljpKtyYz6axR/Rv/J+wHvE+gTCMCRI15MmFCTU6c8cXDQ9Op1ir59T+HkZPv58OTzlD32eJ5atWq1R2vdMNNCrXWONuBR4IdbXr8DvHNbnfVA81te/4yRpO547AYNGmh7s3XrVrNDyBfkPN1dYGCgrlOnjtlh3FFySrLutLiT5kP0vD/naa21tli0Hj9eaycnrUHratW0/v333I1DPk/ZY4/nCQjVWfyOt8UtuN1AVaXUA0opF+A5YM1tdU4DbQCUUr5AdeC4DdoWQuSiPVF72PjPRqa3n06/uv24cAHatTOe8aSkwNChxi23xo3NjlTkRznuBae1TlFKDQF+AByBuVrrg0qpQanlM4GxwDyl1F+AAkZqrS/ltG0hRO5qUr4JR4Yc4YHiDxASAs8/D1FRUKoULFhgJCMh7pdNZkLQWm8ANty2b+Yt30cCj9uiLSFE7tJa8/ZPb1OvbD2ee/g5KhZ5gLFj4cMPwWqF5s1h8WLwy/5SPyIXtWzZkocffpjp06ebHco9k5kQhBDpjNk2hom/TWRXxC4uXjSuct5/35jTbfRo+Pnn/J98Ll68yGuvvYa/vz+urq74+vrSpk0bNt+cM+guQkJCUEpx6VLe3ciZN29eph0MVq5caZOJQc0gc8EJIdJ89ttnjNk2hv51+9PbZzING8Lp08YS2AsXwuMF5D5G9+7diY+P56uvvqJKlSpcuHCBbdu2cfny5TyPJTk5GRcXl/t+f4kSJWwYTd6SKyAhBACz98xm+Obh9AjoQdu4r2jR3IHTp6FJEwgLKzjJJzo6mu3bt/PJJ5/Qpk0bKlWqRKNGjRg+fDjPPfccAAsXLqRRo0Z4e3tTunRpevTowdmzZwE4efIkrVq1AowBm0op+vfvDxi3w4YMGZKuvf79+9OxY8e01y1btuTVV19l+PDh+Pj40KxZMwAmT55M7dq18fT0xM/Pj5deeono6GjAuOJ64YUXuH79OkoplFJ8+OGHmbbp7+/PuHHjeOWVVyhSpAjly5fn008/TRdTeHg4gYGBuLm5Ub16dTZs2ICXlxfz5s2zyTnOLklAQggATsecpv2DHan4x2J693YgIQEGDICQEChXzuzobMfLywsvLy/WrFmTbpG1WyUnJzNmzBj27dvHunXruHTpEj179gSgQoUKfPfddwAcPHiQqKgopk7Neh2kzCxcuBCtNdu3b+ebb74BjCWug4ODOXjwIN9++y1//PEHr79uDKds2rQpwcHBeHh4EBUVRVRUFMOHD8/y+FOmTKFWrVrs3buXkSNHMmLECHbu3AmA1Wqla9euODk5sWvXLubNm8eYMWPSLbmQV+QWnBCFXFJKEq5OrrxVbxzP/c/KZz864OQEwcHGEtkFbaV7Jycn5s2bx8svv8zs2bOpV68ezZo1o0ePHjzyyCMADBgwIK1+5cqVmTFjBjVr1iQiIoLy5cun3fYqXbo0pUqVuucYHnjgAT777LN0+95888207/39/Zk4cSKdO3dm/vz5uLi4ULRoUZRSlClT5q7Hf/zxx9Ouil5//XU+//xztmzZwqOPPsrmzZs5cuQIP/74I36pD/OmTJmSdiWWl+QKSIhCbPOxzVSdVpWNv//DI4/Ajz86UKoU/PQTDB5c8JLPTd27dycyMpK1a9fSvn17fvvtN5o0acLHH38MwN69e+ncuTOVKlXC29ubhg2NgfynT5+2SfsNGjTIsO/nn38mKCiI8uXL4+3tTbdu3UhOTubcuXP3fPzatWune12uXDkuXLgAwOHDhylXrlxa8gFo1KgRDg55nw4kAQlRSO04vYMuS7vgEtmK3h0e5J9/oE4dCA2FwECzo8t9bm5uBAUF8f777/Pbb7/x4osv8uGHHxITE8MTTzyBh4cHCxYsYPfu3WzatAkwbs3diYODQ7oZpIEMSxoAaRN83nTq1Ck6dOhAzZo1Wb58OXv27GHu3LnZajMzzrete6GUwmo1pt/UWqPs5C8LSUBCFEJ7o/by5LdPUvSfl4mYNo8rlx148knYvh1ssNJyvhQQEEBKSgphYWFcunSJjz/+mBYtWlCjRo20q4ebbvZas1gs6fb7+PgQFRWVbt++ffvu2nZoaCjJyclMmTKFRx99lGrVqhEZGZmhzdvbux81a9bk7Nmz6Y4fGhqalqDykiQgIQqZfy7/Q9A3j+Pw6yii5gWTlKR49VVYvRq8vc2OLvddvnyZ1q1bs3DhQvbv38+JEydYvnw5EydOpE2bNgQEBODq6sr06dM5fvw469ev57333kt3jEqVKqGUYv369Vy8eJG4uDgAWrduzcaNG1mzZg1Hjhxh2LBhnDlzJrMw0qlatSpWq5Xg4GBOnDjB4sWLCQ4OTlfH39+fxMRENm/ezKVLl4iPj7+vnz8oKIjq1avTr18/9u3bx65duxg2bBhOTk55fmUkCUiIQqa0ux8lflpB9PqRKAWTJsEXXxjr+BQGXl5eNGnShKlTpxIYGMhDDz3EqFGjeP7551m6dCk+Pj7Mnz+f77//noCAAMaMGcPkyZPTHcPPz48xY8YwevRofH190x74DxgwIG1r1qwZXl5edO3a9a4x1a5dm6lTpzJ58mQCAgKYM2cOkyZNSlenadOmDBo0iJ49e+Lj48PEiRPv6+d3cHBg1apVJCUl0bhxY/r168fo0aNRSuHm5nZfx7xvWc1Sag+bzIadf8l5uru8ng377LWzOvJytO7QwZjF2s1N6xUr8qz5HJHPU/bc73kKCwvTgA4NDbVtQPrOs2EXkr95hCjcLly/QOCMLlz8v7nEHC1KyZKwbp0xyFQUPqtWrcLT05OqVaty8uRJhg0bRp06dahfv36exiEJSIgC7mrCVVpN68Wx4Lno8w9Tvjxs3gw1apgdmTBLbGwsI0eO5MyZMxQvXpyWLVsyZcqUPH8GJAlIiAIsLjmO1lMG8ven/wfR/tSoAT/+CBUqmB2ZMFPfvn3p27ev2WFIAhKiIHtu2v8I+3g6XPelcWNYv95Yy0cIeyAJSIgCascO+GXMGLjuQFAQrFwJmczmL4RppBu2EAWMxWphxOyNPPGEJjbWgaefhrVrJfkI+yMJSIgCxKqtdPjocz4d3JLr1xV9+xqrl7q6mh2ZEBlJAhKigNBa0/nDufwwdjCkuPPKK/D114VngKnIfyQBCVFAdH9/GevG9QOrC0OHambMABMmOBYi2+TjKUQBMGnGBVaNfxqszowcqQkOVgV2KQVRcEgCEiKfW7AARgwuDdqRD8dYmTBBko/IH2ySgJRS7ZRSR5RSR5VSb2dRp6VSKkwpdVAptc0W7QpR2L368Q769dNoDePHwwfvO0jyEflGjh9PKqUcgS+AICAC2K2UWqO1/vuWOsWAL4F2WuvTSqnSOW1XiMLuP5/uZua7TUArxo61MmqU3NAQ+YstPrGNgaNa6+Na62RgCdD5tjrPAyu11qcBtNYXEELct3eC9xE8sj5oR979IIl335XkI/IfW3xq/YBbV1yKSN13q2pAcaVUiFJqj1LK/EmIhMinPpx+iE+GPQzakbffTWDshzLIR+RPthghkNkdZ33bayegAdAGcAd2KqV2aa3DMxxMqYHAQABfX19CQkJsEKLtxMXF2V1M9kjO091FR0djsVju6TyFhPjw0dgaoB15ptdhnmhzjsJwmuXzlD357TzZIgFFALfOrVseiMykziWt9XXgulLqF6AOkCEBaa1nA7MBGjZsqFu2bGmDEG0nJCQEe4vJHsl5urtixYoRHR2d7fO0bLmVceMc0FYY8U4yn4yvgVKFY00F+TxlT347T7a4BbcbqKqUekAp5QI8B6y5rc5qoLlSykkp5QE8AhyyQdtCFAozF0by7HMWLBYYNQo+Ge8ivd1EvpfjKyCtdYpSagjwA+AIzNVaH1RKDUotn6m1PqSU2gTsB6zAHK31gZy2LURhsHDVBV7tXxKszgwYcolx40pJ8hEFgk1midJabwA23LZv5m2vPwU+tUV7QhQWq3+4Qt9nvcHiyrMDLjDn89KSfESBIX03hbBTP/1yjW5dXNA33On47Hm+/T9JPqJgkXlyhbBD+/fDM128sSYqWnY8x/eLysjEoqLAkY+0EHYm7K9E2rS1cvWqonNn+HFlGRwdzY5KCNuTBCSEHTlyNJkmgXFcuuhAUJCVpUvB2dnsqITIHZKAhLATJ0+n0KDZVZKulqJqvSi+/95BVjIVBZokICHsQNQ5K3WaXuT6BV8qBkQRGlIWDw+zoxIid0kCEsJkV65AvWYXuXa2LL6Vz/Hn9rIUKWJ2VELkPklAQpjIYvGkXTs4f9wXn4pX2PebLyVKmB2VEHlDumELYRKLxY1/IiaSFA0PPADbt5fA19fsqITIO3IFJIQJEhNhz9F3SYpuTDGfOLZsAb/bFzERooCTBCREHrtxAx5td5qEc4+h3C+wPcSVBx4wOyoh8p4kICHykMUCLTudJmxbRZTrVR6s9CoPB8hAH1E4yTMgIfKI1Qr9BiTx26aKOLpd5+GHRkDKMbPDEsI0cgUkRB7QGoYOhUXfuOLmbmHdOk0xr38y1Dt48CDTpk1jw4YNhIeHk5ycbEK0QuQNuQISIpdpDX0HR7JwRjlcXWHtGkfatvHik7EZ64aHhzN06FCKFCmC1WolISGB4sWL4+/vT0BAAA8//DBVqlShSpUqVK5cGU9Pz7z/gYSwEUlAQuSywSPPsXBGOXC4waIlFtq2dcuybpcuXXj44Yc5cODf9RovXbrEpUuXCA0NxdnZGXd3d7TWJCQk4OnpSYUKFXj55ZcZOnRoXvw4QtiM3IITIhe9/dFFZnxaBpSF6V9dpXuXrJMPgFKKWbNm4ZHFPDw3btzg2rVrxMbGkpKSQkxMDOHh4ShZKEjkQ5KAhMgl4z67zP8+8AFgwufnGdy/dLbe17RpU1q0aIFjNtZgcHR0pGHDhgwePDhHsQphBklAQuSC+fPhveElARj1SQRvDyl3T++fNm0aztlYh8Hd3Z2lS5fiIKvViXxIPrVC2NiSJZoBA4zvP/w4jvEjy9/zMapUqUK/fv1wvct6DBaLhe+++w6r1Xo/oQphKklAQtjQ0pXxPN/bgtUKH30EH7zjdd/HGj9+PE5Od+4nlJCQwKhRo2jWrBlnzpy577aEMIMkICFsZO3GRHo+64S2ONH95aO8+27OjleyZElGjx6dZYeEm+Lj49m9ezcBAQHMnz8frXXOGhYij9gkASml2imljiiljiql3r5DvUZKKYtS6mlbtCuEvfhpazJduoBOcaHd8+Esn1UFW3RM+89//pNhrI+bmxtubul701ksFuLi4hg8eDDt27fn4sWLOW9ciFyW4wSklHIEvgDaAwFAT6VUQBb1/gf8kNM2hbAnIdtSaNfegjXZjcc6h7N+QTWbJB8wks2UKVPSkpCrqysdO3akV69euLu7Z6h//fp1fv75Z6pWrcrq1attE4QQucQWV0CNgaNa6+Na62RgCdA5k3qvA98BF2zQphB24ddfoWMHRyxJ7jR68m9CvquGrTuk9ezZkwoVKgBQtGhRvvrqK+bMmcOqVasoUaIELi4u6erfuHGDmJgYnn/+eXr27Mm1a9dsG5AQNmKL/yp+wK1PPyNS96VRSvkBXYGZNmhPCLvw66+a9u01168r+vTR7FwTQDaG7twzBwcHZs6ciVKKZcuWUSR1ve4nnniCo0eP0qFDh0yfE8XHx7Nq1SqqVKlCSEiI7QMTIodsMRVPZjcbbn8KGgyM1Fpb7jZiWyk1EBgI4Ovra3f/ceLi4uwuJntU0M/TX395M+y/AaQkuRPYOoJ+/Y6yffu9HSM6OhqLxZLt87R06VK01hnqDx06lFq1ajFp0iSSkpKwWCxpZUlJSVy8eJF27drxxBNP8Nprr921a7c9KuifJ1vJd+dJa52jDXgU+OGW1+8A79xW5wRwMnWLw7gN1+Vux27QoIG2N1u3bjU7hHyhIJ+nnTu1dvVI1KB11cDf9Y0b1vs6TmBgoK5Tp47N4oqKitKtWrXSnp6eGuOPwHSbu7u7rlChgt69e7fN2swrBfnzZEv2eJ6AUJ3F73hb3ILbDVRVSj2glHIBngPW3JbkHtBa+2ut/YEVwGta6+9t0LYQeer336FV2ySS4l3xb76TA5sb4uRkH/OwlSlThi1btqR1Wrh9doSEhATOnDlDixYtGDVqFDdu3DApUiEMOU5AWusUYAhG77ZDwDKt9UGl1CCl1KCcHl8Ie/HHH9CqbTKJ113xe/RXDv3UCBdn+xpKp5Ti5Zdf5q+//qJu3bqZLteQkJDA1KlTqV27NocOHTIhSiEMNvnfo7XeoLWuprV+UGs9PnXfTK11hk4HWuv+WusVtmhXiLyyYwcEBUFCnAtVHvuTIz83ws3FflczeeCBB/jjjz947733cHd3zzBbdnx8PEeOHKFBgwZMmjRJpvIRprCvP99EtrVs2ZIhQ4aYHUahsHUrBD1u4do1eOYZ+Pvneni62f+DfEdHR0aOHMnu3bupWrVqhp5yOnVNoQ8++IAmTZpw6tQpkyIVhVWhSkAXL17ktddew9/fH1dXV3x9fWnTpg2bN2/O1vtDQkJo1aoVly5dyuVI/zVv3jy8vDLOJ7Zy5UomTJiQZ3EUVps2Qbv2FhLiHXk4aA/ffgvZmKTarjz00EP89ddfvP7665kOXo2Pj2fv3r089NBDfPXVVzKVj8gzhSoBde/enT/++IOvvvqK8PBw1q1bR/v27bl8+XKex5KcnJyj95coUQJvb28bRSMys3o1dOpsJTnJkeKPLePnlf65Ms4nL7i4uPDJJ5+wdetW/Pz8MiQii8XC9evXGTp0KEFBQZw/f96kSEWhklX3OHvYbNkN++rVqxrQmzdvzrLOggULdMOGDbWXl5f28fHRTz/9tI6IiNBaa33ixIkM3Vr79euntTa60w4ePDjdsfr166c7dOiQ9jowMFAPGjRIv/XWW7pUqVK6YcOGWmutP/vsM12rVi3t4eGhy5Urp1988UV99epVrbXRpfL2Nj/44INM26xUqZIeO3asHjhwoPb29tZ+fn564sSJ6WI6cuSIbtGihXZ1ddXVqlXT69ev156envrrr7++n1N6R/bYHfReLF2qtZOTVYPWRQO/1mdjIm3ehq27YWfX9evX9csvv6w9PDwy7a7t7OysixQpolesWJHnsWUlv3+e8oo9nidyuRt2vuDl5YWXlxdr1qwhMTEx0zrJycmMGTOGffv2sW7dOi5dukTPnj0BqFChAt999x0ABw8eJCoqiqlTp95TDAsXLkRrzfbt2/nmm28AY5R7cHAwBw8e5Ntvv+WPP/7g9ddfB4yVMYODg/Hw8CAqKoqoqCiGDx+e5fGnTJlCrVq12Lt3LyNHjmTEiBHs3LkTAKvVSteuXXFycmLXrl3MmzePMWPGkJSUdE8/Q2Ewdy707KlJSVF4t5lO2KpWlCtS1uywbMbDw4PZs2ezZs0aSpUqlWFg6s1lv/v27cvTTz9NdHS0OYGKgi+rzGQPm60Hoq5YsUIXL15cu7q66iZNmui33npL79q1K8v6hw4d0oA+c+aM1vrfK5KLFy+mq5fdK6BatWrdNcaNGzdqFxcXbbFYtNZaf/3119rT0zNDvcyugJ577rl0dapUqaLHjh2rtdZ606ZN2tHRMe2KTmutd+zYoQG5AkpltWr9v/9pDcb2yvAIffjikVxrz6wroFtdvXpV9+jRI8urIVdXV12yZMk73jnIC/nx82QGezxPyBWQoXv37kRGRrJ27Vrat2/Pb7/9RpMmTfj4448B2Lt3L507d6ZSpUp4e3vTsGFDAE6fPm2T9hs0aJBh388//0xQUBDly5fH29ubbt26kZyczLlz5+75+LVr1073uly5cly4YMz9evjwYcqVK4ef37/T9DVq1EiWck5ltcJ//wsjR4JSmunTYeanflQvVc3s0HJVsWLFWLZsGQsWLKBIkSIZlgFPSkri8uXLdOrUiYEDBxIfH29SpKIgKnS/fdzc3AgKCuL999/nt99+48UXX+TDDz8kJiaGJ554Ag8PDxYsWMDu3bvZtGkTcPcOAw4ODhl6DmU2yvz2QYGnTp2iQ4cO1KxZk+XLl7Nnzx7mzp2brTYzc/svD6VU2vgOrXWGsSDCcOMGDBgAn30GON7A5ZkX6NLnrNlh5alu3boRHh5OYGBgphObJiQksGDBAqpVq8bvv/9uQoSiICp0Ceh2AQEBpKSkEBYWxqVLl/j4449p0aIFNWrUSLt6uOnmtPe3TvYI4OPjQ1RUVLp9+/btu2vboaGhJCcnM2XKFB599FGqVatGZGRkhjZvb+9+1KxZk7Nnz6Y7fmhoaKEfgJiQAN26wfz54OCSgHPvrvzwyQv4FfG7+5sLGF9fX3788UemTZuGp6cnjrd1+UtMTOTs2bO0aNGCrVu3mhSlKEgKTQK6fPkyrVu3ZuHChezfv58TJ06wfPlyJk6cSJs2bQgICMDV1ZXp06dz/Phx1q9fz3vvvZfuGJUqVUIpxfr167l48SJxcXEAtG7dmo0bN7JmzRqOHDnCsGHDOHPmTGZhpFO1alWsVivBwcGcOHGCxYsXExwcnK6Ov78/iYmJbN68mUuXLt33LZCgoCCqV69Ov3792LdvH7t27WLYsGE4OTkV2iujy5fh8cdh3Tpw8oxB9WvL6tGDCfQPNDs00yilGDBgAAcPHqRBgwaZTuXj5eVF3bp18z44UeAUmgTk5eVFkyZNmDp1KoGBgTz00EOMGjWK559/nqVLl+Lj48P8+fP5/vvvCQgIYMyYMUyePDndMfz8/Ojfvz+jR4/G19c3bSaCAQMGpG3NmjXDy8uLrl273jWm2rVrM3XqVCZPnkxAQABz5sxh0qRJ6eo0bdqUQYMG0bNnT3x8fJg4ceJ9/fwODg6sWrWKpKQkGjduTL9+/Rg9ejRKqQzLOxcGR4/Co48aC8qV9E1Av9CcJW/+h/ZV25sdml2oVKkSO3fuZMyYMXh4eKT9keLh4cG3335L8eLFTY5QFAhZ9U6wh02WY8hdYWFhGtChoaE2P7Y9n6cdO7QuVcro6VanjtYREVofu3Isz+Owh15w2XHo0CFds2ZN7ezsrF944QVTYrDnz5M9scfzxB16wdnvbIrC5latWoWnpydVq1bl5MmTDBs2jDp16lC/fn2zQ8szy5dDnz6QlATl6//F2K8v4+fXEqhsdmh2q0aNGuzbt4/FixfTvXt3s8MRBUihuQUnIDY2liFDhhAQEECvXr2oWbMmP/zwQ6F4BqQ1TJxoTCaalAQ124UQ0aEeh2L/MDu0fMHZ2Zm+fftm+kxIiPslV0CFSN++fenbt6/ZYeS5pCR47TVjhgOA5i+uZXv5TrzT/B1GNBthbnBCFGJyBSQKtMhICAw0ko+bGzz9wTK2V+jE64+8zvjW480Or9Dx9/fP0NFGFF5yBSQKrF27jDE+UVFQsSKsXKmZHrGR/vQnuF1wobj1aIb+/ftz6dIl1q1bl6Fs9+7dchtPpJEEJAqkuXPh1VchORlatICFi5OoUM6Vr+ob6904KLn4N4OPj4/ZIQDGTCM3B5YL8xS4/4WRkZFZznYtCr7kZHj9dXjxReP7wYNhwORvaf3dw5yJOYODcsDRIZ8u6lMA3H4LTinF7Nmz6dGjB56enlSuXJmFCxeme8/Zs2f56KOPKF68OMWLF6dDhw78888/aeXHjh2jc+fOlClTBk9PT+rXr5/h6svf358PP/yQAQMGUKxYMXr16pW7P6jIlgKVgFJSUqhbty4+Pj7897//zdZsBKLgOHECHnsMpk8HFxeYMweChqzmxfV98fP2o5RHKbNDFJn46KOP6Ny5M/v27ePZZ59lwIABacuDx8fH06pVK1xcXNi2bRs7d+6kbNmytG3bNm1WkLi4ONq3b8/mzZvZt28f3bt3p1u3bhw+fDhdO5MnT6ZGjRqEhoamTUAszFWgEtDq1atJTEwkLi6Ozz//nKpVqxbKXl+F0apVUK8e7N4NlSrBL79AxZabeWbFMzQo14C1Pdfi7pxxOWphvj59+tC7d2+qVKnC2LFjcXJyYvv27QAsWbIErTUjR46kdu3a1KhRg1mzZhEXF5d2lVOnTh0GDRpErVq1qFKlCqNHj6Z+/fqsWLEiXTuBgYGMGDGCKlWqULVq1Tz/OUVGBSoBjR8/ntjYWMC4x5uUlJRhck9RsCQnw5tvGp0NYmKgc2f4809Q5f+gy9Iu1ChVg429NuLtKsuX26tblxFxcnLCx8cnbSLgPXv2cOLECZ588sm0RSWLFi3K1atXOXbsGADXr19nxIgRBAQEULx4cby8vAgNDc2wjMrN5VWE/bBJJwSlVDtgKuAIzNFaf3JbeS9gZOrLOOBVrfXdp4u+B/v37+fIkSPp9nl5efHOO+/YshlhR44fh2efhdBQcHY2Bpq+8QYoBVXcqtC5ememPDGFEu4lzA5V3MGdlhGxWq3UrVuX//znPzzyyCPp6pUoYfy7Dh8+nE2bNjFp0iSqVq2Kh4cHffv2zbCkifS+sz85TkBKKUfgCyAIiAB2K6XWaK3/vqXaCSBQa31VKdUemA08kvFo92/ixIkZOh+UKFGC1q1b27IZYQe0Np7v/Oc/cP06+PvD0qXQuDEcv3ocP28/SriX4Nvu35odqsih+vXrs3jxYooWLUqVKlUyrfPrr7/St2/ftGmCEhMTOXbsGNWqFezFBAsCW1wBNQaOaq2PAyillgCdgbQEpLX+7Zb6u4DyNmg3zeXLl/nuu+/SrW3j6enJyJEjZaxHARMVBS+9BBs2GK+feQZmzoTixeHYlWM0/7o5bSu35Zuu35gbaCF37do1wsLC0u0rVqzYPR+nV69eTJo0idGjR+Pt7U3FihU5c+YMq1evZtCgQVStWpVq1aqxatUqOnfujLOzM2PGjJGesPmELRKQH3Brd7MI7nx18yKw0Qbtppk9e3aGRKO1lg4IBczy5TBoEFy5AsWKwYwZ8NxzRlnEtQjaLmhLkiWJkc1G3vE4Ivdt376devXqpdt3PxOZenh48Msvv9C/f3969OhBTEwM5cqVo1WrVmlLQkyePJkXX3yR5s2bU7x4cd58801JQPmE0rctJX3PB1CqB/CE1vql1Nd9gMZa69czqdsK+BJ4TGt9OYvjDQQGAvj6+jZYsmTJHdu3WCx0796dmJiYtH2Ojo489dRTvPHGG/f5U2UtLi4OLy8vmx+3oLHleYqJcWbatCps2eILQKNGV/jvfw/j42Pc47+afJU3wt7gcvJlJteZTHXv6jZpN7e9+eabWCwWpk2bZnYodk/+32WPPZ6nVq1a7dFaZ94DJKt1GrK7AY8CP9zy+h3gnUzq1QaOAdWye+zsrAe0cuVK7e3trYG0zc3NTR87ljvru9jjehv2yBbnyWrV+uuvtS5Z0li7x8ND6xkzjP23aj2/tXYf565/OflLjtvMS/llPSB7IP/vsscezxO5vB7QbqCqUuoB4CzwHPD8rRWUUhWBlUAfrXW4DdpM8/HHH6d1vb6pSZMmVK4s67vkZ0eOGLfbQkKM123aGM96MnsOPeWJKZyPO0/zSs3zNEYhRM7keByQ1joFGAL8ABwClmmtDyqlBimlBqVWex8oCXyplApTSoXmtF2AAwcOcPDgwXT7vLy8GDVqlC0OL0yQlAQffQS1axvJp1Qp+OYb2Lw5ffJJTElkwb4FaK2p7VuboAeDTItZCHF/bDIOSGu9Adhw276Zt3z/EvCSLdq61cSJEzP09S9WrBht27a1dVMil2kNa9bAf/8LN6f5GjDAGNtTsmT6ujcsN+ixvAfrw9fzUOmHqF+28KzoKkRBkm9nw7569SrLly/HYrGk7fPw8GDEiBHS9Tqf2bfPGNOzdavxukYN43ZbYGDGuharhT6r+rAufB0zOsyQ5CNEPpZvp+L5v//7PxwcMobfv3//vA9G3Jdz5+Dll4053LZuNcbyTJ0K+/dnnnys2sor615h6cGlTGw7kUENB2WsJITIN/LlFZDFYmHSpElps+GCMYdU37598faWOb/sXXQ0TJkCkydDXBw4ORnLJrz/PpS4w6w5u8/u5uuwr3mvxXv8t9l/8yxeIUTuyJcJaP369RkGmjk5OTFs2DCTIhLZERtrXOF89pmRhAA6doRJk6B6NobuPFL+EUJfDqVumbq5GaYQIo/ky1twmXW9btSokUyxbqeuX4f//Q8eeADee89IPi1bGksmrF179+QzZecU1oevB6Be2XryjE+IAiLfJaC///6b/fv3p9snXa/t08WLMGaMMVno22/D5cvQrBn8/LPxzKd5NobtzNg9g2E/DmPJwTvPiCGEyH/y3S24SZMmZeh6XaRIER5//HGTIhK3O3oUpkypyubNkJBg7HvkEWN8T1CQsVxCdizYt4DXNrxGx2odmdtpbu4FLIQwRb5KQDExMSxZsiTTrteZ9YgTeUdr2LHDeMazciVYrX4APPmkMbYnMDD7iQdg5aGV9F/dn9YPtGZ5j+U4Ozrf/U1CiHzFrn9rR0RE8MUXXxAXFwfAnDlzMp31+oUXXjAjPIGxCun06VCrlnFLbcUKcHSEdu2iOHAA1q83nvfc62ObbSe30divMaufW42bk1uuxC6EMJddXwFdvXqVt956ixEjRtCvXz9WrlyZoet1r169KFKkiIlRFj5aG6uQzpoFixfDzX8SX1948UWjS3V4+BEeeqjsPR/bqq04KAeC2wUTfyMeTxdZxVKIgsquE5CjoyMJqQ8R5syZg5NT+nCdnZ0ZPny4GaEVSqdOwaJFsHAhHDr07/42bYyJQzt1AhcXY1/4fUw5GxoZSv/v+7Py2ZVUK1lNko8QBZzdJ6Cbbty4wY0bN9KV+/r6Uq5cubwOq1C5etW4rbZwodFt+iYfH+jdG155JXtjeO7mwIUDPLHwCYq6FsXTWRKPEIVBvklAmblw4QJlypShd+/eDB8+XMYB2UhkJKxebXQm2LoVbvb5cHODLl2gTx+jN5uzjfoF/HP5H4IWBOHq6MpPfX/Cr4ifbQ4shLBrdp2Abr/ldrubz4PmzJnD3LlzuXjx4n2tO1/YaQ1//w0bNsCqVbBz579ljo7GLbbevaFbN7D147abS2mnWFPY1n8blYvLOk5CFBZ2nYCcs/kntpubG4sXL5bkcw+io+Gnn2DTJvjhB4iI+LfMzQ0ef9xIOB07ZlwOwZa8XbypVboWH7X6iACfgNxrSAhhd+w6ATk6OuLk5ERKSkqm5U5OThQtWpSffvqJunXr5m1w+cy1a8Y4nW3bjG337n9vrYHRg+3xx42OBO3aQW4vK3814SquTq4UdSvKuufX5W5jQgi7ZPcJyMXFJdME5Obmhr+/P1u2bJGOCLfR2rii+eMP43batm2wdy9Yrf/WcXIyBoe2awdPPAF16kBejeWNTYql3aJ2FHMrxqZem2RuNyEKKbtPQJl1RPDw8KBFixasWLECT0/pMXX1qjEu548//t3OnUtfx8kJGjc2kk5goDEnmxnDpxJuJPDU4qfYE7mHlc+ulOQjRCFm9wno9il2PDw8GDhwIJ999lmhm34nJcVYrvqvv4ztwAHj67FjGesWL24knEaNoEULePTR3L+tdjfJlmS6L+vOL6d+YVG3RXSq3sncgIQQprLrBHR7Lzh3d3eCg4N5+eWXTYoob8TFGYnmn3+MAZ3h4UaiOXQIkpIy1nd1hfr1jYRzc3vwwXuf/ia3Dd04lI1HNzK742x61uppdjhCCJPZdQJydHTEarWilMLLy4s1a9bQsmVLs8PKsZQUY6zN6dPGduoUHD/+b8KJisr6vZUqGfOu3bpVr267MTm56c0mb1KvTD1eblCw/4AQQmSP3Seg2NhYypUrx9atW6lWrZrZId1VfLzx/OXWLSLCSDI3E87Zs+l7oN3OxQWqVIGqVf/dHn7Y2PLbtHdaa9aFr6NjtY7UKFWDGqVqmB2SEMJO2HUCcnJy4u2332b48OGUzM3BKFmwWIzZni9fhitXjO3XX33Zv9/Yd/kynD//b6I5f95Ydjo7ypY1rmYqVjQ2f3+oVs1INhUqGANAC4LRP49mwq8TWPXsKrrU6GJ2OEIIO2KTBKSUagdMBRyBOVrrT24rV6nlTwLxQH+t9d5sHJcJEybcUyxaG89JEhKMq5HYWGO7di3z729/HRPzb7K5etU4Xno179i+qyuUKWOMq7n5tUKFfxNNpUrg52fUK+gmbJ/AhF8n8EqDV+hcvbPZ4Qgh7EyOE5BSyhH4AggCIoDdSqk1Wuu/b6nWHqiauj0CzEj9ekdnz8LQoUYyyc4WHw+JiZkljftXtKgxE0CJEsaWknKemjV9KVHC2H8z0dzciha1v4f/Zlh1dhWfH/2cXrV68WWHL6W7tRAiA1tcATUGjmqtjwMopZYAnYFbE1Bn4ButtQZ2KaWKKaXKaq3v8LjduK01bdq9B+TsDO7u4OEB3t7GcxNv7+x9X6QIacmlWDFj/MytQkIO0bKl770HVYicij7Fl8e+pHP1znzd+WscVOHqLi+EyB5bJCA/4MwtryPIeHWTWR0/4I4JyMXlIhUqTMPBIQlHxyQcHJJxcEhK2xwdM3+tlDXDsbQ2brVdu3ZfP2Oa6OhomXMuGyo5VuLK1isEzQwyOxS7FRYWRkpKSoHo2Znb5P9d9uS382SLBJTZvZXbb4Jlp45RUamBwEAwJiP18voq00a1Nroz5zWLxUJ0dHTeN5wPXPO9htXRSrHIYrhb3LnmmMNsX8ClpKSgtZbPUzbI/7vsyW/nyRYJKAKocMvr8kDkfdQBQGs9G5gN0LBhQx0aGmqDEG0nJCRE/mLNRMjJENovak+t0rXY+eJOtv+yXc7TXbRs2ZLo6GjCwsLMDsXuyf+77LHH83Sn57+2uDm/G6iqlHpAKeUCPAesua3OGqCvMjQBYu72/EfkH3+c/YOnFj9F5eKV2dBrA44OBaQPuRAiV+X4CkhrnaKUGgL8gNENe67W+qBSalBq+UxgA0YX7KMY3bBfyGm7wj7sP7+fdgvbUdqzNJv7bKaURymzQxJC5BM2GQektd6AkWRu3Tfzlu81MNgWbQn7svrwajycPdjSdwvlvGVZDCFE9kn/WHFfdOpgq3dbvMufr/yJfzF/cwMSQuQ7koDEPYuKjSJwXiAHLhxAKYWPp4/ZIQkh8iG7ngtO2J/L8ZcJWhDEyeiTxCXHmR2OECIfkwQksu1a0jXaLWrH0StH2dBrA03KNzE7JCFEPiYJSGRL/I14On7bkbBzYXz/7Pe0fqC12SEJIfI5eQYkssWqrbg7u7Oo2yI6VOtgdjhCiAJAroDEHaVYU0hMScTLxYtNvTbJrNZCCJuRKyCRJau2MmD1AIIWBJFsSZbkI4SwKUlAIlNaa4ZsGMKC/QvoWLUjLo4uZockhChgJAGJDLTWvP3T28wIncGIpiMY1XyU2SEJIQogSUAig6m/T2XibxN5teGrfNL2E7n1JoTIFdIJQWTQqXonomKjmNB2giQfIUSukSsgkWZXxC601lQuXpn/Bf1PltIWQuQq+Q0jAFh6YClNv2rK9D+mmx2KEKKQkAQkWBe+jt6revNYxcd4sf6LZocjhCgkJAEVcj+f+Jmnlz1N3TJ1Wff8OjycPcwOSQhRSEgCKsRik2LpsbwHVUpUYVOvTRRxLWJ2SEKIQkR6wRVi3q7erOixghqlalDSo6TZ4QghChm5AiqEDl86zNIDSwFo9UArynqXNTkiIURhJFdAhcyJqydo+01brNpKh2od8HLxMjskIUQhJQmoEImMjaTtgrbE34hnW/9tknyEEKaSBFRIXIq/RNCCIC5cv8CWvluo5VvL7JCEEIVcjp4BKaVKKKU2K6X+Sf1aPJM6FZRSW5VSh5RSB5VSb+SkTXF/Vvy9guNXj7Ou5zoa+zU2OxwhhMhxJ4S3gS1a66rAltTXt0sB3tJa1wSaAIOVUgE5bFfco0ENB3HwtYME+geaHYoQQgA5T0Cdgfmp388HutxeQWsdpbXem/p9LHAI8MthuyIbElMSef6759kbtReAysUrmxyREEL8K6cJyFdrHQVGogFK36myUsofqAf8nsN2xV3csNzguRXPsfjAYg5eOGh2OEIIkYHSWt+5glI/AWUyKRoNzNdaF7ul7lWtdYbnQKllXsA2YLzWeuUd2hsIDEx9WR04cscA814p4JLZQeQDcp6yR85T9sh5yh57PE+VtNY+mRXcNQHdiVLqCNBSax2llCoLhGitq2dSzxlYB/ygtZ583w3aAaVUqNa6odlx2Ds5T9kj5yl75DxlT347Tzm9BbcG6Jf6fT9g9e0VlLGi2VfAofyefIQQQthOThPQJ0CQUuofICj1NUqpckqpDal1mgF9gNZKqbDU7ckctiuEECKfy9FAVK31ZaBNJvsjgSdTv/8VKEjrOs82O4B8Qs5T9sh5yh45T9mTr85Tjp4BCSGEEPdLZsMWQghhCklA90kpNVwppZVSpcyOxR4ppT5VSh1WSu1XSq1SShUzOyZ7opRqp5Q6opQ6qpTKbAaRQk+m8bo3SilHpdSfSql1ZseSXZKA7oNSqgJGp4vTZsdixzYDD2utawPhwDsmx2M3lFKOwBdAeyAA6CnTU2VKpvG6N29gzDSTb0gCuj9TgBGAPEDLgtb6R611SurLXUB5M+OxM42Bo1rr41rrZGAJxrRW4hYyjVf2KaXKAx2AOWbHci8kAd0jpVQn4KzWep/ZseQjA4CNZgdhR/yAM7e8jkB+sd6RTON1V8EYfxRbTY7jnsh6QJm4y/RDo4DH8zYi+3Sn86S1Xp1aZzTGrZRFeRmbnctsWIJcTWchdRqv74A3tdbXzI7H3iilOgIXtNZ7lFItTQ7nnkgCyoTWum1m+5VStYAHgH3GBA+UB/YqpRprrc/lYYh2IavzdJNSqh/QEWijpb//rSKACre8Lg9EmhSLXUudxus7YNGd5pAs5JoBnVIH+LsBRZRSC7XWvU2O665kHFAOKKVOAg211vY2+Z/plFLtgMlAoNb6otnx2BOllBNGx4w2wFlgN/C81lqmLb9F6jRe84ErWus3TQ4nX0i9Ahqute5ocijZIs+ARG6ZDngDm1OnX5ppdkD2IrVzxhDgB4wH68sk+WRKpvEq4OQKSAghhCnkCkgIIYQpJAEJIYQwhSQgIYQQppAEJIQQwhSSgIQQQphCEpAQQghTSAISQghhCklAQgghTCEJSIg8ppQakbqY4e3bR2bHJkRekpkQhMhjSilvwPOWXcOBXkBzrfVRc6ISIu9JAhLCREqpkcBQoLXW+ojZ8QiRl2Q5BiFMopR6B2NS0lZa63Cz4xEir0kCEsIEqQv1DcJYrkJuu4lCSRKQEHlMKfUe8DLQUmt9zOx4hDCLJCAh8lDqlc8bQCfgulLq5pLm0VrrRPMiEyLvSScEIfJI6gqf0UCRTIrbaq235G1EQphLEpAQQghTyEBUIYQQppAEJIQQwhSSgIQQQphCEpAQQghTSAISQghhCklAQgghTCEJSAghhCkkAQkhhDCFJCAhhBCm+H+7xj5SCcm/twAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# extra code – this cell generates and saves Figure 11–1\n",
"\n",
"import numpy as np\n",
"\n",
"def sigmoid(z):\n",
" return 1 / (1 + np.exp(-z))\n",
"\n",
"z = np.linspace(-5, 5, 200)\n",
"\n",
"plt.plot([-5, 5], [0, 0], 'k-')\n",
"plt.plot([-5, 5], [1, 1], 'k--')\n",
"plt.plot([0, 0], [-0.2, 1.2], 'k-')\n",
"plt.plot([-5, 5], [-3/4, 7/4], 'g--')\n",
"plt.plot(z, sigmoid(z), \"b-\", linewidth=2,\n",
" label=r\"$\\sigma(z) = \\dfrac{1}{1+e^{-z}}$\")\n",
"props = dict(facecolor='black', shrink=0.1)\n",
"plt.annotate('Saturating', xytext=(3.5, 0.7), xy=(5, 1), arrowprops=props,\n",
" fontsize=14, ha=\"center\")\n",
"plt.annotate('Saturating', xytext=(-3.5, 0.3), xy=(-5, 0), arrowprops=props,\n",
" fontsize=14, ha=\"center\")\n",
"plt.annotate('Linear', xytext=(2, 0.2), xy=(0, 0.5), arrowprops=props,\n",
" fontsize=14, ha=\"center\")\n",
"plt.grid(True)\n",
"plt.axis([-5, 5, -0.2, 1.2])\n",
"plt.xlabel(\"$z$\")\n",
"plt.legend(loc=\"upper left\", fontsize=16)\n",
"\n",
"save_fig(\"sigmoid_saturation_plot\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Xavier and He Initialization"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"dense = tf.keras.layers.Dense(50, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\")"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"he_avg_init = tf.keras.initializers.VarianceScaling(scale=2., mode=\"fan_avg\",\n",
" distribution=\"uniform\")\n",
"dense = tf.keras.layers.Dense(50, activation=\"sigmoid\",\n",
" kernel_initializer=he_avg_init)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Nonsaturating Activation Functions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Leaky ReLU"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAADkCAYAAADeiPCXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAg9ElEQVR4nO3deXhU9fk28PuZJEASEkBCggEFDLuILBGoCA2kUMrqqwgIBamWaHHtJQj059JXARdc3rpc1pZaqlB/gtVWLVqhJULQBhITQJIQliIFQ1LAGLLQJDPP+wdkZJjJZCaznDMz9+e65oI5y/c888zJ3DkzJ2dEVUFERBRsFqMLICKiyMQAIiIiQzCAiIjIEAwgIiIyBAOIiIgMwQAiIiJDRBux0aSkJO3Zs6cRm3arpqYG8fHxRpcRMtgvzx04cABWqxUDBw40upSQEan7V00NUFoK2GxAp05Ar16AiCfrmbNf+fn5p1S1i6t5hgRQz549kZeXZ8Sm3crOzkZGRobRZYQM9stzGRkZqKysNOV+b1aRuH/985/AxInnw2f2bGD9eiDaw1dps/ZLRL5qbh7fgiMiMoGm8Dl71vvwCVUMICIig0Vi+AAMICIiQ0Vq+AAMICIiw0Ry+AAMICIiQ0R6+AAGnQXXkqqqKlRUVKChoSGo2+3QoQOKi4uDus1Qxn557rHHHoOqhlW/YmJikJycjMTERKNLCTkMn/NM95CrqqpQXl6Obt26ITY2FuLJCfB+cvbsWSQkJARte6GO/fKcxWJBY2MjBgwYYHQpfqGqqKurw4kTJwCAIeQFhs93TPcWXEVFBbp164a4uLighg8ReU5EEBcXh27duqGiosLockIGw8eR6QKooaEBsbGxRpdBRB6IjY0N+lvloYrh48x0AQSARz5EIYI/q55h+LhmygAiIgoXDJ/mMYCIiAKE4eMeA4iIKAAYPi1jAIWYZcuWYcKECUaX4VfffPMNUlJScPjwYY+WnzlzJp5//vkAVxV83vYh0MK1z8HA8PEMA8jPxo8fjwULFgRs/MLCQgwZMsSnMcaPHw8RgYggJiYGffr0wW9/+1uvx5k6darbx9q5c2esWbPGafojjzyCHj162O+vXr0akydPRlpamkfbfeyxx7By5Up8++23XtdsZt72IdDCtc+BxvDxHAPIzwoKCjB8+PCAjb9nzx4MHTrUpzEKCgqwevVqlJWV4dChQ5gzZw7uvPNOFBQUeDXO3r17m32sR48exZkzZ5Cenu40Ly8vz75ebW0t1q5dizvuuMPj7V5zzTW46qqrsH79eq/qNbPW9CHQwrHPgcbw8Q4DyI8OHz6MysrKZl+UT5w4gQULFqBz587o2LEjbr75ZpSXlzsss3LlSgwePBjt27dHly5dsHDhQtTV1QEATp48ifLycvsRUE1NDebMmYNhw4bh6NGjAIDu3bs7vW2yb98+tGvXDkVFRfYaJ02ahK5du6JHjx648847oarYv3+/x/W29Fjz8/MhIhg2bJjLeU3BtHnzZlgsFowePdphmWeeecZ+lHbx7dFHHwUATJ8+HW+99ZbLbfvi+PHjEBG8/fbbGD9+POLi4nDttdeipKQEeXl5GDt2LOLi4jBixAgcO3bMvp675w0APv74YwwdOhRfffXdd3Pdf//9SEtLQ3l5ebN9AM73KzMzE7Gxsejduze2b9+OjRs32pdtqVfN8WS9QPU5HDF8WkFVg34bPny4NqeoqKjZeYFWVVXl0/pvv/22WiwWPXv2rNO8I0eOaHJysi5fvlyLioq0oKBAx44dqzfeeKPDco899pjm5OTo0aNHdcuWLZqamqqrV69WVdXNmzdrbGysNjY2aklJiQ4cOFDnzZuntbW19vVnzpypc+bMcRhz/Pjxevfdd9trTExM1MbGRlVVLSsr0zlz5qjFYtH9+/d7XK+7x6qqumLFCu3bt6/T9KNHjyoA/fjjj1VV9b777tMJEyY4LVdVVaVlZWX224MPPqhdu3bVgwcPqqrqRx99pDExMQ6PvcmqVas0Pj7e7W379u0u6/7ggw8UgH7/+9/X7Oxs3bt3r/br109Hjhyp48aN0x07dmhhYaH26tVL77//fvt67p43VdXi4mIdOHCg/vSnP1VV1TVr1miXLl20tLTUbR927dqlsbGx+sQTT2hpaanOmzdPMzIydPDgwfr3v//do141x5P13PW5SaB+Zrdt2xaQcQPh889VExJUAdXZs1UbGoJfg1n7BSBPm8mCkAggwJibtx566CHt37+/y3kTJ07UFStWOEzbsmWLJiQkuB1z0aJFumDBAlVVXb16tY4YMULfeecdveyyy/SFF15wWv65557TtLQ0+/333ntPO3XqpKdOnbLXaLFYND4+XmNjYxWAtmnTxmmslup96KGHXAbMxevPnTvXafo777yjAOz1zJgxw/74mvPUU09pamqqlpSU2Kft2bNHAeihQ4eclj99+rQePHjQ7a25F9SVK1dqhw4d9OTJk/Zp99xzjyYlJdlrVlVduHChzpo1q9maL37eVFVLSkr0tdde0+joaH3yySe1ffv2umvXLvv85vowZswYh+389a9/VYvFohkZGS6366pXnmhuPXd9bhLpAWSG8FE1b78YQEEKoMzMTJ03b57T9K+++koBaGxsrMNv4e3atdPLLrvMvtyxY8f03nvv1UGDBmmnTp00Pj5eo6Oj9Re/+IWqqs6aNUs7deqkiYmJmp2d7bKGnTt3KgA9ffq0njt3TtPS0hzCJTMzU7OysvTgwYOan5+vP/zhD+1HR97Um5mZ6fYFOCkpSZ9//nmn6StWrNAePXrY70+cOFGzsrKaHWf16tWampqqBw4ccJheWlqqAHTfvn3NrtsaM2fOdHoOp0+f7lTj+PHjddmyZara8vOmej6AvvzyS/3e976nUVFRunnzZofxXPWhrKxMATi8sGzZskUB6I4dO5xqb65XLXG3nid9juQAMkv4qJq3X+4CKCTeoVQNznZ8vbpzQUEBHn74YafphYWFSExMRH5+vtO8Nm3aAABOnz6N6667DmPHjsWzzz6L7t27IyoqCtddd539M5/CwkLcdNNN+OMf/4jTp0+7rGH48OFo06YN8vLyUFBQgOjoaNx9990ONc6fPx+9e/cGALz22mvo1asX7rzzTlxzzTUe11tQUIAlS5a4rOH48eM4deoUBg0a5DRvy5YtDp9zJCUl4ZtvvnE5zqpVq/DrX/8an376qb3eJmfOnAEAdOnSxWm91atXY/Xq1S7HbPLRRx9hzJgxTtP37NmD++67z2FaQUEBHn/8caflsrKyPHremuTm5mLPnj1QVaSkpDjMc9WHpq9uuO666+zTDhw4gH79+uGGG25wWNZdr9xpaT13fY50/MzHD5pLJk9vANoB2AVgD4D9AP5vS+uE42dAR44cUQAuj0w2b96sUVFRzX5eoqr6hz/8QTt06KA2m80+bd26dQpAS0tLtaamRi0Wi+7atUs3bNig8fHxmp+f73KskSNH6j333KMJCQn6wQcfONV46XrDhg3TpUuXelxv0ziX/hZ/6fy//OUvDtNzc3NVRBymr1mzRq+++mqnMR5//HG94oormn3rZ+3atZqamupyXmvfgquurlaLxaI5OTkOYwHQwsJC+7Rjx44pAD1w4ECLz1uTP//5z5qQkKC/+93v9KabbtKJEyc6bNtVH959910VEXutVVVVevnll+vQoUO96lVzPFnPXZ+bROIRkJmOfJqYtV8I5FtwAARA+wv/jwGQC2CUu3XCMYA2bdqkAHTnzp26b98++62oqEjPnDmjSUlJeuONN+oXX3yhhw4d0k8++UQXL16sVqtVVVU//PBDjYqK0nfffVcPHjyoL774onbt2lUTEhLUZrPpZ599plFRUfYXo4cfflhTU1P1+PHjTrU88MADKiJOL3KbNm1Si8Xi9OK7bNky7d27t/1+S/U2PdYtW7Y4PVZVVZvNpgMGDNBBgwbpJ598onv27NHXX39dU1NTddq0aQ7b3rt3r1osFofPV1auXKmdO3fWnTt3OnxIXldXZ1/mtttu09tvv701T1WzPvvsM6cTK7Zu3apt27bV+vp6+7T3339f27dvr1artcXnTfX8iRfJycn2tzr37dunIuLwguGqD01B/sgjj2hJSYlOmzZNhw0bpomJifZw86RXrni6nid9jrQAMmP4qJq3XwENIIfBgDgAXwAY6W65cAyg5cuXKwCn26BBg1RVdffu3Tpu3Djt0KGDtm/fXgcPHuxwlpTNZtPFixdrQkKCdunSRe+991598MEHdfTo0aqq+uqrrzqc4GCz2fSWW27RYcOGaU1NjUMt69at06ioKP3yyy+dauzTp49T7Vu3blUADsu7q7elx6p6/sXz5ptv1qSkJE1ISNAhQ4bor371K/3vf//rtP1Ro0bpyy+/bH9ciYmJLsffunWrqqrW1dVpYmKifv755x48M5579dVXtV+/fg7T1qxZo5fur0888YT9eWnpeTt9+rT2799fZ82a5dDfWbNm6ahRo5rtQ5NVq1ZpUlKStm3bVufOnatnzpzR66+/XpOSkjzq1e9//3sFoP/617/sY3qynqrnfY6kADJr+Kias1+qQQggAFEACgFUA3i6peXDMYDMZMKECbp48eKAb8df/froo4+0b9++9lPDW/Lyyy+7PGXZzJpOQnDH2z544tFHH9WBAwdqQyteKT3tc6QEkJnDR9V8/WriLoD88pGZqloBDBGRjgDeE5FBqvrlxcuISBaALABISUlBdna2y7E6dOiAs2fP+qMsr1mtVsO27SubzYZTp05hw4YN2Lt3L9auXRvwx+Kvfo0ePRp33HEHSkpKcOWVV3q03aeeeiqknqvGxkYAcFuzt33wxAcffIBnnnnG4Y9iPeVpn8+dO9fsz7MvqqurAzJuaxQVJWLp0sGorY3GuHEVWLSoGDk5QTo7ykNm6pfHmkum1t4APAZgibtleATkf9u2bVMR0f79++vOnTuDss1Q7leweXIEFKrC/QjI7Ec+TczSr0shkEdAItIFQIOqVopILIAfAHja13HJOxkZGbDZbEaXQRRWeKp1YPmjlZcD+IOIROH8teU2quqHfhiXiMgwDJ/A87mdqroXgG+XZyYiMhGGT3DwathERBdh+AQPA4iI6AKGT3CZMoD4YTpRaAinn1WGT/CZLoDi4+Nx4sQJ1NfXN53WTUQmo6qor6/HiRMnEB8fb3Q5PmP4GMN0Le7evTtOnTqFr776yv7He8Fy7tw5tGvXLqjbDGXsl+dOnjwJVYXFYrrf+VotOjoaHTp0QFJSktGl+IThYxzTtdlisSA5ORnJyclB33Z2djaGDuUJfZ5ivzz3s5/9DJWVlSgsLDS6FLoIw8dY4fPrGBGRFxg+xmMAEVHEYfiYAwOIiCIKw8c8GEBEFDEYPubCACKiiMDwMR8GEBGFPYaPOTGAiCisMXzMiwFERGGL4WNuDCAiCksMH/NjABFR2GH4hAYGEBGFFYZP6GAAEVHYYPiEFgYQEYUFhk/oYQARUchj+IQmBhARhTSGT+hiABFRyGL4hDYGEBGFJIZP6GMAEVHIYfiEBwYQEYUUhk/4YAARUchg+IQXBhARhQSGT/hhABGR6TF8whMDiIhMjeETvnwOIBG5QkS2iUixiOwXkfv9URgREcMnvPnjCKgRwIOqOgDAKAB3i8hAP4xLRBGsqCiR4RPmfA4gVS1T1S8u/P8sgGIA3Xwdl4gi1z//CSxdOpjhE+b8+hmQiPQEMBRArj/HJaLI0fS2W21tNMMnzPntaRWR9gD+BOABVa1yMT8LQBYApKSkIDs721+b9pvq6mpT1mVW7JfnKisrYbVa2a8WFBUlYunSwaitjcaYMSewaNEh5OSo0WWFhFD8eRRV359cEYkB8CGAv6nq8y0tn56ernl5eT5v19+ys7ORkZFhdBkhg/3yXEZGBiorK1FYWGh0KaZ16QkHixZ9iszM7xtdVsgw68+jiOSrarqref44C04A/A5AsSfhQ0R0KVdnu0VF8cgn3PnjM6DRAOYDGC8ihRduk/0wLhFFAJ5qHbl8fppVNQeA+KEWIoowDJ/IxishEJEhGD7EACKioGP4EMAAIqIgY/hQEwYQEQUNw4cuxgAioqBg+NClGEBEFHAMH3KFAUREAcXwoeYwgIgoYBg+5A4DiIgCguFDLWEAEZHfMXzIEwwgIvIrhg95igFERH7D8CFvMICIyC8YPuQtBhAR+YzhQ63BACIinzB8qLUYQETUagwf8gUDiIhaheFDvmIAEZHXGD7kDwwgIvIKw4f8hQFERB5j+JA/MYCIyCMMH/I3BhARtYjhQ4HAACIitxg+FCgMICJqFsOHAokBREQuMXwo0BhAROSE4UPBwAAiIgcMHwoWBhAR2TF8KJgYQEQEgOFDweeXABKR10WkQkS+9Md4RBRcDB8ygr+OgNYBmOSnsYgoiBg+ZBS/BJCqbgdwxh9jEVHwMHzISPwMiChCMXzIaEHb3UQkC0AWAKSkpCA7OztYm/ZYdXW1KesyK/bLc5WVlbBarabpV1FRIpYuHYza2miMG1eBRYuKkZOjRpflgPuXd0KxX6Lqn51ORHoC+FBVB7W0bHp6uubl5fllu/6UnZ2NjIwMo8sIGeyX5zIyMlBZWYnCwkKjSwmZIx/uX94xa79EJF9V013N41twRBEkVMKHIoO/TsN+C8DnAPqJyHERucMf4xKR/zB8yGz8svup6q3+GIeIAoPhQ2bEt+CIwhzDh8yKAUQUxhg+ZGYMIKIwxfAhs2MAGUhE8M477xhdBoUhhg+FAgaQGwsXLsTUqVONLoPIKwwfChUMIKIwwvChUMIAaqWioiJMmTIFCQkJSE5Oxq233oqTJ0/a5+/evRsTJ05EUlISEhMTccMNN+Dzzz93O+bTTz+NpKQk5ObmBrp8CkMMHwo1DKBWKCsrw9ixYzFo0CDs2rULW7duRXV1NaZPnw6bzQYAOHv2LObPn48dO3Zg165dGDJkCCZPnoxTp045jaeqWLJkCV566SV8+umnGDlyZLAfEoU4hg+FIu6irfDqq6/i2muvxdNPP22f9sYbb+Cyyy5DXl4eRowYgfHjxzus89JLL+FPf/oTPv74Y/z4xz+2T7darbj99tuxc+dO5OTkoGfPnsF6GBQmGD4UqribtkJ+fj62b9+O9u3bO807fPgwRowYgYqKCjzyyCPYtm0bysvLYbVaUVdXh2PHjjksv2TJEkRHRyM3NxfJycnBeggUJhg+FMq4q7aCzWbDlClT8OyzzzrNS0lJAQDcdtttKC8vxwsvvICePXuibdu2yMzMRH19vcPyEyZMwFtvvYXNmzdj4cKFwSifwgTDh0Idd9dWGDZsGDZu3IgePXogJibG5TI5OTl48cUXMWXKFABAeXk5ysrKnJabPHkybrrpJtxyyy0QEdx2220BrZ3CA8OHwgFPQmhBVVUVCgsLHW5TpkzBt99+i9mzZyM3NxdHjhzB1q1bkZWVhbNnzwIA+vbti/Xr16OoqAi7d+/GnDlz0KZNG5fbmDp1KjZt2oS77roLb7zxRjAfHoUghg+FC+62LdixYweGDh3qMO3mm2/Gzp07sWLFCkyaNAnnzp3DlVdeiYkTJ6Jt27YAgNdffx1ZWVkYPnw4UlNT8ctf/hL/+c9/mt3O1KlTsXHjRsyaNQsAsGDBgsA9KApZDB8KJ9x13Vi3bh3WrVvX7Hx3l9G59tprnf6eZ/78+Q73L/022mnTpqGurs77QikiMHwo3PAtOKIQwPChcMQAIjI5hg+FKwYQkYkxfCicRWQANTY24uDBg0aXQeQWw4fCXcQFUFFREQYPHoyrr74axcXFRpdD5BLDhyJBxASQ1WrFk08+ifT0dJSUlKCxsREzZ85EY2Oj0aUROWD4UKSIiAAqLS3FsGHDsHLlStTV1UFVoao4evQoVq1aZXR5RHYMH4okYR1ANpsNzz33HIYMGYJ9+/ahtrbWYX5tbS2efPJJNDQ0GFQh0XcYPhRpwnb3PnLkCGbPno3i4mKXf9wZGxuLzp07Y9OmTc1ez40oWBg+FInC7ghIVfHKK6/gmmuuwRdffIGamhqnZWJjY/GTn/wEpaWlGDVqlAFVEn2H4UORKqx283//+9+YM2cO9uzZ4/R2GwC0a9cOHTt2xMaNGzFmzBgDKiRyxPChSBYWR0CqirVr12LAgAHIzc1t9qhn7ty5OHjwIMOHTIHhQ5Eu5Hf3r7/+GvPmzcPu3btdBk/btm2RmJiIt956C5mZmQZUSOSM4UMUwkdAqoo333wT/fr1Q05OjsvwiYuLw8yZM3Ho0CGGD5kGw4foPL8EkIhMEpEDInJIRJb7Y0yr1YrFixejvLzcaV5FRQV+9KMf4a677kJ1dbXTH5O2bdsWnTp1wqZNm7B+/XokJib6oyQinzF8iL7jcwCJSBSAVwD8CMBAALeKyEBfx127di1ee+01zJ8/3+F7czZt2oQ+ffrgH//4h8sTDeLi4jBt2jQcPnwYkydP9rUMIr+prY1m+BBdxB9HQCMAHFLVI6paD+B/AczwZcCqqiosW7YMNpsNO3fuxBtvvIHTp09jxowZWLhwIaqqqpz+eLRNmzbo0KEDNmzYgE2bNqFTp06+lEDkF6pARQVw5Ahw6FA8w4foInLpt3J6PYDITACTVPWnF+7PBzBSVe9pbp2EhAQdPnx4s2MePnwYX3/9NWw2GwDAYrHAYrHAarU6fYto0/yOHTuif//+Pv1RaWVlJTp27Njq9SNNpPersRGorwf++1/Hfy+ddn6XLQQAdO06BH37AiJGVh4aIn3/8pZZ+/Xpp5/mq2q6q3n++B3M1Y+SU0qISBaALACIiYlBZWWly8Hq6+tx4sQJh6Cx2Wz2MLqUxWJB9+7d0alTJ5cnInjDarU2Wxc5C9d+Wa2ChgYLGhoEjY3n/z1/33GaqmcpIqIQUbRpY0VKSiW+/TbADyBMhOv+FSih2C9/BNBxAFdcdL87gK8vXUhVfwPgNwCQnp6ueXl5LgebMWMGSktLYbVa3W40Pj4e119/Pd58802kpKS0tnYH2dnZyMjI8MtYkSCU+qUKVFYCZWXA119/9+/F/2/699w5z8aMjwdSU7+7XX65479N/09IADIyMlBZWYnCwsJAPsywEkr7lxmYtV/i5nDfHwG0G0AfEekF4ASAOQDmtmag3NxcbNmyxW34REdHo127dnjllVcwf/58tw+Owp/RwUJEredzAKlqo4jcA+BvAKIAvK6q+1sxDhYtWuTywqEXa9OmDYqLi9G9e/fWFUwhgcFCFP78ch6Oqm4GsNmXMd5++20cOXKkxeVEBO+99x7uvfdeXzZHBmGwEFETU5wIeu7cOdx3330enURQU1OD5cuXY/LkyUhLSwtCdeQJBgsRecsUAfTss896fAZbTEwMGhoa8POf/xzvv/9+gCsjVeDbb12HSWHhQDQ2MliIqHUMD6Dy8nKsWrUK5y68eokI4uLiEB0djfr6ejQ0NKBLly7o0aMH+vfvjwEDBiAtLQ3p6S5PKycPuQuWS6c1HyzJDvcYLETkDcMDKDs7G7GxsUhLS0OfPn1w9dVXo3fv3rjqqqvQq1cvdOvWDRZLyF4zNej8EyyOmguWb74pwg9+MJDBQkStYngAzZ49G7Nnzza6DNMLZrB4esSSnV2BjAyfL/tHRBHK8ACKdGYMFiKiYGAABQiDhYjIPQaQlxgsRET+wQC6QBWoro5GURGDhYgoGMI+gLw7YrnBozEZLEREvgvZAArEW2Ht2llxxRVRDBYioiAwXQAZ+RlLfv4OU17OnIgoHBkSQGfPAm++yc9YiIgimSEBVFoKLFjQ/HwGCxFR+DMkgNq3B6ZNY7AQEUUyQwKoXz/gj380YstERGQWvMonEREZggFERESGYAAREZEhGEBERGQIBhARERmCAURERIZgABERkSEYQEREZAgGEBERGYIBREREhmAAERGRIRhARERkCAYQEREZwqcAEpFbRGS/iNhEJN1fRRERUfjz9QjoSwA3Adjuh1qIiCiC+PR9QKpaDAAi4p9qiIgoYvAzICIiMkSLR0AishVAVxez/kdV/+LphkQkC0DWhbvVInLA03WDKAnAKaOLCCHsl3eSRIT98hz3L++YtV89mpshqurz6CKSDWCJqub5PJiBRCRPVXkyhYfYL++wX95hv7wTiv3iW3BERGQIX0/D/j8ichzA9wD8VUT+5p+yiIgo3Pl6Ftx7AN7zUy1m8BujCwgx7Jd32C/vsF/eCbl++eUzICIiIm/xMyAiIjIEA6gZIrJERFREkoyuxcxEZI2IlIjIXhF5T0Q6Gl2T2YjIJBE5ICKHRGS50fWYnYhcISLbRKT4wqW+7je6JrMTkSgRKRCRD42uxRsMIBdE5AoAEwAcM7qWELAFwCBVHQygFMAKg+sxFRGJAvAKgB8BGAjgVhEZaGxVptcI4EFVHQBgFIC72bMW3Q+g2OgivMUAcu0FAA8B4AdkLVDVT1S18cLdfwLobmQ9JjQCwCFVPaKq9QD+F8AMg2syNVUtU9UvLvz/LM6/sHYztirzEpHuAKYAWGt0Ld5iAF1CRKYDOKGqe4yuJQTdDuAjo4swmW4A/n3R/ePgi6nHRKQngKEAcg0uxcz+H87/wmwzuA6v+XQadqhyd3khAL8AMDG4FZmbJ5djEpH/wfm3TjYEs7YQ4OpKvTyy9oCItAfwJwAPqGqV0fWYkYhMBVChqvkikmFwOV6LyABS1R+4mi4i1wDoBWDPhSt8dwfwhYiMUNWTQSzRVJrrVxMRuQ3AVACZyvP6L3UcwBUX3e8O4GuDagkZIhKD8+GzQVXfNboeExsNYLqITAbQDkCiiKxX1R8bXJdH+HdAbojIUQDpqmrGC/yZgohMAvA8gO+r6n+MrsdsRCQa50/OyARwAsBuAHNVdb+hhZmYnP/t7w8AzqjqAwaXEzIuHAEtUdWpBpfiMX4GRL56GUACgC0iUigivza6IDO5cILGPQD+hvMfpm9k+LRoNID5AMZf2KcKL/yGT2GGR0BERGQIHgEREZEhGEBERGQIBhARERmCAURERIZgABERkSEYQEREZAgGEBERGYIBREREhmAAEQWQiDx04YsNL709bnRtREbjlRCIAkhEEgDEXzRpCYB5AMao6iFjqiIyBwYQUZCIyDIA9wEYr6oHjK6HyGgR+XUMRMEmIitw/qKk41S11Oh6iMyAAUQUYBe+rO8unP/KCr7tRnQBA4gogETkEQCLAGSo6mGj6yEyEwYQUYBcOPK5H8B0ADUi0vS15pWqes64yojMgSchEAXAhW/1rASQ6GL2D1T178GtiMh8GEBERGQI/iEqEREZggFERESGYAAREZEhGEBERGQIBhARERmCAURERIZgABERkSEYQEREZAgGEBERGeL/AzODoCvOmQ/oAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# extra code – this cell generates and saves Figure 11–2\n",
"\n",
"def leaky_relu(z, alpha):\n",
" return np.maximum(alpha * z, z)\n",
"\n",
"z = np.linspace(-5, 5, 200)\n",
"plt.plot(z, leaky_relu(z, 0.1), \"b-\", linewidth=2, label=r\"$LeakyReLU(z) = max(\\alpha z, z)$\")\n",
"plt.plot([-5, 5], [0, 0], 'k-')\n",
"plt.plot([0, 0], [-1, 3.7], 'k-')\n",
"plt.grid(True)\n",
"props = dict(facecolor='black', shrink=0.1)\n",
"plt.annotate('Leak', xytext=(-3.5, 0.5), xy=(-5, -0.3), arrowprops=props,\n",
" fontsize=14, ha=\"center\")\n",
"plt.xlabel(\"$z$\")\n",
"plt.axis([-5, 5, -1, 3.7])\n",
"plt.gca().set_aspect(\"equal\")\n",
"plt.legend()\n",
"\n",
"save_fig(\"leaky_relu_plot\")\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"leaky_relu = tf.keras.layers.LeakyReLU(alpha=0.2) # defaults to alpha=0.3\n",
"dense = tf.keras.layers.Dense(50, activation=leaky_relu,\n",
" kernel_initializer=\"he_normal\")"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2021-12-16 11:22:41.636848: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
"To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
]
}
],
"source": [
"model = tf.keras.models.Sequential([\n",
" # [...] # more layers\n",
" tf.keras.layers.Dense(50, kernel_initializer=\"he_normal\"), # no activation\n",
" tf.keras.layers.LeakyReLU(alpha=0.2), # activation as a separate layer\n",
" # [...] # more layers\n",
"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ELU"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Implementing ELU in TensorFlow is trivial, just specify the activation function when building each layer, and use He initialization:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"dense = tf.keras.layers.Dense(50, activation=\"elu\",\n",
" kernel_initializer=\"he_normal\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"### SELU"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By default, the SELU hyperparameters (`scale` and `alpha`) are tuned in such a way that the mean output of each neuron remains close to 0, and the standard deviation remains close to 1 (assuming the inputs are standardized with mean 0 and standard deviation 1 too, and other constraints are respected, as explained in the book). Using this activation function, even a 1,000 layer deep neural network preserves roughly mean 0 and standard deviation 1 across all layers, avoiding the exploding/vanishing gradients problem:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAD+CAYAAAB8xdFqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA5cElEQVR4nO3deXgUVfbw8e/NDmQjBEKIbIogIMgmIo4QDJuIuKCijAqjA4KM44bj6wY6DKg4MuMAIowO4k9Fx3EQVEBhIICKQoCIrBJWCSAQCCQkIUuf949KmjTphIR0Ut3J+TxPPemuulV96nanT1fVrXuNiKCUUkpVNz+7A1BKKVU7aQJSSillC01ASimlbKEJSCmllC00ASmllLKFJiCllFK2CLDjRaOjo6VFixZ2vHSZzpw5Q7169ewOw2dofZXfzp07KSgooF27dnaH4jO88vPlcICfX4lZKSmQkQEBAdCmDYSEVH9oXllfwIYNG46LSEN3y2xJQC1atCApKcmOly5TYmIi8fHxdofhM7S+yi8+Pp709HSv/Nx7K6/7fE2YAAsXwpIlEBMDQHY2DBkCmzZZs1asALt+Y3hdfRUyxuwvbZmeglNKqQuZOhUmTYKtW2HDBuBc8lm+3P7k46s0ASmlVFnefBOefhqMgXffhUGDNPl4SKUTkDEmxBizzhjzozFmqzHmJU8EppRStps3D8aNsx7PmgX33qvJx4M8cQ3oLHCDiGQaYwKBb4wxS0Tkew9sWyml7PHJJ/DAA9bjv/4VHnpIk4+HVToBidWbaWbh08DCSXs4VUr5trVrrSZuL74ITz6pyacKGE/0hm2M8Qc2AK2AmSLytJsyo4HRADExMV0/+uijUrcXFBRESEgIxphKx1YRIlLtr+nLtL7K7+TJk4gIUVFRdofiM6rq8yUi5OTkkJube6GCRK1bx4nu3Tmb689zz13Jhg1R1K+fy7RpybRokeXx2CojMzOT0NBQu8MooU+fPhtEpJu7ZR5JQM6NGRMJLAAeEZEtpZXr1q2blNYc9cCBAxhjiImJITAwsFq/4DIyMggLC6u21/N1Wl/lt3PnTvLz82nfvr3dofiMqvh8iQh5eXn8+uuviAjNmjVzLbBxIzRtCg3P3bbiK0c+XtwMu9QE5NFWcCKSDiQCAy92G2fOnCEuLo6goCD9da2U8ihjDEFBQcTFxXHmzBnXhcnJkJAAvXvDsWOA7yQfX+WJVnANC498MMbUAfoCOyoVlJ+2DldKVZ0S3zHbt0P//pCeDm3bQv36mnyqgSdawcUC8wqvA/kB/xaRLzywXaWUqnp79kDfvtZRz8CB8OGHZOcFaPKpBp5oBbcZ6OyBWJRSqnodPGiddjt0yDr19umnZDuCNflUEz3XpZSqnRwO68hn3z7o3h0+/5xsU1eTTzXSBFQDnTx5kpiYGHbv3l2u8nfccQfTpk2r4qjKVtGYi/Tu3ZtOnTrRqVMnAgICWLt2bRVFWFJ5623kyJE89NBDzucOh4OHHnqIBg0aYIwhMTGxCqP0DiNHjmTw4MF2h+HKzw8GD4aOHWHJErIDwjT5VDcRqfapa9euUppt27aVuqyqnT59ulLrjxgxQrBuwnWZrrnmGufym266qdT1e/fuLePGjSsxf+7cuVKvXr1yxzF+/HgZOXJkuctv3rxZ6tevL+np6eVeR6Ty9VVcRWM+30svvSQPP/ywx+IREVm1apXcfPPN0qRJEwFk7ty5LsvLW2/p6emybt062bJli4iIfP755xIYGCjffvutHD58WM6ePevRuCti5syZ0qJFCwkODpYuXbrI6tWrq+R1LvTZd8eTny93tm3bJuJwiGRkSFaWSN++IiASEyOydWuVvnSVWLlypd0huAUkSSm5QI+APKxv374cPnzYZVq8eHG1vX5WVhZvv/02Dz74YLnX6dChA5deeinvv/9+FUZWuouJubjp06eza9cuZsyY4dG4MjMzufLKK3njjTeoU6dOieXlrbeIiAjCw8Odz1NSUoiNjaVnz540btyYoKAgj8Z9vtOnT5Oenl5i/scff8yjjz7Ks88+y6ZNm+jZsyc33ngjBw4cqNJ4bFVQAPv3Q16e9dwYsv1D9cjHJpqAPCw4OJjGjRu7TNV59/vixYvx8/Pjuuuuc5k/depUjDElpgkTJgAwZMgQ5s+fXyUxbdiwgYSEBOrUqUOrVq1YvXo1//73v50xlhYzQGpqKvfffz8NGjQgMjKSoUOH8uuvvzqXv/fee6xYsYK5c+d6/L6xQYMGMWXKFO64445Sbw0oT70VPwU3cuRIHn/8cecN12UNzHih96wsBQUFfPXVVwwfPpzGjRvz448/ligzbdo0Ro4cyahRo2jbti3Tp08nNjaWWbNmXXD7xYkIU6dO5bLLLqNOnTp06NChzKS8evVqevToQWhoKBEREVxzzTVs2bKlxPY6duxYru1BOevK4bBavB07Zv1F7/OxmyagGmbNmjV07dq1xJfx2LFjXY7KnnzySRo3bsz9998PQPfu3Vm3bh3Z2dkltjllyhRCQ0NLTLGxsc7Ha9ascRvP+vXruf766+nTpw+bN2+mR48eTJw4kcmTJzNp0qQyY967dy9dunQhLi6Ob775hsTERI4fP86YMWMAWLBgAR988AEfffQRAQG2jK1YZr2588YbbzBhwgQuueQSDh8+zPr160ste6H3zJ2tW7fypz/9iWbNmjFs2DDq1avH0qVL6dWrl0u53NxcNmzYQP/+/V3m9+/fn++++65c+1Lk+eef55133mHmzJls27aNZ555hoceeogvv/yyRNn8/HxuueUWfvOb3/Djjz/yww8/8Oijj+Lv719ie6+//voFt1fkgnUlAnv3wqlT1rClzZohosnHbvb819ZgS5cuLdEf07hx43j11Ver5fX3799PbGxsiflhYWHObk1effVV5s+fT2JiIq1atQKgSZMm5OXlcejQIS677DKXdceMGcNdd91VYpvF+56Ki4tzG8+TTz7JzTffzPPPPw/A8OHDufnmm+nVqxc33HBDmTGPGTOGBx98kClTpjjnvfDCC9x+++0APPDAAzRs2JBrrrkGgIkTJ3LbbbeVUTueV1a9uRMREUFYWBj+/v40bty4zLIXes+KpKWl8cEHH/Dee++xefNmBg4cyN///neGDBlCcHCw220fP36cgoICYgpH9iwSExPD8uXLL7gfRc6cOcO0adP4+uuvuf766wFo2bIl69atY+bMmdx0000u5YtOB958883O+rriiivcbq9Tp06EhYWVub1y1ZWI1dLt5Enw94fWrXEE1+HoUU0+dvOJBFR9PfK49jt1Md3k9erVizlz5rjMi4yMrERMFZOdnV3iS6W4l19+mRkzZrBy5Upat27tnF90jcPdL/moqCi3pxEv1FfXkSNHWLNmDStXrnTOCwoKwuFwOI9+Sov5wIEDfP3116xZs4Z//OMfzvkFBQXUrVsXsFrOXcjzzz/P5MmTyyyzcuXKi+5Dq6x685TS3rMi06dP56WXXqJnz57s2rWL5s2bl3vb5x91SgU7AN22bRs5OTkMHDjQZb28vDy3pxejoqIYOXIkAwYMICEhgYSEBO68806aNm16Uds7X4m6EoEDByAtzWr1dvnlOELqkpICOTmafOzmEwnIl9StW7fEL9TyCg8P59SpUyXmp6enExERUa5tREdHl/rFPHnyZN566y1WrVpVIsYTJ04A0LBYJ4xFpkyZ4nIU4s6SJUucv4CLbN++HYCrr77aOW/nzp20adOG3/zmN2XGnJycTHh4OBsKhz8uriIX7R977DHuvffeMsuU6JCyAsqqN08o6z0rMnr0aAIDA3nvvfdo3749t912G/fddx8JCQkup7aKi46Oxt/fnyNHjrjMP3r0aJk/YM7ncDgA+Pzzz0vUY2BgoNt15s6dy2OPPcbSpUtZtGgRzz33HJ999hkDBgxw2V5UVJTL2YTStlfEbV2dOmVd8zEGWrXCUTeUlBQ4fdrKR5p87OUTCciDHXaXye7endu0acPixYtL/ArduHEjbdq0cSm7adMmHnvsMVJTUxk/fjwrVqzg73//O507d+bdd98tse1Jkybxz3/+k8TERLenirZs2UKTJk3cfvlc7Cm49PR0jDHOC/gZGRlMnjy5xKkndzEHBgZy5swZGjduXKku5qOjo4mOjr7o9S+krHqrrAu9Z0WaNGnCc889x3PPPcf333/PvHnzuPvuuwkODmb48OHce++9dO7s2llJUFAQXbt2ZdmyZdx5553O+cuWLWPo0KHljrFdu3YEBwezf/9+5ynV8rjqqqu46qqrePrpp7nxxhuZN28eAwYMcNne1VdfXe7/x1LrKiICYmOhXj0coeHO5BMYCI0ba/Kxm08kIF9y9uzZEr8q/f39nb+QT58+TXJyssvyyMhIWrRowdixY5kxYwaPPPIIo0aNIiQkhMWLFzN//nwWLlzoLJ+VlcWwYcP44osviIqKonPnzgwePJgmTZowYMAAnn76adLS0mjQoAFg/TJ84403WLRoEfXq1XPGFxkZSUhICGA1BBg40H0n5hd7Cq5Tp06ICC+//DK//e1veeqpp4iNjSUlJYVdu3Zx+eWXA7iNuUePHtSvX5/77ruPCRMmEB4ezp49e/jss8+YPn16tXRYm5mZSUpKCmD90j9w4ADJyclERUU5f+2XVW+VUZ73zJ0ePXrQo0cP/v73v/P5558zb948unfvzooVK0ocoT7xxBPcd999dO/eneuuu4633nqLQ4cOORt5lEdYWBjjx49n/PjxiAi9evUiMzOT77//Hj8/P0aPHu1Sfu/evcyePZshQ4YQFxfHnj172Lx5M2PHji2xvezsbPr371/m9kqtK4eDyKgoq67i4nA4cEk+rVtbl4WUzUq7Qagqp9p2I2pcXFyZy4cOHercxrp166R///7SqFEjCQ8Pl+7du8uCBQtcXuezzz6TBx54wPk8ODhYUlJSnM979OghM2bMEBERh8Mh4eHhbl93+fLlIiKSnZ0t4eHhsnbt2grtb3nqa/LkyRIdHS3BwcEyfPhwOXHihPTs2VOio6NdyhWPucj69eulT58+EhERIaGhodKxY0eZMmVKhWKsjJUrV7qttxEjRohI+ettxIgR0rt3b+eNqK+99po0b9681PLlec8q4tdff5Xjx4+7XTZz5kxp3ry5BAUFSZcuXWTVqlUuy+fOnSuA7N27t8x4//GPf0jbtm0lKChIoqOjpW/fvvL11187yxTdiHrkyBG57bbbpEmTJhIUFCRNmzaVp556SnJzc0tsr02bNqVur3jZUutqyRIRESkoENm5U2T9epHkZJGsLGtdO79rqoIv3oiqCaiYqr7z2lNeeukl55f1jh07pHv37i7LlyxZIq1bt5b8/PxybW/GjBnSr1+/CsfhyfqqaMzeoCL1tmPHDmcC8iUTJkyQdu3aSV5eXrW/9kV9vo4ftzLN+vUix46VmnxENAFVl7ISkN4H5IMCAwNJTU1FRHj22WfJz893WT5w4EDGjRvHwYMHy7296dOnV0Wo5VbRmL2BN9RbVVu8eDEzZsyw7T6rCjl50rrXB6zTblHRJU67uenQQtlIE5APuvfee1m6dClXXHEFgwcPpn79+iU6xfzjH/9Y7ua4o0ePLtHIwQ4VidkbeEu9VaX169fTp08fu8O4sFOnnL0bEBuLIyZWk48P8IGfNep8TZs2ZePGjc7nv/vd72yMRimbZWRYLQxEICYGR+Mmmnx8hB4BKaV8W0aGlXyio3E0uYSU3UaTj4/QIyCllG+LjYU6dXCER2ry8TF6BKSU8j05OZCbaz02BkdEfU0+PkiPgJRSvuXsWfj5Z6t7nTZtcAQE6TUfH6VHQEop35GbayWf3FwIDMRh/DX5+DBNQEop35CXZyWfs2ehbl0cl7UiZa+/Jh8fpglIKeX98vNh1y7r2k+dOjhatSZlb4AmHx+nCUgp5d0cDiv5ZGVBcLCVfPZp8qkJtBGCUsq7GQPh4ZCXh+Py1qTsD9TkU0NoAlJKeTdjrL7dGsbokU8No6fglFLeR4Sgo0ed9/o4HGjyqYE0AdVgJ0+eJCYmht27d5er/B133FGiU1PlfWr8+yoC+/YRfPIk7NqFo0C0qXUNpQnIg44dO8bDDz9MixYtCA4OJiYmhoSEBJYtWwbAyJEjMcaUmHr06OHcxsiRIxk8eHCprxEfH88f/vCHEvPffffdEkNXT5kyhUGDBpU5nHNxEydO5C9/+QunTp0qV/mKWL16tXMUTGOM22HD3XnzzTdp2bIlISEhdO3alTVr1rgsf/HFF0vU5/lDfhep7vqvKt70vnqcCPzyC6SlIcbgaNpceziowSqdgIwxTY0xK40x240xW40xj3oiMF80dOhQ1q1bxzvvvMPPP//MF198wY033khaWpqzTN++fTl8+LDLtHjxYo/HkpWVxdtvv82DDz5Y7nU6dOjApZdeyvvvv+/xeDIzM7nyyit54403qFPOb5CPP/6YRx99lGeffZZNmzbRs2dPbrzxRg4cOOBSrk2bNi71+dNPP5W6zeqq/6ribe+rR4lAaiocPQrGkNUkjpQjoZp8ajBPHAHlA0+KSFugBzDOGNPOA9v1Kenp6axZs4ZXXnmFhIQEmjdvztVXX8348eO5++67neWCg4Np3LixyxQVFeXxeBYvXoyfnx/XXXedy/ypU6e6PQqYMGECAEOGDGH+/Pkej2fQoEFMmTKFO+64Az+/8n3spk2bxsiRIxk1ahRt27Zl+vTpxMbGMmvWLJdyAQEBLvXZsGHDUrdZXfVfVbztffWoI0esyRgcLS9j/8loTT41XKUTkIgcFpGNhY8zgO1AXGW362tCQ0MJDQ1l0aJF5OTk2B0Oa9asoWvXrhhjXOaPHTvW5df/k08+SePGjbn//vsB6N69O+vWrSM7O7vENqdMmeLcz9DQUGJjY12eh4aGljhFdrFyc3PZsGED/fv3d5nfv39/vvvuO5d5e/bsIS4ujpYtW3L33Xezp2hgshqoKt5Xr5CZaR39AI7mLUk5HklWVoAmnxrOo82wjTEtgM7AD26WjQZGA8TExJCYmOh2GxEREWRkZJSYHxYeXurr5rzxBnmFg7IFzp1LyKOlnwXMOH3a+bhur174Jyefe41SypXXrFmzeOSRR5gzZw4dO3akR48e3HrrrVx99dUA5OXlsXTp0hLXCkaNGsWf//xnZ5n8/Hy3dQBQUFBAbm5uieVFSa9o/u7du4mOjna7nXr16gHwt7/9jQ8//JAvv/ySmJgYMjIyiIiIIC8vj59//plLL73UZb3f/va3DBo0yPnc4XCUOJpp0qRJqbGfH29Z5Q4fPkxBQQFhYWEu5SIjIzl06JBzXocOHZg1axatW7fm2LFjvPbaa1x77bX88MMPNGjQwGWb1Vn/w4YNY+3atfTu3Zv/+7//cw6bfv56+/fvZ9y4cfz6668YY1iyZAlHjhxh9OjRzjK7du3iX//6F4MHD66S99UriBAUHY3DP4C9R8PJygJ/fwdxcdnk5zsox0eqwnJyckr9HvJFmZmZPrc/HktAxphQ4FPgMREp8e0tInOAOQDdunWT+Ph4t9vZvn07YWFhbpeVJiQkhJCidUJCyizrsu0yTgVVNAawhsq+4447WLNmDWvXrmXp0qVMnz6dyZMn8+yzzxIYGEivXr2YM2eOy3qRkZHO1wsMDCQgIKDU1/f39ycoKKjE8pDC/S6an5eXR1hYWKnbefnll5k9ezaJiYm0bt3aOT86Otr5OuevGxYW5jJkdkZGxkXVU1G8Za1b9AVbr149l3JBQUEusQ0dOtRlvRtuuIFLL72U//73vzzxxBMuy6qz/p966ikyMzOZN28eYWFhBAQEkJ+fX2K9cePGMWnSJHr16sWpU6cICQmhRYsWbN682VkPLVu2ZMiQIdSrV69K3ldbORzO/0NHaDgpKVaHB4GBEBeXTXR0vSp76ZCQEDp37lxl269uiYmJlPa96q080grOGBOIlXw+EJH/emKbJYiUPhX7tcjo0WWXLW7DBpdlGadPuy9XASEhIfTr148JEybw3Xff8eCDD/Liiy+SW3g/Q926dWnVqpXLVPTlUB7h4eFuWzOlp6cTERHhfB4dHc3JkyfdbmPy5Mm8+eabrFq1yuVLCuDEiRMAbq+jVOcpuOjoaPz9/Tly5IjL/KNHjxITE1PqeqGhobRv355du3a5XV5d9d+nT58LftFv3brVmRTBOvoPDg52KbNw4UL69u3rPLqpivfVNunpsG0bnD1r3edzXlPr4GCH3RGqKuaJVnAGeAfYLiI+dLNB9WjXrh35+fkeuy7Upk0bNm7ciJyXJDdu3EibNm2czzt37sy2bdtKrD9p0iTnL+RWrVqVWL5lyxaaNGni9kt+zJgxJCcnO6dvvvnG5XlycjLdunXzwF5aRzpdu3Z1NmEvsmzZMnr27Fnqejk5OezYsYPY2FiPxHG+8tZ/eezatYuwsDBuueUWOnfu7GwwUNzHH3/MsGHDnM+r4n21xenTsHs35OTgSDuh9/nUUp44BXcdcB/wkzEmuXDesyLiO21bPSAtLY0777yTBx54gI4dOxIWFkZSUhJTp04lISGB8MJrWGfPni3xq97f39/ll+np06dJLnZtCqzTRC1atGDs2LHMmDGDRx55hFGjRhESEsLixYuZP38+CxcudJYfMGAATz/9NGlpac5rIZMnT+aNN95g0aJF1KtXzxlHZGSk8xTSmjVrGDhwoNt9jIqKcmkxVpFTcJmZmaSkpADWtaMDBw6QnJxMVFQUzZo1A2DGjBnMmDGDHTt2APDEE09w33330b17d6677jreeustDh06xJgxY5zbHT9+PDfffDPNmjXj6NGjTJo0iTNnzjBixAi3cVRX/ZdHfn4+iYmJJCcn06RJE2699VYWLFjAbbfdBlhHVd9//z2ffPKJc52qeF+rXUaGdbgjgjRsREpGY05naPKplUSk2qeuXbtKabZt21bqsqp2+vTpi143JydHnnnmGenWrZtERkZKnTp1pFWrVvL4449LWlqaiIiMGDFCgBJTXFycczullRk6dKizzLp166R///7SqFEjCQ8Pl+7du8uCBQtKxNSjRw+ZMWOGiIg4HA4JDw93u+3ly5eLiEh2draEh4fL2rVry7XPFamvlStXun3tESNGOMtMnDhRrI/kOTNnzpTmzZtLUFCQdOnSRVatWuWyfNiwYRIbGyuBgYHSpEkTuf3222Xr1q1uY6ju+l+5cqVzvR07dsiWLVtclq9du1YSEhKcz998802ZMGGC8/k777wj9913X4ntVvX7WqUyM0U2bhRZv14ce/bKzh0OWb9eJDlZJCvLtWhl/h/Lw87vmqqwcuVKu0NwC0iSUnKBJqBiqvoDX92WLFkirVu3lvz8/HKVnzFjhvTr16/c269p9eVpF0pA+fn50qlTJzl+/Lg4HA4ZNmyYfPTRR87l/fv3ly+//LLEdqv6fa0yWVkimzZZySclpczkI6IJqKJ8MQFpVzw12MCBAxk3bhwHDx4sV/nAwECmT59exVHVDn379uXOO+9k8eLFXHLJJWzatAmATp06Ocv4+/vz6quv0qdPHzp27Eh0dDR33XUXAMePHyc5OZl+/fqV2LbPvq+nTkF+PhIRwa78lpzOMHrarZYzUokWXxerW7dukpSU5HbZ9u3badu2bTVHZKlMs+LaSOur/Hbu3El+fj7t27e3OxRbOdJOsPt4JKcy/C6YfKr682Xnd01V8NZm2MaYDSLitnWSjgeklKo6eXnWbQ1BQVZT67QobXCgnDQBKaWqRn4+/PwzFBTguLwNKb8Ea1Nr5UKvASmlPK8o+WRnI35+7N3vp8lHlaAJSCnlWQUFFPWpI8HB7AlozcnMQE0+qgSvTEB2NIxQSnmAw2H1cJCZiQQGsTewNSczg7wu+eh3jHfwugQUGBjovV3GK6VKJwJ79sDp00hAAPuCWnMiM9jrkg9AdnY2gYGBdodR63ldAmrUqBGpqalkZWXprxSlfIkxUK8e4u/P/uDWpJ0J8brkIyJkZWWRmppKo0aN7A6n1vO6VnBFfaYdOnSIvLy8an3tnJwcZ99Z6sK0vsrvyJEj1p3f5RwN1leJwPH0QLLO7sfPDxo3hn37Lm5bVfX5CgwMJCYmxvldo+zjdQkIrCRkx4cjMTGxRo0PUtW0vspv7NixpKenl+jk1OeJwJQpMHw42Y1bMmQILF8OMTGwYgW0a3fxm9bPV81Xs3+OKaWq1gsvwPPP40joy+2Dcz2WfFTtoAlIKXVxXnkFJk9G/P15Kex1lq4I0uSjKkQTkFKq4mbMgGeeQYzhlSvm8efNt2ryURWmCUgpVTFz58IjjwDwtytm8+zW32ryURdFE5BSqvy2b4ff/x6AWZdP48ntozT5qIvmla3glFJeqm1bcl95nQ/fyuDhXY9r8lGVoglIKXVhBQXg7092Ngz5+jGW79HWbqry9BScUqps330HHTqQ89Muj97no5QmIKVU6TZuhEGDYPt2vrzpTU0+yqM0ASml3Nu2Dfr3h1OnWN1oKMN+eU2Tj/IoTUBKqZJ274a+fSEtjXUNbqTv0Q+JjgnQ5KM8ShOQUsrVL79AQgIcPkxy/Xh6p31KVEyQJh/lcZqAlFKuvvoK9u9ne/g1XH9yERExdTT5qCqhCUgp5SL7t79nypUf0vP0EurFhGnyUVVG7wNSSsHp03DiBNkxLaym1lvu0QYHqsppAlKqtsvKgptvRnal8IcWy1m+tq0mH1UtNAEpVZudPQu33w6rV3M8OI4Vh0M0+ahq45FrQMaYfxljjhpjtnhie0qpapCfD/fcA199xcnAhlx/djnZMS01+ahq46lGCO8CAz20LaVUVXM4YORIWLCAjIBI4vOWkR5zhSYfVa08koBEZDVwwhPbUkpVMREYOxY++IAs/1D65S/h15irNPmoaldt14CMMaOB0QAxMTEkJiZW10uXW2ZmplfG5a20vsovPT2dgoICr6mvxiaIFqYOgwo+5+f6XZj2yjqOHs3i6FG7IztHP18V44v1ZUTEMxsypgXwhYhceaGy3bp1k6SkJI+8riclJiYSHx9vdxg+Q+ur/OLj40lPTyc5OdnuUKwhFYbAtuWpFMTEee2Rj36+KsZb68sYs0FEurlbpjeiKlVbzJ5Nzo87nUMqeHPyUbWDNsNWqjaYNQsefpgzQY35IXcnMTHhmnyU7TzVDHs+sBZoY4w5aIx50BPbVUp5wHvvwcMPA/Bc7kTqavJRXsIjR0Aico8ntqOU8rBPP0V+9zsMMJ7X+CxmjCYf5TX0GpBSNdWSJcg992AcDl5kIu/HjNfko7yKJiClaqLUVGToUExeHn/lSd5qNFGTj/I62ghBqRooOyqOOc2nEbhjM39t9BorVhpNPsrraAJSqiYRITvHWE2td4zRjkWVV9NTcErVFDt24LiqM3+4YRvLl6PJR3k9TUBK1QR79yIJffH76UcSvv+LJh/lEzQBKeXrUlNx3JCAOZTKaq7nuUZva/JRPkETkFK+7OhRHAl98du3l3VczQMNv+DLlXU1+SifoAlIKV918iSOfgPw27mDzXTg/oZLWZQYrslH+QxNQEr5qNwvl+G3OZmdtGZ49DL+mxilyUf5FG2GrZQPys6GIfPuIpoCdkb/hn+vitHko3yOJiClfEluLjm7Uxnyx5aFTa3v0QYHymfpKTilfEV+PgXDhpN91TUcW56sTa2Vz9MEpJQvcDjIH/kg/p99isnLpWGUQ5OP8nmagJTydiLkj/kDAR+8Ryb1uLf+Yt5Y00WTj/J5moCU8mYi5D35NAH/nEUOwYyMXMjUb3pq8lE1giYgpbxY3sS/EPi318gjgFGR/+HP3yZo8lE1hiYgpbxUdjbM+CSGPAL4Q8T7PPPtYE0+qkbRZthKeaHsbAqHVBjN/Oh+vLuqpSYfVePoEZBSXubsB5/wxxu2OIdU0OSjaio9AlLKi5z9aAEB993DKxLB99Fb+XhFY00+qsbSIyClvMTZRV9hht+NvxTwXr2xfLxKk4+q2TQBKeUFzi5bjdx2G0GSy5y6jzLgh0mafFSNpwlIKZvlrF5H/o2DCXFk836d3/ObdX+jXXtjd1hKVTlNQErZKPtwOmf7DqJeQQb/DbmHLj+8pclH1RraCEEpm2Rnw5D7I2mS9zp3Bi/iiu/n0a6Dv91hKVVt9AhIKRtkZ4l1n89y+CpmBJdu+A/trgq0OyylqpUmIKWqWfbuQ+xpcp3rkAp62k3VQpqAlKpG2QeO8etV/Wh/ai3/CBqvQyqoWs0jCcgYM9AYs9MYk2KM+X+e2KZSNY3kFXDwygG0OLONHf7tiVnxkSYfVatVuhGCMcYfmAn0Aw4C640xi0RkW2W3rVRN4cgXAnfs53JHBnv8W+H3v2W0vi7a7rCUspUnWsF1B1JEZA+AMeYj4Bag1AS0c+dO4uPjPfDSnpWenk5kZKTdYfgMra/yKSiAjd99T6Dkci1BONo3oM7Ee+wOy+vp56tifLG+PJGA4oBfij0/CFxzfiFjzGhgNEBgYCDp6ekeeGnPKigo8Mq4vJXW14UVFBgOpfgRKLkAnGneAj+Tw9n0HJsj8376+aoYX6wvTyQgd813pMQMkTnAHIBu3bpJUlKSB17asxITE73yyMxbaX2V7eRJGDAATuRATEAH4qLS2bBvp91h+Qz9fFWMt9aXMaW38PREAjoINC32/BLgkAe2q5TPOnQIBg2CH3+Eli2hUaMG5OToTaZKFeeJVnDrgcuNMS2NMUHA3cAiD2xXKZ+0bRtcey3k/LiD25omsWoVhITYHZVS3qfSR0Aikm+M+QPwFeAP/EtEtlY6MqV80OrVcMstkJ4ubAgbQ+eDqzHffmh3WEp5JY/cByQii0WktYhcJiKTPbFNpXzN/PnQrx+kp8PMTm/TJWMVpkED6N/f7tCU8kraE4JSlZSfD089BcOHQ24uPDvyEGP3PmUt/Mc/ICrK3gCV8lLaG7ZSlZCWBsOGwf/+BwEBMO114Q8rx2FOnYKbboK777Y7RKW8liYgpS7Shg1wxx2wbx80agSffAK9jn4Kn30GYWEwaxaU0QRVqdpOE5BSFeRwwLRp8OyzkJcHV18Nn34KTeMccOUEq9Crr0LTpmVvSKlaThOQUhVw+DCMGAHLllnPH3kEpk4tambtZ42tMGsWPPSQnWEq5RM0ASlVTv/5D4wdC8ePQ3Q0zJ0LgwefV6hxY3jpJVviU8rXaCs4pS7gyBEYOhTuvNNKPn37wubNxZLPgQMwZYrVHE4pVW56BKRUKUTgvffg8cetft1CQ+G112D0aPAr+ulWUAD33WfdgZqZaSUipVS5aAJSyo1Nm6zrO99+az0fOBBmz4Zmzc4rOHWqlXxiYqxMpZQqNz0Fp1QxaWnWdZ5u3azk06gRzJsHixe7ST7r1sGEwlZv8+ZBw4bVHq9SvkyPgJQCsrJg+nR45RWrKx1/f3jsMXjxRYiIcLPCsWPWTUD5+fDoo9a4C0qpCtEEpGq1/HyrNduLL1pDKAAkJMAbb0D79mWsdM898MsvcM011j0/SqkK0wSkaqXcXKuBwSuvwO7d1rzOna3n/fpdoAODrCyrFUKjRlbb7ODgaolZqZpGE5CqVbKy4F//stoO/FI4kHyrVjBpEtx1V7HWbWUJD4clS2DvXrjkkiqNV6maTBOQqhUOHoSZM2HOHDhxwprXvr3Vnc5dd1kdiV7Qvn0QG2sd8fj7W5lLKXXRNAGpGksEVq2ymk9/8ol1yw5Yfbc984w1cFy5jnjA6oOnd2+49FJYsAAiI6sqbKVqDU1AqsZJTYV337UaFxRd3/H3t450HnsMevSoYCfVmZlWtwcHDkBcnF7zUcpDNAGpGiEry7pXZ+5cWLrU6rEarEs0I0davRdcVOfU+fnWgD8bN1qn3BYuhDp1PBm6UrWWJiDls4qSziefwJdfwpkz1vzAQKvvtgcesFq0+ftf5AuIWN0hLF4MDRpYDQ/0ZlOlPEYTkPIpJ07A11/Df/9rJZ2srHPLrr7aGhb73nut3qor7fnn4a23rFNuixZpowOlPEwTkPJqDoc18uiSJda0bt2502tg3Qd6551WpwTNm3vwhQsKYNcu6/Dpo4+gZ08PblwpBZqAlJdxOGD7dqv12qpVsHKl1etNkcBAiI+Hm26ykk6J/tk8xd8fPvzQyniafJSqEpqAlK1yciA5Gb7/3upUevVqq0PQ4po1gxtvtKYbboCwsCoMaMEC68JRaKh1c5AmH6WqjCYgVW3y8mDnTkhKsg4s1q2DH38sOY5bkybWLTdFU5s2FWw2fbH++ld46im47jpITCzn3alKqYul/2HK40Sse3E2b4affjo3bd9uJaHijIErr7QaEPzmN+fu9ayWhFM84BdegMmTref33KPJR6lqoP9l6qKdOePPhg3Wtfri044d1pAG7lx2GXTqZDUe6N4dunSp4lNqF+JwWMMpzJhhXfeZO9ca4VQpVeU0ASm3RKwkcuCANf3yy7m/+/dDSgr8+uv1pa4fFQUdO0KHDuf+tm9vXVrxGmfPwoMPwgcfQFAQfPwx3Hqr3VEpVWtoAqplHA7rXppff3U/paaeSzZFN3aWJjDQQevWflx+OS5T69ZWn53VehrtYsyZYyWfevWsHg4SEuyOSKlaRROQj3I4ICPDSiYnT577W/xx8XlpaXD0qDUVdcp5IaGhVgu0Zs2sbmyK/73sMkhJWc0NN8RX6X5WqYcfhi1brDG4O3WyOxqlap1KJSBjzJ3Ai0BboLuIJHkiqJpAxDrDk5MD2dkX/puZaU0ZGdbk7nHxeRc6OilL/foQE+N+io09l2giIso+itmz5+JjsM2KFdb5wIYNrWs+s2fbHZFStVZlj4C2ALcDFfovLiiA48etX/EFBdZUXY/z8s5Nubmuz3fvvpSFC90vK2s9d0klJ6eSNVsO9epZ11qioqyk4u5v8ceNGllTUFDVx+Z1HA74y1+ssbevvdZKRNqrtVL2EpFKT0Ai0K0C5cWapNg0uHDeomLzZhfOG1VsXmrhvNjz1u9SOD+p2LyJhfMmFpuXVDivy3nrxxbOTy02b1ThvNnF5i0qnDf4vPVL7pMx1j5FRi6S5s1FrrhC5JJLrH2KjR0lN94ocvvtIrfeau1TvXqxMnGiyF//KvLWWyItWlj79NZbSbJ+vcj27SKPP27t0wsvTJQiSUnWPnXp0kWKi4219ik1NdU5b9Qoa59mz57tnLdokbVPgwcPdlm/aJ+KGzzY2qdFixbJypUrRURk9mxrn0aNGuUsl5qaWrifsS7rd+li7VNSUpJz3sSJ1j5NnFhF+3TsmMiAASLO9wiRggK3+1TE0/vUu3dvufzyy215n6pqn0Ts++zpPpXcp6L/R2/bJyBJSskF1XYNyBgzGhhdfF5kZC7GgL+/cPq0g9xciIo6S926WRgDZ86c5cQJCAvLo3HjDPz9hfz8DFJSICBAaN8+HT8/wd9f2LatgKws6NgxnbCw4/j5CQcOnGH/fmjZMpNWrY7g5wdnzhzjm28gMjKPAQMO4O8vBAQIn3xSQHY23H//XiIizhAQIPzvf+ls3gw33XSIq6/eQkCA8PPPe3nvPWjf/hRjx24kIEAICnLwwAPWPi1c+A1BQQ6Cghw8/3waa9fCU0/9RM+eVlvjzz/fybRp0KXLIcaPTwTg+PHjfPYZhITkEh+f6KyfwMCMwkdJZGZmkJkJJ07sA+DAgX0kJlpld+7cCUBGRoZzHkBubi4A3333HdGFvXMeOnTIuU5R2Z9++gmAtLQ0l/WLFJ+XVthNwU8//UTHjh1JTEx0vv6hQ4ecZY8fP+6Mofj6GRnWPiUlJTkf79u3z/nX0/t0au9ectq3J+ToUfLCw+H0aWufVq92u09hhW3CPb1P6enpOAo7savu96mq9qn4NnWf7N+nuLg4EhMTvXKfSmOsBFVGAWOWA43dLHpORBYWlkkExks5rwF169ZNkpK873JRYmIi8fHxdofhM7y+viZMsE67iVij0P373xc5KFDlxcfHk56eTnJysi2v74u8/vPlZby1vowxG0Skm7tlFzwCEpG+ng9JqWoQHGy1onjmGZg4sZZe/FLKe2kzbFVznD0L27ZB587W86eftnow7dLF3riUUm75VWZlY8xtxpiDwLXAl8aYrzwTllIVtGaNdS9PQoJ1sxNY/blp8lHKa1UqAYnIAhG5RESCRSRGRAZ4KjClyuXXX+H3v4devaxO6Bo2hCNH7I5KKVUOlUpAStnm7FmYOtXq++edd6zrOxMnWuM7dOxod3RKqXLQa0DKN917L/znP9bjm26C11+3Bg5SSvkMPQJSvkHEeQ8PYA2hcOWV8NVX8MUXmnyU8kGagJR3czisBNO9O867fcEave7HH6F/f/tiU0pVip6CU94pLw/mz7eu82zdas07eNAapCgy0nrup7+flPJl+h+svEt6Orz6qjXew4gRVvKJi7Ou8ezefS75KKV8nh4BKe9y9iy88IJ1BNS2LfzpTzB8uPZioFQNpAlI2ef4cfjoI/jyS+s6j7+/NSjRyy9byWfgQD3NplQNpglIVa/MTFi0CD780GrBlp9vzV+61GpODfDkk/bFp5SqNpqAVPU4fRrGjIGFCyEry5rn72/11TZyJPTrZ2t4SqnqpwlIVY3Dh+G772DoUOt5WBisXWsln549res6d95pDdGqlKqVNAEpzzh7Fr7/Hv73P+t02vr11vxDhyA21hoW4d13oVkzaNnS1lCVUt5BE5CqnD17YOxYqzfq7Oxz80NCoG9fOHXKSkAAvXvbE6NSyitpAlLlc/y4dYSzdq117ebPf7bmR0dbRz0FBVbXOAkJ56a6de2NWSnl1TQBKfd+/NFqHp2cDJs2QUrKuWWNGsFLL1mPw8Otcp06WU2olVKqnDQB1WanT8POndY4Olu3wi23wLXXWstWrIDnnjtXNiTE6o/t2mutRgQi55YN0GGglFIVpwmopsvNde1F4JFHYMsWK/EcPuxatm7dcwkoPh4ee8w6srnqKmjfHgIDqylopVRtoAmoJjh82DplduAA7N0L+/ad+5ufb12/KfLVV7Brl/U4JARat4YrrrB6Hih+JNO5szUppVQV0QTkzdatg2PH4OhRK8kcOnRueuIJuOsuq9znn8NDD7nfRmCg1ftAaKj1/K9/heBga/ycZs20qxullG00AVUFhwMyMqwmyOnp0Lw5RERYy779FlatspadOgVpadYRyvHjVplvvjm3nX79XAdhK27nznOP27Sxmjw3awYtWpybWra0mkD7+58rO2SIZ/dVKaUuUu1LQHl5cPKkdePkeVNkcjL06GGdmgLrdNWOHXDmjDVlZZ173K7duYv0x45ZF+azsqyjjYwM14v0ixbBzTdbj5cvhxdfdB9bVJTr8z59rNiio6FJk3NTbKx16qxI7956j41SyufYk4COHLF6PM7PLznde++5aw+LF1u9JZ9fJi/PShILFpzb5q23Wtc8cnNLJpdHH4VJk6xyK1eW2mqrE1hHCJdeas345z/h00/d78MNN5xLQEFBrs2UwTrlFRlpHdUUv3h//fXw9NPW/IgIaNDASjBFU3GffVZWLSqllG8TkWqfAME6RnBOgwvnLXrySSkye+hQAWRUsXKpheVijZHiugQHCyBJxcpOLCw7sUcPZ7mkt98WQLoEBoq0bClyxRUiV10lsYGBAkjqunXOsqN69RJAZg8cKDJpksjrr8uihx8WQAZfe+25Fy8oEOc+nTghkp8vIiKDBw+29mnRonP7NHu2tU+jRjnnpaamWvsUG+u6T126WPuUlOScN3HiRGufJk48t09JSdY+denisn5sbKy1T6mp5/Zp1Chrn2bPds5btGiRtU+DB7us79ynYorv08qVK2vcPhXx9D717t1bLr/88hq1TyI1733y5X0q+n/0tn0CkqSUXGDvFejnn7dOR/3lL1ZLLDh39AHWnfVgnYr6+GPraORf/7LmnX+6qmi9jz+2WnkdOADjx1vz+vY9V65TJ+tvhw5WNzLbt1s3WxYdfcTFnSvbpo3197bbrFifeMIaowasI5cixS/k16/ves1FKaWUW0aKX6uoJt26dZOkpKRqf90LSUxMJD4+3u4wfIbWV/nFx8eTnp5OcnKy3aH4DP18VYy31pcxZoOIdHO3TNvgKqWUsoUmIKWUUrbQBKSUUsoWmoCUUkrZolIJyBjzmjFmhzFmszFmgTEm0kNxKaWUquEqewS0DLhSRDoCPwPPVD4kpZRStUGlEpCIfC0i+YVPvwcuqXxISimlagNPdsXzAPBxaQuNMaOB0QAxMTEkJiZ68KU9IzMz0yvj8lZaX+WXnp5OQUGB1lcF6OerYnyxvi6YgIwxy4HGbhY9JyILC8s8B+QDH5S2HRGZA8wB60ZUb7xhyltv5PJWWl/lFxkZSXp6utZXBejnq2J8sb4umIBEpG9Zy40xI4DBQILY0a2CUkopn1SpU3DGmIHA00BvEcnyTEhKKaVqg8q2gpsBhAHLjDHJxpi3PBCTUkqpWqBSR0Ai0spTgSillKpdtCcEpZRSttAEpJRSyha2jAdkjDkG7K/2F76waOC43UH4EK2vitH6qhitr4rx1vpqLiIN3S2wJQF5K2NMUmkDJ6mStL4qRuurYrS+KsYX60tPwSmllLKFJiCllFK20ATkao7dAfgYra+K0fqqGK2vivG5+tJrQEoppWyhR0BKKaVsoQmoFMaY8cYYMcZE2x2LN9NRcS/MGDPQGLPTGJNijPl/dsfj7YwxTY0xK40x240xW40xj9odk7czxvgbYzYZY76wO5aK0ATkhjGmKdAPOGB3LD5AR8UtgzHGH5gJ3Ai0A+4xxrSzNyqvlw88KSJtgR7AOK2zC3oU2G53EBWlCci9vwF/AvQC2QXoqLgX1B1IEZE9IpILfATcYnNMXk1EDovIxsLHGVhfrHH2RuW9jDGXADcBb9sdS0VpAjqPMWYIkCoiP9odiw96AFhidxBeJg74pdjzg+iXabkZY1oAnYEfbA7Fm/0d6wezw+Y4KsyTQ3L7jLJGeQWeBfpXb0TezVOj4tZSxs08PbIuB2NMKPAp8JiInLY7Hm9kjBkMHBWRDcaYeJvDqbBamYBKG+XVGNMBaAn8aIwB63TSRmNMdxE5Uo0hehUdFbdSDgJNiz2/BDhkUyw+wxgTiJV8PhCR/9odjxe7DhhijBkEhADhxpj3ReRem+MqF70PqAzGmH1ANxHxxg7+vELhqLjTsEbFPWZ3PN7GGBOA1TgjAUgF1gPDRWSrrYF5MWP9+psHnBCRx2wOx2cUHgGNF5HBNodSbnoNSFWWjopbhsIGGn8AvsK6mP5vTT4XdB1wH3BD4WcqufAXvqph9AhIKaWULfQISCmllC00ASmllLKFJiCllFK20ASklFLKFpqAlFJK2UITkFJKKVtoAlJKKWULTUBKKaVsoQlIqSpkjPlT4cCG509/tjs2peymPSEoVYWMMWFAvWKzxgO/Ba4XkRR7olLKO2gCUqqaGGOeBv4I3CAiO+2ORym71crhGJSqbsaYZ7A6Je0jIj/bHY9S3kATkFJVrHCwvjFYQ1boaTelCmkCUqoKGWNeAEYB8SKy2+54lPImmoCUqiKFRz6PAkOAM8aYomHN00Ukx77IlPIO2ghBqSpQOKpnOhDuZnFfEflf9UaklPfRBKSUUsoWeiOqUkopW2gCUkopZQtNQEoppWyhCUgppZQtNAEppZSyhSYgpZRSttAEpJRSyhaagJRSStlCE5BSSilb/H9ma6MAFqxXiAAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# extra code – this cell generates and saves Figure 11–3\n",
"\n",
"from scipy.special import erfc\n",
"\n",
"# alpha and scale to self normalize with mean 0 and standard deviation 1\n",
"# (see equation 14 in the paper):\n",
"alpha_0_1 = -np.sqrt(2 / np.pi) / (erfc(1 / np.sqrt(2)) * np.exp(1 / 2) - 1)\n",
"scale_0_1 = (\n",
" (1 - erfc(1 / np.sqrt(2)) * np.sqrt(np.e))\n",
" * np.sqrt(2 * np.pi)\n",
" * (\n",
" 2 * erfc(np.sqrt(2)) * np.e ** 2\n",
" + np.pi * erfc(1 / np.sqrt(2)) ** 2 * np.e\n",
" - 2 * (2 + np.pi) * erfc(1 / np.sqrt(2)) * np.sqrt(np.e)\n",
" + np.pi\n",
" + 2\n",
" ) ** (-1 / 2)\n",
")\n",
"\n",
"def elu(z, alpha=1):\n",
" return np.where(z < 0, alpha * (np.exp(z) - 1), z)\n",
"\n",
"def selu(z, scale=scale_0_1, alpha=alpha_0_1):\n",
" return scale * elu(z, alpha)\n",
"\n",
"z = np.linspace(-5, 5, 200)\n",
"plt.plot(z, elu(z), \"b-\", linewidth=2, label=r\"ELU$_\\alpha(z) = \\alpha (e^z - 1)$ if $z < 0$, else $z$\")\n",
"plt.plot(z, selu(z), \"r--\", linewidth=2, label=r\"SELU$(z) = 1.05 \\, $ELU$_{1.67}(z)$\")\n",
"plt.plot([-5, 5], [0, 0], 'k-')\n",
"plt.plot([-5, 5], [-1, -1], 'k:', linewidth=2)\n",
"plt.plot([-5, 5], [-1.758, -1.758], 'k:', linewidth=2)\n",
"plt.plot([0, 0], [-2.2, 3.2], 'k-')\n",
"plt.grid(True)\n",
"plt.axis([-5, 5, -2.2, 3.2])\n",
"plt.xlabel(\"$z$\")\n",
"plt.gca().set_aspect(\"equal\")\n",
"plt.legend()\n",
"\n",
"save_fig(\"elu_selu_plot\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using SELU is straightforward:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"dense = tf.keras.layers.Dense(50, activation=\"selu\",\n",
" kernel_initializer=\"lecun_normal\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Extra material – an example of a self-regularized network using SELU**\n",
"\n",
"Let's create a neural net for Fashion MNIST with 100 hidden layers, using the SELU activation function:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"tf.random.set_seed(42)\n",
"model = tf.keras.Sequential()\n",
"model.add(tf.keras.layers.Flatten(input_shape=[28, 28]))\n",
"for layer in range(100):\n",
" model.add(tf.keras.layers.Dense(100, activation=\"selu\",\n",
" kernel_initializer=\"lecun_normal\"))\n",
"model.add(tf.keras.layers.Dense(10, activation=\"softmax\"))"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"model.compile(loss=\"sparse_categorical_crossentropy\",\n",
" optimizer=tf.keras.optimizers.SGD(learning_rate=0.001),\n",
" metrics=[\"accuracy\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's train it. Do not forget to scale the inputs to mean 0 and standard deviation 1:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"fashion_mnist = tf.keras.datasets.fashion_mnist.load_data()\n",
"(X_train_full, y_train_full), (X_test, y_test) = fashion_mnist\n",
"X_train, y_train = X_train_full[:-5000], y_train_full[:-5000]\n",
"X_valid, y_valid = X_train_full[-5000:], y_train_full[-5000:]\n",
"X_train, X_valid, X_test = X_train / 255, X_valid / 255, X_test / 255"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"class_names = [\"T-shirt/top\", \"Trouser\", \"Pullover\", \"Dress\", \"Coat\",\n",
" \"Sandal\", \"Shirt\", \"Sneaker\", \"Bag\", \"Ankle boot\"]"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"pixel_means = X_train.mean(axis=0, keepdims=True)\n",
"pixel_stds = X_train.std(axis=0, keepdims=True)\n",
"X_train_scaled = (X_train - pixel_means) / pixel_stds\n",
"X_valid_scaled = (X_valid - pixel_means) / pixel_stds\n",
"X_test_scaled = (X_test - pixel_means) / pixel_stds"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2021-12-16 11:22:44.499697: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/5\n",
"1719/1719 [==============================] - 13s 7ms/step - loss: 1.3735 - accuracy: 0.4548 - val_loss: 0.9599 - val_accuracy: 0.6444\n",
"Epoch 2/5\n",
"1719/1719 [==============================] - 12s 7ms/step - loss: 0.7783 - accuracy: 0.7073 - val_loss: 0.6529 - val_accuracy: 0.7664\n",
"Epoch 3/5\n",
"1719/1719 [==============================] - 12s 7ms/step - loss: 0.6462 - accuracy: 0.7611 - val_loss: 0.6048 - val_accuracy: 0.7748\n",
"Epoch 4/5\n",
"1719/1719 [==============================] - 11s 6ms/step - loss: 0.5821 - accuracy: 0.7863 - val_loss: 0.5737 - val_accuracy: 0.7944\n",
"Epoch 5/5\n",
"1719/1719 [==============================] - 12s 7ms/step - loss: 0.5401 - accuracy: 0.8041 - val_loss: 0.5333 - val_accuracy: 0.8046\n"
]
}
],
"source": [
"history = model.fit(X_train_scaled, y_train, epochs=5,\n",
" validation_data=(X_valid_scaled, y_valid))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The network managed to learn, despite how deep it is. Now look at what happens if we try to use the ReLU activation function instead:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"tf.random.set_seed(42)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"model = tf.keras.Sequential()\n",
"model.add(tf.keras.layers.Flatten(input_shape=[28, 28]))\n",
"for layer in range(100):\n",
" model.add(tf.keras.layers.Dense(100, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\"))\n",
"model.add(tf.keras.layers.Dense(10, activation=\"softmax\"))"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"model.compile(loss=\"sparse_categorical_crossentropy\",\n",
" optimizer=tf.keras.optimizers.SGD(learning_rate=0.001),\n",
" metrics=[\"accuracy\"])"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/5\n",
"1719/1719 [==============================] - 12s 6ms/step - loss: 1.6932 - accuracy: 0.3071 - val_loss: 1.2058 - val_accuracy: 0.5106\n",
"Epoch 2/5\n",
"1719/1719 [==============================] - 11s 6ms/step - loss: 1.1132 - accuracy: 0.5297 - val_loss: 0.9682 - val_accuracy: 0.5718\n",
"Epoch 3/5\n",
"1719/1719 [==============================] - 10s 6ms/step - loss: 0.9480 - accuracy: 0.6117 - val_loss: 1.0552 - val_accuracy: 0.5102\n",
"Epoch 4/5\n",
"1719/1719 [==============================] - 10s 6ms/step - loss: 0.9763 - accuracy: 0.6003 - val_loss: 0.7764 - val_accuracy: 0.7070\n",
"Epoch 5/5\n",
"1719/1719 [==============================] - 11s 6ms/step - loss: 0.7892 - accuracy: 0.6875 - val_loss: 0.7485 - val_accuracy: 0.7054\n"
]
}
],
"source": [
"history = model.fit(X_train_scaled, y_train, epochs=5,\n",
" validation_data=(X_valid_scaled, y_valid))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Not great at all, we suffered from the vanishing/exploding gradients problem."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### GELU, Swish and Mish"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAADsCAYAAAAy23L6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABbDElEQVR4nO2dd3hURdfAf5PeSCMkEECKIUiXYoiAEHpVFLChIIIiCIIKSrGAH0XBV1+VgIKgqPBSFUSqoCRUKUoLHamBQEhICOltvj9udrObbJINKZsyv+e5z+6de+7Mmbt377kzc+aMkFKiUCgUCkVpY2VpBRQKhUJROVEGSKFQKBQWQRkghUKhUFgEZYAUCoVCYRGUAVIoFAqFRVAGSKFQKBQWocgGSAhRWwixUwhxWghxUggx3oSMEEJ8JYS4IIQ4LoRoVdRyFQqFQlG+sSmGPNKBCVLKf4QQVYC/hRDbpZSnDGR6Aw2ytrbA11mfCoVCoaikFLkFJKWMkFL+k/X9HnAaqJlDrD/wo9T4C3AXQtQoatkKhUKhKL8URwtIjxCiLtASOJDjUE3gmsF+eFZahIk8RgIjARwcHFo/8MADxalimSczMxMrq8o1NKfqXPG5du0aUkoq2/8ZKt9vDXDu3LkoKWW1guSKzQAJIVyAn4E3pZRxOQ+bOMVkDCAp5SJgEUDDhg3l2bNni0vFckFISAhBQUGWVqNUUXWu+AQFBREbG8vRo0ctrUqpU9l+awAhxBVz5IrFLAshbNGMz3Ip5S8mRMKB2gb7tYAbxVG2QqFQKMoOUVHmyxaHF5wAlgCnpZSf5yG2ARia5Q0XCNyVUubqflMoFApF+eXqVWjZ0nz54uiCaw8MAU4IIY5mpU0FHgCQUn4DbAb6ABeARODlYihXoVAoFGWEyEjo3h3Cw80/p8gGSEq5B9NjPIYyEhhT1LIUCoVCUba4GHORLadDWTL2Zc6dgxYt4Ngx884tVi84hUKhUFQeIu5F0O3H7lyKvQjOt/Dzm8y2bVC9unnni7K8IF1+XnCZmZmEh4eTkJBQylqVLMnJyTg4OFhajVJF1bnic/PmTaSU1KhhPP3P1tYWb29vXF1dLaRZyVORveCG/PISy078qO2kOxD6zEk6NquPEOJvKWWbgs4vty2gqKgohBA0bNiwQvnY37t3jypVqlhajVJF1bniY2VlRXp6Oo0aNdKnSSlJSkri+vXrABXaCFVEMjMh/ddgENegzi6CO66hY7P6hcqj3D65Y2Nj8fHxqVDGR6GoTAghcHJyombNmkRGRlpaHUUhkBLeegtW/lgFp3WbmR/4O2O69yt0PuW2BZSRkYGtra2l1VAoFEXE0dGRtLQ0S6uhKAQzZsBXX4GdHfz6swPdunW5r3zKrQEC7Q1KoVCUb9T/uHwgpWTKH1NIOdGPL6Z1wMoKVqyAbt3uP0/Vf6VQKBSKAvko9CPm7J3DF9E9oMFmFi2CAQOKlqcyQAqFQqHIl+jEaL7c+422Y5tEq5dWMmJE0fNVBkihUCgU+XL2aFWS5u+BmLrUz+jF/smLiyVfZYAUCoVCkSdhYdC3L6RE+PF84j6OvbcWO2u73IJ//glnzhQqb2WAFADExMTg4+PDv//+a5b8oEGD+PzzvGLPlj0qev0UiuJGSsmVK9CzJ8TGwpNPwo8LauBi75xb+K+/4PHHoUMHuHzZ7DKUAbIAt27d4q233qJBgwY4ODjg7e1Nu3btmDdvHvHx8QAMGzYMIUSuLTAwUJ/PsGHD6Ncvb9/7oKAgxo4dmyt96dKluLi4GKXNnj2bPn368OCDD5pVh2nTpjFz5kzu3r1rlnxJcvjwYYQQXM7nxi/P9VMoSpujN48SsLA9nfuHc+MGdOqkebzZmPKbPnkS+vSBxETo1w8KseigMkClzOXLl2nVqhVbt25lxowZ/PPPP/z5559MnDiRP/74g82bN+tlu3XrRkREhNFmeLy4SExMZPHixYwoxKhis2bNqF+/PsuWLSt2fczl1KlTDB06lEGDBgHa9Ro+fDg5wzeV1/opFJbgwp0L9PypF4dv7edS5/Y81P4cv/4KJiNHXb4MPXpATAw88QQsXgyFCA5QYQyQEJbZCsvo0aOxsrLi8OHDPPfcczRu3JimTZsyYMAA1q9fz9NPP62Xtbe3p3r16kabp6dnMV41jc2bN2NlZUX79u2N0ufOnWuyFfbhhx8C8MQTT7BixYpi16egcgF++eUXWrRoQUJCAm+++SYAU6ZMITo6mmbNmrF+/foyWz+Foixz9PpJbsffAcDK6S7BC5NwczMheOuWtv6Crom0cmUeTaS8qTAGqDxw584dtm3bxpgxY3B2NtGPimUm5e3evZvWrVvnKnv06NFGra8JEyZQvXp1hg4dCkBAQAAHDx4kKSkpV56zZ8/GxcUl32337t0m9Smo3NTUVF577TV69uzJzz//TIcOHQDo2rUrv/76K926dWPkyJH62fUlUT+FoiKSkQGrZ/RHLv8NkeTF//ptpGuTFrkF09K0brcLF7QV6H79FRwdC11euY6EYEgZDuqt5/z580gpadiwoVF6rVq1iI2NBeDZZ59lyZIlAGzdujXXWM2YMWOYM2dOsep15cqVXFGKAapUqaIPmDlnzhxWrFhBSEgIfn5+APj6+pKWlsaNGzdyja2MGjWKZ555Jt9ya9asaTK9oHJPnDhBVFQUQ4YMMXn+kCFD2LJlCydOnKBVq1YlUj+FoqIhJYwbB2vWgKtrT7YMukS7Ni6mhW1t4fXX4T//ga1bMd1EKpgKY4DKM7t37yYjI4ORI0eSnJysT+/YsSOLFi0yknV3dy/28pOSkvDx8cnz+Mcff0xwcDA7d+7E399fn+6Y9cZjqoXg6elZ5O7CvMrVLSGSV2tRF6BWJ1cS9VMoKgqpGalYCStmzbBhwQKwt4cNG8jb+OgYMQJefFE74T5RXXCliJ+fH0IIzuTwla9Xrx5+fn44OTkZpTs5OeHn52e0eXl5mV2eq6urSS+u2NhY3AzeWLy8vIiJiTGZx6xZs1iwYAGhoaFGD2fQuhQBqlWrluu8onTBFVRu8+bNqVq1KsuXLzd57rJly/Dy8qJZs2YlVj+FoiKQkZnB0HVDaTVnINNnJmFlpQ3ldOpkQjgzEyZMgOPHs9OKYHxAtYBKlapVq9KjRw+Cg4N54403cnWvFTcNGzZk8+bNSCmNWgv//POPUTdgy5YtWbp0aa7zZ8yYwbfffktISIjJLqiwsDB8fX1Nti6K0gVXULl2dnYsWLCAwYMH8+yzz/LYY48BEBoayttvv83WrVtZuXIldnZ2JVY/haK8I6XkjS1vsOrkKi3hxd4Et9/Ck0+aGMuRUjM+X3wBq1ZpYz/FsKCiMkClzIIFC2jfvj2tW7dm+vTptGjRAhsbG/7++2+OHTtG586d9bIpKSncvHnT6Hxra2ujN/K4uDiOHj1qJOPu7k7dunUZPXq03ti9+uqrODg4sHnzZlasWMGvv/6ql+/ZsyeTJk0iOjqaqlWrAlrL4Msvv2TDhg04Ozvr9XB3d9ev5Ll792569eplsp732wVnTrkAzzzzDI0aNeKTTz5h7ty5gDZ3p2PHjhw7dozGjRuXaP0UiorAnYjsRQAfrd+MUSPyMCqzZ2vGx9YWvvuuWIwPoFnBsrr5+/vLvDh16lSex8o6ERERcty4cfLBBx+UdnZ20tnZWbZp00bOnj1bXr9+XUop5UsvvSSBXFvNmjX1+eQlM3DgQL3MwYMHZY8ePaS3t7d0dXWVAQEBct26dbl0CgwMlMHBwVJKKTMzM6Wrq6vJvHfs2CGllDIpKUm6urrK/fv3F/l6xMXFmV2uKQ4dOiQBeenSpTxlLFk/U+jqXFk4c+aMDAsLy/N4ef4/F8TOnTstrYJJDh6U0tlZStrPkQ9NHSzTMzJMC379tZQgpRBSrlplVt7AYWnGM97iRia/raIaoPyw1INpy5Yt0t/fX6anp5slHxwcLLt3714sZZdGnS1ZP1MoA2RMRf0/S1k2DdCZM1J6eWkW4MUXpUxPzzQtuHKlZnhAym++KTjjjAwpk5PNNkDF4oQghPhOCBEphAjL43iQEOKuEOJo1vahKTmF5ejVqxdjxowhPDzcLHlbW1vmzZtXwloVHxW9fgqFOVyOvcz161rwgqgo6N1b61GztjbhURoeDkOHauM/s2bBa6/ln3lGBrzyihY0zkyKawxoKRAM/JiPzG4pZeEXDVeUGuPGjTNbduTIkSWoSclQ0eunUOTHriu76PlTT6ocn8Ttq9N49FHBmjXasI5JatXSrNOxYzBlSsEFCKF5yoWGmq1TsbSApJS7gDvFkZdCoVAoipeTkSd5/H+Pk5yRzO0mH1Ft4Cw2bgSTAVkyM7O/v/ACzJ1rXtwxKytYsgQOHTJbr9L0gntUCHEMuAFMlFKeNCUkhBgJjARt/kVISIjJzNzc3Lh3714JqWo5MjIyKmS98kPVueKTnp4OkGedk5OT8/yvl3fi4+MtXrd7KcmI649AtT+wSqjOh089xPHjuXVyDA+n6QcfcGbyZO7liNhiCqvUVGqtWcO1Z55B5tmUygdzBorM2YC6QFgex1wBl6zvfYDz5uSpnBAqB6rOFR/lhGA5MjI0RwOsU6TDoJHy17+OmRa8elXKBx7QHA6eesq8zJ9/XpMfMcIomdJ0QjDDyMVJKeOzvm8GbIUQ5k/pVygUCkWhkRImToRly8DZwY5d7y7kibbNcwtGRkK3bnD1Kjz6KPyY33C+ARMnQoMGWhC5+6BUuuCEENWBW1JKKYQIQBt7ii6NshUKhaKyEZ8aT1hkGKHLA/nvfzVHg3Xr4JFHTAjHxmrLnp47By1awKZNkF+UFimzx4RatYJTpwq9DIOO4nLDXgHsBxoKIcKFECOEEKOEEKOyRAYBYVljQF8Bz2U10xQKhUJRjKSkpzBg1QAeWxLE5KXrEQJ++klbuicXCQnQty8cPQr+/rBtG3h45J15dDR07AiGY1r3aXygmFpAUsrnCzgejOamrVAoFIoSZMLvE9h+cbu288wgpnqG8eyzD5kW3rUL/voLateG7duhoLiH33wDe/bAm2/CP/8UavVTU6hYcAqFQlGBeMx6IgvubEN6XqATHzJzfB7GB7SZqGvXQpMm8MADBWc+ZQokJWlrARXR+IAyQAqFQlFhOHYMRj5TF5mxh0dfW8afn76dWygzE65cgXr1tP2nnso/0/BwqFZNW3rBygpmziw2fdV6QOWcYcOG0a+feQEmgoKCGDt2bIFyMTEx+Pj48O+//5qV76BBg/j888/NkrUkFbVeCgXAxYuaL0FcHAzq7cPuuROwssoxgVRmLXvaqhXs319wphcuQLt28Oyz2jLcxYwyQKXM7du3ef3116lbty729vb4+PjQtWtXtm/ffl/5ffnllyxbtqxYdZw9ezZ9+vQxexnqadOmMXPmTJOL35UlKmq9FJWbn0/9zKHzl+nRA27dgi5dNLdra2sTwu+/D/PnQ2KithVEYiLEx8Pt21rXWzGjuuBKmYEDB5KYmMiSJUvw8/MjMjKS0NBQoqPvzyvd7T7XYs+LxMREFi9ezG+//Wb2Oc2aNaN+/fosW7aMMWPGFKs+xUVFrZeicvPb2d94du2zWCX6kBb3O61bN2H9+jwWKp07V1vXx9oaVq+Grl0LLqB5c81RoW7d/F2z75OK1QISIu9t0aJsuUWL8pc1pHVr8+TMIDY2lt27d/PJJ5/QtWtX6tSpwyOPPMLEiRN57rnn2LJlC76+vvqwJefPn0cIwejRo/V5vPfee3Q38KfM2QW3a9cuAgMDcXFxwc3NjbZt2xIWlh2kPDMzk6lTp+Ll5YW3tzcTJ04k0yD20+bNm7GysqJ9+/b6tLlz5yKEyLV9+GF2UPMnnniCFStWFPqa5Ic55eo4ePAg3bt3p1q1arnk//333zJVL4WiOLibfJeh64aSITNIc7yB48A32LwZqlQxIfzNNzBpkvbc+uEH6N8/74z/+ENzx9bRtGmJGB+oaAaojOPi4oKLiwsbNmwgOTk51/HHHnuM5ORkDh8+DEBISAheXl7s3LlTLxMSEkJQUJDJ/NPT0+nfvz8dOnTg2LFjHDhwgPHjx2Nt0BZfvnw5NjY27Nu3j+DgYL744gtWrVqlP757925at25ttIT36NGjiYiI0G8TJkygevXqDB06VC8TEBDAwYMHSTLRTJ89e7a+7qa2GjVqsHv37lznmVMuaEtnBwUF0ahRI0JCQvjzzz+pXr06AQEBLFu2jPr165dIvRQKS+Ji60bTk79AShWs79YndNz/8PY2Ibh8uea1BvD111qA0bw4fhz69NEcE86cKRG9jTAnXo+ltooYC27t2rXSw8ND2tvby8DAQDlhwgT5119/6Y+3bt1azp49W0op5eDBg+X06dOlg4ODvHHjhkxISJC2trZyz549evmXXnpJ9u3bV0opZXR0tARkSEiIybI7deokAwMDjdK6desmRxjEcerfv78cOnRonvp/8skn0tfXV545c8Yo/dixYxKQFy5cyHVOdHS0PH/+fJ7bkSNHZGJiYp5l5leulFJ26dJFDhgwwCht8uTJ0s/Pr0TrVRRULDhjyuv/2RxKIhZcZqaUr76qhWFz8T8sN+3L5/5csUJKa2sp58wxL+OXX5Zy7FgtiNx9gpmx4NQYUCkzcOBA+vbty+7du9m/fz9bt27ls88+Y9asWUydOpXHHnuMkJAQpkyZQmhoKOPHj+fPP//Ut4ZsbW0JCAgwmbenpyfDhg2jZ8+edO3ala5du/L0009Tu3ZtvUzz5sZxoHx9fYmMjNTvJyUl4ZPHZLSPP/6Y4OBgdu7cib+/v9ExR0dH/fmm9PL09Mzzmty7d09/fmHLjYqKIjQ0lG2GXQaAs7OzUWunJOqlUFiKDz6Ab78FBwfY+l1r2j+aj/Bzz2ljOY0b5y2Tnq5FNBBCy9jK6r6GGQqL6oKzAA4ODnTv3p0PP/yQffv2MWLECKZPn05qaiodOnRg7969nDp1inv37tG6dWuCgoLYuXMnISEhtGvXDtt8wp5///33HDhwgI4dO7Jhwwb8/f2NHs45zxVCGI0BeXl5ERMTkyvfWbNmsWDBAkJDQ3M9pAHu3NGWg6pWrVquY/fbBWdOuX///TcZGRm0aNHCKP3w4cM8YhD4qiTqpVCUJneS7jBuyzjmfpHArFmaL8GaNWAwrJnN9u1w8GD2fl7GR0qYPh0efxxSU7U0a+tSMT6gvODKBI0bNyY9PZ3k5GTatWtHSkoKc+fOpUOHDlhbWxMUFMTIkSPx9vamT58+BebXokULWrRowaRJk+jduzc//PADPXv2NEuXli1bsnTpUqO0GTNm8O233xISEpKnC3NYWBi+vr4mWxmjRo3imWeeybPM+Ph4GppYe8SccjMyMgDjFsqFCxfYtm0b69atK9F6KRSlRUJqAv3+14/94fvh2mFw3MiSrz0xOQUwNFRzMrCxgb//1qJV50VkJCxYoMV427NH8+EuRVQLqBSJjo6mS5cuLFu2jOPHj3Pp0iXWrFnD3Llz6dq1K66urri4uNCqVSuWLVtG586dAXj00Ue5du0aBw4cyNMBAeDSpUtMnjyZffv2ceXKFXbu3Mnx48dpnF/TOwc9e/bk9OnTerfwWbNm8eWXX7Jy5UqcnZ25efMmN2/ezOVEsXv3bnr16mUyT09PT/z8/PLcHnzwwVxdcOaW27ZtW5ycnHj33Xc5ffo027Zto2/fvjz33HNG+pREvRSK0mL9mfWa8QGovZ9h03bz0ksmBPft04KLJiVpk0cLmvPm4wNbt2pNqVI2PoByQihNkpOT5ZQpU2SbNm2ku7u7dHR0lH5+fvKtt96S0dHRUkptcHrSpEkSkIcOHdKf26lTJ+nk5CRTU1ON8jR0Qrh586Z86qmnpK+vr7Szs5O1a9eW77zzjv6cTp06yTFjxuR5vo7AwEAZHBwsMzMzpaurqwRybTt27NDLJyUlSVdXV7l///77ui45B+TNLVfHpk2bZMOGDaWtra2sW7eunDFjhkxLS8slV9r1KkydKzrKCaFo7NsnpW2HLyXTkV2nfmVa6NAhKV1dNc+EF1+UMj3dtFxSkpZhCYKZTggWNzL5bRXNAJlDWXgwbdmyRfr7+8v0vG7gHAQHB8vu3bvfd3mlVefSrld+lIXfuTRRBuj+CQuT0sNDe1r3H3lcZmaaEDp6NFvo6aelNPECJqWUMjlZyq5dpbS3l/LPP4ukV36Ya4BUF5wiF7169WLMmDGEh4ebJW9ra8u8efNKWKuiU1Hrpai4XL2qxXeLiYEnnoC185vl9g9ISNCiWsfEaGM/y5fnvUaPnR089JC25o+X5RelFpqxKps0bNhQnj171uSx06dP06hRo1LWqOS5d+8eVUxOZa64qDpXfM6ePUt6ejpNmjQxebyi/p8h/8njefHVga+4EHmDbe98zLmzgsce04IT5Dlb4eeftWW0V6/OIw6PARkZcPMm1KxZKJ0KgxDibyllm4LklBecQqFQlCGWHV/G+K3jtZ2GUTRz+IYNG2xyG5/MzOw1eQYOhAEDTLtP//MPfPKJZqAcHDQ36xI0PoVBdcEpFApFGUFKyS+n1uv37WudZt1vqbi75xC8dk2LU7l3b3aaKeOTkQGDB2tebp99VhIqFwllgBQKhaKMIKXAZv1KOPoS1tFN2fv6Rh6s7WQsdOOGFsn66FEtJEJ+wyi62aqvvQbvvFOiut8PqgtOoVAoygBSwvjxsGaVDS5VvmPj9ru0buJhLHTjBgQFwfnz8PDD2nLaplo+Z8+CbnJ3s2ZaNOwyiGoBKRQKhQVJSU8BtJWug4M1R7XfNljRqa0J49O5s2Z8WrSAHTsgZ4xFKbVlF5o2hd9/L6Ua3D/FYoCEEN8JISKFEGF5HBdCiK+EEBeEEMeFEK2Ko1yFQqEoz1y9e5UmC5ow4ouVfPih5lOwYoXWyDEiIkIzPufOacbnjz+gatXcGQqhjfuAtoppGae4WkBLgfzilfQGGmRtI4Gvi6lchUKhKJfcir9F95+682/Mv3wXOxhaLuGbbzRntlwcOQIXL+ZvfHTMnasFIs1v3Z8yQrEYICnlLuBOPiL9gR+zJsn+BbgLIWoUR9kKhUJRHsmQGaQkZQ3DZ9jy8oAHePXVPIT79IGNG00bn/BwGDZMm5AKWjOqZcuSUrtYKS0nhJrANYP98Ky0iJyCQoiRaK0kqlWrRkhIiMkM3dzcuHfvXrEramkyMjIKVa9Ro0YRHR3NmjVrCpTt06cPjRo14rMy5o6ZV51jYmJo06YN27dvp379+gXmM2TIENq2bcvYsWNLQs1io7D1Kg1K+trplpnP695OTk7O879e3omPjzdZt7Nnq3Drkz9gwJMEZoxgyBBbIzm76Gjsb93ini6YsL09nDiRK5+Hx4/H/fhxwuPiuDBuXAnVooQwJ16PORtQFwjL49gmoIPB/h9A64LyrIix4CIjI+Xo0aNlnTp1pJ2dnfT29pZdunSRv//+u5Sy8DHCYmNjZUxMjFmypoKRlgXyqvPEiRPlsGHDzM7n+PHj0sPDQ8bGxhaXaiXCxIkT5QsvvJArff78+bJu3brS3t5etmrVSu7atavAvG7cuCGHDh0qvby8pL29vWzUqFGeK+LmR0lfOxULzpizZ6X08tJCtz0/OCP34qMREVI+9JCULi5SGqyYbJIzZ6Ts31/KrIDGZQHKWCy4cKC2wX4t4EYplV2mGDhwIAcPHmTJkiWcO3eOjRs30rt3b/0yAYXFzc0N91yz1Mo/iYmJLF68mBEjRph9TrNmzahfvz7Lli0rQc2Khq5eQ4cONUpftWoV48ePZ+rUqRw5coR27drRu3dvrl69mmdesbGxtG/fHiklmzZt4vTp08ybNw9vb+9C61Uerl15Jz0znSuxV7h+HXr0gKgo6NULln5vpQ9oAGhhcjp3hjNnoF4900sqnDmT/b1hQ1i/PrdHXHnAHCtlzkb+LaC+wBZAAIHAQXPyrGgtoJiYGAnI7du3mzy+efNm6eLiol9K4Ny5cxKQo0aN0stMnTpVduvWTb+fczmF0NBQ2bZtW+ns7CxdXV1lQECAPHHihJRSawGNHj1aTpkyRVatWlVWq1ZNTpgwQWYUYe334sBUC2jNmjXS09NTZuYI/TtnzhyTyyh88MEHUkopP/roI9m+ffti17GgcnUcOHBAduvWTXp5eeWSvXDhgr5ed+/eNTovICBAvvLKK0Zpfn5+cvLkyXnqNGXKFNmuXbti07+krp2UqgWUkZkhh/wyRHrNqSbrtzsiQcrAQCnj43MI37ihtXxAymbNpIyMzJ3hBx9IaWUl5YYNpaL//UBptoCEECuA/UBDIUS4EGKEEGKUEGJUlshm4CJwAfgWeL04yi1v6Jag3rBhQ66FzwAee+wxkpOTOXz4MKAFMfTy8mLnzp16mfwCG6anp9O/f386dOjAsWPHOHDgAOPHj8fa2lovs3z5cmxsbNi3bx/BwcF88cUXrFq1qngrWgzs3r2b1q1bI3JMshs9ejQRERH6bcKECVSvXl3foggICODgwYNGK6TqKGhpcBcXlzyXBi+oXNBWTw0KCqJRo0aEhITw559/Ur16dQICAli2bBn169c3Wa/U1FT+/vtvevToYVRmjx492LdvX57XaP369bRt25Znn30Wb29vHn74YYKDg3UvfYXWP79rpygak7ZP4qfjPxGVdJuLHYNoEHCRjRvB2dlA6No16NhRa900bao5HJhaCl7XXIqMLBXdSxRzrJSltkK3gLRpWMZp/fppaYZvCwsXammvvpqddv26llajhvH5rVpp6YcPZ6dNm5a7HDNZu3at9PDwkPb29jIwMFBOmDBB/mXQx9u6dWs5e/ZsKaWUgwcPltOnT5cODg7yxo0bMiEhQdra2so9e/bo5Q1bQNHR0RLIcwygU6dOMjAw0CitW7ducsSIEUZpb7/9tmzWrJn09/eX69atu696btq0Sfr7+0s/Pz85f/78POUuXbokO3bsKBs1aiQbN24sb9++LaWUsn///nLo0KH5lvHJJ59IX19feebMGX3asWPH9K2NnERHR8vz58/nuyUmJhZYN1PlSillly5d5IABA4zSJk+eLP38/PT7unoZtvquX78uARkaGmp07kcffSTz+w/Y29tLe3t7OXnyZPnPP//I7777Tjo7O8t58+bdl/75XbuiUtlbQDvP75M277lLpiOrDB4pw8NzLOqTmiplw4bac+Xhh6XM+h+YJDNTyiNHSlTnokIZGwNSZDFw4EBu3LjBb7/9Ru/evdm3bx+BgYHMnj0b0FpBOk+Y0NBQevfuTUBAACEhIezduxdbW1sCAgJM5u3p6cmwYcPo2bMnffv25fPPP+fatWtGMs2bNzfa9/X1JdLgTWrdunVcv36dY8eO8dVXX7Fly5ZC1zE9PZ3x48ezY8cOjh07RnBwMBERuRweAXjppZeYMmUKp06dYt++ffolCpKSknBwcMizjI8//pivvvqKnTt30lAXcgT0S3ubeosvaGlwPz+/XEuDm1tuVFQUoaGhvP66cePe2dnZqLWTX71ytvaklLnSDMnMzKRVq1Z8/PHHtGzZkpdffplx48Yxf/78QusP+V87xf2Tni747K1HSV8cglPYGP75vwXUrJnjd7W11SJWt28Pf/5pvFbPhQtatOu4OG1fCC0MTwWgYhkgXRvIkN9+09Iefzw7beRILW3Rouw0X18t7UYO34i//9bSW7fOTps+Pf8AgAXg4OBA9+7d+fDDD9m3bx8jRoxg+vTppKam0qFDB/bu3cupU6e4d+8erVu3JigoiJ07dxISEkK7du2wtbXNM+/vv/+eAwcO0LFjRzZs2IC/vz/btm3TH895rhCCzMxM/f7GjRt5+eWXSUhIYNGiRTz55JOFrt/Bgwdp3LgxtWvXxsnJiaeeeoqNGzfmkjt58iS2tra0b98e0Bwq7LPWMvHy8iImJsZk/rNmzWLBggWEhobi7+9vdOzOHW06WjUTXRdF6YIrqNy///6bjIwMWrRoYZR++PBhHnnkEf2+qXp5eXlhbW3NzZs3jdIjIyPx8fHJU58aNWrQWOeim0WjRo3ydFzIT3/I/9op7o/MTPjkk4fYuBE801pwcHowfg9md4mTmpr9/cknYdcubbE4HVLC8OHwyy9a4NEKRsUyQOWUxo0bk56eTnJyMu3atSMlJYW5c+fSoUMHrK2tjQyQOQtbtWjRgkmTJunlf/jhB7N1OXLkCNeuXcPLy4ubN2/SvXt3o+Nt27bl4YcfzrX9+++/epkbN25Qu3a202OtWrW4fv16rrLOnz9PlSpVeO6552jZsiUffvih/ljLli05depUrnNmzJjBwoULCQkJwc/PL9fxsLAwfH19TT64R40axdGjR/Pd2rQxvYZWQeVmZIU/MWw9XLhwgW3btjFkyJB862VnZ0fr1q3Zvn27Ufr27dtp166dSX0A2rdvT84FG8+dO0edOnUKrT/kf+0UhWPD2Q3EJMUydiz88YcPLi6wdSsYrcd39Cj4+4PhS49VjkeyELB0KQwZogWLq2iY009nqa2iecFFRUXJzp07y59++kkeO3ZMXrx4Ua5evVr6+PjoPdvi4uJkQECAtLa2lp9++qmUUsqkpCRpb28vra2tjcZ/pDQeA7p48aKcNGmS3Lt3r7x8+bL8888/pa+vr5wxY4aU0vQ8IMPz09LSZL169fR69OnTRy5ZsqTQ9Vy9erVROQsWLJDTp0/PJbdmzRrp7u4uw8LCZGpqquzTp4/85ZdfpJTavBQrKysZFRWll585c6asWrWq3Lt3r4yIiNBvSUlJRvUZPnx4oXXOD3PKjYqKkk5OTvK5556Tp06dklu3bpX+/v655jHp6nXp0iWj9JUrV0pbW1v57bffylOnTslx48ZJZ2dnefnyZb3MvHnzZMOGDfX7Bw8elDY2NnLmzJny/PnzcvXq1dLV1VUGBwcXWn8pS+ba6ahMY0D/O/4/KaYLWf3DVhLHKGlrmyFzTQU6eFBKDw+tz+aZZ4yPpadLmYenbHkBM8eALG5k8tsqmgFKTk6WU6ZMkW3atJHu7u7S0dFR+vn5ybfeektGZ00ii4uLk5MmTZKAPHTokP7cTp06SScnJ5mammqUp6EBuXnzpnzqqaekr6+vtLOzk7Vr15bvvPOO/pyCDNCxY8dk79699cemTZuWywAFBATIFi1a5NoMB6737t0rn3jiCf3+1KlT5aJFi3Jdj/3798uuXbvqB+QXLFggP/zwQ/3xwMBA/cM0MzNTurq6mnQl3rFjh5RSM9Surq5y//79ef8IhcSccnVs2rRJNmzYUNra2sq6devKGTNm6F3qDQkMDJT/+c9/cqXPnz9fP0G5VatWuZwSpk2bJsnh/LJx40bZvHlzaW9vLxs0aCC//PJLI9d1c/UviWtnSGUxQJdiLkmb/7ORTEcyHSkGvChnzTpuLLRnj5Surtrjt39/KZOTs49lZEj5+OPasbVrS1X34kQZoHJKYSMhFCc//vij7Ny5s8zMzJT37t2TLVu2lNevXy90PmlpafLBBx+UV69elYmJibJRo0Ym80lPT5cPP/ywvHTpkszMzJTPPvusXLlypf74li1bpL+/v0xPTzer3ODgYNm9e/dC61vabNmyRfr5+Zldr9KgpK9dZTFAUko55L+LJdOE5PUm8psfbxtHQti5U0pn5+yWT44XSimllJ9+KqWnp5T3EdGirGCuAVJjQAo9R48eJSYmhvr169O5c2c++OADfH19C52PjY0N//3vf+natSvNmjVj9OjRRvk8nOXBY21tzZw5c+jXrx/NmzfHy8uLZ555Ri/Xq1cvxowZQ3h4uFnl2traMm/evELrW9r06tWLV1991ex6lQbl5dqVdVauhGVvj4DVa/m40Q5eG2LgzbZ1K/TurQUNHTIEli/XvN8AsmLlATBhApw6BZ06la7ylsAcK2WpTbWASpcuXbrIc+fOlXq5lqyzpahsda4MLaCNG6W0sdEaN1lT+aSUBrHgNm/WBF59VRoFf/vxRy3qQRmK5VZUUC0gRWG5cOECD5qKO6VQKEyy/9p+3tz6JjtDMhk0SGvIvPsuTJ5sQrh3b22dnoULs73d0tPhiy+0KNcrV5am6mWC0lqOQVEOuHLliqVVUCjKDYdvHKbX8l7EpcQx/2QM6alLGDnShk8+0bynkRLmzsXdxiZ7idOc6/TY2GhzfLZvh1deKe0qWBzVAlIoFIr74MdjPxKXokUnSK+7hf5DrrNggYHxeecdmDyZptOmgWG0+6NH4dNPs/fr1KmUxgdUC0ihUCjui+G+X7AoLImU+r/Q7cYO1i6tg7U1Wrfaa6/Bd9+BjQ1n336bJrpVTO/c0ZwL4uK0Wal9+li0DpZGGSCFQqEoJKdPQ4/uVqRELaTrgPfYvKIuNjZASgoMHqx1qzk6wi+/cNsw9p+npxbR4MgR6NLFYvqXFVQXnEKhUJjB3eS7AJw/D127wu3b0LOHFZuW19W8qe/e1Vo0v/wCbm7auE6vXtjExWkWS8fYsbBkCeQTbLeyoAyQQqFQFMDxW8dpMK8Bn/y+mK5dISJCW7T0l18gK36uZmT27AEfHwgN1SJbX7pE61GjNA+4qChNToisgSKF6oJTKBSKfDgZeZIuP3QhOimaKftfharWtH/gZTZsACcnA8HAQM0iNW6sLaUNUKMGaW5uODo7a91zCiNUC0ihUCjywbeKLzWd62o7ya408mrC5s3g4gLs3QubNmUL9+2rLe2iW2bBwYETs2drEa9r1ixt1cs8ygApFApFPiREe3BvwXa42JUGf21nz6oAXF2BdeugWzd4+mk4eVITvnxZ63p79139+WkeHppDgiIXygApFApFHly7ps0hvXTKg4dPbGf/2gA8PYH587VVSpOTYehQ0K0ue+sWHD8Ov/6qOSUo8kWNASkUCoUB+6/tJ1NmUpv2dO4MFy9Cq1awfbvA0z0TprynLZ8Nmkv11KnZTgVt22rjQO3aaZ5winxRBkihUCiyCLkcQr//9cMKa6qs+4MbF9vQurXmUe3hkATPD4PVq8HaGhYv1tyu+/aFadM04wPQr59F61CeUF1wZZxhw4bRz8wbOigoiLFjxxYoFxMTg4+Pj9Ey2nkxaNAgPv/8c7PKL4+UxDUzh8zMTF577TWqVq2KEIKQkJD7zqsw94i5FLa+pu6T8nbvpGak8vKvL5OQlsC9tDhutB1Cm0cy2LEDPDyACxdg40aoUkX7HDYMgoNhyxZtbo+Ulq5CuaNYDJAQopcQ4qwQ4oIQIlccWCFEkBDirhDiaNb2YXGUWx4ZNmwYQgheMRH76d1338XV1dXoYfLll1+ybNmyYtVh9uzZ9OnTx6zI19OmTWPmzJnctWB/trlGoiQpzDUzh23btvH999/z22+/ERERQbt27cpEPXUUtr6m7pOycO8UBjtrOxY89itWyZ4Q50uzU+vY/rs17u5ZAs2awZo1sG8f9Oqlpb3/PowZA+vXq7k990GRDZAQwhqYD/QGGgPPCyEamxDdLaV8OGv7v6KWW56pXbs2q1atIiEhQZ+Wnp7OTz/9RO3atY1k3dzccNf/A4pOYmIiixcvZsSIEWbJN2vWjPr16xe7ESxPFPaamcPFixepUaMG7dq1o3r16tjZ2RVb3kXlfupr6j4pb/fOxYswekBzMn/4nRZHdrF73UO479tsvExCs2aaA4JuTo+dndYKUi7W90VxtIACgAtSyotSylRgJdC/GPKtsDRv3pwGDRqwevVqfdqmTZtwcHCgQ4cORrKG3Su7du0iMDAQFxcX3NzcaNu2LWFhYUbymZmZTJ06FS8vL7y9vZk4cSKZmZn645s3b8bKyor27dvr0+bOnYsQItf24YdaQ/WJJ55gxYoVxX4ddOW6urqaLFdX/9DQUObPn68/fvnyZbZu3cpjjz2Gh4cHnp6e9OzZk9OG4U7QWk6vv/56vtfjfq9Zfr9FSkoKb775Jj4+Pjg4OBAYGMiePXuM6jRlyhSuXr2KEIK6devmWc+goCBGjRrF+PHj8fDwwMPDg3feeSdXHXLWO2dLKmc3XX76m6qv4e9VmPukpO6d4iIlXTMkp0/DY4/BlSsQWKc1u9Y/iNuP8+Dxx+Gll7QVSqWEAQPgm2805wNFkSkOJ4SawDWD/XCgrQm5R4UQx4AbwEQp5UlTmQkhRgIjAapVq5Zn37ibmxv37t0zSpu9bzaf/KV5p0wOnMzUdlONjk8NnUrw38EAzOo4izfavGF0fNz2cSw9sRSAL7t9ycvNXzY6PnzTcNaeXQvA4t6LeabRMxSWtLQ00tPTeeGFF/j2228ZNGgQAIsWLWLw4MFcunSJ9PR0fd108jExMfTv358hQ4awcOFC0tPTOXr0KMnJyXrZjIwMli9fzqhRo/j99985ceIEI0aMoHHjxjz99NMA/PHHH7Ro0YL4+Hi9Ti+++CIDBgzQ73/11VesXr2aAQMGcO/ePZo2bcrMmTOJjIzE0cR8hv/85z989tln+db7559/pl27dkZpunIzMjKwtrbOVS7AzJkzOX36NP7+/kybNg0Ad3d3oqKieO2112jSpAnJycl8+umn9O3bl0OHDulbE+Zcj/u5Zunp6fn+Fu+++y7r1q0jODiYevXqERwcTK9evThy5AjVq1dn5syZ+Pj4sHz5ckJCQrC2tsbW1tZkPXX6DR48mB07dhAWFsa4cePw9PTUGxndPWJ4H6Smphr9PwxlCtLf1D1yv/eJLi0+Ph4HB4dc/1kdycnJRRoHux/Whq9lY8RGxrosYuaUIO7etaN581g+mHKEey99hev69QBcfu45Lt+8CZGRVBk+nDr29pxt3Zo0M/WNj48v9bqVF4rDAJnq+Mw5GvcPUEdKGS+E6AOsBxqYykxKuQhYBNCwYUMZpFvIKQenT5+mSpUqRmn2+qBM2vecx+1s7fI9bqtbnx1wcHDI97ijo2Ou4+Zga2uLjY0Nw4cP5/333+fmzZtUqVKFHTt28PXXXzNlyhRsbGz0eevkpZTExsYycOBAWrRoAUDr1q2N8ra2tqZx48bMmTMHgFatWrFs2TL27dvH8OHDAYiIiKB27dpGuht+nzNnDj///DOhoaE0zJrb4OfnR1paGvfu3cPb2ztXncaPH8+QIUPyrXfNmjVzGS9duffu3WPBggW5ytXJODo64ubmhp+fnz79xRdfNMrrp59+wtXVldOnT+tbkeZcj/u5Znfu3Mnzt0hISGDJkiUsXrxYb8CWLFnC7t27+eGHH5g5cyZVqlTB1dUVGxsbozqZqqe1tTU1atTgm2++QQhB69atuXbtGvPnz2fKlClA9j2i08/a2ho7Ozuj39VQJj/9TdU35+8F5t8nurQ7d+7g6+ub53/GwcGBljkXaytBpu2cxvx/5wMw6fK7ZKb+Se/enqxdmIHT8NmwY4fWvTZjBnWrV6euLnJ1UBCMHo1XIcoKCQkhr+dYZac4DFA4YDhwUQutlaNHShln8H2zEGKBEMJLShlVDOWXSzw8PHjqqaf47rvvcHd3JygoiAceeCBPeU9PT4YNG0bPnj3p2rUrXbt25emnn841ZtS8eXOjfV9fXyIjI/X7SUlJ+Pj4mCzj448/Jjg4mJ07d+Lv769P1xmOpKSkPHXz9PTMv8L58Nlnn/Htt9/mKjc//v33Xz744AMOHDjA7du3yczMJDMzk6tXrxrJFXQ9zJHJec3y+y3+/fdf0tLSjLqvrK2tefTRRzl16pRZdctJYGAgwmCA+9FHH+WDDz4gLi4OV1fXQudX0L2U3z0ChbtPdGnJycmF1rMkaVmjJVZYkUkmmckuPPG4FWsnncC24xNaNANvb1i0CJ5/HtLSoGlTbTKQolgpjjGgQ0ADIUQ9IYQd8BywwVBACFFdZP2DhBABWeVG58qpiEwPmo6cJpHTJNODpuc6/lnPz/THJ7SbkOv4oscX6Y+PbD0y1/H/Dfyf/vjgZoOLrO/w4cP58ccf+e677/Rv2/nx/fffc+DAATp27MiGDRvw9/dn27ZtRjKGrTQAIYTReIGXlxcxMTG58p41axYLFiwgNDQ0lxG4c+cOoHWJmmL27Nm4uLjku+3evdvkubNmzWLx4sUmy82Pxx9/nNu3b7Nw4UIOHDjAkSNHsLGxIVUXgyuLgq6HOTKmrllev4XMcsU1NBiG+ZYGVlZWej10pKWlGe3ndy/ldY9A4e8TXVpRXlBKAnH2ScSmhXD2cV6Uv/PzcndsHW20lUsfeQQOH4b+/eHVV+GFF8CgVaooPopsgKSU6cBYYBtwGlgtpTwphBglhBiVJTYICMsaA/oKeE7m/IdUQrp27YqdnR1RUVE8+eSTZp3TokULJk2apG/W//DDD4Uqs2XLlrnexGfMmMHChQsJCQkx6v7RERYWhq+vb55vxaNGjeLo0aP5bm3atMl1nq7cTZs2mSxXh52dHRkZGfr96OhoTp8+zdSpU+nWrRuNGjXSj22UBKauGZj+Lfz8/LCzszNyOsjIyGD//v00bmzKOTSbnPXUceDAASOD8tdff+Hr65tn66datWpEREQYpR07dsws/fOr7/3cJ7o0L6/CdFqVLEuWaFF0Mg69wljPdfyw2ElbTK5RI21C6ZIloOtZ+PxzWLoU7qOlqSiYYpkHJKXcLKX0l1I+KKWclZX2jZTym6zvwVLKJlLKFlLKQCnlvuIot7wjhOD48eNcunTJaPzKFJcuXWLy5Mns27ePK1eusHPnTo4fP17gQy0nOm+x6Kw16mfNmsWXX37JypUrcXZ25ubNm9y8edOoy2T37t300s17MIGnpyd+fn75bjnHf8wpV0fdunU5ePAgly9fJioqCg8PD7y8vPj222+5cOECoaGhjBo1ChubkgnskfOa5fdbODs7M3r0aCZPnszmzZs5ffo0o0eP5tatW7z++uv5lpOznrpW2I0bN3jzzTc5e/Ysa9eu5dNPP+Wtt97KM58uXbqwZcsWNmzYwNmzZ3n77be5di3bT6igeylnfeH+75OC7p3S4Hrcdd7Y/AYp6anMmgWvvAIZGTDr7Wi+OtsLq28XaoJLl8LEiTBihNbtBlrEA0WJoULxWBhzHRmcnJw4d+4cTz/9NFFRUfj4+PDCCy8wadKkQpXXrFkzAgICWLlyJa+//jpz584lLi4ul8vtjh076Nq1K8nJyaxbty5XV19RkFIWWK4hEydO5KWXXqJx48YkJSVx6dIlVq1axbhx42jatCl+fn589tlnDBw4sNh0NMTwmo0ZM6bA30Ln0PDyyy8TGxtLy5Yt2bp1KzVq1Mi3HFP1BHjhhRfIyMigbdu2CCEYMWJEvgZo+PDhHD9+XN+t+/rrr/PUU08RlbUgWkH656yvOb+XqfukJO6dwnIu+hw9furBlbtX2LY7mvNzliGEFSsmH+PZlU/BpUtw5owWUPTJJ+Hjj7VxH2V4SgcpZZnd/P39ZV6cOnUqz2Plmbi4uBIvY8uWLdLf31+mp6cXKBscHCy7d+9eovqURp2LSmGumTmYW+dOnTrJMWPGFEuZhaGw9TV1nximnTlzRoaFheV5fkn9n7/Y/4VkOtr2gY20qXNQHnr9OykdHaUEKR94QMrLl7NPSEkpdh127txZ7HmWdYDD0oxnvIoFVwnp1asXY8aMITw8vEBZW1tb5s2bVwpalW0Kc80qAoWtr6n7pCzcO0MbjqPG9ZGQ6oTH+jVce2g+bRYMh6QkqF4drl6FDQY+U2UoIkVlQHXBVVLGjRtnltzIkbm9ASsr5l6zikJh6mvqPrH0vXPxIvTrJ4g4O59qjd7g7AOz8diyQlsc7uuvNceCqVMhIMCielZmlAFSKMowaga9+aRmpLIybCVDmg9h3z7Bk09CVBQ0aWLDpk1N8fjnaTh/SFvJtGlT7aR+/SCHG76i9FBdcAqFotxzJ+kOvZb14qX1LzE4+D906QJJUfEEN5rPvr2SOgmntKWzb90ydqlWxseiqBaQQqEo9/x3/3/ZeXknACvvvMtDXt6EJszC+/R5WGkDI0dqgUXr1oUyNCepsqMMkEKhKPe83eYDvtkWQpTTHjr+0ZM/IoZjIzPhoYegQwdtrZ61a5V7dRmjXBsgKWWphTdRKBQlgyxiUJTz52HAADtsLs1j1gNDmXraYN5Rs2bQpIn2XRmfMke5HQOytrbOFd9KoVCUP5KSknLF48uP+NR4fj3zKwC//QZt2oBN2BFOJ3Vh6ukTWiDRpUuhd2/4v0q99mWZp9y2gNzd3bl16xY1a9bEyqrc2lGFotIipSQpKYnr16/nG33bkHPR5xi0ehBhkWE8m7GRlTP6YEsqExtvwfW6hPZ94LvvwMdHW0hOUaYptwbIy8uL8PBwzp49a2lVipXk5GQcHBwsrUapoupc8bl586Y28z3Hy6KtrS0+Pj5mLysx5Y8pnIg8AcDatKfxdP6b/xtXhcFfzUYkJGgtHjONmcLylFsDZGVlle/6OeWVkJCQUl2Yqyyg6lzxGT16NLGxsRw9erRI+bzo9g0b4vcjHG7x9bZE+gZ8RPXZK8A/GGrUgByLNCrKNuXWACkUispDWhpM+1By/ZMt/F4zBveMTFrE2GP1344gJQwbZmkVFfeBMkAKhaJM8uuZX8mUmTSzfYp3B1zg9ROj6MYfcD1L4KXnYPRoi+qoKBrKACkUijJFYloib297m4V/L8RBuNBy4Tf8cWMEjqSQ5lYV2/9+qi0Y162bpVVVFBFlgBQKRZljx4UQAJJlPK6dh8AqZ1KfHIjd11+qSAYVCOW/rFAoygxSwsavohn1+QOIDBtszw0k+GBTHIY9j92385XxqWCoFpBCobAoMUkxeDh6cP54Etd6jmDgzVVYk0mTZQ/RLHQ1tbzToIAl6xXlE9UCUigUFiEjM4PP9n1G7f/W5v+GzsKmRWO63FyBNZnceaAFvUJ+p1ZtK2V8KjDKACkUCovwzvZ3mbh9IglpCaxwfZ8aNpe5VaU+ia+Ow/PCIcQDtS2toqKEUQZIoVCUOocPg+dEe1xStH3rDBsOvjUTnztncVr0pVqnp5JQLAZICNFLCHFWCHFBCDHZxHEhhPgq6/hxIUSr4ihXoVCUL678+S+/P/QGQY/EsybsWT7eZscb51pxcEYkHee+BzZqWLoyUeRfWwhhDcwHugPhwCEhxAYp5SkDsd5Ag6ytLfB11me+pKVZce0aWFlpkdRNfeZME0LbFApF2eDa3WucCD+Ke2QCtbs2oA6Sd629iH97GkPG3MKtjrulVVRYiOJ43QgALkgpLwIIIVYC/QFDA9Qf+FFqC3/8JYRwF0LUkFJG5JfxpUvhPPBA0H0rZmiMDD8LSsv5PedmZVW4dHPO0X0mJsbj6uqiN666TSej2yoSsbGxuLu7W1qNUqWy1Dny9i3OxJ9B3oIYoKOARLtq2D+0GfuDO+l/0NIaljyV5be+H4rDANUErhnsh5O7dWNKpiaQywAJIUYCI7U9Z2xtM9HWqxLo1q3K+Wl4zBApMZletnExS0ozYDLLMEn9vmasZJahkllbQd/zOi5LpTWZkZFBbGxsyRdUhqjodU6JScfh+nU8M2OxqwkpgASu1vbG06MGSaSQFJtiaTVLhYr+WxeF4jBAph5ROR/75shoiVIuAhYBNGzYUJ49u6tQykgJmZmQkaF9Gn7P6zOvY7otLQ3S07XPonzP63haGqSmQnIy3LwZg6OjBykpkJKipRl+6jZD45qRUahLVCgcHcHVtXCbmxt4emZvjo75lxESEkJQUFDJVaIMUhHrHH9PsvK7WBZ/70jI1eq4c5d0rPikYSdm2/1FTceanA87b2k1S52K+FsXhLkrVReHAQoHDP0lawE37kMmF7Fpsaw4sYKGXg1pVcM8vwUhtLGg8rr6bkjIsQJvVimzDVZehiopCRIStC0xsfDf4+Ph3j0tn6QkuHXr/utkb29skDw8jPejony5eVP7XrWqtqBltWpQiZbLKbdICQd3JvD3R6NZXucX3G7V5sCx0yx1GE2Hh6KoN3ME7/Vpy/bOQdyNvWtpdRVljOIwQIeABkKIemhxap8DBueQ2QCMzRofagvcLWj8ByAyOZLBvwxm+EV3lpxvpL1au7mBuzvzhzflg50f4ubgxmsuQUz2ejL7uJsbuxNOcfDuadycPAioGUBzn+bFUNWygRDaQ72k5+dJqRmjuLjCbbGxEBMDd+5AdLRmECMitM00/iZTq1TRDJG3d7ZRyuuzWjWwsyupK6EwREo4fRp2BJ/BZflCHnaYx5jXtWa4XZ0zfDnoLCOHfGz0AiFMdoIoKjtFNkBSynQhxFhgG2ANfCelPCmEGJV1/BtgM9AHuAAkAi8Xpgy3W7Gwf392gpMT0c9PIiY5hpjkGOL3/gZrlhqds7krfPKY9n1m55maATp1Ct59Fzw8mFTzNN85nqaqlQvvew3gxTqPQ8eO4KKNwey+vIsb8RFUdapKM+9m+LhUvlUWhQBnZ22rUeP+8pBSa0HpDJJuM9wPC7uBo6Mv0dGawYqMhNu3tRbYvXtw8aJ5ZXl4aHrWqAHVq2d/N9yqV9e6CZWnZOFIT9f+gr/+ChErQnj1xjTGoXWPyzgIvCb4q7Yk3daKKm324eDQ0MIaK8oDxeJ0L6XcjGZkDNO+MfgugTGFzdfNxpWedTvSuk17eL8D3L2rbamp3E0+oZdzr/sQ9PPIPn73LrFuEUCqlo+DmyZ49Sps2gRAZH+IaglRmYmkfPsNHPkGLlzQG6CvvxrCCrerAPz0Tx1eTGyQ3X/UsiUT653nyM0jeDtW452qj9OqQUejfqOIexE42jriZu9mdn9oRUQIcHLStpo1TcuEhJwjKMjXKE1K7ae8fTvbIEVGGn83/Lx9WzNqMTHae0Z+ODqaNkw507y8Kp7HoblkZMDJk/Dnn/DHDsnpkFv8m+jNM37DeCdjBa1IJ8XGiageL1B90ku873WbJSd+ZFaXWTSq1sjS6ivKCWV61pePY3VWvfSbyWOfZKQx5bEpxCbH4u7gDk7GUXK7n/4Fxyu7iU2KoZl3My2xVSvYsAHu3OFO+KeQfhIAzzYdwNvZKNJudGaC/nvVU1fgwpXszB9/nEMD7rLrivYG+MoPq+BS1jFnZ6hWjYH9b7PfIwFbYcOu4bsJrBUIJ07ApUtQrRpL43aR4exE9Wr16Fi3E1XsqxTtYlUwhAB3d21r0KBg+cxMrfWk6+rTbTdv5k5LTNRaVQW1rKytwcdHM055bTrj5WKe82KZJD0d/v1XMzh//w1//QWHDkHVe5d4iaV8znIO1Elh+EAHVrteIOksrE94Evsfl1LTTXu56wv0bfykReuhKH+UaQOUH7bWtng5eeHlZDo8+4BGAxjQaIBxorc3PP44AGsynudO0h2iE6Op5VoLdK2kLIIGvE2Vq39xJ/42tb4ZB+me2f1GtWtz++qk7Gxr+UNKvPYanjWSH5nloZYm0zUDCfDjj/Cf/wAw7U24mpV8bkU1qtRvA5uzG5FD5gRia+dADbdaTG7zJlVqP6g9jStxayo/rKyyx4Ka5zPcJ6XWrZeXcTJMv3MHbtzQtoJwdjZtoLy8NMcKnYOF7ntpTw9ISoLr1+HaNQgPh8uXtZbiqVNw5ozm1ALwEKcZwC/M5Rda84/+fGnjRbqrthTpbw3hxGvTaeHmZqIkhcJ8yq0BKip21nZUd6lOdZfqJo9P6Tg13/NXR9bjxr0bRCZEUndyf7Cvoj1V4uIgKgrn9b1wTgwnITMZb2dv7aSGDaFPH+TtSCJd/kbniV7j0m1widTnnZmRzsqEA6QnA3HwwWvLIR1tlN3Hh4zZs/C+8SbVnKrhK1zZkfkiVjV89a/rmT7eXMuIoYarL3bWamTeECGy3cUbFjBMkZKief/dvJn/FhGhvXf8+6+2mYOtbUe9cXJzyx5rc3Ex/u7oqEWnsbHRWmS6z8xMzWjoXPjT0jQPyLt3NScQ3WdsrKZjVFT++jzwALzg/TsTTvZkVVPwPwYIe+0itG2L/5499Fn9JHuv7WX4w8Op5lzNvIoqFPlQaQ1QUWnq3ZSm3k2NE4XQe+Edm6DNd0hMS8TRJmsizCuvwCuvkJGZztgdU4hMjCQmPgqX819rf/Qsou9GkJ7lRu6eZoND/fraUyQuDq5d43bmPe4k3eFO0h2ihDNW08YbqRHpAnUnat8beTXi1JisQZElS0BK4qu58Zf9bWrVbkLNOs2o4uJZ7NenImBvrz2YH3ggfznde4cpwxQdne0NqPvUPAOtCvAMLF5sbbUxuFq1oJlXBJ1Sf+eRqK24elpjv3oZVbau4Z2fhlCjN6RZg8vo8Qx9ZhYcOwaPPgpC8HXfr/F09MTZzrl0lFZUeJQBKmGcbJ1ypdlY2fBpj0/zPMelihcbnttARHwEqRmpMHOsdiBrQs7NtGv6MafqjtVg1BDtiZf1uh7ODbS551pXpZ7p0yE8nFM1ofurwAFocROO/lwN3nlH24BrJ/awefOX1KxaDz/fpjz0YFvlPpYPBu8dBbaqdGzduoumTTsSHa0Zr4QEbe6Vbi6W7ntiYvaE6PT07E9ra82o2NpqDWNbW81gZs1SMNp8rKOoFrYTq727ITQU9hxHkjU73NERbJOhfn2q3UkhLevF50frMIY6O0O7dnqda7up5REUxYsyQGUQR1tHHm/4uIkDjlC3Li1kHaLeiSIiPoKU9BTwbW0kFn9pJzV+HszNhFva+JaO4cPh6lWupxwDjgBQKw5t7MqAQwfXMyp5LVyHfn/CbysMyq9enUO/LWTtjR3Ucq1FmzNxPJrmYzwi7+2twukXgINDJrVqaS2SYiUtTfMmcHEBPz8tbdEv8NprxNnDmsbw62CBc4pkxaVWsHKlZrlateKFL/9kyqZuPOL7CIMaD0JKWak9OBUljzJA5RAhBFWdqlLVqarJ40H1OnNjYgRpGWkkpGV78/HRRwA4XdjGY7tncf3edeoP7g2fvW80q/W6a/ZDp6ZzdXjQOdt97PJl9t8NY+6+uQCMvurDo98Zh0lY0wTWPmxPzTpN6TtiDl3rd9UGIZYtMzZUulaV4v64d0+bERoWprmvHT6sdZmlpMCbb8J//5sdGqNVK+L6d+YV+RkgcUyDRLv6OPn56Vu1NVt35kqDK8YvLQpFCaIMUAXG1toWd2v3XOk9/XrS069nnuc91KoHw+3uEH4vnFb9BsK3WbFh4+Ph1i3CLy7Uy9Z8sCW8XMPIfexgzVusbpAC/I3X9QOaATp3Dt56C4CZHeHXhlrra/RxOx6LrQo7dkDjxgBkbtmM1fUbxsbKx6dyrhWTlqa5rl26pPmNv/xy9nXo1Qv27TMSX9UEQpu7csh9NTuSp+O2ZgOMGwddulDrw//w0PxNnIk6Q5It7P6/V+iZo4WjjI+iNKmE/2hFQXR/sDvdH+ye+4CLC7i40Ne6L672roTHhRPY+Gmo39VI7Pra5+HkSsDggVa1KowdCzdvEuYVwuHqURyuCQNOp+IYEaHF3cli8O+vscMunFp/w+fboMsltLf0qlWhTx+uz5uNo60jHpn2iPffz/Zt1vk6677XqFF2A8rpZtrevKm1PuvW1dLPnYOZMzWf6UuXtMnThtFmO3YksX5tTt8+jW9rf2rEx0OjRtocN3t7gq9OZ49rLBBH6JVQnujbFwICoF8/kJKxj4wlKT2JJx96Ej9Pv9Kvt0JhgDJAikLTqW4nOtXtlOfxd9q/S1//foTHhdO2ZtbKHA0bwrx5AFz//jG4ugeAWis2ceBIDG19syMhhFezJzoNop1APPggJMZrIQ+ioiAhgWfWPsO+a/uwETbsXZNOwHXj8n9ooQ2we7/7EY8NmqB5bc2dC19/rXX5VamSvTk6gq8vfPxxdgaffaY99B0cNONg2PJ65JHsiUYXL0JIiHGY8pQUzS86JQVmz84eCxs1Cg4e1Efq6Bgbm21YXnkFvv1W+56UBD/9RJw9pFlB1UyhDRTVqwf16/PO0bl8tvJ7JJJ5XaYwdvwv8OCD2rmbNhGwOZY9WX4Df1z8gycaPgEHDujVHxNQ6IAkCkWJoQyQothpWaMlLWu0zPP4yoEruXL3CuFx4TSv25YTkSeMwpdHuEht9TKg1sotULWB5voVFQVSEr4qEIB0mY7n+MkQa5Xt53znDlPb7OKGYxqcn8blxJc0AxQRoc2+BNqNAJtMcE+GlSvAqY6f3gBlZGawetX7OMUl45wG3XJES8iY8wn3/B/ASlhh/dcunEeMMDqeKSDJBlJsgHfG4OldVztw7hwcOcItZ9j7ANytDd7p9vRNqqWFd8rip+QDvP6RPfEyhfH+Q/niqYVGrTifff9BntHmj4Ut+RjqxcNXX2kHu3XjqUvv4FHfirYNu/JIzUfM+bkUCouhDJCi1KnpWpOarnkEhgPOjT1HZEIk4XHh1HGvoyXa2ED16kgpcbFzwdXelbiUOHzGTdUmAWchpSRyph1kavv6ScAffQRvvEHG3Rj2b2ijl7dfuAScs+PoJKQlMLhvMgAumbbcixhm1AV21d+H+nM0g1HXsQaXhg3LDk1ub89h+9u0tVkKwCPrBnHwtcPaiV98ASkp/JNwlIGh2phajwc70ffFbUZ1d6pSlXipudBfJEYzPqtXQ3AwvPEGTVs2xUpY0cCxFtVEtLETh709HcbOpUOeV1ahKFsoA6Qoc1hbWVOjSg1qVMkdglsIwcnXtRh+yenJ2Fsbr0mRlpnGGwFvcCvhFnEpcTjaZk0Czgp/cDfpjrY4COBm74b1y8ONzk9MS9R/d6riAR8tMjqeeedfOKZ9t3JwhO+/Nzpud/MoLFwKQIpMyz6Q1W3ndi0NQrWku8m518fxCdeafnZWdkjdmo1Xr8Lu3VCvHl0GfEvC1AQcrOxgApU3WqqiQlCmDVCV8+dh4kR9/DQSErRxhDp14PnnswWlVBMkKyEONrkdDOys7fi85+d5nuNq78rR144SkxxDcnpyruPWwprnmj5HYloiVexMB4h1s3cjQ2aYnGTsYOOAo40j9jb2Js/3reJL/4b9SYpNotMDAbB1K3Trph9nCliylTsbwP2LrxDPv6adNHCgNgbUtasKraSoUJRpA4SUxgPA167BlCnaBDtDA9SqlTY+sHNn9uS7nTu1fvdOneChh0pXb0WZxcbKhhbVW+R5vJpzNVYMXJHn8Qc9HyR2cmyexx/yeojE9xJzH0hPh6tXqVu/PuufW68t0/zKK/DvTDhyBB5+GAC7gc9g55kjomq9etqmUFQwynT7Pd7PDyZPzk5wcdFaREOHGgtevaqF+DWMzrtiheZ5tHNndtr27dos/ZEjjc+fPx/WrtUiPCoURSUpSbsndSQmavdm06bavB4dnTtroW4SDQzWM89o3nqPPlp6+ioUFqJMt4CklZUWzEpHrVrwqYkYateva/MpqhpEBujYUTMoWW+Wernbt7UHhI5797T5KY6OWleHjoEDtZAmP/wAbbNciY8d09b0ad1am3uhqNykpWnhr+3soH59Le2vv6BDB+0e0bk/Ozlprt7p6dqLkq41o3O9VigqKWW6BWQ2Dg7aRD7DAdkXX4TFi43fJIcM0RZ3mTMnOy09HUaP1lpVhuNI587B2bPaw0XHunVaHv/7X3ZaWJjWqurf31inefNg4UJt3EpHamrpLwSjKB6OHdNeRi5dyk4LDtZeRD77LDvN31/7lNL4tz5+XDtXdaUpFHoqhgEyF2trbXa8waRHPDxgwQL45htj2V27NONi2NJp0gSefRZaGsxxuXVLa1XFxRmf/957WhegYZfLyJGasVxhMMZw5Ai8+y788kt2Wmam1tIyZyU0RdFIS8u9WM7UqdoLhWHX2Ny5MGyYNvFUR5Mm2ouPoSu0p6cWsujgQeMXGkfHElBeoSjflOkuOIvi4WE0QRCAp5/WNkOCgrTuP4P1fMjMhDfe0AyT4bjU3btaK8gg7Ax//611K778MgzIWsH1zh1tENrDQ/uuY/hwbTJlcLA+bhoHD8LRo1o3YYuswfWUFK1sd/fyvVb0/ZKerk1MdXXNfvD/8w9s2aI5rPTuraWdPAnNmmnXMiws+/w1a+DCBa3F0qSJlta5s9aiqW2wJEGPHsYtIh1lNfyPQlHGqFwtoJLA2loLlGm4apmVFcyaBYsWGb8Fr1undcn16JGd9sgj2ix8nfEBbYyqcePc3nsHDmhOFYbOEuvXw2uvwW+/ZacdPao9KDt3Nj6/c2do397YqP30E7z9thZJWcfly1rk6v37s9MyM7XW2smTxnlGR2vGzrCll5mp7RvGMMvMzF7gxpC4OGzv3NGMhmH5e/ZoY3Y6rl7VnEV+/tk4z4EDoWdP4+6uvn21AKaGrZVDh+D9941bmj4+2qehngD/93+aEappMFn2lVe0rtdu3VAoFMVDkQyQEMJTCLFdCHE+69MjD7nLQogTQoijQojDpmQqDU5OxuNKLVponn79+mWn1a6tPehzRDpm1SotarRuwBugTRsYMUIb9NaRnq51M1bPsdz4gQNangZLL7Blixa2/9y57LS//tLGunQhXkAzHq1aZTtk6Bg4UBsD27MnO+2zz7Q6TpqUnXb2rNYaa9XK+PxWrWg/cKCx19jnn8Njj2meiYbnjx2rdZfqsLLS9P/9d2PD5u0NXl7GrdI2bbTr3LdvdlrVqprM6dPGOj3/PAwaZOwAo1Aoip2idsFNBv6QUn4ihJictT8pD9nOUsoCVqZX5EvTptpmyIABxq0n0Fo513NE6ATNsNy9qxlBHUOHag9nQ8NQu7bmxBEYmJ0mpeZRmHMsw9VVe9gbpltba5uhU4i1tSaTs3vKzY1UDw/sDFshfn6a84i3d3ZavXramJquS0yHbkE1Q6P+44+5Jya3bm1spEGTUQvnKRQWQ8gieGUJIc4CQVLKCCFEDSBESplrUWIhxGWgTWENUMOGDeXZs2fvW7/ySEhICEFBQZZWo1RRda74BAUFERsby9GjRy2tSqlT2X5rACHE31LKNgXJFbUF5COljADIMkLeechJ4HchhAQWSikX5SGHEGIkMBKgWrVqhBj241cC4uPjVZ0rAZWtzrGxsWRkZFSqOuuobL91YSjQAAkhdgDVTRx6rxDltJdS3sgyUNuFEGeklLtMCWYZp0WgtYAq25tDZXxbUnWu+Li7uxMbG1up6qyjsv3WhaFAAySlzNPtRwhxSwhRw6ALLjKPPG5kfUYKIdYBAYBJA6RQKBSKykFR3bA3AC9lfX8J+DWngBDCWQhRRfcd6AGE5ZRTKBQKReWiqAboE6C7EOI80D1rHyGErxBic5aMD7BHCHEMOAhsklJuLWK5CoVCoSjnFMkJQUoZDXQ1kX4D6JP1/SKQd/x7hUKhUFRKVCQEhUKhUFgEZYAUCoVCYRGUAVIoFAqFRVAGSKFQKBQWQRkghUKhUFgEZYAUCoVCYRGUAVIoFAqFRVAGSKFQKBQWQRkghUKhUFgEZYAUCoVCYRGUAVIoFAqFRVAGSKFQKBQWQRkghUKhUFgEZYAUCoVCYRGUAVIoFAqFRVAGSKFQKBQWQRkghUKhUFgEZYAUCoVCYRGUAVIoFAqFRVAGSKFQKBQWQRkghUKhUFgEZYAUCoVCYRGKZICEEE8LIU4KITKFEG3ykeslhDgrhLgghJhclDIVCoVCUTEoagsoDBgA7MpLQAhhDcwHegONgeeFEI2LWK5CoVAoyjk2RTlZSnkaQAiRn1gAcEFKeTFLdiXQHzhVlLIVCoVCUb4pkgEyk5rANYP9cKBtXsJCiJHAyKzdFCFEWAnqVhbxAqIsrUQpo+pcOfASQlS2OkPl/K0bmiNUoAESQuwAqps49J6U8lczyjDVPJJ5CUspFwGLsso+LKXMc2ypIqLqXDlQda48VMZ6CyEOmyNXoAGSUnYroi7hQG2D/VrAjSLmqVAoFIpyTmm4YR8CGggh6gkh7IDngA2lUK5CoVAoyjBFdcN+SggRDjwKbBJCbMtK9xVCbAaQUqYDY4FtwGlgtZTypJlFLCqKfuUUVefKgapz5aEy1tusOgsp8xyOUSgUCoWixFCREBQKhUJhEZQBUigUCoVFKBcGSAgxUQghhRBeltalNBBCzBBCHBdCHBVC/C6E8LW0TiWNEOJTIcSZrHqvE0K4W1qnksbcUFYVgcoYjksI8Z0QIrKyzGUUQtQWQuwUQpzOuq/HF3ROmTdAQojaQHfgqqV1KUU+lVI2l1I+DGwEPrSwPqXBdqCplLI5cA6YYmF9SoMCQ1lVBCpxOK6lQC9LK1GKpAMTpJSNgEBgTEG/c5k3QMB/gXfJZ/JqRUNKGWew60wlqLuU8vcsj0mAv9Dmi1VopJSnpZRnLa1HKaAPxyWlTAV04bgqNFLKXcAdS+tRWkgpI6SU/2R9v4fm9Vwzv3NKIxTPfSOEeAK4LqU8VkC8uQqHEGIWMBS4C3S2sDqlzXBglaWVUBQbhQrHpSj/CCHqAi2BA/nJWdwA5RfqB5gK9ChdjUqHgkIcSSnfA94TQkxBm0c1rVQVLAHMCeskhHgPrSm/vDR1KymKIZRVRaBQ4bgU5RshhAvwM/Bmjt6cXFjcAOUV6kcI0QyoB+haP7WAf4QQAVLKm6WoYolQiBBH/wM2UQEMUEF1FkK8BPQDusoKMkGtGEJZVQRUOK5KghDCFs34LJdS/lKQvMUNUF5IKU8A3rp9IcRloI2UssJHlRVCNJBSns/afQI4Y0l9SgMhRC9gEtBJSploaX0UxYo+HBdwHS0c12DLqqQoboTWUlgCnJZSfm7OOeXBCaEy8okQIkwIcRytC7JAd8YKQDBQBdie5X7+jaUVKmnyCmVV0ShiOK5yixBiBbAfaCiECBdCjLC0TiVMe2AI0CXrP3xUCNEnvxNUKB6FQqFQWATVAlIoFAqFRVAGSKFQKBQWQRkghUKhUFgEZYAUCoVCYRGUAVIoFAqFRVAGSKFQKBQWQRkghUKhUFgEZYAUCoVCYRGUAVIoSgkhxLtZCyvm3P7P0ropFJZARUJQKEoJIUQVtPWddEwEXgAek1JesIxWCoXlUAZIobAAQohJwDigSyVZlE6hyEWZjYatUFRUDNZ46iylPGdpfRQKS6EMkEJRimQtuDcKbdkJ1e2mqNQoA6RQlBJCiA+AV4EgKeW/ltZHobA0ygApFKVAVstnPNoCgwlCCN0y3bFSymTLaaZQWA7lhKBQlDBZK0XGAq4mDneTUv5RuhopFGUDZYAUCoVCYRHURFSFQqFQWARlgBQKhUJhEZQBUigUCoVFUAZIoVAoFBZBGSCFQqFQWARlgBQKhUJhEZQBUigUCoVFUAZIoVAoFBbh/wGYvos0Dz25tAAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# extra code – this cell generates and saves Figure 11–4\n",
"\n",
"def swish(z, beta=1):\n",
" return z * sigmoid(beta * z)\n",
"\n",
"def approx_gelu(z):\n",
" return swish(z, beta=1.702)\n",
"\n",
"def softplus(z):\n",
" return np.log(1 + np.exp(z))\n",
"\n",
"def mish(z):\n",
" return z * np.tanh(softplus(z))\n",
"\n",
"z = np.linspace(-4, 2, 200)\n",
"\n",
"beta = 0.6\n",
"plt.plot(z, approx_gelu(z), \"b-\", linewidth=2,\n",
" label=r\"GELU$(z) = z\\,\\Phi(z)$\")\n",
"plt.plot(z, swish(z), \"r--\", linewidth=2,\n",
" label=r\"Swish$(z) = z\\,\\sigma(z)$\")\n",
"plt.plot(z, swish(z, beta), \"r:\", linewidth=2,\n",
" label=fr\"Swish$_{{\\beta={beta}}}(z)=z\\,\\sigma({beta}\\,z)$\")\n",
"plt.plot(z, mish(z), \"g:\", linewidth=3,\n",
" label=fr\"Mish$(z) = z\\,\\tanh($softplus$(z))$\")\n",
"plt.plot([-4, 2], [0, 0], 'k-')\n",
"plt.plot([0, 0], [-2.2, 3.2], 'k-')\n",
"plt.grid(True)\n",
"plt.axis([-4, 2, -1, 2])\n",
"plt.gca().set_aspect(\"equal\")\n",
"plt.xlabel(\"$z$\")\n",
"plt.legend(loc=\"upper left\")\n",
"\n",
"save_fig(\"gelu_swish_mish_plot\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Batch Normalization"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [],
"source": [
"# extra code - clear the name counters and set the random seed\n",
"tf.keras.backend.clear_session()\n",
"tf.random.set_seed(42)"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"model = tf.keras.Sequential([\n",
" tf.keras.layers.Flatten(input_shape=[28, 28]),\n",
" tf.keras.layers.BatchNormalization(),\n",
" tf.keras.layers.Dense(300, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\"),\n",
" tf.keras.layers.BatchNormalization(),\n",
" tf.keras.layers.Dense(100, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\"),\n",
" tf.keras.layers.BatchNormalization(),\n",
" tf.keras.layers.Dense(10, activation=\"softmax\")\n",
"])"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model: \"sequential\"\n",
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"flatten (Flatten) (None, 784) 0 \n",
"_________________________________________________________________\n",
"batch_normalization (BatchNo (None, 784) 3136 \n",
"_________________________________________________________________\n",
"dense (Dense) (None, 300) 235500 \n",
"_________________________________________________________________\n",
"batch_normalization_1 (Batch (None, 300) 1200 \n",
"_________________________________________________________________\n",
"dense_1 (Dense) (None, 100) 30100 \n",
"_________________________________________________________________\n",
"batch_normalization_2 (Batch (None, 100) 400 \n",
"_________________________________________________________________\n",
"dense_2 (Dense) (None, 10) 1010 \n",
"=================================================================\n",
"Total params: 271,346\n",
"Trainable params: 268,978\n",
"Non-trainable params: 2,368\n",
"_________________________________________________________________\n"
]
}
],
"source": [
"model.summary()"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[('batch_normalization/gamma:0', True),\n",
" ('batch_normalization/beta:0', True),\n",
" ('batch_normalization/moving_mean:0', False),\n",
" ('batch_normalization/moving_variance:0', False)]"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"[(var.name, var.trainable) for var in model.layers[1].variables]"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/2\n",
"1719/1719 [==============================] - 3s 2ms/step - loss: 0.5559 - accuracy: 0.8094 - val_loss: 0.4016 - val_accuracy: 0.8558\n",
"Epoch 2/2\n",
"1719/1719 [==============================] - 3s 1ms/step - loss: 0.4083 - accuracy: 0.8561 - val_loss: 0.3676 - val_accuracy: 0.8650\n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# extra code – just show that the model works! 😊\n",
"model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
" metrics=\"accuracy\")\n",
"model.fit(X_train, y_train, epochs=2, validation_data=(X_valid, y_valid))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Sometimes applying BN before the activation function works better (there's a debate on this topic). Moreover, the layer before a `BatchNormalization` layer does not need to have bias terms, since the `BatchNormalization` layer some as well, it would be a waste of parameters, so you can set `use_bias=False` when creating those layers:"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [],
"source": [
"# extra code - clear the name counters and set the random seed\n",
"tf.keras.backend.clear_session()\n",
"tf.random.set_seed(42)"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [],
"source": [
"model = tf.keras.Sequential([\n",
" tf.keras.layers.Flatten(input_shape=[28, 28]),\n",
" tf.keras.layers.Dense(300, kernel_initializer=\"he_normal\", use_bias=False),\n",
" tf.keras.layers.BatchNormalization(),\n",
" tf.keras.layers.Activation(\"relu\"),\n",
" tf.keras.layers.Dense(100, kernel_initializer=\"he_normal\", use_bias=False),\n",
" tf.keras.layers.BatchNormalization(),\n",
" tf.keras.layers.Activation(\"relu\"),\n",
" tf.keras.layers.Dense(10, activation=\"softmax\")\n",
"])"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/2\n",
"1719/1719 [==============================] - 3s 1ms/step - loss: 0.6063 - accuracy: 0.7993 - val_loss: 0.4296 - val_accuracy: 0.8418\n",
"Epoch 2/2\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.4275 - accuracy: 0.8500 - val_loss: 0.3752 - val_accuracy: 0.8646\n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# extra code – just show that the model works! 😊\n",
"model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
" metrics=\"accuracy\")\n",
"model.fit(X_train, y_train, epochs=2, validation_data=(X_valid, y_valid))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Gradient Clipping"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All `tf.keras.optimizers` accept `clipnorm` or `clipvalue` arguments:"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [],
"source": [
"optimizer = tf.keras.optimizers.SGD(clipvalue=1.0)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer)"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [],
"source": [
"optimizer = tf.keras.optimizers.SGD(clipnorm=1.0)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Reusing Pretrained Layers"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Reusing a Keras model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's split the fashion MNIST training set in two:\n",
"* `X_train_A`: all images of all items except for T-shirts/tops and pullovers (classes 0 and 2).\n",
"* `X_train_B`: a much smaller training set of just the first 200 images of T-shirts/tops and pullovers.\n",
"\n",
"The validation set and the test set are also split this way, but without restricting the number of images.\n",
"\n",
"We will train a model on set A (classification task with 8 classes), and try to reuse it to tackle set B (binary classification). We hope to transfer a little bit of knowledge from task A to task B, since classes in set A (trousers, dresses, coats, sandals, shirts, sneakers, bags, and ankle boots) are somewhat similar to classes in set B (T-shirts/tops and pullovers). However, since we are using `Dense` layers, only patterns that occur at the same location can be reused (in contrast, convolutional layers will transfer much better, since learned patterns can be detected anywhere on the image, as we will see in the chapter 14)."
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/20\n",
"1376/1376 [==============================] - 1s 908us/step - loss: 1.1385 - accuracy: 0.6260 - val_loss: 0.7101 - val_accuracy: 0.7603\n",
"Epoch 2/20\n",
"1376/1376 [==============================] - 1s 869us/step - loss: 0.6221 - accuracy: 0.7911 - val_loss: 0.5293 - val_accuracy: 0.8315\n",
"Epoch 3/20\n",
"1376/1376 [==============================] - 1s 852us/step - loss: 0.5016 - accuracy: 0.8394 - val_loss: 0.4515 - val_accuracy: 0.8581\n",
"Epoch 4/20\n",
"1376/1376 [==============================] - 1s 852us/step - loss: 0.4381 - accuracy: 0.8583 - val_loss: 0.4055 - val_accuracy: 0.8669\n",
"Epoch 5/20\n",
"1376/1376 [==============================] - 1s 844us/step - loss: 0.3979 - accuracy: 0.8692 - val_loss: 0.3748 - val_accuracy: 0.8706\n",
"Epoch 6/20\n",
"1376/1376 [==============================] - 1s 882us/step - loss: 0.3693 - accuracy: 0.8782 - val_loss: 0.3538 - val_accuracy: 0.8787\n",
"Epoch 7/20\n",
"1376/1376 [==============================] - 1s 863us/step - loss: 0.3487 - accuracy: 0.8825 - val_loss: 0.3376 - val_accuracy: 0.8834\n",
"Epoch 8/20\n",
"1376/1376 [==============================] - 2s 1ms/step - loss: 0.3324 - accuracy: 0.8879 - val_loss: 0.3315 - val_accuracy: 0.8847\n",
"Epoch 9/20\n",
"1376/1376 [==============================] - 1s 1ms/step - loss: 0.3198 - accuracy: 0.8920 - val_loss: 0.3174 - val_accuracy: 0.8879\n",
"Epoch 10/20\n",
"1376/1376 [==============================] - 2s 1ms/step - loss: 0.3088 - accuracy: 0.8947 - val_loss: 0.3118 - val_accuracy: 0.8904\n",
"Epoch 11/20\n",
"1376/1376 [==============================] - 1s 1ms/step - loss: 0.2994 - accuracy: 0.8979 - val_loss: 0.3039 - val_accuracy: 0.8925\n",
"Epoch 12/20\n",
"1376/1376 [==============================] - 1s 837us/step - loss: 0.2918 - accuracy: 0.8999 - val_loss: 0.2998 - val_accuracy: 0.8952\n",
"Epoch 13/20\n",
"1376/1376 [==============================] - 1s 840us/step - loss: 0.2852 - accuracy: 0.9016 - val_loss: 0.2932 - val_accuracy: 0.8980\n",
"Epoch 14/20\n",
"1376/1376 [==============================] - 1s 799us/step - loss: 0.2788 - accuracy: 0.9034 - val_loss: 0.2865 - val_accuracy: 0.8990\n",
"Epoch 15/20\n",
"1376/1376 [==============================] - 1s 922us/step - loss: 0.2736 - accuracy: 0.9052 - val_loss: 0.2824 - val_accuracy: 0.9015\n",
"Epoch 16/20\n",
"1376/1376 [==============================] - 1s 835us/step - loss: 0.2686 - accuracy: 0.9068 - val_loss: 0.2796 - val_accuracy: 0.9015\n",
"Epoch 17/20\n",
"1376/1376 [==============================] - 1s 863us/step - loss: 0.2641 - accuracy: 0.9085 - val_loss: 0.2748 - val_accuracy: 0.9015\n",
"Epoch 18/20\n",
"1376/1376 [==============================] - 1s 913us/step - loss: 0.2596 - accuracy: 0.9101 - val_loss: 0.2729 - val_accuracy: 0.9037\n",
"Epoch 19/20\n",
"1376/1376 [==============================] - 1s 909us/step - loss: 0.2558 - accuracy: 0.9119 - val_loss: 0.2715 - val_accuracy: 0.9040\n",
"Epoch 20/20\n",
"1376/1376 [==============================] - 1s 859us/step - loss: 0.2520 - accuracy: 0.9125 - val_loss: 0.2728 - val_accuracy: 0.9027\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2021-12-15 16:22:23.274500: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:Assets written to: my_model_A/assets\n"
]
}
],
"source": [
"# extra code – split Fashion MNIST into tasks A and B, then train and save\n",
"# model A to \"my_model_A\".\n",
"\n",
"pos_class_id = class_names.index(\"Pullover\")\n",
"neg_class_id = class_names.index(\"T-shirt/top\")\n",
"\n",
"def split_dataset(X, y):\n",
" y_for_B = (y == pos_class_id) | (y == neg_class_id)\n",
" y_A = y[~y_for_B]\n",
" y_B = (y[y_for_B] == pos_class_id).astype(np.float32)\n",
" old_class_ids = list(set(range(10)) - set([neg_class_id, pos_class_id]))\n",
" for old_class_id, new_class_id in zip(old_class_ids, range(8)):\n",
" y_A[y_A == old_class_id] = new_class_id # reorder class ids for A\n",
" return ((X[~y_for_B], y_A), (X[y_for_B], y_B))\n",
"\n",
"(X_train_A, y_train_A), (X_train_B, y_train_B) = split_dataset(X_train, y_train)\n",
"(X_valid_A, y_valid_A), (X_valid_B, y_valid_B) = split_dataset(X_valid, y_valid)\n",
"(X_test_A, y_test_A), (X_test_B, y_test_B) = split_dataset(X_test, y_test)\n",
"X_train_B = X_train_B[:200]\n",
"y_train_B = y_train_B[:200]\n",
"\n",
"tf.random.set_seed(42)\n",
"\n",
"model_A = tf.keras.Sequential([\n",
" tf.keras.layers.Flatten(input_shape=[28, 28]),\n",
" tf.keras.layers.Dense(100, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\"),\n",
" tf.keras.layers.Dense(100, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\"),\n",
" tf.keras.layers.Dense(100, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\"),\n",
" tf.keras.layers.Dense(8, activation=\"softmax\")\n",
"])\n",
"\n",
"model_A.compile(loss=\"sparse_categorical_crossentropy\",\n",
" optimizer=tf.keras.optimizers.SGD(learning_rate=0.001),\n",
" metrics=[\"accuracy\"])\n",
"history = model_A.fit(X_train_A, y_train_A, epochs=20,\n",
" validation_data=(X_valid_A, y_valid_A))\n",
"model_A.save(\"my_model_A\")"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/20\n",
"7/7 [==============================] - 0s 20ms/step - loss: 0.7167 - accuracy: 0.5450 - val_loss: 0.7052 - val_accuracy: 0.5272\n",
"Epoch 2/20\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.6805 - accuracy: 0.5800 - val_loss: 0.6758 - val_accuracy: 0.6004\n",
"Epoch 3/20\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.6532 - accuracy: 0.6650 - val_loss: 0.6530 - val_accuracy: 0.6746\n",
"Epoch 4/20\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.6289 - accuracy: 0.7150 - val_loss: 0.6317 - val_accuracy: 0.7517\n",
"Epoch 5/20\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.6079 - accuracy: 0.7800 - val_loss: 0.6105 - val_accuracy: 0.8091\n",
"Epoch 6/20\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.5866 - accuracy: 0.8400 - val_loss: 0.5913 - val_accuracy: 0.8447\n",
"Epoch 7/20\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.5670 - accuracy: 0.8850 - val_loss: 0.5728 - val_accuracy: 0.8833\n",
"Epoch 8/20\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.5499 - accuracy: 0.8900 - val_loss: 0.5571 - val_accuracy: 0.8971\n",
"Epoch 9/20\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.5331 - accuracy: 0.9150 - val_loss: 0.5427 - val_accuracy: 0.9050\n",
"Epoch 10/20\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.5180 - accuracy: 0.9250 - val_loss: 0.5290 - val_accuracy: 0.9080\n",
"Epoch 11/20\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.5038 - accuracy: 0.9350 - val_loss: 0.5160 - val_accuracy: 0.9189\n",
"Epoch 12/20\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.4903 - accuracy: 0.9350 - val_loss: 0.5032 - val_accuracy: 0.9228\n",
"Epoch 13/20\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.4770 - accuracy: 0.9400 - val_loss: 0.4925 - val_accuracy: 0.9228\n",
"Epoch 14/20\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.4656 - accuracy: 0.9450 - val_loss: 0.4817 - val_accuracy: 0.9258\n",
"Epoch 15/20\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.4546 - accuracy: 0.9550 - val_loss: 0.4708 - val_accuracy: 0.9298\n",
"Epoch 16/20\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.4435 - accuracy: 0.9550 - val_loss: 0.4608 - val_accuracy: 0.9318\n",
"Epoch 17/20\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.4330 - accuracy: 0.9600 - val_loss: 0.4510 - val_accuracy: 0.9337\n",
"Epoch 18/20\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.4226 - accuracy: 0.9600 - val_loss: 0.4406 - val_accuracy: 0.9367\n",
"Epoch 19/20\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.4119 - accuracy: 0.9600 - val_loss: 0.4311 - val_accuracy: 0.9377\n",
"Epoch 20/20\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.4025 - accuracy: 0.9600 - val_loss: 0.4225 - val_accuracy: 0.9367\n",
"63/63 [==============================] - 0s 728us/step - loss: 0.4317 - accuracy: 0.9185\n"
]
},
{
"data": {
"text/plain": [
"[0.43168652057647705, 0.9185000061988831]"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# extra code – train and evaluate model B, without reusing model A\n",
"\n",
"tf.random.set_seed(42)\n",
"model_B = tf.keras.Sequential([\n",
" tf.keras.layers.Flatten(input_shape=[28, 28]),\n",
" tf.keras.layers.Dense(100, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\"),\n",
" tf.keras.layers.Dense(100, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\"),\n",
" tf.keras.layers.Dense(100, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\"),\n",
" tf.keras.layers.Dense(1, activation=\"sigmoid\")\n",
"])\n",
"\n",
"model_B.compile(loss=\"binary_crossentropy\",\n",
" optimizer=tf.keras.optimizers.SGD(learning_rate=0.001),\n",
" metrics=[\"accuracy\"])\n",
"history = model_B.fit(X_train_B, y_train_B, epochs=20,\n",
" validation_data=(X_valid_B, y_valid_B))\n",
"model_B.evaluate(X_test_B, y_test_B)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Model B reaches 91.85% accuracy on the test set. Now let's try reusing the pretrained model A."
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [],
"source": [
"model_A = tf.keras.models.load_model(\"my_model_A\")\n",
"model_B_on_A = tf.keras.Sequential(model_A.layers[:-1])\n",
"model_B_on_A.add(tf.keras.layers.Dense(1, activation=\"sigmoid\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that `model_B_on_A` and `model_A` actually share layers now, so when we train one, it will update both models. If we want to avoid that, we need to build `model_B_on_A` on top of a *clone* of `model_A`:"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [],
"source": [
"tf.random.set_seed(42) # extra code – ensure reproducibility"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"model_A_clone = tf.keras.models.clone_model(model_A)\n",
"model_A_clone.set_weights(model_A.get_weights())"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [],
"source": [
"# extra code – creating model_B_on_A just like in the previous cell\n",
"model_B_on_A = tf.keras.Sequential(model_A_clone.layers[:-1])\n",
"model_B_on_A.add(tf.keras.layers.Dense(1, activation=\"sigmoid\"))"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [],
"source": [
"for layer in model_B_on_A.layers[:-1]:\n",
" layer.trainable = False\n",
"\n",
"optimizer = tf.keras.optimizers.SGD(learning_rate=0.001)\n",
"model_B_on_A.compile(loss=\"binary_crossentropy\", optimizer=optimizer,\n",
" metrics=[\"accuracy\"])"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/4\n",
"7/7 [==============================] - 0s 23ms/step - loss: 1.7893 - accuracy: 0.5550 - val_loss: 1.3324 - val_accuracy: 0.5084\n",
"Epoch 2/4\n",
"7/7 [==============================] - 0s 7ms/step - loss: 1.1235 - accuracy: 0.5350 - val_loss: 0.9199 - val_accuracy: 0.4807\n",
"Epoch 3/4\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.8836 - accuracy: 0.5000 - val_loss: 0.8266 - val_accuracy: 0.4837\n",
"Epoch 4/4\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.8202 - accuracy: 0.5250 - val_loss: 0.7795 - val_accuracy: 0.4985\n",
"Epoch 1/16\n",
"7/7 [==============================] - 0s 21ms/step - loss: 0.7348 - accuracy: 0.6050 - val_loss: 0.6372 - val_accuracy: 0.6914\n",
"Epoch 2/16\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.6055 - accuracy: 0.7600 - val_loss: 0.5283 - val_accuracy: 0.8229\n",
"Epoch 3/16\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.4992 - accuracy: 0.8400 - val_loss: 0.4742 - val_accuracy: 0.8180\n",
"Epoch 4/16\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.4297 - accuracy: 0.8700 - val_loss: 0.4212 - val_accuracy: 0.8773\n",
"Epoch 5/16\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.3825 - accuracy: 0.9050 - val_loss: 0.3797 - val_accuracy: 0.9031\n",
"Epoch 6/16\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.3438 - accuracy: 0.9250 - val_loss: 0.3534 - val_accuracy: 0.9149\n",
"Epoch 7/16\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.3148 - accuracy: 0.9500 - val_loss: 0.3384 - val_accuracy: 0.9001\n",
"Epoch 8/16\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.3012 - accuracy: 0.9450 - val_loss: 0.3179 - val_accuracy: 0.9209\n",
"Epoch 9/16\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.2767 - accuracy: 0.9650 - val_loss: 0.3043 - val_accuracy: 0.9298\n",
"Epoch 10/16\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.2623 - accuracy: 0.9550 - val_loss: 0.2929 - val_accuracy: 0.9308\n",
"Epoch 11/16\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.2512 - accuracy: 0.9600 - val_loss: 0.2830 - val_accuracy: 0.9327\n",
"Epoch 12/16\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.2397 - accuracy: 0.9600 - val_loss: 0.2744 - val_accuracy: 0.9318\n",
"Epoch 13/16\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.2295 - accuracy: 0.9600 - val_loss: 0.2675 - val_accuracy: 0.9327\n",
"Epoch 14/16\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.2225 - accuracy: 0.9600 - val_loss: 0.2598 - val_accuracy: 0.9347\n",
"Epoch 15/16\n",
"7/7 [==============================] - 0s 6ms/step - loss: 0.2147 - accuracy: 0.9600 - val_loss: 0.2542 - val_accuracy: 0.9357\n",
"Epoch 16/16\n",
"7/7 [==============================] - 0s 7ms/step - loss: 0.2077 - accuracy: 0.9600 - val_loss: 0.2492 - val_accuracy: 0.9377\n"
]
}
],
"source": [
"history = model_B_on_A.fit(X_train_B, y_train_B, epochs=4,\n",
" validation_data=(X_valid_B, y_valid_B))\n",
"\n",
"for layer in model_B_on_A.layers[:-1]:\n",
" layer.trainable = True\n",
"\n",
"optimizer = tf.keras.optimizers.SGD(learning_rate=0.001)\n",
"model_B_on_A.compile(loss=\"binary_crossentropy\", optimizer=optimizer,\n",
" metrics=[\"accuracy\"])\n",
"history = model_B_on_A.fit(X_train_B, y_train_B, epochs=16,\n",
" validation_data=(X_valid_B, y_valid_B))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So, what's the final verdict?"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"63/63 [==============================] - 0s 667us/step - loss: 0.2546 - accuracy: 0.9385\n"
]
},
{
"data": {
"text/plain": [
"[0.2546142041683197, 0.9384999871253967]"
]
},
"execution_count": 44,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model_B_on_A.evaluate(X_test_B, y_test_B)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Great! We got a bit of transfer: the model's accuracy went up 2 percentage points, from 91.85% to 93.85%. This means the error rate dropped by almost 25%:"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.24539877300613477"
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"1 - (100 - 93.85) / (100 - 91.85)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Faster Optimizers"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [],
"source": [
"# extra code – a little function to test an optimizer on Fashion MNIST\n",
"\n",
"def build_model(seed=42):\n",
" tf.random.set_seed(seed)\n",
" return tf.keras.Sequential([\n",
" tf.keras.layers.Flatten(input_shape=[28, 28]),\n",
" tf.keras.layers.Dense(100, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\"),\n",
" tf.keras.layers.Dense(100, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\"),\n",
" tf.keras.layers.Dense(100, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\"),\n",
" tf.keras.layers.Dense(10, activation=\"softmax\")\n",
" ])\n",
"\n",
"def build_and_train_model(optimizer):\n",
" model = build_model()\n",
" model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
" metrics=[\"accuracy\"])\n",
" return model.fit(X_train, y_train, epochs=10,\n",
" validation_data=(X_valid, y_valid))"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [],
"source": [
"optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.6877 - accuracy: 0.7677 - val_loss: 0.4960 - val_accuracy: 0.8172\n",
"Epoch 2/10\n",
"1719/1719 [==============================] - 2s 948us/step - loss: 0.4619 - accuracy: 0.8378 - val_loss: 0.4421 - val_accuracy: 0.8404\n",
"Epoch 3/10\n",
"1719/1719 [==============================] - 1s 868us/step - loss: 0.4179 - accuracy: 0.8525 - val_loss: 0.4188 - val_accuracy: 0.8538\n",
"Epoch 4/10\n",
"1719/1719 [==============================] - 1s 866us/step - loss: 0.3902 - accuracy: 0.8621 - val_loss: 0.3814 - val_accuracy: 0.8604\n",
"Epoch 5/10\n",
"1719/1719 [==============================] - 1s 869us/step - loss: 0.3686 - accuracy: 0.8691 - val_loss: 0.3665 - val_accuracy: 0.8656\n",
"Epoch 6/10\n",
"1719/1719 [==============================] - 2s 925us/step - loss: 0.3553 - accuracy: 0.8732 - val_loss: 0.3643 - val_accuracy: 0.8720\n",
"Epoch 7/10\n",
"1719/1719 [==============================] - 2s 908us/step - loss: 0.3385 - accuracy: 0.8778 - val_loss: 0.3611 - val_accuracy: 0.8684\n",
"Epoch 8/10\n",
"1719/1719 [==============================] - 2s 926us/step - loss: 0.3297 - accuracy: 0.8796 - val_loss: 0.3490 - val_accuracy: 0.8726\n",
"Epoch 9/10\n",
"1719/1719 [==============================] - 2s 893us/step - loss: 0.3200 - accuracy: 0.8850 - val_loss: 0.3625 - val_accuracy: 0.8666\n",
"Epoch 10/10\n",
"1719/1719 [==============================] - 2s 886us/step - loss: 0.3097 - accuracy: 0.8881 - val_loss: 0.3656 - val_accuracy: 0.8672\n"
]
}
],
"source": [
"history_sgd = build_and_train_model(optimizer) # extra code"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Momentum optimization"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [],
"source": [
"optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1719/1719 [==============================] - 2s 941us/step - loss: 0.6877 - accuracy: 0.7677 - val_loss: 0.4960 - val_accuracy: 0.8172\n",
"Epoch 2/10\n",
"1719/1719 [==============================] - 2s 878us/step - loss: 0.4619 - accuracy: 0.8378 - val_loss: 0.4421 - val_accuracy: 0.8404\n",
"Epoch 3/10\n",
"1719/1719 [==============================] - 2s 898us/step - loss: 0.4179 - accuracy: 0.8525 - val_loss: 0.4188 - val_accuracy: 0.8538\n",
"Epoch 4/10\n",
"1719/1719 [==============================] - 2s 934us/step - loss: 0.3902 - accuracy: 0.8621 - val_loss: 0.3814 - val_accuracy: 0.8604\n",
"Epoch 5/10\n",
"1719/1719 [==============================] - 2s 910us/step - loss: 0.3686 - accuracy: 0.8691 - val_loss: 0.3665 - val_accuracy: 0.8656\n",
"Epoch 6/10\n",
"1719/1719 [==============================] - 2s 913us/step - loss: 0.3553 - accuracy: 0.8732 - val_loss: 0.3643 - val_accuracy: 0.8720\n",
"Epoch 7/10\n",
"1719/1719 [==============================] - 2s 893us/step - loss: 0.3385 - accuracy: 0.8778 - val_loss: 0.3611 - val_accuracy: 0.8684\n",
"Epoch 8/10\n",
"1719/1719 [==============================] - 2s 968us/step - loss: 0.3297 - accuracy: 0.8796 - val_loss: 0.3490 - val_accuracy: 0.8726\n",
"Epoch 9/10\n",
"1719/1719 [==============================] - 2s 913us/step - loss: 0.3200 - accuracy: 0.8850 - val_loss: 0.3625 - val_accuracy: 0.8666\n",
"Epoch 10/10\n",
"1719/1719 [==============================] - 1s 858us/step - loss: 0.3097 - accuracy: 0.8881 - val_loss: 0.3656 - val_accuracy: 0.8672\n"
]
}
],
"source": [
"history_momentum = build_and_train_model(optimizer) # extra code"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Nesterov Accelerated Gradient"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [],
"source": [
"optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9,\n",
" nesterov=True)"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1719/1719 [==============================] - 2s 907us/step - loss: 0.6777 - accuracy: 0.7711 - val_loss: 0.4796 - val_accuracy: 0.8260\n",
"Epoch 2/10\n",
"1719/1719 [==============================] - 2s 898us/step - loss: 0.4570 - accuracy: 0.8398 - val_loss: 0.4358 - val_accuracy: 0.8396\n",
"Epoch 3/10\n",
"1719/1719 [==============================] - 1s 872us/step - loss: 0.4140 - accuracy: 0.8537 - val_loss: 0.4013 - val_accuracy: 0.8566\n",
"Epoch 4/10\n",
"1719/1719 [==============================] - 2s 902us/step - loss: 0.3882 - accuracy: 0.8629 - val_loss: 0.3802 - val_accuracy: 0.8616\n",
"Epoch 5/10\n",
"1719/1719 [==============================] - 2s 913us/step - loss: 0.3666 - accuracy: 0.8703 - val_loss: 0.3689 - val_accuracy: 0.8638\n",
"Epoch 6/10\n",
"1719/1719 [==============================] - 2s 882us/step - loss: 0.3531 - accuracy: 0.8732 - val_loss: 0.3681 - val_accuracy: 0.8688\n",
"Epoch 7/10\n",
"1719/1719 [==============================] - 2s 958us/step - loss: 0.3375 - accuracy: 0.8784 - val_loss: 0.3658 - val_accuracy: 0.8670\n",
"Epoch 8/10\n",
"1719/1719 [==============================] - 2s 895us/step - loss: 0.3278 - accuracy: 0.8815 - val_loss: 0.3598 - val_accuracy: 0.8682\n",
"Epoch 9/10\n",
"1719/1719 [==============================] - 2s 878us/step - loss: 0.3183 - accuracy: 0.8855 - val_loss: 0.3472 - val_accuracy: 0.8720\n",
"Epoch 10/10\n",
"1719/1719 [==============================] - 2s 921us/step - loss: 0.3081 - accuracy: 0.8891 - val_loss: 0.3624 - val_accuracy: 0.8708\n"
]
}
],
"source": [
"history_nesterov = build_and_train_model(optimizer) # extra code"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## AdaGrad"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [],
"source": [
"optimizer = tf.keras.optimizers.Adagrad(learning_rate=0.001)"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 1.0003 - accuracy: 0.6822 - val_loss: 0.6876 - val_accuracy: 0.7744\n",
"Epoch 2/10\n",
"1719/1719 [==============================] - 2s 912us/step - loss: 0.6389 - accuracy: 0.7904 - val_loss: 0.5837 - val_accuracy: 0.8048\n",
"Epoch 3/10\n",
"1719/1719 [==============================] - 2s 930us/step - loss: 0.5682 - accuracy: 0.8105 - val_loss: 0.5379 - val_accuracy: 0.8154\n",
"Epoch 4/10\n",
"1719/1719 [==============================] - 2s 878us/step - loss: 0.5316 - accuracy: 0.8215 - val_loss: 0.5135 - val_accuracy: 0.8244\n",
"Epoch 5/10\n",
"1719/1719 [==============================] - 1s 855us/step - loss: 0.5076 - accuracy: 0.8295 - val_loss: 0.4937 - val_accuracy: 0.8288\n",
"Epoch 6/10\n",
"1719/1719 [==============================] - 1s 868us/step - loss: 0.4905 - accuracy: 0.8338 - val_loss: 0.4821 - val_accuracy: 0.8312\n",
"Epoch 7/10\n",
"1719/1719 [==============================] - 2s 940us/step - loss: 0.4776 - accuracy: 0.8371 - val_loss: 0.4705 - val_accuracy: 0.8348\n",
"Epoch 8/10\n",
"1719/1719 [==============================] - 2s 966us/step - loss: 0.4674 - accuracy: 0.8409 - val_loss: 0.4611 - val_accuracy: 0.8362\n",
"Epoch 9/10\n",
"1719/1719 [==============================] - 2s 892us/step - loss: 0.4587 - accuracy: 0.8435 - val_loss: 0.4548 - val_accuracy: 0.8406\n",
"Epoch 10/10\n",
"1719/1719 [==============================] - 2s 873us/step - loss: 0.4511 - accuracy: 0.8458 - val_loss: 0.4469 - val_accuracy: 0.8424\n"
]
}
],
"source": [
"history_adagrad = build_and_train_model(optimizer) # extra code"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## RMSProp"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [],
"source": [
"optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001, rho=0.9)"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.5138 - accuracy: 0.8135 - val_loss: 0.4413 - val_accuracy: 0.8338\n",
"Epoch 2/10\n",
"1719/1719 [==============================] - 2s 942us/step - loss: 0.3932 - accuracy: 0.8590 - val_loss: 0.4518 - val_accuracy: 0.8370\n",
"Epoch 3/10\n",
"1719/1719 [==============================] - 2s 948us/step - loss: 0.3711 - accuracy: 0.8692 - val_loss: 0.3914 - val_accuracy: 0.8686\n",
"Epoch 4/10\n",
"1719/1719 [==============================] - 2s 949us/step - loss: 0.3643 - accuracy: 0.8735 - val_loss: 0.4176 - val_accuracy: 0.8644\n",
"Epoch 5/10\n",
"1719/1719 [==============================] - 2s 970us/step - loss: 0.3578 - accuracy: 0.8769 - val_loss: 0.3874 - val_accuracy: 0.8696\n",
"Epoch 6/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3561 - accuracy: 0.8775 - val_loss: 0.4650 - val_accuracy: 0.8590\n",
"Epoch 7/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3528 - accuracy: 0.8783 - val_loss: 0.4122 - val_accuracy: 0.8774\n",
"Epoch 8/10\n",
"1719/1719 [==============================] - 2s 989us/step - loss: 0.3491 - accuracy: 0.8811 - val_loss: 0.5151 - val_accuracy: 0.8586\n",
"Epoch 9/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3479 - accuracy: 0.8829 - val_loss: 0.4457 - val_accuracy: 0.8856\n",
"Epoch 10/10\n",
"1719/1719 [==============================] - 2s 1000us/step - loss: 0.3437 - accuracy: 0.8830 - val_loss: 0.4781 - val_accuracy: 0.8636\n"
]
}
],
"source": [
"history_rmsprop = build_and_train_model(optimizer) # extra code"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Adam Optimization"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [],
"source": [
"optimizer = tf.keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9,\n",
" beta_2=0.999)"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.4949 - accuracy: 0.8220 - val_loss: 0.4110 - val_accuracy: 0.8428\n",
"Epoch 2/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3727 - accuracy: 0.8637 - val_loss: 0.4153 - val_accuracy: 0.8370\n",
"Epoch 3/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3372 - accuracy: 0.8756 - val_loss: 0.3600 - val_accuracy: 0.8708\n",
"Epoch 4/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3126 - accuracy: 0.8833 - val_loss: 0.3498 - val_accuracy: 0.8760\n",
"Epoch 5/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2965 - accuracy: 0.8901 - val_loss: 0.3264 - val_accuracy: 0.8794\n",
"Epoch 6/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2821 - accuracy: 0.8947 - val_loss: 0.3295 - val_accuracy: 0.8782\n",
"Epoch 7/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2672 - accuracy: 0.8993 - val_loss: 0.3473 - val_accuracy: 0.8790\n",
"Epoch 8/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2587 - accuracy: 0.9020 - val_loss: 0.3230 - val_accuracy: 0.8818\n",
"Epoch 9/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2500 - accuracy: 0.9057 - val_loss: 0.3676 - val_accuracy: 0.8744\n",
"Epoch 10/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2428 - accuracy: 0.9073 - val_loss: 0.3879 - val_accuracy: 0.8696\n"
]
}
],
"source": [
"history_adam = build_and_train_model(optimizer) # extra code"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Adamax Optimization**"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [],
"source": [
"optimizer = tf.keras.optimizers.Adamax(learning_rate=0.001, beta_1=0.9,\n",
" beta_2=0.999)"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.5327 - accuracy: 0.8151 - val_loss: 0.4402 - val_accuracy: 0.8340\n",
"Epoch 2/10\n",
"1719/1719 [==============================] - 2s 935us/step - loss: 0.3950 - accuracy: 0.8591 - val_loss: 0.3907 - val_accuracy: 0.8512\n",
"Epoch 3/10\n",
"1719/1719 [==============================] - 2s 933us/step - loss: 0.3563 - accuracy: 0.8715 - val_loss: 0.3730 - val_accuracy: 0.8676\n",
"Epoch 4/10\n",
"1719/1719 [==============================] - 2s 942us/step - loss: 0.3335 - accuracy: 0.8797 - val_loss: 0.3453 - val_accuracy: 0.8738\n",
"Epoch 5/10\n",
"1719/1719 [==============================] - 2s 993us/step - loss: 0.3129 - accuracy: 0.8853 - val_loss: 0.3270 - val_accuracy: 0.8792\n",
"Epoch 6/10\n",
"1719/1719 [==============================] - 2s 926us/step - loss: 0.2986 - accuracy: 0.8913 - val_loss: 0.3396 - val_accuracy: 0.8772\n",
"Epoch 7/10\n",
"1719/1719 [==============================] - 2s 939us/step - loss: 0.2854 - accuracy: 0.8949 - val_loss: 0.3390 - val_accuracy: 0.8770\n",
"Epoch 8/10\n",
"1719/1719 [==============================] - 2s 949us/step - loss: 0.2757 - accuracy: 0.8984 - val_loss: 0.3147 - val_accuracy: 0.8854\n",
"Epoch 9/10\n",
"1719/1719 [==============================] - 2s 952us/step - loss: 0.2662 - accuracy: 0.9020 - val_loss: 0.3341 - val_accuracy: 0.8760\n",
"Epoch 10/10\n",
"1719/1719 [==============================] - 2s 957us/step - loss: 0.2542 - accuracy: 0.9063 - val_loss: 0.3282 - val_accuracy: 0.8780\n"
]
}
],
"source": [
"history_adamax = build_and_train_model(optimizer) # extra code"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"**Nadam Optimization**"
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"optimizer = tf.keras.optimizers.Nadam(learning_rate=0.001, beta_1=0.9,\n",
" beta_2=0.999)"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1719/1719 [==============================] - 3s 1ms/step - loss: 0.4826 - accuracy: 0.8284 - val_loss: 0.4092 - val_accuracy: 0.8456\n",
"Epoch 2/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3610 - accuracy: 0.8667 - val_loss: 0.3893 - val_accuracy: 0.8592\n",
"Epoch 3/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3270 - accuracy: 0.8784 - val_loss: 0.3653 - val_accuracy: 0.8712\n",
"Epoch 4/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3049 - accuracy: 0.8874 - val_loss: 0.3444 - val_accuracy: 0.8726\n",
"Epoch 5/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2897 - accuracy: 0.8905 - val_loss: 0.3174 - val_accuracy: 0.8810\n",
"Epoch 6/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2753 - accuracy: 0.8981 - val_loss: 0.3389 - val_accuracy: 0.8830\n",
"Epoch 7/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2652 - accuracy: 0.9000 - val_loss: 0.3725 - val_accuracy: 0.8734\n",
"Epoch 8/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2563 - accuracy: 0.9034 - val_loss: 0.3229 - val_accuracy: 0.8828\n",
"Epoch 9/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2463 - accuracy: 0.9079 - val_loss: 0.3353 - val_accuracy: 0.8818\n",
"Epoch 10/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2402 - accuracy: 0.9091 - val_loss: 0.3813 - val_accuracy: 0.8740\n"
]
}
],
"source": [
"history_nadam = build_and_train_model(optimizer) # extra code"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**AdamW Optimization**"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import tensorflow_addons as tfa\n",
"\n",
"optimizer = tfa.optimizers.AdamW(weight_decay=1e-5, learning_rate=0.001,\n",
" beta_1=0.9, beta_2=0.999)"
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1719/1719 [==============================] - 3s 1ms/step - loss: 0.4945 - accuracy: 0.8220 - val_loss: 0.4203 - val_accuracy: 0.8424\n",
"Epoch 2/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3735 - accuracy: 0.8629 - val_loss: 0.4014 - val_accuracy: 0.8474\n",
"Epoch 3/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3391 - accuracy: 0.8753 - val_loss: 0.3347 - val_accuracy: 0.8760\n",
"Epoch 4/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3155 - accuracy: 0.8827 - val_loss: 0.3441 - val_accuracy: 0.8720\n",
"Epoch 5/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2989 - accuracy: 0.8892 - val_loss: 0.3218 - val_accuracy: 0.8786\n",
"Epoch 6/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2862 - accuracy: 0.8931 - val_loss: 0.3423 - val_accuracy: 0.8814\n",
"Epoch 7/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2738 - accuracy: 0.8970 - val_loss: 0.3593 - val_accuracy: 0.8764\n",
"Epoch 8/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2648 - accuracy: 0.8993 - val_loss: 0.3263 - val_accuracy: 0.8856\n",
"Epoch 9/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2583 - accuracy: 0.9035 - val_loss: 0.3642 - val_accuracy: 0.8680\n",
"Epoch 10/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2483 - accuracy: 0.9054 - val_loss: 0.3696 - val_accuracy: 0.8702\n"
]
}
],
"source": [
"history_adamw = build_and_train_model(optimizer) # extra code"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAtgAAAHoCAYAAABzQZg1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAD0SElEQVR4nOzdd5gUVdbA4d+t6jzdk4c0wBBmQJAgoIiCSjChILqgIihmXOPnrqirmLMsxtXVlaC4LiYUs5hRMYEkRVByhsl5Ond9f3RPzzQzYKMwifM+zzx0Vd2qvl2Oevpy6hxlGAZCCCGEEEKIA0Nr7AkIIYQQQgjRkkiALYQQQgghxAEkAbYQQgghhBAHkATYQgghhBBCHEASYAshhBBCCHEASYAthBBCCCHEAdSgAbZS6lSl1G9KqfVKqX/Uc/xGpdSKyM8qpVRQKZXakHMUQgghhBDiz1ANVQdbKaUDa4GTgO3AEuA8wzBW72X8aOBvhmEMb5AJCiGEEEIIcQA05Ar2QGC9YRgbDcPwAa8AY/Yx/jzg5QaZmRBCCCGEEAdIQwbYmcC2WtvbI/vqUEo5gFOBNxpgXkIIIYQQQhwwpgZ8L1XPvr3lp4wGvjEMo6jeCyk1GZgMYLPZBnTs2PHAzLAZ0Coq0QsLATBsVgKtW8d9bigUQtPkudbfI/cpPnKf4if3Kj5yn+In9yo+cp/iI/cpfmvXri0wDCPj98Y1ZIC9HehQa7s9sHMvY8ezj/QQwzCeA54D6N69u/Hbb78dqDk2ef68PNYff0J4w2ym23ffoTsT4jp34cKFDB069OBNroWQ+xQfuU/xk3sVH7lP8ZN7FR+5T/GR+xQ/pdSWeMY15NeVJUCOUqqzUspCOIh+Z89BSqkk4ATg7QacW7NhbtUKa48e4Q2/n6rvv2vcCQkhhBBCiBgNFmAbhhEArgE+AtYArxmG8YtS6q9Kqb/WGnoW8LFhGJUNNbfmxnnccdHXFV8vasSZCCGEEEKIPTVkigiGYXwAfLDHvmf32H4BeKHhZtX8OI8bQuFzzwFQ8fVXGIaBUvWluAshhBBCiIYmGe3NkP2II9CcTgACO3fh27ChkWckhBBCCCGqNf8Au4Ea5TQlymwm4dhjo9uSJiKEEEII0XQ0aIrIwWCv2MKviz/hsIEn1Xu8rKyMvLw8/H5/A8/s4ApNuoDgmDMA2Gm1krdmze+ek5SUxJo4xrUkCQkJtG/fXsoPCSGEEKLBNPsA20QA95ePQz0BdllZGbm5uWRmZmK321tUnnLI78dbXZ5QKWzduqF0fZ/nlJeX43K5GmB2TUMoFGLHjh0UFBTQqlWrxp6OEEIIIQ4RLWJZr2/FN2xbt7LO/ry8PDIzM3E4HC0quAbQzGY0qy28YRiEKqXoyp40TaN169aUlpY29lSEEEIIcQhpEQG2pgx2Lni0zn6/34/dbm+EGTUMzeWMvg5VVDTiTJous9lMIBBo7GkIIYQQ4hDSIgJsgL4F71OYu73O/pa2cl1bdSURgGB5BcYh+MDn72nJ//yFEEII0TQ1+wDbgxUAm/Kz9r3HG3cyDUxzOFCRh/cMvw/D52vkGQkhhBBCiOYfYJuTo68P2/YK7sryxptMA1OahpZQK02k/ND57EIIIYQQTVWzD7CLTB6WmcIVIlIo56f3n2nkGR0Y+fn5XHXVVXTq1Amr1Urr1q0ZMWIEn3zySXTMxo0b+ettU+l+8skk9+9Px8MPZ9iwYcyZMwdfrdVspRRKKRITE3E4HHTp0oUJEyawaJHUzxZCCCGEONCafZk+d8jNC1m96L/hcwAy18wmGPg7uql5f7SxY8dSVVXFrFmzyM7OJi8vjy+//JLCwkIAfvzxR0aMGEGPww7jkVtuoXvnzlR5vWz0eJgZOWfw4MHR682YMYOhQ4diNpvZuHEjc+bM4fjjj+fhhx/mxhtvbKyPKYQQQgjR4jTvKDTiW7ayRXOSFaqgvbGL5Z/9j36nXNjY0/rDSkpK+Prrr/nkk08YMWIEAFlZWRx11FEAGIbBhRdeSE5ODt9+9x2+DRswvF4AjsrKYsLEiXUeeExOTqZ169a4XC6ysrIYNmwY7dq145ZbbuGss84iOzu7YT+kEEIIIUQL1exTRAC8IR/PZPaLbtuXNO80EafTidPp5J133sHj8dQ5vmLFClavXs2UKVPQNA3dWTsPO1yuL57qGTfccAOhUIi33nrrgM1dCCGEEOJQ1yJWsAG+shVSgplk/BwWWMOvP3wMiR3qjOv0j/cbYXZhmx86Pa5xJpOJF154gcsvv5znnnuOfv36MXjwYM4++2yOPvpo1q5dC0D37t0B0FwuSjdvJnvECFAKlOLWW2/l1ltv3ef7pKWl0apVKzZu3PjnPpgQQgghhIhq9ivYJhX+jlAeqOCp1jWr2O4vH2+kGR0YY8eOZefOnbz77ruMHDmSb7/9lkGDBvHAAw/UGas5HLhcLr6fN4/vX3+ddm3bxjzkuC+GYUitaCGEEEKIA6jZB9iJemL09WfJXvyR130rvyXgb951oW02GyeddBJ33HEH3377LZdeeil33XUXnTp1AuDXX38FwuX6zC4XXTt2pGvHjljifMCzoKCA/Px8unTpcrA+ghBCCCHEIafZp4gkaAmk2lIp8hRR4Cvm2ZTeXFv8M5oyCLrL6oyPN02jKerZsyeBQIDDDjuMHj16MG3aNM455xx0XUdzOglG6mAboVBc13vkkUfQNI0xY8YczGkLIYQQQhxSmv0KtkJxfo/zo9sftLJSHV5aQu5muYpdWFjI8OHDeemll/jpp5/YtGkTr7/+OtOmTWPEiBEkJSXxwgsvsGHDBo455hjefvtt1u/axa8bN/L8G2+wY9cuNC32H21JSQm5ubls3bqVL774gosuuoiHH36Yhx56SCqICCGEEEIcQM1+BRvgnO7nMPPnmVQFqtjuy+MVZxcmVGxEYeApycWZUfdhx6bM6XQyaNAgnnjiCdavX4/X6yUzM5MJEyZw2223ATBw4ECWLVvGgw8+yLXXXsvu3buxW6306taNO6+9liuuujrmmpdffjkAVquVtm3bMmjQIBYuXMjxxx/f4J9PCCGEEKIlaxEBdpI1ibO7nc2c1XMAmNc2gwnrwpUxbP4iQsF2aLremFPcL1arlQceeKDeBxpry87OZtasWdFt/65dBCKNaEy1Vu6ra2KXl5fjcrkOwoyFEEIIIUS1Zp8iUu2Cnhdg0sLfF9YFdvGptTUAJkJUleY15tQajFarHnYwUg9bCCGEEEI0rBYTYLdOaM2oLqOi2y9kdo6+tngK63Q2bIm0hIRwHWzA8HkJxVmqTwghhBBCHDgtJsAGuPjwi1GEA8yVxk58KvzxLPipKitszKk1CKVp4SA7IhSpKiKEEEIIIRpOiwqwuyR3YViHYdHtct0cfa1X5h8Sq9h6rRzrUIWkiQghhBBCNLQWFWADXNL7kuhrHyG8kRVtGx48lXXrYrc0MXnYlZVx18QWQgghhBAHRosLsPtm9GVA6wEAGBjkme3RY0ZFbmNNq8FoVivKYglvhEKEqqoad0JCCCGEEIeYFhdgA1zSq2YVu1wLEoi8tgcr8XpafsCp11rFDkk1ESGEEEKIBtUiA+zjMo8jJyUHCNeAzjPZgHCBDX/p7sacWoPQauVhByvkQUchhBBCiIbUIgNspRQXH35xdLtUJ9o+3REoa5bt0/dHTLk+r5TrE0IIIYRoSC0ywAY4tfOp6Fq4e2PICJGvWwHQVLh9ektWp1yfVBMRQgghhGgwLTbANmtmnOaaXOQSk0Z1kb5w+/Rg40wsDhdddBFKKS677LI6x2666SaUUowaNaqeM2vE5GE3YoCtlGLevHmN9v5CCCGEEA2txQbYAHaTPbqKHTCCFGnhutjNoX16hw4dePXVV6msrIzuCwQC/Pe//6Vjx46/e762Rz1sKdcnhBBCCNEwWnSArSmNVFtqdLvQZIquYjf19ul9+vQhJyeH1157Lbrv/fffx2azMXTo0Oi+UCjEvffeS4cOHbBarfTu3Zu3334bZbGgzGa27NiB/fDDefnFFxk5ciR2u51+/frx008/sWrVKo499lgSEhIYMmQImzZtipnDu+++y4ABA7DZbHTu3JmpU6fiq5XP3alTJ+677z6uuOIKEhMTad++Pf/85z9jjgOcffbZKKWi23fddRe9evWKea8XXngBZ61V9+oxc+bMoVOnTjidTi6++GJ8Ph///ve/6dChA2lpafz9738nJF8ehBBCCNGEtOgAGyDVloqKPPDnJ0iZCq9oN4f26ZdeeimzZ8+Obs+ePZuLL744+nkAnnjiCf75z3/y8MMP8/PPP3PWWWfxl7/8hZUrV8asYt91331cf/31LF++nOTkZCZMmMC1117L/fffz+LFi/F4PFx33XXR8R999BETJ07kmmuu4ZdffmH27NnMmzePW2+9NWaOjz32GL1792bZsmXcfPPN3HTTTXz33XcALFmyBIAZM2awa9eu6Ha8Nm/ezNtvv817773HG2+8weuvv86YMWNYsmQJH3/8MTNnzuRf//oX8+fP36/rCiGEEEIcTKbGnsDBZtJMpFhTKPIUAZD0n+OixxL2dtLBclfpfg2fMGECU6ZMYd26dbhcLhYsWMC//vUv7rjjjuiY6dOnM2XKFCZMmADAPffcw1dffcX06dOZ8+9/R8ddd+GFnHLKKbhcLm644QZGjx7NG2+8wbBh4dby11xzDddcc010/P3338+NN97IxReHq7F07dqVhx9+mPPPP59//vOf0SD/5JNPjp537bXX8uSTT/LZZ59xzDHHkJGRAUBycjJt2rTZ37tFMBjk+eefJykpiV69enHqqafy5ZdfsmPHDiwWCz169GDw4MF88cUXjB07dr+vL4QQQghxMLT4ABsgzZ4WDbCbk5SUFM466yxmz55NcnIyQ4cOjcm/LisrY+fOnQwePDjmvCFDhvDBBx/ElOvr1bkzBMItd1q3bg1A7969o+e0bt2ayspKqqqqcDgcLF26lMWLF/Pwww9Hx4RCIdxuN7t376Zt27ZAOJWltnbt2pGXd2Dy2zt27EhSUlLMHLt164alulNlZN+Bej8hhBBCiAPhkAiwLbqFJGsSpd79W0FuCi655BIuvPBCnE4n99xzT71jaqeM1N6ndB1lD7eKN5nNKI8nZrzZbK5zjep85lAoxJ133snZZ59d59rVK9N7XqP6Or+XE61pWp38d7/fX2dcfdeub1+wCVeEEUIIIcSh55AIsAHS7emUekv55ZpvAcj2+bFiYBjgS+uO1eZo5BnWb8SIEVgsFgoKCjjzzDNjjiUmJtKuXTsWLVrE8OHDo/sXLVpEz549AdAdNYkwmtsd9/v279+fX3/9lezs7D81f7PZXCcAzsjIIDc3F8MwooH9ihUr/tT7CCGEEEI0FYdMgG0z2XBanFT4wjWhc01WOgY80fbpVluXRp5h/ZRS/PTTTxiGgdVqrXP8xhtv5I477iAnJ4cBAwbw0ksv8fXXX7N06VIANGdNgK3cboJlZXG97x133MGoUaPIysrinHPOwWQysWrVKhYvXsy0adPinn+nTp347LPPOOGEE7BaraSkpDB06FCKiop44IEHGD9+PAsXLpRa2UIIIYRoMVp8FZHa0u3p0dcVWojqpARHoAx/E26f7nK5SExMrPfYddddx4033shNN91Er169mD9/Pm+88QZHHHEEAKpWvjKGgW/rVvxx5CyfcsopvP/++3zxxRcMHDiQgQMH8tBDD8VVg7u2Rx55hC+++IIOHTrQr18/AHr06MEzzzzDc889R58+ffjkk0/qVCcRQgghhGiuVFOuBR2P7t27G7/99lu9x9asWUOPHj2i24ZhsKlsE25/OFUiOQiZwXBgXWFOx5nR4eBPuBGEqqrwbd2KEXnIEUCZLZjbZ6InNHgtlQa35+/B71m4cGFMrXFRP7lP8ZN7FR+5T/GTexUfuU/xkfsUP6XUUsMwjvy9cYfUCrZSinRbzSp2ma6ozg62+YsIBgP1n9jMaQ4H1uxsDEdNnrnh9+HbtAn/7t3S5VEIIYQQ4gA6pAJsAJfFhUUPp02EMCjQa9qnu0vzG3NqB5UymQimp2Nu3x6l69H9gYICfBs2ENqPByCFEEIIIcTeHXIBtlIqJhe7WNeoXr+1eAqadPv0P00pTMnJWLKz0Wq1JQ95vXg3bsSfn9+yP78QQgghRAM45AJsgCRrEiYtXEAliEGxVt0+PdDk26cfCJrZjCUrC3PbtqAivwKGQSA3F9/GTYS83sadoBBCCCFEM3ZIBtia0kizp0W3C0xmqtdt9cq8Q2IVVymFKS0Na3ZXtEgzGoCQuwrvhg0EiooOifsghBBCCHGgHZIBNkCKNQUtsnobIESZFn5tw4unMr5a0S2BZrVi6dIFU6tW0bbqhEL4d+7Et2ULoXo6LAohhBBCiL07ZANsXdNJtaVGt/N0S3QV26jIbZxJNRKlFOZWrbB26YKq1cwmVFGBb/16gqXNr8W8EEIIIURjOWQDbIBUe2q0VbdPhaiKvHaEKvF6qhpzao1Cs9uxdu2KKa0mfcYIBvFt24Zv27aYOtpCCCGEEKJ+h3SAbdbMJFuTo9u5ppquh4HS3Y0wo8anNA1z27ZYOnVGmc3R/cHSUrzr1xMsL2/E2QkhhBBCNH2HdIANxDzs6FYG7sgqtr2Jt08/2HRnAtbsbPTklOg+IxDAt2ULvp07MYLBfZwthBBCCHHoOuQDbKtuJdGaGN3OizSe0ZSBt6RxcrEvuugilFLcd999MfsXLlyIUoqCgoI//R7xXEvpOpb2mVg6dkTppuj+YFER3g0bCFYdemk0QgghhBC/55APsIGYxjMVGlSvWzdm+3Sbzca0adPIz2/87pJ6YiLWnGz0xJovIobPh2/jRvy5ufikbrYQQgghRJQE2IDdZCfBnBDdzmsC7dOHDRtGp06duPfee/c6ZvXq1Zx++um4XC5atWrFeeedx+7dNbnjP//8MyNGjCAxMRGXy8Wxxx7LF198webNmxk2bBgAGRkZKKW46KKLADAMg2nTptG1a1fsdju9e/fmpZdeQplMmDt0YEcwiKN3b1774ANGXnopiVlZ/PuBBwhUVXHvvffSoUMHrFYrvXv35u23347O5ZhjjuGGG26ImX9ZWRl2u5358+cfwDsnhBBCCNG4JMCOqL2KXaYrqtetG6t9uqZpPPTQQzz77LNs2LChzvFdu3Zx/PHH06tXLxYvXsynn35KRUUFZ5xxBqFQuPn7hAkTaNu2LYsXL2b58uXccsst2Gw2OnTowBtvvAHAL7/8wq5du3jiiScAuO2225g1axZPP/00q1ev5pZbbuGKK67g/fffDzenSUoC4I4nn2Tyueey7K23OP2443j07rv557RpPPTQQ/z888+cddZZ/OUvf2HFihUAnH/++bzyyivRuQG88cYb2O12Tj/99IN5K4UQQgghGpTp94e0LL3n9G609/75wp/3a/xpp53G4MGDmTp1Kq+88krMsWeeeYa+ffvy8MMPR/e9+OKLpKam8uOPPzJw4EC2bNnClClTOOywwwBo3bo1LpcLgNTUcA3wVq1akZ4e/nJRWVnJo48+yscff8xxxx0HQOfOnVm8eDFPP/10TCB87fXXc86kSfhzc8EwePyFF/i/SZMYd8wxmNu355577uGrr75i+vTpvPTSS4wfP56//e1vfPHFF4wYMQKA//3vf5x99tlYLDXVW4QQQgghmjtZwW7ipk2bxuuvv86PP/4Ys3/p0qV89dVXOJ3O6E+HDh0Aoivef//737nssssYPnw4999/P2vXrt3ne61evRqPx8Opp54ac91nnnmmzir6UUcdhSk9HWvXrlT4A+zKy+OYfv0IVVXhW7+eQFERgwcPZvXq1QCkpaVxyimn8L///Q8Ir8B/8cUXnH/++QfkPgkhhBBCNBUSYDdxRx11FGPHjuXmm2+O2R8KhTj99NNZsWJFzM+6desYNWoUAHfddRerV6/mzDPP5Ntvv+WYY45h9uzZe32v6vSNd999N+aav/zyCx9//HHM2ISEcM66ZrNh6dwJABVpPW9EWq2HyspQtc45//zzeeONN/B4PLz88st06NCBIUOG/JnbI4QQQgjR5BxyKSK/l6ZR5CliV8UuAHQU3XxeNKBKS8DRplsDzLCuBx54gJ49e7JgwYLovv79+/Paa6+RlZWFuVZDmD3l5OSQk5PDddddx2WXXcbMmTO55JJLomkZwVr1rHv27InVamXLli0MHz487vklJSfTrl07Fm/exLDjhmD4wnVYvvn+e7q3b0+wtBQ9KYkxY8YwefJk3nvvPf73v/8xceLEaCdNIYQQQoiWQlaw95BsTUbXdACCGJRp4VvUmO3Ts7OzmTx5cvRBRICrr76a0tJSzj33XH744Qc2btzIp59+yuTJkykvL8ftdnP11VezcOFCNm/ezA8//MB3331Hz549AcjKykIpxfvvv09+fj4VFRW4XC6mTJnClClTmD17NuvXr2fFihU8++yzPPfcc/uc44033sgjTzzBm0uWsLG4hHueeopvli3jukmTwq3Wt2/Hajbzl7/8hfvuu49ly5ZJeogQQgghWiQJsPegKY00W013xzzdTHUNkcZsn37HHXdgMtX8hUO7du345ptv0DSNU089lcMPP5yrr74aq9WK1WpF13WKi4u58MIL6d69O2eddRYDBw7k0UcfBSAzM5O7776bqVOn0rp1a6655hoA7r33Xu666y6mT5/O4YcfzkknncQbb7xB586d9zm/6667jhtvvJGb//EP+o0Yzntff83LTz5J38gDlsGSErzr1zNh3DhWrlxJ//796dGjx0G6W0IIIYQQjeeQSxGJR4othQJ3ASEjhF8ZVCgNlxGKtE/3YjZbD+r7v/DCC3X2tWrVivLy8ph9OTk5zJs3b6/XmTt3bsx2eXl5tIoIwO23387tt98eM0YpxbXXXsu1115b7zU7depUb9lCTdPqXM8IBvHv2kWwpCS87fczJCsL386dmFq33uu8hRBCCCGaM1nBrodJM5FiS4lu50VWjsPt0/Maa1rNTrjVenssHTqgdD26P1BYiHfDBkLSal0IIYQQLZAE2HuRZkuLPoDnUVAVed2Y7dObKz0pCUt2Nnqt1XPD68W7cRP+3DyMWs1nhBBCCCGauwYNsJVSpyqlflNKrVdK/WMvY4YqpVYopX5RSn3ZkPOrzaybSbImRbfz9PAqdmO2T2/ONLMZc8eOmNu1Q2nVv3YGgfw8fJs2EfJ6G3V+QgghhBAHSoMF2EopHXgaGAn0BM5TSvXcY0wy8G/gDMMwDgfObqj51SfdVtM+vVJTeCKr2BZPgay6/gFKKUypqViys9Ecjuj+kNuNd/16AgWN05ZeCCGEEOJAasgV7IHAesMwNhqG4QNeAcbsMWYC8KZhGFsBDMNo1IRnq8mKy1KT1pAfySO2EKCqvKixptXsaRYLls6dMbduDdV1sA0D/+7d+DZvJhSpoy2EEEII0RyphloxVEqNA041DOOyyPYFwNGGYVxTa8zjgBk4HHABTxiG8WI915oMTAbIyMgY8Nprr9X7nklJSWRnZ/+peXtDXnL9udHtHJ8PC+DGit/Zvlk1SgkGg+i1HjZsEnw+9IJClL9WUK00gqkpGAkJNQH4n7B+/XpKS0vjHl9RUYHT6fzT79vSyX2Kn9yr+Mh9ip/cq/jIfYqP3Kf4DRs2bKlhGEf+3riGLNNXX6S0Z3RvAgYAIwA78J1S6nvDMNbGnGQYzwHPAXTv3t0YOnRovW+4Zs2amLJ0f4QLF+Wl5VT5wxUvCnWdtsEgdrygGdidiX/q+g1pzzJ9TYWRkkIgL59AQSS33QihFxai+/3hnG3Tn/s1tdls9OvXL+7xCxcuZG+/U6KG3Kf4yb2Kj9yn+Mm9io/cp/jIfTrwGjJFZDvQodZ2e2BnPWMWGIZRaRhGAfAV0LeB5rdX6faaXOxiXae6hohRkVv/CWK/KE3D3KY1ls5dUJEW7gDBsjK869cTLCtrxNkJIYQQQuyfhgywlwA5SqnOSikLMB54Z48xbwPHKaVMSikHcDSwpgHnWC+n2YnVFG4uYwBFkTQLR6gSr7uyEWfWsugJDqxdu6Knpkb3GYEAvq1b8e3YgREMNuLshBBCCCHi02ABtmEYAeAa4CPCQfNrhmH8opT6q1Lqr5Exa4AFwE/AYmCmYRirGmqOe6OUiqkoUqjrVNcQCZQ1jVXsefPmNZt88FGjRnHRRRfVe0zpOpZ27bBkZcWkhgSLi8Or2ZXyhUYIIYQQTVuD1sE2DOMDwzC6GYbR1TCM+yP7njUM49laY/5pGEZPwzB6GYbxeEPOb18SrYmYNTMAIaA4Usu5un36wbB8+XJ0XWfw4MEH5foAGzdu5LLLLiMrKwur1Uq7du0YNmwYc+bMwdeI1Tx0lwtrdjZ6Uk0tcsPvx7dpE/5du6VMohBCCCGaLOnkGCdNaaTZ06LbBboJg+r26QdnFXvGjBlcddVVrFq1ijVrDnymzI8//ki/fv1YtWoV//rXv/j555/54IMPmDx5MnPmzGHJkiV7Pdfv9x/w+exJmUxYOnTA3L79Hq3WC/Bt2EDI7T7ocxBCCCGE2F8SYO+HZGsyuhYO9AIKyiKr2DZ/8QFvn+52u5k7dy6XX34548aNY9asWTHHX3zxRbKysnA4HIwaNYrc3Nggf8OGDYwZM4Y2bdqQkJBA//79+fDDD6PHDcPgwgsvJCcnh2+//ZYzzjiDbt26ccQRR3Deeefx+eefc+yxxwKwefNmlFK8/PLLDB8+HLvdzn/+8x8KCws577zzaN++PXa7ncMPP5znn38+Zh5VVVVcdNFFOJ1OWrduzQMPPLDf98KUnBxuTlOrhFDI68W7cSP+vHxpTiOEEEKIJkUC7P2gazqptpoH8PJ1HYNI+/SSA9s+fd68eWRlZdGnTx8uuOACXnzxxeiq8Q8//MBFF13E5MmTWbFiBaNHj+aOO+6IOb+iooKRI0fyySefsHLlSsaOHcv555/Pr7/+CsCKFStYvXo1U6ZMQdPq/zXYM6f7lltu4aqrrmL16tWceeaZeDwe+vfvz3vvvccvv/zC//3f/3HFFVfw2WefRc+ZMmUKn3zyCW+88QafffYZy5cv56uvvtrv+6GZzViysjC3bQvV8zUMAnm5+DZulFbrQgghhGgyGrIOdpOw5rAef/oa1eGoH/h1P87r8Wv8aR4zZ87kggsuAOCEE07A4XDwzjvvMHbsWJ544glGjBjB1KlTAejWrRtLliyJWeXu27cvffvWVDicOnUqb731FvPmzeO2225j7dpwafHu3btHx5SWlpKZmRndvvXWW7n11luj29deey3jxo2LmeeNN94YfT158mQ+//xzXn75ZUaMGEFFRQWzZs1i9uzZnHLKKQA8//zztG/fPu77UJtSClNaGprTiX/79miKSMjtxrthA+bWrdFTU5vNw55CCCGEaJlkBbsJWr9+Pd988w0TJkwAwoHlxIkTmTlzJhBuoHPMMcfEnLPndmVlJTfddBM9e/YkJSUFp9PJ8uXL2bp1617f1+VysWLFClasWEG7du3qPOR45JGxjYuCwSD3338/ffr0IS0tDafTyZtvvhl9jw0bNuDz+WLm5nQ66d27937ekVia1YqlSxdMrWq1Wg+F8O/ahW/LFkINkB8uhBBCCLE3h9wKdnMwc+ZMgsEgHTt2jO6rzjPetm1bXDnHU6ZMYcGCBUyfPp2cnBwcDgcTJ06MBs3dunUD4Ndff412OdQ0Ldpa3lKr4Uu1hISEmO3p06fzyCOP8MQTT9C7d2+cTie33noreXl5MXM+GJRSmFtloLsiq9mRFJFQRQW+9esxtW2LKTn5oL2/EEIIIcTeHHIB9v6kaezL9vLtlHpLAXCFQnQMhB9ydCd2we5M2tep+xQIBJgzZw4PPvggo0aNijl2wQUX8Pzzz9OzZ0++//77mGN7bi9atIhJkyYxduxYADweD5s2baJHj3CKzBFHHEGPHj2YNm0a55xzDnqtKh3xWrRoEaNHj46mshiGwdq1a0mOBLbZ2dmYzWa+//57unTpAoRX1letWkXXrl33+/3qo9ntWLp2JZCXR6CgIDyPYDAcdJeXh3O2hRBCCCEa0CEXYB8o6fb0aIBdrml4UVgxMMpz4U8E2O+//z4FBQVcfvnlpKWlxRwbP348zzzzDHPnzmXIkCE8+OCDjBs3joULFzJ//vyYsd26dWP+/PmMGTMGs9nM3XffjbfWg4BKKV544QVOPPFEjjnmGKZOnUqPHj0IBoN88803bN++/XeD7m7duvHqq6+yaNEi0tPT+de//sWmTZuiK+JOp5NLL72Um2++mYyMDNq1a8c999xD8AB3ZAy3Wm+D5nLh374dI5IiEiwtJVRZScjjOaDvJ4QQQgixL5KD/QfZTDaclpqycYWm8K10GH+uffqsWbMYNmxYneAa4Oyzz2bLli3RhwefeeYZ+vTpw5tvvsldd90VM/bRRx+lVatWHHfccYwcOZJBgwbVydMeOHAgy5Yto3fv3lx77bX06tWLQYMGMWfOHO6//35uuummfc71tttuY+DAgYwcOZLjjz+ehIQEJk6cGDNm+vTpDBs2jLPOOothw4bRq1cvjj/++D92c36HnpAQbk6TkhLdZwQCBIuK2HXXXYSkC6QQQgghGoCsYP8J6fZ0KnwVAJRoOhkEMRNun261d/lD13znnXf2eqxLly4xec0XX3xxzPFrrrkm+jorK4tPP/005vgVV1yBy+WK2ZednV2nxvaeOnXqVG8+dUpKCm+++eY+z01ISODFF1/kxRdf3Oe4A0XpOpbMTIIuF/6dOzEiqTslr7xK5bff0e6hh3D079cgcxFCCCHEoUlWsP8Eh8mB3WwHwAAKIykVB7N9uoiPnpgYXs1OTIzu82/dypbzzyfv0ccwGrENvBBCCCFaNgmw/wSlFOm29Oh2sa4T5OC2TxfxUyYT5g4d0JOTa7pAhkIUPvccm845F89vaxt3gkIIIYRokSTA/pNcFhcWPVzSLgQU6QevfbrYf0opNIeDLu+8jWPQoOh+76+/snncOApnzcI4wA9dCiGEEOLQJgH2n6SUIt1es4pdqJkIcXDap4s/ztyuHR1nz6L1rbeirFYADL+fvH9OZ8ukC/Ft29bIMxRCCCFESyEB9gGQZE3CpIWfFw0qKNHCt9XiLcAIhRpzaqIWpWmkTrqAzm++ga1Xr+h+99KlbBpzJsWvv35Qm+MIIYQQ4tAgAfYBoCmNNHtNWb0CXccALASoKi9svImJelm7dqXTy3NJv+YaiDyYGqqqYvftd7D9yqvQSksbeYZCCCGEaM4kwD5AUqwpaCp8O/1KUR5ZxdYr82VVtAlSZjMZ11xNp1dextK5c3R/xcKFpN11N7vuuJOqJUvkbyCEEEIIsd8kwD5AdE0n1ZYa3c6PrGLb8OKpLGu8iYl9svfuTef5b5Iy6YLoPs3tpuS119hywSTWn3gieY88gmetVBwRQgghRHwkwD6AUu2pKKUA8ChFVeS1US4l+5oyzWajza230vGF5zFndYw5Fti5i8IZM9l0xhg2jjmTwpkz8e/a1UgzFUIIIURzIAH2AWTWzCRbk6PbBZH83j/bPl00jIRBg+j64YcUTbmB5HPPRU9Kijnu/e038qY/wvrhI9hywSSKX3uNoORrCyGEEGIPEmAfYLUfdqzQNNyRVexAWfyr2BdddBFKKZRSmEwmOnbsyJVXXklxcXF0TKdOnVBK8dJLL9U5f+DAgSilmD59enTfpk2buOyyy2jfvj1Wq5V27dpx+umns3z58jrXVErhcDjo1asX//nPf/br8zd3StPwZ2fT9u67yPn6K9r/+2lcI0+NlvYDwDCoWrKE3Xfcybohx7HtmmsoW/ARIa907xRCCCGEBNgHnFW3kmitac8d0z7dF38AduKJJ7Jr1y42b97MzJkzeffdd7nqqqtixnTo0IFZs2bF7Fu1ahW//PILaWk1gb7f7+ekk06ioKCA1157jbVr1zJv3jwGDhxIUVFRzPl33HEHu3bt4qeffuLMM8/kr3/9K6+++mq9c/S18HbjymLBNXw47R97jJxvFtH2wQdJOPZY0Gr+tTH8fio+/Ywd11/PuiHHsXPqVCq//16a1wghhBCHMAmwD4LajWdKNQ0fkfbppfGvYlutVtq0aUP79u05+eSTOffcc/n4449jxkyYMIHvvvuOjRs3RvfNmjWLcePG4axuDQ788ssvbNiwgUceeYRjjz2WrKwsjj32WO68805GjBgRc02Xy0WbNm3Izs7mvvvuIycnh7feeguAoUOHcuWVVzJlyhQyMjIYPHgwAF999RVHH300NpuN1q1b87e//S0m+B46dCh//etf+b//+z9SUlJISUnhxhtvJNSMKnToTifJZ51Jx9mzyF74Ba1v+Qe2ww+PGRMqL6f0jTfZetHFrB82nNyHp+FZvVqqyAghhBCHGAmwDwK7yU6COSG6XZ2L/Ufbp2/cuJEFCxZgNptj9qenpzN69Gief/55ILyi/NJLL3HppZfGjMvIyEDTNN555x0Cgf17f5vNht/vj26/9NJLGIbB119/zYsvvsiOHTsYOXIk/fr1Y/ny5cyaNYuXX36ZW265JeY6//vf/wiFQnz33Xf85z//4bnnnuPxxx/fr7k0FeZWrUi98EI6vzGPLh98QPpVV2Lu0CFmTCAvj6Lnn2fTX8aycdRoCp59Ft/27Y00YyGEEEI0JFNjT6ChPf3Xzxvtva++I4WKknycaW1/d+yCBQtwOp0Eg0E8Hg8Ajz76aJ1xl1xyCVdccQV3330377zzDsnJyRx//PExYzIzM3nyySe56aabmDZtGgMGDOD4449n/PjxHL7HKmy1QCDASy+9xM8//8yVV14Z3d+5c2ceeeSR6PbUqVNp27Yt//73v9E0jR49evDQQw9xxRVXcO+99+JwOABo27YtTz75JEopDjvsMNauXcujjz7K3//+99+/cU2YtUtnMq67jvRrr8WzciWl775H2YcfEqyVeuPbsIH8x58g//EnsPfrR+LoUSSOHIkpJaURZy6EEEKIg0VWsBtYvO3Tjz/+eFasWMHixYu59tprOe2007juuuvqjDvllFMwDINPPvmEWbNmcckll9R7vauvvpp169Yxd+5chgwZwttvv80RRxzBf//735hxU6dOxel0Yrfbufrqq7nxxhu54ooroscHDBgQM37NmjUcc8wxaLXykocMGYLP52P9+vXRfYMGDYqWMAQ45phj2LFjB2VlLaNGuFIK+xFH0Ob228j5ciEdnvsPiaNHo+z2mHHu5cvJvede1h13PNv+eiWl779PyO1upFkLIYQQ4mCQALsBBYm0Ty/7/fbpDoeD7OxsevfuzZNPPklVVRX33ntvnXGapnHhhRfywAMP8Pnnn3PhhRfu9Zoul4szzjiD+++/n5UrVzJs2DBuv/32mDF///vfWbFiBVu2bKGiooJp06bFBM8JCQkx4w3DiAmca9vb/pZOmc04jz+ezH9Oo9uir2n3z2kkHH9ctC07AIEAFQsXsvOGKawbPISdN99MxdeLMPYzhUcIIYQQTc8hlyJy9bPDG+y9DMNgfcl6fMHwA3/FgQDpoRB6VT5GUvp+BaB33nknI0eOZPLkybRr1y7m2CWXXMIDDzzAaaedVufY3lSnaixbtixmf1paGtnZ2XHPq2fPnrz22muEQqFoIL5o0SIsFgtdu3aNjvvhhx9igvHvv/+edu3akZiYWO91WwotIYGk0aNJGj2aQGEhZR8uoOzdd3GvXBkdE6qqovTtdyh9+x309HQSR44kafQobL17H7JfUoQQQojmTFawDyKlVExd7EJdJ0R1+/T9a1AydOhQDj/8cO677746x7p06UJBQQGvv/56veeuWLGCMWPG8NZbb7F69WrWr1/PrFmzmD17NmedddZ+zWNPV111FTt37uSqq65izZo1vP/++/zjH//gmmuuieZfA+zcuZPrr7+e3377jXnz5vHPf/6Tv/3tb3/qvZsbU1oaqedPpNOrr9D1449Iv+5aLJ07x4wJFhRQ/N//svmcc9l46kjyn3oa35YtjTRjIYQQQvwRh9wKdkNLtiaTV5VHMBQkoBRlmkZyKIRRngfO5P261t///ncuvvhibr755jrHUlNT93pe+/bt6dKlCw8//DBbt24lFArRsWNHpkyZwj/+8Y/9/UgxMjMz+fDDD7nxxhs54ogjSE5OZsKECTzwwAMx4yZOnEgwGOToo49GKcWll156yAXYtVk6diTjqqtIv/JKPL+spuzddyn94H2C+QXRMb4tWyh46ikKnnoKW58+JI0aReJpIzGlp+/jykIIIYRobBJgH2Sa0kizpZFXlQeES/YlhULR9ulWe0Kdc1544YV6rzVhwgQmTJgAwObNm/f5vrWPp6en89hjj1FeXo7L5YrrnPosXLiw3v3HH388P/zwwz7PNZlMPPXUUzz11FP7HHeoUUph73U49l6H0+qmG6n8/nvK3n2P8k8+IVRZGR3n+eknPD/9RO7DD5NwzDEkjR6F68QT0RLq/v4IIYQQonFJgN0AUmwpFLgLCBkhvEpRoTRcRohAWS5We5fGnp5oIpSu4xw8GOfgwYTuupOKL76g9N33qPj6a6iuRR4MUrloEZWLFqFsNlwjRpA4ehTOwYNRe9RJF0IIIUTjkAC7AZg0Eym2FArd4eohBSYNlz8UbZ9utlgbeYaiqdFsNhJHjiRx5EgCxcWUf/Qxpe+9i/vHpdExhsdD2fvvU/b+++gpKSSOPJXEUaOx9ztCHo4UQgghGpEE2A0kzZZGkacIwzCoUhpVSuHAoKo0F3NGx8ae3kG1t9QSER9TSgop488lZfy5+HfsoPS99yl7712862rqjAeLiyme+zLFc1/G3L49iaNOJ2n0aKy1KrkIIYQQomFIFZEGYtbNJFmTotvV7dPtf7B9ujg0mTMzSb9iMp3feYfOb80n9dJLMLVpEzPGv307hc/+h42nj2LjX/5C4ezn8efmNdKMhRBCiEOPBNgNKN1WU/2hXNPwKIVOCHeJBD9i/yilsB12GK1vvJHszz+j45w5JJ89Dm2Ph1i9q9eQN20a64cOZcvFF1PyxpsEy8sbadZCCCHEoUEC7AZkNVlxWWoCoEI9fPst3sK42qcLUR+laSQcPZC2995LzjeLyPzXk7hOPhllsdQMMgyqvvueXVOnsm7wELb/3/WUf/opIZ+v8SYuhBBCtFCSg93A0u3plPvCK4glmk4GQSwEqCwrJCE5o5FnJ5o7zWIh8aSTSDzpJIJlZZR//DGl775H1eLFYBgAGD4f5R99RPlHH6ElJZF48skkjh6F48gjUZp85xZCCCH+LAmwG5jD7MBhdlDlrwLC3R3bBoN/qH26EPuiJyaSPG4cyePG4d+9m7L3P6D0vffwrlkTHRMqLaXk9dcpef11TG3bkjTqdBJHjcbWvVsjzlwIIYRo3mS5qhGk22tysUt0jQB/rH26EPEyt2lD2qWX0GX+m3R5713SrrgCc2ZmzJjArl0UzpjJpjFj2HjGGApmzMC/c2cjzVgIIYRoviTAbgROsxOrKVz7OoSiKFJRxCj/4w87zps3T1a/RVys2dm0+tv1dP30E7Lm/o/k8eeiJyXFjPGuXUv+I4+yfvgItpx/AcWvvkawVL4ACiGEEPFo9gG22Q1GsHk9IKiUiqkoUqRrhCDaPr3a8uXL0XWdwYMHN8IsRUunlMLRvz9t77qLnK+/ov2//03iaSNRNlvMuKoff2T3nXeydshxbLv6GsoWLCDk8TTSrIUQQoimr9nnYGt+RelHm0k+rXm1HE+0JpJXlYc/5CeIoljTSAvFtk+fMWMGV111FS+++CJr1qyhR48ejTxr0VIpiwXX8GG4hg8jWFFJ+aefUPbue1R+9x1UV7jx+6n47DMqPvsMzekksXdvSisrcQwYgHmPWtxCCCHEoazZr2ADVHy1A/evRY09jf2iKY00e1p0u1DXMQB7oBS/z4vb7Wbu3LlcfvnljBs3jlmzZsWc/+KLL5KVlYXD4WDUqFHk5ubGHN+wYQNjxoyhTZs2JCQk0L9/fz788MOYMZ06deKee+7hoosuwuVy0aFDB1599VVKSkoYP348TqeTnJwcPv7444N2H0TTozsTSD7zTDrOmknOlwtpfest2Hr1ihkTqqjA/t137LxhCuuHDmP98BHsuPEmil95Bc/atVJ2UgghxCGtRQTYAMWv/UagxNvY09gvydZkdC2cf+1XijJNQ1PgLc1l3rx5ZGVl0adPHy644AJefPFF/H4/AD/88AMXXXQRkydPZsWKFYwePZo77rgj5toVFRWMHDmSTz75hJUrVzJ27FjOP/98fv3115hxjz/+OAMHDmTZsmWcc845XHjhhUyYMIHTTjuNFStWcPzxx3P++efjkZSAQ5IpI4PUSZPoPO91unz4AelXXYW5Y8c64/w7d1L27rvsvutuNp0xhrWDjmHbFX+l4LkZVC1dSsjbvP7dFEIIIf6MZp8iQuS5vlBVgKKXfyVjcm+UvvfvDY+cO6qBJlbXDa++F7OtazqptlTyq/KBcPv0xFAIu7+YGTNmcMEFFwBwwgkn4HA4eOeddxg7dixPPPEEI0aMYOrUqQB069aNJUuWxKxy9+3bl759+0a3p06dyltvvcW8efO47bbbovtPOeUUrrrqKgDuvvtuHn30UbKzs5k0aRIAt99+O7Nnz2bVqlUceeSRB+GuiObC2rkzGdddS/q11+D56SdWzXmRVkVFuFeuxHC7Y8aGysqo+PJLKr78EgBlNmPr1QvHgP7Y+w/A3u8ITCkpjfExhBBCiIOu2QfYfocRXocPgW9LGWWfbCHp1M6NPa24pdpSKXAXYBgGHqWoVIrdGzfz7bff8sorrwDhh9EmTpzIzJkzGTt2LGvWrGH06NEx1znmmGNiAuzKykruvvtu3nvvPXbt2oXf78fj8dCvX7+Y8/r06RN97XQ6cTgc9O7dO7qvdevWAOTlSTt3EaaUwt63L5VnjCZr6FAMvx/Pr79StXQp7qXLqFq2jGBhYcw5ht+Pe/ly3MuXA+HfU0t2Vxz9B4SD7gEDMGdmSiUcIYQQLUKzD7BDOiSelEXZR1sAKF+4HWvnJGzdUxt5ZvExaSZSrCkUecI55AW6zsyX3yIYDNKx1l/FG5EufNu2bYu+3pcpU6awYMECpk+fTk5ODg6Hg4kTJ+LbozW22WyO2VZKxeyrDnhCklMr9kKZzdh798beuzdcdBGGYeDfsoWqpcuoWrYU97Ll+DZtqnOeb/0GfOs3UPLaawCYWrXCPqB/NOi2duuGMjX7/0QJIYQ4BLWI/3u5TuiAd2Mp3nUlABS99hut/69/vWP3TNNoCtLsadEAuzQU4oXX3+PBW65l+KhxOBJrvihccMEFPP/88/Ts2ZPvv/8+5hp7bi9atIhJkyYxduxYADweD5s2bZJKJOKgU0ph6dQJS6dOJI/9CwCBwkLcy5dHg27PL6shEIg5L5CXR/mHCyj/cAEAmsOB/YgjwkH3gAHY+/RBczga/PMIIYQQ+6tFBNhKU6Se253cJ5YTKvcRqgxQ+PJvcJz5909uAiy6hSRrEqXeUr765CsKi0q4fMJZJKSmY23bI7qKPH78eJ555hnmzp3LkCFDePDBBxk3bhwLFy5k/vz5Mdfs1q0b8+fPZ8yYMZjNZu6++2688qCZaCSmtDRcJ56I68QTAQi53bh/+hn3sqVULV2Ge/lyQpWVMeeEqqqo/PZbKr/9NrxD17H17Imjf//ISnd/TOnpe76VEEII0ehaRIANoDstpI7vTsHMn8EA36ZSQkclN/a04pZuT6fUW8qb/3uTo4YchTMtBavhxV1Zit2ZDMDZZ5/NP/7xDyoqKpg1axZ33nkn99xzD0OHDuWuu+7i2muvjV7v0Ucf5dJLL+W4444jJSWF66+/noqKikb6dELE0ux2Eo4eSMLRAwEwgkG8a9eGg+1lS6n6cSmBPfP+g0E8P/+M5+efYc4cACxZWdgHRPK4+/fH0qmT5HELIYRodC0mwAawdU0mcURHyj7dCkDIEyDkCaDZmv7HtJlsOC1OnnrpKQAKQ0HaBYIY5bkQCbC7dOkSk3998cUXx1zjmmuuib7Oysri008/jTl+xRVX4HK5otubN2+uM489g3CbzRZXzrcQf4bSdWw9emDr0QPOnxjO496xMxxsL1uGe+kyvOvW1TnPt2ULvi1bKH3zTQD01FTs/ftF87htPXqgLJaG/jhCCCEOcU0/8txPruEd8W4qxbuhFIBAkQdza8c+S/c1Fen2dCp84QC3RNPJIIjDqMLrrsRqT2jk2QnRcJRSWNpnYmmfSdIZZwAQLCmhasWKaKUSz08/YURqw1cLFhVR8elnVHz6Wfg6Nhv2Pn2iD0/a+x2B7nQ2+OcRQghxaGlxAbbSFKnjDyP3iWXhHSGDQJEHU7q9yf/VscPkwG624/a7MQh3d2wTDBIo243V3rWxpydEo9KTk3ENHYpr6FAAQl4vnl9+qSkPuHw5odLSmHMMj4eqxYupWryYQgBNw9q9O47+/WvKA0ZKUQohhBAHSvMPsOvJXtBdFlLP7U5h/ubwEG+QULkPPdHasHPbT0op0m3pbPNvA6BY18gIBrEHyvD7vJgtTXv+QjQkzWoNB8r9+8PlYIRC+DZsqCkPuHQZ/h07Yk8KhfCuWYN3zRqK//c/AMyZmdGHJu39+2PNzkZpTf9vvIQQQjRdzT7A9pRCRbEHZ4otZr8tJwWtfHt0O1jmQ1n0Jp+P7bK4sOgWfEEfIRRFukZGMERVaS7mjLotqoUQYUrTsObkYM3JIWX8uQD4c3NxL1sWDbq9v/4Ge9R09+/YgX/HDsreeRcALTERR79+0Ycnbb16oVnly60QQoj4Ne1oMw5GCN57+if+MqU/lj2CZ81mQll1DG8QgECxB3Orpp2PrZQi3Z7OzoqdABRpOmnBcPv0YLAdut7s/5EJ0WDMrVtjHjmSxJEjAQhWVOBesbKmPODKlRgeT8w59bZ57907WqnE0a8fenJyQ38UIYQQzUiLiNYKt1fwyaxfGHllHzStVp61AlOKDX9eFYQMCDaPfOwkaxJ5VXkEQgECSlGiaaSGQlSU5OFMa9fY0xOi2dKdTpxDBuMcMhgIt3D3rFlTUx5w6TKCRUUx5xh+P+5ly3AvWxbdZ83Jxl7d5r3/AMyZ7Zr0f1OEEEI0rBYRYANs/rmQb+etZ8g5OTH7lUnDlGojUOAGqvOx/eiJTbd0l6Y00uxp5FbmAuGHHVNCIazeQkKhNmiSHyrEAaHM5nCVkT594OJwm3ff5s3RtBL30qX4tmypc5533Xq869ZT8uqrAJhat44pD2jt3h2l6w39cYQQQjQRzT7ANtVKvV75+TaSW9vpdUL7mDGazYTmshAq9wEQLPOirBqatel+/BRrCvlV+YSMED6lKNc0EkMBKssKSUjOaOzpCdEiKaWwdu6MtXNnkseOBSBQUEDV8uU15QFX19PmPTc3ts17QkJNm/f+A1B7pKEIIYRo2ZpuhBknswO69Mtg4/J8AL56dR2J6XY6Hp4WM05PtGB4gxi+SD52UdPOx9Y1nVRbKgXuAgAKdA1XKIRelY+RlC5/HS1EAzGlp5N40kkknnQSEG7h7v7p52ilEvfy5YSqqmLOCVVWUvnNN1R+8w0ArYB1badhzc6u9dMVS9dsdKfUuBdCiJam2QfYACde3JO3ipaRt6UcI2Tw0YxV/OXGATFjlFKYUvfIxy72YkqzNdlgNdWeSqGnEMMwcCuNKqVIMLy4K0qxu5Jjxs6bN4+zzz5bui4KcZBpDgcJg44mYdDRABiBQLTNe3XQXafNOxDYtYvArl1Ufv11zH5Tu7ZYu0rgLYQQLUmLCLDNFp3TrurDvId+pKLYi88T5P2nf2LAxOSYccqkYUqxESiM5GN7AoQq/OiuppmPvWrlKo488kj6DOjDSx+8RIGukxAIYFTkwh4BthCicSiTCVvPnth69iT1gvMjbd531ORxL1uGZ8MG1B7lAasFdu4isHMvgXd2dk3wnZONpUtXCbyFEKIZaNAAWyl1KvAEoAMzDcN4aI/jQ4G3gU2RXW8ahnFPPNdOSLJy+tV9ePOfy/B7g5QXefCU+zBCBqpWZRHNbkJzWghVRPKxS73h+tjWpvdA0owZM7jiyiv474v/ZcPaDXTt1hW3UtI+XYgmLNzmvT2W9u2jbd4Xfvopx3TqhHd9+OFI74YNeNevw7d5S5187mrRwPurvQTe2TlYu3bFmpONtWtXtAT574EQQjQVDZaArJTSgaeBkUBP4DylVM96hn5tGMYRkZ+4gutq6e1dnHzZ4VRnfAQDBmWFnjppE3qSBWWpCagDRR6MYP2rS43F7XYzd+5c/jr5r5x+5um8+b83gXBFEYAXZz5DVlYWDoeDUaNGkZubG3P+hg0bGDNmDG3atCEhIYH+/fvz4Ycfxozp1KkT99xzDxdddBEul4sOHTrw6quvUlJSwvjx43E6neTk5PDxxx9HzwkGg1x66aV07twZu91OTk4O06ZNIxRZnfN4PPTq1YtLLrkkes7OnTtJT09n+vTpB+VeCdHkmUxYs7NJPPVUMq69hvaPP0bX997jsGVL6fLeu2Q+/hjpV1+N65RTsGR3BdPe1z6qg+6i2bPZNXUqm885l98GHMn64SPYOnkyudP+Scmb83H/9BOhysoG/JBCCCGqNeQK9kBgvWEYGwGUUq8AY4DVB/JNOvVOZ/DZOSx6bR0A3io/laUazuSaTmx187FDTS4fe968eWRlZdGnTx8uvvBiJoyfwPW3XU+p2czmZT9zxf/dxF133cn48efxxRdfcOutt8acX1FRwciRI7nvvvuw2+28+uqrnH/++fTp04fDDjssOu7xxx/nvvvuY+rUqTz77LNceOGFDB8+nPHjx3Pffffx4IMPcv7557N161ZsNhuhUIjMzExee+01MjIyWLx4MZMnTyYtLY1LL70Um83G3LlzGThwICNHjmTcuHFMmjSJvn37csMNNzT0bRSiSVMWSzT3mlNr9hs+H74tW8Ir3evWh1e+N6zf54q3f+dO/Dt31lnxNrdrhyUnNtXE2qWLrHgLIcRB1JABdiawrdb2duDoesYdo5RaCewEphiG8cv+vlGfYe0pza0Cwk/2V5V60U0Ku9PC9n98ve+TD6L2Dx0X99iZM2dywQUXAHDy8JOx2+0sXLCQk0afxCOzX2HEkIH8/cqLcWZ0pFu3bixZsoRZs2ZFz+/bty99+/aNbk+dOpW33nqLefPmcdttt0X3n3LKKVx11VUA3H333Tz66KNkZ2czadIkAG6//XZmz57NqlXhfHCz2cw999T8xUKnTp1YtmwZL7/8MpdeeikAffr04aGHHmLy5Ml89913LF++nJ9++qnJfHkRoqlTFku07Xu9gff69XjXb4j8uT5cq/v3Au8vv4rZHw28q/O8JfAWQogDpiED7Pqiqz1LXiwDsgzDqFBKnQa8BeTseZJSajIwGSAjI4OFCxfWvXBrA6UnR7fLCz14fY1bi7a8vDyucRs2bOCbb75hxowZ0XPGnTOON156g5NGn8Sv6zYx9sTjsPuKKS1NQtM0+vXrF/MelZWVPPTQQyxYsIDc3Fz8fn80faN6jGEYdO/ePWZeDoeDnJyc6D6HwwHA5s2b6d69OwCzZs3ixRdfZOvWrXg8Hvx+Px06dIi5ziWXXML8+fN57LHHmDNnDomJiXF//gPN4/HU+zuyNxUVFfs1/lAl9yl+B/xe2e3Qu1f4ByAQQM/Lw7RzF6ZdOzHt2o1p5070vLy9Ply5t8A7mJZGoG3b8E+78J/BNm0wbLZ6r3Mgye9U/ORexUfuU3zkPh14DRlgbwc61NpuT3iVOsowjLJarz9QSv1bKZVuGEbBHuOeA54DyO7c0Rg6dGi9b7h69Wp0s0bQH/4fjL+ycVdQXS5XXONeeeUVgsEgPXvWpKhX55Hv2rELwzCoUgpdhdADlTjT2mGL/M+v+j1uuukmFixYwPTp08nJycHhcDBx4kQMw4iOUUrhdDpj5qWUwuVyRfeZzWYAbDYbLpeLV199lX/84x9Mnz6dY489lsTERJ5++mnmz58fc538/HzWrl2Lruvs2LEj7s9+MNhstugXkHgsXLiQvf1OiRpyn+LXWPcqZsU7mmqyAd/mzRAM1nuOXliIXliIddWqmP3mzEys2dlYsruGH7DMzsbatQta5Ev4gSC/U/GTexUfuU/xkft04DVkgL0EyFFKdQZ2AOOBCbUHKKXaALmGYRhKqYGEH8Is3NdFNVMe65YuIGfAqXWOKaVIbuWgeHcloaCBETKwX9uPlDYOtFoNZoxACH9uFUSCWM1uQk9tnHzsQCDAnDlzePDBBxk1alTMsQnnT+Ctl9+ia/eu/LD8Z4IQbZ/+/fffx4xdtGgRkyZNYmykG53H42HTpk306NHjT81v0aJFHH300VxzzTXRfRs2bKgz7rLLLqNr16489dRTnHfeeZx88skMGDCgzjghxMETk2oysma/4fPh3bwZX0yO974Db/+OHfh37IAvv4zZ3xCBtxBCNDcNFmAbhhFQSl0DfES4TN9swzB+UUr9NXL8WWAccKVSKgC4gfHG73ROUZrB2s03kJjWkdad6hYl0U0aSRl2inPdYBgEAyFK890kt3ZEA2hl0tBTrASLwikkIXcAVelHdzZ8fez333+fgoICLr/8ctLSYrtRThg/gaf+/RQPPfsQk0ZN4s6nnufC00bw0eufMH/+/Jix3bp1Y/78+YwZMwaz2czdd9+N1+v90/Pr1q0bL7zwAh9++CHZ2dm88sorfPnll6SkpETHPPvssyxcuJAVK1bQuXNnLrroIiZMmMDy5cujKSdCiMajLBZs3bph69at/sC7vhzvPxh4W3OysVTneUvgLYQ4RDRoHWzDMD4APthj37O1Xj8FPLW/17WlePjhmwsZkbyAhOS0OsfNVhOJaTbKCsINZvzeIOWFHly1qoboDjOGN0io0g/Uqo9dq5xfQ5g1axbDhg2rE1wDnHPOOdxyyy1UVVZxz+P38O9pT/PIY89x/DFHceedd3LddddFxz766KNceumlHHfccaSkpHD99ddTUVHxp+d3xRVXsGLFCiZMmIBhGIwdO5YbbriB2bNnA/Dbb79xww038Mwzz9C5c2cgXKmkf//+/O1vf+M///nPn56DEOLgiAm8awn5fPiigXdN8B1P4F1RO/BWKhx4R+p3W6rreXfpLIG3EKJFUc29tXb37lbj38+0B6BqezYjz3sPUyRveM2aNTEpEZWlXipLalZxE5KtJCTVlO8zQgaB/CqMSM42Jg1zK0dMo5rGFjJCrC1eSzAU/p9aZiBAciiE29W5Tvv0PZWXlzdqLnRj2fP34PdILlp85D7Fr6Xeq5DPh2/TZnwb1sc00dlX4F2vSOBdkZxMmz59MLdvjzkzM/LTDj05WaoQ7aGl/k4daHKf4iP3KX5KqaWGYRz5e+Oafav0UNAZfe1ov57PXpvMyRNm1/sfY0eihaA/hCeySl1Z4kU3adgSwgG50hR6qp1AXiQfOxAiWOxptHzs+mhKI82WRl5VHgAFuk5SKCTt04UQDU6zWLB174atez0r3nsG3uvX49u6tf7A2zDwb9+Odft2ivd4uBJAS0ioFXBHftpnYom81hITm8x/o4UQAlpAgK3p6ZgCPQiYfgZAb/0V3753P4NH31ZnrFIKV5qNYCCE3xv+j3xZoQfdpGGOtErXzE0nH3tvUmwpFLgLCBkhvEpRoTRc0j5dCNFE/F7g7V2/LuYBy70G3tXnVVbiXbsW79q19b+f01lr1btdOPCutQquH4J/cyeEaFzNPsAGGDx8Lgs/HoGy5aE0qDTN4ZfvuqEl964zVikVfuhxdxXBQAgMg9L8KlLaJKCbwpVFmko+9t6YNBMpthQK3eECKwUmDZc/RKBsN1Z710aenRBC1G/fgfcmln/wAd2SkqP52/4dO/Dt2IFRVbXP64YqKvD++iveX3+t/30TE2tWvNtVr4DXCsCdsjAhhDiwWkSAbTI5OPaEeSz66hR0qxvdGmJr3p20T3i53vGarpHUKhxkGyGDUNCgNK+K5DYJaJF8az3JiuELhvOxDQgWeVBNKB87zZZGkacoUhNbo1Ip7IEy/D4vZov19y8ghBBNRDjw7o531y7S9sgDNQyDYEkJ/u3VQff2aODt37ED//YdGJ59NxELlZXhXV2Gd/Waeo/rSUl75H1HVsLbt8fcrp10txRC7LdmH2AHIu2BHQmZ9O//PMtXTkQzBbG4fPh9RQT9PnRz3fQOk1knKcNOSW54ZSTgD1GW7yaplR2lVCQf20YgL1zezwiECJZ40VOsTSLXz6ybSbImUeIpAaBQ1+loBKgq3Y05I6txJyeEEAeIUgpTSgqmlBTs1Z0razEMg2BRUc2K9/btkdc7o/uM3ylRGiwtJVhaiueXX+o9rqek1Fr1boc5M7MmDaVdOzS7/YB8ViFEy9HsA2y32x2tEpHe+ii6Zt3Lxu23ohQoPUR52UYSU7qhaVqdcy02E640G+WF4dUPnydARbEXV2q4K6Jm1tGTrQSLI/nYVX6UVUePPBTZ2NJt6dEAu1zT8CiF3VdCMJCJbmr2/2iFEOJ3KaUwpaVhSkvD3qdPneOGYRAsKKhZ9d6+IyYFxb9jB4bfv8/3CBYXEywuxlPPA5gAelpaOOhun1n3Ycx27dAaoM28EKJpaRFR2Jtvvskll1xC27Zt6dL9XMpL1lFQ+TwAmtlPefEmElO71LvybHeGK4tUlfkAcJf70E0ajsTwqreeEMnHrorkY5d4URYNzdz4+dhWkxWXxUW5rxyAQl0j0whSUZqHM61dI89OCCEan1IKU0YGpowM7EccUee4EQoRyC+oFXBvj8n/9u/cBb8XgBcWEiwsxPPTT/Ue1zPSsdST+129Gq5Zms5D9EKIA6NFBNh+v5+5c+dy+eWXk5iYSJ+BU/nms3XR48pcRUXJDlwp7es9PyHZSjBg4I0E0RXFHnSTwuoIr1TryZF87MhDkcHCppOPnW5PjwbYpZpOBsFo+/T6Vu2FEELUUJqGuXUrzK1bQf9+dY4bwSCB/Py9p6Ds2gWRVMW9CeYX4M4vwL1yZb3HTa1axZQfjKagZGZibtsWJQG4EM1Osw+wq1ely8vLefnll7n44ouxWCwcO3wmK1d8Ex1naMVUlVtxuDLqvUZimo3iQIiAL1K+r8BDchsNs0UP52On1c3HNqU2/l/7OcwOHGYHVf4qDMK52G2DASrLCklIrvtZhRBCxE/pOuY2bTC3aQMDBtQ5bgQCBPLy9p6Csnv37zbdCeTlEcjLw718eT0TUJhat45Z8bZkZmLJzcPTti2mVq2kEY8QTVCzD7Dt9vBDiYZhsGvXLubPn8/ZZ5+Nppmx2jIwQgqlBUGBP7gbr9uK1Z5Y5zpKU+HKIruqCAVDGIZBaZ6blDYOdJNWbz52sInkY6fb09nq3wpAia6REQyiV+VjJKXX+Y/uNddcw6pVq1i4cGEjzFQIIVoWZTJhbtcOc7t2OI46qs5xIxDAvzu3JuDeXisFZecOArtzIRTa+xsYBoHduwns3o176dLo7hRg07/+Fd4wmzFlpGPKyMDcqlU4Jab6z1qv9ZQUlPzNphANotkH2Lquc9ppp/H+++8D4bbYn3/+OSeeeCJKaSQ4O1NZuR6lDJQGHs82dFM2JnPdUna6rpFcXb7PMAgFQ5Tmu0lu7UDTFJrDhOE1H/R87Isuuog5c+Zw7733ctttNQ1zFi5cyLBhw8jPzyc9PT2632l2YjVZ8Qa8hFAU6Tqtgl7cFaW/2z5dCCHEwaNMJiztww9A1sfw+fDn5u41BSWQmxvuLLwvfj+BnbsI7NzFPgsWmkyY0tNrgu9WGbFBeSQY11NTJRAX4k9q9gE2wFFHHUVBQQE//PADAIsWLSI9PR2r1YrJZMNu64DHG17hVXqIivKNJCZ1Q9PrBsYmi05ihp3SvEj5Pl+Q8kI3ienhlfI6+dhFHlTGgc/HttlsTJs2jSuuuIKMjH2neiilSLels6NiBwBFukZ6MCjt04UQoolTFguWDh2wdOhQ7/GQz0dg167YFJSdOylY+xtOfzg9JVRREd+bBQLR1fB90vVwIF57JbxVRt2V8bQ0VD3/HxVCtJAAG+CUU06hqKiIdevCDze+8847nHXWWQBYrEkEA63xB3MB0EwByks2kpiaXW/emtVuwplqoyLSLt1bFaCyxIszxVZTHzs/ko/tDxEs9WJKObD52MOGDWP79u3ce++9PPnkk3WOB4NBJk+ezOeff87u3btp3749Z048k0lXTQJNo1jTSPaX87frr+OFOf8F4LzzzsPYYyVkwYIF3H///axatQqlFEcddRSPP/44PXr0AGDz5s107tyZl19+mWeeeYbFixdz2GGHMWfOHDRNY/LkyaxcuZJ+/frx3//+l86dOx/Q+yCEEIcyzWLBkpWFJSuL2u1u1i1cSN9IU55QVRWB/PzwT15e9LW/+nVeeDtUVhbfmwaDBHJzw6vn+5ycFi6RuGdayp5/pqWipHSsOMS0mN94TdMYN24cs2bNIi8vj1AoRGVlJYFAAJPJhD2hFcEyDyFKAVBmD+UlW0lMqb8pi8MVLt/nLg+X76sq86GbNexOC5pFR0+yECwJNy8IVUbysR0HLh9b0zQeeughzjzzTP7v//6Prl1jW6CHQiEyMzN57bXXyMjIYPHixVw++XJcyS7Gnj+WQl1n5lOzmT37eWbMnEWfPn147LHHePnll+nfv3/0OpWVlVx//fX06dMHt9vNfffdx+jRo1m9ejWWWk+u33nnnTz22GN06dKFK6+8kgkTJpCRkcH9999Pq1atuPDCC7nuuut49913D9g9EEII8fs0hyMahO9LyOOJDcTz8gnk1/ozPx9/Xj6h0tL43jgUil5v3xPU0NNSw+koGa0iq+GRP/dcETc3/nNNQhwILSbABrBarUyYMIEZM2ZQWVmJYRgUFRWRnp6OpmkkuDpw992zG21+d911136NP+200xg8eDBTp07llVdeiTlmNpu55557otudOnXix6U/8uH8Dxl7/lj8SvH4zJe58cpJnHXmGZgt4ZSTL774IuY6Y8eOjdl+/vnnSUxMZPHixQwZMiS6/+9//zunnXYaADfccAOjR4/mjTfeYNiwYUD44clrrrlmvz6fEEKIhqPZbPtMR6kW8noJ5BfUrIbv+WfkdbCkJL43DoUI5hcQzC/AS/3t6gFQCj01NRJ01+SK13lwMz1dAnHR5LWoABsgOTmZ8ePH88ILLwDhVupFRUWkpaU1yzJG06ZNY9CgQUyZMqXOsWeffZaZM2eyZcsW3G43fr+f9h3Dtb7Ly8rZnZvPMQP64C3NxZyRhaZpHH300Wzbti16jQ0bNnD77bfzww8/kJ+fTygUIhQKsXXr1pj36lOrQ1rr1q0B6N27d8y+yspKqqqqcDgcB/QeCCGEaDia1brPBzOrhXw+gnumo9RKSakOyINFRfG9sWFEm/Z49xGHA+FAfC854ubq/bWKAQjR0FpcgA3QoUMHzjzzTCoiD374fD5KS0tJSkpq5Jntv6OOOoqxY8dy8803c/vtt0f3v/rqq1x//fVMnz6dY489lsTERJ5++mnmz58f80Widvv0+owePZrMzEz+85//kJmZiclkomfPnvh8vphx5lqrBdXXr29faF/lpoQQQrQYmsWCFmmIY9/HOMPnI1BYGBN0+2NWxcOr5cGiot+vmBIRLCoiWFSE97ff9jkuIyGBDa1bo6emYEpNq/VnKqa0VPTUNEypKehpaehJSfLQpjhgWmSADeHV1eW1ivZXVVVhMpmiaRp+XwVV7k1Ux6JGUMfpykE31f/XTn5vkJLcquhDgmarTnKkm2OwwhfNxwbQU20HNB/7gQceoGfPnixYsCC6b9GiRRx99NExaRkbNmwAIMWagpFokNE6g4XLV3H64KOoKM3DMDtZvHgxbdu2BaCwsJA1a9bw9NNPR1M9li1bRuB3upIJIYQQ8VIWC+a2bTFH/t+zN4bfT6CoKDYlJSZPPB9/fh7BwqJ91w6vRausxLdxI2yMZ6IKPSUlNghPTQ3nj6fuEYynpIQDcilnKPaixQbYEC51Z7PZ8HjC1UDKysowmUzYbDbMFifWQDt8gZ0AKD1IRdlGEpNz6v0XxmzVSUy3UZrvBsIBd3mRB1eaDS3BjOENEnKHA9NgsRfNoqNMB+ZfvOzsbCZPnswTTzwR3detWzdeeOEFPvzwQ7Kzs3nllVf48ssvSUlJIc2eRpGniPMnn89/npjJEZ070Ld7d55++UN27doVDbBTUlJIT09nxowZdOjQgR07dnDjjTdikqe9hRBCNDBlNmNu3RpzJA1xb4xAgEBhUb154TF/FhTEHYiHL2xEV8Z9bPj98bqOnpISCb6rg/FIEF69Wh4Jxk1paWguV7NMVRV/TIuPpJKTkyksLMTvDzeHKS4uJj09HbPZjM2RRrDcQ9AI54cpk4/y0s24kjvXX77PYSYhOURlZLXaU+lHN2skJFnRU2yE/FUQqY8dKPRgamU/YP8y3XHHHcyZMye6fcUVV7BixQomTJiAYRiMHTuWG264gdmzZ2PRLSRZk7jwqgspyCvgqin3ojA4Z9w4Jk6cyJo14eQ2TdN49dVXue666+jVqxfZ2dk88sgjdR58FEIIIZoKZTJhbt0Kc+tWwOF7HWcEg3z1wQcMOuwwAoVFBIuLwn8WFREoKiRYVBz+M7IvGG/1lGrBIMGCAoIFBfGNN5sxpYRXwKN/7iUY11NT0RISJCBvxlp8gK1pGqmpqRQUFBAMBmMqi+i6ToIrk/JSL4aqDJ+gV1JZthNnUv05y45EC8FACE9FOGCvLPGimzRsCWZMqTYC+VVggOEPhutjJ+9/fezqBzRra9WqFeXl5TH7Zs2axaxZs2L23XHHHUC4fXqpt5Sb77uZm++7mWy/H2Xo+B2tsSfWPPgxfPhwVq1aFXONilpNCzp16lSndvaRRx5ZZ9+pp55aZ58QQgjRWJSuY7hcWHNysOb8/njD7ydYUkIgsoodDsYLw9uFRQSKi6LBeKCoiNAe/0/+XX5/JO0lD+/vj0ZZLHsPxmsH5ZHUFU0KDDQpLT7AhnA79eog2zAMgsFgtLKIpmk4EztRVroOpYUf7AupItyVVuwJdZ9AVkrhSrURDBj4PeGUkLJCD7pJYbaa0BOtBEsj9bEr/ISsOpq94csJ2Uw2nBYnFb5wsFyoa7QLBLBU7cBftZsqSwrWpFaY62kZL4QQQhxqlNkcrckdj5DPR7C4mGBhIYGi4nqD8eqV8mBhIaGqqv2ajxHp4hnYtSu++dvt0WD8dx/qTE1Fsx3YBnki1iERYEO44kVKSgpFkXJBfr+fkpISUlJSUErD5epKeflalBYEwBfYhe6xYbE561xLKUVSuo3i3VUEIykhpXluUtokoDnNhLxBjEjwHSj2YjYfuHzs/ZFuT48G2CWaTgZBzICZIGZfAaG8Air1RHRXK2wJrgafnxBCCNFcaRYLWhw549VCHk9k9TsSjNeXrlJUHbAXYUSeH4uX4Xbjd7vx79wZ3/wdjmgwnmwY7PzoY/Tk5Pp/UpIxJSejajWgE/t2yATYEH7oMTExkbJIu1iPx0N5eTmJiYlouomEhM5UVm1AKQOlwO3Zgm7KQTfV/YXSdI2kVnaKd1dhhAxCIYOS/CpS2iRgSrHizwtBMAQhg0CRB1PGgcvHjpfD5MButuP2uzGAnaYE2gU8mAl/idAUJITKoLQMT6mNoCMde2J4VV8IIYQQB45ms6G1a4e5Xbu4xoeqqmqlq+yRMx6TT15EsLAQI/KsWbxCVVWEqqrwb9uGFSj96eff/wwOxz6C8JQ6QbmenHzI5pLHFWArpXoCQcMwfotsnwRcCPwCTDMMI3jwpnhgJSQkEAgEqIr8VU1FRQUmkwmHw4HJbMdm7YDXuxUUKC1ERfkGXEnd0LS6tTFNZp2kDDsleeG866A/RFm+m6RW9pp8bMDwBQmW+jAlN2w6hlKKdFs62/zhxjIVmp+tjgSchgWXuwIHNd+ObXigajv+ql1UWVIj6SPyTVUIIYRoDJrDgcXhgPbtf3esYRiEKiujq981gXf9wXiguBj+QEneaFAe5yo5AGYzelISenISenJyOI1lb0F69U9SEqqZVzSLd/azgCeA35RS7YG3gYXA1UAicMtBmd1BoJQiKSmJYDCI1xvOlS4pKcFkMmGxWLDakggGWhEI5YXH6wHKSzeSmJxd7zcwi81EYqqdssJw+T6fJ0BFkRdnqhU9qXY+ti+Sj92wvzAuiwubyYYnEA6mPQEPHjwU23RcejpOjxdXoAJNRep7E8Tsyw+nj5hc4fQRh6SPCCGEEE2VUgrd6UR3OrFkZf3ueMMwCJWVRYPxlV99zWHtMwmWlIR/iktqXlf/lJZC8A+sp/r9+1dtJUJLTKwVdCfVCcJN9ayaa/Z9tTxqWPFGez2AZZHXZwM/GIZxmlJqGPA8zSjAhvAvYkpKCgUFBdGmKtWVRUwmEw5na8pLPRgqnEqidA8VZdtwJXWs93o2p5lAIERVJJh2V/jQzRp215752B7MZkeD5mMrpeiY2JHcylzKvGUYhAPpYChISaiMEh2ctmQcfkj2ldVKHzFICJZBSRnuEhshSR8RQgghWgSlVHhVOSkJOnfGW15OytCh+zzHCIUIVVTEBt3FxeHKK3sG4yWl0deG2/2H5hgqKyNUVoZ/69b4P5fVWm/+eL3BeXVQnph4UBoGxRtg60B17+wRwAeR1xuA+LL7m5jq8n35+fnhb3KhUDTIDlcW6Uh5yXrQwyu/hiqlqiIXh7P+j5uQZCHoD+GtCudAVRSHK4tYUqz484IQNBotH9usmWnvak+JUYLf5KfYU4w/VJOrVeGvpAIosdtxGhYSvVUkGDXpI/Y90kdsSa0wSfqIEEIIcchQmoaemIiemAgd619wrE/I46m7Gr6XlfJASTHBklJCZWXwB0r/Gl4vgdxcArm58Z9U/bniSV1JSY77svEG2KuAK5VS7xEOsKtXrDOB/Vvzb0JMJhOpqakUFhYCEAgEKC4uJjU1FaUUzqQulJeuRemRFehgHl63Das9qc61lFIkptkoCYbwe8OrwGUFHpJbOyL52OFvcIYvSLDMhymp4cvj6Uon2ZFMuj2dcl85xd7iaJURAF/QRxE+SiwaTj0Vp9dHUqACLfJdQNJHhBBCCLE/NJsNrU0bzG3axH2OEQwSLCurP1Wl1sr5nj/7+6AnAKFQ9PwDKd4A+2bgLWAKMMcwjOpHTc8AFh/QGTUwq9VKcnIyJZEb6/V6KSsrIykpCU3Tcbq6UFGxHqWFQIHHuw3dZMFkrpvnozRFUoY9Wr7PMAxK892ktHHE5mOXR/KxbY2TwK+UItGaSKI1EW/AS5G3iBJPCSEj3FI2ZIQoC1RQpkOxNZGEgCLFV4GlvvSR0kj6iEvSR4QQQgjx5yldx5SSgiklJe5zDMPAqKqqm7KytyA98hOq1VzvQIorwjMM4yulVAaQaBhGca1D/wH2r3J6E+RwOAgEAtEOhpWVlZhMJhISEtBNVuz2jrg9m1EKlGZQWbEJV2I3NL3u7dN0LRxk50bK9wVDlOa7SW7tQNXOxy7yYG514PKx582bx9lnn73f3RStJittTW1pZW9Fqa+UIk8R3kBNjyl3wIMbKLFZcGIlyevGadQctxseqNyOv1LSR4QQQgjROJRSqIQEtIQEzJn1d+Ouj+H3Eywt3Uc+eWxOOb/9Gtd1415CjZTiK458CDswGFhnGEZe3J+iCXO5XAQCATyRwu6lpaXouo7NZsNidREMtMUfDHdTUnqQ8rINJCbnoFTdANlk0UlKj5TvAwK+IOUFblxpNgL7kY+9fPlyjjzySAYNGsQ333xzkD55mK7ppNpSSbGmUBWooshTRLm3PPpQZCAUpIQqjjhiKBdePolbrr2ClEAlmoJ7H5vBHdOfYd5z/+Ss006MpI+05tLLr2Dbtm189dVXB3XuQgghhBB/hDKbMaWnY0qv2727/hPie4YuruVTpdQLSqmrIq8thNNCPiZctm9kfDNq2pRSJCcnYzbXtDUvLi7GH8nnsSeko5FaM173UVG6ea/Xs9hNuFJr2pB63QGqynyYau2rzsfemxkzZnDVVVexatUq1qxZ80c+1n5TSpFgTqCDqwM5KTlkODIwaTXfwwYOGcg3X3/Lbt3PBruTXZZEPv9uKR0z2/DFt0ui6SO2knV88fmnDD7maIxQqEHmLoQQQgjRFMSbn3AK8H3k9RmAC2gD3BX5aRGqK4tU5xIbhkFRURHBSN1HZ2ImBB3R8YZWSWX53out210WHK6adImqMh9eXwg9sWZfqNxHyFO32Lvb7Wbu3LlcfvnljBs3jlmzZsUcf/HFF8nKysLhcDBq1Chy93hidsOGDYwZM4Y2bdqQkJBA//79+fDDD2PGdOrUiXvuuYeLLroIl8tFhw4dePXVVykpKWH8+PGkJKUwuO9gNi3ZRHtXexxmBwOHDGTFkhX4vD58IR+7vWV8v+wnrvjb1Xz27dLotddu2MKu3XmcOrA7gd2/UFGwnYB/718mhBBCCCFaingD7BSgOhXkVOCNSGrIK0DPgzGxxqLrerSKCEAwGKSoqCia2+xK7owRqFnlDhqFeKoK93q9hBQr1lrNZcqLPATNOspa0xkyUOTBCMau8s6bN4+srCz69OnDBRdcwIsvvhhdTf/hhx+46KKLmDx5MitWrGD06NHccccdMedXVFQwcuRIPvnkE1auXMnYsWM5//zz+fXX2Nyhxx9/nIEDB7Js2TLOOeccLrzwQiZMmMBpp53GihUrOP7445l0wSSshpXOSZ0ZN3IcHreHVctWAbDyx5UkpSRx8tmnsmHzVhaXecnTE/js2yXYbTYG9e+NmQBOXz5a3moqczfiqTo4DxQIIYQQQjQF8eZg7wZ6KaV2EV7NnhzZ7wT+QE2UxvPZ510b7b17d10OQGmBm5QMO4bfA6Fa+djpNfnYM2fO5IILLgDghBNOwOFw8M477zB27FieeOIJRowYwdSpUwHo1q0bS5YsiVnl7tu3L3379o1uT506lbfeeot58+Zx2223RfefcsopXHXVVQDcfffdPProo2RnZzNp0iQAbr/9dmbPns2qVas48sgjOfyww2nfvj3rl67ntBNPY+m3Szlq8FHYHXZ69u3JV18vIuGskbz3wwoGDOyPYbVDTPWRUigpxV1qJ+RIx+FKPSgF3oUQQgghGku8kc1s4FXC9bCDwGeR/UcD8T1OKdD0cPBshAxKCz3oyTW1sA1vkFB5OIVi/fr1fPPNN0yYMAEI50VPnDiRmTNnArBmzRqOOeaYmGvvuV1ZWclNN91Ez549SUlJwel0snz5crbu0RGpT58+0ddOpxOHw0Hv3r2j+1q3DjfWycureZZ12LBhfLnwS9Lsafz0/U+cOuJUXBYXRx17FIu/CVdt/GHRD/QfciQbrSY2W12UKCu165vYDTcJldsi6SM7CPyR2pVCCCGEEE1QvGX67lFK/QJ0BF43DKM6mTYAPHywJtfSJGU4KMmtwjAMgoEQZWU+XC5LNLAOlvlQFp2ZM2cSDAbpWKtTUnWKyrZt2+IqxTdlyhQWLFjA9OnTycnJweFwMHHiRHy+2Dzo2g91QjiYr72vekU9VOtBxeHDh/PXv/6V4uJifvjhB2bPnk3HxI6cefKZXH/99Wxet5nC/EIGDhmIYRhU4qXSDPlaAq6AQUbAjR4Jt80EMPvyCOXlU2lKxJTYCqvduT+3VQghhBCiSdmfMn1v1LNvzoGdzsE3YviG/RpvGAaFhYUxgWl6ejoWS/hBxcry3QSN/JoTQnZcSV33WnovMd1GaaSro98bpFLXcFh1jEj3R09+BXPmzOHBBx9k1KhRMedecMEFPP/88/Ts2ZPvv/8+5tie24sWLWLSpEmMHTs2fF2Ph02bNtGjR4/9+vz1GTZsGF6vl0ceeYSMjAy6dg2n3Qw9YShbNm/h2/e+xel0ctTAo/AbNSvTvpCfQg2KrVachok0nxvHnukjxaW4S+wYjnTskj4ihBBCiGYo7gBbKdWHcCfHnoABrAam1+rq2CIppUhJSaGgoCBaTaSoqIj09PRwMxpXG8qKPaCXh0/Q3FSWb8eZ2KHe61kdZpwpISqKw81avFV+TC4LZi0EIYMPPl5AQUEBl112Gel71GQcP348zzzzDHPnzmXIkCE8+OCDjBs3joULFzJ//vyYsd26dWP+/PmMGTMGs9nM3Xffjdfr5UDIysqic+fOPPnkk4wZMya6v7payb/+9S+OO+44uqV1wx1wU+QpotRbGl15DxkhyvBRZtGxKxtJfj+pIR/VX0nshhsqt+Gv3IXXkootuRUmk7memQghhBBCND3x1sE+A1gGdAA+BBYQThdZppQaffCm1zTsWVkkFApRVFQUTZtwJWdhBGryqUOU4K7ce/8du8uC3VlTqq+y3EfIEf6u88IrL3LCMceRYnHVOe/ss89my5YtVFRUMGvWLJ555hn69OnDm2++yV133RUz9tFHH6VVq1Ycd9xxjBw5kkGDBtXJ0/4zhg0bRnl5OUOHDo3ZP3ToUMrLyxk+fHj4s5rsZDoz6ZbSjdYJrbHosV0e3Yaf3SZYa7Wz02THT83Kf7j6SB5a7i9U5m7E65bqI0IIIYRo+lQ8+bxKqZ+A+YZh3LnH/nuAMYZh9K3/zIOve/fuxm+//VbvsTVr1hyQlIhqXq+XwsKaknxWqzUaeIeCQcpL16JMkZrWBthsWVisifVeyzAMSvPc+KprYCtITjCDu6YmtinDjmaN+y8Zfld5eTkuV93AvSEZhkGFv4IiTxEVvroBs1IKh2Ei3e/BaQTrHHcrO4YjA3ti6l7TcPa0v78HCxcurPPFQdQl9yl+cq/iI/cpfnKv4iP3KT5yn+KnlFpqGMaRvzcu3gTXbsB/69n/X6D7/kysObNarSQlJUW3vV4vZWVlAGi6ToKrC0YwEvQpcHu2Egh46r2WUorEdBu6OfKPwICyqgCY910fu7lTSuGyuMhKzCInJYc0exq6VvOZww9F+tli1llndZCvWah9B+yGG0flVvy7VlFRuINAQKqPCCGEEKJpiTfAzgMG1LN/AJBbz/4WKyEhgYSEhOh2ZWUllZWVAJjMVmy2LKr/UkApg8qKjYRCdTs1Ami6RnKGA02rTj0xqAiEILJN0CBQ7I2rakhzZNEttEloQ7eUbrRztsNmssUc9xkB8kzwm9XGdpMdb630EQsBnN48VO4vVORuwuuubOjpCyGEEELUK978gxnAf5RS2cC3hB9yHEL4ocd/HqS5NVmJiYkEAoHoQ4OlpaWYTCasVitWu4tgoA0BYzcASgtSUbYBV1K3elMadLNGUis7xbluMAwCgRAei44tUsbO8AQIVfjRXZY657YUmtJIsaWQbE2OPhRZ5iuLeSiyVINSixk7JtL8XhKNIArQlYEzWALFJZHqI/uXPiKEEEIIcaDFG2DfB1QANwD3RvbtBO4EnjwI82rSalcWCQTCq9PVlUXMZjMOVwblJR4MrSR8guajomwzrqTO9V7PbDWRmGajrCBcvs/jC2Ky6pgC4eSIYKkXZdHRarVXb4mUUjjMDhxmB4FQgGJPMcWeYvyhmjQQNwG2m3VMWEgOBkkP+qi+K+HqI1vxVe7EZ03DlpQh1UeEEEII0eDiShExwh4zDKM9kAQkGYbR3jCMJ4wmnr9wsKanaRqpqalokTrNhmFQVFQULeXnTGqPEXDUzENVUFWxa6/XsyWYSUiqqURS4Q1i6DWrsC0xH3tfTJqJDEcGOSk5dHB1IMGcEHM8QJACHX6zWNlqsuFWe6aP5KJyf6F890YC/gNTnlAIIYQQIh773cXDMIxywzDKD8ZkDjSz2Yzb7T5o1zeZTKSmpka3g8EgxcXFGIYRfpgvuTMhf80KaiBUgNddtNfrOZIs2BJqxpf7QlAdOAZDLTofe2+UUiRaE+mU1Ins5GxS7aloqubX1sCgXAux0WxmvcVOkaZHH4rUlYHVX4KtZAM/PziUFZ++TDBQfz68EEIIIcSBstcUEaXUz0Bc0ZxhGH0O2IwOoFatWrFjxw4yMzOx2+0HJS/XYrGQkpJCcXExAD6fj9LSUpKSktA0DVdSF8rL1qGZwmGf17cDzWTFvMeKLEQqbKTaCAZC+L1BQkBl0CChutDIIZCPvS9Wk5W2pra0srei1FdKkacIb6BmddpLkF0mnTzMJAZDpAW85BVXkLT5fXK8y2HRX9nxzZ1syz6fHqddRVJK+j7eTQghhBDij9lXDva8BpvFQZKYGK5BvXPnTvz+g1vOzePx4PHUlOSz2+1YreGUj6Dfj89fiNIiDy4aeVitrdG0+nOqjZBBZZkPIxgeb9GI5hmjQHdaUKb9byHu8Xiw2Wy/P7AZCQQDVPor8QQ8GLW+D+4gvLpdULqOTjs/IgNQQKaRS+a6R6h6/Cl+yDidNideS9Zh/Rtr+kIIIYRogfYaYBuGcXdDTuRgSUxMjAbaB5NhGLz11lusXLkyum/8+PEcdthhAKxZ/Brb8qeiW8Ir2aGqNgw79TNMpvoD3uLdlbwxbSneqgAKGJZqwRUKB5B6ipXW1/VHs+9fE5qFCxfSr1+/P/Dpmr78qnzmrZvH67+9Tr47P/Zg21TahdpzXlEeZ1cUk2AYOJSXowvehFfe5CfbAIyBk+l9wtloest+kFQIIYQQB9/+L4OKeimlGD16NB07dozue+ONN9i1K/xgY4+B5+AMXYgRSRDWHLv55rMJe82pTmmTwKlX9EbTFAbwfYmP6h42wWIvRfPWHnL52PuS4cjgyr5X8tG4j5h+wnSObB3bZGmnVsUj6U6GdurCzWkd2Wiu+XLSx7OUvl9dwc77evL93HvxSUt2IYQQQvwJEmAfQCaTiXPPPZeUlBQA/H4/c+fOjXZ7PGbkVEIFx0fHB8wr+XHRjXu9XvvuKZwwMdwosyoESytqHtDz/FJI5Xd7r0pyqDJrZk7pdArPn/o8b57xJud2Pxe7yR497sHPB4kwpn07zm2Xw8d2B9V3tb2xm0FrpzP0+wtZ9eAJfDdnKutWfE0oWLdluxBCCCHE3kiAfYAlJCQwYcKEaP51eXk5L7/8Mj6fD6UUw8f+B/fOrtHxZf75/PbzjL1er+fgdvQ/JbwqvstvsNFbE+yVvL8R3w5Zbd2bnJQcbht0G5+d/Rn/GPgPOu9Rh3y11csNbdIZnpXNk8npFERKLlpUgF7eFRyz6Sly3hpF6b1ZLJ0+hsVvPMauLb81xkcRQgghRDMiAfZBkJGRwTnnnBOtWrJr1y7mz59PKBTCZLYw/IxXcOfWlPfbtvthdm37Yq/XGzSmK136ZQDwiztESeThR4IGhXPXEPJI6bl9cVlcTOwxkbfHvM2Mk2cwouOImFJ/xZqPGSkOTszqyNWtslhis1L7kdgUyhlQsZCBP99F2+cHsu3uHvzw1MUs//glykoKG/4DCSGEEKJJkwD7IOnatSunnXZadHvNmjV88UU4iHYkpjJw8It4S8IPOCrd4OdfrqKsdH2911Ka4sSLe9Iqy0UI+LEygD+Sfx0s9FD85jrJx46DUopBbQfx+LDHWfCXBVze+3JSbbXqmBPiqwSDS9q2ZlDnLozPPIwnklqxwmqJCbg7GDs5uuBN+n17NY7Hsvn1/mP4btYUfv3hY/w+aWojhBBCHOriKkOhlJq0l0MG4AHWG4ax/IDNqoU46qijKCgo4IcffgDg66+/Ji0tjSOOOIJWHXuQU/gIG3Zeh9keRLf4+OGbcxl64ueYLUl1rmW26Jx2VR/mPfQjFcVeVlQFOSoh/I/P/VMBlV124xzUtkE/X3PW1tmW6/pfx1/7/pWPt3zMK7++wsr8mgowPgL8YgnwS6qNmbTBgk53r8ZxlYUc66mip9eHGTCpEIf5V8O21bBtBhUf2FmV0A9fx+Np238kHbL7oDT5HiuEEEIcSuKt8/Y0YAHMEG2Up0F0Yc+slFoOnGoYRn495x+yTjnlFIqKili3bh0A77zzDikpKWRlZZHT71RK8q+lJPAEmslAs5Ww6POzOeHkD9C0uv9oEpKsnH51X97851J2eoNs8gbpbA2XlSt5bwOWji4s7ZwN+vmaO4tuYVSXUYzqMorVhauZt3YeX2z8goJAQcw4H0F+tgb52ZrIv0nEamj09Bic4C7iKI+Xnl4fJsCp3PSr+hZ+/RZ+fYjdpLM1eSBaznC6DhxFSoZ8CRJCCCFauniX1s4BlgODAVvkZzCwFDgL6Ee4j8ejB2GOzZqmaYwdO5ZWrVoBEAqFeOWVVygqCrdMP+rka9GKa1JJQpYN/PDVFXtN+Uhv7+Tkyw5HKVjlDlFanY8dMCia+yshr+Rj/1E903pyxzF3cGfmnXw09iPuH3I/Z2afSaYzs85Yrwqx3G7weGoKE9u1YVCnTlzSOpPZSS5+tliilUnaUMDAkg84cskUkp7qwfp7+/Pdf65l1ddv43FXNuwHFEIIIUSDiHcF+1HgIsMwfqi17zul1N+B5w3D6KGUugH47wGfYQtgs9k477zzmDlzJpWVlbjdbubOncull16K3W5n2LjH+PB/W7G1/xmAqtBCfln2ML0G/KPe63Xqnc6Qc3L4+tV1/FgZ4ASXCZNSBArcFM9fT+q53Q9KW/hDSTtnO85wnsEZXc8AYEfFDn7c/SOLdy9mye4l7KqMLZHoVSGWOHSWOMIlGm0hRX+Ph0GeKga6vXT3+TApg+zgBrJ3bYBdL+L+1MJP9t5UtT+OjL4j6XL4QEknEUIIIVqAeAPsTkBVPfurIscANgEpf35KLVNKSgrjx4/nhRdeIBgMUlBQwOuvv87EiRPRdZ0RY1/i4zdPISFzNwC7i2eQuKE7HbueVe/1+gzrQEmum58XbmdlVZAB1fnYK/Kp6ppMwlFtGuyzHQoynZlkZmcyJnsMEA64F+9azI+54aB7d+XumPEezeBbh5VvHeFyjfYQHOl2c7THw5EeD4f5/NiVjz6epbB+Kax/nMI3ktiUeBShzkPpNHAUrTI77zkNIYQQQjQD8QbYi4FHlVIXGIaxG0Ap1QaYDlSvaucA2w/8FFuODh06cOaZZ/LGG28AsHHjRj788ENOP/10bAlOhoz4H998PQZHRgVKg9823IwrqRMp6fW3Nx9ydjal+W62/lJIujdEljW8+lnyzgYsHVyY2yQ02Gc71GQ6Mzkr5yzOyjkLwzDYXrGdH3f/yJLdS1i8ezG5Vbkx490afJ1g5+uEcNObhJDBkR4PR7k9HOXx0N3nJ41S0so+hZWfwsrb2Kx1YHf6MdgPG0H2UaeS4EpuhE8qhBBCiP0Vb4B9GfAWsFUptZNw9ZBMYC1wZmRMAnDfAZ5fi9O7d28KCwtZuHAhAD/++CPp6ekMGjSI1Had6H3406zZcBkWlx/NFOTHJZMYMvRj7Pa6D8dpusYplx3OG/9cys87K0kxKRJ1heEPUTh3Da2u6Ydm0Rv4Ex56lFJ0cHWgg6tDTcBdvp0luUuiAXdeVV7MOZWa4kuHnS8d4YDbGQyFA26PNxpwdwpto1PeNsh7Dd+XOqutPSltO5jU3qeQfcTx6KZ4//UVQgghREOK6//QhmGsU0r1Ak4GuhN+oHEN8IkReRrPMIy3fu86SqlTgScAHZhpGMZDexl3FPA9cK5hGPPimWNzcsIJJ1BQUMCqVasA+Oijj0hNTaVbt2506jWEkvxbyfPci24NoVmr+PbLsxl60ifour3OtSx2E6df3Yd5Dy9lSYWvJh87z03JW+tJPad7Q3+8Q55Sig6JHeiQ2IG/5PwFwzDYVr4tGmwv2b2EfHdssZ0KXWNhgoOFCQ4AEoMhBkQC7oEeDzk+Pz19P8OWn2HLs5S9l8D6hP74s06g/ZGnkdnl8Mb4qEIIIYSoR9xLYJFA+qPIz35TSumEy/2dRDiVZIlS6h3DMFbXM+7hP/o+zYFSijFjxlBSUsL27dsxDIN58+Zx6aWX0rp1a44YNomFr68lYH4ZpQGWXXz35SQGD3sVpeo+BJeYZuf0K/sw/9Fl/FQVpH8kH7tqWR7WrskkDGjdwJ9Q1KaUomNiRzomdmRst7EYhsHW8q3RYHvJ7iUUuGPLApbpGl8kOPgiEnAnBYMMiATbR7q95Pgr6V/5Naz+Glbfx07Vmm2pgzDnDKfrwNNJSs1ojI8qhBBCCPYjwFZKHQ2MAFqxR3k/wzCui+MSAwk3pNkYud4rwBhg9R7jrgXeAI6Kd27NkdlsZvz48cyYMYPS0lJ8Ph9z587l8ssvx+l0csLYe/jgxU3YOn4PgJdlrPjhH/QbNK3e67XunMiJF/XkoxmrSPeF6GiJ5GO/tR5Leyfm1pKP3VQopchKzCIrMYuzu52NYRhsLtvMkt1LopVKCj2xLdhLdZ3PExx8Hgm4k4NBjvR4I3ncXrL9ubQrfBsK3yb43fWsNedQ2HowiYefTM6A4Vistsb4qEIIIcQhScXTYlspNQWYBqwHqnOwqxmGYQyP4xrjCDeiuSyyfQFwtGEY19QakwnMBYYDs4D36ksRUUpNBiYDZGRkDHjttdd+9zM0VRUVFSxfvpxgMAhAYmIiffv2Rdd1gn4fxRvvJSVnZ3R8wH0u1oST93q9/NUGhT8ZnOAy4dLDpfq8ToPtx4Qod1fgdEojmt9TUdG498kwDHIDuaz3rGetZy3rPespD5Xv85yU6oDb7WGgx0tXv5/qQo2VhpXVpp7kJvaFtv1ISO94QMoBNvZ9ak7kXsVH7lP85F7FR+5TfOQ+xW/YsGFLDcM48vfGxRtgbwMeNgzjqT86IaXU2cApewTYAw3DuLbWmNeBRwzD+F4p9QJ7CbBr6969u/Hbb7/90Wk1CWvXruXll1+ONpfp1asXY8eORSlFaf5uvvxoFAntigEwQorePWfSut3Qeq9lGAaf//dXdny/ixNcJvRIPeyEo9qwMm0HQ4fWf56osXDhwiZ1nwzDYFPppmhKyY+5P1LkKdrnOanVKSWRKiVd/IFowJ1HKluSjkJ1HU6ngaeT3qbDH5pXU7tPTZncq/jIfYqf3Kv4yH2Kj9yn+Cml4gqw400RSQQ++HNTYjtQ+//k7Qmvhtd2JPBKpElKOnCaUioQzwOUzVm3bt045ZRTWLBgAQCrVq0iPT2doUOHkpTRhv4DZ7Dip/Oxp3pQmsFPP1/JINe7uFzZda6llGLohO68k+/mpy2l9HOE/xFXLtlNYk+FETJQmjShaU6UUnRJ7kKX5C6MP2w8hmGwoWRDtErJj7t/pNhbHHNOka7zSYKDTyIpJanBYKQkoJejPGUcWfoRatlHsOxmNmqdyMs4BkePk8g56mTsCa7G+JhCCCFEixFvgP0ycCrw7z/xXkuAHKVUZ2AHMB6YUHuAYRjRzhq1VrDf+hPv2WwcffTRFBQU8OOPPwLhb5NpaWn07t2b9t36UZL//+39d3gk933n+75/VdU5A42MyZEzQ3LIYRBJicHKlhxkeS3Ta3ttX9nrPetd+5yzd9Nzn72795x7dn033LN+du/xsWVpj5MsWpa1CrRkUdIokAoMEjWcyOFE5Bw6d1X97h9VaHSjG0DPEANght/X8+Dp7qrqRqE5BD744lvf3//K0Ow/IxB1MAIVvvf8z/P4j32FYLB5bR/TMnj/b97Np//di1zPV9nh92N3nzEY+90XiZ7oJnZ/D1a2eSqJ2P6UUuzP7Gd/Zj9PH34aV7te4PYvmHxp/CXmynMNz5kxTb4cj/HluNeL32k7PFgbCzjEw+OfRI1/ksrXLV4LH2Ox/+1k730f++5+FMOUUY9CCCHEjWg3YF8H/o1S6jHgR0C1fqfW+j+t9wJaa1sp9Vt400FM4ONa69NKqd/09//+DZ35HUYpxfvf/35mZma4dOkSAJ/97GdJp9Ps2LGDY499iLm/fp2i/QcYlkYFZ/nON5/mHT/2OQwj2PR64ViAD/7Wcf76//MSGUcT9/uxnfkyi1+7zuLXrmP0x0i+rY/ovV0YIZmpfLsylMGBzAEOZA7wC3f9Aq52uTh3sSFwz5fnG54zbZl8KR7jS37gztYCd4kHi69x9PIPUZf/C7OfTXApfgJnz1PsfPAD9O48sBVfohBCCHFbabcH+/Iau7XWeu/GndKNuRN6sOsVi0X+6I/+iKkpb2xbLBbjox/9KJlMBq01f/Pf/gGhXV+pHZ8I/RgPPvoH+G01TUZen+Vv/vMP2R9QDAYMQi3aQxwFpc4IoXuydD3cRywVujVf3G3iTutFc7XL67OvNwTuhcrCms/ptm0e8Be9ebBYZqft9XBfV/2MdL6N4MF3MuVmePf7PrA5X8Rt7k77N3WryPvUPnmv2iPvU3vkfWrfhvZg17duiFsrEonwC7/wC/zhH/4hxWKRfD7PJz/5SX7t136NcDjMu//uf+aL/+3DJPafBWCx/DXOnfr33HXPP235ev0HMrzrH9zDd/76DU4P5egNKHYEDXosheGHclNDbKoIX7vO0FeuMWEaVHcm6DiYoWdPiq4dcSxZEfK2ZSiDQx2HONRxiF888ou42uXC7IWGwL1YaZxSMmFZPBu3eNavcHfbttdOUpznwbnPseOFz+Bogzde3MV06hgMnKDz4CPsOnw/VqD5LypCCCHEW4n0BWxDHR0d/PzP/zx//Md/jOM4TExM8OlPf5qnn36aQDDEu/7OH/OVv/4gyd3jAAxP/p8krh5kcNdPt3y9XUc72XW0k69+5esc2nEv41cWOHdxDuvaIr1akzSXq9oxU7EHDdcWmHxjjhcrLmMOpAfj9OxJeh+7k6S7o3Kx5G3KUAaHOw5zuOMwv3Tkl3BcpyFwvzz+MovV5sD9xbjFF/3A3eMH7ntLkxzLfZmDpz5P8BQUdZDXgweY77iHwM4H6L3rMfp3H9qQsYBCCCHE7WLVgK2U+j3gX2it8/79VbW50Iy4Abt27eInfuIn+OxnPwvAxYsX+du//Vve//73E8908MhTH+f73/k5Yr15lIJzF/4pseRuMpnjq76mGVAMHu5g8HAH4I1/W5guMvmDSSo/miI6VcCq6xjqChh0BQxsrRmeLHBtOMdr3xgGIBS16N69HLh79iSJxKVyeTsyDZO7Ou/irs67+OWjv4zjOpyfPd8QuHPVXMNzxi2LL8QtvuAH7oDWHC5XOFaucHflMsdmzrNr/JMYL8IsCa6FD1PoOk50z0PsOPYYHd0DW/GlCiGEEJtirQr23UCg7v5q1m/iFjfl+PHjTE9P861vfQuA733ve2SzWR588EF69xzm0OR/5I3hf0QoVUWZDi9//5d5+xNfJhzua+v1lVKkslFS794F796Ftl3yr02x8MIozrWF2txkSyl2hRS7QgY5R3O94nK9aHP9zAzXzyzPY052RWphu2dPkq7BBGZAKpe3G9MwOdJ5hCOdR/h7R/8ejutwbvYcL46+yIvjXuDOV/MNz6kqxalwiFPhEJ/EG/MXd12OlivcXS5zrPwad4+8Qvf1P4RvwojqZjR2hGrvfST3v43dxx4hGk9txZcrhBBCbLhVA7bW+qlW98Xmeuqpp5iamuLsWa/n+tlnn6Wjo4N9+/Zx6KF3M/v532a+/J+wQi4qkOc73/o53vHUl7CsG18aXVkG8ePdxI934yyUKfxggvxL49iTxdoxcVNxV8TkcNhg0vbC9mhV4wALk0UWJou8/qLXumJYiq4diYbQncxGVr0gU2xPpmFytPMoRzuP8ivHfgXbtTk/c55PvfApSukSr029xvXF603PyxkG34uE+V5keZn2btv2qtzlEsfK3+HoG98gcVHj/I3isrmLydRRdP8JsoceYefhEwSCb+0LboUQQtyepAd7mzMMgw996EPMzc0xOjqK1ppnnnmGj370o3R1dfHwB3+Tv/3ji+j+z6JMcM0Rvv/Cr/LIOz6JUjd/YaKZDJF4YgfxxwepXF+k8PI4hVcn0SVvSXelFN0BRXfAwAaGqy5XSy6zzvIfNFxbM355gfHLC/B1b1s4HmgI3D27k4SigRZnILYry7A4mj3KjyV/jCcffxKAudIcr02/xqmpU5yeOs2pqVMtV5ucsCy+Zll8zV8AB2BPpepXuac4lv8Kh05/keBpr5/7YnA/85m7sXY+QN+Rx+jffZf0cwshhNj22g7YSqmPAO8EuoGGn3Ba65/c4PMSdYLBIE8//TQf+9jHWFhYoFwu82d/9mf8+q//OrFYjHf93X/H5z92ncTBlwEo2i/zo1f+Jfee+N03/bmVUoR2JgntTJL+4F6KZ6bJvzRO+eJcrTnIAnYFDHYFDJyoxXQ0wMW5CpNTpabXK+WqXH1tmquvTde2pXuiDb3cnYNxTFNC1O0kHU7z9oG38/aBtwNef/9ofpRTU6d4bcoL3memz1C0i03PvRwMcDkY4HP+ApKW1hyu+P3c5ascm73A7olPYbwEc8S5Fr6LfNe9RHY/yI5jb6ezZ3Azv1QhhBBiXW0FbKXUvwd+B68OOYL0XW+6ZDLJ008/zcc//nGq1Spzc3N86lOf4pd/+ZexrADv/bsf42/+/KdIHbgGwNT8p7n0+iH2Hvi1DTsHFTCJ3ttN9N5u7PkyhVfGKbw8gT21HJrMgk13waZbQeD+LKW+OGOuZuLqIuNXFigX7KbXnRsvMDde4Px3x7zXCBhea8me5Up3oiMsrSW3EaUU/fF++uP9vHf3ewFwXIdL85dqgfu1qde4MHsBRzsNz7WV4rVQiNdCIf7C37bUz32sXObu8mscG3mFnusfg2/BKF2Mxo9Q6b2PxN6H2X33o8QS6c39goUQQog67Vawfxl4Wmv96Vt5MmJtfX19fPjDH+Yv/sKLHdeuXePzn/88P/3TP00kkeTxD3yCb3/1wyR2zAFw6er/Rjy1l+7uJzf8XKxUiORTO0k8uYPK1QUKL09Q+NEkuuyHJQ3VS/OYl+YZDFscvDdL9Kf3UgiaTFzx2kbGrywwdT2H6zb+vuZUXcYuzTN2aXn1wUgy2NhasitJMCIdTrcT0zBrK05+6MCHACjZJc7NnGsI3dcWrzU9d7V+7uWLKL/L0UvfJHlR43xZcdncyWTyKHppPvddD0g/txBCiE3TbkIxgB/ewvMQbTp8+DDvfve7+cpXvNUcX331VbLZLO94xzvIDu7mnuP/hdPnP0o0W0IZmh/98B/wtkc+Rzxxa5a4VkoR2p0itDtF6if2Ujw9TeFlv4XEp0s2+e+Nkf/eGFZ3lP4TPez/wB7MZBC74jB5Pcf45XnG/eC9ON3cWlJcqHDlR1Nc+dGU/4kh0xujt67K3dEXw5DWkttK2ApzvPs4x7uP17bNleY4PX26ob1ktX7uCcvi63X93LsrVe4uVzhWnuZY8TkOnXmW0GkofSbAG4H9zHXcjbXDm889sPeI9HMLIYS4JdoN2H8A/CLwr2/dqYh2Pfroo0xNTfGDH/wAgK9+9at0dnZy5MgR9t77CLNj/5KJ/P+LYMxGWRW+/92nefsTf3vLz8sImsTu6yZ2Xzf2bInCKxPkXx7HmVkOzPZEgfm/ucz8ly4TPpgh+kAPvXd10rdveURbYaHih+15xi8vMHFlgUqpsY0ADbOjeWZH85x9YRQAK2jQvStZV+lOEc9I1fJ2kw6neWzgMR4beAzw+rnH8mMNgfv09OmW/dxXggGuBAN8PuFN0bG05lCtn/sax2ZfZ8/EMxgvwzwxroYPk8/eS2T3Qwwee4xs785N/VqFEELcmdoN2GngF5RS7wZ+BFTrd8pCM5tLKcUHPvABZmdnuXLlCgCf+cxnSKVSDAwMcOK9f5ev/ulFnOCfYAY02pzle8//Apr/edPO0cqESb5zJ4mndlC5skD+5XGKpybRFdc7QEPp/Cyl87MYUYvIvV3ETvQQGIgTTQbZc0+WPfdkvUNdzexYgfEr87XWkunhPHpFa4ldcRl5fY6R1+dq22LpUMMFlN27kgRCsuz77UQpRV+8j754H+/Z/R6gsZ97KXS/Pvs6tm7s8beV4nQoxOlQiE/522IN/dxnODb6Q3qGPo76NozRxUjsLio9x0nsexu77n6UeDKzyV+xEEKI2127AfsIyy0ih1fskwset4BlWfzcz/0cH/vYx5iZmcG2bT75yU/y67/+66RSKZ56+v/B5//gCrGD30QpqOjXcSr/lrHxEtnOH7upOdk3QxmK0N4Uob0p3J/cR/HUFPmXx6lcXu6vdgs2+e+Mkv/OKIHeKNETPUTv68b0V4ZUhqKjP0ZHf4y7Hu0HoFp2mLy26I8B9NpLcrPlps+fnytz6QeTXPrBpPdaCjr64w0XUGZ6Yxiy7PttZb1+7temveB9deFq03PzhsH3I2G+X9fP3dXQz/19jl75FqlLv4f7t4or5g4mksfQ/ffRcfARdt31IMFQuOl1hRBCiCVtBWxZaGZ7ikaj/MIv/AIf+9jHKJVK5HI5PvnJT/Krv/qrhEIh3vf3/guf//jPkjlyAQAzeJXTp38HtEUm9XYGdnyIzs6nNi1sGyGT2AM9xB7owZ4ukn9lgsLL4zhzy8G4OlZg/ouXmf+bK4QPZYg90EP4UAfKauyVDYRM+g+k6T+Qrm3Lz5X9Crdf6b66iF1ubC3RGqaHc0wP5zjz7RHvtcKm11riV7rLi5pSvkowYknwvo206ueeL8/X5nIvVbqnS9NNz520LE5aFidX9HMfq1Q4Vp7hWPE5Dp99ltAZKP91gPOBfcxm7sbacYKeux5lYO8xDFP+MiKEEMIjYxhuc9lslo985CP8yZ/8Ca7rMjY2xmc+8xk+8pGPEIrGeOeH/4ivfObv0HnX2PKTlM3swklmT58EHaAj8zj9Az9FNvsUphld9XNtJKszQurdu0i+cyflS/MUXh6n+NoUuuq3kLia0tkZSmdnMGIW0ePdRE/0EOyPr/qasXSIvfd1sfe+Lu8lXM3saJ7xywuM+f3cM6P5pr+5VEsOw+dnGT4/W9t28Yve8vTBiEUoYhGKWYSiFqFoYPk24m+LLW8PRwPec2KWzPLeBlKhFI8OPMqjA48CXj/3eGGcU1OnaqH79NRpCnah6blL/dxfiC/3cx+sVPyLKK9z99xFdk/+JeYrsECMq6FD5Jb6uY8+RrZ/16Z+rUIIIbaPVQO2UupzwC9qrRf8+6uShWa21p49e/jABz7A5z//eQDOnz/Pc889x3ve8x4yff2886c+ybc/83sslp4nvWeeSGddK4WqMjP3VWbmvgoE6Ox4kr7+nyTb+eSmhG1lKML704T3p3F/ym8heWmcytWF2jFu3ib3/Ai550cI9MWIPtBD9Hg3ZmztFSANQ9E5EKdzIM6Rt3utJZWS7c3k9gP3+OUFCguVVV+jUrSpFG0Wm4dYrMsKmYSjK4J5xL+/MrCv2G4FpBp6Kyil6I310hvr5d273g14/dyX5y/XLp48NXWKCzMXWvZznwmFOFPXzx2t9XNXuLt8lrvHXqVn+BOo52GcToZjRyj33MdcNc7VngTdOw8SiSU2+asWQgix2daqYE+zXOtr/puq2FZOnDjB9PQ0L7zwAgAvvPAC2WyW+++/n87BnfzUP/4PPPflL9EbNLnwyucp2N8nvXehMWxTZXrmK0zPfAVFkM7Op+jt+wk/bEdu+ddghC1iD/YSe7CX6lTRW579lXGc+eUAXB3NM//5S8w/e5nI4Q6iJ3oIH8qg2qwWB8MWg4cyDB7yLlzTWpObLTf0ck+NzqNci0qxeVGcG2GXHXJlp2Vv+HrMgNFeII9ahOsq6MGIRSBkyqI8N8A0TPZn9rM/s7/Wz112yk3zuVv1cxcMgxcjYV6s6+futB2vl7tS4e7yixy98m0ecTV86n8DYIo0U1YfuegA1cQOzI7dRHv20jFwkO7BvViB4OZ84UIIIW6ZVQO21vpXW90X29e73vUupqenOX/+PABf+MIXyGQy7NmzBwArFObYk09y7Kl3k5+b5cL3nuf173+BivoB6X2LRDqWg6CmwtT0l5ma/jKKENnsO+nt/QCdnU9sStgOZCOk3rub5Lt3Ub44500hOT0Ntt9C4miKp6cpnp7GiAeI3tftTSHpvbF+cqUUiY4wiY4w+090A3Dy5EmefPJxXFdTKdqUCzblQtW/rb/fvK3k368UbPSbuPzXqboU5isU5levrq/GMNWKdpYVLS0rA3ut1SVAMCzhHCBkhri3617u7bq3tm2+PM/p6dO10H1qsnU/97RlctKKNvRz99g2A7bNYNVm0LYZtK8yWH6DXXmb7LCLOuUdZ2uDESPLbKCPfGwQJ7WTQOce4r37yO44SGf3oMzuFkKI24D0YN9BDMPgZ37mZ/jEJz7B2NgYruvyqU99io9+9KNks9mGY2PpDPe994Pc994PsjA1wfkXvsXF57+IEzrtVbY7loOdpszk1LNMTj2LIkxX9zvp6V4K27d2moIyFOGDGcIHM7hFm8KPJim8PE7l2mLtGDdXJfetYXLfGiYwGCd2oofovV0Y0bVbSNZjGIpwLEA4FgBu7JcK7WqqZYdSq2CetykXWwV2/37eblrd8ka4jqa4WKW4WF3/4BWUguAqgTzs3w/623NjmomrC7XHwcid3XeeCqV4tP9RHu1v7Oeur3K/NvVay37uccti3LJ4pcX/LmHXpd92vOBdtRmwiwzaFxjMnWFwziZ6ZfnfQlEHmTB7mAv1UYoNojO7CWb3kurbR9fOQyTTnbfs6xdCCNG+tgO2Uuop4GlgJ9DwN0yt9Y9t8HmJmxQKhXj66af5wz/8Q3K5HKVSiT//8z/nox/96KrPSWa7efAnP8yDP/lhZkaGOP/Ct3jj619EJV4nvW+BcKY+bJeYmPgiExNfxFARurreSU/PB+joeALTvLWLuhgRi/jDfcQf7qM6UaDw8jj5VyZwF+taSIZyzA3lmPvCJSJHOr0WkgMZlLm5VVllKIIRL3Ryg5lHa41dcRsq5aW8F8ArRbsutNcH9uXHztKFojdBa7zXy7fXHnP15EsNj62QWRfMvVaWYNQiFFluYVlucWkM7Lfb1Jb6fu537XoX4PVzX1m40hC4z8+ex3ZXfz9LhsGloMGlYOtfCDscxw/efvW7OsOgPcGOuZfpmXIwX18+dp4Yk2YvC+EBKokdqI7dRLr2kh7YT/eOA4QjmzMxSAgh3uraCthKqV8Bfh/4a+BJ4L8DB4E9wJ/eonMTNymVSvH000/ziU98Atu2mZmZ4ZlnnmHnzvVXqevoH+SRn32at33455m8epmzL3yDy899iUDmCum9jWHb1UXGJ77A+MQXvLDd/W56un+cjo7Hb3nYDnRHSb1/D8n37KZ0cdabQnJ6Ghy/2udoiqemKJ6awkgEid7vt5B0b86UlDdDKUUgZBIImcRvYo0Tu+osB+9C1Q/fKwJ5oUrJD+z17S0rxxre8OcuO9hlh/zcjfedgzcyMVQXuOt7zoP1wXwpsNdtC4Yt1BYHdNMw2Zfex770Pn56/08DUHWqfPZrn2Xg6ABDi0MM5YYYWhxiODfM0OIQC5WFNV9zxjSZMU1+RPP/U5bW9PnBe2Cp/aQ6wmD1Gvsmvk1yzKX+HZmgg+lAL/nIAHZyJ0bnHmI9++gcPEBX/x5MS/6oKYQQG6Hd76b/BPgtrfXHlFKLwL/QWl9SSv0XIHfrTk/crIGBAT70oQ/xl3/5lwBcuXKF+fl5ent72b9/P8Hg2hdSKaXo3r2X7t17eVz/CqOvn+PsC9/gyne/TLh7mPS+RcLpFWF7/HOMj38Ow4jS3fVuurt/nI6Od9zSsK1MReRQB5FDHbiFKoVXJ8m/PE51aPmfpbtYIfeNIXLfGCK4I+FNIbmnCyNyZ4YJK2BipUxiqRt/3x3bXQ7jxca2lYaQXrQZH54kEoov96kX7Te97FS15FAt3dyFoSjvItamMO4H8oYwXt/+4m8PhMxbEtADZoCuQFettWSlhcoCw4vDteBdC98573at6retFNcDAa4HAi27mBKO64XvWv93hUH7EgOFC/Qv2ASHlo+taJNRo4vZYB9Fv/87mN1Lom8/2cEDZLJ90v8thBBtajdh7AWe8++XgaVhxP8FOAn88409LbERjh49yvT0NF/72tcAmJ2d5ZlnniEQCHDw4EGOHDnCgQMH2grb/Qfvov/gXbjur3P99CnOv/ANLn/7q0T6x0jvXRG23QJj4/+dsfH/7oft99Dd8+N0drwdw7h1YduIBog/0k/8kX6qY3nyL49T+MEEbm65F7lyfZHK9UXmPn+JyNFOYid6CO1Pb3nlc7swLYNoMkg0uf4kC+9i0Idqj5f6zldWzOsDeKWw3H/utbssbbPf9NQW9PJYRW5irKJSNLSr1F8E2lg9twg29Ki/uektyWCSZGeSuzrvatrnuA6TxUmuL16vVb+XKt9Di0MtL7Kst2ganDWDnA01//dUWtPjOMuVb9tmsLrAoD3D3vlX6ZxxUZeXjy/oEONmL/PhfkqxQcjsIpTdQ3rgAN07DxFLpG/4axdCiDtVuwF7Glga3joMHAN+hNddeutHSoib9o53vIPFxUVefPHF2rZqtcrp06c5ffo0gUCAAwcOcOTIEQ4ePLhu2DYMk113H2fX3cdx7P+BK6/+gHPPn+TiN04S3zFFZt8CodRyoPXC9mcZG/8sphHz20g+QEfHY7c0bAd6Y6Q/sJfU+3ZTOu+3kJybWW4hsV2Kr05SfHUSMxUken8P0fu7CXRt/xaS7aq+7zzRcePPd11NtbQyjDdWzZe31Qd3b3+19ObaW7SmdrHpzVCGalk9D0Ysxidcvj3xOoapUKbCMBSG6X0oY+mx4W0z/G3+fsMwyJq76TL2cCKkMCIKo9c7tuyWmS5PMVEaZ6I0zlhxjNHCCKPFEUYKIxScPK5y0MrF9T9Q3v8DWinGLIsxy+LlFl9P2HX9vm+nNv1kwJ5gsDrCwanvEp3UcGH5+FmSTFq9LIb7qSR2YnTsItq9l/TAAXp2HJDl5YUQbyntBuxvAe8BTgHPAL+nlHo38E7gK7fo3MQGUErxgQ98gAceeIBnn32WfD7P1NRUbX+1WuXMmTOcOXMGy7IawnYotHYANq0A+048xL4TD1Et/2MuvfIi557/BteufZvkrhnSexvDtuPmGRv7LGNjn8U0E3VtJI9hGLdm9q8yDSJHOokc6cTJVyn80FuevTqSXz6v+QqLX7/O4tevE9yVJBlTVEbzBHqiUtneRIah/JaNm5v+4joulZJTF76rzWHcD+Stgnr1Tfafa1dTylcp5VtPb5l5/fqbev31dROjm/3cw/41jtIshW3H/3DrAvhSGHeWH+OSVy5nlcNp5aL9fRY2YRzC2ibqf8R1lbi2iY1UMRliUV0lx3MM41BUIUpmjGowjo6kMWMZQqksyWwviUwnpmUyf01z+UdTWJaBGTSwAgamZWAFDUzL9B4HDSzLkP83hRDbWrsB+7eApfLDvwVs4DG8sP2/3oLzEhusp6eHPXv28MQTTzAxMcGZM2c4ffp0Q9i2bZuzZ89y9uxZLMti//79HD16tK2wHQiFOfTIOzj0yDsoF/JcfPG7nHv+JFdHvktqz3xz2HYWGR37DKNjn8Eyk3R1vZvu7vff0rBtxgIkHhsg8dgAlZGct5DNDydw6yZmVK4u0I3BxJlXUCGT4I4EwZ0JgjuThHYm3vToP3HrGKZBOGb4YxVvnOO4VIveaMX6QF5fKa8F8+Ly9JaKH+Ttys1Pb9lMCgNTG5h6Y69BKPofkzf8zCn/wzP0wo/aepZhKi9wBwysgIlZu2+sut0MeOG8FtiD3rblEO8/P2hiWiue57+eYSmZFS+EWNe632GVUhbw88BnAbTWLvC7t/a0xK2ilKKnp4eenh6eeuqphrA9Obn8o9G2bc6dO8e5c+cwTbMhbIfDa/+pNxSNcfSJd3L0iXdSWJjnwnef5/wL3+DK5Euk9y6S3rdAKLkctm1ngdGxv2J07K+wrCRdXe/xwnbm0VsWtoP9cYL9cVLv30Pp/Az5l8YpnZ+Buoykyw7li3OUL87VtlldEYI7kwR3JgjtSmJ1S5X7TmGaBmbcIBy/yYBuu0395kth/dzZC+zbuw/X1biORvu3rqNxXY32b13HXfG47tbRaNdd8bhxf+25/ussP14+7k7hOpqK40DJAW585vtNU3gV9obgbjYEe7NFyG+uypu1QN/y+KbtBsYdPGdeiDvNugFba20rpf498MVNOB+xybq7u+nu7ubJJ59kcnKyFrYnJiZqxziOw/nz5zl//jymabJv3z6OHj3KoUOH1g3b0WSK4+/5cY6/58dZnJ7i/He+xbnnv8HCwinS+xa9ynZ92LYXGB39NKOjn8ayUnR1vYee7h8nk3kEw9j46rGyDCJHs0SOZnEWKxR+NMnw998gWQg1zNeund9kEXvSW8YdaKxy70oS2iFV7rcq0zKIJIJEEs2/FE46r3P8yfXHZG4GXR/cl0J9q9Df9NjFcbzAbtsOs8U5pvJTTBdmmC7MMFucZbY4z3xxnmKliKFNDG2g/FtDmyhtePcxMVwDxfI+QxsYrknADRB1TKKORdi1CLoWphvAcAO4Ooijg9gEgC0Kmxrsqov9JubN3yxlLFftHddl+OvfxQwYBIJ+yA8uV+CXKu5msG5/wPCPMWu35srnBJePk0AvxM1r92+E3wVOAFdv4bmILdbV1cUTTzzBE088wdTUVC1sj4+P145xHIcLFy5w4cKFWtg+cuQIhw4dIhJZ+3rXRGeWBz74IR744IeYHRvh/PPf5Ny3vkGhfIHMvgXSexcJJurD9jyjo3/J6OhfYllpbxpJ94+TybztloRtMxEk8dgAY9XXOfTEQzhzZSrXFqhcXaR8bcHr216xwuKaVe5dCUI7pcotthdlKExDYb7pV+rGWw6hWaFaYDg3vDzxZMXs75JTuqnPmHIcdldtdleq7KpqdlUUgxVFT8XAIIitAyzoJOPGDnKRQZz4TqzUDuIdO4jFszgOOFUHu+ri+CG56dZ2sCsuju16t1UX23ZxKg627eLaW/dXgKVJPUvXC8wVmlcN3UiGodYM4A2hful+cLkCbwWXg3wt3K+xX75PijtJuwH7D4H/oJTaCbwM5Ot3aq1f2egTE1srm83y+OOP8/jjj9fC9pkzZxgbG6sdUx+2DcOohe3Dhw+vG7Yzvf287cM/z8M/8xGmrl/l/Avf5NxzJ6nqq6T3LZDeu0AwsdwbbdtzjIw+w8joM8thu+cDZNJvwzA2fp61UgorE8bKhIne2w2ArjpUhnO1wF25toDbYjnyllXuuj7uoFS5xR0uGohyIHOAA5kDTfu01kyXphuC99L9i1MXmXfmV33dedPkVdPk1XDjNSGG1gzYNrureXZX59hdvcSeapXdC1Wysy7qijdmcDiwk7nYXnTnIWJ9d5Hdcw99uw/f0AI72tVe4K6F8tUD+1KYb97v1D1/Kdi72BV/+1K4t+v2Vxz0Jmd719W4/mz6zWjDMSxFYGUAr7s1byK0F6Y0Y5fnvek8ypvQowxaP/a3Gf42tTTRR9U9lv570Sal1/g/Vin1ceB3gLk1XkNrrd98MeQmHTp0SJ8/f36rPv1txZtb/OSbeo3p6ela2B4dHW15jGEY7N27txa2o9H2Rt9prRl74wLnnv8m57/zTXRghPTe5rBdLxDooKvLG/2XTj+8IWG73fdJa40z61e5r61e5W7F6o4Q3HF7V7k34t/TW4W8V+05efIkDz72IFcWrnBl/krD7dWFqxTt4g2/Zsx12V2tepVv/3ZPtcrOqk1Ea8o6wJA1yGx0L9WOA4T67qJz9z307z1KIHhrV6S9UY7jh/KKy7e/9TwPnngIu+KH8IpTu3UaHvu/BFSWg7q96n6n9vr2FgT624LCD9zLoXs5rK/xuG7bqo9rAZ/VHxsKQzXur32ets+r+XOcv3COu+89SiBoYgW9lYSXfnkJBE2skHftgPyCAUqpl7XWD6x73DoB2wH6WGfWtdZ6y1pHJGC3b6N/yM/MzNTC9sjISMtjDMNgz549HDlyhLvuuqvtsO26DsNnT3Pu+W9y4XvfxohOLle242uFba9n+82E7TfzPrVb5V6pqcq9M7ntV5qU0Ng+ea/as9b75GqXicIEl+cvNwXw0fwo+iaWEe2z/dBdWQ7fu+0qvbaDo02GzX5monsoZw4S6D1Mx6576N93jHAk9ia/0jfvVv+b0trrv6+F8RYB3K7WhfM19jtNvwSsCPWVrelpFzdGKWp/IfACuB/GgwZWyMQKmARCdaHc374U2uufF2h6He95xm1QaGo3YK/3E1zB1gZosX11dHTw9re/nbe//e3Mzs7Werbrw7brurzxxhu88cYbfOELX2DPnj0cPXqUw4cPE4ut/kPKMEx2HL2HHUfv4cd+7Te5euoHnH/+m7z+V98hkJrxppGsCNvV6gwjI3/ByMhfEAh00N39Prq73k8m8zBKbc4fWVTAJLQ7RWh3igTtV7l12aH8+hzl1+dY9LdZ3RE/cHuVbqvr9qtyC7FRDGXQG+ulN9bLI/2PNOwr2SWuLlxtWfnOVXOrvuaoZTFqWXxnRQkp7LrsrNrsrhbYXX2V3TMvsWfcpvulKgEXrht9TEV2U0ofwOo5THrX3Qzsv4doPHUrvvQtoZTCtBSmZRDahF/2tdbLLTOrBHC7qfrut+esUsWvVhzmZudJxBPehbraa/HR/v2li3mXPtz6/S0ev9Ur+lpTuwag2Ebh6GaYftvPWqHcCpkEAl5VvVVYXyvcG+bmtfm083/NW/yflGhHJpPhscce47HHHmNubq4WtoeHh2vHaK25dOkSly5d4gtf+AK7d++uhe14PL7qa5uWxd77HmTvfQ/yrkqZyz94iXPPf4Pzz3yfcMcC6X0LpPYuEow1hu3h4T9nePjPCQQ6vbDd/X4y6Yc2LWyD38vdEcbqCBM97vVyuxWH6nCOyrUFylcXvSp3rkUv90QRe6JI4SW/lzu8NLEkSWhX0uvl3uZVbiE2Q9gKc6jjEIc6DjVsX+r3vjx/2QvgS+F74QpDi0M4uvXiQiXD4EIoyIUWS8xnbcevdp9ld+4Ue2ar9JyyCdo2I6qbyfBuiqkDmN2HSOw8Rv/+4yTTnbfk676TKKVqFVE28A8EXqX/wQ15La0bQ/rK0O4Fdv84V9cFeP8vAnVBfTm0a1x3ndfU/mu4za+x6udo+UsEuLpx/9L90dExOtJZ7xeUskN16a8WFS9QL10TcKstXZtQzt/cirrrUYbyKu5B0w/gxnIl3m+LWW6TMVpW6dvVzk/nsfXS/lb2YIvtJ51O8+ijj/Loo48yNzfH2bNnOX36NENDQ7VjtNZcvnyZy5cv88UvfpFdu3Zx9OhR7rrrrjXDdiAY4uDDj3Hw4ccoFwq88dJ3OffCNzn3yVeIdOX8nu1FAg1he5rh4T9jePjPCAazdHW9j57u95NOP7ipYXuJETQJ7UkR2tNc5S5f9Srd1dFcw0xuAF1aUeVWYHVFazO5gzulyi1EPaUU2UiWbCTLg72NIavqVLmeu95U8b4yf4XZ8uyqrzllmUxZJi9FGkeUWlr7Ve9L7C6fZ/elz7LnfBX3WZsJN814aDf51H5U1yESO7zgnc723pKvW9waSimUAu7A77EnT07w5JP3rHmM63h/KVgK3HbFD+L+Yy+UO1TLdfsqrr9/+XG19nip/Wj5/q0u6WpXUyk5VEpvbuXedrQTsH+DtS9yFGJV6XSaRx55hEceeYT5+flaz/b168vLRmutuXLlCleuXOHZZ59l165dtZ7tRCKx6muHolGOPP5jHHn8xygszHPx+9/h3PPf4PSfnSLWU/B6tvcsEIgt/49UqUwxPPynDA//KcFglu6u99Pd/eOk0ye2JGzDGlXuoZzfx71KlVuDPVHAnijUVbktL3D7fdzBnQmMsFS5hVgpYAbYm9rL3tTepn3z5fmWvd7XFq9RdVv/adxWikvBAJeCzROC0o7D7uowu6tX2D30N+y+bBP+apVSNcZ0cBe55H7IHiQ2eJTe/cfp7B5EGTKDWmwvhmkQNA2Ct+hnylKb0MogXgvgK4J5dUWw9yrvdaG9RdB32xhCsFHaeZc+r7WeWP8wIdaWSqVqYXthYaEWtq9du1Y7ZrWwfeTIkTXDdjSZ4p53vY973vU+cjPTnP/Otzn/wjc5/cI5Yr1Fr7LdImwPDf8JQ8N/QjDY5beRfABvsdKtZQRNQntThPZ6PZ1aa5yZUq2Pe/Uqt035wizlC34FToHVHfX6uP3FcKxsRKrcQqwhFUpxvPs4x7uPN2x3XIeR3AiXFy43tJtcmb/CZHH1ReLnTJMfmiY/bDlecJLd1VF2T3yN3cM2xW9Xma8GKBs7yCX243YeJDpwlO5999IzsFeCt7hjNbQJ3SKO4y5X3MsOdrUuiPuP7XJdaG+ouHv72v562pkisp0DtkwRad92nWSwsLDA2bNnOXPmDFevrn497c6dO2thO5lMtvXac+Nj3oztF77J1PXLxHoL3gqSexYIRFf7E1GcbPZtpFMnSKVPkEwcwzC217guWKpyL1K+tkjFby1x821MLKmvci/1ct9ERWK7/nvajuS9as/t/D7lKjmuLlxtDN/zV7i6cIWSU77h14u7LrtWjBfsqRhY9FGK7mPaiZIe2I+V7Cac6iHe2U+ys49UR88NzfW+093O/6Y2k7xP7dvQKSJC3ErJZJKHH36Yhx9+mMXFxVrP9sqwfe3aNa5du8aXvvQlduzYUQvbqdTqV+6ne3p5+EM/x8Mf+rnlBW2e/ybDz48Q7/XaSFJ7FleE7RxTU88xNfUcAEoFSSaPkUqd8EJ36n6Cwa2/aMmrcqcJ7U0DUuUWYivFg3GOZo9yNHu0YburXcbz403B+8qCN15wNTnD4HQoxOnQyl/uy/TZP6LXtsnOfpPOKYes49Ble7cZ2yXoRjF1klKgg3Kwg2q4Ex3LYsa7CSS7iXT0kejoJ93VRzSWlKq4ELfAmgFbay3/14lNlUgkeOihh3jooYdYXFzk3LlztbBd/9eW69evc/36db785S8zODhYu0AynU6v+trZHbvIfuSXePTnfpHxSxe9SSTf+RZDz08R7yuQ3utNIwlEGivbWleYn3+F+flXuMYfAhCJ7K5VuFOp+4lF96HU1v7vopTC6oxgdUaI3lffy71OlVuDPV7AHi+Qf9FbqVNFLII73nyVW4i3OkMZ9MX76Iv38Wj/ow37inaRawvXmsL35blLFJzVF9VZGi+4FqU1GXeSTmfMC985h855l+w1P5A7DmXHIW6b2DpJzspQDGQohzpxIllUPIuV6CaU6iHa0U+qs49UtnfbLb4jxHYlPzHFtpVIJHjwwQd58MEHyeVytTaSK1euNITtoaEhhoaG+PKXv8zAwABHjx7lyJEjq4ZtpRS9+w7Qu+8AT/zirzF87gznXvgmF777bYa+PU8oXSHWWyDeWyTaUyScrjS9RrF4hWLxCqNjfwWAZaVIpe4nnbqfVOoEyeQ9mObay8VvhtWq3MuBe4HqWL65yl1cpcrtTysJ7kzKAE8h3qSIFVl3vGCt4j1/mTdm32Ck0N6iOlopZkyTGdPk9eZpgw1CrkvWydHpzJN13iBbcchOOmTHHLKOS9Z2iDkOOA5zxJk30uStDKVghmo4ixvNYsS7CCS7Cad7SXT2kcwOkEx1SHVcvGVJwBa3hXg8Xgvb+Xy+FrYvX77cELaHh4cZHh7mb//2b+nv76+F7Uwm0/J1lWEweOQYg0eO8dSv/AbXz5zi+Wc/j5FfZOhbr+M6NlbYJtZbJNpT8G67Shhm4w84255nevrrTE9/3XtdZZFIHCXlB+506gShUPete4PaVF/ljtVVuSvXF2vTSirXFnBXziCtr3J/36ty7wkYTF46RXAgTmAgTnAggZkJyVK6QrxJ640XHMoN8dx3nmPw0CBTxSkmi5NMF6e9+4VJJvMTzFcX2l7dsmwYDBsGw4H1I0HKccg6NllnnKwzQtZxyM45ZKe9ynjScUk4DknXpapN5lSKBTNDIZChHOrACXdCrAtjqTqe6SWR7Sed7dsWK2QKsVEkYIvbTiwW44EHHuCBBx4gn89z7tw5zpw5w6VLlxrC9sjICCMjI3zlK1+hv7+/1rPd0dHR8nVNy2L3PfdxZWaeJ598kmqlzNjr5xk6d5qhs6cZ/eE5RssllOkSyZaI9RaJ9xSI9hZbtJXYLCy8ysLCq1y//gkAwuFBv4fbayuJxw9u2WjAekbQJLwvTXhfGvCr3NOl5RGBV/0q94qf1WZV1eZy114ravlhW0K3ELdCwAywJ7WHA+EDPLnnyVWPq7pVZkuzTBWn1vyYLExSckptf/5502TeNHljneMsrb3w7Thk7XmyzgxZ53WyBYfsokN22KntD/nfWxZ1hHkjXWtXqYQ7caNZVMyrjofSPSQ6+klm+0l1dGOYW//9U4jVSMAWt7VYLMaJEyc4ceIEhUKhIWy77nLfw1LYfu655+jr6+PIkSMcPXp01bAN3qI2S8u1Azi2zcSVNxg+e5qhc6cZPneGyVcXAU0wWSXeW6hVuiMdzW0lpdIQY6Uhxsb/OwCmGSeVus+vcN9PMnkvlrX6IjubRSmFlY1gZSPE7u8BwC07VIb8CvfVRSrXW1S5AbdgS+gWYhsIGAG6o910R9f/y1mhWmCyONkQvKeL07VtS9Xx6dI0bptjTG2lGLMsxiwL1mnbTjjuchh3KnQ6I3Q518lWXLJFh45xr2c87bosNZw4WjGlUiwaafKBNKVgJ3akE6JdGIkugskewpke8rMTlAo5wtGt/94q3lokYIs7RjQa5f777+f++++nUChw/vx5zpw5wxtvvNEQtkdHRxkdHeWrX/0qvb29tbDd2bn2ZBDTsujbf4i+/Yd44Cd+Bu26TA9dY+jcGYbOvsbwudPMXJj2jg05xPyWklhPkWhXESPQWAJ2nBwzM99iZuZb/haDRPwuv63kftLpBwiH+zf0PbpZRqi5yv3833yD+wePesu+D+eoDOXQJQndQtxuooEouwK72JXcteZxjuswW55dbkdZEcDrW1Vy1Vzbn3/RNFg0DS7TvEhPPVNrOh2HTselyw/knc4sWXuKLuc82XmH7IxLp+MQ9f+aeRjg1d9kgShzRoZFq4NSKEs10oWOdWOl+gil+4h1DpDqGiCT7ZMxh2JDyL8icUeKRqPcd9993HfffRSLRc6fP8/p06ebwvbY2BhjY2N87Wtfo6enhyNHjpDL5XBdF2Odi3OUYZDduZvszt0cf8+Po7VmfmKc4XOna4F79Psj/rGaSGfJD9xe8K5fzt3jspg7zWLuNEPDfwJAKNTrXzzptZbE43dhGFv/v61SCjsK0Xu64J4uoG5MoB+4q28qdCcIDsQldAuxjZiGWesNP8ShNY8t2sVa2F7rY7o4ja2bv0e04ijFhGUxYcHZdY6Nua4fwL1AnrUdss4inc4cWft1snMundPe/vpYv1QZnzM7yQc6KIe7cKJdqEQPgVQvkY4BEtkBMt2DxOIpuYhTrGrrf1ILcYtFIhGOHz/O8ePHKRaLXLhwoRa2HWe5d3p8fJzxcW/J8VOnTtHf38/g4CADAwMMDg4Sj6/9J0alFOmeXtI9vRx94p0A5OdmGTq7HLgnX7vC5KkOQBNMVIn1FIn5rSXhjjIrs2S5PMbExLNMTDwLgGFESCXvJZX2LpxMJu8jEGhv0Z1brWFMoIRuId7SIlaEwcQgg4nBNY9ztct8eX7NAL5UKV+oLLT9+fOGQd4wuBpYuyoOkHT8aSm1QO7Q6UySdcbIlh06Cw7ZcYeM4zaEpoIOMWukWbQ6KQSzVCNZ3Fg3ZrKXYLqPWEc/qe4dZLr6ZbzhW5AEbPGWEolEuPfee7n33nsplUq1NpKLFy82hO1yuczly5e5fPlybVsqlWoI3H19fQTW+eYdS2c49MjbOfTI2wEo5XOMnD/L0NnXGDp3mvE3Xmf2ordQjhFw/MDtVbmjPUXMFW0lrltkdu67zM5919+iiMUO1BbASaVOEIns3DYB9NaG7kQtfEvoFuL2ZCiDTDhDJpzhQObAmsdWnEpD4F6tV3yyOEnVXX9V2yULpsmCaXJpnRYVb7a4Wwvh3gjDMp3OdTqdK3TmXK9N5XpjvzjALEnmjAy5QAflUBY72g2JHqxkL+FMH4nOAdLdgyQzXVIVv0NIwBZvWeFwuCFsX7hwgbNnz3Lx4kWq1eZvzvPz88zPz3P69GkADMOgp6enFrgHBgbo7Oxcs7UkHIuz9/4H2Xu/N3qrWi4x+vqFWoV75PVzLA75yyorTaSj7FW4/Up3MLEyhGry+Qvk8xcYHvkkAMFgtjapJJ06QSJxFMNYZxDuJpLQLYS4GUEzWFu0Zy1aaxYqC0wVp/jqd7/KjkM7GgL5VMm7nS5OM1OawdHOmq9Xe9362eLrHGtqTYffntJZ6xef86epnKfTH23Y6bgkXbe2bHZFW8yoDAtWhnwwSyXsVcWNRA/BdD+Rjj6S2UE6ugfkws1tTgK2EHhh+5577uGee+7h61//OsePH2d4eJihoSGGh4cZHR3FthvDnuu6tQsmX3rppdrrrGwticVWn+0aCIXZeewedh6rm1Ry+Y1ahXv43GmmToeZOu0fH6s29HFHOkusXECyUplicvLLTE5+GQDDCJJI3LO88mTyPoLB1aenbAUJ3UKIjaKUIhVKkQqluB6+vuY4Q8d1mCvPMV2argXw+qkpS8F8pjTDbGm27dnijlJMWhaTbaSsQO3izeVWlQ5nnKw9QmfJIZt3yY56+2Na18L48oWbncsXbsZ7sJI9hNJ9xLODtQs3ZaTh5pOALcQKSikymQyZTIZjx44B4DgO4+PjtcA9NDTE9PR003NLpRKXLl3i0qVLtW3pdJrBwcFa6O7t7V21tcS0LPoOHKLvwCEe/MkPo12XqaFrXoXbHw8498YMc294fdeG5RLtLi73cvcUMUONY7Rct8L8/EvMz78E17xt0eje2gI4qdT9RKN7t13wlNAthLjVTMOkM9JJZ6STg5mDax5ru3bDbPGmUF5arpLfSL94tX6k4TrCrruiKl4g6+TIOpfozDl0zjt0XvOCesSfpGJrg2mVYt7sWL5wM9aNivcQSPUR6egjNz1FITdPNJ5q+7zF2iRgC9EG0zTp7++nv395bF6xWKytHLkUvAuFQtNz5+bmmJub47XXXgO81pLe3t6GKndHR0fLkKcMg66du+nauZv73vtBb1LJ+FhDhXtuZJTcyFKVXBPOlL0qt1/pDqWa210KhUsUCpcYHf1LAAKBjNfDnbyfVPoEycTdmGZ4A965jSWhWwixVSzDoivaRVe0a91jK06FmdJMY2tKXTCfLk7XAnq+mm/7HEo3sOpmzK0P4i6dzgxZZ5JO54w3znDSqV3YeQTg1D9kUUeYNTtYtLIUw13Y0R5I9BJI9xPpGCDZtYOO3h0SxNsgAVuImxSJRNi/fz/79+8HvKA3Ozvb1FpSf/EkeK0lSwvf1L/WwMBAQz93NBpt+pxKKdK9faR7+zj21LsByM1M18L20NnTTF2/Smk2zPRZb3l4K2LX9XEXiWSLGCv+WlitzjI19VWmpr7qf54AicTR5ZUn0ycIBbMb9t5tpLZC99AileH8zYfuwThmWkK3EKI9QTNIb6yX3ljvusfWjzSsD961UF7XM34jq24uTVK51sYklYTj0u3YZB2XbrtE1rlKt3OJ7KJD96xD12UvjC9VxReIMmt0sBjIUgp1eRdtJvtqQTzVvYOOnp1EYom2z/dOIwFbiA2ilKKjo4OOjg7uvvtuAGzbbmotmZmZaXpusVjk4sWLXLx4sbato6OjIXD39vZitfgTYryjk8OPPs7hRx/3Xiu3yMj5MwydPc3w2dOMX77I/GWL+cteW4kyXaJdpYaFcKympd6rLCz8kIWFH8L1PwIgEt5JKn1/rbVEt7mi21a41aE7OOgtkCOhWwjxZrU70lBrTb6abw7gfo/4ytYV221vvjgsLfYT5I11jkv4i/x4H0W67Ct0OW/QtejQNevQdckL4tFaEI/5QbyTYrgbO9qNSvQRSA8Q7ewn2bWTzt4dd+QFmxKwhbiFLMuqVaaXFAqFptaSYrHY9NyZmRlmZmY4deoU4LWprGwtyWQyTQEvEk+w78TD7DvxMADVUomR188x7Fe5Ry6cJz9mkB+LwqsAmlCq4reVeJXucKZ5qfdi6RrFsWuMjX126TPx0st3EYvuJxbzPqLR/YTDfaiVV15uA+uG7qEc1WEJ3UKI7UkpRTwYJx6Mr7vq5tIklVYXbL6ZSSpLq26uN9Iw7nqL+3Q7DlmnQLe9SNZ5g+4Fh+ysQ/aSF9Lrg/jMUkU83IUd7UUlewmm+4l2DJDs3klH7w7CkdWHBmw3ErCF2GTRaJQDBw5w4IA381VrzczMTEOVe2xsrGHFSfAutFwK5vWvtbK1JBKJNDwvEA6z6+7j7Lr7uPc6dpXxS2/URgMOnz9DeT5PeT7EzPk0AGbY9lpK/Isno10lDGvl1fNF5udfYX7+lYatphklGt3rhe668B0O79gWq1DWaz9059Cl5h9Aa4buwURtKfg2Bw8IIcSGqJ+ksje9d81jXe3yxa99kYP3HWSiMMFUcYqJwgSTxUkmC96M8YnixA1VxXOGQS5ocGWdIB5zXbpsL2xnnQLdzmW67It0LXgV8Y43vH0xP4jPE2PW6FwO4rEeVKKPYGY5iHf27iAUbm6x3Gzb66edEG9BSik6Ozvp7Ozk3nvvBaBarTI2NtZQ5Z6dnW16bqFQ4PXXX+f115ensnZ2djYE7p6enobWEtMK0H/wMP0HD8NP/aw3qeT6Vf/CyTMMn32N/NwsC1cTLFz1+ueUoYlkSw293IFo62+0jlNgcfE1FhdfW/F1BolGdzcF72h0N4axfVY5WzN0Dy1dSHljoXuvaTD+o1ewshGsjjBWZwSz079NBlGGVLyFEFvDUAYJM8GhjkMc6ji06nGudpkrzzFZmKyF74bbuvvtBvG8YZBvI4hHXb81xXbocvJ0OQt02a/TNe/QNeOQueTtWxpjOEfcC+LBLOWQF8SNZB+BdB/RzkG/R/zWBnEJ2EJsQ4FAgB07drBjx47atnw+3xC4h4eHKZWaL3iZnp5menqaH/3oR4DXWtLX19fQWpJOp2ttDMow6Nq1h65de7jvfT+B1pq5sRHvwsmzZxg69xrz42MUJiIUJiJMAqAJxGxC6TLhTIVwpux/VLDCrf/UqHWltihOI4NIZGctcHvhex/R6D4sa3v8ObAhdN9746HbcBTV0TzV0RbTAixVC91WR9gL4f59MxNCmduv3UYI8dZjKIOOcAcd4Q4OsXoQ11p7QbxVCF9x2+6KmwXD4KphcHWdCzYjDUE8R5czT5dzoRbE0/6+uB/EZ0kwtxTEw8tBPJjuI9I5SLp7Jx09OwiGbnyqlgRsIW4TsViMgwcPcvCgN6vVdd2m1pLx8fGWrSVDQ0MMDQ01vNbK1pJw2PsGopQi0zdApm+Au596DwCLM1P+HG6vwj11/SrVfIBqPkBuuOHTYYZtL3Snl0N3KFMmGFutouFSLF6hWLzC1NRzDXvCoX6vyl0L3/uIxfYTCKRv/o3cIKuG7un6kYGr93TX2Bp7oog90dyHjwFmOozlV7sbbjvCqIAsHiGE2F6UUmTCGTLhzJqzxbXWzJfnV62C199W3ObrglopGgbX2picEnG9BX28ML5IlzNHl3OerjmHrmmHlD/eMOEuBfEks0YHueD6IxqXSMAW4jZlGAbZbJZsNsvx48cBr7VkdHS0odI9NzfX9Nx8Ps+FCxe4cGG5mpzNZhuq3N3d3Zj+6l+JjiyHH3uCw489AUBxcYGvfPYzDGY7mRm+zszIEDMjQ+RnZ3BKFvlRi/xo45/ejKBDOL1c7Q7594OJKqtdE1gqj1AqjzA9882G7cFgllh0ZfA+QDCY3dILDJVSXgU62xi6v/WVb/Dw4fuxp0s400Xs6RK2f+vm16jguODMlHBmSg0tJ0vMVBCzI9IygBth+fYuhNi+lFKkw2nS4TQHMgdWPW7pgs31QvhkcZKyU27rcxcNg+uGwfV1gnjYD+LdjkPWXqTbmWv765PvwELcQQKBADt37mTnzp21bblcrqm1pFxu/iY0NTXF1NQUP/zhDwFvAkp/f39DlTuVSqGUIpJIktyxm/uffLLhNcqFvBe2h4eWb4evMzc+iluh1mZST1ku4ZRX5Q5nyrUQHkpVmpaBX1KpTFGpTDE7992G7ZaVbOjxjsb2EYse2NLJJkop3CCEdiYJ7Uw27XdLdkPgXrp1Zoo482tXbZz5Cs58hcrl+aZ9RizQELjNusq3EQvIpBMhxG2h/oLN/Zn9qx6ntWaxurhuW8pkYbLteeIlw2DIMBhqY5b4ShKwhbjDxeNxDh06xKFDXs+c67pMT083tZZo3TjqwrZtrl27xrVr1xpeaylwz87Osri4SDwer4W1UDRG3/5D9O1v7M9zbJv5ibHG4D1ynZnhIcqFPMXpMMXpxh43ZWiCybr+7qXgna60mGiydM4LLSebGEaEWGxf3cWVXsV7O0w2McIWQX8FyZV01cGeKbUO4HMlWGMUuZuvUslXqVxbbNqnQmbDBZdeEPfuGwm56FIIcftRSpEMJkkGk+xL71v1OK01uWquFrjrJ6fU304WJynaLVr32iQBW4i3GMMw6Orqoquri/vuuw+ASqXS1FoyP99cFc3lcpw/f57z588D8OqrrxIMBmtTUOo/Ojo6aiMDTcuio3+Qjv7GhRS01hTm55bbTJYC+MgQC5MTlOdClOdCzF+ue5LSBBNVv8fbr3z7981g68TpusU2J5t4wXu7TDZRAZNAT4xAT/PFntpxcWbLXuCeKWFP+bf+Y+zVZwPqskPVX1a+iWUsV747wljZMJbfhmKmwyhTwrcQ4vallCIRTJAIJtYcYbi0sM9EcYKpwlTt9lf51bY+z6YGbKXU+4D/DJjAx7TW/27F/p8C/he8uowN/I7W+tubeY5CvBUFg0F27drFrl3LixcsLi42tZZUKs0tC0vhfHR0tGlfLBZrGb4zmQyBgNemEEtniKUz7Dh6T8Nzq6USs2MjTA9frwXv2eHrzIwOU1lQVBaCLFyrf8Yqk03SlaaVKmvPuJ0nm5hGrd97Je1qnIUK9nQRp1b1XqqAl9CVNRaVsF3s8QL2eKF5n6GwMqHldpP620wYFZCJJ0KIO0P9wj57U8tBfNsFbKWUCfxX4N3AEPCiUupzWuszdYd9Ffic1lorpe4BngEOb9Y5CiGWJRIJDh8+zOHD3v+CrusyNTVVC9wXLlygUqm07Odeks/nyefzDW0mS9LpNB0dHU3hO51OYxgGgXCY7t176d7dWGFwXYfFqcla6K4P4MWF+TUmm5TrLrK8MyebLFGGwkqHsNIhWPGXUq01bq7aVPVeCuJuYY2JJ66uhfSm/+oKzGTIC9zZCOaK9hMjJH8wFUK8dWzmd7yHgIta60sASqm/AH4KqAVsrXX93ytjyPpnQmwbhmHQ3d1Nd3c3999/PydPnuSJJ56gUCjUZm+v/HCc1Sulc3NzzM3NcenSpYbtpmmSyWRaVr7j8TiGYZLq7iXV3cue+x5oeG5xcYGZkeFaf7fXdnKd+fFxf7JJY/V5syabbCdKKcxEEDMRJLSrxUWXRbup33up7cRdWOOiSw3OfBlnvkz5UouLLuOBhhGDS/O+zY4wRtSSiy6FEHeUzQzYA8D1usdDwMMrD1JKfQj4t0A38IHNOTUhxM1QShGLxYjFYg2TS8CreC8sLLQM3nNzc00XVS5xHKc20WSl1fq9Ozs7CYfDRBJJBg4lGTh0V8Pz7GqVubGR5gknI0MUJortTzZJVlCrjJ5ea7KJ43Zy6tSnCUcGCIcHiYQHCPsfltV8ceNWMiKWt8z7YKJpn1txcGaaL7i0p4s4c+U1SyJurkolV6VydaFpnwpbWJ1hehzFfPGyV/3uCGNlwpjpEMqS1hMhxO1FrfZDbsM/kVJ/B3iv1vqj/uNfAh7SWv+jVY5/HPhXWut3tdj3G8BvAHR1dZ145plnbt2J30FyuRzx+Pb6Yb4dyfvUnjfzPrmuS7FYpFgsUigUGm5b9Xm3IxAIEI1GiUQiDbfhcLg2z3slrTXVfI7S3Ayl2Wn/dobS3AzVfPMFgDcz2WR9MSALdKLoBNWJIru8Td26pXw3lAuBIgQKECioFbeg9M1VqDUaOwx2BKoRTTXaeN8JAW/B4rd8n2qPvE/tkfepfU899dTLWusH1jtuMwP2I8C/1lq/13/8LwC01v92jedcBh7UWjeXsnyHDh3SSxMNxNpOnjzJkyvmFotm8j6151a9T6VSiZmZmZaV77X6vdeSTqdbVr1TqRSG0bo6Wi4UmB0dbppwMjs6guus6FNeMdnEC91rTzZpl2Ul/Gr3IOHwgF/9HiQc7icSGcSy0tu+vUK7Gme+3HChZX3ft66+iffIVF6lOxPyl5evq37fwe0n8n2qPfI+tUfep/YppdoK2JvZIvIicEAptQcYBn4e+IX6A5RS+4E3/Isc7weCwPQmnqMQYouFw2H6+/vp7+9v2K61Jp/PtwzeMzMzbfV7v/HGGw3bTdNseaFlZ2cnsViM3n0H6N3XuMKY6zjeTO+60O1daHmdhWv5lpNNgomq9xGv1u4H/PuGuXaRw7YXyeXOkcuda7nfNGOEw/2NATyyfD8Q6NzygKkMLwRbmTCsWCeidtHldJHXnv8h+7t348yWsGdKOLMlnIXK2lfjONq7WHOq2HzhJf7Mbz9sW5mQf+uFcLMjjBGU5eaFEBtv0wK21tpWSv0W8GW8MX0f11qfVkr9pr//94EPA7+slKoCReAjerNK7EKIbU0pRTweJx6PN4wTBK/lZH5+ftV+79U4jsPk5CSTk5NN+0KhUMvZ3p2dnWT6Bsj0DbDvxPJlJFprigvzdbO8vQstp4eHWBifID/W6luZxoraTeG7/vF6rSeOkyeff518/vWW+w0j7FfA+4mEBxuq3+HwAMFg15atcgmNF10uXtGknmz8b6ttF3uu7PV+z5Zqt0sB3M2vMfUEf+b3WJ7qWL7lfiMWqIXvpgq49H8LIW7Sps5N0lo/Czy7Ytvv193/XeB3N/OchBC3P8MwyGQyZDIZ9u9vLJFWq1VmZ2dbVr1zuRYLrfjK5TIjIyOMjIw07VttvndHRweDR44xeORYw/Ff++pXOX7XYebGR5gbG/U/RpgbH2N+YoxCIUBhvNVZaKyw41W8W4TvUKKKEVi7vcJ1SxQKb1AovNFyv1JBP3wvX3gZjgz6YbyfUKgHtdqVnZtAWQaBbIRAi3nfAG7Z9hbcmfFD94oArivrvD/5Km6+SvV684qXS6MHzY5QQ9V7KYDLqpdCiNXIYFIhxB0tEAjUxguutFq/99TU1JoXW64231spRSqVagre5UqFVE8vHf0DTa/lOg6L05PMLgXv8eUAPj8+hl2qYJcsmGwVMDVmyGnZghJM2oQSNkZgnQqvrtRmfreilEUo1NfUerLUFx4K9W7pcvNGyMLotQj0tljtUmvcfHU5gNdVwJ2ZEvZcGZw1/kJQN3qwcrl5+gmWwkqHW1bAzcyd2/8thFifBGwhxFvWRvd7a61X7fd+8cUXyWQydHR01G6XPtIdWVLdvXDPfY2v57rk5mYaq95jo8yNjzE3PkKlWMQpWxTLFsWp1hVeM+jU+r2XPkJJm3BaE4iXMay1p7ZobVMqXadUug5zzfuVMgkFe/zw3e+PIRys6wvvwzCCa36OW0UphRkPYsaDBHc0jx1cWvHS8avfKwO4s7hO/7ct/d9CiNYkYAshxAq3ot/bdd3aca0+XyqVagjdSx+ZTIYdHVl2HLm74TlLPd+zde0m3q0Xxks5r+XBqZg4MyalmXDL8zICzor+b5top0E47WBGiiizuOZ7pbVTW3ynNUUo1FNrP6mvfkcig4RC/ZhmaM3PcavUr3gZ2ptq2q9t1wvbq1TA11z1kjb6v+OBugAebmxFSYdQpvR/C3G7koAthBA34Gb6vcfGxtZsOamvfK9c2RIgmUw2he6l+wOH7mpaWAeglMv5YXtkufXED9/5udnacW7VpDRrUpptHcCV5S4H8HiVeLdFtEN5F2CG8mijRe9y41dHuTxGuTzG/PzLLY8IBrsIhwdw3QAXL35vuRc83L+li/EoyyDQFSXQ1XoWuVuysWfLtQr40vSTpQC+3vjBpcV3WK3/OxWqu+gy1ND/LescC7G9ScAWQogNslq/98mTJ3nkkUeYnZ1lZmam6WNhoUV/b52FhQUWFha4cuVK075YLNay8t3R0dFyzCBApVRs7Peu9X2PsjjdOFFF2wbluRDlOa/KPH22fm8GZbr+KMIKie4A8e4A4YwmEC1DYAGXOdZLg5XKJJWK93mvXnuxab9lperaTgaaAnggkNmSXmcjbBHss6Bv9f7v5eBdbgzgs2Vw1+n/nivjzJWpXG5een4fBiPPfxczEcTwp7CYiQBGvO6+v12FTOkFF2KTScAWQohNEAqF6O3tpbe3t2lftVplbm6uZfhea1l5WL7g8vr16037IpHIquG7a9ceunfvbXqOXakwPzFem3gyOzbKvB/A5yfH0W5jVVY7BpWFIJWFILnh+j1RIIoyegjEqsS7LJL9EWJZg1DKwQwX0eYctjMNrD7DHMC251nMzbOYO91yv2lGCYX6iUT84B3yw7f/OBTs3vRRhPX93+xMNu33+r+Xqt9l7JliQyuKu7B2b7xC4eaquLkqjLZuQakdGzC8sB1fDt0N95dCejwgYwmF2CASsIUQYosFAgG6urro6upq2mfbNvPz8y3D9+zsLK67ehtCsVhkeHiY4eHhpn3BYHDV8N0xMEjn4I6m5zi2zcLURNO0k7mxUeYnxnDs5p5k7Soqi0FmFmHmUv2lgCbQCaqDaMYkszMBsTK9u5IEElWMYB5XzVKpjuG6a4dNxylQKFykULjYcr9SAcKhvlrgrq9+R8ID/iSUzb0Q0+v/DmOlw4Saf89BV13sufqxg+WGBXjW6/9e+VqOP8JwPUbU8qvgfgD3K+JG0gvgS2HciFgyolCINUjAFkKIbcyyrNq4v5WWLrhcCtsrA7jdIvAuqVQqjI2NMTY21vJzrha+U929ZHr7m57jug65melaq8nsUvAeH2V2fBR7tWXutaIw41KY8doghp+vD4ExgpGjdOzoJL0jRrwrQDjtEoiW0dY8VXuCUmkEx1l9njmA1lWKpWsUS9dWOWLpQszlFTFXzgY3zdZTWm4VFVi7//vk107y9hOP4C5WvEkouQruYhVnseJtyy3fv5Gl6N2CjVuwsSfWOdBQLargfgD3A3otjMu0FPEWJAFbCCFuU/UXXK7kui65XK5l5XtmZmbNiy5t22ZiYoKJieaUZZpm7XOuDN/pTCfJbDc7j93b8BytNfm52bppJ6MNE0/KhdVbHCrFEmMXhhm70LwvFEuS6T1EeqCTVF+EaKdBMFHFDBWoupOUSsOUSiNUqzNrvIvQeCHmKy2PCAQ66qrezf3glpXc3D5nA6xUCFJrT2DRWqMrDs6CH8BzFT94+wHcf+wsVnFz64wlrLc04nChQnWdQ1XQxEwGMfwKeC2Mx1f0j8eCKFOq4uLOIAFbCCHuQIZhkEwmSSaT7N69u2Hf0pzvVi0nMzMzFIurj+ZzHIepqSmmpqaa9imlSKfTLSvf6XSawbuOMXhX4yqXWmtKuUVmR0f4zsmv0ZtOMTc2wuzoCHNjI2uG73I+z9gbFxl7o7k1JJxIkundT6bvCdK9WeI9ASJpCMTKdeHbC+Dl8jjrJctqdYZqdYbFxdda7jfNeEPoblgZMzxAMJjdkgsNlVKokIXRZUFzB1ID7foL8yxWcP0KeH1F3F2s1Lbp0tp98w2vW3Gwp4owtfbIR5S/dH28rhpea1NprJarsFy4KbY3CdhCCPEWUz/ne+fOnU37C4XCqhNP8vnVA6/WmtnZWWZnZ5sW2gEaZn2vrID3HzxM58gYjz35ZMPrFRfmmR0d8VtORpbvj45QLa/eU1xaXGB0cYHR18837Yum0mT6+kn3Pkamt5+uvh5iWYtgoortTNWFby+Al8qjaL12ndZxcuTzF8jnW5TaAcMIehdiNvSAD9YCeCjUs6UrYoLXF74UYNejqw6OXxF3/Qr4ckW8MYyvuVpmw4uyfOFmc+dSI0st94cnGltS6gO6av/3ACE2lARsIYQQDaLRKNFolIGB5qXdy+Xyqm0ni4trz8Sen59nfn6ey5cvN+2Lx+MYhsHExASJRIJ4PE4ikfDuZzrZu3M3kUikVrWstZ34gXt2dLhW9Z4bG8Wurt4CU5ifozA/x/C5M83n0dFJprefdN8OMr0P09XXT3qwh2jGoup4Pd/1AbzoP3bdtauzrrvekvT+ipj1AbxuafqtXJCnFRUwsTpMrI7W89OXaK3RRbuhJ7wxmC9vcwvV9ltUbF0bY7iWfZgMf+s7dSMMAw1TUxou5IwFpEVFbBgJ2EIIIdoWCoXo6+ujr6+vaV+lUll13OD8/Pya4wZzOe9CxTNnmkPvEtM0G4J3LYhnuujfuZeD/r5IOExudqax4u3fnx8fbTntpHYeM9PkZqa5fuZU4w6lSGa7SPf2k+ntJ9P3AOnen2Tnzn6SXd24OkepNFQL4MX6CnhpGNtunmVdr2FFzPnmWeAAwWC2FsBdt8zlK6cJBrMEA53ebbCTYLAT02x9YeRWUEqhogGMaIBA99rnpR3Xb1Gpv1hzuV+8Pozryg20qJRs7JINk220qEQDDRNUloL4yos4ZYqKWI8EbCGEEBsiGAy2XGgHvAsn15r1vda4wSWO49Sq4GsxDGNFBbyHzM797EwkiMWiGLaNnVugODvNfG3iyQjzE+O4zirBTWsWJidYmJzg2qkfNuxShkGyq9sP3gNeCO87Tk+vF74N08S2F1uE7+UAvrTQzloqlSkqlSkWFl4F4NKlr7Y8zjSjBAKdfuDOEgx01O4Hatu8/d4iPdtj9rUyDcxkCDO5fqXeLTtNF2g6C5Wm/nFnsYzSbQZhDW6+ipuvwlhh7WNXTFGpXcDZYpss9PPWJAFbCCHELWdZFtlslmw227RvKTh/4xvfYN++fSwuLpLL5VhcXGy4v9bkk3qu69ZWv1zLUi96oqOf+M6DZGMxAqaBqlZwC3kqi3MUpybJjY+wODmJ1q1/CdCuy/z4GPPjY1x5tXEKiWFapLp7/J7vper33fT0vZfE3izKMPz3oEy5PLIifC8H8HJ5DK3bq9o6TgHHKVAqNS8+1MwgGOyoVcEDfhU8GMguB/RgZy2wm+baLSGbxQiZGKEIVufa4xNPfv0k73josVoYr1XDc8tTVW6qReVGpqjUL/SzZptKABWQkYZ3CgnYQgghtpRpmrULH++5555VjyuXyy2D98r75dVmbq+gta49Z42zQ2V3Ett5mEg4TNA0MFwHyiXs3AKl2RnKs9Mou4KybdSKhOY6tt8f3rzYjxkIkO7p8yveXvj27h+mv7ejFr4BXNemUpmgWByiVB7h7NnvsXNHmkplmkp12q9uT1OpTKN1e7+I+K9cq4yTb74gtOmczbgXyP0qeCBY356yXBkPBrP+6MItro4rMGMBzFiAQE/zkvb1tLM8RaUWvBump1RrF3Hq0q1Z6EeFzYaLNOvbVBpHGgZQ5vb4y4NoTQK2EEKI20IoFCIUCrVcdKdepVKpBe61Avla4wjraa3J5fPkmiaoWJDu9j58QcvEApTtVcGdQg5lV1F2FaPq3SqnitIap1pleuga00PNC+BYwRDp3r7l4N23VP0+QLrnQc6fS7N//5Mtz9Vxcg2Buz6AVyv+fX+bba9d5V/JcXIUizmKxdUW7VmmlEUg0LEcwGvhu65aXusf78AwtvYiTmUqzGQQM9nOFBV3eaZ4rsUCPze50I8uOdilNkYaAkZsadXNxpaUhr7xuNf/Lv3im08CthBCiDtK/TLwa6lWq+RyuYbg3SqIFwrr9OPWqdgOXv3YhGjS+2hBOQ6qWl4O3/7t0odrV5i8fpWpa1eanhsIRzBCYUa/9izRZJJIMkU0ma7d9x6niCb3kOi4F9MKrHq+rluhUp2hUpnyw3d9AJ/2t8/Utml9A5Vb7VXdK5X1loX0WFbCC94N/eN1bSrBTkL+fstKbGlfswoYWJkwVqaNKSoVZ5VFfpqr5W2PNATcvI2bt7HH1+sXZzmI19pU6qvkAQJ5cCuOrLq5gSRgCyGEeEsKBAKrroRZz7btpiDe6v5aM8JX0qaJbmfah2M3hG+jWqFiV1GOTf76da8abtsox2a1uBmKxogkk0STaT98J+tCeKoulB8llU1hBVoHcq01tr2wXB2vTtcF88ZQXqlMr7uE/Uq2vYhtLwLNYxxXUirot6p0rugfb2xT0XoG285jmtGtXegnZGFl1+4Xr400XNGO0jjS0H+cv5F+cXAXKrhr9IvvwmTkWy9gxCzMdBgzFcJKhzDrPqx0CCMelGp4myRgCyGEEGuwLIt0Ok06nV7zOMdxyOfz6/aI5/P5NUcWNjAtXNOC0NrhDK29kG3bdaHbu604VXK5Isb8Qm07rrtqIA9Gon7wbq6OL4fyDiLJPXR2p7CCrVsqHKdMdUV/eGMon6ZS9VtXqjNtX8TpfbmV2vL26/nGN//vKGViWcnaR6B2P4EVSNU9TmIFko2PreSmXNzZMNKwZ+1jtaNxCy1aU+r7xpf6xYvt/9XBq4rnqA6v8suRqTBToVUDuJkOYYQkWoIEbCGEEGJDmKZZW55+La7r1oL4elXxtoO4UmgrgLYCwDph3DuJpiBu+JXxql2lkMuj5uaWQ/sa5dJAOEI0lSKaqAvlqTTRxHIojyQHiaeOEsmmCASbe621dqlW55oDeENl3O8hr07jOO3/tcB7fYdqdZZqdfaGnrfEMIKtA3qgVWhfGdITGMb6fd03Qpk3sOqm7TbOE28xQSU/MU+gYqzfouLo2gWbq11Kq8LWquHbTIcwE6G3xII+ErCFEEKITWQYRm1G91pc16VQKLQM3leuXCEajZLP58nn821fsFl3EmgjhA60eWFhrVWluUJedaoUF3LMzs6ibBtcZ9XqOEAgFPZDuB++/WAe9YN5JJkkmjhIMpki2p0iEGquHjtOgUpdb3i1rjVl+aLOKfL5CQyjjOuuP8FjLa5bWZ62chMMI+IF7pWBvM2ArtTN90Yry8BKhyG9ehX+5MmTPPH4O3BzVey5Um2VTGeujD1Xxpkv48yVcPPrV8N1yaY6ZlMdW+WXIIU377xV+E6HsVJBVMS67WeHS8AWQgghtqGlBXPi8Ti9vb0N+06ePMmTTz5Ze+w4DoVCoRa46z9yuVzTNnuN1SxbqrWqtHGsdhuDuF8ZV7aN4VSx7SqlxUXmZme87etU6a1QqBbEo8mkH8JTRBL1oXyQTDJFpNcL5EvhbOl9ct1yrce7Wp3Hthew7QWq9oK33d9W9bfX9le9W63Xm3a9NtctUq4UKVfGb+r5phm/gUDu7w+ksKwEphlvK6wqo26Kys5Vvo6K44ftuvBdC+Bl7LkS2OtUwTXe8fNluLrKuQTNlgHc8kO4mQyirO09plACthBCCHGbM02zrao4eBfTVSqVlmG81UehUGi/VQVAGehAEB1osy3CsRsC+HIo9+7bjk15foG5mZk1L+ZcYgWCRPzqeNF2KJ5+xWtZWWpdSaaIpvqJJO+iM7N6D/kSrbUf0OeXA3jVC+ZVezmst9xmL/oBvf3+8pZvkZPzLhotj9zEsw0sK0HASmEFEi1bXVw9wfhEkVCwi1Com2Cwu2XfuRE0MbqiBLpaX6CrtT9LvEUA9+6XcBfX/2VFVxzsiQL2RIGWU+2VNxmldStK2OsFj25tFVwCthBCCPEWopSqzRRfb5QheK0qxWJx3ar40ke7K27WmBbatNBBWHdidO1iztbtKoZTxbFtKvPzLExPo7TLmWtrTyUJRiLehJWUf0Fn3W3tws6lYJ7Yi2HeWLuGN5s8XwvcVXuhVjH3quaLfkCfb6yq14L6WgshtcP1X2se1uiUee21P2t47I1N7CYU7CIYWrrtIhTsJlgXxOtHJiqlvFUp40EYbP3LnrbdusDdHMCd2fL6s8M1uIsVKosVuN76/VEBwwveqdVaUUKowK2rgkvAFkIIIcSqDMMgFosRi629EuKSarXadnU8n8/juu0vxNJ4MWcbXKflnPHlxxXKlQrl8VHmxkfb+vyReGK5V3xFIF8O6t79UCyGUgrLimNZ8fa/zjpaO9j2UkCvr6Iv+oF8RcW8Wr9t8YYvCF2y1FJTKLyx5nGGEfICdy2Idy8H8bpAHgx2oJTp9YR3rr7M/dK4QntlH/hSb/h8GWehsu6YQl11sSeL2JOrX59gxANe+E6trIT7VfDYzS/SIwFbCCGEEBsmEAi0NdYQvDBVKpXWrIjXf5RKN3ixomGigyZOcJ0xe9pdddGflY+LiwsUF9tbAdMwLW/EYUOLSpJIi2AeTaYIhJvPUymTQCBJIJAEBm/s6wdc18Zx6nvPm6voV6+eoTMboFKZpFyeoFKZarvv3HXLlEpDlEpDax7nfR2dhEJda1TGuwmFshjREMFoAPpb/1KiHRdnodIygC8Fc11evy3HzVVxc1WqQ2uMJVwRwNslAVsIIYQQW0IpRSQSIRKJ0NXVte7xtm2vejFnq4s7HafN3mdloAPeVJU16+kNLSpVDLuyehjXGtexyc3OkJudaes0vAs607UJK/W94yvbVSLJFKa1fowzDAvDyBAIrL6g0vXrJ7n3nifrvkx/bGItcE9QrkxRKU9QrkxQKU96t5VJHKe9lU61dupW9jy99vtgpQnVV79XVMRDoW6CiS6C6eSqfdZuyW7sA/dDeO3xQnn9niRH40yXcKZvfAqNBGwhhBBC3BYsy2pr1jh41fHnnnuO48ePN4w4bHVbLre8lK7ZihaVteK7chzUGgF8KZzXL/pjl8ssTI6zMNnetJFwLO5Xx5N1feMrq+P+ap3xBMpor+dYKcNfKbODePzQmsfads4P4pN+EJ9sCuLl8iS2PdfW5/Zecw7bniOff33N4wwjslwRD/m94cHu2rZgvItQRxfhQA9KNX7t2tXeXPBa9bvSOKJwvoxbuMFpO3UkYAshhBDijqOUIhAI0NXVtW51vFKpkMvl1gzhi4uLFArtVWsBtGmizci6q3AqrTEcG6plVLWyamtKqwkqpXyOUj7HbBvDRZRheKMNG6riXjCfHBvjQiRAOJYgHI8TjieIxBNYodC6kziW+suj0T1rHue6ZSqV6eWK+FIVvOyH8soklfIklepU21NXXLdIsXiNYvHa2l+7sggGs34V3G9LWaqMR7sJZbxwHgvuwjCW+/vdstMwgtCZK8PvtnVqErCFEEII8dYWDAbp6OhYd6qK4zhtBfEbuXhTK4VjBcAKrLkIpwJMvDCuKmXcUrFWIW8M461X3tSuS2F+jsL8XMvXv/aNrzRtMy2LcDzhf3jBuz6Ee49jdcd4+0KRaFO13DBChMP9hMP9a78f2qFSnW3ZjlKrkvu3rtveXx60timXxyiXx2DNoSyKQCCzoj+8m1Cki2Dau4CzXRKwhRBCCCHaYJomqVSKVCq15nH1q3CuF8bbXfRHAzYKzABEAhBZfSpJwFB+GHf8MF7ALRaae8fXmW/u2Db5uVnycze2xLxSBqF4nEg83hzIV96P1d+PY5gmoWCWUDBLgiOrvx9aY9uLfuD2A3itPaXutjJxA6MONdXqDNXqDOTP39DXvJIEbCGEEEKIDVS/CudatNaUy+V1e8QXFxfb7xMHqq6mCqBMCEW9jxa/E1imQdA0ccslQpYFjo2ulHErZexSAV2polzH+3AclGuD4z9eI5xr7VJaXKDU5rSVesFItCmIR2rV81ZBvYNkYhdWYPXRjY5TqusPX9Ga4t+WyxNesF5v/l+bJGALIYQQQmwBpRThcJhwOLxun3i1Wq0F7rXCeD7f/txr23GxHReUScnRgAnBqPcRX33qiHfuYCmFAbUAru2qH9Ar/jYb5TrePHJnOajXHrcIs5VigUqx0PaFnkusUMgL4ytCeCi23FPuhfI+wvGDpFLe9kAoXOszd12banV6uSJeVwVf2gaX2jufGzp7IYQQQgix6QKBQNt94vl8ft2qeC6Xu7FFflbQGqq1KrYBpuG1r4RaL6PeikJjaO2FcNtGL13M6TrLlXL/tvZ4ZTXdfy27XCZXLpObnrqhr8Mwraa2lUhD+8pewvF7SMUThDMJ4K/bel0J2EIIIYQQdwjTNNsaZei6LsVikVwux/PPP8/hw4cplUqUSiXK5XLDbattbyacL9EoHKXqwvnaE1dW+UJq1fJVK+VrVNMdx17z4s+bJQFbCCGEEOItxjAMYrEYsViMjo4OjhxZ/YLClbTWVKvVptC9ViBvtU2vc5Flm18I2jBqs8lvSl11nBWBvBbUl+63SQK2EEIIIYRom1KKYDBIMBgkkUjc1GtoralUKm8qpJfL5Y0J6abpzS1/869UIwFbCCGEEEJsKqUUoVCIUCjU1sqcrbiuuyEh/VaQgC2EEEIIIW47hmHUprCsN5t8Na7r1oJ2O4G8XRKwhRBCCCHEW5JhGEQiESKRm7jAcq3X3dBXE0IIIYQQ4i1OArYQQgghhBAbSAK2EEIIIYQQG0gCthBCCCGEEBtIArYQQgghhBAbSAK2EEIIIYQQG0gCthBCCCGEEBtIArYQQgghhBAbSAK2EEIIIYQQG0gCthBCCCGEEBtIArYQQgghhBAbSAK2EEIIIYQQG0gCthBCCCGEEBtIArYQQgghhBAbSAK2EEIIIYQQG0gCthBCCCGEEBtIArYQQgghhBAbSAK2EEIIIYQQG0gCthBCCCGEEBtIArYQQgghhBAbSAK2EEIIIYQQG0gCthBCCCGEEBtIArYQQgghhBAbaFMDtlLqfUqp80qpi0qpf95i/99VSv3I/3hBKXXvZp6fEEIIIYQQb9amBWyllAn8V+D9wBHgaaXUkRWHXQae0FrfA/wvwB9s1vkJIYQQQgixETazgv0QcFFrfUlrXQH+Avip+gO01i9orWf9h98FBjfx/IQQQgghhHjTlNZ6cz6RUj8LvE9r/VH/8S8BD2utf2uV4/8JcHjp+BX7fgP4DYCurq4TzzzzzK078TtILpcjHo9v9Wlse/I+tUfep/bJe9UeeZ/aJ+9Ve+R9ao+8T+176qmnXtZaP7DecdZmnIxPtdjWMt0rpZ4C/m/A21vt11r/AX77yKFDh/STTz65Qad4Zzt58iTyXq1P3qf2yPvUPnmv2iPvU/vkvWqPvE/tkfdp421mwB4CdtQ9HgRGVh6klLoH+Bjwfq319CadmxBCCCGEEBtiM3uwXwQOKKX2KKWCwM8Dn6s/QCm1E/gM8Eta6wubeG5CCCGEEEJsiE2rYGutbaXUbwFfBkzg41rr00qp3/T3/z7wr4BO4P+nlAKw2+lzEUIIIYQQYrvYzBYRtNbPAs+u2Pb7dfc/CjRd1CiEEEIIIcTtQlZyFEIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA0nAFkIIIYQQYgNJwBZCCCGEEGIDScAWQgghhBBiA21qwFZKvU8pdV4pdVEp9c9b7D+slPqOUqqslPonm3luQgghhBBCbARrsz6RUsoE/ivwbmAIeFEp9Tmt9Zm6w2aAfwz89GadlxBCCCGEEBtpMyvYDwEXtdaXtNYV4C+An6o/QGs9obV+Eahu4nkJIYQQQgixYTYzYA8A1+seD/nbhBBCCCGEuGNsWosIoFps0zf1Qkr9BvAb/sOyUuq1mz6rt5YsMLXVJ3EbkPepPfI+tU/eq/bI+9Q+ea/aI+9Te+R9at+hdg7azIA9BOyoezwIjNzMC2mt/wD4AwCl1Eta6wfe/Ond+eS9ao+8T+2R96l98l61R96n9sl71R55n9oj71P7lFIvtXPcZraIvAgcUErtUUoFgZ8HPreJn18IIYQQQohbbtMq2FprWyn1W8CXARP4uNb6tFLqN/39v6+U6gVeApKAq5T6HeCI1nphs85TCCGEEEKIN2MzW0TQWj8LPLti2+/X3R/Dax25EX+wAaf2ViHvVXvkfWqPvE/tk/eqPfI+tU/eq/bI+9QeeZ/a19Z7pbS+qesMhRBCCCGEEC3IUulCCCGEEEJsoNs6YK+39LrwKKU+rpSakHGGa1NK7VBKfV0pdVYpdVop9dtbfU7bkVIqrJT6vlLqVf99+jdbfU7bmVLKVEr9QCn1ha0+l+1MKXVFKXVKKfXDdq/SfytSSqWVUp9WSp3zv1c9stXntB0ppQ75/5aWPhb867rECkqp/9H/Xv6aUuqTSqnwVp/TdqSU+m3/PTrdzr+l27ZFxF96/QJ1S68DT69Yel0ASqnHgRzwx1rrY1t9PtuVUqoP6NNav6KUSgAvAz8t/6YaKaUUENNa55RSAeDbwG9rrb+7xae2LSml/ifgASCptf7gVp/PdqWUugI8oLWWWbxrUEr9X8C3tNYf8ydyRbXWc1t8WtuanxeGgYe11le3+ny2E6XUAN738CNa66JS6hngWa31f9vaM9telFLH8FYgfwioAF8C/oHW+vXVnnM7V7DXXXpdeLTW3wRmtvo8tjut9ajW+hX//iJwFllttIn25PyHAf/j9vxN/RZTSg0CHwA+ttXnIm5/Sqkk8DjwRwBa64qE67a8E3hDwvWqLCCilLKAKDe5Rskd7i7gu1rrgtbaBr4BfGitJ9zOAVuWXhe3jFJqN3Af8L0tPpVtyW97+CEwAXxFay3vU2v/O/BPAXeLz+N2oIG/VUq97K/WK5rtBSaBT/htRx9TSsW2+qRuAz8PfHKrT2I70loPA/8BuAaMAvNa67/d2rPall4DHldKdSqlosCP07h4YpPbOWBv2NLrQtRTSsWBvwJ+R2awt6a1drTWx/HGaj7k//lM1FFKfRCY0Fq/vNXncpt4TGt9P/B+4B/6rW2ikQXcD/wfWuv7gDwg1x+twW+j+UngL7f6XLYjpVQG76//e4B+IKaU+sWtPavtR2t9Fvhd4Ct47SGvAvZaz7mdA/aGLb0uxBK/p/ivgD/TWn9mq89nu/P/PH0SeN/Wnsm29Bjwk35v8V8AP6aU+tOtPaXtS2s94t9OAH+N1wYoGg0BQ3V/Mfo0XuAWq3s/8IrWenyrT2SbehdwWWs9qbWuAp8BHt3ic9qWtNZ/pLW+X2v9OF7b7ar913B7B2xZel1sKP/ivT8Czmqt/9NWn892pZTqUkql/fsRvG/Q57b0pLYhrfW/0FoPaq13431/+prWWipDLSilYv6FxfgtD+/B+5OsqOMvxnZdKXXI3/ROQC7CXtvTSHvIWq4Bb1NKRf2fge/Eu/5IrKCU6vZvdwI/wzr/rjZ1JceNtNrS61t8WtuSUuqTwJNAVik1BPw/tdZ/tLVntS09BvwScMrvLwb4l/4KpGJZH/B/+VfmG8AzWmsZQSfejB7gr72f71jAn2utv7S1p7Rt/SPgz/zC0iXgV7f4fLYtv1f23cDf3+pz2a601t9TSn0aeAWv5eEHyKqOq/krpVQnUAX+odZ6dq2Db9sxfUIIIYQQQmxHt3OLiBBCCCGEENuOBGwhhBBCCCE2kARsIYQQQgghNpAEbCGEEEIIITaQBGwhhBBCCCE2kARsIYQQLSmltFLqZ7f6PIQQ4nYjAVsIIbYhpdR/8wPuyo/vbvW5CSGEWNttu9CMEEK8BTyHt/hRvcpWnIgQQoj2SQVbCCG2r7LWemzFxwzU2jd+Syn1RaVUQSl1VSnVsBS7UupupdRzSqmiUmrGr4qnVhzz95RSp5RSZaXUuFLqv604hw6l1F8qpfJKqUstPse/8j93WSk1ppT641vxRgghxO1EArYQQty+/g3wOeA43vLGf6yUegBqy0R/CcgBDwEfAh4FPr70ZKXU3wf+T+ATwD3AjwOnV3yOfwX8d+Be4FPAx5VSu/znfxj4J8D/ABwAPgh8f+O/TCGEuL3IUulCCLEN+ZXkXwRKK3b9V631P1NKaeBjWutfr3vOc8CY1voXlVK/DvwHYFBrvejvfxL4OnBAa31RKTUE/KnW+p+vcg4a+Hda63/hP7aABeA3tNZ/qpT6n4C/DxzTWlc36msXQojbnfRgCyHE9vVN4DdWbJuru/+dFfu+A3zAv38X8KOlcO17AXCBI0qpBWAA+Oo65/CjpTtaa1spNQl0+5v+Evht4LJS6st4FfPPaa3L67ymEELc0aRFRAghtq+C1vriio+pNp+rgNX+RKn9/e1YWZnW+D87tNbXgUN4VewF4D8CLyulYm2+thBC3JEkYAshxO3rbS0en/XvnwHuVUol6vY/ivd9/6zWehwYBt75Zk5Aa13SWn9Ra/0/Ag8CR4HH3sxrCiHE7U5aRIQQYvsKKaV6V2xztNaT/v2fUUq9CJwEfhYvLD/s7/szvIsg/1gp9a+ADN4FjZ/RWl/0j/l/A/9fpdQ48EUgCrxTa/0f2zk5pdSv4P0c+R7exZQfwat4v36DX6cQQtxRJGALIcT29S5gdMW2YWDQv/+vgQ8DvwdMAr+qtX4RQGtdUEq9F/jf8SZ7lPCmgfz20gtprf8PpVQF+J+B3wVmgGdv4PzmgH+GdzFlAK9q/jNa68s38BpCCHHHkSkiQghxG/InfPwdrfWnt/pchBBCNJIebCGEEEIIITaQBGwhhBBCCCE2kLSICCGEEEIIsYGkgi2EEEIIIcQGkoAthBBCCCHEBpKALYQQQgghxAaSgC2EEEIIIcQGkoAthBBCCCHEBpKALYQQQgghxAb6/wPr1k6vXFAjhgAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAtgAAAHoCAYAAABzQZg1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAADxsklEQVR4nOzdd3hUVf7H8fednslMei+kQICEXkQURIqKBdsiFixgw13XtmsXe5dVd9Xdn+4KKqxrRbGLlaig0ntCD6GkkN6n398fkwwZkkCAkPp9PQ8PmXvPvXPmEjKfnPnecxRVVRFCCCGEEEK0DU1Hd0AIIYQQQojuRAK2EEIIIYQQbUgCthBCCCGEEG1IArYQQgghhBBtSAK2EEIIIYQQbUgCthBCCCGEEG2oXQO2oihnK4qyVVGUHYqi3NfM/rsVRVlX/2eToihuRVHC2rOPQgghhBBCHA+lvebBVhRFC2wDzgT2ASuBK1RVzWqh/fnAX1RVndguHRRCCCGEEKINtOcI9ihgh6qqu1RVdQDvARcepv0VwLvt0jMhhBBCCCHaSHsG7Hhgb6PH++q3NaEoihk4G/ioHfolhBBCCCFEm9G143MpzWxrqT7lfGCZqqqlzZ5IUWYBswBMJtOIXr16tU0PuzmPx4NGI/e1Holcp9aR69R6cq1aR65T68m1ah25Tq0j16n1tm3bVqyqauSR2rVnwN4HJDZ6nADktdD2cg5THqKq6n+A/wD069dP3bp1a1v1sVvLzMxk/PjxHd2NTk+uU+vIdWo9uVatI9ep9eRatY5cp9aR69R6iqLktqZde/66shJIUxQlRVEUA94Q/dmhjRRFCQZOBz5tx74JIYQQQgjRJtptBFtVVZeiKLcA3wBa4A1VVTcrivLH+v2v1Te9GPhWVdWa9uqbEEIIIYQQbaU9S0RQVfUr4KtDtr12yOO3gLfar1dCCCGEEEK0HaloF0IIIYQQog11/YDdTgvlCCGEEEII0RrtWiJyIugKCnCVlaELDW12f2VlJQcOHMDpdLZzzzqf4OBgsrOzO7ob7SowMJCEhASZfkgIIYQQ7abLB2zF4WTPzGvp9dabTUJ2ZWUlhYWFxMfHExAQgKI0NxV3z1FVVYXVau3obrQbj8fD/v37KS4uJioqqqO7I4QQQogeolsM69m3bmXPddfjLi/3237gwAHi4+Mxm809Plz3RBqNhujoaCoqKjq6K0IIIYToQbp+wK7PzfbsbG/IbhSmnE4nAQEBHdQx0Rno9XpcLldHd0MIIYQQPUiXD9jusHCoH522ZWWx5/obcFdW+vbLyHXPJv/+QgghhGhvXT5geyyBxD75hO+xbdMm9txwI+6qqg7slRBCCCGE6Km6fMAGCJk6lZjHH/M9tm3YwN4bbkT1eDqwV0IIIYQQoifqFgEbIPTSS4l59BHf47r163GXlqK63R3Yq2NXVFTEzTffTHJyMkajkejoaCZNmsR3333na7Nr1y5uuOEGkpKSMBqNxMXFMWHCBObPn4/D4fC1UxQFRVEICgrCbDaTmprK9OnTWbp0aUe8NCGEEEKIbq3LT9PXWOjll6O63RQ+8SQAqsOBIzcXQ1ISilbbwb07OlOnTqW2tpZ58+bRp08fDhw4wE8//URJSQkAq1atYtKkSaSnp/PKK6/Qv39/amtryc7O5vXXX6dPnz6MGTPGd77XX3+d8ePHo9fr2bVrF/Pnz2fcuHE899xz3H333R31MoUQQgghup1uFbABwq68EjwqhU89BYCnthZH7h4MSb26TMguLy/nl19+4bvvvmPSpEkAJCUlcdJJJwGgqiozZswgLS2NX3/91W8RlaFDh3LFFVegHrLCZUhICNHR0VitVpKSkpgwYQJxcXHcf//9XHzxxfTp06f9XqAQQgghRDfWbUpEGgu7+iqi77/P99hTW4Njz54uU5NtsViwWCx89tln2Gy2JvvXrVtHVlYWd911V4srFLZm9ow777wTj8fDJ598crxdFkIIIYQQ9brdCHaDsBkzyFu50vfYU+MN2f1e29xhfdr97HmtaqfT6Xjrrbe48cYb+c9//sOwYcMYM2YM06ZN4+STT2bbtm0A9OvXz3dMRUUF8fHxvscPPPAADzzwwGGfJzw8nKioKHbt2nUMr0YIIYQQQjSnW45gN9BaLOiio32PPdXVHdibozN16lTy8vL4/PPPOeecc/j1118ZPXo0Tz/9dLPtrVYr69atY926dcTFxfnd5Hg4qqrKXNFCCCGEEG2oWwdsAH1kJLqo6CM37IRMJhNnnnkmDz/8ML/++ivXX389jz76KMnJyQBs2bLF11aj0dCnTx/69OmDwWBo1fmLi4spKioiNTX1RHRfCCGEEKJH6rYlIo3poyIBFdeBA2RflQSAxmrFkJiI0kINc2eUkZGBy+Wif//+pKenM2fOHC699FK0x3jz5gsvvIBGo+HCCy9s454KIYQQQvRcPSJgA+ijokAFV9EBADxVVTj37kXfCUN2SUkJ06ZN47rrrmPw4MFYrVZWrVrFnDlzmDRpEsHBwbz11lucccYZnHLKKcyePZv09HTcbjfLli1j3759TUJ3eXk5hYWFlJWVsXPnTubPn8+CBQuYM2eOzCAihBBCCNGGekzABtA1jGQXFQF4l1Pfuw99YkKnCtkWi4XRo0fz0ksvsWPHDux2O/Hx8UyfPp0HH3wQgFGjRrFmzRqeeeYZbr31VgoKCggICGDw4ME89dRT3HDDDX7nvPHGGwEwGo3ExsYyevRoMjMzGTduXLu/PiGEEEKI7qxHBWxFUdA1jGQXN4TsSti3D31C5wnZRqORp59+usUbGhv06dOHefPmHfF8DXNiV1VVYbVa26SPQgghhBCieZ0jUbYjRVHQRUehi4jwbXNXVuLct6/LzJMthBBCCCE6rx4XsKEhZEejCz8kZO/f32QFRCGEEEIIIY5GjwzYUB+yY6LRhYf7trkrKrwj2RKyhRBCCCHEMeqxARsaQnZMMyFbRrKFEEIIIcSx6dEBGxqF7LAw3zZ3RbmUiwghhBBCiGPS4wM21Ifs2Fi0jUN2eTnO/XkSsoUQQgghxFGRgF1PURT0sbFoQ0N929zlZTjzJGQLIYQQQojWk4DdiKIo6OPi/EN2WRnOvHwJ2UIIIYQQolUkYB/CF7JDQnzb3GWlOPMlZAshhBBCiCOTgN0MRVHQx8f7h+zSUlwSsoUQQgghxBFIwG6BL2QHB/u2uUpLcRUUnPCQPXPmTBRF4YYbbmiy75577kFRFKZMmXJC+9BWFEVh4cKFHd0NIYQQQoh2IwH7MBRFQZ+Q4B+yS0raJWQnJiby/vvvU1NTc/C5XS7++9//0qtXrxP63EIIIYQQ4thJwD4CX8gOOiRkFxae0JA9ePBg0tLS+OCDD3zbvvzyS0wmE+PHj/dt83g8PPHEEyQmJmI0Ghk0aBCffvqpb//u3btRFIX33nuPc845h4CAAIYNG8aGDRvYtGkTp556KoGBgYwdO5acnBy/Pnz++eeMGDECk8lESkoKs2fPxuFw+PYnJyfz5JNPctNNNxEUFERCQgJ/+9vf/PYDTJs2DUVRfI8fffRRBg4c6Pdcb731FhaLxfe4oc38+fNJTk7GYrFw7bXX4nA4+L//+z8SExMJDw/nr3/9Kx6P55ivsxBCCCFEW5OA3QrekB2PNijIt81VXHzCQ/b111/PG2+84Xv8xhtvcO2116Ioim/bSy+9xN/+9jeee+45Nm7cyMUXX8wf/vAH1q1b53euRx55hDvuuIO1a9cSEhLC9OnTufXWW3nqqadYsWIFNpuN2267zdf+m2++4corr+SWW25h8+bNvPHGGyxcuJAHHnjA77x///vfGTRoEGvWrOHee+/lnnvu4bfffgNg5cqVALz++uvk5+f7HrfW7t27+fTTT/niiy/46KOP+PDDD7nwwgtZuXIl3377LXPnzuWVV15h0aJFR3VeIYQQQogTSdfRHWh3jwYfuU0zFMBw3M9dcVTNp0+fzl133cX27duxWq0sXryYV155hYcfftjX5vnnn+euu+5i+vTpADz++OP8/PPPPP/887z99tu+dn/961+ZPHkyVquVO++8k/PPP5+PPvqICRMmAHDLLbdwyy23+No/9dRT3H333Vx77bUA9O7dm+eee46rrrqKv/3tb76Qf9ZZZ/mOu/XWW3n55Zf54YcfOOWUU4iMjAQgJCSEmJiYo71auN1u3nzzTYKDgxk4cCBnn302P/30E/v378dgMJCens6YMWNYsmQJU6dOPerzCyGEEEKcCD0vYHchoaGhXHzxxbzxxhuEhIQwfvx4v/rryspK8vLyGDNmjN9xY8eO5auvvvLbNnjwYN/X0dHRAAwaNMhvW01NDbW1tZjNZlavXs2KFSt47rnnfG08Hg91dXUUFBQQGxvb5LwAcXFxHDhw4DhfuVevXr0IblT/Hh0dTd++fTEYDH7b2ur5hBBCCCHaggTsTu66665jxowZWCwWHn/88WbbNC4ZaWmbXq9vsq+5bQ31zB6Ph0ceeYRp06Y1OXfDyPSh52g4z5FqojUaTZPSGqfT2aRdc+dubpvb7T7s8wkhhBBCtKeeF7CPskyjJarHg2PPHjzV1b5tuqgo9FFRbXL+BpMmTcJgMFBcXMxFF13kty8oKIi4uDiWLl3KxIkTfduXLl1KRkbGcT3v8OHD2bJlC3369Dmu8+j1+iYBODIyksL6+vWGYH9ozbgQQgghRFfV8wJ2G1E0Ggy9evmFbNeBA6Ao6BuN8B738ygKGzZsQFVVjEZjk/133303Dz/8MGlpaYwYMYK3336bX375hdWrVx/X8z788MNMmTKFpKQkLr30UnQ6HZs2bWLFihXMmTOn1edJTk7mhx9+4PTTT8doNBIaGsr48eMpLS3l6aef5vLLLyczM1PmyhZCCCFEtyGziByHhpCtaTS9nKuwEFdRUZs+j9VqJajRDCaN3Xbbbdx9993cc889DBw4kEWLFvHRRx8xdOjQ43rOyZMn8+WXX7JkyRJGjRrFqFGjePbZZ496Du4XXniBJUuWkJiYyLBhwwBIT0/n1Vdf5T//+Q+DBw/mu+++azI7iRBCCCFEV6V09aW/+/Xrp27durXZfdnZ2aSnp5/wPqgeD47cXDyNFoXRx8Sgi4g44c99NKqqqrBarR3djXZ3tN8HmZmZfnONi+bJdWo9uVatI9ep9eRatY5cp9aR69R6iqKsVlV15JHayQh2G/CNZAcG+rY5CwpwFRd3YK+EEEIIIURHkIDdRhSt1huyzWbfNmdBAa6Skg7slRBCCCGEaG8SsNuQotViSEryD9n5+RKyhRBCCCF6EAnYbcwXsgMOCdmlpR3YKyGEEEII0V4kYJ8AilaLITkJTUCAb5szL09CthBCCCFEDyAB+wTxhuzkpiG7rKwDeyWEEEIIIU40CdgnkC9kmxqF7P37JWQLIYQQQnRjErBPMF+5iMnk2+bcvx9XeXnHdUoIIYQQQpwwErDbgaLT1Y9kNwrZ+/ZJyBZCCCGE6IYkYLcTX8g2+odsd0VFB/ZKCCGEEEK0NQnY7UjR6TCkJKMxGn3bHHubhuyZM2eiKApPPvmk3/bMzEwURaG4DVaIbMtzCSGEEEKIgyRgtzNvyE5B8YVstdmQbTKZmDNnDkVFRe3fyaPkdDo7ugtCCCGEEJ2GBOwOoOh0GJOT/UP2vn24Kyt9bSZMmEBycjJPPPFEi+fJysrivPPOw2q1EhUVxRVXXEFBQYFv/8aNG5k0aRJBQUFYrVZOPfVUlixZwu7du5kwYQIAkZGRKIrCzJkzvT1RVebMmUPv3r0JCAhg0KBBvP32275z7t69G0VRePfdd5k4cSIBAQH8+9//xuPx8MQTT5CYmIjRaGTQoEF8+umnvuNOOeUU7rzzTr/+V1ZWEhAQwKJFi471UgohhBBCdDoSsDuIotdjSE5GMRi8G1QVx969vpCt0Wh49tlnee2119i5c2eT4/Pz8xk3bhwDBw5kxYoVfP/991RXV3PBBRfg8XgAmD59OrGxsaxYsYK1a9dy//33YzKZSExM5KOPPgJg8+bN5Ofn89JLLwHw4IMPMm/ePP71r3+RlZXF/fffz0033cSXX37p9/z3338/N998M1lZWVx00UW89NJL/O1vf+O5555j48aNXHzxxfzhD39g3bp1AFx11VW89957vr4BfPTRRwQEBHDeeee16bUVQgghhOhIuo7uQHsbNH9Qhz33xhkb/R5r9HoMKSk4cnJQHQ5fyFbrSy7OPfdcxowZw+zZs3nvvff8jn311VcZMmQIzz33nG/bggULCAsLY9WqVYwaNYrc3Fzuuusu+vfvD0B0dDRWqxWAsLAwAKKiooiIiACgpqaGF198kW+//ZbTTjsNgJSUFFasWMG//vUvvyB86623cskll/geP//889x1111Mnz4dgMcff5yff/6Z559/nrfffpvLL7+cv/zlLyxZsoRJkyYB8L///Y9p06ZhaPglQwghhBCiG5AR7A7WELIbj2R7ampQXS4A5syZw4cffsiqVav8jlu9ejU///wzFovF9ycxMRHAN+L917/+lRtuuIGJEyfy1FNPsW3btsP2JSsrC5vNxtlnn+133ldffbXJKPrIkSN9X1dWVpKXl8eYMWP82owdO5asrCwAwsPDmTx5Mv/73/8A7wj8kiVLuOqqq47mcgkhhBBCdHoSsDsBTUO5iL5RyK6rw11VxUknncTUqVO59957/Y7xeDycd955rFu3zu/P9u3bmTJlCgCPPvqor4Tj119/5ZRTTuGNN95osR8N5Ruff/653zk3b97Mt99+69c2MDCwyfGKohx221VXXcVHH32EzWbj3XffJTExkbFjx7buIgkhhBBCdBE9rkTk0DKNzkJjMGBIScaRk+Pb5tizB0NSEk8//TQZGRksXrzYt2/48OF88MEHJCUlodfrWzxvWloaaWlp3Hbbbdxwww3MnTuX6667zleW4Xa7fW0zMjIwGo3k5uYyceLEVvc9KCiIuLg4li5d6nfc0qVLycjI8D2+8MILmTVrFl988QX/+9//uPLKK5sN5UIIIYQQXVmPC9idmTdkp4Cm/oMFVcWRm0tKUhKzZs3y3YgI8Oc//5nXX3+dyy67jHvvvZfIyEh27drFBx98wAsvvIBOp+Ouu+5i2rRpJCcnU1hYyG+//cYpp5wCQFJSEoqi8OWXX3L++ecTEBCA1Wrlrrvu4q677kJVVcaNG0d1dTW///47Go2GWbNmtdj3u+++m4cffpi0tDRGjBjB22+/zS+//MLq1at9bUwmE3/4wx948sknWb9+vd/sJEIIIYQQ3YWUiHQyGoMBjdV6cGRXVXHk7mH2nXei0x38fSguLo5ly5ah0Wg4++yzGTBgAH/+858xGo0YjUa0Wi1lZWXMmDGDfv36cfHFFzNq1ChefPFFAOLj43nssceYPXs20dHR3HLLLQA88cQTPProozz//PMMGDCAM888k48++oiUlJTD9vu2227j7rvv5p577mHgwIEsWrSIjz76iKFDh/q1u/rqq1m/fj3Dhw8nPT297S6cEEIIIUQnISPYndD8BQvw2O04cnajupygegipq6O8oABto9rntLQ0Fi5c2OJ53nnnHb/HVVVVvllEAB566CEeeughvzaKonDrrbdy6623NnvO5ORkVFVtsl2j0TR7vkNNnDix2eOFEEIIIboLGcHupDRGI4aUZJSGUWuPB0duLu6amo7tmBBCCCGEOCwJ2J2YN2Sn+IVsZ24u7traju2YEEIIIYRoUbsGbEVRzlYUZauiKDsURbmvhTbjFUVZpyjKZkVRfmrP/nVGh4Zs1ePBuXs3HgnZQgghhBCdUrsFbEVRtMC/gHOADOAKRVEyDmkTAvwfcIGqqgOAae3Vv85MYzR658nWHgzZDgnZQgghhBCdUnuOYI8CdqiquktVVQfwHnDhIW2mAx+rqroHQFXVA+3Yv05NYzJ5a7K1WqA+ZOfm4qmr6+CeCSGEEEKIxpT2mtFBUZRLgLNVVb2h/vHVwMmqqt7SqM0/AD0wALACL6mquqCZc80CZgFERkaO+OCDD5p9zuDgYPr06dPGr6SDORzoCguhftVFNBpcUdFgNBzxULfbjbY+oPckO3bsoKKiotXtq6ursVgsJ7BH3YNcp9aTa9U6cp1aT65V68h1ah25Tq03YcKE1aqqjjxSu/acpq+5JfsOTfc6YAQwCQgAflMU5XdVVbf5HaSq/wH+A9CvXz91/PjxzT5hdna237R03YXHbMaxezeq2w0eD/qiAxiSk9EEBBz2uEOn6espTCYTw4YNa3X7zMxMWvqeEgfJdWo9uVatI9ep9eRatY5cp9aR69T22rNEZB+Q2OhxApDXTJvFqqrWqKpaDPwMDGmn/nUZmoCA+prs+nIRt9tbk22zdXDPhBBCCCFEewbslUCaoigpiqIYgMuBzw5p8ylwmqIoOkVRzMDJQHY79rHL8IVsTaOQnZMjIVsIIYQQooO1W8BWVdUF3AJ8gzc0f6Cq6mZFUf6oKMof69tkA4uBDcAKYK6qqpvaq49djTdkJ6FoNCz69lsCMjJw5HT+kewpU6Ywc+bMju6GEEIIIcQJ0a7zYKuq+pWqqn1VVe2tqupT9dteU1X1tUZt/qaqaoaqqgNVVf1He/avs1m7di1arZYxY8a02EZjNmNITgbFW+Kuul3echG7vVXPsWvXLm644QaSkpIwGo3ExcUxYcIE5s+fj8PhaIuXIYQQQgjRo8hKjp3Y66+/zs0338ymTZvIzm65UkZjNqOLjPI9Vl0ub7nIEUL2qlWrGDZsGJs2beKVV15h48aNfPXVV8yaNYv58+ezcuXKFo91Op1H/4KEEEIIIXoACdidVF1dHe+88w433ngjl1xyCfPmzfPbv2DBApKSkjCbzUyZMoWiinLvDo33n3RnTg4XnnsuMTExBAYGMnz4cL7++mvf8aqqMmPGDNLS0vj111+54IIL6Nu3L0OHDuWKK67gxx9/5NRTTwVg9+7dKIrCu+++y8SJEwkICODf//43JSUlXHHFFSQkJBAQEMCAAQN48803/fpZW1vLzJkzsVgsREdH8/TTT5+4iyaEEEII0QlIwO6kFi5cSFJSEoMHD+bqq69mwYIFvlHj5cuXM3PmTGbNmsW6des4//zzefjhhwEwJCWBRkN1bS1njhnDF//5D2tXrmTq1KlcddVVbNmyBYB169aRlZXFXXfdhUbT/LeBovjPrHj//fdz8803k5WVxUUXXYTNZmP48OF88cUXbN68mdtvv52bbrqJH374wXfMXXfdxXfffcdHH33EDz/8wNq1a/n5559PxCUTQgghhOgU2nMe7E4hu396hz13+pbWT4gyd+5crr76agBOP/10zGYzn332GVOnTuWll15i0qRJzJ49G4C+ffuycuVK5s2bhzYwEEOvJAarMLhfPwAUrZb7776bTz75hIULF/Lggw+ybZt3avF+9W0AKioqiI+P9z1+4IEHeOCBB3yPb731Vi655BK/ft59992+r2fNmsWPP/7Iu+++y6RJk6iurmbevHm88cYbTJ48GYA333yThISEVl8HIYQQQoiuRkawO6EdO3awbNkypk+fDnhHkq+88krmzp0LeBfQOeWUU/yOafxYawnEGRHO7BdfZPiFFxJ70kkEhYaydu1a9uzZ0+LzWq1W1q1bx7p164iLi2tyk+PIkf4LF7ndbp566ikGDx5MeHg4FouFjz/+2PccO3fuxOFw+PXNYrEwaNCgY7gqQgghhBBdQ48bwe4K5s6di9vtplevXr5tDUva7927l9Ysb3/vo4+y+McfefqOO+idmIg5IIAbHngAW3kFHoeDvn37ArBlyxbfKocajca3tLzB0HTp9cDAQL/Hzz//PC+88AIvvfQSgwYNwmKx8MADD3DgwAG/PgshhBBC9CQ9LmAfTZlGR3C5XMyfP59nnnmGKVOm+O27+uqrefPNN8nIyOD333/323fo46VLl3LNjBlcev31OPbswWazkbN3L2lJSdi3bSMjLIz0fv2YM2cOl156Kdr6VSGPxtKlSzn//PN9pSyqqrJt2zZCQkIA6NOnD3q9nt9//53U1FQAampq2LRpE7179z7q5xNCCCGE6Ap6XMDu7L788kuKi4u58cYbCQ8P99t3+eWX8+qrr/LOO+8wduxYnnnmGS655BIyMzNZtGiRX9u+ffuyaNEiLrzwQjROJ48/8gi2RiUfnqoqXnvkEabceCOjR41i9kMPkZGRgdvtZtmyZezbt++Iobtv3768//77LF26lIiICF555RVycnJ8I+IWi4Xrr7+ee++9l8jISOLi4nj88cdxu91tdLWEEEIIITofqcHuZObNm8eECROahGuAadOmkZub67t58NVXX2Xw4MF8/PHHPProo35tX3zxRaKiojjttNOYMnUqp0yaxCmjR4Ne72tz0qBB/PrBB2QkJXHrzTczcOBARo8ezfz583nqqae45557DtvXBx98kFGjRnHOOecwbtw4AgMDufLKK/3aPP/880yYMIGLL76YCRMmMHDgQMaNG3fsF0gIIYQQopOTEexO5rPPPmtxX2pqql9d87XXXuu3/5ZbbvF9nZSUxPfff++3/49//CNWqxWPzYarpAR3eTm9e/Xitccf97VRNBq0oaFow8PR1NdhJycnN1tPHRoayscff3zY1xMYGMiCBQtYsGDBYdsJIYQQQnQXErB7II3JhCE+HjUqCldpGe7SEtT6sg3V48FVUoKrpARtUDC6iHA0ZnMH91gIIYQQouuQgN2DKXo9+ugodJERuMvLcZWUoDZaXt1dWYG7sgJNgNkbtIOCmiw+I4QQQggh/EnAFigaDbqwMLShoXiqq3GVlOCprvbt99TV4thbi6LXowsPRxsainIMs44IIYQQQvQEErCFj6IoaK1WtA112sXFuCsqoL7+WnU6cRYU4DpwAG1YGNqwMF+dthBCCCGE8JKALZqlMZkwJCTgiY7GXVqKu7TUv067uBhXcQna4CB04VKnLYQQQgjRQAK2OCyNXo8mOhpdZKS3Tru4BNXRUKet4q6owF1RgcZsRhcRgcZqlTptIYQQQvRoErBFqzSp0y4uxlNT49vvqa3FsWcPisHgrdMOCZE6bSGEEEL0SBKwxVHxq9Ouq/POp924TtvhwJmf763TbphPu9HiNkIIIYQQ3Z0EbHHMNAEBB+u0S0pxlzWq03a7vXXajefTDgjo4B4LIYQQQpx4ErDFcdPo9Whiov3n03Y4vDtVFXdFOe6KcjTmQG/QljptIYQQQnRjmo7ugGhq5syZKIqCoijodDp69erFn/70J8rKynxtkpOTURSFt99+u8nxo0aNQlEUnn/+ed+2nJwcbrjhBhISEjAajcTFxXHeeeexdu3aJudUFAWz2czAgQP597//3ep+K1otuvBwjGlpGHr1ajKziKe2BseePdi3b/eG8PrRbiGEEEKI7kQCdid1xhlnkJ+fz+7du5k7dy6ff/45N998s1+bxMRE5s2b57dt06ZNbN68mfDwcN82p9PJmWeeSXFxMR988AHbtm1j4cKFjBo1itLSUr/jH374YfLz89mwYQMXXXQRf/zjH3n//feb7aOjYZT6EIqioA0KwpiaijG1N9rgYODgiHVDnbZ92zachYV4nM6juTRCCCGEEJ2alIh0UkajkZiYGAASEhK47LLLeOutt/zaTJ8+nX/84x/s2rWL1NRUAObNm8cll1zCTz/95Gu3efNmdu7cycKFCxk6dCgASUlJnHrqqU2e12q1+p73ySef5IMPPuCTTz7hsssuY/z48aSnpxMYGMj8+fNJTk5m5cqV/Pzzz9x9992sX7+e4OBgpk+fznPPPYfBYEBjDuCsc6+mX1oaBlXl7Q8/BFVl5tSpPPmXv6AWFeEqLkYbHOydT1vqtIUQQgjRxckIdhewa9cuFi9ejP6Q2TgiIiI4//zzefPNNwHviPLbb7/N9ddf79cuMjISjUbDZ599hsvlOqrnNplMOBuNML/99tuoqsovv/zCggUL2L9/P+eccw7Dhg1j7dq1zJs3j3fffZf777/f7zzvvPceBATw66+/8n9//ztvLFzIP//7X+9OVcVdXo59507sOTm4q6pQ62clEUIIIYToanrcCPa//vhjhz33n1+b2Oq2ixcvxmKx4Ha7sdlsALz44otN2l133XXcdNNNPPbYY3z22WeEhIQwbtw4vzbx8fG8/PLL3HPPPcyZM4cRI0Ywbtw4Lr/8cgYMGNDs87tcLt5++202btzIn/70J9/2lJQUXnjhBd/j2bNnExsby//93/+h0WhIT0/n2Wef5aabbuKJJ57AXF+HHRsby8svv4yiKKQPGMCOvDxeee017vjTn/DU1vrO56mpwVFTg2I0HpxPWyO/BwohhBCi65Dk0kmNGzeOdevWsWLFCm699VbOPfdcbrvttibtJk+ejKqqfPfdd8ybN4/rrruu2fP9+c9/Zvv27bzzzjuMHTuWTz/9lKFDh/LfhlHkerNnz8ZisRAQEMCf//xn7r77bm666Sbf/hEjRvi1z87O5pRTTkHTKASPHTsWh8PBjh07fNtGjx7tN3PIqaeeyv68POwRERhTU5vWadvtOPPysG/dirOwEFXqtIUQQgjRRUjA7qTMZjN9+vRh0KBBvPzyy9TW1vLEE080aafRaJgxYwZPP/00P/74IzNmzGjxnFarlQsuuICnnnqK9evXM2HCBB566CG/Nn/9619Zt24dubm5VFdXM2fOHL/wHBgY6NdeVdUWp9xr7VR8GrMZQ2Iixr5p6MLD/UasVbcbV1ERtm3bcOzbj6d+NF8IIYQQorPqcSUiR1Om0Zk88sgjnHPOOcyaNYu4uDi/fddddx1PP/005557bpN9LVEUhf79+7NmzRq/7eHh4fTp06fV/crIyOCDDz7A4/H4gvjSpUsxGAz07t3b12758uV+Yfz3338nLi6OoKAgXxuNwYAmNhZdVBTusjLvVH4NI9eqiru8DHd5GRqLxXtDpMUi82kLIYQQotOREewuYvz48QwYMIAnn3yyyb7U1FSKi4v58MMPmz123bp1XHjhhXzyySdkZWWxY8cO5s2bxxtvvMHFF198XP26+eabycvL4+abbyY7O5svv/yS++67j1tuucVXfw2Ql5fHHXfcwdatW1m4cCF/+9vf+Mtf/tLsORWtFl1EBMa+fTEkJjaZWcRTXY0jNxf7jh24SktRPZ7jeg1CCCGEEG2px41gd2V//etfufbaa7n33nub7AsLC2vxuISEBFJTU3nuuefYs2cPHo+HXr16cdddd3HfffcdV5/i4+P5+uuvufvuuxk6dCghISFMnz6dp59+2q/dlVdeidvt5uSTT0ZRFK6//voWA3YDRVHQBgejDQ7GXVuLu7gYd2Wlb39Dnbar8ADa8DB0YWEoOvmWFkIIIUTHkjTSCR0633WD6dOnM336dAB279592HM03h8REcHf//53qqqqsFqtrTqmOZmZmc1uHzduHMuXLz/ssTqdjn/+85/885//PGy7lmjNZrS9euFxOHCXlOAuK/ONXKtuF64DB3AVFaENCfGWj5hMx/Q8QgghhBDHSwK26FL86rRLy3CVHlKnXVaGu6y+TjsiAs0hN2UKIYQQQpxoErBFl6RotegiI9CGh+GurMRdUoKnrs6331NdjaO6Go3RhKe2Fo/DgcZg6MAeCyGEEKKnkIAtTriWSkvagqLRoAsJQRscjKe21ls+0qhO22O34S4vZ8fESYRdOZ2Qyy9HFxp6wvojhBBCCCGziIhuQVEUtIGBGHr1wpiWhi4sHBrNp+0uLqbopZfZMWEi+Y8+in1XTgf2VgghhBDdmQRs0e1ojEb0cbGY+vZFFx3tF7RVm43y995n17nnsvePf6Lmd+/83EIIIYQQbUUCtui2FJ0OfWQk+uho4v42B2NGut/+6sxM9sycSc4fplLx6aeoDkcH9VQIIYQQ3YkEbNH9KQrB559Pykcf0WvBfCwTJvjttmdnk3fvfeyYdAbF//4PSk1NB3VUCCGEEN2B3OQoegxFUQgcNYrAUaOw5+RQumABFYs+QbXZAHAVFVH0978TqdGQ8+ZbmEeMIGDkCMwjRqA7zEI+QgghhBCNScAWPZIxJYXYRx4h8rbbKH//A0r/9zbuomIAFI8H26ZN2DZtgvnzATCkpmIeMQLzyBEEjBiJPj4ORVE68iUIIYQQopOSgN1NLFy4kGnTpskNe0dJFxpKxB9vIuy6a6n86ivK/vcOdZs2oRxyHR27duHYtYvyDz/0HhcT4x3hHjEc84iRGNP6oGik4koIIYQQErA7tbVr1zJy5EhGjx7NsmXLOro73ZrGYCDkoosIuegifvrqK4abzdStXk3tqtXUbdoEDatF1nMVFFD55ZdUfvml9/jgYMzDhtWPcI8gYMAAFFnYRgghhOiRJGB3Yq+//jo333wzCxYsIDs7m/T09CMfJI6bajZjHT8e6/jxAHhsNuo2bDgYuNeuxVNb63eMp6KC6sxMqusX1VFMJgIGD/YFbvPQobJsuxBCCNFDyGfanVRdXR3vvPMON954I5dccgnz5s3z279gwQKSkpIwm81MmTKFwsJCv/07d+7kwgsvJCYmhsDAQIYPH87XX3/t1yY5OZnHH3+cmTNnYrVaSUxM5P3336e8vJzLL78ci8VCWloa33777Ql/vZ2ZxmQicNQoIv70J3rNm0vfFctJ/mgh0Q/cj/Wss9CGhzc5RrXZqF2xguL/e5W919/A1lEnk3PJNAqfeZbK777DVVraAa9ECCGEEO1BAnYntXDhQpKSkhg8eDBXX301CxYswFlfprB8+XJmzpzJrFmzWLduHeeffz4PP/yw3/HV1dWcc845fPfdd6xfv56pU6dy1VVXsWXLFr92//jHPxg1ahRr1qzh0ksvZcaMGUyfPp1zzz2XdevWMW7cOK666ips9TNtCO/82gEDBhB2zTUkvPwSaUt/IfXrr4h98gmCL7oIfWJi04PcbmybNlE6fz77b72N7aeOYee555H/0MNUfPopjn37pX5eCCGE6CZ6XInIC5dN6bDnvvP9L1rddu7cuVx99dUAnH766ZjNZj777DOmTp3KSy+9xKRJk5g9ezYAffv2ZeXKlX6j3EOGDGHIkCG+x7Nnz+aTTz5h4cKFPPjgg77tkydP5uabbwbgscce48UXX6RPnz5cc801ADz00EO88cYbbNq0iZEjRx77i+/GFEXBmJKCMSWFkEsuAcBZeIC61auoXbWa2tWrsW/bBq25cXL48PqpAeXGSSGEEKKr6nEBuyvYsWMHy5Yt49133wW8Ae7KK69k7ty5TJ06lezsbM4//3y/Y0455RS/gF1TU8Njjz3GF198QX5+Pk6nE5vNxrBhw/yOGzx4sO9ri8WC2Wxm0KBBvm3R0dEAHDhwoM1fZ3emj45Cf+65BJ17LgDuykpq16w58o2TX31F5VdfAXLjpBBCCNFVScDuhObOnYvb7aZXr16+bQ3lA3v37m1VKcFdd93F4sWLef7550lLS8NsNnPllVfiOGQ5cL1e7/dYURS/bQ1zPXs8nmN+PQK0QUFy46QQQgjRQ/S4gH00ZRodweVyMX/+fJ555hmmTPEvZ7n66qt58803ycjI4Pfff/fbd+jjpUuXcs011zB16lQAbDYbOTk5MhNJJ9Fw42TgqFEAqC4Xtq1bfYG7dvVq3CUlfsc03DhZu2KFd4NWiyk9XVacFEIIITqZHhewO7svv/yS4uJibrzxRsIPmZ3i8ssv59VXX+Wdd95h7NixPPPMM1xyySVkZmayaNEiv7Z9+/Zl0aJFXHjhhej1eh577DHsdnt7vhRxFBpunGy4eVJVVRy7d/sFbufevf4H1d846bfiZErKwRHukSPRx8fLipNCCCFEO5OA3cnMmzePCRMmNAnXANOmTeO+++6jurqaefPm8cgjj/D4448zfvx4Hn30UW699VZf2xdffJHrr7+e0047jdDQUO644w6qq6vb86WI43DMN07m5ODIyaH8w4UA6KKjG41wy42TQgghRHuQgN3JfPbZZy3uS01N9au/vvbaa/3233LLLb6vk5KS+P777/3233TTTVitVt/j3bt3N3mOQ0O4yWSS6eM6iWO6cbKwUG6cFEIIIdqZBGwhuii5cVIIIYTonCRgC9FNyI2TQgghROcgAVuIbkpunBRC9FQet4dtKwspz1Fxj/Gg1cu9J6J9ScAWoodo8xsng4KoCwvD2LcvGpOp3V+PEEI0x+P28N2bWexY5V0g7cN9K5k0I4PIXtYjHClE25GALUQPdjw3TgYBu997D7RajKkpmDIyMKanY0rPwJTeH21QUAe8IiFET3ZouAYo2V/DwmdXMeLcZEack4RWK6PZ4sSTgC2E8DmWGydxu7Fv34F9+w749OAsOPrEREzp6Zgy0jFlZGBKT0cXGdmOr0YI0ZN43B6+PyRcowAqeDwqK7/IYfeGYibNSCc83tJh/RQ9gwRsIUSLDnfjZO633xJUXIIjN7dJWQmAc+9enHv3UvXtt75t2siI+tCd4R3pzkhHn5AgNd1CiOPSEK63NwrXg06Ppy4wj+otQRTsqgCgaE8VHzyzklFTUhh2Zi80MpotThAJ2EKIVmt84+SGXr0YNn48npoabFu3YsvKxpaVhS07G/uOHU1KSwDcRcXUFP1Czc+/+LZprFZv6K4f7Tamp2NMTUXRyY8nIcSRedwevn8ru0m4Pu3yvvz0Uz5n3jWc9d/vZflnu3C7PHhcKr9/souc9d7R7NAYmZpUtD15BxNCHBdNYCDm4cMxDx/u2+ZxOLBv3449O9sbvLOzsW3ZglpX1+R4T1WV/1SBgGI0YuzX7+Bod0a692ZKo7FdXpMQomvwheuVhb5tA+vDdcMnYxqNwrCzepE0MJwf5mdxILcKgMKcSt5/aiWjL0xlyMREFI18kibajgTsbmLhwoVMmzZNVl0UnYLGYPCNdDdQ3W4cubn1gTsLW1YW9qxs3BUVTY5X7XZsGzZg27Dh4EatFmNq6sHAXT/qrbXKzABC9EQthetxjcJ1Y2FxgUy9ZwRrvtnDyi9z8LhV3E4PyxbuYNe6IibNSCc40tyeL0F0YxKwO7G1a9cycuRIRo8ezbJlyzq6O0IcF6U+IBtTUwmech4Aqqriys/3lpY0jHRnZ+MqKGh6ArfbOyq+fTsVn37q26zv1cuvxMSUkYEuIqK9XpYQogM0G67HtRyuG2i0Gkaem0zy4HC+fyubkn3VAOTvqOC9J1Zw6h/6MHBcvIxmi+PWrgFbUZSzgZcALTBXVdVnD9k/HvgUyKnf9LGqqo+3Zx87k9dff52bb76ZBQsWkJ2dTXp6ekd3SYg2pSgK+rg49HFxWM84w7fdVVrqG+m2Z2dj25zlvZmyGc49e3Du2UPVN9/4tukiIzFmpDcqMcmQBXKE6CY8HrX5cH3F4cN1YxEJVqbdN5JVX+1m9eJcVI+Ky+Hh5/e2sWtdEROvSccaJvP7i2PXbrfPKoqiBf4FnANkAFcoipLRTNNfVFUdWv+nx4bruro63nnnHW688UYuueQS5s2b57d/wYIFJCUlYTabmTJlCoWFhX77d+7cyYUXXkhMTAyBgYEMHz6cr7/+2q9NcnIyjz/+ODNnzsRqtZKYmMj7779PeXk5l19+ORaLhbS0NL5tNAuE2+3m+uuvJyUlhYCAANLS0pgzZw4ejwcAm83GwIEDue6663zH5OXlERERwfPPP9/Wl0l0U7qwMCxjxxBx443Ev/givb9ZTN9Vq0j639tEz55N8MUXY+zfH1q4EdJVVETNTz9T8tq/2X/b7ew840y2nTya3BkzKXz2OSo++wz79u2oLlc7vzIhxPHweFTvbCFHOXLdHK1Ow8kXpHLJvSMIjT14o+O+LWW8+/hyspblSdmlOGbtOYI9CtihquouAEVR3gMuBLLasQ9dxsKFC0lKSmLw4MFcffXVXHrppTzzzDPo9XqWL1/OzJkzeeKJJ5g2bRpLlizhgQce8Du+urqac845hyeffJKAgADef/99rrrqKgYPHkz//v197f7xj3/w5JNPMnv2bF577TVmzJjBxIkTufzyy3nyySd55plnuOqqq9izZw8mkwmPx0N8fDwffPABkZGRrFixglmzZhEeHs7111+PyWTinXfeYdSoUZxzzjlccsklXHPNNQwZMoQ777yzvS+j6Ea0lkDMI0ZgHjHCt63hZkpbVtbBGyq3bm3+ZsrKSmqXL6d2+XLfNsVkwtivb32JiXek29g3TW6mFKIT8nhUfnjLP1wPaAjXx1HSEZUUxKUPjGTF5zms+24PqgpOm5sl/93CrrVFTLiqP4Eh8jNBHJ32DNjxwN5Gj/cBJzfT7hRFUdYDecBdqqpubstO7LvvlyM3OkESnj2t1W3nzp3L1VdfDcDpp5+O2Wzms88+Y+rUqbz00ktMmjSJ2bNnA9C3b19WrlzpN8o9ZMgQhgwZ4ns8e/ZsPvnkExYuXMiDDz7o2z558mRuvvlmAB577DFefPFF+vTpwzXXXAPAQw89xBtvvMGmTZsYOXIker2exx8/+MFCcnIya9as4d133+X6668HYPDgwTz77LPMmjWL3377jbVr17Jhwwb5eF60uRZvpty9+2BNd/3UgZ7mbqa02bCt34BtfaObKXU6782U6emYBngXyDGmp6O1yMIUQnSUhnC9bYV/uD79OMN1A51ey6l/6EPq0Ei+fyuLigPeX9JzN5Xw7uPLOe2yvvQdFS3vY6LV2jNgN/ddeehnL2uAJFVVqxVFORf4BEhrciJFmQXMAoiMjCQzM7PZJwwODqaqquo4uty2WtuXnTt3smzZMl5//XXfMdOmTeO1117jrLPOYvPmzZx99tl+5xs2bJjfc9TU1PDss8+yePFiCgsLcTqdvvKNhjaqqtKvXz+/85jNZtLS0nzbzGbvHdW7d++mX79+AMybN48FCxawZ88ebDYbTqeTxMREv/Ncd911LFq0iL///e/Mnz+foKCgDvu3sNlsLX6PNKe6uvqo2vdUnf46WS0w6iTvH1VFU1qKfs9edHv3ot/r/VtbXt70OJcL+7Zt2Ldt87uZ0hUZiSsxEWevRFyJ3j+eVi4H3+mvVSch16n1etK1Uj0q+5erVDS6DSO0N6ixefz0c/5hjz2W6xQ3TkW7AUq3eR/ba118/2YWy7/NIu4kBZ2p+4XsnvT91F7aM2DvAxIbPU7AO0rto6pqZaOvv1IU5f8URYlQVbX4kHb/Af4DkBFvVeODNaQNG9fkCbOzs7EeMoVX0zGs9nNoX1ry3nvv4Xa7ycg4WKLeUAdWXl6OoigYjUa/85lMJr/nuOeee1i8eDHPP/88aWlpmM1mrrzySlRV9bVRFAWLxeJ3HkVRsFqtvm16vd53fqvVyvvvv899993H888/z6mnnkpQUBD/+te/WLRokd95ioqK2LZtG1qtlv3797f6tZ8IJpPJ9wtIa2RmZjK+fqlw0bLucJ1cJSWNZi/xTh3ozN3TbFtdURG6oiJMa9Yc3BYV5R3hzji4OqU+Pq7JKFd3uFbtQa5T6/WUa+XxqPwwP4uK3EYj16fFcfoV/Vo1cn3M1+kM2L+tjB8XZFNZbAOgaj/klus4fXo/+oyIOvpzdmI95fupPbVnwF4JpCmKkgLsBy4HpjduoChKDFCoqqqqKMoovDdhlhzupGa1lrRPz2fN9+MIP/8xkvoPP1zzoyrT6Agul4v58+fzzDPPMGXKFL99V199NW+++SYZGRn8/vvvfvsOfbx06VKuueYapk6dCnhHcXNyco57JpKlS5dy8sknc8stt/i27dy5s0m7G264gd69e/PPf/6TK664grPOOosRjWpnhegMdOHhWE4bi+W0sb5t7upq7Fu2+K9MuXMnNHNDpOvAAaoPHKD6p5982zTBwZj69/fN121KTwe3u11ejxDdicej8uP8bLYtPxiuM44iXB+v+L6hXPbgKH79eCebf94PgK3GyTevb2Ln2ijGXd6XAIvhhPdDdE3tFrBVVXUpinIL8A3eafreUFV1s6Iof6zf/xpwCfAnRVFcQB1wudrKW3iH1/yM+92JrAyZTPzFjxOX3O8EvZIT68svv6S4uJgbb7yR8PBwv32XX345r776Ku+88w5jx47lmWee4ZJLLiEzM5NFixb5te3bty+LFi3iwgsvRK/X89hjj2G324+7f3379uWtt97i66+/pk+fPrz33nv89NNPhIaG+tq89tprZGZmsm7dOlJSUpg5cybTp09n7dq1vpITITorrcWCeeRIzCNH+rZ57Hbs23dgy9rsDdwNN1PabE2O91RUNLmZMkpR2B4ZiS42Bn1MLPqYGHQxMehjY7xfx8aii4hA0Wrb5TUK0dk1hOutyw/OiZ9xWhzj2ylcNzCYdIyf3o/eQyP58b/ZVJd530d3rDrA/m3ljJ/ej9Shke3WH9F1tOs82KqqfgV8dci21xp9/U/gn0dzzmrl4I1HWkXlpIrFON78juWRFxE4/i/H2eP2N2/ePCZMmNAkXIO3Dvu+++6jurqaefPm8cgjj/D4448zfvx4Hn30UW699VZf2xdffJHrr7+e0047jdDQUO644w6qq6uPu3833XQT69atY/r06aiqytSpU7nzzjt54403ANi6dSt33nknr776KikpKYB3ppLhw4fzl7/8hX//+9/H3Qch2pvGaCRg4AACBh5yM2VOTv2NlI1upqysbHK8oqq4DhzAdeCA/w2Vjel06KIiDwbwhjAeG4Ou/m9tWJjcZCW6PY9H5ccFh4Trse0frhtLzAjj8odPZtmH28n+1Vv3XVfp4OvXNtLv5BjGXpqGKVDfIX0TnZPS1ed47Nevn/rlB/Oo/fpRBttW+u3bPPlDkvr0xxQai04n3/hVVVUdWgvdUY52kR6pRWsduU5NqaqKc3+e31Lwti1bcB44gNIGP2sVg8E78h0dfUgAj0Ef6w3mmuDgLhvC5Xuq9brrtfJ4VJYsyGbL74eE6+nHFq5PxHXavbGYJW9vobbC4dsWGGxgwjXpJA1oOjjWFXTX76cTQVGU1aqqjjxSu26xVHqfIWNhyPdk/fY1yo+Pk+70Tq2tQcXiLMF9oIxqQzgBoTFotd3iJQshOiFFUTAkxGNIiCfozDN92zO//55T+/fHlZ+Ps6AAZ34BroIC79cF+bjyC3CXlR3x/KrD4Vu5ssU+BASgry8/0TU7Gh4jUw6KTqnZcD0m9pjD9YmSPCiCKx4+mV/e3+abNrCmwsEXr6wnY0wsYy5JwxAgWaOn61bfARmnnIN68mTWZy7Esuxp33YtHiyOIpyFpdQZIzGHRqHRSK2jEKKd6HQYEhIwJCS02MRjsx0M3fkFuAryceYfDODOggI8rZjqUq2rw5GTgyMnp8U2GovlsAFcHxuLxiTLRIv24/GoLPlvM+H6yv6dKlw3MAXqOfO6AfQeFkXmO1uoq3ICkLUsnz3ZpUy6Jp2E/mEd3EvRkbpVwAZQNBqGTLwUz+lT2bh+NQ70GPB+4+txo7cX4CgoxhkQhTkkqst+lCqE6F40JhOG5GQMyckttnFX1+AqbDmAOwsKUGtrj/hcnupq7Nt3YN++o8U22pAQdPVlJ43rwHXR0d5ylOhoFIPMoCCOny9c/9Y1wnVjqcMiie0TzE/vbGXn2iIAqkvtfPqPdQw6PZ5T/tAHvVEG9HqibhewG2i0WgwBVnQx/aipKMJQdwA93mm2DLgw1OVhryvGFRiNOShcgrYQotPTWgLRWnpj7N272f2qquKprKwfBc/3jog3hPGCQl8YVx2OZo9vzF1ejru8HHt2dsv9iYhoUo7iF8YjI1F03fZtRrQB1aOy5O0tfuE6vYuE6wYBVgOTZw1kx6oD/PTeVuw13qyx8af95GaVMmlGOnF9Qjq2k6LddfuffBqNhsDQaDzBEVSXFWKyF6PDOyetEQfGmr3Yag7gscYSYAmRoC2E6LIURUEbHIw2OBhTv+anKlVVFXdZWdMAXj8K7srPx3ngQLPzfh/KXVyMu7gY26ZNzTfQaNBFRR0sQ4k+JIDHxHinJ9Rojudliy5K9aj8+PYWtvx6cDXG9DGxTOhC4bqBoiiknRRNXN8QMv+3ld0bvOvjVRbVseiFNQyZlMjoC1LRGWQ0u6fo9gG7gUajxRIeh9sdRXVZAQH2ErSKBwATdqjaTV11AATFERDYuuWPhRCiq1EUBV1YGLqwMBgwoNk2qtuNq7jEN/LdJIAXFOAqKgKP5/BP5vHgKvDe0Mm6Ftro9eijotDFxhCkaDiwerXfrCi62Fi0ITL40d00G65P7ZrhurHAYCPn/mkQW38v4JcPtuOoc4EK67/fy55NJUyckU5MSnBHd1O0gx4TsBtotTosEQm4nNFUl+VhdpahUbzTZwWodVCxk9rKQLQhcRgD5E57IUTPo2i16KOj0EdHETCk+Taq04mrqKhJOYqvJrywEHdx8ZGfzOnEuX8/zv37CQBKVq5s0kQxmeqnJoz1Bu9DpibUxcbKzChdSIvh+qquHa4bKIpC/1NiSegfyo//3cLerFIAygpq+XjOaoZNTmLUeSlo9fLJTXfW5QN2njOP+Zvnc1Gfiwg2tv63Qp1ejyUqCacjhtqyPMyuchr+X5vVGtTS7dRorehD4jCYZPVBIYRoTNHr0cfFoY+La7GNx+HAVVjYfACvHw13V1Qc8blUmw1Hbi6O3NwW28jMKF2Dr+a6Ubju343CdWOWUBPn3zqErKV5LFu4A6fdjarCmsW55G4sZtKMDCJ79by1KXqKLh+wXaqL51c9z7/W/YspqVO4ov8VpIWmtfp4vcGIPjoFh60OZ/l+zO4qFAUUBQI9VaglW6nWBWMMjUNvkB/OQgjRWhqDAUNiIobExBbbeGprfWUoGzMz6R0c3CSMe2pqjvhcrZ4ZJS625dUyo6NQ9LIo2YnSEK6zDwnXE7thuG6gKAoDTosnMT2MHxdks39bOQAl+2tY+OwqRpybzIhzktBqZTS7M6sus7FrXTE564tafUyXD9gN6lx1fLjtQz7c9iGjYkYxvf90YtXYVh9vMAVgiOmDva4ad0UeZo/3B7qigMVdgaeokmp9qHdVSH3XnZrqlltuYdOmTWRmZnZ0V4QQAo3ZjDE1BWNqCjaHg8hmVpNzV1W1OAruzM/DVVCIarcf8bl8M6NktTAziqKgi4hodnrChlIUXUQEilZuVDtaqkdlyf96VrhuLCgigAvvGMbGn/bx28c7cTk9eDwqK7/IYfeGYibNSCc8XsqcOpPS/Bp2rSsiZ10RB3KPvAbBobp8wA7ThdE3tC/byrb5tq0oWMGKghW8MvAVouqiCDGGoNO07qUaAywQ0Je6mkqozPPWZQMaRcXiKsV9oJxqQxgBobFoT9D0UzNnzmT+/Pk88cQTPPjgg77tmZmZTJgwgaKiIiIiIk7IcwshRGejtVrRWq3Qt2+z+9tsZhRVxVVUhKuoCNuGDc230enqb8psZlrC+lIUbWio3JTZiC9cL2sUrk+J6THhuoGiURg8IZFeGeH8MD+bgl3e8qiiPVV88MxKRk1JYdiZvdDIaHaHUD0qhbmV5KwrZte6IsoLj7ymwOF0+YBt0VhYeP5CVheu5p0t7/Djnh9xq95p+NweN4U1hRyoPUCwMZhwUzgmXevKPAICg1DNVmqrK9BU5XlnGgG0igeLsxjXgVKqjZGYQ6LRnIDRDJPJxJw5c7jpppuIjIxs8/MLIUR3cdQzozQeBW+YFSU/H1dxMajq4Z/M5cKZl4czL4+6lvpjNKKLifYvRYmN8w/h1p5Re6t6VDKbCdcTrk7vUeG6sZBoMxffNZz1P+xl+ae7cLs8eFwqv3+yi5z13tHs0JjAju5mj+B2e8jbVu4dqV5fTE1585+EaTQK8f1CSB0aCf9u3bm7fMAG7w/XkTEjGRkzkoKaAt7f+j4Lty307VdVlXJbOeW2csx6M2GmMIIMQUccYVAUBbM1BNUSTE1lKbqaAox4F2jQ4cFiL8RZWEytybsqpKYN53KdMGEC+/bt44knnuDll19ust/tdjNr1ix+/PFHCgoKSEhI4MYbb+Suu+7y9cPtdnPvvfcyb948AK644grUQ948Fi9ezFNPPcWmTZtQFIWTTjqJf/zjH6SnpwOwe/duUlJSePfdd3n11VdZsWIF/fv3Z/78+Wg0GmbNmsX69esZNmwY//3vf0lJSWmzayCEEG3Ff2aU5qdGUR0OnAeKWlgl0/u1u6zsiM+l2u04c/fgzN3TYhtNYGDTGzEb14PHxqAJCDjm19sZNITrrMbherQ3XGt6aLhuoNEoDDuzF0kDw/nhrSxfCUJhTiXvP7WS0RemMnhiYo+/TieC0+5mT1YJu9YVkbuxBHtt859s6QwakgaEkzI0kqSB4ZgCj+7+jG4RsBuLCYzh9uG388chf2TT5k2YdCZsLptvf62zllpnLXqNnlBTKKGm0COWjyiKQmBwOGpQGDXlRejrCjHUrwqpx43elu9dft0cjTk4ok0+GtRoNDz77LNcdNFF3H777fQ+ZOU2j8dDfHw8H3zwAZGRkaxYsYJZs2YRHh7O9ddfD8ALL7zA66+/zuuvv87gwYP5+9//zrvvvsvw4cN956mpqeGOO+5g8ODB1NXV8eSTT3L++eeTlZWFodEyyI888gh///vfSU1N5U9/+hPTp08nMjKSp556iqioKGbMmMFtt93G559/ftyvXQghOoJiMGBIiMeQEN9iG4/N5i1DKSjAmZfvPytKfTD3VFcf8bk8NTU4duzEsWNni220wcG+UpTGo+D6mBi0hQdwV1ejCQzslOUoqkcl852tfuG63+gYJlwj4bqxsNhApt4zgjXf7mHlFzl43Cpup4dlC3ewa10Rk2akExwpM5kdL1u1k5wN3psU92SV4nY2P4e/KVBP8uBwUodGkpgedlwLA3W7gN3AqDVi1ptJDU6lzlVHia2ESnslH/7zww7r06OPPnpU7c8991zGjBnD7Nmzee+99/z26fV6Hn/8cd/j5ORk1qxZw7vvvusL2P/4xz+45557uPTSSwGYM2cOS5Ys8TvP1KlT/R6/+eabBAUFsWLFCsaOHevb/te//pVzzz0XgDvvvJPzzz+fjz76iAkTJgDemydvueWWo3p9QgjR1WhMJgzJyRiSk1ts466u9i3I09LNmarN1uLxvvNUVOCuqMC+ZUuTfRHAtkce8ZajhIejjYxAFx7h/Toi3HuzZngEuohwtOHh6CIj2y2M+8L10jzftn6jY5go4bpZGq2Gkeckkzwogh/mZ1G81/sLWv6OCt57YgWn/qEPA8fF99iSmmNVVWojZ30Ru9YVkbe9AtXTfPmXJcxI6pBIUodGEtsnuM1q4LttwG6gKApmvRmz3ozT7Ozo7hy1OXPmMHr0aO66664m+1577TXmzp1Lbm4udXV1OJ1OkpKSAKioqCA/P59TTjnF116j0XDyySezd+9e37adO3fy0EMPsXz5coqKivB4PHg8Hvbs8f9oc/Dgwb6vo6OjARg0aJDftpqaGmprazGb5bdtIUTPpbVY0KalYUxrfspYVVVxl5cfPoQfOADOI79nqXa7ryb8SNojjKselcx3DwnXJ0u4bo2IBAuX3DuSVV/tZvXiXFSPisvh4ef3trFzbRETr+lPUHjXLhs6kVRVpSy/ll3rvKG6aE/LM3+ExQWSOtQbqiMSLSfkF89uH7Ab02u73vymJ510ElOnTuXee+/loYce8m1///33ueOOO3j++ec59dRTCQoK4l//+heLFi06qvOff/75xMfH8+9//5v4+Hh0Oh0ZGRk4HA6/dvpGc8M2fCM2t81zpKWThRCih1MUBV1oKLrQUEwZGc22UT0eXMXFLc6KUrV/H/rqmlaNhPvOeSxhPCKiPoAfOYyrHpWf3t1K1i+HhOsZEq5bS6vTcPIFqaQMieD7t7Ipy/dOGbx/axnvPbGCsZekkT4mtlOWBXUE1aNSuLvSd5Pi4Wb+iEkNIqV+pDok+sQPBLYqYCuKEgmgqmpR/eNBwGXAZlVV3z1x3Wt7jcs0VFWlzlVHqa2USkdlkxsAFUUhyBBEuCmcAP2Rf2t0Oh3Yy/IwO8t9y683qNFYsNfVYAw4+juDn376aTIyMli8eLFv29KlSzn55JP9yjJ27jxYyxccHExsbCy///47EydO9L3eFStWEBvrnR+8pKSE7Oxs/vWvf/lKPdasWYPrSFNZCSFEG7PXubBXqaiqKuEBUDQa9FFR6KOiCGj0CWKDzMxMTj/9dDw1tbhLinGVlOAqKsZVUoy7uMQbzktKcNf/7SouPqFhXBseTnavi9lrOvgLQ3JkLSNj9mJbU9vuZSpdXVRSEJc+MJKVX+Sw9ts9qCo4bW6WvL2FnWuLmHBVfyyhxo7uZodwuzzs31ZGTv3CLzUVjmbbaTQK8f1DSR0aScqQCAKD2/d6tXYE+wPgv8AbiqJEAD8DecCtiqLEqar6wonq4InkVz7idlJmL6PMVobL4w2YqqpSYa+gwl5BgC6AsADv7CMapfn6HL3egD4qGYfdhrM8D7OrgoafI4GeatTSbdTogtCHxGMwtn5VyD59+jBr1ixeeukl37a+ffvy1ltv8fXXX9OnTx/ee+89fvrpJ0JDQ31tbr/9dp555hn69u3LoEGD+Mc//kF+fr4vYIeGhhIREcHrr79OYmIi+/fv5+6770Z3gub3FkKI5mxfVciSt7fgtKnUbF3D6AtTiUsLPfKBPZyiKGgtgWgtgRjqywNboqpqy2G8PoAfaxj32B1sCRzH/kbhOrpgBSmZC8j/8JCBq5ZGxsPrHzeMjEdEoLGcmI/uuwqdXsspF/chZUgkP8zP9o3O7tlcwntPLOe0S9Poe3JMj7hGTrubPZu9M3/s3liCo66FmT+MWpIGhJFaP/OH0dxxlQutTVKDgd/rv74E2KGq6kmKolwI/A3okgG7Mb1WT5Q5ioiACCodlZTaSqlzHpzltM5Vx/6q/RRqCr2zjxhDWyw5MRhNGKJTsdtqcZXvJ9DjvWFBUSDQXYmn2LsqpDE0Fr2+db9RPfzww8yfP9/3+KabbmLdunVMnz4dVVWZOnUqd955J2+88YavzZ133klBQQE33HADAJdddhlXXnkl2dneVcw0Gg3vv/8+t912GwMHDqRPnz688MILTW58FEKIE8E7W8J2Nv6037ctf0cFi15YS0L/UE6+IJWY1OAO7GH3caLCuLO4hG2JF7I//jTf8dEFK8jYsgCFpjeVHXOZSn3o7olhPCY1mEtnn8TyT3axfsleUMFe6+L7t7LZubaI8Vf2xxzUdVeYbkldtYPdG4rZta6YvdmHmfnDoidlcAQpQyNJ7B96XDN/tCXl0LKIZhspSi3QX1XVPYqiLATWq6r6hKIoicA2VVU7rOq+X79+6tatW5vdl52d7ZvP+VjUOetnHzlM+UiYKYwAXcBh/2PbaqvwVORhVv1rg9yqQp0hzLv8uu7E/5ZVVVWFtYcsbtDY0X4fZGZmMr6Z5ZqFP7lOrSfXqqmKojq+eX3TYW9EAkgaFM7J56cS2avn/ew6nM7wPaV6VH56byubfz4YllMSPYxKPoBacvwj40ejpTCeU1JCv2HD0QZZ0VitaIOC0FqtaIKCvOUqbbh+RXvI217GD/OzqSw+eB1NgXrGXdGXtJHRx3zezvD9BFBZUucr/cjbXt7iuk+WMKPvJsXY3m0380drKIqyWlXVkUdq19oR7O3AHxRF+Qg4C++oNUA0UH5MPewCAvQBJOgTcHqclNvKKbWVNls+YtKZCDeFE2RsvnzEZLaCuR911RVQlUeA6v2PoVVULM4S3AfKqDaEExAag1Yr5RlCiO5t17oifpif7fcxb+qwSIgqxlAZzdbfC3xvrLkbS8jdWELvYZGcdH4K4XGWDuq1aExVVX5+b5tfuO47KppJMzNavKGxych4cXF9AD/+MN7SyHgQkP9BC9PzKoo3dFssaBoFb+/fVrTWoPpgHoTGajn4uKGNxYJyAlZyPpy4tFAue3AUv328k00/ez/5sdU4+XbuZnatLWLcFX0JsHSd0WxVVSnNq6mfTq+4w2f+aEutTXOPAe/iLQX5QVXV5fXbJwNrT0THOhO9Rk+kOZLwgHCqHFWU2kqpdR4cjba5bOyv3k9BbQGhplDCjGHNlo8EWIJRA4OorSpDU52PqX5VSC0eLI4inIWl1Jki61eF7BwfcQghRFtxuzz8tmgn6384OFWoRqtw6tQ+DJ6QwE8//cT4izMYPjmJlV/uZvuqQhqqDHauLWLnuiLSRkYzakpKu8wCIJqnqio/v7vNF/AA0k46fLiG4yxTacMw3uhJ8FRW4qmshFaUrDRHY7F4w7ilaSj3+9tiPWS7Fa3VinIM9zwZTDpOn96P1GGR/Lggm+oy7/LeO1YfYP+2MsZf2d+7pHcn5Zv5Y613Or2KorrmGyoQkxJMytAI78wfUV3r/3yr/mVVVf1YUZReQBywvtGu74GPTkTHOiONoiHYGEywMdg7+0hdKRWOCl/5iNvjpri2mJLaEqxGq3f2kUPKRxRFwRwUhmoNpaaiGH1tIQa8c516V4Us8K4KGeBdfr2z/4YmhBCtUVVq45vXN1GYU+nbZg0zMfnGgUSnBPm1DY0J5KzrBzDi7CRWfJHDrrVF3h0qbF9ZyI7VB+g3OoaTzk0mKELmBW5PLYXrM649fLg+Wm0VxvdkZRETZMVTWYW7qgpPZaXvb09ty1O6tZanuhpPdTUu8o/cuBkas7l+RNziDd/NjKJ7/25a4pLQ28LlD5/Msg+3k/2r9/nrqpx8/dpG+p0cw9hL0456ee8Txe3ysH9rmW86vdrKFmb+0Cok9AslpYNm/mhLrf7VSVXVQqCw4bGiKH3w1mKfmIKqTi5AF0C8NZ5oTzRltjL/8hFUKu2VVNorMelMhJnCCDYG+5WPKIpCYEgknqBwaiqKMNQdQF+//LoBF4a6POx1xbgCozEHhUvQFkJ0Wbs3FvP9W1nYaw6WhCQPjmDSjPTDBoDweAvn3DSIA7mVrPg8h9xNJYB3BGzLr/lsW15A+pg4Rp6ThCW09TMziWPTUBbSJFzP7Nh5rg8XxjdnZjKihdpi1eXCU12Nu6oKd2UlHr+/q7x/HxLK3VVVuKsq8VRV46k6/P0DreGprcVTW4ur4NiOVwICSLRaCYgczObws7BrvFMBb11eQO6qvYxKLSYhQX8wwDceRQ8KQmM4ceUkDpuLPZtL2bWuiNxNR5r5I5zUYREkDYzAGNA9SmVbOw/208BWVVXnK96k9y0wCahQFOXsRiUjPY5OoyPSHOk3+8ih5SN51XkU1hYSagwlzORfPqLRaAgMjcYTHEF1WSEmezE63AAYcWCs2Yut5gAeaywBlhAJ2kKILsPj9rD8sxzWfJPr26ZoFE65qDdDz0xs9c+zqKQgptwyhIJdFSz/bBf7tpTVn19l88/72fJrPgPGxTHi7ORuOZtCZ+AL1z81E67b8QaztqTodGhDQtCGhBzT8arbjaempj6MV/r/XV3l99hdVek/il5d7S1NacVEE4ftQ10drro6Qg58zyjdr2zrM43CmFEA2Nx6ft4eS+zPy0jb8TE6d9PxUMVoRBNkJVyrY3fc62hDQ9GGBKMLDa3/OqT+71C0od5rpQ0ObvHm0LoqBzkbislZV8Te7DLcrsPM/DEkgtQhkSSkh6LTd7+y2Nb+mnAl3oVlAM4BhgKj67c/C0xo8551MYqi+MpHbC4bpbZSyu3l/uUjdcUU1xUTZPTOPmLWmX1vMBqNFkt4HG53FNVlBQTYS9Aq3m9ME3ao2k1ddQAExREQGNRiP4QQojOoLrPz7bxN5O+o8G0LDDEy+YYBxPYJOaZzxqQGc+Edw9i/tYzln+/yndvt8rDhx31kLc1j8IQEhp2ZhMnSOT4a7w5UVeWXbhau24Ki1XpLNoKCgPijPl71eLwj2IeOkFd6R8gPhvLKpiUu9aPruN2+8+ldtQzYMp+o4nVs6XsFToN35p382DGUhqaTvuVtwsr9Z11T7XbcRXZ0QF1BK4fRNRq0wcG+8G0PiafQlEq+J5YSmwWV5n9xtoaZvDcpDosgpndIt1/ds7UBOxrYV//1ucAHqqquUBSlFFh1QnrWhZl0JuIscUSZoyi3l1NaV4rT4/TtbygfMeqMhJvC/cpHtFodlogEXM5oqsvyMDvLfKtCBqh1ULGT2qpAtMFxGAP876ZXVRVVBY/Lg8et4nZ7//a4VVSPihsV1SKrpAkhTqy9WaV89+Zm6qoO/tzrlRHGGddmEGA9/hHm+H6hXNx3OHuzSln+2S4O5Ho/qnc5PKz5Zg8bf9rPkEmJDJ2U2KELTXQHDeG68VzlaSOjeny4bguKRoPWYkFrsXAs36WqqqLW1jYpcYmrqqJPSQ0rt8C+Cm/ItpvCWDf0NnrVbaJv4XcolaW4KyvhGFZuVj0eKhwBFHn6U6QMptrTC1ooZw+s3k9k8XoiSzYQrKtGtzYExyeh7D/OkfKuoLUBuwRIwhuyzwLub3S8pLUW6DQ6IgIiCDcdnH2kxlnj22932b3lIzWFhJhCCDOFYdB633x0ej2WqCScjhhqy/IIcFWiosWjatF4FFzFRdQq1aC3gKocDNJH+LipzFmLNcyI3tg9apyEEJ2Hx6Oy8sscVn212zf7h6LAqAtSGTE5CaWNb4LrNSCcxIwwdm8sYflnuyjZ513Uy2lzs+rL3Wxcso+hZ/Zi8IQEDCb5mXe0VFXll/e3Nw3X12ZIuO4EFEVBCQxEExiIPibGb18wcIGqsmPVAX56b6vv/oc9AQMpHzmSSTMyiO0TjGqz4a6s4vcfvmdEWhqusjLcZeW4y8txl5XhLivDVV6Gq7ycktpACrWJHAjOoC6ghVlKVA/BlTlEFq0nong9Zluxb5cHcJSVQU5O617gISPlx1u+0t5a+xPnI+AdRVG2AWHA4vrtQ4EdJ6Bf3YqiKAQZgwgyBvnKRyrsFXhUbwmI6oGKmiqqq+oI0Jgxa81oVV2jUegIaj3hzZ/c3Xx9U0tcDjdlBbWYAvUEhhrRyg9JIUQbqK108O28zezfWubbZg4ycNb1A4jvd+KWPVcUhZTBESQPDGfn2iJWfL6LsgLvcJq91sXyT3ex/oe9DJ+cxKDT4zvNKm+dnS9cZ+7zbZNw3bUoikLaSdHE9Q0h839b2b3BG3Yri20senENQyYmMvrCVPTRUbjj4jCfdJLf8W6Xh32NZv6oa2nmD0UlOtRJfGA5Mep+9FVFuIKcuMPjcJeZcZeVHdsNoR6PL+R3xVDe2oD9VyAX6AXco6pqwzBsLPBqm/eqG1BV1Teq7HE3lGyoeFwqZncIRneQt/i/mQFnJypOnE13HMbnX33CDTfPoDC3Aq1Wg0aroNFq0GoVNDrvCHfjaXFsNU7stS4CQwwEWA1SNiKEOGb7t5bx7bzNfj9j4vuFctb1A9rtpkNFo9BnRBSpwyLZvrKQFV/kUFk/v66t2smvH+1g3fd7GHF2MgPGxqHVS0hsiaqq/PKBf7juI+G6ywoMNnLunwax9fcCfvlgu3c2DxXW/7CX3E0lTJp5cKVjh81F7qYSchpm/rC5mz2n3qglaWA4qUMjSRoYjuEIM3+oTifuigpvYC4vb3Gk3Pu4vFOH8tZq7TzYLryLzBy6/e+tfqZuQlVVPJ764Ozy+IVo9yGB+nht3LSesy4Yz4hhJ/HFR996A7NOQaNVUD0uFGclRurQ4Mai8f5mGq7Noc4YRkBILNpDJrB3Kw5w6LDXHVyNsrrMTl21E2uo6Yj/QQBiY2O55ZZbmD17tm/bE088wcMPP8zChQuZOnWqb/uVV17J3r17+fnnn4/7WgghOh/Vo7J6cS4rPt91cDIEBU46N5mR56V0yE1MGo1Cv5Nj6DMyiq2/F7DyyxyqS70LcdRWOPjl/W2s/TaXk85Lod8pMfIp3iF84XqJf7g+U8J1l6YoCv1PiSWhfyhL/ruFPVmlAJQX1vLxnNWE9IYvNq9n32Fm/giw6kkZHEHK0EgS+h/dzB+KXo8uIgJdRESrj+nUobwVWl2UpihKNPBnIAPvuGsW8C9VVQ+0WW86kKp6bwRsCMyNbxD0uD24XQe/bmsNo81oVBzYsXnqcCsu3vjwNa6YOZ1PP/yE3/dnMnTgUMJMYQTqvbOPqKrVt/y6VvH+lqlVPFgcxbgOlFJtjMQcEo2mfilXjVbBGmXGXueiutTm+0/kdnooP1CL0azDEmI67MjOhAkTWLJkiV/AzszMpFevXixZssQvYGdmZnLDDTe0+fUSQnS8umoH37+R5XujBu8b8JnXDiAxI6wDe+al1WrIGBNHv1ExZC3LY9XXu6mt8I6wV5fZWfL2FlYv3s2oKSmkjYrp9jMatIaqqiw9NFyPkHDdnVhCTUy5dQhZS/NYtnAHTrsbVYWyHVBGSZP21nCTb3nymN7B7fr/pFOH8lZo7TzYY/DWXRcCv9VvvhL4i6Iok1VV/a3FgzsB74hzo8Dsajra3JobBI+WRuMNzpr6UWdfyUb915r6r/3LMyy4PW7yy/P5fNHnvPXZW9Taavj4fx/T+7HeVDmqMGgNfPfRdzz3xHMUFRUxceJEJo4f5/fcubtz+etjt7N87Saqamrp168/9913H5deeinGAB2GuECSk5K5fNpV5Obu5ovFnxEcFMwjs5/kvCnncOc9t/PFl18QGxvLv/71L8466yzAG7Bvv/127HY7RqMRu93Ob7/9xosvvsgrr7zie/5t27aRl5fHxIkT2/SaCiE6Xv6Ocr6Zu5macrtvW2yfYCbfMJDAkM618ppWr2HQ+ATST41l08/7WfNNrm92k8piG9+/lc3qxbmcNCWFPsOj2vRGzK5EVVWWfridDYeG6+skXHc3iqIw4LR4EtPD+PG/2ezfWu63PzzeQurQCFKHRRIeb+lSJaQnKpS7y8u9wbysHLZuadV5WzuC/TzwLvBHVfXemacoigZ4DW/pyKmtfiVtTPV4f9jXVDioKbdTU1H/p9xBwmiFoj1VbR6cFY03GGsbheSGIK1tFJ6P9ZtSq9Gy5MslpCSncOYpZ+KodvDnmX/mjgfvQK/Xs2rFKm6ZdQu33n8r06ZNY91v63jkwUcAqAmIR19XSHVNHedMGMOT99xMgMnIO5/9wFVXXUXf3ikMHjYcjUaLolF4/c1XmX3fw9xxy13M/98b3PbXP/LBwnFcfOElPPTgI7zw979x1VVXsWfPHkwmExMmTKCuro7ly5czbtw4fv/9d8LDw7nmmmu4/fbbKSwsJDo6miVLlhAQEMDo0aPb8tILITqQqqqs+24vv32yE9Vz8Ofq8MlJnHxBSqcOYjqDlqFn9CJjbBwbM/ex9ts92Gu95XJlBbV8O3czq+NzGXV+CilDIrpUqDhevnD948Fw3Xu4hOvuLigigAtvH0bWsjzW/LSVgSf3IXVoJMGRAR3dtXZ11KG8lT8bWhuwhwIzG8I1gKqqHkVRXgTWtvIcJ4S7Aj5+fk2z++JOCm0SrjfuHNYe3WrWpIk7W9127ty5XH311VgNVi479zLuDbyXVT+uYszZY3j7P29z8riTmfWXWQBMmDaBpb8t5f3/vo85JBI1OILepggyBqSjr18V8uHbr+Wr75bwxftvMThGh00xoXpcTJo4ntvu+BOqR0tcwoO8NvefJCenMu3iywG440938+abb7Jp0yZGjhxJnz59SEhIYMmSJYwbN44lS5Ywfvx4zGYzI0aMIDMzk8suu4wlS5Zw6qmnYjR2rtEsIcSxsdU4+WF+tm8mAgBjoI4zZmaQPKj1o0UdzWDSMeLsZAaensD67/ew7oe9OOtv5CrZX83Xr20kKsnKqAtS6ZUR1u2DtqqqLPtwR5Nwfdb1Eq57AkXjHc0ucm9n2PheHd2dbqW1/3sqgJRmtqcA5W3Wm2MQqFUYb9URpz/MD0EFNDoNemPXmJ5px44dLFu2jOnTpwPej3OuuvIqPnr7I/qG9mXvjr2MOGmE3zEZwzO8x5bvoMxehkcfyP0vLqD/hMsIzTgdS9oYVm3IZk9ePhoFTNhQVA/D0+LRF2dBaTbhlmrMAWYy+mf4zhsc5K2l3L1zH576EauGOmzAF7ABxo8f79uemZkp5SFCdBOFOZV88NRKv3AdnRLEZbNHdalw3ZgxQMeo81O55slTGT45CZ3h4NvhgdwqvnhlPYueX+M37WB30xCu1/+417et9/AozpRwLcRxa+0I9nvAPEVR7gF+xXuT41i8y6S/e4L61mrBWoWTAnU4jFrqkoLR9A0hMNREpaeAiAQLiqZRuUZWx/a1NebOnYvb7aZXr4O/TTaMxOftz0OjaAg1hZIUlESJrYRqR7WvncPtoKCmgCfufoJfl/zKnDlz6NsnDZ3q5MabbqbO4T/ljl7v/RYw4MLgqURRVEIM5Zg0ldg8QSj16wjZahyU7q8mMNTIhAkT+NOf/kRZWRnLly/njTfeAOD000/n9ttvJzs7m8LCQiZMmHBCr5MQ4sRSVZUNS/bx60c7/GZGGnJGIqdc1ButruuHMJNFzykX92bIpETWfJPLpp/2+24Az99ZwSd/X0t8v1BOviCV2N7BHdzbtqOqKssWHhquIznz+gyZWUWINtDagH0P3hUb32h0jBPvHNj3nYB+HROD3Y1hWynaAzVYxydQE6Q0+S38aMo0OoLL5WL+/Pk888wzTJkyxW/f1VdfzZtvvklGRgbLly/HYrBgMViwu+1sXb/Vr+2a5Ws4b9p5DJo4CIvBghkzOXv2kT5wMK7IgTjqqlAVLU70eFTFtxw7gAaVIG0RAZoKipWDy7F7PCpVJTaGpo/Abrfz/PPPExkZSe/evQEYO3Ysu3bt4n//+x8Wi4WTDpm0XgjRddjrXCxZkM3OtUW+bYYAHZNmpJM6tIVV3Lowc5CBsdPSGHpGL1Yv3k3W0jzfLxX7t5bx8d9W02tAOCdfkEJUUlAH9/b4+ML1D4eG6wESroVoI62dB9sB3K4oyv1Ab7xhe4eqqi2sPt9+HFYV68REqpflodq9o7Pucjvln+zEfZEVd5UDTaC+y9wZ/uWXX1JcXMyNN95IeLj/6o2XX345r776Ku+88w5jx47lmWee4ZJLLiEzM5NvPv8GgNjAWEptpSSlJvHDVz8w8eyJ6PQ6Xv3bq9TZ6qhz1eHASYA1FEWjRR8UBbGDsdlqcNVVAwpuvKU0esVBiDYfAA0HR77jY5PplZjEKy+/zHmTJ1FdvB+tyUJAgIXhw4fz8ssvc9ppp6HTydLEQnRFRXuqWPz6Jt9CLQCRvaxMvnFgt78ByhJq5PQr+jHsrF6s+mo3W34r8N3QuWdzCXs2l5A6NJJR56cQHm85wtk6H1VVWfbRIeF6mIRrIdraUf1vUlW1VlXVjaqqbugM4RpAVSD4rGRi7x9F0OQkNOaDoU71qLgr7DgLanBX2v3ueu+s5s2bx4QJE5qEa4Bp06aRm5tLdXU18+bN49VXX2Xw4MF8/PHHPProowCEBYTRO6Q3L/39JaIio5hxwQz+dPmfGDxyMMNHD8fuspNTkcO2sm24VTcOtwNFUTCZrVjCY0FR0IYm4IxIpyYwgRq9d4njQG0xZk0ZDUtPjjnlNKqqqzl51CQ0dXZMFbugYANjRw6kqqqKsaeejNNhb/IahBCdl6qqbPp5Px/NWe0XrgeNT2Dq3SO6fbhuLCg8gIlXpzP90ZPpe3I0NBqj2bWuiPeeXMG3czdRVlDT8kk6GV+4/v6QcH2DhGsh2prS0hR2iqJ81tqTqKp6QZv16Cj169dP3br1YHmEx+GmZnkBVT/vo2SCgX69+hxsrChoLHq0Fj1KD/lh4nA7KLWVUm4vx+1pYclTjZ4gYxDBhmBMOlOzd8173G7sddW46mpw2PS4Pf6zg2gVB1ZNMQZNnd92JzocmgBUfSC6AAsGUyAaTfte++zsbNLT04/csF5mZqbvxk3RMrlOrdcVrpXD5iLzf1vZvrLQt01v0jLhqv6kjYxulz505utUmlfDii9y2LnGf201RYF+J8cw8ryUdv0F5Givlaqq9cvF96xw3Zm/pzoTuU6tpyjKalVVRx6p3eE+w2+6pE8XoDFosZ4Wj2V0LGWbs0CrgYbVF1UVT5UDT7UTTaAerbX7B22D1kBMYAzR5miKKotwaV1UOir9wrbT46SkroSSuhL0Wj3BhmCCjEGYtAfDtkarJcASDBbvTT72WidVpTZfjaJbNVDujsPoqcaiLUGreOeX1eNC76kCexXYwa0q1GlMuHVmNEYLBrMVnU7fzldFCNFYyf5qFv9nE+WFBz+YDE+wcPaNAwmJNndgzzqPsLhAzp41kKK9Vaz4PMc3o4qqwpbfC9i2opD+Y2IZeU4y1jBTB/fWX3PhOrUHhGshOlKLAVtV1WvbsyNtTdFr0Bi16GPMeGpdeKocqK5GQbvagafGgcasR2s1oHSDu+EPR1EUAjQBWC1WYtQYap21VDgqqLRX4jk4vTlOt5PiumKK64oxaA0EGYIINgZj1Br9RraNZj2GAB21lQ5qKxy+WU7sqgW7KxCDthqrUoxW8V9aXquoBKh14KwDZwlUgx09Lq0ZDIHoAqwYjAHdfu5ZITqL7F/z+PndbbicB/+vZoyN47RL09AZusbUpu0pMtHKeTcPpjCnkuWf72Jv/VLxHo9K1i95bPktnwGnxTPi7CQCgzt+HQBVVfn1451NwvVZEq6FOKG6/V1oiqKgDdSjMevw1NUH7YY3EhU8NU48NU40Zj0aqx6Nvvu/oWgUjW8GktjAWGqcNVTaK6l0+Idth9vhF7aDjcEEGYJ8YVtRFAKDjZgC9dSU27HVOOuPVHC4rZRpgzGZVfBUozhr0HvqMOBq0h8jTozuCqirgDpwocGhBODRm9GaLBgCLGi13f5bVYh25bS7+fndrWz5vcC3TWfQMP7K/vQ7OaYDe9Y1RKcEccFtQ8nbXs7yz3aRt70cAI9LZeOSfWQvzWPg+ASGn9WLAKuhQ/roC9ff7fFtSx0q4VqI9tBjUouiKGjNejQBOlSbG3eVA7XRnNCeWieeWieaAB0aqwFNDxm50SgarAYrVoOVWNUbtivsFVQ5qpqE7aLaIopqizBqjb6abaPOiFanISgiAJNFT3WZHVf9dfW4PdRWgsEUjCU8Cp1ei9Nh904RaK9G56rFoNo5dIIXHR50ag04asBRhFoBNsWIS2dGMQRiCLCiN3auj2CF6EpK82v45vVNlOYdvEEvLC6QyTcOJCw2sAN71vXEpYVw0V+HsW9LGcs/20VhTiUALqeHdd/tYfPP+xk8MYGhZ/TCFNh+5XCqqvKbhGshOky3D9iqqvqVGyiKghKgQzFpUe31QdveKGjXufDUuVBMOrRBPSdog3/Y9qgeqh3VVDoqm4Rtu9t+MGzrjN4yEkMwRpOR0Bgttmon1eUHZ21x2FyU5rkwWw2YQ4wEBkcA3tXfPG43dXXVuG3VaFw1GDw2dPjfjKkoYMIOLju4yqC28c2TZnQmK4aA5m+ebOkmXiF6qq3LC8h8ZyuuRj/3+o+OYdwV/brMaredjaIoJKaHkdA/lNxNJSz/bBfFe70LgDntblZ/ncvGzP0MPSORIRMTMQSc2LfehnC9tlG4ThkS4Q3X3bwcUojOolsHbL1eT11dHWZz05t0FEVBMenQmHR47C7cVU5U28HyBdXmwmVzoRi1aIMMKAZtj6oL1igagoxBBBmDfGG7wlFBtaPaP2y77BS5vGHbpDMRZAgiyBxEuDmQmnIHddUOX9vaKge2WieBId6yEkVRmtw8qaoqDnsdzrpqVEc1encdRhxN+ud/82QhnvLGN096R7l1egNOp1Pm4xYCcDnc/PLBdrKW5vm2afUaTr+iL+mnxnVgz7oPRVFIHhRB0sBwdq0rYsXnOb5PCRx1LlZ8nsOGH/cxbHIvBo1PQH8CBnBUVeW3RU3D9eQbB0q4FqIddevkERUVxf79+4mPjycgoOUb5zRGHRqjDo/D7Z1lpK5R0La7cRXVoRjqg7axZwVt8A/bbo+baufBke3GI8Q2lw2by8aB2gPesB0QhDXQiq3cjdPeUDbiXQ2yrsqJNczUZMRMURQMJjMGkxmIAsDlcuKorcJjr0FbP8qtVfxHpjXN3DxpU3XsK7NTnreL3a4KevUdhkYrI3Si5ykvrGXx65so2Vft2xYSbWbyjQOJSOh6i6V0doqi0HtYFClDItmxupAVn+dQccA7hamtxukt3fh+LyPOTmLAaXHo2ujeH1+4/lbCtRAdrdUBW1EUMzAUb+rx+5+qqurHbdutthEU5F3ONi8vD6fTeYTWB6luD6rNjcfRdN5oRatBY9Ki6LV+Cw90BTabDZOp7WqXVVXF7rZT56rD7rKj0nw5hl6jx4QZxa5FbTypyE7QG7UYzbqjWmlTVQ24nA48Thu4Heg8DrQ0M8e36iGwZAMj1sxB86uLSgLJMWVQGz0Ca9oYUoaMI9AacnQvWoguZsfqA/z432yctoP/R9JGRjH+qv4YTN16jKXDaTQKfU+Koc/wKLYuL2TllzlUldgAqKt0sPSD7az7bg8jzkkm/dTY4wrBqqry+ycSroXoLFr101VRlDOAd4Gmywt6l/brwGHB/WzcdCtWSzoWSzoWazpGQ7RvlDkoKMgXtI+Wq7iOqp/2UbOmENz+4VEXbSZoQiIBgyJRtF0jaWdmZjJs2LATcu5qRzWZ+zL5Zvc3LNu/DKen6S80OreBs0uvJGHXEPAcvGaGAB2jzk9h0OnxaI7x5puivN3s3ZCJI+d3QkvWkuLcjkHxD91B1DDEthJyV0Lua7i/U9ihS6UkdAjapNHEDRpPbK+0Y3p+ITobt9PDso92sDFzn2+bRqdw2qV9GXBaXI/7JK4jabQa0k+Npe+oaLJ/zWfVV7upKfeudFtdZuend7ay9ttcRp6bQr+To4/656A3XO9izTcSroXoLFo7fPES8CXwgKqqeUdq3L6cHDjwFQcOfOXboteHYrH0x2JJx1r/d2BgHzSao5sqSRcRQOjUNKyTelH98z6qVxRA/VzarsJaSt/biva7XILGJ2IeFtXt59I+HIvBwpTUKUxJnUKVo4ole5fwze5v+DXvV1web8mNS+vgi8g3CbJEMGb3H0gqHwB4axOX1teGjrusL/H9Qo/6+SPjkomMmwnMBMBWV8OWDcso37YUY/4qkmo3Ekal3zFaRaWPeyd9indC8cewGooIRa/PYJ27gAGnXYze0PHz2ApxtCqL6/jm9U0cyK3ybQuKMHH2rEFE9rJ2YM96Nq1Ow8Bx8fQ/JYbNv+SxenEudZXee0wqi238uCCbNd/kctKUZNJGRLfqk72D4TrXty15sIRrITpai0ul+zVSlBpgsKqqO098l45Ov35G9f9eTThiO0XRExjYu1HwTsdi6Y/B0NygfPPcVQ6qlu6n5rd8vyn+ALTBRqynJxB4UrS3fKQT6oilUCvsFfy450e+yf2G5XnLcakH69t7lWUwZvcfCLZF+h3TZ0QUp07t06aroakeD3m7s8nbkIln73KiytaT5M5Fo7T8/V9KENsiJxN+6jX0GTIWpZ2XeO9sPG4PRXur2b+1jP1byyjYV0py/2iSB0WQmBHWrlOQdTXt+X9v17oiflyQjb324P+13sMimXBNOsYTPHvF8eppyzU77W42/rSPtd/sabSOgFdYXCAnn59KytCIZj9tyMzM5PTTT+f3T3exZrF/uD57loTrBj3te+pYyXVqvdYuld7agP0t8A9VVb86YuN21q9fsvrjj89SXb2Fqupsqqu34HZXH/nAekZDNBarN3RbLP2xWtIxm1NQlJZDsqfWSdWyPKqX5fnNPAKgseixnpZA4OgYNMb2fzNTPR6cDjtOmw2n3Y7TbsNps6F6PGzZu49JZ57V7n1qUG4r58e9P7I4ZzErClbgVt1oPFoG509gxL6z0HsajRbrVAadGcup5/ZrsxuADlVZXkLu+p+o3vErlgOrSbFlY1Hqmm2bq0kgL+lCUiZcS0wPKSNRPSrF+w8G6rzt5ThszdS6A4pGIbZ3MEkDw0kaFE5YbKCUIDTSHm9ebreH3xbtZH2jFfs0WoVTp/Zh8ISELvHv0VPf5B11Ltb/uJd13+/FUef/nhLZy8qo81NIGhju92+4ZMkSTBW9JFwfQU/9njpacp1ar60D9h+AJ4EXgY2A36/aqqquOcZ+Hrd+/fqpW7dubdwXbLZ9VFVnUV21herqbKqqt2Cz7T3MWfxpNCYsgX29o93WdKyWDCyWfuh0/h+temwuqn/Pp3rpfjzV/qMPSoAO65g4LKfGoTEfHNlTVRW3y4XTbsPVKAA77f6BuOFrXxu7Daet0deHPPa2s+Ny2Ft8XVqjidMuu4ohZ52HTt+xo42ltlJ+2PMD3+R8w8rClQTYrIzecwFpxf7fs3ZzNVFneDh3wjjCAsJOaJ/cLhc5m39n17f/YWjNL0RR2qSNR1XINg6iJn0a6ROvwhp8YvvUnlRVpSy/lv3byti3tYz928qw1zRdebM1rOEmkgeGkzQ4gvi+ISfsl6Su4kS/eVWV2vh27iYKdh0sg7KGmZh840CiU47tHpSO0NPf5G01TtZ9v4cNP+7zzbzUIDoliJMvTCWhvoTug1eWUJx1cL+E6+b19O+p1pLr1HptHbA9h9mtqqraYe+ehwbslrhcVVRVewN3dZV3pLu6ZiseT8uB9FA6TRQGTS+0ahwadyyqPRK3zYy7zklAoYnQ4lD0bv86b5fqZI9rCztq11FTV47T7h1N7khBkVGMufQq+o89HY2m44NPcV0xP+T+wDe537BvWyljcv5ARK1/2c/ekC3UjtrF+EGnMqnXJIKNwSesP5mZmZw2dixZv36JbfX/GFCeiVlp+n1iU/VsDjoN3fDpDBh7ITp9xyyHfKxUVaWiqM43Qr1vW7mvHrQlgSFGEvqFEt8vhF37txJlTiZ3UwmFuytpYRIZdAYNCf3DSB4UTtLACCyhPa+u/US+ee3eWMz3b2X5/TKUPDiCSTPSu1zZjrzJe9VVOVjz7R42Zu7D7fR/v4hLCyE0xszmXw7eDpU8OIKzbxyIVi/h+lDyPdU6cp1ar60DdtLh9quqmnu4/SdS37Q09feflhzjKHAdHk0JmoBitOZy9EGVGINr0Qe2fko/t11DXamRuhIT9pIAQuoG0Fc9A6s22q+dy+NkV9V6tlSsoM5d1cLZ2obOYERvNKI3mdAbTeiNRqrLy6guKfZrF9krmdOmzyR56IhO8/FxUW0R3+Z8x7qfdhOTNQiT6+CyzW7FzcaYTNYn/sDwXkM5O/lsJiROaPOwfegPmtrqCrJ+fAdD1ocMqFvTZA5ugGJC2BE1mYgx19B70Kmdtl67sqSO/VvL2b/NG6qryw7/C2aAVU98v1Di+4aS0C+U4KiD88k3vk61lQ72bC5h98Zi9maVtlhKAhCRaCFpYDjJgyKISg5CcxRTNHZVJ+LNy+P2sPyzHL+b2xSNwikX9WbomYmd5v/00ZA3eX81FXZWL85l8y/78biaf69OHhTO2bMGSbhugXxPtY5cp9Zr04DdmSWGhah3nDm2Tc+pNbkICLcTEG7z/W0KsXOYsmw/qgc8VVYCq3tjqe2DsSoRU1UvtPYQPKqH3JpN7LCtx6mzozeZ0BkPBmHvn/rHJu/XusaPDY2Ds+mQIG1CZzA0G+5cTicfvfoyxRtWY6vyn00jMWMQp105k9g+/dri8rWZPQf2882Ha7BvDEBpNPV6jb6C35M+Y3vEanRaLafGncrZyWczPnE8VsPxz5BwuB80RXm72fnjW0TtWkSqZ3ezbXZrepGffBGpE2cSndD7uPtzPGoq7AdHqLeWUVlsO2x7o1lHfN9Qb6juF3LYWuqWrpPb5SF/Rzm7N5WQu7GE8sLaFp/PZNF767YHhtMrIwyjuWuNuLZWW7951ZTb+XbeZvK2l/u2BYYYmXzDAGL7hLTZ87Q3eZNvXlWpjVVf72bLsnw8noPv2RKuj0y+p1pHrlPrtXnAVhRlMHAXkIH3w+As4HlVVTceT0ePV3xMjHrzpRejtdWisdWisdtQWvqs+ijoGgddoxG9SY8p1IExpAadtRKtuQyNqQS0zd8U1xytw4qxKtH7p6YXIbFDiRgzGmNsyHH3tzUyMzM5ZdQoVn3xMau+WITL7j962ffkMYy5/BrC4uLbpT+tVbSnih/e3URJjv+1LrDksDRlIcUW7zy/eo2eMfFjmJw8mQmJEwjUBzZ3uiNq7Q+aXZuWc2DpfHoXfEUkZU32e1SFLNMQ6tKnkT7xSixBRz/94NGqq3Z4R6jra6jLCloOtwB6k5b4tBDfKHVEgqXVi/609jqVH6gld6N3dDtvezked/P/PzUahdg+wSQNjCB5cDgh0eYuOQrbnLZ889qbVcp3b26mrurgJ229MsI449oMAqxdq0zpUPImf3gVRXWs+jKHnI3FBES5uPyv4yVcH4F8T7WOXKfWa+sSkQuAj4FfgKX1m8fW//mDqqqfH0dfj0tcXJw6a9Ys32MFlQCNhkC9DmuAgRBzICHWQIwBZr9R4UNHiRuPHLc0CnwoVVWxOwrra7qzfbOY1Nbm0GJB6qE8WkzuXgSFDyQ4apBvGkGDoe1vnmv8H6i6rJTfP3qPDT8s9qsJVzQaBk+azOipV2AJ7Tw38KmqyvaVhfz60Q5qKg7WCat4yI76jRW9vsSmr/FtN2gMnJZwGpOTJ3N6wumY9eZWP9fR/qBxu1xkLfsM++p3yKj4udl67TrVwObgcRiGXUHG2AvarF7bXuskb3u596bEreWU7D/8DDo6vYbYtBDi+4aQ0C+MyF6WY17c51h+IDtsLvZml3oD96aSw9Z8B0WYSB4UQdKgcOLTQrt0kGiLNy+PR2XVlzms/Gq378eLosCo81MZcXbSUa2G2lnJm3zrybVqHblOrSPXqfVaG7BbO4/ck8BTqqo+csiTPF6/r8MC9qFUFGo9KrV2J0V2J5TXoCgKkZGRxMbGEhMS4f07Jua4lw1XFAWTMQaTMYaIiAm+7W53HdU126iuyjp4Y2X11uanD9S4sWlysFXmcKDy4GU0GmMOmbM7HbM56bDTBx4NS2gYZ9xwM8PPvZBl7y1g2/JlgHeav/Xffc3mn39k5HkXMfL8qRjNrQ+nJ4qiKPQdFUPy4AhWf72bdd/vxeNWUdCQcWAMaaUj+T3hc7JilqEqHhweBz/s+YEf9vyASWvyhe3T4k87qrDdGlqdjkGn/wFO/wM1VeWs/OF/mLI/ZIBtnW+e7QDFwcjK7+Gn7yn+KYQd0ecQOeYaUgeOPqp6bYfNRf6OCl/JR/HeKg73O7JGpxCbGlxf8hFKdHJQh84yYDDp6D0sit7DolA9KkV7q9i9sYTcjcV+i6KAd+GNDUv2sWHJPnRGLYn9Q72Be2A4gSE960bJ2koH372xmX1bDn5SYg4ycNb1A45pYSYhhBAnVmtHsG3AQFVVdxyyPQ3YqKpq260IcpR69+6tvvHGG+Tn55Ofn09FRUWrjw0LCyM2NtYXuGNjYwkMPLaygiNRVY93+sD60e6Koo1UV2Th0B1o9Tk0mgAslr5NFsvR6SytOv5wv6Hmb9/Kz++8yb6sTX7bA6xBjJ56OYPPOKfDp/ZrrLywlqUfbid3U4nfdiXMzpq0r1muWdLscQG6AE5POJ3JyZMZGz8Wk67pt25b/SZfuG8nOT++RfTuT0nxNH8fcI4micKUi0ideC1R8SlN9rscbvJ3VfjqqA/srvKrwTyURqMQlRxEQv9Q4vuGEJMajM5wYmaKafO64go7uZtKyN1Uwt6s0ibTlDUW2ct68EbJJGunH709nmu1f1sZ387dTG2j0f74fqGcdf0AzEFduyTkUDKK1npyrVpHrlPryHVqvbYuEdkD3K2q6vuHbL8ceE5V1cPOMnIiHTpNX21tLQUFBb7AnZ+fT0lJyWHO4C8oKMgXuhuCd1BQ0AmrBa3dm0/Rb0upLN6I3bIXu3Uvdss+VG3rZzIJMPXyLZbTMNptMsU36fOR/gOpqsrudav5+Z23KN6z229fcFS0d2q/Mad3qhkydm8o5pcPt1NZ5F+fHTs4kMIhG/m2+Et2lO9o9lizzsz4xPFMTp7MmPgxGLXeUdG2/kGjejzs2vQ7RcsW0KfwayIob9LGoypsNg2ltv+lRKZeQPFep3e1xJyKFmcPAG+JQGQvq2+EOrZ3MAZT+yxwdCJ/ILudHvJ2lPtqtyuKWr7XIcDacKNkBL0ywjB0wtUKj+VaqR6V1d/ksuKzXQc/pVBg5LnJnHReSrecfUXe5FtPrlXryHVqHblOrdfWAfsh4E7gb8CveCsAx+K96fFvqqo+1cpOnQ28BGiBuaqqPttCu5OA34HLVFVdeLhztmYebLvdTmFhoV/oLioqwtPK+ajNZrNf6I6NjSU0NLRNQ7fzQC1VmXupXXcAVXXjMBdit+7Bbt2DIyIPe/A+nJ7iI5+onk5nxRLY3y94r15dzIQJE494rMfjZsvSn1j6/n+pKi7y2xeZnMq4K2aQNGR4p7kBze30sO6HPaz6ajcux8F/U51Bw4hzkgka6eL7/d+yePdicipymj1HoD6QCYkTODv5bBw7HJw54cwT0leX00HW0s9wrH2XARU/Y8TFAWdv9jsGsd8xiHxHOi4OX/4QnmAhoa93lo+4tJAOm3mjPX8glxfWsntjMbs3lpC/vbzFUXyNRiE2LYTkQd7R7ZDoji9vgqO/VnXVDr5/M4s9mw8udhRg1XPmtQNIzOg890a0NXmTbz25Vq0j16l15Dq1XlsHbAW4A2/IjqvfnIc3cL+stuIkird4eBtwJrAPWAlcoapqVjPtvgNswBttEbCb43Q6OXDggN9od2FhIS5X61auMxqNvrKShj/h4eFotcf3cbyrpI6qn/dRs6oQDp1tIdb5/+39d5yk2V3ffX9O5Vydc5wO0xM2x9mo1UpaBRCSCBJBRhiQ/djwyAHbgG2SwcDrNrfxc9/YGIPICCQhQCChVVxt3p3VpsnTPZ1zV4fK8brO80dVV3d1rJ6p6e6Z+b1fr3pVuqrqmjPVVd861++cg+XBBNnGGWLxfG13PHEFrctdba+Fe+/5HwSDd5e3L5kMb33ty7z8hb8iFSutj+04eTuP/tCP0dRzeJYNjy2nePELVxg8PVdye6DezSPf30fnyRquhK/w9OjTPD36NKOR0S2fx67snKg7kT/V5s+7Al1Y1LX33JumZnEyxuSlZcbPzTM9uIxp7PyecTkjNPTXcPzUAC39Vbh9h6Ms4KA+kNPJHBPnlxg7G2Ls7GLJbBobBevdxYGSLX1VB1Z/vpe2mrkS5qu/f7ZkjvLm3iDv+fGTN/0iPfIlXz5pq/JIO5VH2ql8120ebKWUH0BrvafVUpRSp4Bf0lo/Vbj+c4Xn+fUN2/0r8kux3wf8w/UK2FsxDINQKMTMzExJ8M5kdl7dbpXNZqOxsbEkdDc0NGCz7f2QtRFO54P2q7PoDSt5WWtc+B9vw3tPI9qSJR6/kl8aPnaRWDS/NHwut7LNMyva236UI0f+DTZbefXm6USc01/8a77zpb/btBT70VOP8vDHPk51U8s2j95/04PLPPuXg5tm1Og4UcujP9BHVaMHrTWXly/z9OjTfGX0K0xEJ3Z8Tq/dy/Ha45ysPcnxuvx5q29zGc5GWmuWpuP55ccvLjM9uEI6sfMPooB1hjbHGVodZ2h1nMNrzQ9su2LtZqH7Q/S+88eoazmwqqyiw/CBrE3N/FiU0TP5sL0wvv3Hkt1lpf1YfkXJjhO1eIP7F1bLaSutNW9+bYKX//ZKSQ/93U918sAHu696tpcbyWF4T90opK3KI+1UHmmn8h26hWaUUt8HvFdr/ROF6x8HHtBa/9S6bVqBvwDeCfwB2wRspdQngU8C1NfX3/PZz372uu231ppkMkksFiMajRKLxYjFYmSz5dVIK6Xwer34fD58Ph9+vx+fz1d2T7c1DcExRdWYwmKUhrmcU7PcrYm0a9YvVp//P10GJtBMgJ5AcwZYH47rsKh/glInytoPgEw8xsxrLxK6cIaSqSssFuqP307zPaewe67PING90qZm6QosnNEY634fKQvU9EP9CYXVnm9PrTWT2Ulej7/Om4k3CeXKK8XxWrx0ODryJ2cHnY5OAtYAmSjE5yE+p4nPg7HzYonYPeBtAG+jwlOnScWG8Ex8i7viz1OrIpu2N7TiLdttTNS/A2/3Q9ic7rLbpZJisRg+X3kDbPdLNqmJTUN0WhOfA3OH3zKuGvC3KPwt4KrmupY87dZWRkYz9YomOrV2m9UBrQ8q/C2HoxRrPxzG99RhJW1VHmmn8kg7le+JJ564toCtlHobeFxrvayUOsMOEztrrW/f9YWU+n7gqQ0B+36t9U+v2+ZzwG9prV9WSv0R+9yDXS6tNZFIpKSme2Zmhmi0/E79urq6ktlLmpubcbu3D0pmIkvspRliL0xhbugBtXjt+B5pxXeqGcs2A9ySySlefOlfAKWzhDQ3fz99vT+H3V7+cuOLUxO88Jd/yuCrL5bcbne6uPe7P8y93/VhHO7DUfuajGV45e+GOff8dMk72BN08NBHeum/v3FTsPriN75I1dEqzoXOcXbxLGdDZ1lKLbEdf6qGlkgfreE+2qJH8aQDO+6TJ+AoLOxSRdtANYE695bhLpfNcO65vyX3xmc4EXkOl9r8oy6hnZyregeue36Y4w99AOtVHC25Woe9x8PImkwNLhenAdxpFUtPwJEfKHlbLe3Haio+UHSntpobjfD0/zlLdHFt/xq7Azz1kyfx1xzYBE0H4rC/pw4TaavySDuVR9qpfNfcg62U+kXyAxgTSqlfYueA/ctl7NCuJSJKqRFgNWnUAQngk1rrv93ueQ8iYG8nFottmsFkeXnzCn/bqaqq2jRtoN9fuvS3mTaIvzJD9NlJzFhp4FIuG76HmvE93IrVu3ng27e+9S0GBla4PPirJSUkDkc9R4/+Mg31T+3p3zt9+SLP/vkfMnXxXMnt7kCQU9/7MW5/13ux2g7H1H4L41Ge/cvLzA6XTuPY3BPk0Y/2U9+x1s4bP2i01swl5oqB+9LEMOGRLNWLbbRG+gika3d87Yw9QaYxTHWPg/4Tbdx99AQB584hfKNoeImL3/hTPBc/z4nM21tuM08NV5rfT9Ojn6D7+H17ev6rcSN9IGut8wMl315k7GyI6aEweruBklZFS19VsXa7quHafyxu1VZaa848M8kLnx8qWd3yjne1c+pDPQc6X/lBuZHeUwdN2qo80k7lkXYq32EsEbGRH+T4JDBFfpDjD2mtz22z/R9xSHuw9yKVSm0K3aFQiHLb3efzbZo2sKqqCnIm8dfmiH57EmOltAZBOSx4H2zG/2gb1nVLJ6/+AaUzIS5f/mXm579c8riGhvfT3/+LOB11Zf/7tNYMv36a5z/zx4QmSud6DjY28chHP87RU48eiqn9tNZcfnWOF78wRCK8vm4ETjzayoMfPILLZ9/0QZOIZJi6vMzU5fwS5CtzOy8/nrYmmQkMMRUcZDowyKJnBlTp/3dXoKs4iPJk3UkGagZw28or9ZgZu8TYt/6Y5vG/o9Oc3HKbK9YjLBz5ML1PfoK6po6ynnevbuQP5HQiy/j5/IqSY+cWScW2L/mqavTQeVstXSdrae69uoGSG9sqnczxrT+9wJXX12bpcbhtPPmjxzhyZ/2en/9mcSO/p/abtFV5pJ3KI+1UvkrPIvJN8kuir2y4PQD8rdZ697nf8tu/H/ht8tP0fVpr/WtKqX8OoLX+3Q3b/hE3QcDeSiaTYW5uriR4z8/PYxjbL6yxnsvlWgvcDU1ULzuwnY5gLG0o9rUpvPc14X+8DVuVa9Mf0MLCV7l46RfIZNa+5G22IP19/4mmpg/vqSbVNA0uPPcML/zVnxFdLJ3ar6G7h8d+6MfovP3Osp/vesqkcrz2pVHe+uZESc+h02PjgQ8eYS49yJHGk4Xlx5dZmo7v8Gxgc1pp6Q3S3BfE0pZk3DHI+eVznAud4+LSRTLm7oNkLcpCT1UPJ2tPFkN3X3UfDuv2M4Zo02ToredZfPFP6F94mhq2rtc+576XzMkf4MQTP4jb69/ima7OzfKBbJqa+dFIcaBkaGL75eYdLivtx2vouq2OjhO1ZS/0sr6tFsajfOX/nC2Zu72+w89TP3mSYP3B1NMfFjfLe2o/SFuVR9qpPNJO5at0wDaBJq31/IbbG4AprfWB1QHciAF7K7lcjoWFhZIZTGZnZ8seTOlwOKj31VAdc1MTd1Fr+qnWXixYwKLw3N3AINPc+9RDWHz2YnjOZsMMDf0G0zOlA0Vrax5jYODXcLn2NjtILpPhjaf/gVf/5rOk4qVBpfP2u3j0hz5BY3fPnp7zelmejfP85wZL5houh9VmoaknSNvRKlqP1tDQ5ce6zQwPWTPL0PIQ5xbPcTZ0lvOL5xlcHiRXxtSKNouNo9VHi4H7eO1xeqp6sFk21wdnM2nOPfc3GG98hpPRF3BuUa8d024uVL8D970/xPFTH8ByjVNK3qwfyLHlFGNnFxk9s8jkhSVy2W3my1fQ0Bkozrld1+7b9kfpM888w+OPP86556Z5/rODGLm157zt8VYe/r4+rPaDP8pz0G7W99T1IG1VHmmn8kg7la8iAVsptTph8mvAe4D1ScQKPAX8hNa66+p39drcLAF7K6Zpsri4WOzlXg3eqdT2g7XWs2hFjfZRa/qp037qzSA12ofNY8fW6MXe5MHe5MXe6CHqfItLI/+ZVGqt5MBq9dLT8+9oa/1h1B7ngE7FYrz6xc/zxpe/SC5b2oM78PDjPPzRj1PV2LSn57wetNaMvh3i+c8NbjsIzmJVNHYHaO2vpu1oNY1HAtjsVx9OU7kUl5cvczZ0lnOL+Z7u4fAwevthDkUuq4uBmoFi4D5Zd5LOQGfJHN3h5RCXvvmn+C79NcczZ7Z8nlnqGGl5Py2PfoLOY/dc1b/jVvhAzmUMpi6vMHYmxOjZxZKBiBt5g6sDJetoG6guGSj5ja99C2O8oWSOdrvLyhM/MkDfvY3X9d9wI7kV3lOVIm1VHmmn8kg7la9SAdtkbXDjVl0zSeCntdafvqq9rICbOWBvRWvNysrKphlM4vGdyxhW2bWVBjNIoxmkSVdRbwaxkw+Lqtok1P83hAL/UFIzHAzey7GBX8frPbLn/Y0uhnjxc3/BuWe+jtZrvXYWq4073vM+HvzIx/AEyp/B5HrJZQ3e/NoEb3xtnEwqR0NngLaj+dUSm3uqsDuvrbd3N/FsnAuLF4qB++zi2V3n5l7ls/s4Xnu8uCjOybqTtHhbUEoxPXqJsWf+kLaxv6NdT2/5+CFrD6Gej9D35CeobWwre59vtQ9krTVLM/Hi8u2zw5HtB0raFK391XTdVkt1s5ev/MGbZNZNMlTb6uO9nzx5aFaaPCxutffUtZC2Ko+0U3mknXZmGgaLUxPMj1zh5DveVZGA3Uk+WA8D9wPri2szwLzWurzC4evkVgvY24lGo5tCdzgc3vVxSitqtY8ms4pGs4pGM4gKTjB74tNkfNPrtrPTYv4obQ2fwNkUwFbnRu1h4YvFyXGe+8yfcOW1l0tut7vc3PfBj3DPBz6Ew3Xw9adaa771zWd455NPHPSuEE6HObd4jvOL54u93bPx2bIeW+2s5nhdPnSfrD3J8ZpjhC9fZPnFP6E/9FWq2TylZE5bOOe5l+zJH+DkEz+Iy7PznKi3+gdyKp5l4vxSvnb73CLpeHkrqh5/pIVHf6APm+P6/mi7Ed3q76m9kLYqj7RTeaSd1hi5HIuT48yNDDE3fIX54SEWxkaKR+N/5rNfOlyziFwvErC3l0gkimUl09PTDA4OlrUqZcB004iPxs43UJ3fBstaz7Mz0knTuX+KK9GFvd6NrcmLvTFfZmJv8mKtcqIs2w+OnLp4nmf//A+Zvnyh5HZPsIpT3/dD3PbO9+zrXM5bOcwfNKFkiHOhc8Wa7nOL53aco3u9BncDx+uOc6xqAN/UEp2XX+JU5BUcanMwjGo3F6qfwHvfj3DswfduWa99mNtpv5mmZm44zOjZ/Jzbi1ObjyjZHBbe8cMDHH3g4EujDit5T5VP2qo80k7luVXbychlCY2PFcL0EPMjV1gYH8XYYfxbxQN2YZq9+4EOoGTovNb6T8p6kutAAnb5vvWtb3HnnXcyPj7OxMQE4+PjzM/P7/gYr3eJo/0v4/Uvrt1oWqgZfT+1wx/EYpbOoqAclnx9d6MnH7wLdd7rB1ZqrbnynVd57i/+iKWp0jKIqqZmHvnYj9L/4MPXdWW9ndxIHzRaa2bjsyWB+9ziOaKZzT3UW2n2NNGadnF0cZp3Jic4ns7g2/CZMEs9I60foOWxH6Pz6J3F22+kdtpv0aXVgZIhpi6vYPMafPinHqCm5XCsdHpYyXuqfNJW5ZF2Ks+t0E65TIbQ+Gg+TI9cYW54iND4GKZR3tFHX20djd29fPjf/+eyAnZZXYVKqQHg74Fu8iUjRuGxWfLrbx9YwBblU0pRXV1NdXU1d9xxBwDJZLIYtsfHx5mamiqZLjAer+H1N95La9sFOjvfwmo1wGKydOQfWGh8Cff5j9GyfCdO8hPJ6IxJdiJKdqI04Fk8trWBlY1e2psG+Piv/DYXTn+bFz/758SW8gF+ZXaGf/jt36Cpp49Hf+jH6Di56yKhtzSlFM2+Zpp9zbyr811APnSPR8eLtdznQue4sHSBZC656fEziVlmgNeq4M+r8oPt2rKaO9MJTqQznEynOZoJcWrqj+Azf8SgrY/Fno/Q/+SP7t8/8gbkr3Fx8rFWTj7WCuS/vCRcCyHE/shm0iyMjjA/cqXYO704OY5Z5nTIgfoGGrt7aejuofFIL43dPXiCVfk7//1/Lus5yj0W/9vAd4A7gdnCeRD4X8B/KvM5xCHkdrvp7++nv78fyE8XODMzUwzc4+PjJJNJpiZPsLTYTl/fSwSr8r3eVu8i6Xt/hxemjxKZeIz6TB0NmQBNugqfdqHWjYs1EzkyI2EyI6V14XXBIB++59+yFJ/m0qWXWIxOEskuMntlkM/9l5+n6857ePQHf5SGrr0PsLxVKaXoDHTSGejk/UfeD4BhGoyER4qB+9xifo7urLn5MNikXTFp9/IPvnwgtGpNTybLiUyGk+kZToz8d1y/89/w2u7gpeEvYa/vJdA6QGP3CYLV5S9SJIQQQlyrbCrF/NhIocQj3zu9ODmONreZYnWDYGMTjV09NBzppfFILw1dR7acfGFubo6XXnqp7P0qN2DfBzyutY4XZhaxaa1fV0r9e+D/AaSb8SZhs9lob2+nvb2dhx9+GK01oVBoXS93J/MLr9Dd/To2WxaloLX1ErW1kwwOPsiF5fy82T6nh2ZXHY1GkPqIl+qMOz8n9wZGOIMRzuDDyz2Bd0Eg3wMbyy0TzoQIjy7wzK/8T2pv6+aej30vVc1Sv3o1rBYrvdW99Fb38qHeDwGQNbIMrgyuzVwSOsvQyhDGhnHLhlJcdjq47HTwN4V1auxa05eZojU1TuNIjqZBg8ZvGLhzTiyqEZuzE6qP4GjoI9g6QPORE3j9Vfv7jxZCCHFTySQTzI8O5wcfFsL00tRkySxlO6lqaqaxuxCku3to7O7F5dt+QL/WmitXrvDSSy9x5cqVPe1ruQFbAavrQy8ArcAlYBLo3dMrihuKUor6+nrq6+u5++78tOixWIyRke8wO/dbWCz5le5drji33fYNZmd7GBm+h1gaBtPjDAJYwOF30FzTSIu7jiYjSG3UAwsZMDaPAVBK4bfX4LfX0ObN96wzB5Hfvsii6wz+niZc7cFCjffuAyvF1uxWO8drj3O89jjf3//9QH6O7otLF4uh+9ziOUbCI5vm6M4qxXmnk/POrZ45gdLnqTPO0Dhp0Dhq0PhtA59hx6mrcTuaqPL30th4gvr2kzR1Ha/oCpNCCCFufOlEvFDika+Xnhu5wvLMFJQzdlApqptbaezuyZ8KgdrpKa9UL5fLcebMGV566aVdx6ptp9yAfRa4g/x0fa8C/0EpZQA/CQxd1SuLG5bP5+O22x7n5MnHmJv7Ipcu/wq53AoATU1XqKmZZmjoPhZDncXHZLIZxuYmGCM/qFEpRVNHE231LTS762g0q3AtabJzCXKLSbZac8WirDjTVjLnV8icXynevuXAykYvFr/9wAZK3qhcNhd3NtzJnQ13Fm+LZWJcWLpQMnvJZGxy+ycBtFIs2Gws2GycLQnhSWAkf5r7GjXTBo0vGFTlLPi1F5+thip3K43VfXS33cnxvgcJ+mquw79UCCHEYZGKxZgbyc/iMTc8xPzoFZZntl67YSOlLNS0ttHY3UNDdy+NR3po6DqCw733dQYSiQSnT5/m1Vdf3bS+iFKKY8eOlf1c5QbsXwNWY/9/Av4B+BYQAn6g7FcTNxWlFE1N30NNzcNcvvxfmJv/BwAcjiTHjz+LzXo/odB7GBtbJhKJlDxWa12cr3tVdXU17Ufa6Xi0nWZvPcG0i9x8ktjQLKmpCC62/mMpd2ClvcmDvcGDxWOvcEvc3HwOH/c13cd9TfcVbwunw3zuW5+jub+ZucQcc/E5ZuIzTK2MM5+YI2zEyliXEpasVpasVnBCfrz0DOgZWHoNlj4Db0PA0FSbNoLKR5WjlgZfO+0NR+lvv5PWYBuNnkY8dlmwRQghbgTJaKTYKz0/PMTc6BXCc+Wt8aAsFmrbOtaF6V4aOruxu1zXtE+hUIiXX36ZN998k1yudFYRu93O3XffzQMPPEBNTQ0f/ehHy3rOsgK21vrpdZeHgeNKqRpgWd/oE2mLa+Zw1HHy5P+gceG7uXTpF0hn8stB54xXqau/xKmHfh6368liHffExARzc3Obnmd5eZnl5WXefvttID8As729nY67Omj/4ACxsTHO/e1XsUQ0QXsdQUc9QUc9LuvW4Wq7gZXWgCM/f/dq8G70YGvwYJGFP8oWdAbpdfXyjiPv2PL+rJFlIblQDN/T0SlG5y4ztTJKKDnHkhEhrLLoMg4wRKyKiNUAwmCGITIMkW+XHDvzaCvVFh+1znpaqjroqu+nxd9Co6eRRm8jjZ5GfI6dF84RQghRWYlIOF/eUZhjem5kiMhCeSUXFqs1H6aP9BZn9Kjv7MLuvLYwvUprzdjYGC+99BJbTffs9/t54IEHuOeee3C7974Q3lWv6KG1Lm91C3HLqK9/F1VV9zN05TeYnv4rAHK5MBcu/Adqqh9hYODXuP32/HjYZDLJ5ORkyfSAG381JpNJLl++zOXLlwGwWq20nOjGrQ3GzrxObuEZlGHgtHgIOuppbzlOb8+9ODJOsrMJdGbr6XiMSAYjkiF9eXntRgW2Ghe2Ri/VGUW6O4KjzY+ySonJ1bBb7bT4WmjxtWy7Tc7MEUqGmApPcnnsDcbnLzAfGWM5vUDYjLJizbJkhVwZZT4JZZDQYaZSYd6eHYLZb27axq2c1LvraQm20eRtosnblA/g60J4wBGQsiJxqBmGQSwWwyxzhgQh9kt8ZblQK51fAXFuZIjYYqisx1qsNuo6OotT4jV291LX0YXN4dj9wXtkGAbnz5/nxRdfLDmKvqqpqYmHHnqI48ePY7uGhe+2faRS6ltsWQm7mdb6nVe9B+KmYrcHODbwX2ls+C4uXvyPJFPjACwtP88rr76PniM/Q1vbx3G73fT19dHX1wesTQ+4fk7uRCJR8tyGYTAxUViYproJqpuwZlKk4lFiyRizE9/gteF/pOfu+3j4x/8JNYFmsnMJsrNxcrPx/OX5xJYDK9GQW0yRW0xRi4WFobdQTivOnipcfVU4e6vyy8NL+KoYm8VWDLr3tGw9Z38iEeXS0GlGJl5ndukSK/EJYrkQcRUnbM0xa7Mxb7OSLeP/JanTjCcmGU9sXz/usrrWgnchdG+8XuWskveB2FeGYTAyMsL58+e5ePEiiUQCu92Ow+Hgvvvuw+nccrSxENeF1prY8mKxXnq1dzq2XF6/q9Vup76ja90c073Utndis1/f8s1UKsV3vvMdXnnllU1lqwD9/f2cOnWKrq6uinzG7xTNz667bAV+mPwc2K8UbrsfaAb+7Jr3Qtx0amoe4oEHvsTw8G8zPvGHgIlhJLg8+CvMzf8DxwZ+Ha93bQKa9dMDPvTQQ2itWVxcLIbtiYkJFhcXN72O4XBhOFxkq+sBULkM5+YWufgbv0rfwDHe9YM/QtVAe3F7bWhyi0myc3Gyswlyc/ngnQttHlip0wap84ukzudf11rlxNlbhauvGmdvFVav1HJfbx6Pn7tufyd33b75N3wyHmV29DxLExcIzZ0nEh0kmZkmo5eI29LMWW3M2azM2azMWq2kLZunidwoZaQYjYwyGhnddhuHxbFt+F49r3HVYFG7v54Q28nlcly5coXz589z6dIlUqlUyf3ZbJavf/3rPP/88zz44IPcf//9eDwyFkFUltaa6GIoPwBxeG0FxER4pazH2+wO6ru6i4MPG7t7qW3rwHoNPcN7tby8zCuvvMLrr79OJpMp3T+bjTvuuIMHH3yQ+vr6bZ9Da41hJDCMWNmvW9ZS6Uqp/04+ZH9qfc21Uuq3C8/xqbJfscJkqfTyHdRSqOHwm1y4+LPE44PF25RycKT7p+no+EkslvKCaiwWK+nhnpmZ2f0wqWlQ5fVw4s67ONLbR1tb25a9PTprkJ1Pkp2NM/biRapjLoxwZosnXP0HgL3Fh6u3CmdfFc7OIMp+awWqw7y0biyyzOzIecJTF8nMD2JdHkalxtF6hpQ1xazNxpzVWgzgc1YbszYryTJCeDlsFtta+YmnkXAoTGtLKxZlQaGwKEv+slJYWHd5l/utylqyzepjLKy7XMZrbNz+ap6jnNcoub+M13ju28/xzidu3QOi2WyWoaEhzp8/z+XLl0mn01tuZ7FYNn32rfZmnzp1Ct8O8/reag7z59Rh8swzz/D4448TWZgvmc1jbuQKyUh49ycAbE4nDZ1H8r3ShWnxalvbsVgPZnzTxMQEL7/8bQYHz2C1ZrBac1ht+XOvx0L3kWba2uqwWjLkjBhGLl5ynsvFMYwYuVwMw4iz2gv3rieHy1oqvdyAvQic0lpf3nB7P/Cy1vrA5tGSgF2+g/ygMc00o6O/y+jY/0TrtVprn+84x479OgH/yT0/ZyaTYXp6uqSXe7svpFX5mU+a8oMnOzro6OggEAiUbLP6QZNbSJIeXCY1tEL6Snjbmm4AbBac3YFi77a9yXvTz819o35xhZdDzI/mw3d2fgj7yhUCiXEaclNYLIm1nm+rlTnb+sv5IB613lo/pPabVVmxW+zYLDbsFnv+ZN1wfd39Nqtt023lPM5u3eK21ccoW8n9Wz1u9bpVWa/pcHI6nS4J1dns5tVVAYLBIMePH+fYsWM0Nzfzuc99joWFBZaXl0u2s9ls3HPPPTz00EMEg5tXo7vV3KifU/slk0xw7tvf4PTXvkJ2ZYlULLr7gwC7y01D15G1mukjvVS3tGKxXFuY1lpjmsmScLtV+DW2uT2Xi5FKrZDLRlGWDEpVfh6OcgP2XhaauQ24vOH22/a6Y+LWZLE4OXLkUzQ0vJfzF/4D0egZAGKx87z22kfo6PhJurt+Gqu1/NHBDoeDrq4uurq6ADBNk/n5ecbHx7l09gxjY2PkNhymXz894KuvvgpAVVVVMWy3t7ejtUYphb0hP62f7+FWtGGSmYiSGlwhPbhMZjIK6zuQcibpwRXSgyv5f6/Pni8n6a3G1VeFNSg1kodFsLqOYPVjcNdjm+5bCc1ijJ7DOXWJ1oUhusIjBKLjNOWm8KskAHGlCmUntnU94IUwXgji4QPqsbkZGNrAMAzY4ffsYaJQu4f5Dfc7TAfuZTeOkAPrkhVlbh3QrV4rvjYfgfYAvjofCWuCt7JvcX7qPMvVyzzyzkdYHFnkwmsXWF7MB+1cLscrr7zC6dOnueuuu3j44YepqZG55EWp8Pwsb3zl7znzza+RSSZ23Nbh9hSmxesp9k5XN7WgCkf88qE4RTa3jJGLkjPihQC8/jx/+2pvcOl5bN15nNIv16tzjTl/S0o5UZSfUcrtwf5vwI8Dvwm8XLj5QeDfA3+otf63e9/VypAe7PIdll/yppljYvKPGB7+vzHNtR5nj6ebYwO/QVXVrj8My6K15s1vf5Pnv/R3RNI5DI8P0+mGXXqbrFYrHR0dtLS0FE9VVaUD28xUjvSVlXzgHlrJ13DvwNbgxtVbnS8nORLE4ty/+rPr5bC8n/aDNk0W56dYGDtPdPoSxsIQzsgIweQEzblpPKr0yElSqXUlKDbSCjQKk/xXh1ZgFq5rwCxcX7u87v5119ffX3w+VXgO8gv8FF8DMNW61yhc1+Sz6/r7DRQmCkOpwmULhgITS/FxJmrdeeH1lSo+jy7ZF4Uu2S/QhfVA8ydduE+vu64xKvDFeljZDTvNiWZaE600JhqxsnUCiNqjTHommfJOEXaE891bu9HQkmjh2MoxqjJVG+7SGE0Gjh4H/ho/PrsPn8OHz+7D7/CvnRdu89q92Cw3/ufTqlvpc2o3Wmsmz5/h9X/8Ildee7W4vLiyaJxVaaxOA5ffQXVLHYHGanx1fjxVbmxOMMzEul7j9eE5H5S1Ppy/iE3DisW0YDEUqnCy5sBiaKw5jc0wseVM7LkcDsPEkcviNHK4DAOrobEVtrMamtXuOvXLkYqWiFiAnwE+RX5gI8AM8D+A39IH2LISsMt32D5oEolRLlz8eVZWXim5va314/T0/Aw2W2XqCI1cjrPf+iovfu4viEejGG4vhseH4fZhevzoMg7vut1uWlpaaG1tLYbu9aUlueUU6cEVUkPLpIdWMBO57Z/MonB0+PPlJH1VOFpvzOkAD9v76aBo0yQ0O8786HniM5cwQkO4wiNUpyZoMmZwqa0P+YvNVsN/VimyCrIocquXlSKLIomNuMVOUtlJYSNtsZGy2EkrGxmLlYzFRtZiJausZK1WshYrhrKSs1rIWSzkLApTWTAsipyl8KNCgWkBA42hwFAmJpocGgODHAaGNsnpHDkzR9bMkjWz5Mwcxg5ffw7DQUuihdZ4Kw3JBixsXV4UtoeZ8k4x6Z0kao+WF6q3acCmZBMDKwPUpms33KWZ8kxxseoiYefONbVumxu/vRC6HT78dj9eu7cYyH2OtXBeDOyF7VbP7dbDMQhcPqcgl8lw8YVv8/o/fpGFsZHCrRpPQ4rq/jA1fTGsjsPzOWUxdDHc2nJrl62GWbyeNDxMGG1M5FrJGU5yhh3DsGHk7NgMk9uNy9yXO0OQ+O4vuEcVDdglD1AqAKC13jzHyQGQgF2+w/hBo7XJ9PRfMTj0GyWjc13OFgYGfpXa2scr9lrZVIrvfPnvOP3Fz5NJ5nucNQrT5aHq6HFcLR3MLYQ2LY+6HZ/PV9LL3dLSgs/nQ5ua7HQsX7s9uEx6NLL11IAFyrU2HaCrtxprreuGmAbuML6fDhvTMFiYGeX0S89z/NgARjaNkctgZDOYuSxmLo2Zy1/WRgYzlwEjf1kbWchl0GYWjPxJGRkwcygzWzxZCtctOoelcG5dPS+cbDqLFaN4bieHTeewkcOhDmfP02GV0VYyOMgoOxkcpJSDtLKTtDrIWOxElIcZs4kFo4GIEWC7tGxzZrH7s9ircuAB02rBtFjRVoVpsWBa80cKTGvhyIQFTKXJYZAxMkzOTWL324llYsQyMaLZKMlc4UiahvpUPQMrAzSkGja99ox7hotVF1lyXb/lLJxW5+ZQbl/rKd/Yi14M8g5vcTuX9do/C2/lz6nY0iJvfe3LvPX1rxQHKtp9War7wtT0hXFV7zCQf48s5mqPsC7p+d0YjG2b7tsYojWWbb4uNTBEFy9yDyN0bLq/mhVO8Tp3cg4HO3RylUEDUxYnV+xORhwOxux2JuxWJu0Wnv7Js9cnYB82ErDLd5g/aFKpGS5d+gVCi6ULhDQ1fZj+vv+I3V5dsddKRMK88jef5c2nv4RprP0RKmXh+OPvJF3bTO+JE8zMzDI9Pc309PSmKbK2EwwGN4Vup9VBZjRCanCZ9OAK2dmdA7y12lkcLOnqrTq0S7sf5vfTYXOY20qbJrlcllw2QyaTxsimyWUz5LIZjGwKI5cll0ljGtniDwQzm8E0suhcuvBDIf/DwMwVfhis/kAwsmDmryszV/yBYCn+OMiizBwWnf+hoLNJnBYTm85g1xnsZgY7WRxkcOoszkN6NCCClwv0cZ5exmhju1DdwizHGeQ4g9RQ3swMG+W0hTQOllQVYWcTCXcLhr8VW3U7jrp27A3NuOrqyFpMopkoU5NTDL8xTHhq8+tlghlCLSFCrhCxbD6ox7KxQuHOwbMpW0kQ99q9m4L6+vKWTYHd4efV51/liSeeOOh/yr5IxMLMjV9m4s2XGf3Oa4SmFtEaLDaT4JEINf1hfC2JLasknWkDV2qrIGxuE5pLe5i3C8VXI63tZLCRUQ6y2Eni4rLq4bzuI4x/0/bV1gRd7igNbgPsTrA6weZE2ZwouwvL6rndidXhwmp3Y3W4sDncpCw5Zs0ws9kQM5kFplOzTManGY+Nr/1g3eDsJ64xYCul3gYe11ovK6XOsMOiM1rr23d7oetFAnb5DvOXPOTrw+bm/p7Lg/+FbHatZ8Vur+Xo0V+iof59Fe3ZDc/P8sJf/RkXnn9m031KWfAEg3ira/BWVWMNVJGzu0hqiKTSLEeiZHPl/UKurq4uhu3W1lYa/LUwniRVmKHEjOwyHWCrb61+uzOAsh2OWSwO+/vpMJG2Ks9u7WQaBplMinQqSTadIJNKksskyaZT5NIJcpkURiaJkUliZlPFk86m0Lk05FKQS6FyaZSRxmKkUUYGq5m/bDMzWM3MWsAvnBxkcOgsTrJYCrMSrODnAr2cp58Jtl+xtJ1pjnOZYwxRRXkzNGz6dwNJt5W4J39KuSz44gb1ixlc6a3r1kNUsWRrIOZsIuNrJeJsZSTmYmZp8w/89vZ2HnvsMXp7e9FoEtkEsWyMaCZKPBsnmokWr68G8eJ92WgxmK+/f6fSmf1kV3Za/a20+Fpo9jbT6mul2Zc/b/G2UO+pv2HmrDdyORZmRlmcuEx8bghjcQR7ZBxfYoqa7AzLURuvL7UwnQyC0vhaEtT0hwl2R7DaN0c4a86kIZSheS5NVThLZkOwzSoHOWUnZ8mfGxZH4eTEtDoxLQ601YG2OtG23YOtxebG5swHW5vDic3pwe5wYXe6sTvdOF1uHA5XcfBkPB7n9OnTnD59etORZaUUx48f59SpU7S1te3YbqlcivHoOGORMcYiY4yGR4uXl9PLOz52K5UI2L8I/F9a60Th8ra01r+85z2sEAnY5btRvuQzmUUuD/4qc3NfLLm9vu7dHD36Kzidmw95Xou5kSs8/5k/ZvSt18t+jAZMhwvT7UUFqjHdXjJWe9n9PnV1dcXQ3eiuIbjixByJkR5eQWe2H+il7BYc3cF8OUlfNbZGz4GVk9wo76fDQNqqPIe9nRZDIc6eO8uF8xeYnZvbdrvagJvmKheNfgtOlcXM5oP9ashfDfgql8JiZrCshnwzhXYk0a4UhidLzp0j6zFJe0BvM+1nIJKlPpShIZTGk9p9kOgCNTzHfZzRA+gNwTLgtnKyv5vjt99NQ9sRHM7yZ0xYT2tNykgVS1fWl7GsD+yrQX1973kxqGdiZMzKlTBsx2ax0extpsXbkg/hhfC9GsYbPA37OugzGl5ifuwi4ZkrZBauoFbGcMcmqE5P0WjO41ClHTvJnI23V5p4c7mFWM6JsypNTX+Y6r4wDt8WnUAabFEvlkQvdvv9eBuPUdPWz8XhSZ5817v36V+5s4WFBV5++WXeeustchs6shwOB3fffTcPPPAA1dVrR7ZzZo7p2DSjkbXwvHp5Nj57Vfvhd/jpDnTTGejMn4KddPo7OV53XEpERKnD/uW10ULoG1y69Auk02t/HDabn77e/0hz8/dVPFiOnXmTV//2s0wPDZJL7TwryFY0YDrdGC4vptuTP3e5oYzeEaUUtTU1tLa20uSuoybhITCjMKYSOxw7Aovfnu/dLqwwaQ049rzfV+tGez8dJGmr8hzGdgqFQpw/f57z588zO7v1F7VSiq6uLo4fP87AwAB+/+bD2OuZZpZkcoxYfJB4fIh4fJB4fJBEYgStr74MxpawUbWQo3MhTDCR3XGs5BJBnuc+3uQ45oYZTeoJ8Yg+TaNaIGyrJ+ZqIutrRVW146ztxN/YRV1bH4Gq2mJv4/WQMTJrAXw1hK8L7avnm0L7ul717Q7zl8uqrDR6GjcF72ZfM63eVpq8TXsa0JnLZpifGmFp8hKJuSsYiyM4ouP4k1PU52aoLvMoRyjt4Y2lFs6HG9AOTXVvhOr+MN6GrcsZXa5u2lq/n8am78HlbNp0/0H/7WmtGR0d5cUXX2RwcHDT/YFAgAceeID2Y+3MpmdLgvRYZIzJ6CQ5vfe6a5fVRUegYy1EBzrpCnTRGeikylm1Zc5QSknAFqUO+g/oauRyUYaGfpOp6c+U3F5T/TADA7+G292+zSOv3jPPPMOjjzxMfGWZ2NIS8ZUlYstLxJeX113On++2wpVWanPoLmOqQACFxmOzUWcP0KRqaUrVUJvwbjsTAYCt0VNYXbI6Px2g4/rNx3wjvp8OirRVeQ5DO2mtWVhYKIbq+fn5LbezWCwcOXKE48ePc/ToUbxe76ZtTDNDIjlWCNDrg/TonoO009GI19uH19uL09nA0JUvodSlkoW71nPYm3FwEh1txFzIYo1M4UrMEEjPUWfMFed1D+PjRe7lO9xGbsPSGDUs8winuZ0L2LaYQjGuXSxY6wk7mkh5W9D+Nmw17Xgauqlu7qauuQu742DXAPjHb/4jPXf1MB2bXjvF1y5fTYnAegpFvae+JHxXKT/uWAb3cgz/8iKu8ASe+AQ1mWkazBD2qxxYrDW8HWvlzZU2FhN2Ah0xqvtXCHTEtpz32W6vobHxu2lu+jB+/8kdO6UO6m/PMAzOnj3LSy+9tOUPWGvQSrw1zhXXFcaiY1f1g8mqrLT6WksDdDB/3uBp2HOJULkBe9vjHrvVXa93kDXY4uZms/kZGPhVGho/wMWLP08yOQ7A0vILvPzK++jp+be0t/0TlKpskLTa7ATqGgjU7VyOYuSyxFdWiK/kA3hsuRDIS4L5EolwCFbyf05aWTBd+dBtuDyYbi+mw7UpdGsU8ZxBPLfMGIUvAYeJJwt1pp92SwtN1BDUXiyFvqrcXILYXILYC9NgAXuHH3d/Da6+auytvpt+dUkhrobWmtnZ2WKoXlxc3HI7q9VKT09PMVS73W4gH6RjsUuFEF0I0omhQo/03nrVnM6mQpDOh2mftw+Ppxe7vXTF2ZGRozz88F2EQt9kfuErLC09V7KuQCY7Q4YZcIGzt4na+ndTX//PqQreh8ViI7KyyOLUEOHZEWpCYzyyOMXkSo7RTE0xaC9RzRd5D8/wII/wGndxFvu6FYC8KoXXnIDUBKSARWB0bR8NrZhVtSzbG4i7msn6WrFUt+Oq6yDQ2E1dWx/+4PVdBMdtcdNf3U9/df+W9yeyCWbiM5vDd+E8lAzt+PwazXxinvnEPG/wxpbb1HoNWp05mnM5WnJ+WnI5WnI5WnM5mnMGnnUdnWltZ9bayIqzhZS3HV3dhTXYzspCnLHX3yCjR6m5Y4kTPRFs7s1BXSkHdXXvpLnpw9TWPo7FcjgHyS/HlnnmpWe48MYFMonSUiCNZsYzw2BgkJArBEnyp100eBqKvc/re6Jbfa0HMm3kToVFn9+3vRBiFzXVp3jg/i8zPPLbjI9/GjAxzSSDg7/K/NyXGDj26/i8ffu+X/kgXk+grn7H7UzDIB5eXgvhhR7wfDBfIrK8RDieJG6YGE43htuLdmxR/2ixkHDCOHHGyR9Gs5hQZbhopo4GHaROBwhoN8pUZEejZEejRL46Ro4scXeUTI2BarHjbqnGW12Nr7oWb3X1gfc0CbGftNZMT08XQ/XGJcdX2Ww2+vr6OHbsGL29XZjmDPH4INMz3yqG6WRydM8LbTidzYUA3Y/X21sM1DbbzuUl69ntQZqbP0xz84fJ5eIsLj3LwsLThELfKpn2NJ2eZXLyT5mc/FPs9mrq6t5FQ/1TdB57CMuJB0qeM5FI8Morr/DKK68UZ0+KEODLvJOvq8fot05wp/EmLebMpgWWNrIqTRMhmrIhyJ6HKPkVNNaJ4GHR0kDE2UjK24oZaMNe046voZvqliPUNXVitV2/GmiP3UNPVQ89VT3F27RpshyaITQ5yOL0BeZCF1iOjxDNzhNTYcLWDLN2K1M2G/NW665rKSzarCzarLzN1p+xPuWiwVFHa7CDrtqe/KBMbwtVKScLL7zJlS//Hb72WZrfsf3UeoHAXTQ3fZjGxg9gt1dddXtUUtbMMh2bLhlYOLkwiR7TNCw1YNOl/685lWPMN8ZQcIiYPbblcwYcAbqCXXQFuujwdxR7ojv8HXjsnv34Z5Vt23ftQQ5cFGIrVqubvt6fo6Hh/Vy48LPE45cBCEfe4NVXP0h317+ks/OfHcpf7BarFX9NHf6auh23Mw2DRCRMfHmJxbk5pqcmmZtfYCkSIZLOkN3imJJpgSVLiiUmOcckAHZtod7Mh+1600+dDuDTLoLJGpgCpiCWXWIo+TpzyVHmUmNY3Da8VTX4qmuKs6eshu/V23xVNdhdVzfwSYiDZpomk5OTXLhwgfPnzxMOb13i5XAoBgaq6eiwEQjESaW/TTz++7z8ytieg7TL2YLX14fX04u3GKZ79hSky2GzeWlseB+NDe/DMNIsL7/I/MLThEJfJ5td+/GQzS4zM/M5ZmY+h9Xqo67uCerrn6K25jFsNi8ej4cnnniCU6dO8dprr/Hiiy+SSOSX0s5oG2dz3VxxH+eBBx7gWP8RYqFJIrPDpBfHITyJIzaJNzVLTW6eenYvvwiQIGCOQnI030sZAobX7s9qK/OqhmVHEwlXE1l/K5aqDtx1nQSbuqlr68Hrr9pze6WSceYnBlmeGiQ1fwW9NIozOk4wNUWjMUuNSlFO33oWmLVZmbbZiqcJm4Mxu5tZu5Uli4G5y4HDmE4RS08yPD/Jc3Mv0rTk5OSEj5PeHLX9Yfq+d+up9az2ehoaP0hH60fxeXs2b7APtM734K8fVLhVXXRNqoa+cB+tiVbUhtEBKWuKK/4rDAeGyVgzuKwu+gP9Jb3Qq5erXFUH8K+8OlKDfQs5DPWNlWKaGUbH/jejo79TUsvo8w1wbOA3CARuu+rnPsztFIvFmJmZYWpqksnxCaZnZkgky6tJc2k7dWaAOu2n3gxQZwbwFnpUTG2ynJ5lNjXKXHKUxdQU5jbLVjvcHnzVNWSVhYF77qPl6HFa+gfwBIIV+3febA7ze+owuR7tZJom4+PjnD9/ngsXLhCNrg0iU8rA4wnj8YTx+6PUN+Rwu1cwjBnY47LtLldrsRe6WOLh6anYirQbldtWppljJXyahfmnWVj4KunM1rOfWCxOamseo77+Kerq3ondnv97zmQyvP7667zwwgslbQfgdDq57777OHXq1JY16OlUgtD0CMszwyQXxsgtjWONTuJOzBDMztFgzFdkpdMVfCxaG4g4m8h4W9DBNuw1Hfgaurl48QKtAQvZ0Ai28BjexCS12Rnq9VJxysW9MrViQdWw6Ggh7mkjF+zEXtuNr7mXuvaj1Da0Fgd/5swcC4kFpmJTzMRnSs9jM0zHp8mZOawGdE97eDhpp7MtQfDI1lPrpU14M2nldNzGlbQFjcJj89Diy8+CsjobyvrLNa6aXScF2O39FE6HiwF6/TR349Ht54tGQ2uilb5w36aVRQESrgS59hyNPY10V6/N1nE1ddH7qeKDHJVSPwb8INABlExVoLU+cjU7WQkSsMt3M37Jx2KXuHDx54hE3lp3q4XOjp+gu/tTWK1772290dopEokUF8RZPa32OO3Gox350G36qdf50O3GQdbMsJCaYC45ymxylEh25zpEgOrmVlr6j9FydIDWo8epaWm7rjMM3EhutPfUQalUOxmGwdjYWDFUJ5Nh3O4IHu9KIVDnz93uGGqPIcvlai/WRq+GaY+nB5ttc8C8nq6mrbQ2iUTeYn7haRbmnyaZGt9yO6VsVFefoqH+Kerq343TUUcul+PNN9/k+eefZ2VlpWR7u93OPffcw0MPPUQgENjyObfcn0IpxuL0MLG59b3gU/jT+V7w2qtckOdaxbSbOVsTEVcraV87qqYbd0MP1W19NLT34XRVphwhshTi5a/+HkvzXyLYGcLh31yzrzVcSlk4nbBxJmklo/c2lsZlddHsa94yfLf4Wqhz1/Hst5/l/ofvZyI6sWmqu/HIOCvplbJfz2ba6Ix20hfpw5vb/HfR0tnC4488Tn9v/w2xavFGFQ3YSql/B/wc8L+Bfw38T6AXeAz4b1rrX7223b16ErDLd7N+yWttMDHxx1wZ/i1Mc22KIre7i2MDv0519f17er4bvZ201oTDYaampkpCdzq9c73kKp925QO3GaBO58M35FjITjEducxMfJiUsfty8i6vj+b+AVr6j9F69BhNPf23bHnJjf6e2i/X0k65XI4rVy4wOPgsM7NvYLfN58O0N4zLFS1n4p51FO5CkC7tle7Bat3fOs90Ik5oYpzFiTFChVN4fg5td3Di1MO0Dpygpe8oDvfe9ktrTSx2kYWFp5lfeLpYcreZoip4L/UNT1Ff9x4cjibOnj3Lc889RyhU+sPbarVy11138fDDD5fMUXwtUokYC1NXWJkZJRUaJbc8ji06hTs5Q1VmlgYztGlu6HIYWjFnqWfJ0ULC04ZR1Ymj7gj+5j7q2/upqm28rh0Ek5de4+zp/5es7TU8DVv3ADsdnbS1/QCOqkdZzJpMxfO93qu94NOxaaZiU9c8FaHdYseFi6h5dQshBZ3BfAmHo4vq+WqSI0mMbGkpldVq5bbbbuPUqVM0NjZe0/4etEoH7MvAz2utP6+UigJ3aK2HlVL/GejQWv/kte/y1ZGAXb6b/Us+kRjj4sWfZ3nl5ZLbW1t/mN6ef1d2zePN2E6mabK8vLyppzubLe/wbMB0F8N2vRmgrqqGRUIob4aJ2QuMjbxFLrfzohDKYqGh6wgtR48VQvdx/LU716TfLG7G99T1UG47GUaCePwKkchFpqe/w9LyOQxjAqfzKoK0u32tNtrTW6iX7sFqdV/tP+Oq5DIZFqcmSoJ0aHyM6OLCro9d/dtqPXqc1mMnaD16HG/V3gJuIjHC/PzTLCw8TST69rbb+f235Xu2697N2FiaZ599lrkNi+4opbj99tt59NFHqau7vn/jpmGwtDDF0vQw0dkRsktjEJ7EGZ/Cn57DMDVxbztpfyeWmi48jUeoaeunoa133wd257JJzr7ye0xNfh5H7fSWU+sp7aGp8YO0dXwUv/+2XXt4tdaE0+GSqQc3Xo5mri44r+eyukrnig4WaqP9nSSXk7z44oucO3cO0ywtrXK73dx7773cf//9u84Pf6OodMBOAANa63Gl1DzwHq31m0qpXuBVrfX1nWdnBxKwy3crfMlrrZme+SyDg/+1ZBS909nEwNFfpa7uiV2f41ZoJ8iH7lAoVBK4Z2dnN6ycpbHZMtjtKRyOFHZ7Eocjid2ewmMFZ7Sd6pUBWhLHcAfcJGwxFuPTTM5eYH5ldNeebn9t/brAfYz6zm4s1us3d/dBuVXeU9dqYzvlcnESiSvFeaRjhXmkU6kpypxFtkDhdncUa6NXyzs8np6rKiO7FqZhsDw7XRqkJ8ZZmZlG673Vfe+kurmF1oETxdBd1dhc9uH4VGqahYWvMr/wNCsrp9murb3ePurrniKROMrLL08yNTW9aZsTJ07w6KOP0tS0eXGT/XDQf3taa0Jzr3Dhzd8hpU9jdW7u1NCmwmO/m56BH6e+/gkslsouGBbJRPL13luE7+nYdLH8w6ZstPnbSoL0dnXRpmkyODjISy+9xOjo6KbXrKmp4dSpU9xxxx04HPu3ANp+qHTAHga+T2v9ulLqNPBprfX/Ukq9F/hzrfXm6vV9IgG7fAf9QbOfUulZLl36RUKhr5fc3tT4PfT1/Sccju1/E97s7aS1JpeLkMmE1p0WyGRCpNILxGLTJJNzZLOLKKIoy+6zJuRyNtLRRqzhToIrR2lauQ1XpgbToUlaYizGp5hdGmYlM08ku4ixzdzANqeT5t6jtBZCd3P/AC7v9Rkktp9u9vfUtTLNDLH4ZV577W9pb7cVA3UqNbmn59FaAfX4fP3U1d6Gz9dfCNJH9j1Ia62JLi4QGl8fpMdYmprAKPPIEYDFaqOmtY269s78qaOTYH0jzz79FQIWzfTFcyxMjOULdXfgrarOh+2B47QOnCj7x2w6EyK08HUWFp5mafmlbRfIcbnacTgeYPCyn8uXs7BhpoijR4/y6KOP0tbWVva/vRIO6m8vlZ5l+NIfMz31eZRzacttzGQDra0/QN+JT2C3V6ak5moksgm+8u2v8N3v/G7su8zClc1meeutt3j55Zc3lQgBdHZ2curUKfr7+7HcpGNwrnmhmQ2+CXwQeB34A+C/K6V+ALgb+OxV76UQ14nL2cTtt/0u8/Nf4tLlXyabzX/Azc79HYtLz3G0/xdpaPjADTnAYis7heZMZpFMJkR63XWtdy7nANjLZ6PNlsNWPQXVU6R5kTEgmwygIu34V/qpXTlGm/1dWEwHGk1SxVlKTLOcmmUls8BKZp5ELkIunWbi3NtMnCscolaK2tZ2Wo8eL/R0D1DV1HLT/L/dilbDdDRyhkj0LNHoWWKxy8X35PjW4+5KaK1IJv0kEkFMs4m6uts50v0InZ33Y7Ptf51/IhImND5aEqQXJ8bIlDnDDwBKUdXYVAjRXcVAXdXUsuU80DV9A8XgmIrFmB68wNSFc0xdOs/s0GWMXOmP2PjKMpdfeYHLr7wAgN3lpqV/gNaB47QNnKCptx+7c3PbOR11tLZ+jNbWj5HNRggtfpOFhadZXHy2ZMxLKjVBKjVBYxO0tNYSiRxh+EqQlZVGwMKlS5e4dOkSR44c4bHHHqOrq6v8trlBGEaC+bmnGR76Y5LZMygFakMVSjbhwMUDnLz3U9Q13XUwO7qBx+6hxlazY7iOxWKcPn2a06dPbxpEr5TixIkTnDp1itbW1uu9uzeMHQO2UupJrfU3gE9Cfn1mrfXvKqWWgYeBvyY/8FGIQ0cpRWPjd1Fd/RCDg7/G7NzfApDNLnH23Keom/t7Bo7+Ck7n4RxwkQ/N0XVB+dpD89WwWr04HLU4HPU4HHU4HHVYrdVcvjyEyzlHzhjEZtu8KIDdHQH3OZKN55jkb9CmBR1twRvuIbhylIZwL63JvuKcqFmdYSU9T7gQuPPnCyxOjrM4Oc7b3/gKAO5AsFhS0tJ/jMYjvdhuskOQN4vVFQ6j0bNbhundFIN0vIpEIkgiESQer8Jj76C/7Ri3d/XSUF2HKhy6NsZTGCq91nmqFCjWfpCptduAtZVNN9yOUmu13Ku3K0U2nWRpZoql6UmWpify51OTJCIr6NUyCr16SWNVtsJNmvz6dPl7vDU1m4J0TWvblgG3HC6fjyN33ceRu+4D8vXcs8ODxcA9fekC6URpuVY2lWTs7TcYezu/+qDFaqPxSE++rGTgBK1Hj+H2l84IYrcHaG76EM1NH8IwEiwuPsfCwtMshL5RUpJnGIt4vYvcdjuYppv5+RYWQx0sLzczPDzM8PAwHR0dPPbYY/T09NzQP5i1NlleeYXpyc8zN/+PUFh8Z/0/ycgq0vMttLZ/lNve8xM31KJe8/PzvPTSS7z99tsYRunRTKfTyd13380DDzxAVVXVwezgIbZjiYhSyiS/8OkfAH+otd5cYHXApESkfLf6YepQ6FtcvPSfSKdni7fZbH56e3+OluYfKH7IX8922jo0rwbngw3NqyfnhuvbzZqw2k5aayKRUYaHv878wqtkMhdxOuewWstYkCPjwRnuwb/ShyvcgyvSjTVX+nqx7EoxbIez+fAdyy4Xw4rVZqPhSG++l7swa8leB3hdb7fC355pponFLhfC9Jl1Ybq8cohsMkA4VkU8UUUing/TyWQArfNlDNWmj26jnm6zgWp945cNAesCPWuJbDXcq8Idau3+4u0WRcyeofHODpzdQRwdfizO7fvLTNNgcWKcyYvnmLp4nqmL54gtbb0c/Hq1bR0lAycD9Q1bhmHTTLO0/FJ+ru3Q14tHDDfK5WwsL7USCnWytNSCadppaWnhscceu24lBdfrby+RGGFm5gtMT/81mezmucW1htiUF4dxDyfv+xe0n7j3UP+QWN9OWmtGRkZ48cUXGRoa2rRtMBjkwQcf5K677sJ1C84MVZEabKXUMeDHgR8BaoGvAv8H+Hu91+WsrhMJ2OW7Fb7kd5PLRRm68n8xNfXnJbdXV5/i2MB/xe3u2HM7lROa14Lz4Q7Ne7FTO62sLHLlyreZnX2JRPIcLtc0Hk8ZI9m1wh5vwh3uxb3Sgyt8BGe8FaVLa0VzZpZINpQP3avhOzNP2swfkq9qbM6H7UJpSV1bx4HOyX2z/e3lw/QlItGzRFbOEA2fIZ4aRFPedGnZZIBorIZwrJpYtIZYrJZcbnOvXq3pp9tooMusp0rv7zzTNxQL2Jt9OLsCOLqCOLsCWP3bH9XRWhNZmGeqELgnL55jaWpi15fx1dbRejRfUtI6cJy69s5Nf1emmSMcfi0/1/bCV0s6NEq3s7C81EJosYOlxTZqatp59NFHOXHiREWDdiX/9rLZFebmv8zMzF8Tiby55TapZQeR0Xqamj/E3e/6YYINh/MI6UbPPPMMjzzyCGfPnuWll17aNDMMQGtrK6dOneLYsWNYb8LB6OWq9CBHG/ka7H8KPAUsAn9MfrDjgaZbCdjlu9m+5K/F8vIrXLj4cySTY8XbLBYXPUf+LVeudPGOdzxxA4Xm+kJo3t9pxcp9P2mtWVxc5Mrw20xNPU8sdga3exa/P4Tdvnu7qZwTV6QLd7gH10oP7nAPtkzVltsmc7Fi2F7J5sN3JLOI3eOiue9occaS5r6jOFz711436t+e1ppcLEFk/iyRpbeIxs4Tz10gaRlFlzn3cC4ZJBqtZSUWJBarJRar2TJMr2p01tDrbeOIr42gzZfvClz9mtI6P5Zv9Xtr9bJevamw7bptVrc3sjmMTJpcJkMuk8HIZPN1yloDqliqpArdxIVr+Utq9RaF1WrDYrVisVixWK0oZcmXqGx8fdbtl14rH1m7//qy1blxdAVwdgVwdgWx1rp27EFNRMJMX7rA1KXzTF04x9zIEKaxcz+a0+stjI/Ih+7Gnj5s9rU6Xq1NItEzLMw/zfzCV0o+b9czTUU43EQo1IFp3MapU+/l9ttvr0iIu9a/PdPMsrj0LDMzXyAU+saWR2RyKSvLQwFyS72cOPUjnHj8yX39fLlaWmtSqRQrKyt89atfZWFhgVhsc7nfwMAAp06doqOj41D3wu+Xiq/kuO6JW4BPAD8GHAFe0Fo/djU7WQkSsMt3o37JXy+GkWJk5H8wNv77lC6L7EWp7C0Zmvfiat9PpmkyOzvLlStDTEy8RjT6Fh7vPAF/CK9vuayV9WzJWtzhI/myknAPrkgnFnPrHjtTm0Szi8XBlOHMAuHsIr62OlqO5nu5W/uP4a+rv25fHof1b09rjRnLkltOYaykyS5FiEYuEcucJ64vkXRcIe2dhDJmkgEwklXEorUsxYLEYjW7hmmnw0lrawstra20trYyPj7OU089dU3/pkQkvDYF3roZPDLJ8lY3BfIDDhuaqF2duaO9g7qOLqqbW7Dadp5lYS+03vyDYO1HBGz8wVDcJG3w5tOv0ONpJTMaITsb3zW0W/x2nF1BHJ0BnN1B7E1elHX793s2nWJm8DJTl/K93NOXL5JN7Txo02q309TTXxw4uX4WIK018fjlQs/208RiF7dpE4hE6onHjtLf/wPcffd7sNuvvs2vbsVLTTR2jpmZLzA39/dblrxoA8LjPpYvB6kKPsI97/sIXXfcfahWrzVNk3g8zsrKCuFweMvzTGbr7zm7zc6dt93Ogw89RG39gU0Udyhdt4BdePIq4OPALwFVWusDO1YgAbt8h/VL/qBFIm9z4cLPEotf/fvoZgnNe1Gp91M2m2VycpLh4WFGRvID4nz+BQL+EP5ACKezjGCkrbii7bhW8oHbHT6CPdFY7IfcSsZIrQ2mzC6QcaXxdTfQfOwoLf3HqO86suXsDVfjoP72tKkxohmM5RTGcprcSuF8OUU2HCOeHSLtHSYVGCUVGCPt20OYTlSTiNWyGAsSjVXvGqatVitNTU20FsJ0a2srNTU1JeUAe2mnTDLB4uTEukVZ8rN4JMIrZT1+la+6Zl2Qzp9q2zoO/aqj69vKTOZIj0fIjERIj4bJTEYht/N3u3JacXT486G7K4Cj3Y/Fsf1XuWkYLIyNMHXxXLGWe9e2Vor6jq7i1ICtA8fx1+QXnkkkRgpzbX9123KL/HZ1BALv4I7bP0FV1bGdX28Le3lPpdNzzM7+HTOzXyAeH9x6f+ZdLF0OEp2oZ+CB93DXez9IbVv7nverEgzDIBKJbBugw+HwpoGJu/FoB8dz7QwYrbjI/7BRdgvKZcPismJx2VCr5878ucVlLd6vnJu3s7isYLPcNL3f1yVgK6XeRb5M5ENACvgM8Pta6zeucj+vmQTs8knA3p5pZhgb+z1Gx34H08z/or8VQ/NeXK/3UyqVYnR0tBC4RwiHR/EHQvj9IQKBED7fYlkDKK2GH3esB+dCd768JNyNNbd7HW80u0w4M0/EWEbV2fEdqafxtj6ajx7D7bu6lciuV1tpQ2OE0xgrKXLLaYzlwvnq9XAaDI1pyZD2TZIOjBbC9Chp31TZYdpM1ZCM17MUDbASCe4apgHq6+tLwnRDQwO2XX6wbNVOuWyW5enJkinwQuNjRBY214juxOn1UtfeVRqkOzqv+v/0oO30ntJZk8xUlPRohMxohPRoBJ3apaTHqnC0+gplJfmebqt3+55jrTUrs9PFGu7pS+dZntl9HoRgQ+O6gZMnqGltI52eZWHhq8zO/SORyGts3x3fTFvbd9Pc/AH8vhNlBbbd/vYMI8nCwteYmf0CS0svUHo0My8Ts7E8GGTpchCHtY07n/oAtz351HV/72QymW17nsPhMNFolKvpJF3Ppi34tAu/dtNtNNJjNmLlOvTCWxUWp7U0qDs3B/Gtg3rhNod1beafA1SxgK2U6iBfDvIJoBN4Fvh94PNa69QOD90XErDLJwF7d4aR4tln/57HHvuuWzI078V+vZ8ikQgjIyOMjIwwPDxMNLqCx7OCPxAi4F/AHwjh8UTKei6n0Y471otzvhNXqBtnrG3TAMqt5MwM4UyIpD2Btd6Jr6eBhrv6qOkurybxattK50yMlXSxhCO3vNYDbawUAvSGj/DVMJ0KjBYD9V7CNLk6Uqkmllf8LCx5iUVrMYydp0EMBoO0tLQUw3Rzc/OeZhcwDYNcNsM3v/IVeloai6sbLk6MsTwztWst8Ho2h5Pato610o5CkPZV1940PWiwt/eUNjW5+QTpkXAhdIcxwruXwNkaPPmBk92FgZNVzh3bML6yXKzhnrp0nvmR4V1Xp3T5A4WBk/le7qrWahZCX+fK8OfJ5c5gsWz9eIejhcbG99JQ/xTB4N3FqRo32qqdtDZZWTnNzOzfMD//jyVTDK4ysorwSICly0Fi0x5a+k9wz/s/SO99pyqy2qzWmmQyuWOA3jjf9NVwahs+7canXSUnf+HcqezY6z2sqAR1VTWYaQOdymGmDMxUDp029mXMQFkUKId1XRBfDePrgriz9P5N4d1pRdmu7QdERRaaUUp9DXgCmCc/qPEPtNab52wR4iZhtbpQql7C9SESCAS44447uOOOO4oDJlfD9sjICKlUCpstjc+/WCgrWdh2AGXaOkE6OAFBoA8syoVH9+OJ9WKfbccx3YkjtXmVT5vFQa2rJX9lKX9Knh5nyDhP2pXGWu/E39NI7e3duFqCKGt5H+BmxsgH5Y09z8spcitpzGhmxy8305Ih7b/6MK1UI5lMM8tLfubmXESjNbuGabvNRrXPS5XHTcDlxOewYUOTS0dJXTrD5bPf4Xwmg5HNkstmMLIZcpnVy4XzTIZcLpc/z2bQ5lqIOlfWnoPFaqW6ubWkN7quvZNgQyMWy607w8FWlEVhb/Jib/LiO9WC1hpjJZ0P24XQnZvfHOZy8wly8wnir+ZnArEGHcVZSpzdQWwNnpIeRW9VNf0PPEz/Aw8D+TKe6cFL+dlKLpxjZugyuUy65DVS0QhXXnuZK6+9DOR/IDX3HaV14Htp7f1XzMdfZW7hq/j9oyVHrjKZaSYmPs3ExKdxOOqpr3839fVPUV31AJZtFkxJJEaYmf1bZmf/dttVQqNTHpYuBwmP+MF0MvDQo9z9U99D45HePbR4vv45FovtGKC3q3/eC492bhuevdqFY13MU3YL9hYv9mYv9hYfjhYftkYPFoeVi888w8A7Tm56fq01OmNgptaCdzGAp3Po1SBeOC/evyGoY1Qgpev8+AMjbUAZPxC3ZbNsLmNZ17u+FtTXlcOsL3cp92V2uT8JfAT40mGZlk8IcetSSlFXV0ddXR333XdfccDk6uIV4+Pj5MZzgMblihZ6ufOh2+tdxmIp/ZA3dYoYbxPzvQ29QC84bI34OIYj0o11sgnHVBuO7NalJW6rD3fWB9PAdJzF585iYpJ1ZrDWu/D1NuLrrsMzD7EXp0tqoI3lNGa8/CWziz3TwRHS/tUwPV12mM4kvMQj1USiNYQT9cRidbuGaUwTSyqBNRnHmopjTcZR2TRJ8l8OM2Xv/bUJrq5w2N5ZrJeubm4tmbFClE8pha3aha3ahfeuBgCMeJbMWKGGezRCZjIGZunfixHOkHxrgeRbC/nncdkKUwPmZytxtPlLegcdbg9dt99F1+35FQuNXJa54Sv5Xu5CHXcqVjp9Zy5Tupqrslio7zpOpu3dxB3DeHyD1NZOYrOt/e1kMgtMTf0FU1N/gc0WpL7uSerrn6Km5hG0jjM59RfMzv4N4fDrW7ZHasXB0uUgy4NBsjE7nmAV93/wfdzx7vfhq978gxsgl8sRiUS2DdCRSGTP9c8bWZQFv9OLDxferANv2lEMz6sBertyDovXjr3Fi6PFVwjVPmx17j2XWCil8uUaThsEr36BHJ0zSwP4TkE9vfV2OlOhGJozMWMmZqz8z9+rcVWDHA8TKREpn5SIlEfaqTyHsZ3WD5gcHh5menq6WKNoseTw+ZbwFwZP+v0hXK74Ls8ISlnxuo/iYQDLfBN6rB7bbDXunK+4Ul+l5XumJ0j6R4j7h0gHRjH887DN4fKNUhEXsXB+WrxIqolYon73MK01lnQSayqOJZkP05Z0CrUPx4eVsuRX47TZaOnpo669oxika9s6bogpz/bb9f77MzMGmYlooYY7TGYsunvAsSkcbYWBk90BnB0BLO7t/0a0abI0PVms4566eH7H2noN5PzVmC2teGqXqKsbp7Z2Aodj62pVq9WDYWRgi/nZcykLy1eCLF8Okph3AYr6riPc/b4PMvDQY5iw4+wb0WgZ8/rvwm63UxWsIuD24bO48eWceOI23CsKb8qOGyeWHQZqr7LVuQu90oWe6WYfFr99TyVRh/HzfCNtanR6u57y1dtWg3r+vvW966uP3aLMfk/af/Oxay8REUKIG4ndbqe7u5vu7m6efPLJTQMmFxZsRCINMFXY3pHI93AXQ/ciVmvpl7HWBrHEeWKcBx9wAmx3BAn4b8eW7sSYqsa4UoVlwYnH9OO1B8veX1MbxM0QEe8QSf8w2eAEunoOSzCCspQXbFMRJ7GVfJiOJpuIphp2D9OAyqSLvdK2TAqnmcNms2GzO7C5HVgDPqw2OzaHHavdgc1ux2Z3YHXkL1vtDmzrL9vthfscWO32dbc7sDryj920TeHyak3rjfAlf6uwOKy4eqpw9VQB+QG12dk46ZFwvqd7JLy5BzCn873foxF4BlBgb/IWB046uwJY1/WCKouF2rYOats6uP1d7wUguhgqzFRynumL51iYGCtOYagAe3QZfWmZtC/IcN1tDLnvJxBcoK52nNq6cVyutVIXwygte9GmIjLuZfFykPBkFabVhbY7qbmjj+rOI+SsNl64dIUvvfIdksmdpyQsh9vtpqqqimAwSFVVFQGvH5/hwpu04QpbsM5lyM0kyy+fsCnsjet7pfOnnVbxvJkoi0K5bTv+aNuN1hqdNdeVtKz1mm8qc1kX4tcH9XLdGv8rQohbksvlYmBggIGBAWDzgMlIBBYXO1hc7Cg8wsTrXSkG7qqqZVyuzXPg5nJhlpafA56DauBe8Hi6sbiPk4g0kBp2kx5yYFlW+Gw1mDpHPBsmrpdI+yYwqqexVC/iqIviqk6zOj5rt8rtXCpIOlFHPFbLcrSGxZVgWWHa5XTSUFdLU0MDzc3NtLa04q8KFkKwXWqWxa7U6iwjrT54pDVfx72YIj0aLs5WkgttCKUasjNxsjNx4i/lC4qsNa51ZSVBbPXukp5Wf20dAw8/zsDDjwOQisWYHrxQHDg5O3QZI5fDFgtjjYUxvAHitc1EwvcxPHwvPt9ivme7bry4emx8yc/CVCvzc51ktR/T4YDetff8VMZkanDvw8v8fn9JgF5/7lduWMySnYqRnYmRORPHWEwBidWm2XHdU+W24SiUdqyWetjq3WWP7xBbU0qhHFZwWLEGdv/s3NJ/Lm8zCdhCiFtGOQMm4/Ea4vEaZmf7AbBaM8XAXV8fxeOZR6nNsw4kEiMkEiP5K+1g6XTi9p3AQhcLc6P4gis4M2M4Ka+O0OHoADpIxOuZn3czMQGZzO5B2G6350P0uinyqqqqbqoZNMTBU0phq3Njq3PjvbcJACOaKc5Skh6NkJ2ObRqkayylSCylSLw+D4DFa8PRuTZw0t7iLQmRLp+PI3fdx5G77gMgl8kwOzxYDNzTly5gG79Ezu0jU9dMjDpisTpGR+/C5YqhtSKdzi92wx7ylMVi2TI4r54HAgFsNlt+hpaFBNmZOJnpONm3Y2SnQyzGy+/ptFY7sTf78oG60DttDe48Y4s4/CRgCyFuSVsNmJyZmSkG7vHx8fxMF4aDlZUWVlZamBiH/ADKGMHgEs0tSfz+EDDBxv4o00wTibwOvI7NDaltB70rXK5OFJ0kEvUshDyMj0MysfthY6UUjY2NJVPk1dfXV2SJaSH2yup34LmtDs9t+cVkzHSOzHg0X1YyGiEzEUVnSwtgzXiO1PlFUucXgfxMF44Of3G2EkdHAItz7f1sczhoGzhB28CJ/ONNg8WJ8WIN98jQIGGHh5y/mlRq+3mqHQ7HjgHa5/OVLIIE+br07Gyc7OU40ekRMjNxcrPxTf+mbVkU9kZPcRYPe7MXR7MXi0cG696MJGALIQT5HqvVkPrII4/sMGBSkUr5SaX8zBXGYyllUFMTpb3DoKpqCYtljGx2qzk2FG5XFxZLF8lUI4shL+PjEImsT9/bB+vq6uqSnummpiYcjqs8zCnEdWZx2nD1VePqqwbyM0lkpmPFxW8yo2HMxIYxD1mT9JUw6SthogAWsLf4cBaWeHd0BrD6197zFouV+s5u6ju7ueup70JrTWRhnnOvvcrb584TTSbpO36Cprb2kgDtdrt37CE2YhlS03Ey07F8mct0LF8CU2a5tHJa8wF63Swe9kbPNc/BLG4cErCFEGILGwdMJpNJxsbG1g2YXChuq7WVxcUqFhcBaoE+qqosdB+Burok83NhLJYjTE4qFhbWzz6w/VyuXq+3JEy3tLTg8Xiu1z9XiOtO2Sw4O/Kzi/gfKyyAE0oWe7jTo2GM5dJ5sjEhOxkjOxkj9kJ+pUhbnbt04GStqxiWlVIEGxp56P3fzUPv/+5dB85qU5NbSpFdF6Qz0/H8HPRlsgYcxdKO1VIPa7XrUKw6KA6OBGwhhCiD2+3ecsDkauCOREpXk1xZMXnjdQAn0ABsrtteZbfbS8o8WltbCQaDUoMpbmrKorA3eLA3eOCBZgBy4XSxhjszEiE7F9/Ua5wLJcmFkiReyx9Csvjt+akBC6Hb3uzdMtzqrEl2Lk52Ok5mJkZ2Oj8As+z5lRXY6j1r80sXZvGw+uQokthMArYQQlyFrQZMrobt1QGTW7FYLDQ2NpaE6bq6uk31nkLcimxBJ7Y7GvDckV8Ax0zmSI+tDZzMTEQ3TWtnRrMkz4RIngkB+fIMR0d+Pu6qccXS3CUy0zFyC4my50BWdktprXSLD3uTB2WX8Q2iPBKwhRDiGq0fMHn//feXDJicmJhgeXmZu+++u1g3bZcVCIUoi8Vtwz1Qg3sgv6KizppkpqKkRwqheyyCTpX2QOu0QXpwhfTgCnVYSDC/82v47IUFWtZm8bDV7n3VQyHWk4AthBAVtn7AJOQXUHnwwQcPeK+EuPEpu6VQex0E2tGmJjuXWFdWEsaIbF8/batzrw06bPHiaPZd/XzIQuxAArYQQgghbkjKonAUprvznWrJL4CznC4s7x5hemqaznv68j3TTZ5bZtVDcfDknSaEEEKIm4JSCluNC1uNC+/djbz1zBQnTrUc9G6JW5CMqhFCCCGEEKKCJGALIYQQQghRQRKwhRBCCCGEqCAJ2EIIIYQQQlTQvgZspdR7lVKXlFJDSqmf3eL+71FKva2UelMp9ZpS6pH93D8hhBBCCCGu1b7NIqKUsgK/A7wbmAROK6W+qLU+v26zbwBf1FprpdTtwGeBgf3aRyGEEEIIIa7VfvZg3w8Maa2HtdYZ4C+B71m/gdY6prVeXQPVC2iEEEIIIYS4gexnwG4FJtZdnyzcVkIp9WGl1EXgS8A/3ad9E0IIIYQQoiLUWofxdX4hpb4feEpr/ROF6x8H7tda//Q22z8G/ILW+l1b3PdJ4JMA9fX193z2s5+9fjt+E4nFYvh8voPejUNP2qk80k7lk7Yqj7RT+aStyiPtVB5pp/I98cQT39Fa37vbdvu5kuMk0L7uehswvd3GWutnlVI9Sqk6rXVow32/B/wewNGjR/U73vGO67C7N59nnnkGaavdSTuVR9qpfNJW5ZF2Kp+0VXmkncoj7VR5+1kichroU0p1K6UcwMeAL67fQCnVq5RShct3Aw5gcR/3UQghhBBCiGuybz3YWuucUuqngKcBK/BprfU5pdQ/L9z/u8D3Av9EKZUFksBH9X7VsAghhBBCCFEB+1kigtb6y8CXN9z2u+su/ybwm/u5T0IIIYQQQlSSrOQohBBCCCFEBUnAFkIIIYQQooIkYAshhBBCCFFBErCFEEIIIYSoIAnYQgghhBBCVJAEbCGEEEIIISpIArYQQgghhBAVJAFbCCGEEEKICpKALYQQQgghRAVJwBZCCCGEEKKCJGALIYQQQghRQRKwhRBCCCGEqCAJ2EIIIYQQQlSQBGwhhBBCCCEqSAK2EEIIIYQQFSQBWwghhBBCiAqSgC2EEEIIIUQFScAWQgghhBCigiRgCyGEEEIIUUESsIUQQgghhKggCdhCCCGEEEJUkARsIYQQQgghKkgCthBCCCGEEBUkAVsIIYQQQogKkoAthBBCCCFEBUnAFkIIIYQQooIkYAshhBBCCFFBErCFEEIIIYSoIAnYQgghhBBCVJAEbCGEEEIIISpIArYQQgghhBAVJAFbCCGEEEKICpKALYQQQgghRAVJwBZCCCGEEKKCJGALIYQQQghRQRKwhRBCCCGEqCAJ2EIIIYQQQlSQBGwhhBBCCCEqSAK2EEIIIYQQFSQBWwghhBBCiAqSgC2EEEIIIUQFScAWQgghhBCigiRgCyGEEEIIUUESsIUQQgghhKggCdhCCCGEEEJUkARsIYQQQgghKkgCthBCCCGEEBUkAVsIIYQQQogKkoAthBBCCCFEBUnAFkIIIYQQooIkYAshhBBCCFFBErCFEEIIIYSoIAnYQgghhBBCVJAEbCGEEEIIISpIArYQQgghhBAVJAFbCCGEEEKICpKALYQQQgghRAVJwBZCCCGEEKKCJGALIYQQQghRQRKwhRBCCCGEqCAJ2EIIIYQQQlSQBGwhhBBCCCEqSAK2EEIIIYQQFSQBWwghhBBCiAra14CtlHqvUuqSUmpIKfWzW9z/w0qptwunF5VSd+zn/gkhhBBCCHGt9i1gK6WswO8A7wOOAz+olDq+YbMR4HGt9e3AfwF+b7/2TwghhBBCiErYzx7s+4EhrfWw1joD/CXwPes30Fq/qLVeLlx9GWjbx/0TQgghhBDimimt9f68kFLfB7xXa/0ThesfBx7QWv/UNtv/DDCwuv2G+z4JfBKgvr7+ns9+9rPXb8dvIrFYDJ/Pd9C7cehJO5VH2ql80lblkXYqn7RVeaSdyiPtVL4nnnjiO1rre3fbzrYfO1Ogtrhty3SvlHoC+HHgka3u11r/HoXykaNHj+p3vOMdFdrFm9szzzyDtNXupJ3KI+1UPmmr8kg7lU/aqjzSTuWRdqq8/QzYk0D7uuttwPTGjZRStwO/D7xPa724T/smhBBCCCFERexnDfZpoE8p1a2UcgAfA764fgOlVAfwBeDjWuvL+7hvQgghhBBCVMS+9WBrrXNKqZ8CngaswKe11ueUUv+8cP/vAr8A1AL/UykFkCunzkUIIYQQQojDYj9LRNBafxn48obbfnfd5Z8ANg1qFEIIIYQQ4kYhKzkKIYQQQghRQRKwhRBCCCGEqCAJ2EIIIYQQQlSQBGwhhBBCCCEqSAK2EEIIIYQQFSQBWwghhBBCiAqSgC2EEEIIIUQFScAWQgghhBCigiRgCyGEEEIIUUESsIUQQgghhKggCdhCCCGEEEJUkARsIYQQQgghKkgCthBCCCGEEBUkAVsIIYQQQogKkoAthBBCCCFEBUnAFkIIIYQQooIkYAshhBBCCFFBErCFEEIIIYSoIAnYQgghhBBCVJAEbCGEEEIIISpIArYQQgghhBAVJAFbCCGEEEKICpKALYQQQgghRAVJwBZCCCGEEKKCJGALIYQQQghRQRKwhRBCCCGEqCAJ2EIIIYQQQlSQBGwhhBBCCCEqSAK2EEIIIYQQFSQBWwghhBBCiAqSgC2EEEIIIUQFScAWQgghhBCigiRgCyGEEEIIUUESsIUQQgghhKggCdhCCCGEEEJUkARsIYQQQgghKkgCthBCCCGEEBUkAVsIIYQQQogKkoAthBBCCCFEBUnAFkIIIYQQooIkYAshhBBCCFFBErCFEEIIIYSoIAnYQgghhBBCVJAEbCGEEEIIISpIArYQQgghhBAVJAFbCCGEEEKICpKALYQQQgghRAVJwBZCCCGEEKKCJGALIYQQQghRQRKwhRBCCCGEqCAJ2EIIIYQQQlSQBGwhhBBCCCEqSAK2EEIIIYQQFSQBWwghhBBCiAqSgC2EEEIIIUQFScAWQgghhBCigiRgCyGEEEIIUUESsIUQQgghhKggCdhCCCGEEEJUkARsIYQQQgghKkgCthBCCCGEEBUkAVsIIYQQQogKkoAthBBCCCFEBe1rwFZKvVcpdUkpNaSU+tkt7h9QSr2klEorpX5mP/dNCCGEEEKISrDt1wsppazA7wDvBiaB00qpL2qtz6/bbAn4/wIf2q/9EkIIIYQQopL2swf7fmBIaz2stc4Afwl8z/oNtNbzWuvTQHYf90sIIYQQQoiK2c+A3QpMrLs+WbhNCCGEEEKIm8a+lYgAaovb9FU9kVKfBD5ZuJpWSp296r26tdQBoYPeiRuAtFN5pJ3KJ21VHmmn8klblUfaqTzSTuU7Ws5G+xmwJ4H2ddfbgOmreSKt9e8BvweglHpNa33vte/ezU/aqjzSTuWRdiqftFV5pJ3KJ21VHmmn8kg7lU8p9Vo52+1nichpoE8p1a2UcgAfA764j68vhBBCCCHEdbdvPdha65xS6qeApwEr8Gmt9Tml1D8v3P+7Sqkm4DUgAJhKqX8FHNdaR/ZrP4UQQgghhLgW+1kigtb6y8CXN9z2u+suz5IvHdmL36vArt0qpK3KI+1UHmmn8klblUfaqXzSVuWRdiqPtFP5ymorpfVVjTMUQgghhBBCbEGWShdCCCGEEKKCbuiAvdvS6yJPKfVppdS8TGe4M6VUu1LqW0qpC0qpc0qpTx30Ph1GSimXUupVpdRbhXb65YPep8NMKWVVSr2hlPqHg96Xw0wpNaqUOqOUerPcUfq3IqVUlVLq80qpi4XPqlMHvU+HkVLqaOG9tHqKFMZ1iQ2UUv+68Fl+Vin1GaWU66D36TBSSn2q0Ebnynkv3bAlIoWl1y+zbul14Ac3LL0uAKXUY0AM+BOt9cmD3p/DSinVDDRrrV9XSvmB7wAfkvdUKaWUArxa65hSyg48D3xKa/3yAe/aoaSU+jfAvUBAa/1dB70/h5VSahS4V2stc/HuQCn1x8BzWuvfL8zI5dFarxzwbh1qhbwwBTygtR476P05TJRSreQ/w49rrZNKqc8CX9Za/9HB7tnhopQ6SX4F8vuBDPAV4P+jtR7c7jE3cg/2rkuvizyt9bPA0kHvx2GntZ7RWr9euBwFLiCrjW6i82KFq/bC6cb8pX6dKaXagA8Av3/Q+yJufEqpAPAY8AcAWuuMhOuyPAlckXC9LRvgVkrZAA9XuUbJTe4Y8LLWOqG1zgHfBj680wNu5IAtS6+L60Yp1QXcBbxywLtyKBXKHt4E5oGvaa2lnbb228C/B8wD3o8bgQa+qpT6TmG1XrHZEWAB+MNC2dHvK6W8B71TN4CPAZ856J04jLTWU8B/A8aBGSCstf7qwe7VoXQWeEwpVauU8gDvp3TxxE1u5IBdsaXXhVhPKeUD/hr4VzIH+9a01obW+k7y02reXzh8JtZRSn0XMK+1/s5B78sN4mGt9d3A+4B/WShtE6VswN3A/9Ja3wXEARl/tINCGc0Hgc8d9L4cRkqpavJH/7uBFsCrlPqRg92rw0drfQH4TeBr5MtD3gJyOz3mRg7YFVt6XYhVhZrivwb+XGv9hYPen8OucHj6GeC9B7snh9LDwAcLtcV/CbxTKfVnB7tLh5fWerpwPg/8DfkyQFFqEphcd8To8+QDt9je+4DXtdZzB70jh9S7gBGt9YLWOgt8AXjogPfpUNJa/4HW+m6t9WPky263rb+GGztgy9LroqIKg/f+ALigtf6/D3p/DiulVL1Sqqpw2U3+A/rige7UIaS1/jmtdZvWuov859M3tdbSM7QFpZS3MLCYQsnDe8gfkhXrFBZjm1BKHS3c9CQgg7B39oNIechOxoEHlVKewnfgk+THH4kNlFINhfMO4CPs8r7a15UcK2m7pdcPeLcOJaXUZ4B3AHVKqUngF7XWf3Cwe3UoPQx8HDhTqC8G+PnCCqRiTTPwx4WR+Rbgs1prmYJOXItG4G/y3+/YgL/QWn/lYHfp0Ppp4M8LHUvDwI8d8P4cWoVa2XcD/+yg9+Ww0lq/opT6PPA6+ZKHN5BVHbfz10qpWiAL/Eut9fJOG9+w0/QJIYQQQghxGN3IJSJCCCGEEEIcOhKwhRBCCCGEqCAJ2EIIIYQQQlSQBGwhhBBCCCEqSAK2EEIIIYQQFSQBWwghxJaUUlop9X0HvR9CCHGjkYAthBCHkFLqjwoBd+Pp5YPeNyGEEDu7YReaEUKIW8DXyS9+tF7mIHZECCFE+aQHWwghDq+01np2w2kJiuUbP6WU+pJSKqGUGlNKlSzFrpS6TSn1daVUUim1VOgVD27Y5keVUmeUUmml1JxS6o827EONUupzSqm4Ump4i9f4hcJrp5VSs0qpP7keDSGEEDcSCdhCCHHj+mXgi8Cd5Jc3/hOl1L1QXCb6K0AMuB/4MPAQ8OnVByul/hnwv4E/BG4H3g+c2/AavwD8HXAH8FfAp5VSnYXHfy/wM8C/APqA7wJerfw/UwghbiyyVLoQQhxChZ7kHwFSG+76Ha31f1BKaeD3tdY/ue4xXwdmtdY/opT6SeC/AW1a62jh/ncA3wL6tNZDSqlJ4M+01j+7zT5o4De01j9XuG4DIsAntdZ/ppT6N8A/A05qrbOV+rcLIcSNTmqwhRDi8HoW+OSG21bWXX5pw30vAR8oXD4GvL0argteBEzguFIqArQC39hlH95evaC1zimlFoCGwk2fAz4FjCilnibfY/5FrXV6l+cUQoibmpSICCHE4ZXQWg9tOIXKfKwCtjtEqQv3l2Njz7Sm8N2htZ4AjpLvxY4AvwV8RynlLfO5hRDipiQBWwghblwPbnH9QuHyeeAOpZR/3f0Pkf/cv6C1ngOmgCevZQe01imt9Ze01v8auA84ATx8Lc8phBA3OikREUKIw8uplGracJuhtV4oXP6IUuo08AzwfeTD8gOF+/6c/CDIP1FK/QJQTX5A4xe01kOFbX4N+O9KqTngS4AHeFJr/Vvl7JxS6hPkv0deIT+Y8qPke7wH9/jvFEKIm4oEbCGEOLzeBcxsuG0KaCtc/iXge4H/H7AA/JjW+jSA1jqhlHoK+G3yM3ukyM8G8qnVJ9Ja/y+lVAb4t8BvAkvAl/ewfyvAfyA/mNJOvtf8I1rrkT08hxBC3HRkFhEhhLgBFWb4+H6t9ecPel+EEEKUkhpsIYQQQgghKkgCthBCCCGEEBUkJSJCCCGEEEJUkPRgCyGEEEIIUUESsIUQQgghhKggCdhCCCGEEEJUkARsIYQQQgghKkgCthBCCCGEEBUkAVsIIYQQQogK+v8DUmYc3Z6AKJwAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# extra code – visualize the learning curves of all the optimizers\n",
"\n",
"for loss in (\"loss\", \"val_loss\"):\n",
" plt.figure(figsize=(12, 8))\n",
" opt_names = \"SGD Momentum Nesterov AdaGrad RMSProp Adam Adamax Nadam AdamW\"\n",
" for history, opt_name in zip((history_sgd, history_momentum, history_nesterov,\n",
" history_adagrad, history_rmsprop, history_adam,\n",
" history_adamax, history_nadam, history_adamw),\n",
" opt_names.split()):\n",
" plt.plot(history.history[loss], label=f\"{opt_name}\", linewidth=3)\n",
"\n",
" plt.grid()\n",
" plt.xlabel(\"Epochs\")\n",
" plt.ylabel({\"loss\": \"Training loss\", \"val_loss\": \"Validation loss\"}[loss])\n",
" plt.legend(loc=\"upper left\")\n",
" plt.axis([0, 9, 0.1, 0.7])\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Learning Rate Scheduling"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Power Scheduling"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```lr = lr0 / (1 + steps / s)**c```\n",
"* Keras uses `c=1` and `s = 1 / decay`"
]
},
{
"cell_type": "code",
"execution_count": 66,
"metadata": {},
"outputs": [],
"source": [
"optimizer = tf.keras.optimizers.SGD(learning_rate=0.01, decay=1e-4)"
]
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1719/1719 [==============================] - 2s 915us/step - loss: 0.6818 - accuracy: 0.7678 - val_loss: 0.4840 - val_accuracy: 0.8276\n",
"Epoch 2/10\n",
"1719/1719 [==============================] - 2s 877us/step - loss: 0.4702 - accuracy: 0.8361 - val_loss: 0.4421 - val_accuracy: 0.8398\n",
"Epoch 3/10\n",
"1719/1719 [==============================] - 2s 886us/step - loss: 0.4242 - accuracy: 0.8491 - val_loss: 0.4110 - val_accuracy: 0.8534\n",
"Epoch 4/10\n",
"1719/1719 [==============================] - 2s 880us/step - loss: 0.4012 - accuracy: 0.8580 - val_loss: 0.3900 - val_accuracy: 0.8574\n",
"Epoch 5/10\n",
"1719/1719 [==============================] - 2s 894us/step - loss: 0.3821 - accuracy: 0.8636 - val_loss: 0.3835 - val_accuracy: 0.8626\n",
"Epoch 6/10\n",
"1719/1719 [==============================] - 2s 927us/step - loss: 0.3685 - accuracy: 0.8687 - val_loss: 0.3836 - val_accuracy: 0.8614\n",
"Epoch 7/10\n",
"1719/1719 [==============================] - 1s 854us/step - loss: 0.3580 - accuracy: 0.8706 - val_loss: 0.3709 - val_accuracy: 0.8646\n",
"Epoch 8/10\n",
"1719/1719 [==============================] - 1s 851us/step - loss: 0.3490 - accuracy: 0.8756 - val_loss: 0.3736 - val_accuracy: 0.8614\n",
"Epoch 9/10\n",
"1719/1719 [==============================] - 1s 852us/step - loss: 0.3413 - accuracy: 0.8786 - val_loss: 0.3536 - val_accuracy: 0.8698\n",
"Epoch 10/10\n",
"1719/1719 [==============================] - 1s 844us/step - loss: 0.3343 - accuracy: 0.8801 - val_loss: 0.3546 - val_accuracy: 0.8698\n"
]
}
],
"source": [
"history_power_scheduling = build_and_train_model(optimizer) # extra code"
]
},
{
"cell_type": "code",
"execution_count": 68,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEbCAYAAADwPQLqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAwgUlEQVR4nO3deXxU9dn38c+VkJCwhp0QVhUCKCiggmtxRbQKj0sf21rXp9RW77u2rlRt3arWpVXvm2ptS63WfalSpVIQU0WLIosgIBCRHVmEAIEASbieP84JDsMkGUgyk2S+79drXpk55/xmrjnGXPx2c3dEREQOVlqyAxARkYZNiURERGpEiURERGpEiURERGpEiURERGpEiURERGpEiUSkHjKzy82suI7e+zMzu+MAyywzsxsqey2pTYlE6i0ze8rMPHyUmtlSM3vIzJonO7bqmFkvM/ubma0ys11mtsbM3jKzQcmOrZYcA/w+2UFI/dAk2QGIVGMK8AMgAzgJ+BPQHPhxMoOqYGYZ7l4afQyYDHwBfAdYDeQBZwBtEx5kHXD3DcmOQeoP1Uikvtvl7l+5+0p3fw54FhgNYGZNzewRM1tnZjvNbLqZnVhR0Mw+MrObI14/G9ZuOoevm5nZbjM7IXxtZnaTmX1hZiVmNs/MLoko3zMs/10zm2pmJcCPYsR8OHAocI27f+juy8Ofd7r7OxHv18rMHjeztWH8C83s/0a+kZmdFjZFbTezd82sV9T5c81sZlj+SzP7tZllRpzvaGZvhN9nuZldGR1s+J0ujDpWZdNVjKYuN7MxZvZyGOvSyHsXXjPUzGaFsc42s7PDcsMr+xxpGJRIpKEpIaidADwA/F/gSmAQMA9428xyw/MFwCkRZb8FbASGh69PAEqBj8PX9wBXAdcA/YH7gD+Y2TlRMdxH0KzTH3g9RowbgD3ABWYWs9ZvZgb8M4zpivC9fg7sjrisKTA2/H7HATnAExHvMYIgsf4vQfK6ErgQuDfiPZ4CDgNOJ0jAlwI9Y8VUC34JvAEcCbwIjDezHmGsLYA3gc+BIcBNwIN1FIckmrvroUe9fBD8EXwz4vWxBIngRYLmrd3ApRHn0wmak+4JX48EigmacHsD24BfA38Iz/8amBw+b06QpE6KiuERYGL4vCfgwPVxxH4NsD38/H8DdwOHR5w/gyDZ9Kuk/OXhZ+VHHPt++J3TwtfvAbdHlRsdfqYBfcL3OCHifA+gHLgj4pgDF0a9zzLghgN47cB9Ea+bADuAS8LXPwI2AdkR13wvLDc82b9retTsoRqJ1HdnmVmxme0E/kPwx/O/CJqOMoAPKi509/Lwmv7hofcJ/lV/DEEt5H2CPpfh4fnhBLUWwjJZBDWa4ooHQV/MoVExfVJd0O4+DuhM8MdyGjAKmGNmPwgvGQSsdfeFVbzNLndfFPF6Tfidc8LXQ4Bbo+J9jiApdgb6ESSrihoX7r48fJ+6MDfic8oIamYdw0N9gc/cvSTi+o/qKA5JMHW2S333HjCGoAlqjYcd2xHNV7GWrw7+iexebGazCJq3DgfeJUg0PcysN0GCuSksU/GPqnOBFVHvVxr1ens8gbv7NmACMMHMbgMmEdRMniGoMVSnLPoto2JNA+4EXo5RdkOcn1HxvtHXZsS6sBrR98n5JlYj9n8raQSUSKS+2+HuhTGOFxI085wILAUws3SCvoTnIq4rIEgk/YBH3H2nmX0E3Mq+/SMLgF1AD3efWttfwt3dzD4HBoeHZgG5ZtavmlpJVWYBfSu5P5jZQoI/5McAH4bHugNdoi7dAORGlOsU+bqWLAQuNbPsiFrJsbX8GZIkSiTSILn7djN7HLjfzDYCXwI/Azqx7/yGAuB6glrErIhjtwLvVtRw3H2bmT0EPBR2hL8HtACGAXvc/cl4YzOzowhqCs8QJKjdBJ3qVwLPh5e9Q9C086qZ/QxYTNAp3tzdX4/zo+4C3jSz5cBLBDWYI4Bj3f0md19kZm8TDBgYQ9AH9NvwZ6SpwDVm9iFB/8m9wM54v2+cniUYzPBHM7uXIJn9IjynmkoDpz4SachuJvgD+hdgDjAQOMvd10Zc8z7BH6r3wz4UCJq40vmmf6TC7cAdwA3AfIK5IBcQJKkDsYqglvRLYHoY2/XAQwT9O7j7HoLBAB8AfyP4F/ujQOb+bxebu08CziGocX0cPm5h36a5y8P4pwL/IKitLYt6q+vDeAuAVwjm6qyPN444Yy0maDY8HJhNMGLrjvB0bSctSTBz1z8GRCTxzGwU8Hego7tvTHY8cvDUtCUiCWFmlxHUfFYSNME9AvxDSaThS2jTlpmdZWaLzKzQzG6Jcd7M7LHw/FwzGxxxbryZrTezz6LKtDWzyWa2JPzZJhHfRUQOWCeCfqNFwDiCCZmXVFlCGoSENW2FI2oWE0zEWgXMAL7r7gsirjmboA35bGAo8Ki7Dw3PnUww0eppdz8ioswDwCZ3vz9MTm3cfe+yGCIiUrcSWSM5Fih096Xuvht4gWCSVqRRBInC3X06kFMxX8Dd3yOYGRttFPDX8PlfCddhEhGRxEhkH0keQdtohVUEtY7qrskD1lK5ThWjdNx9rZl1jHVROPxxDEBadqshTVp/c1nPVhq8BrBnzx7S0nQvIumexKb7Eltjvy+LFy/e6O4doo8nMpHEmmUb3a4WzzUHJZwH8CRA09zennvZIwDk5WTzwS2n1sZHNHgFBQUMHz482WHUK7onsem+xNbY70s4Z2k/iUydq4BuEa+7sv+aP/FcE21dRfNX+DPu8e/ZGencOCI/3stFRCSGRCaSGUBvC3aOywQuJliHKNIEgmUUzMyGAVuiJpfFMgG4LHx+GcEy1nH5+Rl9GD0oL97LRUQkhoQlknA10GsJFq5bCLzk7vPN7Gozuzq8bCLBOPNC4I/ATyrKm9nzBAvu5VuwfelV4an7gTPMbAnBiLD7q4ulW8s0MtPTWLV5Ry19OxGR1JXQCYnuPpEgWUQeeyLiuRPs4xCr7HcrOf41cNqBxJFu8O0ju/DyzFX8/Mx8WmcfzEKnIiICKbzW1hUn9GTH7nJemrGy+otFRKRSKZtIjshrzdBebXnqw2WUle9JdjgiIg1WyiYSgCtP7MXqohImL1iX7FBERBqslE4kp/frRLe22Yz/4EBXCRcRkQopnUjS04zLj+/FjGWbmbuqKNnhiIg0SCmdSAC+c3RXWjRtwl8+WJbsUEREGqSUTyQtszK46OiuvDl3Deu2aqM2EZEDlfKJBODy43tStsf52/SYy8iIiEgVlEiAHu2ac3q/Tjz70Qp2lpZXX0BERPZSIgldeUIvNm3fzRtzVic7FBGRBkWJJDTskLb0y23F+GnLSNSukSIijYESScjMuPKEnixat40Pv/g62eGIiDQYSiQRzj2yC+1bZDJ+miYoiojES4kkQlZGOt8f2oN3Pl/Plxu3JzscEZEGQYkkyveHdSczPY2ntGyKiEhclEiidGyZxbnhXiVbSkqTHY6ISL2nRBKD9ioREYmfEkkM2qtERCR+SiSV0F4lIiLxUSKphPYqERGJjxJJJbRXiYhIfJRIqqC9SkREqqdEUgXtVSIiUj0lkmpcfnxPSsud0x/+N71ueYsT7p/K67O1QrCISIUmyQ6gvpu9oog0g227ygBYXVTC2NfmATB6UF4yQxMRqRdUI6nGg5MWsSdqVfmS0nIenLQoOQGJiNQzSiTVWFNUckDHRURSjRJJNbrkZB/QcRGRVKNEUo0bR+STnZG+z7GmTdK4cUR+kiISEalf1NlejYoO9QcnLdrbnJXfqYU62kVEQkokcRg9KG9v4hj3biEPTlrEtCUbObF3+yRHJiKSfGraOkBXndiL7m2bccc/5lOqlYFFRJRIDlRWRjq3f7s/heuL+euHy5IdjohI0imRHITT+3XkW3068OiUJWzYtivZ4YiIJJUSyUEwM355bn92lpXz4KTPkx2OiEhSJTSRmNlZZrbIzArN7JYY583MHgvPzzWzwdWVNbOjzGy6mc0xs0/M7NhEfJdDO7TgyhN68dInq5izsigRHykiUi8lLJGYWTowDhgJ9Ae+a2b9oy4bCfQOH2OAx+Mo+wBwp7sfBfwyfJ0Q1556GB1aNuVXE+azJ3odFRGRFJHIGsmxQKG7L3X33cALwKioa0YBT3tgOpBjZrnVlHWgVfi8NbCmrr9IhZZZGYwd2ZdPVxbxyqxVifpYEZF6JZHzSPKAlRGvVwFD47gmr5qy1wGTzOwhgsR4fKwPN7MxBLUcOnToQEFBwcF8h/3kuHNYThr3TJhLi82FNMuwWnnfZCguLq61+9JY6J7EpvsSW6rel0Qmklh/YaPbgyq7pqqyPwZ+5u6vmtl3gD8Dp+93sfuTwJMA+fn5Pnz48DjDrl6H3ls4b9w0Zu7qxO1nRLfWNRwFBQXU5n1pDHRPYtN9iS1V70sim7ZWAd0iXndl/2aoyq6pquxlwGvh85cJmsESakDX1lx8TDf++uEylqzbluiPFxFJqkQmkhlAbzPrZWaZwMXAhKhrJgCXhqO3hgFb3H1tNWXXAN8Kn58KLKnrLxLLDWfm0ywznTv/sQB3dbyLSOpIWCJx9zLgWmASsBB4yd3nm9nVZnZ1eNlEYClQCPwR+ElVZcMyPwQeNrNPgXsJ+0ESrV2Lpvz8jD5MK9zIpPnrkhGCiEhSJHTRRnefSJAsIo89EfHcgWviLRsenwYMqd1ID84lw3rw/McrueetBQzP70BW1PLzIiKNkWa216Im6Wn86rz+rNpcwh/+vTTZ4YiIJIQSSS07/tD2nDMgl98XFLJq845khyMiUueUSOrAL87phxncO3FhskMREalz2tiqDuTlZPOT4Yfx28mLOfqeyXxdvJsuOdncOCJfOyuKSKOjRFJHcltnYcDG4t0ArC4qYexr8wCUTESkUVHTVh15ZMqS/abtl5SW8+CkRUmJR0SkriiR1JE1RSUHdFxEpKFSIqkjXXKyD+i4iEhDpURSR24ckU921ITENIMbzuiTpIhEROqGEkkdGT0oj/vOH0BeTjYGtM5uwh6H9cXa411EGheN2qpDowfl7R2h5e5c+9xsHpi0iEHd23Bsr7ZJjk5EpHaoRpIgZsb9FwygW5ts/uv5WWxUzUREGgklkgRqmZXB778/hKIdpVz3whzKtc+7iDQCcScSMxtpZm+a2QIz6xYe+39mdlrdhdf49O/SirtGHc60wo089k5Stk4REalVcSUSM/s+8BLBplG9gIzwVDpwU92E1nh95+huXDC4K49NXcL7SzYkOxwRkRqJt0ZyE/BDd/8ZUBZxfDpwVG0H1diZGXePPpzeHVtw3Qtz+GrLzmSHJCJy0OJNJL2B/8Q4Xgy0qr1wUkezzCb8/vtDKCkt59rnZlFavifZIYmIHJR4E8kaINZMupOBL2ovnNRyWMcW3Hf+AD5ZvpmHtAaXiDRQ8SaSJ4HHzOyE8HU3M7sMeAB4vE4iSxGjjsrjkmHd+cN7S5m8QHu9i0jDE1cicfcHgNeAyUBz4F3gCeAJdx9Xd+GlhtvO6c8Rea24/qU5rNykXRVFpGGJe/ivu98KtAeOBYYBHdz99roKLJVkZaTz++8NwYFrnpvFrrLyZIckIhK3uJZIMbPxwE/dfRvwScTx5sD/uPuVdRRfyujerhkPX3QkY56ZyVVPzeDLjTtYU1SinRVFpN6Lt0ZyGRBr/fNs4NLaCye1nXl4Z07Jb8+0wq9ZXVSC883Oiq/PXp3s8EREYqoykZhZWzNrBxjQJnxd8egAfBtQD3EtWrSueL9j2llRROqz6pq2NgIePhbEOO/Ar2o7qFS2tij25ETtrCgi9VV1ieQUgtrIVOACYFPEud3AcndfU0expaQuOdmsjpE0tLOiiNRXVSYSd/83gJn1Ala6u6Zf17EbR+Qz9rV5lJR+M3LLgJ8MPzR5QYmIVCGuUVvuvhzAzLoA3YHMqPPv1X5oqalidNaDkxaxpqiEdi0y2bx9Ny/MWMm5R3WhVVZGNe8gIpJY8Q7/7QI8R7AkihP8IzlyM430WOXk4ETurAgw9fN1jHl6Jlf+ZQZPX3UszTK1saWI1B/xDv99BCgH+gM7gJOAi4CFwFl1EpnsdWrfTjx68SBmrdjMmKdnsrNUExZFpP6IN5F8C7jZ3T8nqIlscPfXgJuBu+sqOPnGOQNzefDCI5lWuFGrBYtIvRJvIskmGAoMwcitjuHzBcDA2g5KYrtgSFfuHn0EUxau57oXtVWviNQP8Ta2fw70BZYBc4CrzWwlcA2gKdcJ9INhPSjZXca9Ez8nOyOdBy4YSFqaJTssEUlh8SaSR4HO4fO7gLeB7wK7CJZPkQQac/Kh7NhdziNTltAsM507zzscMyUTEUmOeJeRf9bdnwqfzwJ6AscA3d395Xg/zMzOMrNFZlZoZrfEOG9m9lh4fq6ZDY6nrJn9V3huvpk9EG88DdlPT+vNmJMP4en/LOf+tz/HXc1cIpIcBzWO1N13ALPMLNvMbnH3+6srY2bpwDjgDGAVMMPMJrh75NIrIwm29e0NDCXYNGtoVWXN7BRgFDDQ3XeZWUdSgJkxdmRfduwu4w//XkqLzCb812m9kx2WiKSgahOJmbUn+KNeCrzj7uVmlkHQPzKWYA5JtYmEYB+TQndfGr7vCwQJIDKRjAKe9uCf19PNLMfMcglqQJWV/TFwv7vvAnD39XHE0iiYGXeddwQ7dpfz8OTFfLGhmBnLNmv5eRFJqCoTiZkdD7wFtCYY9jvDzC4H/g5kEAz9HR/nZ+UBKyNeryJIUNVdk1dN2T7ASWb2a2AncIO7z4jxXcYAYwA6dOhAQUFBnGHXf+e0d2a3Ml6f882yZ6uLSrjp5TksWLiA47vENxu+uLi4Ud2X2qB7EpvuS2ypel+qq5HcDUwC7gGuBK4D3iTocH/GD6xhPlZvcHT5yq6pqmwToA3Bro3HAC+Z2SHRsbn7kwR7z5Ofn+/Dhw+PP/IG4PaP3iHIo9/YvQfeWpHOL743PK73KCgooLHdl5rSPYlN9yW2VL0v1XW2Hwnc7e6fAbcR/PEe6+5PH2ASgaAW0S3idVcgeuXgyq6pquwq4DUPfAzsIdgSOKWs3aLl50UkOapLJG2BDbC3g30HMPsgP2sG0NvMeplZJnAxMCHqmgnApeHorWHAFndfW03Z14FTAcysD8GCkhtJMZUtM5/bOivBkYhIqoln+G/FzojtCGokraJ2Smwbzwe5exlwLUFT2ULgJXefb2ZXm9nV4WUTgaVAIfBH4CdVlQ3LjAcOMbPPgBeAyw6ittTg3Tgin+yM/dfOTE8z1m+NXVsREakN8Qz/jRxVZQS1g8jXTpyr/7r7RIJkEXnsiYjnTjAaLK6y4fHdwCXxfH5jFr38fJecbEYe0ZnnPl7B6HEfMP6KY+jbuVWSoxSRxiieHRKlgYhefr7i2FV/ncGFj/+H//neIE7JT4lpNiKSQHHtkCgN1xF5rXnjmhO58qkZXPXUDO4873B+cFzPZIclIo1IvKv/SgPWuXUWL199HKf27cjtb8znrn8s0MrBIlJrlEhSRPOmTfjDD47myhN6Mf6DL/nRM5+wfVdZssMSkUZAiSSFpKcZvzy3P3ePOpypn6/noif+w9otmmciIjWjzb9T0A+O60m3ts249rnZjB73AZcM68ELH69kdVEJedOnao0uETkgqpGkqOH5HXnlx8exq3QPD/9rMavDGfCri0oY+9o8Xp+t/cpEJD5x1UjMrLKFGZ1ggadC4EV3j17yROqxvp1bkZWRDiWl+xwvKS3nwUmLVCsRkbjE27TVATiJYB2rz8JjRxBMSJwJnA/cZWYnufuc2g5S6s66Sma9a40uEYlXvE1bHwD/BLq6+8nufjLBwokTgX8BPQiWm3+4TqKUOlPZGl05zeJbel5EJN5E8lPgrnDhRmDvIo6/Bn4WLlPyG+CoWo9Q6lSsNbrSDDbvKOW/n5/NlqhmLxGRaPE2bbUAcgkWTIzUOTwHsPUA3k/qicg1ulYXlZCXk831Z/RhzZYSfjdlCTOXb+a33zmSoYe0S3KkIlJfxfuH/+/An83sJoJFG51g69wHgNfCa44FFtd6hFLnKtboit6U58TeHbjuhdlc/Mfp/GT4oVx3eh8y0jXQT0T2Fe9fhasJlnD/G/AFwVLvfwPeJlzqnaC28sPaDlCS56huObz13yfxnSHdGPfuF1zw+Ics3VCc7LBEpJ6JK5G4+w53v5pgo6tBwGCgrbv/2N23h9fM0Yitxqd50yb85sKBPP79wSz/egfnPDaN5z9eQQpu+SIilTigPo0wacyto1ikHhs5IJdB3dtw/ctzGPvaPN79fD0n9+nA4wVf7N3/RDPiRVJTvBMSswhGbp0GdCSqJuPuA2s/NKlvOrfO4pkrh/LnaV9y/z8X8q8F6/aeq5gRDyiZiKSYePtIfg/cAiwj2CP91aiHpIi0NOOHJx9C2xZN9ztXMSNeRFJLvE1bo4GL3H1KHcYiDcjGbbtiHteMeJHUE2+NZAewsi4DkYalshnxTdKNWSs2JzgaEUmmeBPJA8DPzUyTCASIPSM+I93Izkjj/N9/yE2vfMrXxbFrLSLSuMTbtHUGwaKNZ5nZAmCfdTPc/bzaDkzqt8gZ8ZGjtk7v34nH3lnC+Glf8vZnX3HjiHy+N7QH6WmW5IhFpK7Em0g2EsxuF9mrYkZ8tF+c3Y+LhnTlVxPmc/sb83lhxkruGnUEQ3q0SUKUIlLX4kok7n5FXQcijUvvTi159v8N5a15a7nnzYVc8PiHXDSkKzeP7Mu0JRv3q8loyLBIw6VFFqXOmBnfHtiFU/I78tjUJfz5/S/5x6erKXcoLQ9mxmv+iUjDV2nnuZnNNbM24fN54euYj8SFKw1R86ZNGDuyH29fdzKO7U0iFTT/RKRhq6pG8ipQMezmlQTEIo3cYR1bsLtsT8xzmn8i0nBVmkjc/c5Yz0VqoktONqtjJI3MJmnMXL5ZHfIiDZDmhUhCVTb/JCPduODxD7n8Lx8zd1VRcoITkYMSVyIxs7Zm9riZLTazIjPbGvmo6yCl8Rg9KI/7zh9AXk42BuTlZPPghUfy8a2nc/NZfZmzsojz/vcDfvj0Jyxcq18tkYYg3lFbfybYh+RJYA3BDokiB6Wy+Sc/Hn4olwzrzl8+WMYf31/KyEff55wBuVx3em96d2rJ67NXa9iwSD0UbyI5DTjD3T+qy2BEWmZl8N+n9eay43ryp2lLGT/tSyZ+tpYh3XOYt3oru8LOeg0bFqk/4u0jWQ9oj1VJmNbNMrj+zHzev/lUxpx8CJ8sL9qbRCpo2LBI/RBvIrkVuMvMWtRlMCLR2jbPZOzIflS2UpeGDYskX7yJ5DbgTGC9mS3UhERJtMqWrU9PM17+ZCW7ysoTHJGIVIg3kbwCPAT8BniBg9wh0czOMrNFZlZoZrfEOG9m9lh4fq6ZDT6AsjeYmZtZ+3jjkYajsmHD7VtkcuMrcznh/nd5dMoSLV0vkgTVdrabWQbQHBjn7ssP9oPMLB0YR7Ak/SpghplNcPcFEZeNBHqHj6HA48DQ6sqaWbfw3IqDjU/qt8qWrR91VBc+KPyaP09byu+mLGZcQSHnD8rjyhN70adTSwCN9hKpY9UmEncvNbMfE+zbXhPHAoXuvhTAzF4ARgGRiWQU8LS7OzDdzHLMLBfoWU3Z3wE3AW/UMEapxyobNnxi7/ac2Ls9heu3Mf6DZbw6cxUvzFjJyX060C+3JU9/uIySUo32Eqkr8Q7//RdwKjC+Bp+Vx77b9a4iqHVUd01eVWXN7Dxgtbt/alb55klmNgYYA9ChQwcKCgoO6ks0ZsXFxQ3+vpzZBo47OYt3V5byzvKNvLd4w37XlJSWc/cbn5KzZUm179cY7kld0H2JLVXvS7yJ5B3gXjMbCMwEtkeedPfX4niPWH/loyc2VnZNzONm1oxgRNmZ1X24uz9JMKGS/Px8Hz58eHVFUk5BQQGN5b6cC+wqKyf/trdjnt+00+P6ro3pntQm3ZfYUvW+xJtI/jf8+d8xzjmQHuN4tFVAt4jXXQlmycdzTWYlxw8FegEVtZGuwCwzO9bdv4ojJmnEmjZJJ6+SRSLT04w/vb+U8wd3pW3zzCREJ9J4xDVqy93TqnjEk0QAZgC9zayXmWUCFwMToq6ZAFwajt4aBmxx97WVlXX3ee7e0d17untPgkQ0WElEKlQ22is3J4t73lrI0Hun8JNnZ1KwaD3le7Tyj8jBSNgOie5eZmbXApMIajDj3X2+mV0dnn8CmAicDRQCO4ArqiqbqNil4apstNfoQXks+mobL85Yyd9nr2LivK/o0jqLC4d05aKjuzFz+WYenLSI1UUl5E2fqpFeIlWwYIBUHBeatQXOAroTNDXt5e531X5odSc/P98XLdLSGtFStX13V1k5Uxas58VPVvL+kg24Q5pBZAUlOyOd+84foGQSStXfleo09vtiZjPd/ejo43HVSMJmprcIdkzsAKwGcsPXy4AGlUhEIjVtks45A3M5Z2Auq4tKOOuR99i2s2yfa0pKy3lg0udKJCIxxDuz/UHgWYJhuDsJhgJ3Bz4hmO0u0ijk5WRTHJVEKqwp2sltr89j+tKv2aP+FJG94u0jGQhc5e5uZuVAU3dfamY3A88RJBmRRqGy7YCzM9J4ZeYq/jZ9BZ1aNeXsAbmce2QXBnXLoao5TCKNXbyJZHfE83VAD2AhwdLyXWo7KJFkunFEPmNfm0dJ6TcLQVb0kZzRvxPvfL6eNz9dw7MfreAvHywjLyebbw/M5dsDu1C4fhsP/WuxlmORlBJvIpkFHAMsBgqAe8ysE3AJoNV/pVGJHOm1uqiEvKiEcN6RXTjvyC5s21nK5AXr+Mena/jztC/5w3tLMb6ZZavlWCRVxJtIbgVahs9vA54G/ocgsVxRB3GJJFXFul5VjcJpmZXB+YO7cv7grhTt2M3wBwsoKind55qS0nLueWsB5wzMJSM93i5JkYYlrkTi7p9EPN9AsEqviIRymmWyJSqJVNhYvJvBd03mW/kdOL1fJ4bndyCnmWbTS+NxQBMSzexogmVJ3nT37WbWHNjl7rGHuYikkMo66ds2z+TM/p2YsnA9b85dS3qacXSPNpzRvxOn9etEr/bNtdS9NGjxziPpRLB8yTEETcC9gaXAbwmGA/+0rgIUaSgq66T/5bf7M3pQHnv2OHNXb2HKgnVMWbiOe95ayD1vLaRjy0w2bS+lLBxSrL4VaWjirZH8DvgKaMe+m0e9TNBXIpLyqlqOBSAtzTiqWw5HdcvhhhH5rNy0g6mfr+feiQv3JpEKJaXl3PfPhUok0iDEm0hOA05z981R4+W/IJiYKCJUvvlWLN3aNuOy43tyx4TYy8at27qLUx4q4MTDgo27jju0Ha2yMmozXJFaEW8iyWbfuSQVOhA0bYnIQaqsb6V1dhN6tW/Oq7NW8cz05aSHNZoTD2vPSb3bc2S3HN6au1Z9K5J08SaS94DLgV+Erz3cR/1mgk2vROQgVda3cud5RzB6UB67y/Ywa8Vmpi3ZyPuFG3ls6hIefWcJTdON0j2+d3FJ9a1IssSbSG4C/m1mxwBNgYeBw4HWwAl1FJtISqiubyWzSRrDDmnHsEPaccOIfIp27ObDL77mhpc/ZVd5+T7vVVJazq8mzKdvbkv6dGxJWpqWbpG6F+88kgVmNgD4McGKv1kEHe3jwo2nRKQGDqRvJadZJmcPyOWaZ2fFPL+lpJSzHnmfnGYZHNOzLUN7teXYXm3pn9uKJuGkSA03ltoU9zyScNfBX0UeM7MeZvaSu3+n1iMTkSpV1rfSqVVTbhzRl4+//JqPvtzE5AXrAGjRtAlDerShRdN0Ji9cz+6yPYCaxKTmarpDYg5wQS3EISIHqLK+lbEj+zF6UB4XDukKwFdbdvLxsk1BYlm6iSXri/d7r5LScu5/W/utyMFJ2Fa7IlK7qutbqdC5ddbehSYBet3yFrF2U/lqy05OemAqg7u3YVC3HAZ1b0O/3FZkNvlmjbCKJjFtQSyRlEhEGrAD6VupUFmTWKusJhzRpTXTl37NG3PWANC0SRoD8lozqHsOZeV7eP7jlexUk5hEUSIRSTGVNYndNeqIvQlhTVEJs1cUMXvFZmavLOKv/1m+t08lUklpOb9Rk1jKqzKRmNmEasq3qsVYRCQB4mkS65KTTZecbM4ZmAvA7rI95N/2z5hNYmu37OTE30xlQF5rjshrzYDw0ab5Nysca5RY41ZdjeTrOM5/WUuxiEiCHGiTWGaTtCqbxI7slsNnq7fwz8++2ns8LyebgV1bk2YwecF6dperSayxqjKRuLs2rRIRIL4msS0lpcxfvYV5EY/lX+/Y771KSsu58x/zGdi1NT3aNSe9komTqsk0DOojEZG4VLcFMUDr7AyOP6w9xx/Wfu+xykaJbd5RyqkP/5vsjHT6dG5J/9yW9O3cin65rcjv3JJ3P1+/T+JSTab+UiIRkbjFswVxtMqaxDq0bMpNI/JZuHYbC9du5Z+ffcXzH6/cez7djHLff3n9ByctUiKpZ5RIRKROVdYkduvZ/fZJCO7Ouq27WPjVVhau3coDby+K+X6ri0q45tlZHNaxBX06taRPpxb0bN+cjPT957uoSSwxlEhEpE7FO3HSzOjcOovOrbM4Jb8jz05fEbMmk9UkjflrtjDxs7VUVFiapBm92jenT6eWlJXvYeqi9ZSWa8fJRFEiEZE6dzATJyurydx3/gBGD8pjZ2k5heuLKVxfzOJ121i8rpj5a7awrJLO/dte/4xdZeUc0qEFh7RvTtvmmURt1KeazEFSIhGReqm6mkxWRjpHhHNXIlXWuV+8q4ybX52393Xr7AwO7dA8SCwdmrNx2y6e/WgFuzRz/4ApkYhIvVWbS8B0ycnihR8exxcbi1m6YTtfbChm6YZi3lu8gVdmror5XsH+Lp/RKrsJPds1p2ubZvusPVYh1dcgUyIRkUalsiaxm0b0pXu7ZnRv14xT8vcts21nKQPv+FfMmsyWkjKufOoTANIM8tpk07Ndc3q2a06Pds34astOnpm+PKVrMkokItKoxNu5H6llVkalNZnOrbMY973BLNu4neVfb2fZ1ztY/vV23pizmq07y2K+X0lpObe/8RmO071tM7q1aUaHlk0bbZ+MEomINDq12bl/y1l9GdKjDUN6tNmvzObtuxl89+SYNZltO8v42Yuf7n2dlZFG1zbNwsSSzZaSUibOW8vugxhdVt8SkBKJiAgHV5Np0zyz8j6Z1lk8fdVQVm7ewcpNwWPFph2s3FTCjC83sW3X/rWZktJybnl1LnNWFtG1TTZ5OdnkhT8rRpm9Pnt1vZvxn9BEYmZnAY8C6cCf3P3+qPMWnj8b2AFc7u6zqiprZg8C5wK7gS+AK9y9KCFfSEQaldqsydx0Vl8O69iCwzq22K+Mu3PI2IkxazI7y/bwysxVFEclmuyMdLrkZLFqc8ne/pgKJaXlPBDHcv51VZNJWCIxs3RgHHAGsAqYYWYT3H1BxGUjgd7hYyjwODC0mrKTgbHuXmZmvwHGAjcn6nuJSGqLZw2yaGZWaU0mLyebaTefwtaSMlZu3sHqohJWby7Z+/OLDdtjvueaLTs5+p4pdMnJIrd1Frmts8Pnwc+5q7bwm7c/Z2fpgQ8KqEhAmZ0PGxLrfCJrJMcChe6+FMDMXgBGAZGJZBTwtLs7MN3McswsF+hZWVl3/1dE+enAhXX+TUREIhzMGmSV1WRuHJGPmdG6WQatm+0/T+aE+6fGTEAts5pwWt+OrNkSJJtpSzayfXf5ftdFKikt55dvfAZAp1bBqgKdW2WRnZm+95roprRYEplI8oCVEa9XEdQ6qrsmL86yAFcCL8b6cDMbA4wB6NChAwUFBQcQemooLi7WfYmiexKb7ktsB3JfcoAf9Evn1cV7+Hqn0y7LuKBPOjlbllBQsKTScud0L+eprbA7onUrMw2+2yeN49tvgnDhZfem7CiDTTudTTv38LuZu2K+39adZVz34px9jjVrAm2yjDZN01hSVM6uqvNRQhNJrA0HopsIK7um2rJmditQBjwb68Pd/UngSYD8/HyP918NqeRA/jWVKnRPYtN9ie1A78tw4BcH+BnDgf4H0dfx0hexazK5rbN45qqhrNu6k6+27OSrrTv3Pl+3dSe7vt5SbUyJTCSrgG4Rr7sCa+K8JrOqsmZ2GfBt4LSwWUxEpNGqzUEBN1cxKAAqb0qLtP9c/7ozA+htZr3MLBO4GIjeE34CcKkFhgFb3H1tVWXD0Vw3A+e5+/6rtYmICKMH5XHf+QPIy8nGCDr1KxbArMqNI/LJzkiv8pqE1UjCUVXXApMIhvCOd/f5ZnZ1eP4JYCLB0N9CguG/V1RVNnzr/wWaApPDWaPT3f3qRH0vEZGG4mBqMpGj0tZWck1C55G4+0SCZBF57ImI5w5cE2/Z8PhhtRymiIhEqEhANrZwZqzziWzaEhGRRkiJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREakSJREREaiShicTMzjKzRWZWaGa3xDhvZvZYeH6umQ2urqyZtTWzyWa2JPzZJlHfR0REEphIzCwdGAeMBPoD3zWz/lGXjQR6h48xwONxlL0FeMfdewPvhK9FRCRBElkjORYodPel7r4beAEYFXXNKOBpD0wHcswst5qyo4C/hs//Coyu4+8hIiIRmiTws/KAlRGvVwFD47gmr5qyndx9LYC7rzWzjrE+3MzGENRyAHaZ2WcH8yUaufbAxmQHUc/onsSm+xJbY78vPWIdTGQisRjHPM5r4ilbJXd/EngSwMw+cfejD6R8KtB92Z/uSWy6L7Gl6n1JZNPWKqBbxOuuwJo4r6mq7Lqw+Yvw5/pajFlERKqRyEQyA+htZr3MLBO4GJgQdc0E4NJw9NYwYEvYbFVV2QnAZeHzy4A36vqLiIjINxLWtOXuZWZ2LTAJSAfGu/t8M7s6PP8EMBE4GygEdgBXVFU2fOv7gZfM7CpgBXBRHOE8WXvfrFHRfdmf7klsui+xpeR9MfcD6moQERHZh2a2i4hIjSiRiIhIjaRUIqluiZZUZWbLzGyemc0xs0+SHU+ymNl4M1sfOcdIS/BUel/uMLPV4e/MHDM7O5kxJpqZdTOzd81soZnNN7OfhsdT8vclZRJJnEu0pLJT3P2oVBwDH+Ep4KyoY1qCJ/Z9Afhd+DtzlLtPTHBMyVYGXO/u/YBhwDXh35OU/H1JmURCfEu0SApz9/eATVGHU34JnkruS0pz97XuPit8vg1YSLACR0r+vqRSIqls+RUJVgn4l5nNDJeSkW/sswQPEHMJnhR1bbhK9/hUacKJxcx6AoOAj0jR35dUSiQ1XmalETvB3QcTNPtdY2YnJzsgqfceBw4FjgLWAg8nNZokMbMWwKvAde6+NdnxJEsqJZJ4lmhJSe6+Jvy5Hvg7QTOgBLQETwzuvs7dy919D/BHUvB3xswyCJLIs+7+Wng4JX9fUimRxLNES8oxs+Zm1rLiOXAmoJWRv6EleGKo+GMZ+j+k2O+MmRnwZ2Chu/824lRK/r6k1Mz2cIjiI3yzzMqvkxtR8pnZIQS1EAiWzHkuVe+LmT0PDCdYCnwd8CvgdeAloDvhEjzunlIdz5Xcl+EEzVoOLAN+VNE3kArM7ETgfWAesCc8/AuCfpKU+31JqUQiIiK1L5WatkREpA4okYiISI0okYiISI0okYiISI0okYiISI0okYg0cGbmZnZhsuOQ1KVEIlIDZvZU+Ic8+jE92bGJJErC9mwXacSmAD+IOrY7GYGIJINqJCI1t8vdv4p6bIK9zU7XmtlbZrbDzJab2SWRhc1sgJlNMbMSM9sU1nJaR11zWbj52C4zW2dmT0XF0NbMXjaz7Wa2NPozROqSEolI3buTYA2mo4AngafN7GgAM2sGvA0UEyx8+H+A44HxFYXN7EfAH4C/AAOBs4H5UZ/xS4J1nY4EXgTGm1mPOvtGIhG0RIpIDYQ1g0uAnVGnxrn7zWbmwJ/c/YcRZaYAX7n7JWb2Q+AhoGu4QRJmNhx4F+jt7oVmtgr4m7vH3G0v/Iz73X1s+LoJsBUY4+5/q71vKxKb+khEau49IHpDsKKI5/+JOvcf4JzweT9gbkUSCX1IsBBgfzPbSrAB2zvVxDC34om7l5nZBlJkUyVJPiUSkZrb4e6FB1nWqHyDNSf2hmyxlMYoq6ZrSQj9oonUvWExXi8Mny8AjqzYEyZ0PMH/mwvdfR2wGjitzqMUOUiqkYjUXFMz6xx1rNzdN4TPzzezGUABcCFBUhgannuWoDP+aTP7JdCGoGP9tYhazq+B35nZOuAtoBlwmrun5Pa2Uv8okYjU3OkE+5ZHWk2wnTPAHcAFwGPABuAKd58B4O47zGwEwYZrHxN02r8B/LTijdz9cTPbDVwP/AbYBEyso+8icsA0akukDoUjqi5y91eSHYtIXVEfiYiI1IgSiYiI1IiatkREpEZUIxERkRpRIhERkRpRIhERkRpRIhERkRpRIhERkRr5/x62Q+98ej++AAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# extra code – this cell plots power scheduling\n",
"\n",
"import math\n",
"\n",
"learning_rate = 0.01\n",
"decay = 1e-4\n",
"batch_size = 32\n",
"n_steps_per_epoch = math.ceil(len(X_train) / batch_size)\n",
"n_epochs = 25\n",
"\n",
"epochs = np.arange(n_epochs)\n",
"lrs = learning_rate / (1 + decay * epochs * n_steps_per_epoch)\n",
"\n",
"plt.plot(epochs, lrs, \"o-\")\n",
"plt.axis([0, n_epochs - 1, 0, 0.01])\n",
"plt.xlabel(\"Epoch\")\n",
"plt.ylabel(\"Learning Rate\")\n",
"plt.title(\"Power Scheduling\", fontsize=14)\n",
"plt.grid(True)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exponential Scheduling"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```lr = lr0 * 0.1 ** (epoch / s)```"
]
},
{
"cell_type": "code",
"execution_count": 69,
"metadata": {},
"outputs": [],
"source": [
"def exponential_decay_fn(epoch):\n",
" return 0.01 * 0.1 ** (epoch / 20)"
]
},
{
"cell_type": "code",
"execution_count": 70,
"metadata": {},
"outputs": [],
"source": [
"def exponential_decay(lr0, s):\n",
" def exponential_decay_fn(epoch):\n",
" return lr0 * 0.1 ** (epoch / s)\n",
" return exponential_decay_fn\n",
"\n",
"exponential_decay_fn = exponential_decay(lr0=0.01, s=20)"
]
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {},
"outputs": [],
"source": [
"# extra code – build and compile a model for Fashion MNIST\n",
"\n",
"tf.random.set_seed(42)\n",
"model = build_model()\n",
"optimizer = tf.keras.optimizers.SGD(learning_rate=0.001)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
" metrics=[\"accuracy\"])"
]
},
{
"cell_type": "code",
"execution_count": 72,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/25\n",
"1719/1719 [==============================] - 2s 913us/step - loss: 0.6795 - accuracy: 0.7682 - val_loss: 0.4782 - val_accuracy: 0.8314\n",
"Epoch 2/25\n",
"1719/1719 [==============================] - 1s 869us/step - loss: 0.4656 - accuracy: 0.8376 - val_loss: 0.4407 - val_accuracy: 0.8404\n",
"Epoch 3/25\n",
"1719/1719 [==============================] - 2s 873us/step - loss: 0.4194 - accuracy: 0.8505 - val_loss: 0.4118 - val_accuracy: 0.8520\n",
"Epoch 4/25\n",
"1719/1719 [==============================] - 1s 857us/step - loss: 0.3953 - accuracy: 0.8601 - val_loss: 0.3850 - val_accuracy: 0.8616\n",
"Epoch 5/25\n",
"1719/1719 [==============================] - 1s 868us/step - loss: 0.3754 - accuracy: 0.8667 - val_loss: 0.3769 - val_accuracy: 0.8620\n",
"Epoch 6/25\n",
"1719/1719 [==============================] - 2s 878us/step - loss: 0.3611 - accuracy: 0.8717 - val_loss: 0.3782 - val_accuracy: 0.8630\n",
"Epoch 7/25\n",
"1719/1719 [==============================] - 2s 895us/step - loss: 0.3501 - accuracy: 0.8743 - val_loss: 0.3665 - val_accuracy: 0.8690\n",
"Epoch 8/25\n",
"1719/1719 [==============================] - 2s 883us/step - loss: 0.3407 - accuracy: 0.8785 - val_loss: 0.3694 - val_accuracy: 0.8638\n",
"Epoch 9/25\n",
"1719/1719 [==============================] - 2s 879us/step - loss: 0.3328 - accuracy: 0.8814 - val_loss: 0.3477 - val_accuracy: 0.8708\n",
"Epoch 10/25\n",
"1719/1719 [==============================] - 2s 895us/step - loss: 0.3259 - accuracy: 0.8834 - val_loss: 0.3495 - val_accuracy: 0.8728\n",
"Epoch 11/25\n",
"1719/1719 [==============================] - 2s 884us/step - loss: 0.3200 - accuracy: 0.8855 - val_loss: 0.3483 - val_accuracy: 0.8722\n",
"Epoch 12/25\n",
"1719/1719 [==============================] - 2s 903us/step - loss: 0.3148 - accuracy: 0.8877 - val_loss: 0.3459 - val_accuracy: 0.8772\n",
"Epoch 13/25\n",
"1719/1719 [==============================] - 2s 894us/step - loss: 0.3107 - accuracy: 0.8892 - val_loss: 0.3366 - val_accuracy: 0.8766\n",
"Epoch 14/25\n",
"1719/1719 [==============================] - 2s 885us/step - loss: 0.3068 - accuracy: 0.8904 - val_loss: 0.3409 - val_accuracy: 0.8772\n",
"Epoch 15/25\n",
"1719/1719 [==============================] - 1s 853us/step - loss: 0.3034 - accuracy: 0.8921 - val_loss: 0.3404 - val_accuracy: 0.8766\n",
"Epoch 16/25\n",
"1719/1719 [==============================] - 2s 884us/step - loss: 0.3000 - accuracy: 0.8934 - val_loss: 0.3332 - val_accuracy: 0.8774\n",
"Epoch 17/25\n",
"1719/1719 [==============================] - 2s 887us/step - loss: 0.2978 - accuracy: 0.8933 - val_loss: 0.3342 - val_accuracy: 0.8788\n",
"Epoch 18/25\n",
"1719/1719 [==============================] - 2s 890us/step - loss: 0.2953 - accuracy: 0.8945 - val_loss: 0.3323 - val_accuracy: 0.8770\n",
"Epoch 19/25\n",
"1719/1719 [==============================] - 2s 918us/step - loss: 0.2934 - accuracy: 0.8951 - val_loss: 0.3291 - val_accuracy: 0.8774\n",
"Epoch 20/25\n",
"1719/1719 [==============================] - 2s 923us/step - loss: 0.2915 - accuracy: 0.8966 - val_loss: 0.3292 - val_accuracy: 0.8776\n",
"Epoch 21/25\n",
"1719/1719 [==============================] - 2s 888us/step - loss: 0.2899 - accuracy: 0.8968 - val_loss: 0.3273 - val_accuracy: 0.8766\n",
"Epoch 22/25\n",
"1719/1719 [==============================] - 2s 874us/step - loss: 0.2882 - accuracy: 0.8975 - val_loss: 0.3298 - val_accuracy: 0.8790\n",
"Epoch 23/25\n",
"1719/1719 [==============================] - 2s 879us/step - loss: 0.2870 - accuracy: 0.8978 - val_loss: 0.3287 - val_accuracy: 0.8780\n",
"Epoch 24/25\n",
"1719/1719 [==============================] - 2s 890us/step - loss: 0.2857 - accuracy: 0.8986 - val_loss: 0.3288 - val_accuracy: 0.8786\n",
"Epoch 25/25\n",
"1719/1719 [==============================] - 2s 877us/step - loss: 0.2849 - accuracy: 0.8985 - val_loss: 0.3285 - val_accuracy: 0.8792\n"
]
}
],
"source": [
"lr_scheduler = tf.keras.callbacks.LearningRateScheduler(exponential_decay_fn)\n",
"history = model.fit(X_train, y_train, epochs=n_epochs,\n",
" validation_data=(X_valid, y_valid),\n",
" callbacks=[lr_scheduler])"
]
},
{
"cell_type": "code",
"execution_count": 73,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEbCAYAAADwPQLqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAyaUlEQVR4nO3deXxU1fnH8c+TQCBhC0vYAoosIogKbriLWhXUirVo9dfWrdVatdraulerVVuVulRrRWytWrUWN6SKIkqDVkVBkEV2EDRh3w0J+/P7497gOMwkA8nMJJnv+/Wa18zce+6d5x6GeXLPufccc3dERET2VFa6AxARkbpNiURERKpFiURERKpFiURERKpFiURERKpFiURERKpFiUSkhpnZRWZWupvbFJnZX5IVU/gZi8zsN0nY7xAz2637CKLraE/qTGoPJRKpMWb2lJl5jMeEdMeWLOHxDYla/G+gaxI+66dmNsXMSs1svZlNM7O7avpz0iQpdSap0SDdAUi98w7w46hlW9IRSLq4ezlQXpP7NLNLgIeBXwHvAjnA/sCRNfk56ZKMOpPU0RmJ1LTN7r4s6rEGwMyON7OtZjagorCZXW5mG8ysa/i+yMyGmdmfzWxt+BhqZlkR27Q0s6fDdeVm9o6Z7R+x/qLwr/aTzGyGmW00s/+a2T6RgZrZd83sUzPbZGZfmNndZpYTsX6Rmf3WzB4PYyw2s+si14cvXwzPTBZFfn5EuW5m9pqZLQtjmWxmZ+xmvZ4JvOLuj7v7fHef6e4vuvu1Ucd0upl9HNbLajP7j5k1jijSON7xhNu3MLPhZrbCzL42s/FmdmhUmQvMbLGZlZnZ60C7qPW3m9mMqGWVNl3FqLPbw3+788xsQRjLSDNrE1GmgZk9GPE9edDMHjOzoqqrU2qSEomkjLuPB4YC/zSzVma2H3A/8At3XxhR9IcE380jgZ8BlwG/jFj/FNAfGAwcDpQBb5lZbkSZRsBNwCXhfvKBYRUrzexU4DngLwR/2V8CDAH+EBX2r4DpwMHAvcB9ZlZxFnBY+Hwp0CHifbSmwJvAycBBwMvAK+HxJ2oZcHhFwo3FzAYCrwFjgUOAE4DxfPv/edzjMTMD3gAKgTOAfsB7wDgz6xCW6U9Q/8OBvsB/gN/vxnHsji7AD4DvAaeE8dwdsf43wEXAT4EjCI7z/5IUi1TG3fXQo0YeBD8w24DSqMe9EWUaAhOBV4DJwL+j9lEEzAUsYtlvgeLwdQ/AgeMi1rcA1gM/Dd9fFJbpGVHmhwRNbFnh+/eAW6M++6wwXgvfLwL+FVVmHvDbiPcODIkqcxFQWkVdTYjaTxHwl0rKdwA+Cj9vHvAscAHQMKLMB8ALleyj0uMBTgyPPzeqzGfA9eHr54GxUev/FvyU7Hx/OzCjsjpJ4P3twCagRcSyW4D5Ee+XAjdGvDdgNlCU7v8LmfbQGYnUtPcI/lKNfAytWOnuWwn+ajwDaEtwxhFtgoe/DKGPgEIzaw70AnaEyyr2uZ7gr+zeEdtsdvc5Ee+XECSx/PD9IcAtYRNYadis8jzQBGgfsd20qNiWhHEnzMyamNl9ZjYzbIIpBQ4F9kp0H+6+1N2PBA4AHiL40Xwc+MTM8sJi/Qj6TypT2fEcAuQBK6PqpQ/QLSzTi4i6D0W/rymLw3/bXWI1sxYE/06fVKwMvzMTkxSLVEKd7VLTytx9fhVlKpoh8oECYN1u7N8qWReZfLbFWZcV8XwH8GKM/ayMeL01xn529w+wPwEDCZpi5hE0xT1D0GG+W9x9BjADeNTMjgHeB84lOBtMRGXHkwUsB46Nsd2G8Lmy+q+wI0a5hgnGFymRutfw5bWAzkgkpcysC0G/xJUEbfnPmVn0HzT9w/b6CkcAS9x9AzCTb/pPKvbZnOAv9Zm7EcpkYD8POq6jH9FJqDJbgewqyhwDPOPuL7v7NKCYb/7Cr46K420aPk8BTqrG/iYTdJzviFEnKyI+84io7aLfrwTaRf0b9q1GXLsIz1SWEfSRATv7eOL1U0kS6YxEalojM2sftWy7u680s2yCtv3x7v64mb1E0CT1O+DWiPIdgYfM7K8ECeI64C4Ad59nZq8Bj5vZZQRnM3cT/MX8/G7E+XvgdTNbDIwgOIPpAxzu7tfvxn4WASeZ2XiC5rS1McrMBb4Xxr2V4HgbxygXl5k9RtC0M44gEXUg6DsqA94Oi90N/MfM5hPUhRF0Uj/u7mUJfMw7BP0sr5nZ9QT9De0Jzqbecff3CS5B/tDMbgJeAgYQdIZHKgJaATeb2Qthmeh7bWrCn4HrzWwuQYL7GUG9LE3CZ0kldEYiNe07BP+RIx9TwnU3A92BnwC4+2rgQuDGsJmmwnMEf+V/DDwB/B14MGL9xQRt46PC5zxgoAf3IiTE3ccApxNc2fRJ+LgR+DLxQwXg1+E+vuKb44x2LbCCoBnqTYKO9vd383PGElypNoIgMb0aLj/Z3ecCuPtogh/1QWEs48PYdiTyAWEfw2kEyeoJYE74eT0JkhjuPoHg3+/nBP0tZxN0jEfuZ1a4/rKwzMnsejVcTfgT8E/gHwR1CkG9bErCZ0klKq5OEakVwnsAZrj7VemOReoeM5sMfODuv0h3LJlETVsiUieZ2d7AqQRnXg0IzoAOCp8lhZRIRKSu2kFwL81Qgmb6mcAgd5+U1qgykJq2RESkWtTZLiIi1ZKRTVv5+fnevXv3dIdR62zcuJEmTZqkO4xaRXUSm+oltvpeL59++ukqdy+IXp6RiaRdu3ZMmqRm1GhFRUUMGDAg3WHUKqqT2FQvsdX3egnvu9qFmrZERKRalEhERKRalEhERKRalEhERKRalEhERKRalEhERKRalEhERKRalEhERKRalEhERKRalEhERKRalEhERKRalEhERKRalEhERKRalEhERKRalEhERKRaUppIzGygmc0xs/lmdmOM9WZmD4frp5nZwRHrnjSzFWY2I2qbVmY21szmhc8tq4pj0YYdHH3POEZOKamZAxMRyWApSyRmlg08CgwCegPnm1nvqGKDgB7h4zLgsYh1TwEDY+z6RuBdd+8BvBu+r1LJunJuemW6komISDWl8ozkcGC+uy909y3AC8DgqDKDgWc8MAHIN7MOAO7+HrAmxn4HA0+Hr58Gzko0oPKt2xk6Zs7uHYWIiHxLKqfaLQS+inhfDPRPoEwhsLSS/bZz96UA7r7UzNrGKmRmlxGc5ZDT/pv52kvWlVNUVJTYEdRzpaWlqosoqpPYVC+xZWq9pDKRWIxlvgdl9oi7DweGAzTq0GPnPgvzc+v1HMu7o77PN70nVCexqV5iy9R6SWXTVjHQOeJ9J2DJHpSJtryi+St8XpFoQA2zjetO7ZlocRERiSGViWQi0MPM9jGzHOA8YFRUmVHABeHVW0cA6yuarSoxCrgwfH0h8FoiweRkZ5FlcGS31okfgYiI7CJlicTdtwFXAWOAWcAId//czC43s8vDYqOBhcB84AngiortzexfwEdATzMrNrOfhKvuAU42s3nAyeH7SnVpnsVbvzwWMG4dOQP3Gmk9ExHJSKnsI8HdRxMki8hlwyJeO3BlnG3Pj7N8NXDS7sbStaApvzp5X+55czZvzljGaQd02N1diIgIGX5n+0+P2Yc+hc257bUZrN24Jd3hiIjUSRmdSBpkZ3Hf9w9iXdlW7nxjZrrDERGpkzI6kQD07ticnw/oxiuTSyiak/AFXyIiEsr4RAJw1Ynd6VbQhFtenUHp5m3pDkdEpE5RIgEaNcjmviEHsmR9OUPfmp3ucERE6hQlktAhe7fiwiO78MyExUxcFGtILxERiUWJJMJ1p/akY4tcbnh5Gpu2bk93OCIidYISSYQmjRrwx7MPYOHKjTz87rx0hyMiUicokUQ5bt8ChhzSicffW8iMkvXpDkdEpNZTIonht6f3omVeDte/NI2t23ekOxwRkVpNiSSG/Lwc7hy8PzOXbuCJ9xemOxwRkVpNiSSOQQd0YFCf9jz0zjwWrCxNdzgiIrWWEkkl7hi8P40bZHHjy9PYsUMjBIuIxJLS0X/rmrbNGnPrGb257qVp9LtzLBvKt9IxP5frTu3JWf0K0x2eiEitoERShQZZRpbB+vKtQDDH+02vTAdQMhERQU1bVfrT23OJbtUq37qdoWPmpCcgEZFaRomkCkvWle/WchGRTKNEUoWO+bm7tVxEJNMokVThulN7ktsw+1vLDLjqxG7pCUhEpJZRIqnCWf0K+ePZB1CYn4sBbZrmAPDhgjUEU8yLiGQ2XbWVgLP6FX7rCq2/jJvHn96ey7E92nDuoZ3TGJmISPrpjGQP/HxAd47o2orfvfa57noXkYynRLIHsrOMh37Qj8YNs/jF81PYvE1zl4hI5lIi2UPtWzRm6JCDmLl0A/e8qel5RSRzKZFUw3d6t+Oio7rwjw8W8e6s5ekOR0QkLZRIqunGQfvRq0NzrntpGss3bEp3OCIiKadEUk2NG2bzyPn9KN+ynV++8BnbNUqwiGQYJZIa0L1tU+44c38+WriaYeMXpDscEZGUUiKpIecc2okzDuzAA2Pn8unitekOR0QkZZRIaoiZ8YezD6BDi8Zc/a8pO4edFxGp75RIalDzxg15+Px+LNuwiZtfna4hVEQkI6Q0kZjZQDObY2bzzezGGOvNzB4O108zs4Or2tbM+prZBDP7zMwmmdnhqTqeWA7eqyW/PmVf3pi2lBGTvkpnKCIiKZGysbbMLBt4FDgZKAYmmtkod58ZUWwQ0CN89AceA/pXse19wB3u/qaZnRa+H5Ciw4rp8uO68cH8Vdzy6nTuf3suK7/erCl6RaTeSuUZyeHAfHdf6O5bgBeAwVFlBgPPeGACkG9mHarY1oHm4esWwJJkH0hVsrKMU3q3Y9sOWPH1ZpxvpugdOaUk3eGJiNSoVI7+WwhEtvUUE5x1VFWmsIptfwmMMbM/ESTGo2J9uJldBlwGUFBQQFFR0Z4cQ8IeLirbZVn51u3c+dpU8tfPS+pn76nS0tKk10tdozqJTfUSW6bWSyoTicVYFt0bHa9MZdv+HPiVu79sZucCfwe+s0th9+HAcICePXv6gAEDEgx7z6x5643Yyzc5yf7sPVVUVFRrY0sX1UlsqpfYMrVeUtm0VQxETt7RiV2boeKVqWzbC4FXwtcvEjSDpZ2m6BWRTJFwIjGzQWb2upnNNLPO4bKfmtlJCe5iItDDzPYxsxzgPGBUVJlRwAXh1VtHAOvdfWkV2y4Bjg9fnwjUinajWFP0AlxyTJfUByMikkQJJRIz+yEwguBHeh+gYbgqG7g+kX24+zbgKmAMMAsY4e6fm9nlZnZ5WGw0sBCYDzwBXFHZtuE2lwL3m9lU4A+E/SDpFj1Fb9tmjWjUwHhxUjFlW7alOzwRkRqTaB/J9cCl7v6Cmf00YvkE4PeJfpi7jyZIFpHLhkW8duDKRLcNl/8POCTRGFIpeore8XNXcvE/PuE3L07l0f87GLNYXT8iInVLok1bPYCPYiwv5ZtLb6UKx+9bwE2DejF6+jIeGTc/3eGIiNSIRBPJEmDfGMuPAzTc7W746bH7cHa/Qh4YO5e3ZixLdzgiItWWaCIZDjxsZkeH7zub2YUEd5E/lpTI6qmKwR0P6pzPtSM+Y/ayDekOSUSkWhJKJO5+H8EltmOBJsB/gWHAMHd/NHnh1U+NG2Yz/MeH0LRRAy59ZhJrNm5Jd0giInss4ct/3f0WoA3BfRpHAAXufmuyAqvv2jVvzOM/PoTlGzZz5XOT2bp9R7pDEhHZI4le/vukmTVz9zJ3n+Tun7h7qZk1MbMnkx1kfdVvr5bcc/YBfLRwNXe9PrPqDUREaqFEz0guBGLdkp0LXFBz4WSesw/uxKXH7sPTHy3mhU++THc4IiK7rdL7SMysFcE4Vwa0NLPIO+mygdOB5ckLLzPcOKgXc5aXcutrM+jWtimHdWmV7pBERBJW1RnJKmAFwQCJM4GVEY9lwN+AvyYzwEyQnWU8cl4/OrXM4+fPfkrJuvJ0hyQikrCq7mw/geBsZBzwfWBNxLotwGJ3T/v8H/VBi7yGPHHBoXzv0Q84d9iHuMPS9Zs0IZaI1HqVJhJ3Hw9gZvsAX7m7Li1Kou5tm3L+4Z0Z/v4XO5dVTIgFKJmISK2U0Fhb7r4YwMw6AnsBOVHr36v50DLTG9N3vdu9fOt2ho6Zo0QiIrVSQokkTCDPEwyJUjHRVOSkVLuOly57ZEmc/pF4y0VE0i3Ry38fArYDvYEy4FjgHIIh3QcmJbIMpQmxRKSuSTSRHA/c4O6zCc5EVrr7K8ANwJ3JCi4TxZsQ68yDOqYhGhGRqiWaSHIJLgWG4MqttuHrmcCBNR1UJoueEKtDi8Z0aN6Ipz9axNSv1qU7PBGRXSQ6sdVsYD9gEfAZcLmZfUUwCVVJUiLLYNETYq3YsInvD/uQi/7xCS9efhTd2zZNY3QiIt+W6BnJn4H24evfA6cQTIl7BXBzEuKSCG2bN+afl/QnO8u44O8fs3S9Ot5FpPZIdBj559z9qfD1ZKALcBiwl7u/mLToZKcubZrw1MWH8/Wmbfz475+wVkPPi0gtkfAw8pHCUYAnAxvN7MYajkni6FPYguEXHMqXa8q4+KmJlG3ZVvVGIiJJVmUiMbM2Zna6mZ1iZtnhsoZm9kuCPpPfJDdEiXRkt9Y8cn4/phWv4/JnJ7NlmwYbEJH0qjSRmNlRwDzgP8CbwAdmth8wDbiK4NLfvZIdpHzbqfu3549nH8B7c1fymxensmOHV72RiEiSVHXV1p3AGOAu4BLgl8DrBB3u/3R3/YKlyQ8O24s1G7dy71uzaZnXkNvP3B8zS3dYIpKBqkokBwHHu/vnZvZb4BrgJnWw1w6XH9+V1aWb+dv/vqB100ZcfVKPdIckIhmoqkTSimDuEdy9zMzKgClJj0oSYmbcfFov1pRt4YGxc/lyTRkfLVjNknXlGn5eRFImkRsSK2ZGrBiosXk4c+JO7r4m5paSdFlZxr3fP5DZSzfw0qfFO5dr+HkRSZVELv+tmBlxBdAUmMg3sySuCp8ljRpmZ7GubOsuyyuGnxcRSaZEZkiUOmDp+k0xl2v4eRFJtoRmSJTar2N+bsy53jX8vIgk2x7d2S61T7zh50/q1TZGaRGRmpPSRGJmA81sjpnNjzW0igUeDtdPM7ODE9nWzH4RrvvczO5LxbHUNrGGn+/apgnPTljMK5OLq9xeRGRPJTqMfLWFw6s8CpwMFAMTzWyUu8+MKDYI6BE++gOPAf0r29bMTgAGAwe6+2Yzy9g/waOHny/bso2fPj2JX784lW3bnXMP65zG6ESkvkrlGcnhwHx3X+juW4AXCBJApMHAMx6YAOSbWYcqtv05cI+7bwZw9xWpOJi6IC+nAU9edBjH9ijg+pen8c8Ji9MdkojUQyk7IwEKga8i3hcTnHVUVaawim33BY41s7uBTcBv3H1i9Ieb2WXAZQAFBQUUFRXt8YHUNT/a29mwNptbR85g5uy5nNqlYcxypaWlGVUviVCdxKZ6iS1T6yWhRGJmT8ZZ5QQ/3vOBf7v7ksp2E2f7RMpUtm0DoCVwBMEcKSPMrGv0OGDuPhwYDtCzZ08fMGBAJaHWPwOO38E1L0zhXzOWsVeXrvx8QLddyhQVFZFp9VIV1UlsqpfYMrVeEj0jKQCOBXYAM8JlfQh+4D8FzgZ+b2bHuvtncfZRDEQ20ncCohNPvDI5lWxbDLwSJo5PzGwH0AbdKPktOQ2yeOT8fvxqxFTufWs2W7bt4OqTumugRxGptkT7SD4gGEa+k7sf5+7HEfyYjwbeBvYG3gDur2QfE4EeZraPmeUA5wGjosqMAi4Ir946Aljv7kur2HYkcCKAme1LkHRWJXhcGaVBdhYP/aAv3z+4Ew++M5c/vT0HDeAsItWV6BnJNcCJ7l5WsSAcxPFu4F13v8/M7gXeibcDd99mZlcRDEufDTwZjip8ebh+GEFiOo2gqawMuLiybcNdPwk8aWYzgC3AhRrePr7sLGPokAPJaWA8+t8FbNm2g5tP66UzExHZY4kmkqZAB2BW1PL24TqADVXtz91HEySLyGXDIl47cGWi24bLtwA/qjx8iZSVZdx91gE0zM7iife/YPbSDSxYtZEl6zZROGGcRg0Wkd2SaCJ5Ffi7mV1P0MzkBJfk3ge8EpY5HJhb4xFKUmRlGXecuT9frt5I0dxvWgI1arCI7K5E+0guJ2hWehZYACwMX78FXBGWmQVcWtMBSvKYGfNWlO6yXKMGi8juSOiMJOwbudzMfg10I7haa767b4wo81lSIpSkWrJOowaLSPXs1g2JYeKYlqRYJA3ijRrctnmjNEQjInVRQk1bZtbYzG4ws7fN7LNwQMWdj2QHKckTb9Tgr8u38uliTXwpIlVL9Izkr8D3gBeBD9n1jnSpoyo61IeOmUPJunIK83O56KguPP/Jl5z/xMc8cO5BnHFgxzRHKSK1WaKJ5CzgHHePe5+I1F0VowZHDu8w5JBOXPrMJK56fgrFa8v52XFdda+JiMSU6FVbZXx70ESp51o2yeHZn/bnuwd15J43Z3PLyBls274j3WGJSC2UaCK5D7jWzDSjYgZp3DCbP/+gL1cM6MbzH3/JT56eROnmbekOS0RqmUSbtk4mGLRxoJnNBLZGrnT3M2s6MKkdsrKM6wfuR+dWefx25AzOGfYR/7joMNq3aJzu0ESklkj0DGMVwd3t44BlwOqoh9Rz5x++F09edBhfrSnjrEc/YOaSDekOSURqiURvSLw42YFI7Xf8vgWM+NmRXPLURM4Z9iE/OnJvXp+6lCXryumYn6sxukQylPo8ZLf07tickVceTYvchjw+fiEl68pxvhmja+SUknSHKCIpFveMJLzR8Hh3X2tm06nk3hF3PzAZwUnt1L5F45hfhooxunRWIpJZKmvaehnYHL5+KQWxSB2ybL3G6BKRQNxE4u53xHotAvHH6GrdNCcN0YhIOqmPRPZIrDG6DFhVuoVH/zufHTs0io5Ipkjoqi0zawXcDZwEtCUqAbl785oPTWqzyDG6Kq7auvrE7nywYDVDx8xh8uK1PHBuX1rkNUxzpCKSbInekPh3oB8wHFiCBm0UvhmjK9K5h3XmkL1bctcbMznjL+/z2A8PoU9hizRFKCKpkGgiOQk42d0/TmYwUveZGRce1YU+hS248rnJnP3Yh9w1uA/nHtY53aGJSJIk2keyAth1TlaROA7ZuyVvXH0Mh3dpxfUvT+P6l6ayaev2dIclIkmQaCK5Bfi9mTVNZjBSv7Ru2oinLzmcX5zYnRGTijn7rx/y5eqydIclIjUs0aat3wJdgBVmtphdB23UDYkSU3aW8etTetJvr3x+9e+pnPHI+5x7aGfenLFMQ6uI1BOJJhLdkCjVcuJ+7Xj9F8dw/vCP+Nv/vti5vGJoFUDJRKSOqjKRmFlDoAnwqLsvTn5IUl91bpVHrNtLNLSKSN1WZR+Ju28Ffk5wv5lItSzV0Coi9U6ine1vAycmMxDJDB3zc2Muz2mQFXf8LhGp3RJNJO8CfzCzh8zsx2Z2duQjmQFK/RJraJWG2caOHc4pD47ntc80DL1IXZNoZ/tfwuerY6xzIDvGcpFdxBpa5bpTe3JQ53yuHfEZ17zwGWNnLufOwX1o2UQDQIrUBYnOkKjBHaXGxBpaBeDFnx3J4+8t5KF35vLJF2u4d8iBnNCzbRoiFJHdoQQhtUaD7CyuPKE7I688mpZ5OVz8j4nc/Op0Nm7elu7QRKQSiTZtVYwAPBDYC/hWm4O7/z7BfQwE/kzQFPY3d78nar2F608DyoCL3H1ygtv+BhgKFLj7qkSPS2qf/Tu24LWrjuaBsXN54v2F/G/eKh449yCK15bv0iSmS4ZF0i/RYeSPAN4gmDGxACgBOoTvFwFVJhIzywYeBU4GioGJZjbK3WdGFBsE9Agf/YHHgP5VbWtmncN1XyZyPFL7NW6Yzc2n9eKk/dry6xenMmTYRzTIMraFN6LoRkaR2iPRpq2hwHNAIbCJ4FLgvYBJwL0J7uNwYL67L3T3LcALwOCoMoOBZzwwAcg3sw4JbPsgcD0a3r7e6d+1NW/98jjycrJ3JpEKFTcyikh6Jdq0dSDwE3d3M9sONHL3hWZ2A/A8QZKpSiHwVcT7YoKzjqrKFFa2rZmdCZS4+9SgZSw2M7sMuAygoKCAoqKiBELOLKWlpbW2Xsq2xB45uGRdeVJjrs11kk6ql9gytV4STSRbIl4vB/YGZhEMLd8xwX3E+pWPPoOIVybmcjPLIxiZ+JSqPtzdhxNMzEXPnj19wIABVW2ScYqKiqit9VI4YVzMOeJb5jXk+OOPp7I/IqqjNtdJOqleYsvUekm0aWsycFj4ugi4y8wuBB4GpiW4j2IgcnajTgSzLSZSJt7ybsA+wFQzWxQun2xm7ROMSeqImHPEG6wt28oP//YxC1ZquhyRdNmd+UgqfvR/C6wEHgFaEjYXJWAi0MPM9jGzHOA8YFRUmVHABRY4Aljv7kvjbevu0929rbt3cfcuBAnnYHdflmBMUkec1a+QP559AIX5uRhQmJ/L/UMO4s6z+jC9ZD2DHnqf+9+eo8mzRNIg0RsSJ0W8XklwddVucfdtZnYVMIbgEt4n3f1zM7s8XD8MGE1w6e98gst/L65s292NQeq2eDcyDty/PX8YPYtHxs3ntc+WcMfg/XUjo0gKJXwfCYCZHUrQnPS6u280sybAZndP6I4xdx9NkCwilw2LeO3AlYluG6NMl0TikPqloFkjHvxBX845tBO3jpzBxf+YyKA+7bntu73p0CL2IJEiUnMSvY+kHUGz02EEnd89gIXAAwSXA1+TrABFEnVUtza8ec1xPPH+Qh5+dx7vzV3Jr07el1Z5Dbl/7DzdyCiSJImekTwILANa8+2b/l4k6CsRqRVyGgTDrJx5UEd+N+pz7npjFsY3lwfqRkaRmpdoZ/tJwC3uvjZq+QKCGxNFapXOrfL4+4WH0qpJzi7XmOtGRpGalWgiyeXb95JUKCBo2hKpdcyMtRtjfW01I6NITUo0kbwHXBTx3sPxr24gmPRKpFaKNyMjwOPjF+hyYZEakGgiuR641MzGAo2A+4GZwNHATUmKTaTaYt3I2KhBFr06NOOPb87mpPvH88rkYnbs0DBtInsqoUQSjrJ7APAhwfztjQk62vu5+4LkhSdSPbFuZLz3+wcy+prjeP7S/rRqksO1I6ZyxiP/4/15K9MdrkidlPB9JOHd4r+LXGZme5vZCHc/t8YjE6kh8W5kPKpbG1678mhen76UoWNm8+O/f8KxPdpw46D9mLe8lKFj5lCyrpzCCeN0ybBIJXbrhsQY8oHv10AcImmRlWWceVBHTt2/Hc9O+JJHxs3j9If/R7YZ211zn4gkQlPtigCNGmTzk2P2Yfx1J9C0UYOdSaSCLhkWiU+JRCRCi9yGceeI1yXDIrEpkYhEiXfJsAO3j/pcCUUkSqV9JGYWPcx7tOY1GItIrXDdqT256ZXplEfcY9KoQRZ9O7fg2QmLee7jxQw5pDNXDOhG51Z5aYxUpHaoqrN9dQLrv6ihWERqhYoO9Z1XbUUM9Fi8toxh4xcwYmIxIyZ9xff6FXLFgG50LWia5qhF0qfSROLuF6cqEJHapOKS4eipUzu1zOOusw7gqhN6MPy9hTz/yWJemVzMGQd25KoTuzNzyQaGjpmjkYYlo1T38l+RjNS+RWNu+25vfj6gG3/730L++dFiRk1dQpZBxU3yumxYMoU620WqoaBZI24a1IsPbjiRZo0aED3Sii4blkygRCJSA1o2yaE0zmXDJevKNTik1GtKJCI1pLKRho/847sMHTObZes164LUP0okIjUk1kjDuQ2zuOKEbhzapRV/LVrAMfeO4+p/TWHKl9FzxInUXepsF6khkZcNx7pq68vVZTz90SJGTPyKUVOX0LdzPhcf3YXTDujAG9OW6movqbOUSERqULyRhgH2ap3HrWf05lcn78vLnxbz1IeLuOaFz7h15AzKtmxn2w4NEil1k5q2RFKsaaMGXHhUF9699nievOhQNm/bsTOJVNDVXlKXKJGIpElWlnHifu3Ysm1HzPUl68pZE2fOeZHaRIlEJM0qu9qr/x/e4YrnPmX83JVs13TAUkupj0QkzWINEpnbMJurT+rOqtItvDK5mNHTl9GxRWOGHNqZcw7ppMEipVZRIhFJs6qu9rp+YE/embmCf0/6ikfGzeORcfM4ulsbzj2sM5u3buehd+bpai9JKyUSkVqgsqu9GjXI5vQDO3D6gR0oWVfOS5OCkYev/teUb5XT1V6SLuojEalDCvNzueY7PXj/+hNo3SRnl/XlW7fzxzdnpSEyyWRKJCJ1UFaWxb2ia/mGzZzxyPsMf28BJZrNUVIgpYnEzAaa2Rwzm29mN8ZYb2b2cLh+mpkdXNW2ZjbUzGaH5V81s/wUHY5IWsW72qtFbgOys7L4w+jZHH3POIY89iFPf7iIlV9v3llm5JQSjr5nHPvc+AZH3zOOkVNKUhW21EMp6yMxs2zgUeBkoBiYaGaj3H1mRLFBQI/w0R94DOhfxbZjgZvcfZuZ3QvcBNyQquMSSZd4V3vdcWYfzupXyOLVG3l92lL+M3UJvxv1OXf853OO6taGwpaNee2zJWzaGty/or4Vqa5UdrYfDsx394UAZvYCMBiITCSDgWfc3YEJZpZvZh2ALvG2dfe3I7afAAxJ+pGI1AJVXe21d+smXHlCd648oTtzl3/Nf6YuYdTUJfxv/qpd9lVxJ70SieyJVCaSQuCriPfFBGcdVZUpTHBbgEuAf8f6cDO7DLgMoKCggKKiot0IPTOUlpaqXqLU9jrJB+4+IgtoEixYP4+ionkxyx6SAwcfChePib2vknXlvDh6HAV5Vbd41/Z6SZdMrZdUJhKLsSz6Vt14Zarc1sxuAbYBz8X6cHcfDgwH6Nmzp0fOwy2B6PnJpX7WSeHH4+J2wl/3Xjn7tW/GKb3bcXLv9vQpbI7Zrv/96mO91IRMrZdUJpJioHPE+07AkgTL5FS2rZldCJwBnBQ2i4lIHPH6Vq49eV8Axs5czl/+O5+Hx82nQ4vGfKdXO77Tux1Hdm3N6OnBcPcl68opnDBON0AKkNpEMhHoYWb7ACXAecD/RZUZBVwV9oH0B9a7+1IzWxlvWzMbSNC5fry7l6XmUETqrqr6Vi49ritrNm5h3OwVjJ25jJc+LeafExbTKNvYtsPZHv6ppk56qZCyRBJeVXUVMAbIBp5098/N7PJw/TBgNHAaMB8oAy6ubNtw138BGgFjw1PwCe5+eaqOS6QuquxOeoBWTXIYckgnhhzSiU1bt/PhglVc9fwUNm//9tzz5Vu3c9cbMznjwA40yNZtaZkqpUOkuPtogmQRuWxYxGsHrkx023B59xoOU0QiNG6YzYn7taN8y/aY61eVbqHfnWM5qltrjtu3gON6FOwyqOTIKSWaAbIe01hbIpKQjvm5MTvpW+U15NQ+7Xlv7irGfL4cgC6t8zi2RwHH9mjDmo2bueM/s3b2yahJrP5RIhGRhMTrpL/tu/tzVr9C3J0vVm3k/XmreG/uSl6eHPStxKL7VuoXJRIRSUhkJ33JunIKo5qozIyuBU3pWtCUC4/qwpZtO5j85VrOGz4h5v5K1pXz8cLVHNQ5n8YNs1N2HFLzlEhEJGEVnfSJ3C+R0yCLI7q2pjBOkxjAD4ZPIKdBFn0753PEPq3o37U1/fbKJy8n+GlS30rdoEQiIkkVr0ns1jN6UdCsMZ98sZqPv1iz896VBlnGgZ1a0DIvh/fnrWLLdo0JVtspkYhIUlV138rJvdsB8PWmrXy6eC0ff7GGT75Yw7uzV+yyr/Kt2/nD6FmceVBHsrJiDXgh6aBEIiJJV9V9KwDNGjdkQM+2DOjZFoB9bnxjlzGUAFZ8vZl+d46lb+d8+nbOp99ewXN+3jcTfalJLLWUSESkVop3uXF+bkMGHdCeKV+u4+Fx86gYFKlrQRP6ds4ny4z/TF3C5m1qEksVJRIRqZXi9a3cfub+OxNC6eZtTCtex5Qvg8d7c1eyqnTXmSMrmsROP7ADDXUHfo1TIhGRWqmqvhWApo0acFS3NhzVrQ0A7k7Xm0bHbRLb/7Yx9GzfjD6Fzdm/Ywv279icXh2a77z8WE1ie0aJRERqrUT6ViKZWdwmsZZ5DTn30M7MWLKe0dOX8a9PgimOsrOM7gVNadY4m6nF69kajkqpJrHEKZGISL0Sr0nsd9/9pknM3SlZV86Mkg18vmQ9M0rWM37uSnZEncqUb93Ob0fOYIc7+7ZrRve2TWPePFlxJpOpw+srkYhIvZJIk5iZ0allHp1a5jGwT3sguEosltLN27h2xFQAsgy6tG5Cz/bN2LddM3q2b8ZXa8t4cOxcNm3N3M59JRIRqXd2t0kM4l8l1jG/Mc9ccjhzlpUyZ/nXzFm2gdnLvuatz5cRbxq9is79gX3aVzr8S33pk1EiEREhfpPY9afuR/e2zejethmn02HnuvIt21mwspQzHvlfzP2t+HozvW57i8L8XLoWNKVbQZPguU0TurVtyofzV3HzqzPqxajISiQiIiTWJBYpNyebPoUt4o4l1jKvIRcdtQ8LVpaycFUpkxatoSxiTheDXa4uK9+6nXvems3gvh0JJ+qLqbadySiRiIiE9qRJLJHOfQg6+Jdt2MTClRtZsLKU2177PNbuWLZ+Ewfc/jZ7tcpj79Z57N26SfDcKo+9WufxycI13DKydp3JKJGIiFRDVcPrVzAzOrTIpUOLXI7u3obHxy+MeSbTIrcBZ/UtZPGaMuYs+5p3Zi3feUlyPOVbt3P3G7M4unsb2jTNiXs2k6wzGSUSEZFq2p3h9SvEO5O548w+3/px377DWbq+nC9Xl7FodRk3vzo95v5Wlm7msLvfoVGDLApb5lKYn0unlrl0aplHYX4uC1eVMnz8QjbtwdAxFQkop333Q2KtVyIREUmDRPtksrO+uVT5qO7w6H/nx57yuEkO15zUg+K1ZZSsK6dkbTlvL9nA6o27DhlTIbhPZjqrN26hQ4vGtG/RmA4tGlPQtBENwqFkRk4p2SXhRVMiERFJk5rsk7ntjN4x91W+ZTsl68r5zgPjY+6vdPN27nx95reWZRm0bRYkltnLNuy8RyYeJRIRkTpkT64u6962adyrywrzG/PG1ceydP0mlq3fFD6XB88bNlWZRECJRESkzqnJM5nrTt2P/Lwc8vNy6NWh+S7bHX3PuLhTJVfQeMoiIhngrH6F/PHsAyjMz8WAwvxc/nj2AVUmpOtO7UluJXfng85IREQyxp6cyUQ2pS2NU0ZnJCIiUqmz+hXywY0nsmXZ/E9jrVciERGRalEiERGRalEiERGRalEiERGRalEiERGRaklpIjGzgWY2x8zmm9mNMdabmT0crp9mZgdXta2ZtTKzsWY2L3xumarjERGRFCYSM8sGHgUGAb2B882sd1SxQUCP8HEZ8FgC294IvOvuPYB3w/ciIpIiqTwjORyY7+4L3X0L8AIwOKrMYOAZD0wA8s2sQxXbDgaeDl8/DZyV5OMQEZEIqbyzvRD4KuJ9MdA/gTKFVWzbzt2XArj7UjNrG+vDzewygrMcgM1mNmNPDqKeawOsSncQtYzqJDbVS2z1vV72jrUwlYkk1pRd0dN+xSuTyLaVcvfhwHAAM5vk7ofuzvaZQPWyK9VJbKqX2DK1XlLZtFUMdI543wlYkmCZyrZdHjZ/ET6vqMGYRUSkCqlMJBOBHma2j5nlAOcBo6LKjAIuCK/eOgJYHzZbVbbtKODC8PWFwGvJPhAREflGypq23H2bmV0FjAGygSfd/XMzuzxcPwwYDZwGzAfKgIsr2zbc9T3ACDP7CfAlcE4C4QyvuSOrV1Qvu1KdxKZ6iS0j68Xcd6urQURE5Ft0Z7uIiFSLEomIiFRLRiWSqoZoyVRmtsjMppvZZ2Y2Kd3xpIuZPWlmKyLvMdIQPHHr5XYzKwm/M5+Z2WnpjDHVzKyzmf3XzGaZ2edmdk24PCO/LxmTSBIcoiWTneDufTPxGvgITwEDo5ZpCJ7Y9QLwYPid6evuo1McU7ptA37t7r2AI4Arw9+TjPy+ZEwiIbEhWiSDuft7wJqoxRk/BE+ceslo7r7U3SeHr78GZhGMwJGR35dMSiTxhl+RYJSAt83s03AoGfnGt4bgAWIOwZOhrgpH6X4yU5pwYjGzLkA/4GMy9PuSSYmk2sOs1GNHu/vBBM1+V5rZcekOSGq9x4BuQF9gKXB/WqNJEzNrCrwM/NLdN6Q7nnTJpESSyBAtGcndl4TPK4BXCZoBJaAheGJw9+Xuvt3ddwBPkIHfGTNrSJBEnnP3V8LFGfl9yaREksgQLRnHzJqYWbOK18ApgEZG/oaG4Imh4scy9D0y7DtjZgb8HZjl7g9ErMrI70tG3dkeXqL4EN8Ms3J3eiNKPzPrSnAWAsGQOc9nar2Y2b+AAQRDgS8HfgeMBEYAexEOwePuGdXxHKdeBhA0azmwCPhZRd9AJjCzY4D3genAjnDxzQT9JBn3fcmoRCIiIjUvk5q2REQkCZRIRESkWpRIRESkWpRIRESkWpRIRESkWpRIROo4M3MzG5LuOCRzKZGIVIOZPRX+kEc/JqQ7NpFUSdmc7SL12DvAj6OWbUlHICLpoDMSkerb7O7Loh5rYGez01Vm9oaZlZnZYjP7UeTGZnaAmb1jZuVmtiY8y2kRVebCcPKxzWa23MyeioqhlZm9aGYbzWxh9GeIJJMSiUjy3UEwBlNfYDjwjJkdCmBmecBbQCnBwIffA44CnqzY2Mx+BjwO/AM4EDgN+DzqM24jGNfpIODfwJNmtnfSjkgkgoZIEamG8MzgR8CmqFWPuvsNZubA39z90oht3gGWufuPzOxS4E9Ap3CCJMxsAPBfoIe7zzezYuBZd4852174Gfe4+03h+wbABuAyd3+25o5WJDb1kYhU33tA9IRg6yJefxS17iPg9PB1L2BaRRIJfUgwEGBvM9tAMAHbu1XEMK3ihbtvM7OVZMikSpJ+SiQi1Vfm7vP3cFsj/gRrTuwJ2WLZGmNbNV1LSuiLJpJ8R8R4Pyt8PRM4qGJOmNBRBP83Z7n7cqAEOCnpUYrsIZ2RiFRfIzNrH7Vsu7uvDF+fbWYTgSJgCEFS6B+ue46gM/4ZM7sNaEnQsf5KxFnO3cCDZrYceAPIA05y94yc3lZqHyUSker7DsG85ZFKCKZzBrgd+D7wMLASuNjdJwK4e5mZnUow4donBJ32rwHXVOzI3R8zsy3Ar4F7gTXA6CQdi8hu01VbIkkUXlF1jru/lO5YRJJFfSQiIlItSiQiIlItatoSEZFq0RmJiIhUixKJiIhUixKJiIhUixKJiIhUixKJiIhUy/8DSzqshL63lrYAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# extra code – this cell plots exponential scheduling\n",
"\n",
"plt.plot(history.epoch, history.history[\"lr\"], \"o-\")\n",
"plt.axis([0, n_epochs - 1, 0, 0.011])\n",
"plt.xlabel(\"Epoch\")\n",
"plt.ylabel(\"Learning Rate\")\n",
"plt.title(\"Exponential Scheduling\", fontsize=14)\n",
"plt.grid(True)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The schedule function can take the current learning rate as a second argument:"
]
},
{
"cell_type": "code",
"execution_count": 74,
"metadata": {},
"outputs": [],
"source": [
"def exponential_decay_fn(epoch, lr):\n",
" return lr * 0.1 ** (1 / 20)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Extra material**: if you want to update the learning rate at each iteration rather than at each epoch, you can write your own callback class:"
]
},
{
"cell_type": "code",
"execution_count": 75,
"metadata": {},
"outputs": [],
"source": [
"K = tf.keras.backend\n",
"\n",
"class ExponentialDecay(tf.keras.callbacks.Callback):\n",
" def __init__(self, n_steps=40_000):\n",
" super().__init__()\n",
" self.n_steps = n_steps\n",
"\n",
" def on_batch_begin(self, batch, logs=None):\n",
" # Note: the `batch` argument is reset at each epoch\n",
" lr = K.get_value(self.model.optimizer.learning_rate)\n",
" new_learning_rate = lr * 0.1 ** (1 / self.n_steps)\n",
" K.set_value(self.model.optimizer.learning_rate, new_learning_rate)\n",
"\n",
" def on_epoch_end(self, epoch, logs=None):\n",
" logs = logs or {}\n",
" logs['lr'] = K.get_value(self.model.optimizer.learning_rate)"
]
},
{
"cell_type": "code",
"execution_count": 76,
"metadata": {},
"outputs": [],
"source": [
"lr0 = 0.01\n",
"model = build_model()\n",
"optimizer = tf.keras.optimizers.SGD(learning_rate=lr0)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
" metrics=[\"accuracy\"])"
]
},
{
"cell_type": "code",
"execution_count": 77,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.6804 - accuracy: 0.7679 - val_loss: 0.4803 - val_accuracy: 0.8276\n",
"Epoch 2/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.4683 - accuracy: 0.8361 - val_loss: 0.4410 - val_accuracy: 0.8412\n",
"Epoch 3/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.4216 - accuracy: 0.8493 - val_loss: 0.4108 - val_accuracy: 0.8536\n",
"Epoch 4/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3974 - accuracy: 0.8591 - val_loss: 0.3858 - val_accuracy: 0.8584\n",
"Epoch 5/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3770 - accuracy: 0.8657 - val_loss: 0.3784 - val_accuracy: 0.8624\n",
"Epoch 6/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3625 - accuracy: 0.8713 - val_loss: 0.3784 - val_accuracy: 0.8626\n",
"Epoch 7/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3512 - accuracy: 0.8736 - val_loss: 0.3662 - val_accuracy: 0.8674\n",
"Epoch 8/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3414 - accuracy: 0.8779 - val_loss: 0.3699 - val_accuracy: 0.8638\n",
"Epoch 9/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3333 - accuracy: 0.8810 - val_loss: 0.3470 - val_accuracy: 0.8714\n",
"Epoch 10/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3260 - accuracy: 0.8827 - val_loss: 0.3463 - val_accuracy: 0.8718\n",
"Epoch 11/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3197 - accuracy: 0.8852 - val_loss: 0.3509 - val_accuracy: 0.8718\n",
"Epoch 12/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3140 - accuracy: 0.8877 - val_loss: 0.3463 - val_accuracy: 0.8764\n",
"Epoch 13/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3093 - accuracy: 0.8893 - val_loss: 0.3345 - val_accuracy: 0.8762\n",
"Epoch 14/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3049 - accuracy: 0.8907 - val_loss: 0.3397 - val_accuracy: 0.8778\n",
"Epoch 15/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3010 - accuracy: 0.8925 - val_loss: 0.3400 - val_accuracy: 0.8788\n",
"Epoch 16/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2969 - accuracy: 0.8934 - val_loss: 0.3318 - val_accuracy: 0.8792\n",
"Epoch 17/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2942 - accuracy: 0.8939 - val_loss: 0.3337 - val_accuracy: 0.8780\n",
"Epoch 18/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2913 - accuracy: 0.8960 - val_loss: 0.3290 - val_accuracy: 0.8766\n",
"Epoch 19/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2889 - accuracy: 0.8962 - val_loss: 0.3264 - val_accuracy: 0.8778\n",
"Epoch 20/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2865 - accuracy: 0.8970 - val_loss: 0.3262 - val_accuracy: 0.8794\n",
"Epoch 21/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2845 - accuracy: 0.8980 - val_loss: 0.3226 - val_accuracy: 0.8798\n",
"Epoch 22/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2822 - accuracy: 0.8985 - val_loss: 0.3262 - val_accuracy: 0.8814\n",
"Epoch 23/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2807 - accuracy: 0.8998 - val_loss: 0.3254 - val_accuracy: 0.8790\n",
"Epoch 24/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2788 - accuracy: 0.9006 - val_loss: 0.3258 - val_accuracy: 0.8816\n",
"Epoch 25/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2776 - accuracy: 0.9008 - val_loss: 0.3249 - val_accuracy: 0.8808\n"
]
}
],
"source": [
"n_epochs = 25\n",
"batch_size = 32\n",
"n_steps = n_epochs * math.ceil(len(X_train) / batch_size)\n",
"exp_decay = ExponentialDecay(n_steps)\n",
"history = model.fit(X_train, y_train, epochs=n_epochs,\n",
" validation_data=(X_valid, y_valid),\n",
" callbacks=[exp_decay])"
]
},
{
"cell_type": "code",
"execution_count": 78,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEbCAYAAADwPQLqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA8FUlEQVR4nO3deXwU9fnA8c+TE5JAICGQcIYzyCFyX4pBRQG1aMX7QDwobamt1npU21qt/VmPqlQrXlStJ1ZURBQQjQpyhFuukAABwhXOQAgEQp7fHzPBdc2xuXZzPO/Xa17ZnfnO7DPf7O6z8/3OfEdUFWOMMaaiggIdgDHGmNrNEokxxphKsURijDGmUiyRGGOMqRRLJMYYYyrFEokxxphKsURiahURuVlEcsu5ToqIPFddMbmvkSkid1fDdseKSLnO0feuo4rUWWWIyEMiMtVfr1fM66uIjA3A65ZZzyIySURm+Csmf7FEUkuIyGvuB8R7WhTo2KpLCV8I7wEdquG1bhORFSKSKyI5IrJaRP5W1a8TINVSZ8URkebAXUCtrjs3Ga6phk2/DPQTkXOqYdsBExLoAEy5fAHc6DXvRCACCRRVPQYcq8ptisgtwGTgTmAeEAZ0BwZX5esESnXUWSluA5ao6ubqfiERCVXVk9X9OlVJVfNF5G3gDuDbQMdTVeyIpHbJV9XdXtMBABE5V0ROikhyUWERmSgih0Wkg/s8RUSmiMizInLQnZ4QkSCPdZqKyOvusmMi8oWIdPdYfrP7q/18EVkjIkdF5CsRae8ZqIhcKiLLROS4iGwRkUdFJMxjeaaIPCgiL7oxZonIHzyXuw/fd49MMj1f36NcRxH5WER2u7EsF5FLylmvPwOmq+qLqpqhqutU9X1Vvctrny4WkcVuvewXkU9EpIFHkQYl7Y+7frSIvCQi2SJyRES+FpF+XmVuEpGtIpInIjOBFl7Lf/JLuawmlWLq7CH3f3eNiGxyY/lIRJp5lAkRkac93idPi8gLIpJSRl1eB/yo6cbH912YiPzDrbejIpIqIhd5LE923wejRWSJiJwALqJk8SLyqVuPW0XkBq+YHhORNPd/mSkijxf9L0XkZuAvQHf54cj/ZndZY7cedrnv7fUicrXXtkv9bLj18zMRiSijLmsPVbWpFkzAa8DMMsr8HdgOxABdgaPAOI/lKcAR4F/u8quAHOAujzIfAxuAYUBPnDf9dqChu/xm4CTO0dEA4ExgBTDbYxsXAYeB8UBHYDiQBjzpUSYT2A9MAjoBvwEUGOwuj3Of3wbEA3Eer5/rsZ1ewEQ31k7AAzhHaV299vu5UuptCrAR6FBKmZFAAU6TTTd3v+8GInzcHwHmA5+69dYJeMStpwS3zECg0N2HLsAv3G2qRxwPAWu8YvOuk7KePwTkAh+6+zEY2Aq86FHmPuAgcAWQBDzrvldSSqmjGDf+IV7zUyj7ffcWsAjnfdfBrccTQC93ebJbn98DF7pl4kqIQ916+4Vbjw+4cfXzKPMnYCiQCIwGtgGPuMsaAk/ifA7i3amh+z9cAKxz3w8dgFHA5b5+NtxyEcAp4PxAf69U2fdToAOwycd/lJNICtwvAM/pHx5lQoFUYDqwHHjPaxspOF+Y4jHvQSDLfdzZ/RAO81ge7X7ob3Of3+yWSfIoc737oQ9yn38D/MnrtS9z4xX3eSbwjleZdOBBj+cKjPUqczMeX4ol1NUir+2kUHoiSQAWuq+XDrwJ3ASEepRZALxbyjZK3R/gPHf/G3qVWQnc4z5+G5jrtfwVqieRHAeiPeY9AGR4PN8F3OfxXHC+WFNKqYOz3DpsX873XUecL/q2Xut9BPzbfZzsbvsKHz4rCrzsNe8L4M1S1pnotf/F1fMIN84zStjGzZTx2fCYfwC4tax9qS2TNW3VLt/gfFg9pyeKFqrTXnwdcAnQHOcXmbdF6r6TXQuBViLSGDgD54Oy0GObOTi/Art5rJOvqmkez3fiJLEm7vO+wANuE1iu26zyNhCJ8+uuyGqv2Ha6cftMRCLdZol1bpNJLtAPaOvrNlR1l6oOxjmqeQbnS/NFYIlH80NvnP6T0pS2P31xfonu9aqXHjhfpODU/0KvbXg/rypb3f/tT2IVkWic/9OSooXueya1jG02dP8eL2ZZae+7Pjh1vs6rbi7mh7opsrSMGDy37/389HtYnLPh5rtNornA05T9nukN7FLV9aWUKeuzUeQYP9RXrWed7bVLnqpmlFFmEE7fVxOc5qFD5di+lLLM80ugoIRlQR5//wq8X8x29no89u4oVcrfb/ckTjPD3ThHAHnAGzgd5uWiqmuANcDzInI2TmfoVThHg74obX+CgD1AcWfrHHb/llb/RQqLKRfqY3yefKn78g4Nvs/92xTniMZXQe5r9S8mLu+TBI6WM6afEJFBwLs479E7cT4jP8N5L5W6qg+bL+uzUSSGH38WajU7IqlDRCQReA74NTAXeEtEvH8sDBQRzw/EIGCnqh7GafsNwuNsJfcXY093ma+W4/RRZBQzeX/QSnMSCC6jzNnAG6r6gaquBrL46a/Yiija3yj37wrg/EpsbzlOx3lhMXWS7fGag7zW836+F2jh9T88qxJx/YR7pLIbp50fAPf1+pex6iacpNitmGWlve9W4HxJxxdTNzsquBvF1WPRkcRQYIeqPqKqqaqaDrTzKn+Cn773lgMJInJGBWMCnBNEgAbu9uoEOyKpXcJFJN5r3ilV3SsiwTht+1+r6osi8j+cJqm/4HQsFmkJPCMi/8ZJEH/APedfVdNF5GPgRRGZgPNL7VGcL4e3yxHnw8BMEdkKTMP5ldYDGKCq95RjO5nA+SLyNU6TwcFiymwELnfjPomzvw2KKVciEXkBpwniS5xElIDThp8HzHGLPQp8IiIZOHUhOJ2+L6pqng8v8wVOP8vHInIPP3TkjgS+UNVvcU5B/k5E7gf+h9MvcLnXdlJwfs3+UUTedctUx8V3zwL3iMhGnAT3C5x6KfFIQ1ULReQLnOT+P6/Fpb3vNorIW8BrIvJ7nC/YGJx926yq0ysQ/89FJBWnvsbi/AgY6C7biNOsdj1Ok9dFwLVe62cC7USkD05H/BGcps3FwAcicqe7nU5ApKp+VI7YzsHZr/Ty71bNZEcktcsFOB9kz2mFu+yPOG/qWwFUdT8wDrjPbaYp8hbOL63FOBdHvYrTPlxkPE7b+Az3bwQwUp1rEXyiqrNx2reHu9tYgnMW0DbfdxWA37vb2M4P++ntLiAbpxnqM5yO9vKenz8X50tmGs6Xw4fu/BGquhFAVWfhfKmPcmP52o2t0JcXcPsHRuMkq5dxzmKbhnNG1E63zCKc/98vcfpbfo7T6eu5nfXu8glumRE4Z+tVtSeB/wL/walTcOqluP4PTy8BV7s/bDz58r77D/A4TpKdiXMG19YKxv8Qzhlnq3Hqa7yqpgKo6ic4fYvP8EMd/tlr/Q+AWTjJYy9wraoW4vz/F+D8aFuPk3DL24x6LU4d1BlFZ9CYesC9BmCNqk4KdCym9hGR5cACVf1NGeUW4pxt9V/3eQr2vgNARHrgJKcuXic71GrWtGWM+QkRaYfT5PM1zvfEBJxrdib4sPovcM5wMj/VEripLiURsERijCleIc61NE/gNIGvA0apapmn37onPXifCm0AVZ1Tdqnax5q2jDHGVIp1thtjjKmUetm01aRJE+3UqVOgwyjW0aNHiYyMDHQYxbLYKsZiqxiLrWKqM7Zly5btU9W4nywI9BgtgZi6dOmiNdVXX30V6BBKZLFVjMVWMRZbxVRnbMBStbG2jDHGVDVLJMYYYyrFEokxxphKsURijDGmUiyRGGOMqRRLJMYYYyrFEokxxphKsURijDGmUiyRGGOMqRRLJMYYYyrFEokxxphKsURijDGmUiyRGGOMqRRLJMYYYyrFEokxxphK8WsiEZGRIpImIhkicl8xy0VEJrvLV4tIH49lU0UkW0TWeK0TIyJzRSTd/dvUH/tijDHG4bdEIiLBwPPAKKAbcK2IdPMqNgro7E4TgBc8lr0GjCxm0/cB81S1MzDPfV6qQrtNvTHGVBl/HpEMADJUdbOqngDeBcZ4lRkDvOHejGsR0EREEgBU9RvgQDHbHQO87j5+HbisrED25imFlk2MMaZK+POe7a2A7R7Ps4CBPpRpBewqZbstVHUXgKruEpHmxRUSkQk4RzmExXfi7v/M5Wcdw8q3B36Qm5tLSkpKoMMolsVWMRZbxVhsFROI2PyZSKSYed6HBb6UqRBVfQl4CSA8obN+lHGSK5P7MrhjbFVsvsqkpKSQnJwc6DCKZbFVjMVWMRZbxQQiNn82bWUBbTyetwZ2VqCMtz1FzV/u3+yyAokOEwoV7nh3BXuP5JcZuDHGmJL5M5GkAp1FpL2IhAHXADO8yswAbnLP3hoE5BQ1W5ViBjDOfTwO+LisQJo2EAa0j2HvkXzufG8lp6y/xBhjKsxviURVC4BJwGxgPTBNVdeKyEQRmegWmwVsBjKAl4FfFa0vIu8AC4EkEckSkVvdRY8BI0QkHRjhPi/Tv67tTWxkGPMz9vH8VxlVsIfGGFM/+bOPBFWdhZMsPOdN8XiswK9LWPfaEubvB84vbywtGjfg6avPYtx/lvDMFxvpl9iUIR2blXczxhhT79XrK9uHdYlj0vBOTn/JOyvJPnI80CEZY0ytU68TCcBvz+/MwPYx7MvN53fvWn+JMcaUV71PJCHBQfzr2t40iwrju037eXruxkCHZIwxtUq9TyQAzRs3YPI1vQkSeO6rDOas3R3okIwxptawROIa0qkZ94zsCsDvp61i897cAEdkjDG1gyUSD78Y1oFRPeI5kl/AxDeXcTS/INAhGWNMjWeJxIOI8MSVvegYF8nGPbnc+8FqnDOSjTHGlMQSiZeo8BBevLEfkWHBzFy9i1fnbwl0SMYYU6NZIilGp+ZRPHllLwD+77MNLNq8P8ARGWNMzWWJpASjeibwi3M7cKpQmfT2cnblHAt0SMYYUyNZIinFHy5MYkjHWPblnmDim8s5fvJUoEMyxpgaxxJJKYouVmzVpCGrth/ij9O/t853Y4zxYomkDLFR4bx0U18ahgYzfcUOXv52c6BDMsaYGsUSiQ+6t4zmn1f90Pn+1YYy751ljDH1hiUSH43qmcDvLuiMKtzxzgoysu3Kd2OMAUsk5XLHeZ1PX/l++xtLyck7GeiQjDEm4CyRlENQkPDUVb04I6ExW/YdZdI7yyk4VRjosIwxJqAskZRTRFgIL9/Ul9jIML5N38ffZ20IdEjGGBNQlkgqoHXTCKbc2JfQYGHqgi28vXhboEMyxpiAsURSQf0TY3j0sp4A/OnjNXyzcW+AIzLGmMCwRFIJV/Vvw6+SO3KqUPnVW8tJ230k0CEZY4zfWSKppLsvTOLiMxPIzS/gltdSyT58PNAhGWOMX1kiqaSgIOGpK3vRp20Tdhw6xm1vLCXvhN0QyxhTf1giqQINQoN5+aZ+tI2JYHVWDr99dyWnCm1MLmNM/WCJpIrERoUz9eb+NG4Qwtx1e/i/WesDHZIxxviFJZIq1Kl5FC/e2I/QYOGV+Vt4Y2FmoEMyxphqZ4mkig3uGMv//fxMAB6asZbP1+wOcETGGFO9LJFUg7F9W3PXiC4UKtzx7gpSMw8EOiRjjKk2lkiqyW/O68T1A9tyoqCQW19LZeMeu8bEGFM3WSKpJiLCw2N6cGG3Fhw+XsC4qUvYecju+26MqXsskVSj4CBh8rW9GZAYw66c44ybusSGnjfG1Dl+TSQiMlJE0kQkQ0TuK2a5iMhkd/lqEelT1roicpaILBKRlSKyVEQG+Gt/fFF0jUmXFlGkZ+dy2xupHD95KtBhGWNMlfFbIhGRYOB5YBTQDbhWRLp5FRsFdHanCcALPqz7OPBXVT0L+LP7vEaJjgjl9VsGkBDdgNTMg9zxzgq7YNEYU2f484hkAJChqptV9QTwLjDGq8wY4A11LAKaiEhCGesq0Nh9HA3srO4dqYiE6Ia8ccsAohuGMmfdHv44/XtULZkYY2o/8deXmYiMBUaq6m3u8xuBgao6yaPMTOAxVZ3vPp8H3AsklrSuiJwBzAYEJzEOUdWtxbz+BJyjHOLi4vpOmzat2va1NOkHT/FE6nFOFMJF7UK4pmsYInJ6eW5uLlFRUQGJrSwWW8VYbBVjsVVMdcY2fPjwZaraz3t+SLW8WvGkmHneWaykMqWt+0vgTlX9QESuAl4FLvhJYdWXgJcAkpKSNDk52cewq1Yy0KX7Xm57PZXZWws4o3N7fndBl9PLU1JSCFRsZbHYKsZiqxiLrWICEZs/m7aygDYez1vz02aoksqUtu44YLr7+H2cZrAa7dwucUy+pjdBAs98kc6r87cEOiRjjKkwnxOJiIwSkZkisk5E2rjzbhOR833cRCrQWUTai0gYcA0ww6vMDOAm9+ytQUCOqu4qY92dwLnu4/OAdF/3KZBG9Uzg8bG9AHhk5jqmpW4PcETGGFMxPjVticj1wBTgFeB8INRdFAzcA8wraxuqWiAik3D6M4KBqaq6VkQmusunALOA0UAGkAeML21dd9O3A8+KSAhwHLcfpDYY27c1ucdP8tAn67hv+moiw0OIDHRQxhhTTr72kdwD3K6q74rIbR7zFwEP+/piqjoLJ1l4zpvi8ViBX/u6rjt/PtDX1xhqmpuHtufI8QKemruR3723gt+cFUZyoIMyxphy8LVpqzOwsJj5ufxw6q2poEnndWLCsA6cPKX8a0U+CzL2BTokY4zxma+JZCfQpZj5w4BNVRdO/SQi3D+qK9cNbMvJQrj19VQWbtof6LCMMcYnviaSl4DJIjLUfd5GRMbhXEX+QrVEVs+ICH8b04NzWoVw/GQht7yWypItNvy8Mabm8ymRqOrjOKfYzgUiga9wOt+nqOrz1Rde/RIUJIzvEcYVfVpz7OQpbv7PEpbavUyMMTWcz6f/quoDQDOc6zQGAXGq+qfqCqy+ChLh8bFncnnvVuSdOMW4qUtYtvVgoMMyxpgS+ZRIRGSqiDRS1TxVXaqqS1Q1V0QiRWRqdQdZ3wQHCU9e2Yuf9WrJ0ROnuHnqElZuPxTosIwxpli+HpGMAxoWM78hcFPVhWOKBAcJ/7yqFxf3TOBIfgE3vrqY1VmHAh2WMcb8RKmJRERiRCQWZ6yrpu7zoikOuATY449A66OQ4CCeueYsRnaP58jxAq5/ZTErtlkzlzGmZinriGQfkI0zQOI6YK/HtBvnSvd/V2eA9V1ocBCTr+19Opnc8MpiO5vLGFOjlJVIhuMMiSLAWJyxrIqms4G2qvpotUZoCAsJ4l/X9eZSt89k3NQlfGcXLRpjaohSh0hR1a8BRKQ9sF1VC/0SlfmJ0OAgnrn6LMJDgvjfsizGv5bKizf2JTmpeaBDM8bUc75eR7JVVQtFpKWIDBKRYZ5TdQdpHMFBwuNXnMl1A9uSX1DIhDeWMXeddVEZYwLL19F/WwJv4wyJUnSjKc+bUgVXfWimOEFBwqOX9SA8JIj/LMjkl28uY/K1vRndMyHQoRlj6ilfT/99BjgFdMMZ3v0c4EpgPTCyWiIzJRIR/nxJNyae25GCQmXS28uZvjwr0GEZY+opX4eRPxe4WFU3iIgCe1V1gYjkA4/gDJ1i/EhEuHdkEuEhQTw7L527pq0i59hJxg9tH+jQjDH1jK9HJA1xTgUGOAAU9fCuA86s6qCMb0SEO0d04cGLzwDgr5+s459zN+Lc1sUYY/zD10SyAejqPl4JTBSRdjg3odpRDXGZcrjtnA48MfZMgoOEyfPS+cuMtRQWWjIxxviHr01bzwLx7uOHgc+Ba4F8nOFTTIBd2a8N0Q1DmfTOCt5YuJVDeSd58spehIX4PC6nMcZUiK+n/76lqq+5j5cDiUB/nAsS36+26Ey5XNg9ntfG9ycqPIQZq3Yy4b9LOXbiVKDDMsbUcRX6ueqOArwcOCoi91VxTKYShnRsxju3DyImMoyUtL3c8OpicvJOBjosY0wdVmYiEZFmInKxiFwoIsHuvFAR+R2QCdxdvSGa8urZOpr3Jw6mZXQDlm09yNgp35F1MC/QYRlj6qiyRv8dAqQDnwCfAQtEpCuwGpiEc+pv2+oO0pRfx7go/vfLIXRpEUV6di4///d3rN2ZE+iwjDF1UFlHJI8As3FO8X0W5+6IM4H/Azqr6nOqaj91a6iWTRry/sQhDOoQQ/aRfK6aspCvN+4NdFjGmDqmrETSC3hEVdcAD+IMi3K/qr6hdrFCrRDdMJTXbxlw+m6Lt7yWyrSl2wMdljGmDikrkcTg3HsE98gjD1hR3UGZqhUeEswzV5/FxHM7cqpQued/q3nmC7tw0RhTNXw5a6upx50SFWjsdafEmGqO0VSBoCDhvlFdeWRMd4IEnvkinXs/WM3JU3ZnAGNM5fhyQeI6j8cCpHo9V2z031rjxsGJtGjcgDveXcG0pVnsPHSc56/rQ3REaKBDM8bUUmUlkuF+icL41YXd43nn9kHc/sZS5mfs4/IXFvDquP60bxYZ6NCMMbWQT3dINHVP77ZN+ejXQ7n1taWk7TnCZc8vYMoNfRncMTbQoRljahkbiKkea900gg9+NYTzujYn59hJbnx1Me8u2RbosIwxtYxfE4mIjBSRNBHJKG5oFXFMdpevFpE+vqwrIr9xl60Vkcf9sS91RVR4CC/f1I/bz2lPQaFy3/Tv+dvMdZyy0YONMT7yWyJxh1d5HhiFc6fFa0Wkm1exUUBnd5oAvFDWuiIyHBgDnKmq3YEnq39v6pbgIOGBi7vxjyt6EhIkvDJ/CxPeWEpufkGgQzPG1AL+PCIZAGSo6mZVPQG8i5MAPI0B3lDHIqCJiCSUse4vgcdUNR9AVbP9sTN10dX92/LfWwfSJCKUeRuyufz5BWzZdzTQYRljajjx10VpIjIWGKmqt7nPbwQGquokjzIzcZLCfPf5POBenGHri11XRFYCH+PcO/44cLeqep6iXLTtCThHOcTFxfWdNm1ade1qpeTm5hIVFRXQGPYcLeTZ5cfZeVRpGAITe4XTKy6kRsRWEoutYiy2iqmvsQ0fPnyZqvbznu/Tja1EZGoJixTnyzsDeE9Vd5a2mRLW96VMaeuGAE2BQTj3SJkmIh28h3BR1ZeAlwCSkpI0OTm5lFADJyUlhZoQ2+jzT/L7aauYs24PzyzP5/cj2tE9MqtGxFacmlJvxbHYKsZiq5hAxOZr01Yc8HPgMqCTO13mzksC7gHSROSsUraRBbTxeN4a8E48JZUpbd0sYLrbHLYEKASa+bZbpiSNGoQy5Ya+/H5EFwCenLOR51bmW7+JMeYnfE0kC3CGkW+tqsNUdRjOl/ksYA7QDvgUeKqUbaQCnUWkvYiEAdcAM7zKzABucs/eGgTkqOquMtb9CDgPQES6AGHAPh/3y5QiKEj4zfmdeXVcPxqFh7BszynrNzHG/ISvieS3wMOeQ8a7jx8F7nQ7wP8BnFXSBlS1AOceJrOB9cA0VV0rIhNFZKJbbBawGaep7GXgV6Wt664zFeggImtwOuHH2cjEVeu8ri34eNJQWkYJ6dm5/Oy5+cxbvyfQYRljagif+kiAKCAB50vcU7y7DOBwWdtT1Vk4ycJz3hSPxwr82td13fkngBtKD99UVoe4KP40qCEf7WrE7LV7uPX1pfwquSN3jehCSLBd12pMfebrN8CHwKsicqWIJIpIOxG5EngVmO6WGQBsrI4gTc3QMER44fq+3DMyiSCBf6ds4vpXFpN9+HigQzPGBJCviWQiTrPSm8AmnOanN4HPcZufcI5Wbq/qAE3NEhQk/Cq5E2/fPoi4RuEs3nKA0ZO/5bsM65Yypr7yKZGoap6qTsS50VVvoA8Qo6q/VNWjbpmVqrqy2iI1NcqgDrHMuuMchnSMZV/uCW54dTGT56VTaEOrGFPvlKtxW1WPqupqVV1VlEBM/RXXKJz/3jqQO87rhAL/nLuRm19LZX9ufqBDM8b4kU+JREQaiMi9IjJHRFa6Ayqenqo7SFNzBQcJd12YxH9u7k/TiFC+2biXiyfPZ9Hm/YEOzRjjJ74ekfwbuA/IxLlu4wOvydRzyUnN+fSOc+jbrim7Dx/n2pcX8dScNArsVr7G1Hm+nv57GXClqn5RjbGYWq5lk4a8O2EQz36RzvMpGfzrywwWZOzj2Wt60yYmItDhGWOqia9HJHnA9uoMxNQNocFB3H1REm/fNoj4xg1Yvu0Qo5/9lhmrShuGzRhTm/maSB4H7hIRu/LM+GRwx1g+++05XNitBUfyC7jjnRXc/f4qjtpYXcbUOb4mhhHA1cAWEflMRGZ4TtUYn6nFmkaG8eKNffnbZT0IDwnif8uyuORf81m1/VCgQzPGVCFfE8k+nKvbvwR2A/u9JmOKJSLcMKgdn/zmbLrGN2LLvqP8/IXveHruRk5aR7wxdYJPne2qOr66AzF1W5cWjfjo10N5/PM0pi7YwrPz0vlyQzb/vKoXnVs0CnR4xphKsD4P4zcNQoP586XdePv2gbRq0pDvd+Rw8b/m88q3m+2KeGNqsRITiXuxYVP38ffeFyHaBYmmooZ0bMbnvzuHK/u25kRBIX/7dD3XvryI7Qfyyl7ZGFPjlNa09QFQNNbF//wQi6lHGjUI5Ykre3Fh93jun76axVsOMOrZb/nTJWdwVb82iBR3d2VjTE1UYiJR1b8W99iYqjSiWwv6tB3GAx+u4fO1u7n3g++ZuXoXf7+8p13EaEwtYX0kJuBio8J54YY+PH11L5pEhPJt+j4ueuYbXluwxfpOjKkFfB20MUZEXhCRjSJySEQOe07VHaSp+0SEy3u3Zu6d53JxzwTyTpzioU/WcdWLC9m0NzfQ4RljSuHrWFuv4tyH5CVgJ2A/E021iGsUzvPX9+HSNbv508drWLr1IKOe/Zbfnt+ZCcM6EGq39TWmxvE1kZwPjFDVxdUZjDFFRvaIZ3CHWP726TreX5bFE7PTmPX9Lv5xxZn0aBUd6PCMMR58/XmXDVj7gvGr6AjnzK43bhlAqyYNWbvzMD97bj6PzFxHro3ZZUyN4WsieQB4WESiqjMYY4ozrEscc+4cxvihiQC8On8LFzz1NZ+v2YWqtbIaE2i+JpIHgQuBbBFZbxckGn+LDA/hL5d2Z8akszmzdTS7Dx9n4pvLue31pXYhozEB5msfiV2QaGqEHq2i+fBXQ3l78VYe/zyNeRuyWbBpH5e2D2boOYXWGW9MAJSZSEQkFIgEnlfVrdUfkjGlCw4SbhycyEXd4/nbp+uZsWon728sZNXkb3l4TA8GdYgNdIjG1Ctl/nxT1ZPALwEbs8LUKM0bN2Dytb35760DaBEhbNyTyzUvLWLS28vZeehYoMMzpt7wtR1gDnBedQZiTEWd0zmOR4Y25M4LutAgNIiZq3dx3lMpTJ6XzvGTpwIdnjF1nq99JPOAv4vImcAy4KjnQlWdXtWBGVMeYcHCb5M7M7Zfa/4+az2frt7FP+duZNrS7Tx4cTcu6t7CBoI0ppr4mkiec//eUcwyBYKrJhxjKqdVk4Y8f10fbhi4n79+spYNu48w8c1lDO0Uy18u7U4Xu4mWMVXOp6YtVQ0qZbIkYmqcwR1jmfmbs3lkTHeiG4ayIGM/o579lj9/vIb9ufllb8AY4zM7V9LUWSHBQdw4OJGUu5O5cVA7VJU3Fm4l+YkU/p2SYf0nxlQRnxOJOwLwdSJyn4j82XMqxzZGikiaiGSIyH3FLBcRmewuXy0ifcqx7t0ioiLSzNd4TP3QNDKMRy7rwWe/HUZyUhxH8gt4/PM0znsyhenLs2yoemMqyac+EhEZBHyKc8fEOGAHkOA+zwQe9mEbwcDzwAggC0gVkRmqus6j2CigszsNBF4ABpa1roi0cZdt82V/TP2UFN+I18YPYH76Ph6dtZ71uw5z17RVvDp/Cw+MPoMhnew3iDEV4esRyRPAW0Ar4DjOqcBtgaXAP3zcxgAgQ1U3q+oJ4F1gjFeZMcAb6lgENBGRBB/WfRq4Bxve3vjg7M7NmPmbs3nyyl7EN27A2p2Hue6VxYz/zxLSdh8JdHjG1Driy6B3IpID9FfVjSJyCBisqutFpD/wtqp29mEbY4GRqnqb+/xGYKCqTvIoMxN4TFXnu8/nAfcCiSWtKyI/A85X1d+KSCbQT1X3FfP6E4AJAHFxcX2nTZtW5n4HQm5uLlFRNXNszLoYW/4pZU7mST7dfJLjp5yrbge1DObyTmE0j6iaLsS6WG/+YLFVTHXGNnz48GWq2s97vq+n/57weLwHaAesxxlavqWP2yjuJH7vLFZSmWLni0gEzsjEF5b14qr6Es6NuUhKStLk5OSyVgmIlJQULLbyq0xsFwH35+YzeV467yzZxsKdp0jdfZyr+7fhN+d1Jj66QcBiq24WW8VYbD/m60+u5UB/93EK8DcRGQdMBnwd/TcLaOPxvDXO3RZ9KVPS/I5Ae2CVezTSGlguIvE+xmQMAM2iwnl4TA++/H0yY/u2plCVtxZv49wnvuLRT9dx4OiJsjdiTD1VnvuRFH3pPwjsBf4FNMVtLvJBKtBZRNqLSBhwDTDDq8wM4Cb37K1BQI6q7ippXVX9XlWbq2qiqibiJJw+qrrbx5iM+ZE2MRE8eWUv5tw5jNE948kvKOTlb7cw7PGveHruRo4cPxnoEI2pcXxq2lLVpR6P9+KcXVUuqlogIpOA2ThXwk9V1bUiMtFdPgWYBYwGMoA8YHxp65Y3BmN81al5I/59fV/W7MjhyTlppKTt5dl56by+MJNbh7Zn3NBEGjcIDXSYxtQIvvaRACAi/XCak2aq6lERiQTyVdWn+56q6iycZOE5b4rHYwV+7eu6xZRJ9CUOY3zVo1U0r40fwJItB3hydhpLMg/w1NyNvPztZm45uz3jh7YnuqElFFO/+dS0JSItRGQxsAR4G2jhLvon8FQ1xWZMjTGgfQzv/WIQb982kIHtYzh8vIBnvkjn7Me+5J9z0jiUZ30opv7ytY/kaWA3EIvT5FTkfXw4Y8qYukBEGNKpGe/9YjDvThjEkI6xHMkvYPKXGQx97Ese/3yDdcqbesnXpq3zca7VOOg1FPcmnAsTjalXBnWIZVCHWFIzDzB5Xjrfpu/j3ymbeO27TK4b0JZbz2lPQnTDQIdpjF/4ekTSkB9fS1IkDudKd2Pqpf6JMfz31oFM/9UQhifFkXfiFK/Md87y+sP7q8jItivlTd3nayL5BrjZ47m641/di3PTK2PqtT5tm/Kf8QP4ZNLZXHxmAqcKlfeXZXHBP7/h2eXHWbb1QKBDNKba+Nq0dQ/wtTskSjhOB3t3IBoYWk2xGVPr9GwdzfPX9SFz31Fe/nYz7y/LYkX2Ka54YSH9E5sy8dyODE9qTlCQ3a3R1B2+3thqHdAT+A7n/u0NcDrae6vqpuoLz5jaKbFZJI9e3pMF957HJR1CadwghNTMg9z6+lJGPvsN01K32/1QTJ3h86h0qrpbVf+iqpeo6mhVfRAIE5GaOfqhMTVAXKNwxnYJ47v7z+eB0WfQonE4G/fkcs8Hqxnqnjqcfdi6GU3tVtnhTZsAV1RBHMbUaVHhIdw+rAPf3nMeT13Zi+4tG7P/6Ann1OF/fMld763k+6ycQIdpTIWU68p2Y0zlhIUEcUXf1vy8TytSMw8ydf4W5qzbzfQVO5i+Ygf9E5tyy9D2jOjWgpBguxO2qR0skRgTACLCgPYxDGgfw/YDebz+XSbvpW4nNfMgqZkHadWkIdcPastV/drQLCo80OEaUyr7yWNMgLWJieDBS7qx8I/n89Cl3WgXG8GOQ8d4/PM0Bv/fPO54ZwWpmQfw5SZ0xgRCqUckIuI9zLu3xlUYizH1WlR4CDcPbc+NgxP5ZuNe3ly0lS/TspmxaiczVu0kqUUjbhjUlst6t6KRjTxsapCymrb2+7B8SxXFYowBgoOE4V2bM7xrc7IO5vHOkm28l7qdtD1H+NPHa3nssw2M6d2KGwa2o1tL+y1nAq/URKKq4/0ViDHmp1o3jeAPF3Xlt+d3Yfba3fx30VaWbDnA24u38fbibfRq04Sr+7Xh0l4JdpRiAsY6242pBcJCgri0V0su7dWSjXuO8NairUxfvoNV2w+xavshHp65ltE9E7i6XxsGtI/Ba3BVY6qVJRJjapkuLRrx1zE9uG/UGXy2ZhfTlm5n0eYDTF++g+nLd5AYG8GV/dowtm9rWjRuEOhwTT1gicSYWqphWDA/79Oan/dpTea+o7y/bDv/W5ZF5v48npidxlNz0khOas5V/VozvGtzwkOCAx2yqaMskRhTByQ2i+QPF3XlrhFJfLNxL++lbueL9Xv4ckM2X27IpnGDEC4+syWX925Fv3ZNbdBIU6UskRhTh3ie8bU/N58PVzjNXet2HeadJdt4Z8k2WjdtyGVnteKy3q0CHa6pIyyRGFNHxUaFc9s5HbjtnA6k7T7Chyt28PHKHWQdPMZzX2Xw3FcZtG8cxOaQLVzaqyVxjewKelMxlkiMqQeS4htx36iu3HNREou3HOCjFTuY9f0uthwu4OGZ63h01nqGdIzlkjMTuKh7PE0iwgIdsqlFLJEYU48EBQmDO8YyuGMsfx3TnckffMXG/CakpGXzbfo+vk3fxwMfruHszs24uGcCF3aPJ7qhXZ9iSmeJxJh6qkFoMAPiQ7gnuR8Hj55gzrrdzFy9i+827SclbS8paXv544ffM6xzHBefmcCIbi3sokdTLEskxhiaRoZxdf+2XN2/Lftz8/l87W4+Xb2LRZv3M29DNvM2ZBMWEsS5XeIY3TOe85JaEB1hScU4LJEYY34kNiqc6we24/qB7dh7JJ/P1+xi5updLMk8wNx1e5i7bg8hQcKgDrFc2L0FI7q1ICG6YaDDNgFkicQYU6K4RuHcODiRGwcnkn34OJ+t2c3stbtZvOUA8zP2MT9jH3/+eC29WkdzYfd4Lurego5xUTZESz1jicQY45PmjRswbkgi44YkcvDoCb7ckM2cdbv5euNeVmXlsCorhydmp9GhWSQXdo9nRLcWnNWmCcF28WOdZ4nEGFNuTSPDuKJva67o25pjJ07xbfpeZq/dw7wNe9i87yhTvt7ElK83ERMZxrld4hjetTnndo6zfpU6yhKJMaZSGoYFc2H3eC7sHk/BqUJSMw8ye+1u5m3Yw/YDx/hwxQ4+XLGD4CChb9umJHeN47yuzUlq0ciawOoIvyYSERkJPAsEA6+o6mNey8VdPhrIA25W1eWlrSsiTwCXAieATcB4VT3klx0yxvxISHDQ6etU/nJpNzbtzeWrDXv5ckM2qZkHWOJOj3+eRqsmDUlOcpLK4I6xRITZ79raym//OREJBp4HRgBZQKqIzFDVdR7FRgGd3Wkg8AIwsIx15wL3q2qBiPwDuB+411/7ZYwpnojQqXkjOjVvxO3DOnD4+Enmp+/jyw3ZpKRls+PQMd5avI23Fm8jLDiIvu2acnbnZgzrHEd3u/NjreLPnwADgAxV3QwgIu8CYwDPRDIGeENVFVgkIk1EJAFILGldVZ3jsf4iYGy174kxptwaNwhldM8ERvdMoLBQ+X5HDl+lZfPVhmxW78hh4eb9LNy8nydmp9E0IpTOjQvZE7mNszvH0aqJnV5ck/kzkbQCtns8z8I56iirTCsf1wW4BXivuBcXkQnABIC4uDhSUlLKEbr/5ObmWmwVYLFVTKBjOysEzuoBuV0iWHfgFGv2nWLtvlPszzvJkjxY8sH3AMRHCt1jg+nRLJiuMcE0DAls30qg6600gYjNn4mkuP+8+limzHVF5AGgAHiruBdX1ZeAlwCSkpI0OTm5jHADIyUlBYut/Cy2iqlJsV3i/lVVMvfn8eqn37FHmrJw0352Hy1g99EC5m0rIEigZ6toBnWMZVCHWPonxhAV7t/+lZpUb94CEZs/az8LaOPxvDWw08cyYaWtKyLjcN6H57vNYsaYWkpEaN8skgvahZKc3I+TpwpZtf0Q36TvY376Xla716ysysrhxa83ExwkTmLpEMugDjH0T4wh0s+Jpb7zZ22nAp1FpD2wA7gGuM6rzAxgktsHMhDIUdVdIrK3pHXds7nuBc5V1Tz/7Ioxxl9Cg4PolxhDv8QY7hrRhaP5BSzdepBFm/ezcNN+vt+Rw8rth1i5/RBTvt5EcJBwZuuixBJLn7ZNbLDJaua3ROKeVTUJmI1zCu9UVV0rIhPd5VOAWTin/mbgnP47vrR13U0/B4QDc91z0hep6kR/7Zcxxr8iw0M4t0sc53aJAyA3v4ClmQdYtPkACzfvZ82OHFZsO8SKbYd4IWUTQQJJ8Y3p164p/RKb0i8xxjrvq5hfj/9UdRZOsvCcN8XjsQK/9nVdd36nKg7TGFOLRIWHkJzUnOSk5gAcOX7y9BHL4s0HWLMjh/W7DrN+12H+u2grAC2jG9A3MeZ0cuka39iGcqkEa0g0xtQpjRqEMjypOcPdxHLsxClWZR1i2daDLM08wLKtB9mZc5ydq3byySqnqzUqPITebZvQt11TzmrThF6tm9A00u4S6StLJMaYOq1hWPDp/hKAwkIlPTuXpVsPsCzzIEu3HmTbgbzTd4gskhgb4SSVNk04q00TurVsTHhIcKB2o0azRGKMqVeCgoSk+EYkxTfi+oHtAMg+fJylWw+yfOtBVmUd4vsdOWTuzyNzfx4frXSOWkKDhW4JjenVpgnhuSdpszeX9rGRBFmTmCUSY4xp3rjB6avuAU6eKmTjniPO2WDbDrEq6xDp2bmnTzsGePn7r2ncIIQeraJ/mFo2JrEeJhdLJMYY4yU0OIjuLaPp3jL69FHLkeMnT59qPG9FBlnHQthzOJ/vNu3nu037T68bGRbsrNuqMT1aOgmmY1wkIcFBgdqdameJxBhjfNCoQShDOjZjSMdmdCOL5ORkduccZ82OHNbszGHNjsOs3ZnDrpzjp0c5LhIeEsQZCY3p0aox3VtGc0ZCY7q0iKozIx7Xjb0wxpgAiI9uQHx0Ay7o1uL0vH25+azdeZg1O3JY6yaYbQfyTl80WUQE2sVE0DW+MV0TGjl/4xvRNiai1jWNWSIxxpgq1Cwq/EcXTALk5J1k7a4c1u44zJqdOaTtPkJGdu7pDv3P1+4+XTYiLJguLRrRNd6dEpwE0ySi5p6ObInEGGOqWXTED81iRU4UFLJpby5pu4+wfvdhNuw6wobdh9lzOP8nRy8ALRqH06l5FJ2bN6Jj8yg6N4+iU/MoYiPDAn6nSUskxhgTAGFuv8kZCY25jFan5x88eoINu52ksmHXETbsOUKam2D2HM5nQcb+H22naUQonZpHuTcRi+LYvgKSco4R37iB3xKMJRJjjKlBmkaGnb5dcZFThcqOg8fI2HuE9D25pGfnkuFOB/NOkpp5kNTMg6fLP7n0S6LCQ+jYPIqOcZF0aBZJYrNI2rtTVXfyWyIxxpgaLjhIaBsbQdvYCM7r+kPHvqqy53A+6dlOn0t6di7LNmax90QIB46eYNX2Q6zyaiIDp5msvUdiad8sivbNImgTE1Ghq/ctkRhjTC0lIqfPHDuns9O5n5Kyn+TkZPbn5pORncvmfUfZ4jFt2593upls0eYDP9pekEDrphEkNnOOYtrFRpAYG0nb2AhaNy15xGRLJMYYUwfFRoUTGxXOwA6xP5p/qlDZeegYm/cdJdNNLkWPsw7mse2AM32zce+P1ivtjGRLJMYYU48EBwltYpxmLM9TlAHyC06x/UAeW/blsWWfc3rytv15bD1wlB0Hj5W4TUskxhhjAAgPCXbP/moEtPjRshMFhYQ/Vvx6dXfwF2OMMVUmLKTkdGGJxBhjTKVYIjHGGFMplkiMMcZUiiUSY4wxlWKJxBhjTKVYIjHGGFMplkiMMcZUiiUSY4wxlWKJxBhjTKVYIjHGGFMplkiMMcZUiiUSY4wxlWKJxBhjTKVYIjHGGFMpfk0kIjJSRNJEJENE7itmuYjIZHf5ahHpU9a6IhIjInNFJN3929Rf+2OMMcaPiUREgoHngVFAN+BaEenmVWwU0NmdJgAv+LDufcA8Ve0MzHOfG2OM8RN/HpEMADJUdbOqngDeBcZ4lRkDvKGORUATEUkoY90xwOvu49eBy6p5P4wxxnjw5612WwHbPZ5nAQN9KNOqjHVbqOouAFXdJSLNi3txEZmAc5QDkC8iayqyE37QDNgX6CBKYLFVjMVWMRZbxVRnbO2Km+nPRCLFzFMfy/iybqlU9SXgJQARWaqq/cqzvr9YbBVjsVWMxVYxFtuP+bNpKwto4/G8NbDTxzKlrbvHbf7C/ZtdhTEbY4wpgz8TSSrQWUTai0gYcA0ww6vMDOAm9+ytQUCO22xV2rozgHHu43HAx9W9I8YYY37gt6YtVS0QkUnAbCAYmKqqa0Vkort8CjALGA1kAHnA+NLWdTf9GDBNRG4FtgFX+hDOS1W3Z1XOYqsYi61iLLaKsdg8iGq5uhqMMcaYH7Er240xxlSKJRJjjDGVUq8SSVlDtFTj62aKyPcislJElrrzShzaRUTud2NME5GLPOb3dbeT4Q4lU9xp0WXFMlVEsj2vo6nKWEQkXETec+cvFpHESsb2kIjscOtupYiMDlBsbUTkKxFZLyJrReS3NaXuSoktoHUnIg1EZImIrHLj+msNqrOSYqsR7zd3/WARWSEiM2tKvZVIVevFhNNJvwnoAIQBq4BufnrtTKCZ17zHgfvcx/cB/3Afd3NjCwfauzEHu8uWAINxrqv5DBhVgViGAX2ANdURC/ArYIr7+BrgvUrG9hBwdzFl/R1bAtDHfdwI2OjGEPC6KyW2gNadu40o93EosBgYVEPqrKTYasT7zV3nLuBtYGZN+pwWG2tlVq5Nk1uZsz2e3w/c76fXzuSniSQNSHAfJwBpxcWFc6baYLfMBo/51wIvVjCeRH78ZV1lsRSVcR+H4FxhK5WIraQPtt9j83r9j4ERNanuiomtxtQdEAEsxxmRokbVmVdsNaLOcK6Vmwecxw+JpEbVm+dUn5q2Shp+xR8UmCMiy8QZqgW8hnYBioZ2KW2YmKxi5leFqozl9DqqWgDkALGVjG+SOKNBT/U4nA9YbG4zQG+cX7E1qu68YoMA153bPLMS50LhuapaY+qshNigZrzfngHuAQo95tWIeitOfUoklR5mpRKGqmofnNGLfy0iw0opW23DxFRARWKp6jhfADoCZwG7gKcCGZuIRAEfAL9T1cOlFfV3fMXEFvC6U9VTqnoWzi/sASLSo7Rd8FdcpcQW8DoTkUuAbFVdVlZZf8dWkvqUSHwZoqVaqOpO92828CHOaMYlDe1S2jAxrYuZXxWqMpbT64hICBANHKhoYKq6x/3AFwIv49RdQGITkVCcL+q3VHW6O7tG1F1xsdWkulPVQ0AKMJIaUmfFxVZD6mwo8DMRycQZ6fw8EXmTGlZvnupTIvFliJYqJyKRItKo6DFwIbCGkod2mQFc455V0R7n3ixL3EPZIyIyyD3z4iaqbjiYqozFc1tjgS/VbYitiKIPjutynLrze2zutl4F1qvqPz0WBbzuSoot0HUnInEi0sR93BC4ANhAzaizYmMLdJ0BqOr9qtpaVRNxvqe+VNUbqAH1VlrQ9WbCGX5lI85ZDQ/46TU74JxRsQpYW/S6OO2R84B092+MxzoPuDGm4XFmFtAP5429CXiOinXEvoNzyH4S51fJrVUZC9AAeB9nmJslQIdKxvZf4HtgtfvmTwhQbGfjHPqvBla60+iaUHelxBbQugPOBFa4r78G+HNVv/crUWclxVYj3m8e207mh872gNdbSZMNkWKMMaZS6lPTljHGmGpgicQYY0ylWCIxxhhTKZZIjDHGVIolEmOMMZViicSYWkxEbhaR3EDHYeo3SyTGVAEReU1E1GPaJyIzRaRrObbxkHgMoW9MbWGJxJiq8wXOiKsJOCMYNMQZEseYOs0SiTFVJ19Vd7vTcuBpoKs7BAci8pg4Nx46Js7Nzh4XkQbuspuBvwDdPY5qbnaXNRaRF0Rkl4gcF+cGVld7vrCInC8ia0TkqDg3uWrvzx039VtIoAMwpi5yx1e7GvheVY+5s48CtwA7cG5GNAXIB/4EvAf0AC7BGRYDIMcdI+kzoCkwHmeInyScIS6KhOPck+IW4DjwurvtizDGDyyRGFN1Rnp0fEfi3O/h9K1aVfURj7KZIvJ34G7gT6p6zF23QFV3FxUSkRE4Nynqrqrr3dmbvV43BPi1qqa56zwJ/EdEgtQZxdaYamVNW8ZUnW9w7mNxFs7d9r7EuaFZ0XDdY0VkvojsdpPG00DbMrbZG9jlkUSKk1+URFw7cW4f26QiO2FMeVkiMabq5KlqhjstwRm9uDEwQUQG4dxbYjZwKU6CeBDnC780xd2AyFuB1/OikVjt8238wpq2jKk+inOr1AicmxXt8GzeEpF2XuVPAMFe85YDCSJyRhlHJcYEjCUSY6pOuIjEu4+bApOAKOAToBHQSkSuBxbidIRf67V+JtBORPoA24AjOPedWAx8ICJ34nS2dwIiVfWjat0bY3xkh77GVJ0LcG7MtQvny78/cKWqpqjqJ8ATwDM4N00aAfzZa/0PgFk4yWMvcK3bWT4KWAC8CawHngXCqntnjPGV3djKGGNMpdgRiTHGmEqxRGKMMaZSLJEYY4ypFEskxhhjKsUSiTHGmEqxRGKMMaZSLJEYY4ypFEskxhhjKuX/AandjbShcG4/AAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"n_steps = n_epochs * math.ceil(len(X_train) / batch_size)\n",
"steps = np.arange(n_steps)\n",
"decay_rate = 0.1\n",
"lrs = lr0 * decay_rate ** (steps / n_steps)\n",
"\n",
"plt.plot(steps, lrs, \"-\", linewidth=2)\n",
"plt.axis([0, n_steps - 1, 0, lr0 * 1.1])\n",
"plt.xlabel(\"Batch\")\n",
"plt.ylabel(\"Learning Rate\")\n",
"plt.title(\"Exponential Scheduling (per batch)\", fontsize=14)\n",
"plt.grid(True)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Piecewise Constant Scheduling"
]
},
{
"cell_type": "code",
"execution_count": 79,
"metadata": {},
"outputs": [],
"source": [
"def piecewise_constant_fn(epoch):\n",
" if epoch < 5:\n",
" return 0.01\n",
" elif epoch < 15:\n",
" return 0.005\n",
" else:\n",
" return 0.001"
]
},
{
"cell_type": "code",
"execution_count": 80,
"metadata": {},
"outputs": [],
"source": [
"# extra code – this cell demonstrates a more general way to define\n",
"# piecewise constant scheduling.\n",
"\n",
"def piecewise_constant(boundaries, values):\n",
" boundaries = np.array([0] + boundaries)\n",
" values = np.array(values)\n",
" def piecewise_constant_fn(epoch):\n",
" return values[(boundaries > epoch).argmax() - 1]\n",
" return piecewise_constant_fn\n",
"\n",
"piecewise_constant_fn = piecewise_constant([5, 15], [0.01, 0.005, 0.001])"
]
},
{
"cell_type": "code",
"execution_count": 81,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/25\n",
"1719/1719 [==============================] - 3s 1ms/step - loss: 0.5745 - accuracy: 0.7963 - val_loss: 0.4856 - val_accuracy: 0.8256\n",
"Epoch 2/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.4472 - accuracy: 0.8424 - val_loss: 0.4418 - val_accuracy: 0.8372\n",
"Epoch 3/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.4216 - accuracy: 0.8505 - val_loss: 0.4162 - val_accuracy: 0.8588\n",
"Epoch 4/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.4104 - accuracy: 0.8569 - val_loss: 0.4027 - val_accuracy: 0.8592\n",
"Epoch 5/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3949 - accuracy: 0.8614 - val_loss: 0.4276 - val_accuracy: 0.8510\n",
"Epoch 6/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3483 - accuracy: 0.8747 - val_loss: 0.3907 - val_accuracy: 0.8676\n",
"Epoch 7/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3336 - accuracy: 0.8783 - val_loss: 0.3981 - val_accuracy: 0.8628\n",
"Epoch 8/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3261 - accuracy: 0.8815 - val_loss: 0.4098 - val_accuracy: 0.8574\n",
"Epoch 9/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3241 - accuracy: 0.8818 - val_loss: 0.4197 - val_accuracy: 0.8514\n",
"Epoch 10/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3183 - accuracy: 0.8833 - val_loss: 0.3668 - val_accuracy: 0.8736\n",
"Epoch 11/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3147 - accuracy: 0.8856 - val_loss: 0.3936 - val_accuracy: 0.8698\n",
"Epoch 12/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3111 - accuracy: 0.8864 - val_loss: 0.3854 - val_accuracy: 0.8702\n",
"Epoch 13/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3117 - accuracy: 0.8868 - val_loss: 0.4126 - val_accuracy: 0.8718\n",
"Epoch 14/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3052 - accuracy: 0.8896 - val_loss: 0.3997 - val_accuracy: 0.8722\n",
"Epoch 15/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.3018 - accuracy: 0.8905 - val_loss: 0.4556 - val_accuracy: 0.8678\n",
"Epoch 16/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2640 - accuracy: 0.9022 - val_loss: 0.4124 - val_accuracy: 0.8782\n",
"Epoch 17/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2567 - accuracy: 0.9040 - val_loss: 0.4121 - val_accuracy: 0.8804\n",
"Epoch 18/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2525 - accuracy: 0.9053 - val_loss: 0.4173 - val_accuracy: 0.8776\n",
"Epoch 19/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2497 - accuracy: 0.9065 - val_loss: 0.4279 - val_accuracy: 0.8816\n",
"Epoch 20/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2478 - accuracy: 0.9066 - val_loss: 0.4330 - val_accuracy: 0.8774\n",
"Epoch 21/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2452 - accuracy: 0.9086 - val_loss: 0.4400 - val_accuracy: 0.8816\n",
"Epoch 22/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2430 - accuracy: 0.9099 - val_loss: 0.4522 - val_accuracy: 0.8796\n",
"Epoch 23/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2405 - accuracy: 0.9096 - val_loss: 0.4538 - val_accuracy: 0.8812\n",
"Epoch 24/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2390 - accuracy: 0.9102 - val_loss: 0.4929 - val_accuracy: 0.8830\n",
"Epoch 25/25\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.2377 - accuracy: 0.9105 - val_loss: 0.4720 - val_accuracy: 0.8802\n"
]
}
],
"source": [
"# extra code – use a tf.keras.callbacks.LearningRateScheduler like earlier\n",
"\n",
"n_epochs = 25\n",
"\n",
"lr_scheduler = tf.keras.callbacks.LearningRateScheduler(piecewise_constant_fn)\n",
"\n",
"model = build_model()\n",
"optimizer = tf.keras.optimizers.Nadam(learning_rate=lr0)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
" metrics=[\"accuracy\"])\n",
"history = model.fit(X_train, y_train, epochs=n_epochs,\n",
" validation_data=(X_valid, y_valid),\n",
" callbacks=[lr_scheduler])"
]
},
{
"cell_type": "code",
"execution_count": 82,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEbCAYAAADwPQLqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAp40lEQVR4nO3deZxcVZ338c+3O1unCXToBDp0oDtK7ACCgAgog0ZRWXQGXPCBEVlcIj4woz6Kwui44DDgNo4oghnNACMKOIMYNYqCtowKympYIwECZCELmJClyfp7/ri3k0qlquumq6sqXfV9v1716rrLufWr09X163vPuecoIjAzMxusploHYGZmw5sTiZmZlcWJxMzMyuJEYmZmZXEiMTOzsjiRmJlZWZxIGpyksyWtqXUcA5EUkt5Z6zgsG0lXS/ppBY47If0sTN+JMt1pmSMKLdvQcCKpc+kfdaSPjZKekPQVSa3pLjcAL6lljBlMAn5SyReQNE7SFyQ9LKlP0lJJvZJOl1SVv5NKfsntzLElvU7SbZJWSFon6XFJ10nafajjqoFnSD5P99c4jroyotYBWFXcCrwHGAkcC3wHaAU+FBF9QF8NYyspIp6t5PEltQG/A8YDnwb+BGwA/gb4Z+AOYEElY9hVSDoQ+AVwFfARYC2wP3AKMLpmgQ2RiNgMVPTz1JAiwo86fgBXAz/NW/cfwJL0+dnAmrztfwvcA7wIPAlcAozK2T4K+FfgKWA98ATwjznbDwR+BqwGlgE/ADrSbQcAkbM8luRL++c55T8APJazHMA7c5Y/k/PazwLX5mwT8AngcZIE+QBwRok6+hbJF+bkAtvGAGPS5+OBa4C/pse+FTgoZ9+zgTXAccCD6TF/A0zJ2Wdf4MfA88A64FHgtJz3mfvoTde/CvglsAJ4gSTpvTovzgBmAD9MX/eJ3Pdd7NgF3u9HgIUZPlfTgNnAqvQ93wEcnPuZAz4MLErr6z+BsTvze0rfd//n8D7gLWns09Pt09PlCTllutN1R2Rc7j/GccAf09/J3cDhebG8F3g63f4T4P8CUeu/713l4UtbjamP5OxkB5KOB64DvgkcRPIH9E6SxNHvGuBM4P+RJIb3ASvT8pOA20m+SI8E3gjsBsyW1BQRjwBLSf6AAY4h+TL6G0n9Z8jTgd4i8b0D+DjJH/JU4K0kZxD9/iWN5zyShHYp8G1JbylyvCbgNOC6iFiYvz0iXoyIF9PFq4GjgJPT97YO+IWklpwio4GLSOrt1UAbyX/3/b5FkjxfT1K/HyGtu/SYACeQXH55e7o8DvgvkrPJI0kuy8yRNCEv3M+QJKlXkFyynCWpq8Sx8z0LTJT0+iLbkbQPSTIL4E3A4cAVQHPObscCLyf5/f8f4G0kiaXfgL+n9NLrz0gS4hHAhcBXisU0BC5NX+Nw4DngOklKY3k1yVn8FcChJAn08xWMZfipdSbzo7IP8s5ISL5QVgA3pMtnk3NGQpIE/jnvGKeQ/Ncpki/vAE4o8noXA7flrRufljkyXb4B+Hb6/BLgSpJLR69O1y0E3p1TfusZCUnymgeMLPDarSRJ8ti89f8OzCkS717p8T9aoh773/drc9btQZIE359TlwH05OzzbpIzrqZ0eS7w2SKv0U3Of8sDxCJgCTuecVyaszyCJNGdsZPHbiY5ewiShP+TtM4n5uxzCckZ4agix7iapC1iRM66/wBuzfp7Ijm7WgnslrP9DCp3RnJ8zjGOSddNTpd/APwiL9aZ+Ixk68NnJI3hBElrJL1IcgniduAfiuz7SuBT6f5r0h5d3yf54+8ADgO2kFyyKVb+tXnln0m3vTT92cu2M5Lp6bF+C0yXNBXopMgZCcmlmzHAk5K+K+lUSf3X7g9Mt/0i7/U/lPPa+VRkfb4DSN73Hf0rImIVySWZA3P2Wx8R83KWF5Oc/bWly18HPi3pDkn/IumVpV5Y0l6Svi3pL5JWkVwy3AvYL2/XuTmxbQKWp/tlFhGbI+IcYDLJmd/TwAXAo5IOSnc7DPhdRGwY4FAPpzH0W5wTS5bf0wHA3IjI7VF4B5UzN+f54vRnf7zT2P6sF5LLYJZyY3tjuJ3kP7yNwOKI2DjAvk0kp+0/LLBtOaW/eJtILkl8vMC2penPXuBbadI4Il1uBU4nOVuaHxGLCh08Ip6R1ENyTfuNwFeBz0o6im29EP+W5AswV7H3vJzkGv4BJd7XQO87dwjtTUW2NQFExHcl3QKcRBL/HyRdGhGfG+D41wB7Ax8lOXNbD9xG0laVK/89BoPsmZnW/38B/yXp08BfSBLK2WRLvgPFkuX3lOU1thTYt+Al2wxy493ud5YeP7CifEbSGNZFxPyIeKpEEgG4F5iW7p//2JRubyK5xl+s/EHAUwXKrwaIbe0knyJJGstIzkqOIbnm3jtQgJG0W/wsIj5K0iB7UFr2YZIv2a4Cr/1UkWNtIbnU9m5Jk/O3SxojaUx67CaSdo/+bbsDB6fbMouIhRExMyLeRdKuMSPd1P8ffnNekb8BvpG+54dIzkgm7cxrDnDsLPH+leRS2m7pqntJ2rTyE1lWWX5PDwMH53RTBzg67zjL05+5dXHoIGMayCNsa2Pql7/c0JxILN/FwN9LuljSyyVNk/ROSV8CiIjHgBuB70h6h6Qpko6V9J60/BUkbQc3SDpK0kskvVHSTEnjcl7ntyTXvH+THncByRfD2xkgkaQ3UL5f0sGSpgDnkPw3+ViaqL4CfEXSeyXtL+lQSedKmlHsmMA/kfxn/EdJ50g6KC37HpJeQx3p+/4xSYPwsZIOBr5H0ovq+xnrFklfl3RCWi+HkjR+9yeiZSRtB8dL2lvSHun6vwBnSDpQ0quA69mWGLIqduz8+D4o6UpJb5b00rQuvkiSMG9Od/sWSVK5UdKr0ro6PX0/JWX8PX2f5OxuVhrDm0j+8cg1n+Sy6eckvUzSm0m6bw+1y4E3S7pA0lRJ7yPpPGD9at1I40dlHxTo/pu3/Wx27P77ZuB/SRprXyDpDnl+zvbRwJdIunauJ+nCmbt9KvDfbOsmOw/4Btt3IT6XHbv1Xp2u68yLJ7ex/RSSa+UrSbq53gW8NWdfkbT/9P/Xuxz4FfCmEvW0B0kj8qMk3U2XkSS009jWUJ6p+2/ecaeT0yCc1sNj6WssJ0kKnTn7v58kqW1mW/ffV5Bck+9L6/o9JL3iPleojnLWLQA+PtCxC9TDYel77O+W+xxwJ/CevP0OAuaQdMJYDfwBeHmxzxzwOeDBnfk9kfSQuzfd/meSS2FbG9vTfV5D0outL/1c9HcR3tnG9qIN9um695IkrT6SDggfA/pq/fe9qzyUVpKZmWUk6WvAGyPi4FrHsitwY7uZWQmSLiA5Y1pD0kniXJJLogY+IzEzK0XSDSSXwfYgGe3h28DXw1+ggBOJmZmVyb22zMysLA3ZRtLW1hb7779/rcPY5axdu5bW1tbSOzYQ10lhrpfC6r1e7rnnnhURMTF/fUMmkr333pu777671mHscnp7e5k+fXqtw9iluE4Kc70UVu/1Iqngjb2+tGVmZmVxIjEzs7I4kZiZWVmcSMzMrCxOJGZmVhYnEjMzK4sTiZmZlcWJxMzMyuJEYmZmZXEiMTOzsjiRmJlZWZxIzMysLE4kZmZWFicSMzMrixOJmZmVparzkUg6Afg60Ax8JyIuy9uudPtJwDrg7Ii4N902C3grsCwiXp5TZk/gBqAbWAC8KyL+OlAcC17YwjGX/ZoLju/hlMM6S8Z9832L+PIt81i8so992lrqrpyZWTmqdkYiqRm4AjgROBA4XdKBebudCExNHzOAK3O2XQ2cUODQFwK3RcRU4LZ0uaRFK/u46KYHuPm+RQPud/N9i7jopgdYtLKPqMNyZmblquYZyZHA/Ih4AkDS9cDJwMM5+5wMXBsRAdwpqU3SpIhYEhG3S+oucNyTgenp82uAXuCTWQLq27iZf/rRA/xu/oqi+8x5YAl9GzcP23JfvmWez0rMrKKqmUg6gWdylhcCR2XYpxNYMsBx946IJQARsUTSXoV2kjSD5CyHUR3b5mtft2Ezv3mo+H/t6zZEkfXDo9yilX309vYWLZdrzZo1mfdtFK6TwlwvhTVqvVQzkajAuvxvvyz7DEpEzARmAoyeNHXrMTvbWvj9hW8oWu6Yy37NopV9O6wfTuWyziFd7/NND4brpDDXS2GNWi/V7LW1ENg3Z3kysHgQ++RbKmkSQPpzWdaAWkY2c8HxPQPuc8HxPbSMbK7bcmZm5apmIrkLmCppiqRRwGnA7Lx9ZgNnKnE0sKr/stUAZgNnpc/PAn6cJZjOthYuffvBJdsPTjmsk0vffjCdbS1oGJTbbXRykrlP25hM5czMylW1S1sRsUnS+cAtJN1/Z0XEQ5LOTbdfBcwh6fo7n6T77zn95SX9gKRRfYKkhcBnI+K7wGXAjZLeBzwNnFoqlu7dmwa8TJTvlMM6B/WFXItyfRs3c9FND3DjB1/N5PFjd/oYZmY7q6r3kUTEHJJkkbvuqpznAZxXpOzpRdY/Bxw3hGEOa13tSfJ46rl1TiRmVhW+s73OTJnQCsCTK9bWOBIzaxROJHVm73FjGD2iiaeecyIxs+pwIqkzTU2iq30sC55bV+tQzKxBOJHUoa72Vp+RmFnVOJHUoe72sTz13Dq2bBmSeznNzAbkRFKHutpbWb9pC0tXv1jrUMysATiR1KHu9qTn1oIVbicxs8pzIqlD/feSLHA7iZlVgRNJHdqnrYVRzU1OJGZWFU4kdai5Sey7ZwtP+dKWmVWBE0md6m5v9RmJmVWFE0mdSu4lWUcyfJmZWeU4kdSp7glj6du4meWr19c6FDOrc04kdaqrvwuwh0oxswpzIqlT3e4CbGZV4kRSpzrbWhjRJBZ4OHkzqzAnkjo1ormJffdMxtwyM6skJ5I6lgwn7zMSM6ssJ5I61u0uwGZWBU4kdayrfSxr1m/iubUbah2KmdUxJ5I61j8KsCe5MrNKciKpY1tHAfaYW2ZWQU4kdWzy+LE0yfeSmFllOZHUsVEjmpg8fqzvbjezinIiqXNd7WPdRmJmFeVEUue621t5csVadwE2s4pxIqlzXe1jWf3iJlau21jrUMysTjmR1LnuraMA+/KWmVWGE0md656QdAH2mFtmVilOJHVu8vixyF2AzayCnEjq3JiRzeyzR4uHkzeziqlqIpF0gqR5kuZLurDAdkm6PN0+V9LhpcpKOlTSnZLul3S3pCOr9X6Gi+4JvpfEzCqnaolEUjNwBXAicCBwuqQD83Y7EZiaPmYAV2Yo+yXg8xFxKPCZdNlydLW3+l4SM6uYap6RHAnMj4gnImIDcD1wct4+JwPXRuJOoE3SpBJlA9g9fb4HsLjSb2S46W4fy1/XbWSVuwCbWQWMqOJrdQLP5CwvBI7KsE9nibIfAW6R9BWSxPiaQi8uaQbJWQ4TJ06kt7d3MO9hWFq9dBMAN/3qdqbs0Vx0vzVr1jRUvWThOinM9VJYo9ZLNROJCqzLv9262D4Dlf0Q8NGI+B9J7wK+C7xxh50jZgIzAXp6emL69OkZwx7+Jj27mm/cdzvt3Qcw/RX7FN2vt7eXRqqXLFwnhbleCmvUeqnmpa2FwL45y5PZ8TJUsX0GKnsWcFP6/Ickl8Esx357pveSuOeWmVVA5kQi6URJP5X0sKR903Xvl3RcxkPcBUyVNEXSKOA0YHbePrOBM9PeW0cDqyJiSYmyi4HXpc/fADyW9T01ipZRzXTsPoYn3eBuZhWQ6dKWpHcDVwHfAY4DRqabmoFPALeVOkZEbJJ0PnBLWm5WRDwk6dx0+1XAHOAkYD6wDjhnoLLpoT8AfF3SCOBF0nYQ2173hLG+u93MKiJrG8kngA9ExPWS3p+z/k7g4qwvFhFzSJJF7rqrcp4HcF7Wsun63wGvzBpDo+pub+XWR5bWOgwzq0NZL21NBe4osH4N27re2i6sq72VFWs2sPpFdwE2s6GVNZEsBl5WYP1rgceHLhyrlO52D95oZpWRNZHMBC6XdEy6vK+ks0juIr+yIpHZkOpKh5N3IjGzoZapjSQiviRpD+BXwBjgN8B64CsRcUUF47Mh0pWekXgUYDMbaplvSIyIT0m6hGSsqybg4YhYU7HIbEi1jh7BxHGjPeaWmQ25TJe2JM2SNC4i1kXE3RHxp4hYI6lV0qxKB2lDY0p7KwtW+NKWmQ2trG0kZwEtBda3AGcOXThWSV3tY31py8yG3ICJRNKektpJxroany73PyYCbwV8c8Iw0T2hlWWr17Nuw6Zah2JmdaRUG8kKksERA3i4wPYAPjvUQVlldOV0AT5gkm//MbOhUSqRvJ7kbOTXwDuA53O2bQCeigjP/zFMdG/tArzWicTMhsyAiSQifgsgaQrwTERsqUpUVhH7be0C7AZ3Mxs6We8jeQpA0j7AfsCovO23D31oNtR2HzOS9tZR7gJsZkMq6+i/+wDfJxkSpX+iqdxJqYpPu2e7lK72sTzpeUnMbAhl7f7778BmkpsR1wHHAqcCjwAnVCQyq4juCa0eJsXMhlTWRPI64JMR8SjJmcjyiLgJ+CTwhUoFZ0Ovu72VJate5MWNm2sdipnViayJpIWkKzAkPbf2Sp8/DBwy1EFZ5fR3AX76eZ+VmNnQyJpIHgWmpc/vB86V1EUyCdWiCsRlFdLfBXiB20nMbIhkHbTx60BH+vxi4BfA6SQjAJ9VgbisQro9nLyZDbGs3X+vy3l+r6RukjOUpyNiRdGCtsvZY+xI2saO9JhbZjZksl7a2k46CvC9wFpJFw5xTFZhXe3uuWVmQ6dkIpE0QdJbJL1ZUnO6bqSkjwALgI9XNkQbalN8L4mZDaFSo/++BngM+Anwc+D3kqYBc4HzSbr+7lfpIG1odbW3snhVH+s3uQuwmZWv1BnJF4BbSLr4fh04EvgpcCkwNSK+GRG+RjLMdE8YSwQ883xfrUMxszpQKpG8AvhCRDwIfJrkZsSLIuLaiIiBi9quqitnFGAzs3KVSiR7AsshaWAnGR7lvkoHZZW19V4SN7ib2RDI0v13vKRNbBuocXdJe+buEBHPFyxpu6TxY0cybswIn5GY2ZDIkkhyZ0YUcFfecuDRf4cVSXS3t/qMxMyGRJYZEq0OdbWPZe7CVbUOw8zqQKYZEq3+TJnQypwHlrBh0xZGjRjUfalmZsAg72y34a+rvZUtAYtWuguwmZWnqolE0gmS5kmaX2hoFSUuT7fPlXR4lrKS/iHd9pCkL1XjvQx33Vvnb3eDu5mVJ+vov2VLh1e5AngTsBC4S9LsiMhtzD8RmJo+jgKuBI4aqKyk1wMnA4dExHpJe2Elbb2XZMVa6KlxMGY2rFXzjORIYH5EPBERG4DrSRJArpOBayNxJ9AmaVKJsh8CLouI9QARsawab2a4m7DbKFpHNbvnlpmVrWpnJEAn8EzO8kKSs45S+3SWKPsy4FhJlwAvAh+PiNwuygBImgHMAJg4cSK9vb2DfiP1on10cO9jz9DbuxyANWvWuF7yuE4Kc70U1qj1kimRSJpVZFOQfHnPB26IiMUDHaZI+Sz7DFR2BDAeOBp4FXCjpJfkD+ESETOBmQA9PT0xffr0AUJtDC9fdA+PLllNf1309vbietme66Qw10thjVovWc9IJgLHAluAB9N1Lyf5gr8HeDtwsaRjI+L+IsdYCOybszwZyE88xfYZNUDZhcBNaeL4k6QtwATSoV2suO72Vn750FI2bd7CiGZ34DOzwcn67fF7kmHkJ0fEayPitSRf5nOAXwJdwM+Arw5wjLuAqZKmSBoFnAbMzttnNnBm2nvraGBVRCwpUfZm4A0Akl5GknQ8a2MG3e2tbNoSLF75Yq1DMbNhLOsZyYeBN+QOGR8R69J2idsi4kuSvgjcWuwAEbFJ0vkkw9I3A7Mi4iFJ56bbryJJTCeRXCpbB5wzUNn00LOAWZIeBDYAZ3lk4my6croA75c+NzPbWVkTyW7AJOCRvPUd6TaAF0odLyLmkCSL3HVX5TwP4LysZdP1G4AzBg7fCumekDuc/MTaBmNmw1bWS1s/Ar4r6VRJ3ZK6JJ0KfBe4Kd3nSOAvlQjSKmOvcaMZM7LJXYDNrCxZz0jOBf4N+F5OmU0kl5X652x/BPjAkEZnFdU/CrCHkzezcmRKJGnbyLmSPga8lKS31vyIWJuzz/0VidAqqqt9LI8vdyIxs8HbqT6fEbE2IuZGxJ9zk4gNX93trTz93Do2b3H/BDMbnKw3JI4h6bl1HLAXeQkoIg4Z+tCsGrontLJh8xaWrPIowGY2OFnbSL4FvA34IfAHdrwj3Yap/i7AT7nB3cwGKWsiOQU4NSKK3idiw1N3OgrwgufW0lnjWMxseMraRrKO7QdNtDrRsfsYRo1o8hmJmQ1a1kTyJeD/SfKATHWmqUl07TmWBSvcd8LMBifrpa03kQzaeIKkh4GNuRsj4u+GOjCrnq721uSMZL9aR2Jmw1HWRLKC5O52q0Pd7WP53fzlbIkxtQ7FzIahrDcknlPpQKx2VvZt5MWNW3jvLevo/OOvueD4Hk45rHTT+833LeLLt8xj8co+9mlrqaty/WUWreyj807XidlAqjlDou2Cbr5vEbPv3zYtzKKVfVx00wMAA36x3HzfIi666QH6Nm6uu3LDIcZalDMrRsVGXJc0F3hdRPxV0gMMcO/IcLshsaenJ+bNm1frMHYJx1z2axat3PFmxJHN4sB99iha7uHFq9i4ecePRD2UGw4xVqJcZ1sLv7/wDUXL5WrUmQBLqfd6kXRPRByRv36gM5L/Adanz/+7IlFZzS0ukEQANm4O2lpGFi1X6IuoXsoNhxgrUa7YZ8GslKKJJCI+X+i51Zd92loKnpF0trVwzXuPLFqu2JlMPZQbDjFWotw+bS1Fy5gNxPeFNLgLju+hZWTzdutaRjZzwfE9DVtuOMRYi3JmxWQdtHFP4BKKD9q4+9CHZtXQ37i6tYdSxh48ueV2pufPcCjXSHWyaGUfLSObufTtB7uh3QataGP7djtJPwIOA2YCi8lreI+IayoSXYW4sb2wem8oHIx6r5OzZv2JZavX8/MPH7tT5eq9Xgar3utlMI3tuY4D3hQRfxzasMyslqZ1jOOOx59j4+YtjGz2lW4bnKyfnGXAmkoGYmbV19Mxjg2bt3isNStL1kTyKeBiSbtVMhgzq65pHUnz5qPPrq5xJDacZb209WmgG1gm6Sl2HLRxWN2QaGaJl+7VSnOTmPfsav72FbWOxoarrInENySa1aHRI5p5yYRWHn32hVqHYsNYyUQiaSTQClwREU9VPiQzq6aejnHc/8zKWodhw1jJNpKI2Ah8CFDlwzGzapvWMY6Ff+1jzfpNtQ7Fhqmsje2/BLKN5mZmw0pP2uA+zw3uNkhZ20huA/5V0iHAPcB2fQUj4qahDszMqmNaxzggSSSv7Bpf42hsOMqaSL6Z/vzHAtsCaC6w3syGgc62FnYbPYJ5bnC3Qco6Q6JveTWrU01N4mV778YjvrRlg+QEYWb0dOzOvGdXk2XsPbN8mROJpD0l/b2kCyV9JvexE8c4QdI8SfMlXVhguyRdnm6fK+nwnSj7cUkhaULWeMwsMa1jHKv6NrL0hfWldzbLk3UY+aOBn5HMmDgRWARMSpcXABdnOEYzcAXwJmAhcJek2RHxcM5uJwJT08dRwJXAUaXKSto33fZ0lvdjZtvrSRvcH332BTr2GFPjaGy4yXpG8mXgOqATeJGkK/B+wN3AFzMe40hgfkQ8EREbgOuBk/P2ORm4NhJ3Am2SJmUo+zXgEwwwr7yZFZfbc8tsZ2XttXUI8L6ICEmbgdER8YSkTwLfJ0kypXQCz+QsLyQ56yi1T+dAZSX9HbAoIv4sFb9nUtIMYAbAxIkT6e3tzRByY1mzZo3rJU8j1cn40eK3f55PTzxTct9Gqped0aj1kjWRbMh5vhToAh4hGVp+n4zHKPQtn38GUWyfgusljSUZmfjNpV48ImaSTMxFT09P1PPkM4NV75PyDEYj1ckhTySTXE2fXnqSq0aql53RqPWS9dLWvcCr0ue9wL9IOgu4HJib8RgLgX1zlieTzLaYZZ9i618KTAH+LGlBuv5eSR0ZYzKz1LSOcTy+bA0bN2+pdSg2zOzMfCT9X/qfBpYD3wDGk14uyuAuYKqkKZJGAacBs/P2mQ2cmfbeOhpYFRFLipWNiAciYq+I6I6IbpKEc3hEPJsxJjNLeZIrG6ysNyTenfN8OUnvqp0SEZsknQ/cQnIn/KyIeEjSuen2q4A5wEnAfGAdcM5AZXc2BjMrblvPrdVM3XtcjaOx4SRrGwkAko4guZz004hYK6kVWB8RmYYNjYg5JMkid91VOc8DOC9r2QL7dGeJw8x2tP9eu3mSKxuUrPeR7E1y2elVJI3fU4EngH8j6Q784UoFaGbVsW2SK3cBtp2TtY3ka8CzQDvJJad+PyRDjykzGx56OsYxb6kHb7SdkzWRHAd8KiL+mrf+cZIbE82sDkzrGMczz3uSK9s5WRNJC9vfS9JvIsmlLTOrA57kygYjayK5HTg7ZznS8a8+STLplZnVAQ+VYoORtdfWJ4DfSnoVMBr4KnAQsAdwTIViM7Mq62xroXVUsye5sp2S6YwkHWX3YOAPJPO3jyFpaD8sIh6vXHhmVk1NTaKnY5x7btlOyXwfSXq3+Gdz10nqknRjRLxryCMzs5ro6didnz+4hIhgoIFQzfqVO0NiG/COIYjDzHYR0zrGsXKdJ7my7DzVrpltJ3eSK7MsnEjMbDvuuWU7y4nEzLbTNnYUe+8+2onEMhuwsV1S/jDv+XYfwljMbBcxrWN399yyzEr12nouw/YnhygWM9tFTOsYxx2PP8emzVsY0ewLFzawARNJRJxTrUDMbNfRP8nVkyvWem4SK8n/apjZDnInuTIrxYnEzHaQO8mVWSlOJGa2g9EjmpniSa4sIycSMytomie5soycSMysIE9yZVk5kZhZQZ7kyrJyIjGzgjxUimXlRGJmBXmSK8vKicTMCmpqEi/zJFeWgROJmRU1rWN35i1dTUTUOhTbhTmRmFlR/ZNcLVvtSa6sOCcSMyuqf6iUR5a4ncSKcyIxs6Lcc8uycCIxs6I8yZVl4URiZgPq8SRXVkJVE4mkEyTNkzRf0oUFtkvS5en2uZIOL1VW0pclPZru/yNJbVV6O2YN4YCOccxfvoZNm7fUOhTbRVUtkUhqBq4ATgQOBE6XdGDebicCU9PHDODKDGV/Bbw8Ig4B/gJcVOG3YtZQejrGsWHTFhY8t7bWodguqppnJEcC8yPiiYjYAFwPnJy3z8nAtZG4E2iTNGmgshHxy4joH1XuTmByNd6MWaPY1nPLl7essFJztg+lTuCZnOWFwFEZ9unMWBbgvcANhV5c0gySsxwmTpxIb2/vToTeGNasWeN6yeM6gY1bgibBL//4IOP++hfA9VJMo9ZLNROJCqzLv1222D4ly0r6FLAJuK7Qi0fETGAmQE9PT0yfPr1EuI2nt7cX18v2XCeJl9z/W/pGtzJ9+hGA66WYRq2XaiaShcC+OcuTgcUZ9xk1UFlJZwFvBY4Lj+VgNuR6OsYxd+HKWodhu6hqtpHcBUyVNEXSKOA0YHbePrOBM9PeW0cDqyJiyUBlJZ0AfBL4u4hYV603Y9ZIpu3tSa6suKqdkUTEJknnA7cAzcCsiHhI0rnp9quAOcBJwHxgHXDOQGXTQ38TGA38ShLAnRFxbrXel1kjmDYpmeTqL0tXc/h+42scje1qqnlpi4iYQ5IsctddlfM8gPOylk3X7z/EYZpZnv6hUh5d4kRiO/Kd7WZWkie5soE4kZhZSZ7kygbiRGJmmUzrGOdJrqwgJxIzy6Rnb09yZYU5kZhZJv09t3x5y/I5kZhZJtsmuXKDu23PicTMMumf5OpRD95oeZxIzCwzT3JlhTiRmFlm09JJrjZvcc8t28aJxMwy69k7meRq6TonEtvGicTMMps2KWlwX7ja0+7aNlUda8vMhrdHFic9tr715/X8+Klfc8HxPZxyWGfJcjfft4gv3zKPxSv72KetpW7LLVrZR+ed9Vsvozr2f2Wh7U4kZpbJzfct4p9//NDW5UUr+7jopgcABvwyuvm+RVx00wP0bdzscnVQrhA14nAHPT09MW/evFqHsctp1NndBuI62eaYy37NopV9O6wfPaKJo17SXrTcH594jvWbdrwU5nLDr9ySaz7C+iWP7TBjrc9IzCyTxQWSCMD6TVt4oW9j0XKFvrxcbviXy+VEYmaZ7NPWUvCMpLOthZvPO6ZouWJnMi43vMvlcq8tM8vkguN7aBnZvN26lpHNXHB8j8s1WLl8PiMxs0z6G2S39k7K2Osnt9zO9BYajuXqvV6WFNnHje22lRuWd+Q6Kcz1Uli914ukeyLiiPz1vrRlZmZlcSIxM7OyOJGYmVlZnEjMzKwsTiRmZlYWJxIzMyuLE4mZmZXFicTMzMriRGJmZmVxIjEzs7I4kZiZWVmcSMzMrCxVTSSSTpA0T9J8SRcW2C5Jl6fb50o6vFRZSXtK+pWkx9Kf46v1fszMrIqJRFIzcAVwInAgcLqkA/N2OxGYmj5mAFdmKHshcFtETAVuS5fNzKxKqnlGciQwPyKeiIgNwPXAyXn7nAxcG4k7gTZJk0qUPRm4Jn1+DXBKhd+HmZnlqObEVp3AMznLC4GjMuzTWaLs3hGxBCAilkjaq9CLS5pBcpYDsF7Sg4N5E3VuArCi1kHsYlwnhbleCqv3eukqtLKaiUQF1uXPqlVsnyxlBxQRM4GZAJLuLjQ5S6NzvezIdVKY66WwRq2Xal7aWgjsm7M8GViccZ+Byi5NL3+R/lw2hDGbmVkJ1UwkdwFTJU2RNAo4DZidt89s4My099bRwKr0stVAZWcDZ6XPzwJ+XOk3YmZm21Tt0lZEbJJ0PnAL0AzMioiHJJ2bbr8KmAOcBMwH1gHnDFQ2PfRlwI2S3gc8DZyaIZyZQ/fO6orrZUeuk8JcL4U1ZL0oYqeaGszMzLbjO9vNzKwsTiRmZlaWhkokpYZoaVSSFkh6QNL9ku6udTy1ImmWpGW59xh5CJ6i9fI5SYvSz8z9kk6qZYzVJmlfSb+R9IikhyR9OF3fkJ+XhkkkGYdoaWSvj4hDG7EPfI6rgRPy1nkInsL1AvC19DNzaETMqXJMtbYJ+FhEHAAcDZyXfp805OelYRIJ2YZosQYWEbcDz+etbvgheIrUS0OLiCURcW/6fDXwCMkIHA35eWmkRFJs+BVLRgn4paR70qFkbJvthuABCg7B06DOT0fpntUol3AKkdQNHAb8kQb9vDRSIil7mJU6dkxEHE5y2e88Sa+tdUC2y7sSeClwKLAE+GpNo6kRSbsB/wN8JCJeqHU8tdJIiSTLEC0NKSIWpz+XAT8iuQxoCQ/BU0BELI2IzRGxBfgPGvAzI2kkSRK5LiJuSlc35OelkRJJliFaGo6kVknj+p8DbwY8MvI2HoKngP4vy9TbaLDPjCQB3wUeiYh/y9nUkJ+XhrqzPe2i+O9sG2blktpGVHuSXkJyFgLJkDnfb9R6kfQDYDrJUOBLgc8CNwM3AvuRDsETEQ3V8FykXqaTXNYKYAHwwf62gUYg6W+A/wUeALakq/+JpJ2k4T4vDZVIzMxs6DXSpS0zM6sAJxIzMyuLE4mZmZXFicTMzMriRGJmZmVxIjEb5iSFpHfWOg5rXE4kZmWQdHX6RZ7/uLPWsZlVS9XmbDerY7cC78lbt6EWgZjVgs9IzMq3PiKezXs8D1svO50v6WeS1kl6StIZuYUlHSzpVkl9kp5Pz3L2yNvnrHTysfWSlkq6Oi+GPSX9UNJaSU/kv4ZZJTmRmFXe50nGYDoUmAlcK+kIAEljgV8Aa0gGPnwb8BpgVn9hSR8Evg38J3AIcBLwUN5rfIZkXKdXADcAsyR1VewdmeXwEClmZUjPDM4AXszbdEVEfFJSAN+JiA/klLkVeDYizpD0AeArwOR0giQkTQd+A0yNiPmSFgLfi4iCs+2lr3FZRFyULo8AXgBmRMT3hu7dmhXmNhKz8t0O5E8ItjLn+R152+4A3pI+PwCY259EUn8gGQjwQEkvkEzAdluJGOb2P4mITZKW0yCTKlntOZGYlW9dRMwfZFlRfIK1oPCEbIVsLFDWl66tKvxBM6u8owssP5I+fxh4Rf+cMKnXkPxtPhIRS4FFwHEVj9JskHxGYla+0ZI68tZtjojl6fO3S7oL6AXeSZIUjkq3XUfSGH+tpM8A40ka1m/KOcu5BPiapKXAz4CxwHER0ZDT29qux4nErHxvJJm3PNcikumcAT4HvAO4HFgOnBMRdwFExDpJx5NMuPYnkkb7HwMf7j9QRFwpaQPwMeCLwPPAnAq9F7Od5l5bZhWU9qg6NSL+u9axmFWK20jMzKwsTiRmZlYWX9oyM7Oy+IzEzMzK4kRiZmZlcSIxM7OyOJGYmVlZnEjMzKws/x+Xxe1BqRMGSAAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# extra code – this cell plots piecewise constant scheduling\n",
"\n",
"plt.plot(history.epoch, history.history[\"lr\"], \"o-\")\n",
"plt.axis([0, n_epochs - 1, 0, 0.011])\n",
"plt.xlabel(\"Epoch\")\n",
"plt.ylabel(\"Learning Rate\")\n",
"plt.title(\"Piecewise Constant Scheduling\", fontsize=14)\n",
"plt.grid(True)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Performance Scheduling"
]
},
{
"cell_type": "code",
"execution_count": 83,
"metadata": {},
"outputs": [],
"source": [
"# extra code – build and compile the model\n",
"\n",
"model = build_model()\n",
"optimizer = tf.keras.optimizers.SGD(learning_rate=lr0)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
" metrics=[\"accuracy\"])"
]
},
{
"cell_type": "code",
"execution_count": 84,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/25\n",
"1719/1719 [==============================] - 2s 889us/step - loss: 0.6795 - accuracy: 0.7682 - val_loss: 0.4782 - val_accuracy: 0.8314\n",
"Epoch 2/25\n",
"1719/1719 [==============================] - 1s 841us/step - loss: 0.4670 - accuracy: 0.8368 - val_loss: 0.4440 - val_accuracy: 0.8396\n",
"Epoch 3/25\n",
"1719/1719 [==============================] - 1s 843us/step - loss: 0.4191 - accuracy: 0.8506 - val_loss: 0.4155 - val_accuracy: 0.8522\n",
"Epoch 4/25\n",
"1719/1719 [==============================] - 1s 842us/step - loss: 0.3934 - accuracy: 0.8601 - val_loss: 0.3799 - val_accuracy: 0.8606\n",
"Epoch 5/25\n",
"1719/1719 [==============================] - 1s 830us/step - loss: 0.3708 - accuracy: 0.8681 - val_loss: 0.3664 - val_accuracy: 0.8670\n",
"Epoch 6/25\n",
"1719/1719 [==============================] - 1s 824us/step - loss: 0.3541 - accuracy: 0.8730 - val_loss: 0.3752 - val_accuracy: 0.8654\n",
"Epoch 7/25\n",
"1719/1719 [==============================] - 1s 831us/step - loss: 0.3416 - accuracy: 0.8762 - val_loss: 0.3545 - val_accuracy: 0.8716\n",
"Epoch 8/25\n",
"1719/1719 [==============================] - 1s 832us/step - loss: 0.3300 - accuracy: 0.8807 - val_loss: 0.3597 - val_accuracy: 0.8678\n",
"Epoch 9/25\n",
"1719/1719 [==============================] - 1s 817us/step - loss: 0.3206 - accuracy: 0.8845 - val_loss: 0.3323 - val_accuracy: 0.8804\n",
"Epoch 10/25\n",
"1719/1719 [==============================] - 1s 859us/step - loss: 0.3108 - accuracy: 0.8869 - val_loss: 0.3406 - val_accuracy: 0.8766\n",
"Epoch 11/25\n",
"1719/1719 [==============================] - 1s 849us/step - loss: 0.3033 - accuracy: 0.8893 - val_loss: 0.3551 - val_accuracy: 0.8696\n",
"Epoch 12/25\n",
"1719/1719 [==============================] - 1s 822us/step - loss: 0.2954 - accuracy: 0.8931 - val_loss: 0.3324 - val_accuracy: 0.8810\n",
"Epoch 13/25\n",
"1719/1719 [==============================] - 1s 796us/step - loss: 0.2893 - accuracy: 0.8956 - val_loss: 0.3159 - val_accuracy: 0.8810\n",
"Epoch 14/25\n",
"1719/1719 [==============================] - 1s 806us/step - loss: 0.2826 - accuracy: 0.8969 - val_loss: 0.3435 - val_accuracy: 0.8792\n",
"Epoch 15/25\n",
"1719/1719 [==============================] - 1s 830us/step - loss: 0.2762 - accuracy: 0.8995 - val_loss: 0.3470 - val_accuracy: 0.8792\n",
"Epoch 16/25\n",
"1719/1719 [==============================] - 1s 813us/step - loss: 0.2701 - accuracy: 0.9019 - val_loss: 0.3276 - val_accuracy: 0.8794\n",
"Epoch 17/25\n",
"1719/1719 [==============================] - 1s 821us/step - loss: 0.2656 - accuracy: 0.9025 - val_loss: 0.3334 - val_accuracy: 0.8796\n",
"Epoch 18/25\n",
"1719/1719 [==============================] - 1s 814us/step - loss: 0.2608 - accuracy: 0.9040 - val_loss: 0.3246 - val_accuracy: 0.8844\n",
"Epoch 19/25\n",
"1719/1719 [==============================] - 1s 816us/step - loss: 0.2417 - accuracy: 0.9120 - val_loss: 0.3155 - val_accuracy: 0.8798\n",
"Epoch 20/25\n",
"1719/1719 [==============================] - 1s 821us/step - loss: 0.2387 - accuracy: 0.9135 - val_loss: 0.3149 - val_accuracy: 0.8826\n",
"Epoch 21/25\n",
"1719/1719 [==============================] - 1s 825us/step - loss: 0.2359 - accuracy: 0.9143 - val_loss: 0.3076 - val_accuracy: 0.8844\n",
"Epoch 22/25\n",
"1719/1719 [==============================] - 1s 824us/step - loss: 0.2328 - accuracy: 0.9148 - val_loss: 0.3156 - val_accuracy: 0.8854\n",
"Epoch 23/25\n",
"1719/1719 [==============================] - 1s 821us/step - loss: 0.2304 - accuracy: 0.9157 - val_loss: 0.3225 - val_accuracy: 0.8808\n",
"Epoch 24/25\n",
"1719/1719 [==============================] - 1s 827us/step - loss: 0.2274 - accuracy: 0.9176 - val_loss: 0.3158 - val_accuracy: 0.8834\n",
"Epoch 25/25\n",
"1719/1719 [==============================] - 1s 825us/step - loss: 0.2249 - accuracy: 0.9183 - val_loss: 0.3144 - val_accuracy: 0.8852\n"
]
}
],
"source": [
"lr_scheduler = tf.keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=5)\n",
"history = model.fit(X_train, y_train, epochs=n_epochs,\n",
" validation_data=(X_valid, y_valid),\n",
" callbacks=[lr_scheduler])"
]
},
{
"cell_type": "code",
"execution_count": 85,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAcgAAAEbCAYAAABeCxRrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABM20lEQVR4nO2dd5hUVdKH34JhSCoyYkBRQMS4q4iIcRVUFFlXxIS45lVEBcHVNawJRXfVNaEoiIrKigHFwKcYkTGsCVQyIogJGMGMCAoz1PdH3ZE7Pd3Tt2d6unum632e+/QN55xb99LMr885dapEVXEcx3EcpyINsm2A4ziO4+QiLpCO4ziOEwcXSMdxHMeJgwuk4ziO48TBBdJxHMdx4uAC6TiO4zhxcIF06jQislJETsu2HfUNEflcRC7Kth2Ok01cIJ1aR0QeFBENtlIR+VJERopIy2zblg5EpFvwbK0SXB8aev51IrJURMaJyNaZtjWw57SQPSoiJSIyXkTa17DNlem003GyjQukkyleBVoD7YAzgb8Ad2fToAwzH3v+NkBf4I/A+CzasyqwZ0vgRKATMFFEGmbRJsfJKVwgnUzxm6p+raqLVfVl4HHg0HABETldROaKyK8i8omIXCAiDULXtxOR4uD6fBE5IqZ+u6BH1CXmvIrIsaHjLYMe3HciskpEpotI99D1v4jIB8F9PhOR60WksIbPXxo8/1JVfRO4F9hbRDaqqpKIHC0is0TkNxH5SkQuFxEJXf9cRK4QkXtEZIWILBaRf0SwRwN7SlR1CnAN8AdguwR2/F1EZorILyKyRETuE5GNg2vdgAeA5qFe6dDgWqGI3BjY9YuITBWRw0LtNhSR+4P3vFpEFojIxTH/7g+KyHMx9gwVkdkRntNxqk1Btg1w8g8R2RboCawNnTsLuBYYBHyA/bG+NygzIviD+TTwA7AP0AwYDjRO8d7NgdeB5UAfYAmwW+j6YcA4YDDwBrANMCq4T1rm5ERkC+BooCzYEpXbA3gCuC6waU/gHmAFcGeo6AXA1cB/gMOBO0TkLVV9JwWzVgefjRJcXwcMARYBbYP73wmcDLwdXPsX0CEoXz7c+kBw7kRgMdAL+D8R2VNVZ2A/0pcAxwPfAF2B0cB3wP0p2O846UdVffOtVjfgQaAU+6O5GtBguyBU5kvg5Jh6Q4C5wf6hmJhsE7q+f9DOacFxu+C4S0w7Chwb7J8F/Ay0SmDrG8CVMeeOCmyXBHW6BfdI1ObQwPaV2NBm+fMPT/LexgGvxWlrcej4c+DRmDILgCuqaPc0YGXouA3wDvAVUBhq96Iq2ugJ/AY0iNdmcK4DJqzbxJx/Bri7irZvAF6N+f48F+c9zM72d9u3+r15D9LJFG8A/YGmmEh1AO4AEJFNga2Be0RkZKhOAVA+nLgTsERVvwxdfw/7A5wKuwMzVfXbBNf3ALqKyCWhcw0Cu7cASlK8XzmfYr2nxkBv4Bjgn0nq7AQ8H3PuLeBqEdlIVVcE52bGlFkKbJak7eaBU41gvfEPgaNVdU28wiJyEHBZYFMLoCFQiL2TpQnu0Tlof25oVBjsHbwWansANi/dFnvPjYAvktjvOLWOC6STKVap6sJg/3wRmQJcifUEyuebBmDDdfGQBOfDlItleI4udsgwWTsNsPm4J+Jc+yaCDYlYE3r+OSLSEbgL63klQrCeZjzC59fGuZbMv2AV5pizDlimqr8kNEKkLSbU9wJXYcOfnYFHMZFMRIPAlj3j2Lg6aLsvcDs2fP02Nnx8Hjb8Xc46Kv+7JRoKdpy04QLpZItrgBdEZLSqLhWRJUAHVR2boPxcYCsR2VpVvwrOdaWiEJQLWOvQuU4x7XwInCQirRL0Ij8EdgyJWW0xDJgvIneq6gcJyszFhpHD7I8Nsf5cw/trCs/YBRPCC1S1DCDWQQpYg/Uqw3yECdsWao5A8dgfeE9VR5SfEJEOMWW+ofK/Y+yx46Qd92J1soKqFgNzgCuCU0OBiwPP1R1E5A8icoqIXBZcfxX4GBgrIp1EZB/gNmxus7zN1cC7wCUisouI7AvcHHPrRzAHnWdE5E8i0l5Ejgx5sV4LnCgi1wY27Cgix4rITREe6w+BbeEt7v8xVV0ETMSEMhG3AAcGHpvbi8hfgQuBKLakkwXY34ohwfvqh80Ph/kcaCIiPUSklYg0U9VPsHnUB4N3uK2IdBGRi0Tk6KDeJ0BnETlcRDqKyJXAgTFtvwbsLiJniHkyXwzsV0vP6jjryfYkqG/1fyOOk0Vw/kTM0aNtcNwP68H9inmrvgWcECq/PeaB+hv2R/tIzPHltFCZnYD/YUOIs4A/EXLSCcq0wZaZ/BiU+wjoFrp+KPBmcG0FMA0YWMXzdWO9403stgEJHEqAfYMy+1bR9tHBc6zBnGguJ+QsRBxnGqAYGFFFm6cR41ATp0yFdoHzMW/T1cBkzOtUgXahMiOBb4PzQ4NzjYLnXxQ8w9fYD4M9guuFmLfqD8G/x/3YMO7nMfYMxeZ/f8LWz/4r3jv1zbd0bqKaaIrDcRzHcfIXH2J1HMdxnDi4QDqO4zhOHFwgHcdxHCcOLpCO4ziOE4e8XAfZoEEDbdq0abbNyDnWrVtHgwb+mymMv5P4+HuJT31/L6tWrVJVrb8PGENeCmRhYSG//JIwcEjeUlxcTLdu3bJtRk7h7yQ+/l7iU9/fi4isTl6q/pA3vwQcx3EcJxVcIB3HcRwnDi6QjuM4jhMHF0jHcRzHiYMLpOM4juPEIaMCKUJPEeaLsFCES+NcFxHuCK7PFKFz6NoYEZaLMDumTpEIr4iwIPhsmcyOtr+tZc+tv2bcuGh2jxsH7dpBgwb2WV/rHXTQgXXCzkzU83dSdb1cfy+OkxYyFRUdtCHop6DbghaCzgDdOaZML9AXQAV0b9D3QtcOAO0MOjumzk2glwb7l4LemMyWPUBHcK42a6b68MNaJQ8/rNqsmSqs37xe/a5XF2z0ernJlClTsm1CrQL8olnOsJHJLWPZPETYBxiqymHB8WUm0Pw7VOYeoFiVR4Pj+UA3VUqC43bAc6r8IVTn9zIitA7q71CVLV1E9A2asi2L+LXFFpx/fuKyd9wBP/1U+XyLFni9elqvLtiYz/XatoXPP09cL5vkwTrIVaraPNt2ZIpMCuSxQE9VzgyOTwb2UmVgqMxzwA2qvBUcTwYuUWVacNyOygL5oyobh45/UK08zCpCf6A/wB7IHm9RyP2cyUBGIJLYbns98Qqo16un9eqCjflcT0R57bXXE1fMIitXrmSDDTbIthm1Rvfu3fNKIDPWVQU9DvS+0PHJoHfGlHkedP/Q8WTQPULH7eIMsf4Yc/xDMlv2CMZrfqGpdmlTEjuKUIG2bbXCEE/51rZtldW8Xh2uVxds9Hq5Sd4PsUJPhfkKCxUuraLcngplWp7IHHZQmB7aVigMCa4NVVgSutarShvSuGXkJvZedR/Ql0LHl4FeFlPmHtB+oeP5oK1Dx/EE8vcyoK1B5yezpVwgV1OoHx98bpVfiLoy7+L10levLtjo9XKTvBZIaKjwqcK2CoUKMxR2TlDuNYVJvwtk5etfK7QNjocqXJTwvrWpWxm7EVoAugi0PeuddHaJKfNnKjrpvB9zPZ5A/oeKTjo3JbNlj/D/uE6dkn4pHn7YfrWK2GfU/6B1r966OmJn7dfzd5KsXubeS8uW9l+1TZvcFkfVvBfIfRReCh1fpnBZnHJDFM5TeDCBQB6q8L/Qcf0XSHu32gv0E8yb9fLg3ADQAcG+gN4VXJ8F2iVU91HQEtC1oItB/xac3wQbil0QfBYls2PzggJ79NdfT+GrUf+p7/+5q4O/k/hk8r2MG2f/XefPz9gtq019/760gt8UpoW2/rpeyI5VuC90fLLCCA3//YWtFF4PeomJBHKMwsDQ8VCFzxVmBtdaVqpTS1tGs3moMgmYFHNuVGhfgfMS1O2X4Px3wMGp2LGiYUMoLIRHHoEDDkilquM4GaaoyD6//z67djjwLZSi2iXB5fieWBW5HbgE1bK4HloihcCRYKscAkYCw4K2hgG3AGekZHg1yctIOusAeveGJ56ANWuybY7jOFVQLpDffZddO5ykLAa2Dh23AZbGlOkCPIbI58CxwN2IHBW6fjjwIarLfj+jugzVMlTXAfcCXdNvenzyUiABOPFE+0n68svZtsRxnCrwHmSdYSrQEZH2QU/wBGBihRKq7VFth2o74EngXFSfCZXoB7YO/ndEWoeO+kDFaGq1Sf4K5KGH2v+8Rx7JtiWO41SBC2QdQbUUGAi8BMwDxqM6B5EBiAxIWl+kGdADeCrmyk2IzEJkJtAduCC9hicmo3OQOUVhIRx3HPz3v7ByJdTjxb2OU5dp0QJEXCDrBKqV/ExQHZWg7Gkxx6uATeKUOzlN1qVM/vYgwYZZV62CiROTl3UcJys0bAgbb+wC6WSe/BbI/feHNm18mNVxcpyiIhdIJ/Pkt0A2aAD9+sFLL8G332bbGsdxEuAC6WSD/BZIsGHW0lJ48slsW+I4TgJcIJ1s4AK5226w004+zOo4OYwLpJMNXCBFrBf55pvw5ZfZtsZxnDhssokLpJN5XCDB5iEBHnssu3Y4jhOXoiL44QdYty7bljj5hAskQIcOsNdePszqODlKUZGl3/npp2xb4uQTLpDlnHgizJgBc+Zk2xLHcWLwaDpONnCBLOf4423Zx6OPJi/rOE5GcYF0soELZDlbbAEHH2zDrBqbocVxnGziAulkAxfIMCeeCJ99Bu+9l21LHMcJ4SmvnGzgAhmmTx9o3NiddRwnx/AepJMNXCDDtGgBRxwBjz9u0XUcx8kJWra0TxdIJ5O4QMZy4omwfDm89lq2LXEcJ6CgADbayAXSySwukLH06mX/E32Y1XFyCg8352QaF8hYmjSBY46Bp56C1auzbY3jOAEukE6mcYGMx4knws8/w/PPZ9sSx3ECXCCdTOMCGY/u3WHzzX2Y1XFyCBdIJ9O4QMajYUM44QTrQf74Y7atcRwHF0gn87hAJuLEE2HNGpuLdBwn65QLpAe6ymFEeiIyH5GFiFxaRbk9ESlD5NjQuc8RmYXIdESmhc4XIfIKIguCz5a1+gwhXCATseeeluXDh1kdJyfYZBMoKzP3ACcHEWkI3AUcDuwM9ENk5wTlbgReitNKd1Q7odoldO5SYDKqHYHJwXFGcIFMRHki5ddeg5KSbFvjOHmPR9PJeboCC1FdhOoa4DGgd5xyg4AJwPKI7fYGHgr2HwKOqqGdkXGBrIp+/Ww85/HHs22J4+Q9LpDZpxUUIDIttPUPXd4K+Cp0vDg4tx6RrYA+wKg4zSvwMiIfxLS7OarWS7HPzWr+JNEoyNSN6iQ77QS7727DrEOGZNsax8lrXCCzz7dQGjP8GUbinIudMb4duATVMqRS8f1QXYrIZsAriHyM6hs1s7hmeA8yGSeeCFOnwoIF2bbEcfIaF8icZzGwdei4DbA0pkwX4DFEPgeOBe5G5CgAVJcGn8uBp7EhW4BliLQGCD6jDs3WGBfIZJxwgs1HeiJlx8kqnvIq55kKdESkPSKFwAnAxAolVNuj2g7VdsCTwLmoPoNIc0Q2BECkOXAoMDuoNRE4Ndg/FXi2th+knIwKpAg9RZgvwkKRyp5IIogIdwTXZ4rQOVldEXYT4R0RZonwfyJslFaj27SBAw6AsWPhwAPh66/T2rzjONHwjB45jmopMBDzTp0HjEd1DiIDEBmQpPbmwFuIzADeB55H9cXg2g1AD0QWAD2C44yQsTlIEcpdgHtgXfGpIkxUZW6o2OFAx2DbCxgJ7JWk7n3ARaq8LsIZwD+AK9Nq/Iknwtlnw6JFMGwY3HVXWpt3HCc5jRtD8+YukDmN6iRgUsy5eA45oHpaaH8RsFuCct8BB6fHwNTIZA+yK7BQlUWqJHIB7g2MVUVVeRfYWITWSeruAJRP5L4CHJN2y/fbzz5V4YEHvBfpOFnCo+k4mSSTXqzxXID3ilBmqyR1ZwNHYuPSx1Fxkvh3ROgP9AcoKBCKi4sjG97xttvYUgRRZd3atZQMGMCCeujVunLlypTeSz7g7yQ+2XovhYVd+OSTXykunp28cBbw70v9IpMCGcUFOFGZquqeAdwhwlXYZO6aeDdXZTQwGqBJE9Vu3bpFMBkLEvDyy7/Ht2pQWspWL7/MVqNGwRZbRGujjlBcXEzk95In+DuJT7beyzbbwNq1G+Tsv4l/X+oXmRxijeICnKhMwrqqfKzKoarsATwKfJpWq4cNg3XrKp4rK7PzjuNkFB9idTJJJgVyKtBRhPYixHcBtuNTAm/WvYGfVCmpqq6IRVUQoQFwBfEjNFSfd96xoOVh1qyBt99O620cx0mOC6STSTImkKpUcgFWZY4IA0QodwGeBCwCFgL3AudWVTeo00+ET4CPsV7lA2k1/KOPbHhV1aIkb7SRebV+9FFab+M4TnI8o4eTSTIaak6VSi7Aqut7fKoocF7UusH54cDw9FqagA02gNNPh7vvhltuqXdzkI6T6xQV2QDOqlW25MNxahOPpJMq554La9fC6NHZtsRx8g4PN+dkEhfIVNl+ezjsMBg1yoTScZyMsckm9ukC6WQCF8jqMHCgLf94+ulsW+I4eYX3IJ1qIdKoOtVcIKvD4YdD+/YwYkS2LXGcvMIF0kmKyPmIHBM6vh9Yjch8RHZIpSkXyOrQsKHNRb75JsyYkW1rHCdv8IweTgTOB74BQOQA4HjgRGA6cEsqDblAVpczzoCmTT1wueNkEO9BOhHYCvg82P8L8ASq44GhwN6pNOQCWV2KiuCvf4WHH4Yffsi2NY6TFzRtCk2auEA6VbIC2DTY7wFMDvbXAk1SaSiyQIpwuAjPiTBXxMK+iXCmSHbSkOQE550Hq1dbhg/HcTKCR9NxkvAycG8w97gd8EJwfhfgs1QaiiSQIvwVGA8sANoD5R5BDYGLU7lhvaJTJ9h/fxtmjY3X6jhOreAC6SThPOB/QCvgWFTLvy2dsXjdkYnag7wYOEuVC4DS0Pl3gU6p3LDeMXCgJVJ+8cXkZR3HqTEukE6VqK5AdRCqvVF9MXT+alT/lUpTUQWyI/BOnPMrgY1SuWG9o08faN3al3w4ToZwgXSqRGTnCss5RHog8jAilyHSMJWmogrkUmD7OOcPIN3ppeoahYVw9tnwwguwcGG2rXGceo8LpJOE+4HdARBpAzwLFGFDr9el0lBUgRyNJSXeLzjeWoRTgZuAkancsF7Svz8UFFgQc8dxahUXSCcJOwEfBvvHAe+h2gs4GeiXSkORBFKVm4CngFeA5sAULO/iKFV8IWDr1nDssTBmDPzyS7atcZx6TVGROY+vXp1tS5xKiPQMItYsROTSKsrtiUgZIscGx1sjMgWReYjMQWRwqOxQRJYgMj3YeiWxoiFQnsT3YNZngfoU2DyVx4m8zEOVyzGvoK7YYstNVbkylZvVawYOhJ9+snWRjuPUGuXBAnz5cY5h83t3AYcDOwP9ENk5Qbkbsfy+5ZQCF6K6E6Yv58XUvQ3VTsFWKe1hDLOBcxD5EyaQ5Y46WwHfpvJIUZd5jBFhQ1VWqTJNlfdVWSlCcxHGpHLDesu++9qyjxEjPJur49QiHk0nZ+kKLER1EaprgMeA3nHKDQImAMt/P6NaguqHwf7PwDxM0KrDJcBZQDHwKKqzgvNHAu+n0lDUHuSpQNM455sCp6Ryw3qLCAwaBLNnwxtvZNsax6m3uEBmj1ZQgMi00NY/dHkr4KvQ8WJiRU5kK6APNkUXH5F2mJPNe6GzAxGZicgYRFpWaaTqG1gknVaonhG6cg9wTpV1Y6hSIEUoEmETQICWwXH5tilwBLAslRvWa/r1s/+9vuTDcWoNzwmZPb6FUlS7hLZw5niJUyV2OO124BJUy+LeQGQDrHc5BNUVwdmRQAdszX0JUQKOW/urEfkDIrsg0gTVz1FdnrRuiIIk17/FHlCBufHMAK5O5Yb1mqZN4W9/g1tvhcWLoU2bbFvkOPUO70HmLIvBwpAGtMGWCIbpAjyGCJhPSy9ESlF9JsjZOAEYh+pTv9dQXd8JE7kXeK5KK0QKgH8DA4FCTLh/Q+RO4HJUI2e6TzbE2h2b5BTgWOCg0LY/sI0q10e9WV5wzjkWdu6ee7JtiePUSzzlVc4yFeiISHtECoETgIkVSqi2R7Udqu2AJ4FzA3EUbP3iPFRvrVBHpHXoqA/mhFMVNwEnAQOw9fsdsaHVkzHhjEyVPUhVXjf7aA98pYoHHE1G+/ZwxBEwejRccQU0bpxtixynXtG8OTRq5D3InEO1FJGBmHdqQ2AMqnMQGRBcTzzvCPthAjYLkenBuX8GHqs3IdIJG7H8HDg7iSUnAmfEeLt+isg3wH3ARVEfKdkQKwCqfAEgwpbANli3NXzdvVLCDBwI//d/8OSTlhLLcZy0IeLBAnIWE6VJMefiC6PqaaH9t4g/hwmqJ6doRQviR3j7FNg4lYaiLvPYUoRibIz5f5j77JTQ5oQ55BDYfnt31nGcWsIF0qmCGcD5cc4PBqan0lDUZR63A2XY4s9VwJ+wED7zgJ6p3DAvaNDAckW++y5Mm5Ztaxyn3uEC6VTBxcCpiHyCyEOIPIjIfGxe8h+pNBRVIA8ELlHlY2wc+BtVnsIWZA5L5YZ5w6mn2mTJf/4DBx4IX3+dbYscp97gAukkxNZBbg88AWyAZZx6AtghGMqNTFSBbMr6ED3fA5sF+3OBXVO5Yd7QogWccorNQ771Fgzz3xGOky5cIJ0qUV2K6uWoHoPq0aheATRCZHwqzUQVyI+BHYP96cAAEdpi6UOWpHLDvOL4423Jx7p18MAD3ot0nDThAulUg42BY1KpEFUghwNbBPvXAocCi4BzgX+mcsO8Yvx4ggWxUFbmvUjHSRNFRbByJaxZk7ys41SXqOmuxqnyYLD/IdAO2BMLFPBErVlXlykpsV5jeeDyNWu8F+k4acIzejiZIHK6qzBBVo8PgV9ESJzzK58ZNsyGVsN4L9Jx0oKHm3MyQdJAASK0AvYC1gKTVSkToRE2/3gZFjHhhig3E6EnNlzbELhPtWI9ESS43gtbTnJaIMQJ64rQCYsM3wTLKXauamopTWqFd96pPP6zZg28/XZ27HGceoQLpFMJkYlJSmyUapPJsnnsCywA/g94AfifCDsCM7FAsMOwyDpJEaFSMk0RYpNpHo7FzesI9MeiuCerexNwjSqdgKuC4+zz0Uc2vKoKI0fauRkz7LzjODXCBdKJw3dJts+Asak0mKwHOQyLq3cdcAYwBIukfi3wX9VKqUyqoiuwUJVFACK/J9MMZwnpDYwN2n1XhI1FaI3NeSaqq6z/ZdCCytHjs8/RR1vggPHjYVdfFeM4NcUF0qmE6unpbjKZQO4GHKjKHBGuwEL1XFZNx5x4yTT3ilBmqyR1hwAviXAz1iPeN97NReiP9UopKBCKi4ur8QjVZ9fdd6fJgw/y/sEHr/dszTFWrlyZ8feS6/g7iU+238vKlQ2BP/H++wtp23Zx1uyIJdvvxUkvyQSyCPgGzDFHhFVAdccIoyTTTFSmqrrnABeoMkGE47GUKYdUKqyMBkYDNGmi2q1bt4hmp4kBA+Css+jWogV07pzZe0ekuLiYjL+XHMffSXyy/V5UoWFDaNlyO7p12y5rdsSS7ffipJcoXqwtRSgSYROC4czg+Pct4r2iJNNMVKaquqcC5ck1n8CGcnOPPn2goAAefzzbljhOnUcEWrb0IVandokikHOxXuRyLK7d1OD4Gyz83DcR7zUV6ChCexHiJ9O041NEEBH2Bn5SpSRJ3aVYrFiwRM4LItqTWTbZxLJ8jB+/fm2k4zjVxqPpOLVNsiHW7um6kSqlIlRIphnMbQ4Iro/C8oj1AhZiyzxOr6pu0PRZwHARCoBfCeYZc5K+feH002HqVOiamx1dx6kruEA6tU2VAqnK6+m8mSqVkmkGwli+r9j6ykh1g/NvAXuk085a46ij4OyzbZjVBdJxakRRESxblm0rnJxEpBnQCUusUXGkVPWpODXikjRQgJNGNt4YDjvMhln/8x/LG+k4TrUoKoJ587JthZNziBwCPApsEueqYqOQkfC/0Jnm+ONh8WJLpuw4TrXxIVYnAcOB54E2qDaI2SKLI7hAZp4jj4TGjd2b1XFqSFER/PQTlJZm2xInx2gHDEO1xkFjXCAzzUYbQa9e8MQTFrzccZxqUR5N58cfs2qGE0akJyLzEVmISOJEFiJ7IlKGyLFJ64oUIfIKIguCz5ZJrPgfsEMNnwRwgcwOfftaOqy33sq2JY5TZ/FwczmGSKWY2YjExtsuL3cjtiohSt1LgcmodgQmB8dVMQq4GZEzEdkLkc4VthSI5KQjwpgElxRbWrEQeFw1B+Og5iJ//jM0bWrOOgcemLy84ziVcIHMOboCC1FdBIBIvHjbAIOACVhO4Sh1ewPdgnIPAcXAJVXY8WTwOTrOtZScdKJ6sW4K/AlYB8wOzv0BCwH3AXA0cK0If1JletSb5y0bbABHHAFPPgnDh1uEHcdxUsIFMvO0ggJEpoVOjUa1XIiSx9sW2QrogwV1CQtkVXU3R7UEANUSRDZLYmb75E8Sjah/mf8HrAT+psoqABGaAfcCM7DF/WOBW4CD02VcvaZvX5uHfP11ONhfmeOkigtk5vkWSlHtkuBylHjbtwOXoFoWk7QhSt1oqH5RrXpxiDoHORi4tlwczQZWAddjgcLXYGPKndJlWL2nVy9o3ty9WR2nmrhA5hxR4m13AR5D5HPgWOBuRI5KUncZIq0Bgs/lSS0R2RWRsYhMQ2QqIg8h8sdUHyiqQG4AtI5zfovgGsAKPPBAdJo2tSUfTz0Fa9dm2xrHqXNsvLEFLf/uu2xb4gRMBToi0h6R+PG2Vduj2g7Vdthc4bmoPpOk7kQsKQXB57NVWiFyJPAhJrgvAC8C2wAfIvKXVB4oqkA+DdwvwnEitBOhrQjHYamlysP2dAU+SeXmeU/fvva/+7XXsm2J49Q5GjY0kfQeZI6gWgq/x8yeB4xHdQ4iAxAZUK26xg1AD0QWAD2C46q4Drge1e6oXhls3YF/B9ciE7XHNwC4FXg4VKcUGANcFBzPwwKHO1Hp2dPWRT7+uIWgcxwnJTyaTo6hWjlmtuqoBGVPS1rXzn9Har4t2wP/jXP+v8DFKbQTrQepyipVBmAJlHcHOgNFqpyjyi9BmenuwZoijRtbAPOnn4Y1a7JtjePUOVwgnTgsJ34Ciz2AlMLbpxQoQJVfVJmpyoxyYXRqyPHHWyiQV17JtiWOU+dwgXTicC9wDyKXI9IdkW6IXIEFEIi3NjIhUQMFNME8WQ8mTvoQVXZN5aZOiB49LDX6449bAAHHcSJTVASffpptK5wc4zpsWeKFwLDg3FLgauCOVBqKOgd5N7a48wngbaq7PsWpTGEh9OljayJ//RWaNMm2RY5TZ/AepFMJVQVuA25DZMPg3M/VaSqqQB4FHKfKq9W5iZOEvn1hzBh48UWbk3QcJxJFRfDDD7BunadXdeJQTWEsJ6pArqJiGCAnnXTvDptsYrFZXSAdJzJFRaBqaa9aJsvx4NRfRGYCB6L6AyKzqGqUUzXylGBUgbwJ+LsI56iyLmrjTkQaNYJjjoFx42DVKmjWLNsWOU6dIBxNxwUyr5kA/BbaT8s0YFSB7IEFK+8pwlygQugXVY5MhzF5Td++MHo0TJoExx6bvLzjOBUEskOH7NriZBHVa0L7Q9PVbNRR+2+xaDqvAV8D38VsTk058EDYbDOPzeo4KeDxWJ1KiLyGyMZxzm+ESEphyyL1IFU5PZVGnWrQsKH1HB94AFautJRYjuNUiQukE4duQGGc802wkdDIuN9XLtG3L6xeDc89l21LHKdO4ALp/I5IZ0Q6B0e7/n5s255Af2BJKk0m7EGKMBM4UJUfRKjSK8gDBaSJ/feHLbe0YdYTTsi2NY6T85Q75rhAOsA0TKcUeDnO9dXAoFQarGqINewV9GQqjTrVpEEDOO44GDUKVqywQOaO4ySkUSPYcENPeeUA0B5LvLwIyy71TejaGmA5qmWpNJhQIFW5Jt6+U8scfzwMHw4TJ8JJJ2XbGsfJeTbZxHuQDqD6RbCXtqlDT3Cca+y9N2y9NTz0ENx7rw23brFFtq1ynJzFw805lRApwHqR2xDrsKM6NmozUYOVFwHXkzhYuY8FposGDawXeeutli592DC4665sW+U4OYsLpFMBkR2B/2P9kGsZpnVrsWnDyAIZtSt6P3A48CAwBJvoDG9OOjn4YIuftW6dLfv4+utsW+Q4OYsLpBPD7cAHQAssTOpOQBdgOnBMKg1FHWI9GOihynupNO5Uk4kTrfeoCqWl3ot0nCpwgXRi2BOLy/oLIuuAAlQ/RORi4E6Ivuoiag9yOZZfy6ltSkrgwQdNHAHWrrVMH96LdJy4lAukehI+xxCs5wjmybpVsL8Y2C6VhqIK5OXAtSLUKLyLCD1FmC/CQhEujXNdRLgjuD5ThM7J6orwuAjTg+1zEabXxMasM2yYDa2G+fVXuPzy7NjjODlOURGUlcHPNUps5NQjZgO7BfvvA5cgciBwDbAwlYaiCuQVwKHAchHmBeL1+xalAREaAndhc5k7A/1E2Dmm2OFAx2DrD4xMVleVvqp0UqUTtnbzqYjPlJu88w6sWVP5/GOPWaaP2qKkhE6DB3tP1alzeDSdHEKkJyLzEVmISKVOECK9EZmJyHREpiGyf3B+h+Bc+bYCkSHBtaGILAld65XEiuuxXiSYdm0NTME07PxUHifqHGQ6AgV0BRaqsghAhMeA3sDcUJnewFhVFHhXhI1FaA20S1ZXBAGOBw5Kg63Z46OPKp976imL09qvH0yYAAW1sDrnmmtoMWuWz3c6dY6wQLZrl1VT8huR8o5MD2w4cyoiE1EN/42fDExEVRHZFRgP7IjqfKBTqJ0lWIKMcm5D9eZIdqi+FNpfBOyMSBHwA5raQHzSv7QiNAKaA3ep8kWy8lWwFRWTLi8G9opQZquIdf8ELFNlQbybi9Af65VSUCAUFxenaH4WKSpiy0GD2P6OO1h69NF8csEF5sSTDlRpM348He65x/yh77+f9w4+mDXlf3XynJUrV9at70qGyKX38sUXLYDdmTJlBitW/JBVW3LpvWSBrsDCQJRApHInSDXsy9Kc+CFMDwY+DS38rzmq1RpfSCqQqqwV4Rzg7urcIES8v+ixLydRmSh1+wGPJrq5KqOB0QBNmqh269YtoaE5Sbdu0Lw5W/7732zZtStccUXN2/zhB+jfH55cP0DQUJV9J0/2XmRAcXExde67kgFy6b1suql9br31bmTbpFx6L7VBKyhAZFro1GhURwf7UToyINIH+De2pv7PcW5zApX/lg9E5BQs3uqFqFb8JSQyhahJklUjjzJGnYN8mZoPXS7GxoLLaQMsjVimyroiFABHA/U7meL118Mpp8CVV9r6yJrwv/9Bp07w9NMVh2zXrPG1l06dwucgM8e3UIpql9A2OnQ5SkcGVJ9GdUfgKGBYhWsihcCRwBOhsyOBDtgQbAlwS5z7zAbmBNvHwB6YYC8Oti2Dc/OSPWOYqAI5GfiXCLeLcLIIR4e3iG1MBTqK0F6EQuxXwsSYMhOBUwJv1r2Bn1QpiVD3EOBjVRZHtKVuIgL33QeHHgpnnQWTJqXeRmkpXHMNHHCACWPv3ha9J7bMsGHx6ztOjuEZPXKGKJ2g9ai+AXRApFXo7OHAh6guC5VbhmoZquuAe7Gh3Ni2Bv2+WbSch7C5zVOCbUfgAaA0lQeK6u0xIviM5wGkQMNkDahSKsJA4KWg/BhV5ogwILg+CpgE9MJccVeBJWpOVDfUfLwuef2kUSMbEu3WzTJ/FBfDnntGq/vllxYA/c037fOuu+DAAyt7za5daz1Mx6kDNGkCzZq5QOYAU4GOiLTHnGxOAE6sUEJkO2x+UYPcjYVAOBdL5akykdaolgRHfbDeYlWcAuwTxyHnbuBdYHDUB4okkKrpiY6uyiRMBMPnRoX2FTgvat3QtdPSYV+dYcMN4fnnYd994c9/hrffhu2SrH+dMAHOPNN6h2PHwskn2/mQ12xxcTHdli6Fv/4VBnkEQafuUFTkKa+yjmopIhU6MqjOQWRAcH0UFurtFETWYvkZ+/4uZCLNMA/Ys2NavgmRTlhn7PM412MR4I/AJzHn/5jqI3k2j7rKFlvAiy/CfvvBYYeZSG6+eeVyq1bBkCGWGWTPPeGRR6oW03794O674bLL4JhjYOONa+sJHCdteLi5HEG1ckfGhLF8/0bgxgR1VwGbxDl/copWjAHuQ6Qj1mME2Bu4GBtmjUxkgQwyevQkTvoQVa5N5aZOmth+e3juOejeHY44AqZMgQ1CwY5mzDDBmzcPLr7Y5hULCxO3BzbPeccd0KULXHutZRVxnBzHc0I6IS7GwqMOBv4VnCsBbiC+g09Coqa72ht4Hpv83BQbX24dHH8OLpBZY6+9YPx4c7Y57ji45x4bPj30UBPEli3hlVfgkEOit9m5szkB3Xmnfe60U+3Z7zhpoKjIfgc6TuDMcxM2NLtRcG5FdZqKOrf4H2Ac5jb7K7bkYxtsTUr87rKTOY44woTxxRdNCN94w9ZJHnIIzJyZmjiWc9110Ly5Dc96FGgnx/EhVicuqiuqK44QXSB3BUYETjRlQGNVlgGXAEOre3MnjZx5Jlx4ISwIAgk1amTzjuWrqFNl001tiPXlly39luPkMJ7RI8+x+K4tg/1ZwXH8LQWizkGG1wEsA9piCy5XYgswnVzgl1+gYUNLbSBivcCaRMQ55xzrmf797+YI1KRJ+mx1nDRSVGSrlVatsoEPJ++YgE35QXpihwPRBfJDLAnlJ0AxcJ0ImwMnQbRsHk4tU55HsqzMjssj4lx5pXm8VodGjWD4cOjRw5x1/vnPtJnrOOkkHE3HBTIPUb0m7n4NSSUfZHlEhCuwJJR3Ai0JAoA7WSZeHsmysppHxDnkEDj6aAtzt7h+Bypy6i4ebs6pDaIGCpgW2v8GCwfk5BLx8kiuWWPrI2vKzTdbYIJLLoFx42renuOkGRfIPEdkFtGDle8atdmUAgWI0AULGvucKr+I0Bz4TTW1+HZOLRAvj2S6aN9+/TrKc8+14ASOk0O4QOY9aZt3DBN1HeTmWHDwPTGV7ggsAm7Fln1Ejm3n1FEuucTmNAcNgqlTzRmoJpSUwAknwOOPV3+O1HECXCDznDTOO4aJOgd5G/A1FgZoVej8E8Ch6TbKyUGaN7eh1o8+gjFjataWKvzjH/DWW541xEkLLpBObRBVIA8GLlclNl33p1jAACcfOP54S5P1z39asuVU+fVX64X+8Y82l7luneeedNJC06bQuLELpBMgcjoiLyPyMSKLKmwpEFUgm1JxLWQ5m2JDrE4+UB6n9fvvLadkVJYsscg+W28NZ5xhw6vlpMPT1sl7RDyjhxMg8g8s5uoHQDvgGSxFVhEWyDwyUQXyDaiQUkpFaIhF0pmcyg2dOs5uu8HZZ8OIETBnTuJyquZZ268ftGsH//qXOfc88YSt5i6nfL2m9yKdGuLh5pyAs4D+qF4GrAVGoHokJpptU2koqkBeDJwlwitA4+BGc4H9gMtSuaFTDxg2DDbaCAYPrhzb67ff4OGHLYj6vvvCCy/A+efDwoXwzDPw2mu1s16zNigpodPgwS7edQQXSCegDfB+sL8a2CjYfxTLRxmZSAKpylws2eTbwMtAE8xBZ3dVPk3lhk49YJNNTNAmTzaHnQMPtKDo11wDbdtaNpEVKyzM3eLFcMstsO22Vrc212umm2HDaDFrVm6Kt1MJF0gn4GugVbD/BbBPsL8dUddKBkTtQaLK16pcrcoRqvRS5QqgUITxqdzQqSecfbY52wwZYtlDOnWCoUNhjz0sq8jcubZmMpyfEswLVtW25cvNs+Lss2t3HWd1KCmBMWMQVfsR4L3InMdzQjoBrwFHBvv3A7ciMgV4HHgqlYYiC2QCNibFLqtTTygogKuugpUr7bhBA1u28fzzFti8QYSv1qabWm9z7Njc86649FIbLgbr4XovMufxHmSeI3JwsNcfuA4A1VGY/8wsLGTquak0WVOBdPKZ116zgOZggQMeeST1NgYPhtWrYfTo9NpWExYtgv/+d/3xunVw//3ei8xxiorsq7R6dbYtcbLEK8EyjsuAzX4/q/o4quejOgLVtak06ALpVI+SEvM+XRt836rrjfqHP1hA9BEj1reVTVStBxzrfLR2rfcic5zyYAHVWaLr1At2wYZQBwFfIPI8IkchUu2wXy6QTvVIZ/aQCy6ApUvhyVoJp5ga111nHrexrFsHr76aeXucyHg0nRxApCci8xFZiMilca73DhIXT0dkGiL7h659HiQ7tmvrzxch8goiC4LPlnHvrToP1YswL9a+mEPOE8ASRG5EZIdUH6dKgRRhYlUbMDzVGzr1hHR6o/bsCdtvD7fdlt2U8E8+afOqp5xigqhK8ZQp8OOP9te3ffvs2eYkxQUyy1hP7S4s29POQD9Edo4pNRnYDdVOwBnAfTHXu6PaCdUuoXOXApNR7RjUryy8YVRLUX0K1SOwdY93AEcDcxF5I5VHStaD/C7J9hkwNpUbOvWEsDdqeKuON2qDBjYXOXWqCW82+OADE8Z99oF77rHQLOW0aGHh9V56CaZMyY59TlJcILNOV2AhqotQXQM8BvSuUEJ1Jfr7r+DmRFt20Rt4KNh/CDgqskWqS4G7MZH8EVu7H5kqBVKV06NsqdzQceJyyimw8cZw++2Zv3dJCfTubV61Tz8NTZpULnPeedCmDVx2WXZ7uemkpMTWsNYT5yMXyNqnFRQEQ6PlW//Q5a2Ar0LHi4NzFRHpg8jHwPNYL7IcBV5G5IOYdjdH1eJT2udmREHkEEQeAZYC12CC3aXqShXxOUgnN9hgA+jfHyZMgC++yNx9V682cfzxR5g4ETbfPH65Jk0sEMJ778Gzz2bOvtpk2LB6lVHFBbL2+RZKUe0S2sLu5xKnSuVfk6pPo7oj1hMMf/n2Q7UzNkR7HiIHpGygyDaIXI3IZ1hQmy2xZR9bonoeqikNcblAOrnDeefZ0OaIEZm5n6oFT5861cLj7bZb1eVPOQV23NGGW8vKMmNjbbFokS1dqUcZVTbYwJbnukBmjcXA1qHjNljvLT6qbwAdEGkVHC8NPpcDT2NDtgDLEGkNEHwuj9ueyCtYnuKzsd7i9qh2Q/VhVKuVVMMF0skdttkGjjkG7r13fQCC2uT66+GxxyyQ+lFHJS9fUGB15s2z4AZ1jS++gLvvhiOOMKeociererKEpTyjhwtk1pgKdESkPSKFwAnAxAolRLZDggl+kc5AIfAdIs0R2TA43xzLMzw7qDURODXYPxVINISzGnPG2RrVy1CN446eGi6QTm5xwQXw00/w0EPJy9aECRPgyivhpJMsak5U+vSBPfeEq6+2/Ja5QKK5xLVr4fXX4ZJLbL1pu3bWS589u2K50tJ604v0lFdZRLUUGAi8BMwDxqM6B5EBiAwISh0DzEZkOubx2jdw2tkceAuRGVig8edRfTGocwPQA5EFQI/gON79j0R1IqppG95xgXRyi733tkwgw4dXXmeZLj780ELc7b239VYl3tRJAkTghhvgq69g5MjasS9VwnOJy5fbj4u+fc3pqFs3Wz6zxRZw663w8cdw+OEW+ShMPQmn5z3ILKM6CdXtUe2A6vXBuVFByDdQvRHVXYKlHPug+lZwfhGquwXbLr/XtWvfoXowqh2Dz4z9C2dUIEXoKcJ8ERaKVF7LIoKIcEdwfaYInaPUFWFQcG2OCDdl4lmcWmTIEFiwACZNSn/bJSVw5JHQqpWl34rnsZqMgw6CHj1suHXFirSbmBLlEY3WrTPB3nxzOO00ePNNOPZYeOop+PZbC3JwwQWwww7w7ruV17CWlVnowDqOC6STTjImkEGC5QqLSEWIXUR6ONAx2PoDI5PVFaE7tk5mV1V2AW6u/adxapVjjrElFele8rF6tc01/vBD1R6rUfjXv2ws75Zb0mZetRg2rKLYde1qazqXLIH77rMh4Y02qlgndg3r11/bEpsttqjzS1hcIJ10kskeZFdgoSqLVIm/iNSOx6qiqrwLbCxC6yR1zwFuUOU3ANUEHk5O3aFRIxg40PJNzpyZnjaXLrVcle+/bx6rnTrVrL0uXeC440wgl2fpKxfuPYKJ26xZsOWWqQ0bb7453HgjFBdXDNJeB3GBdNJJQQbvFW8R6V4RymyVpO72wJ9EuB74FbhIlamxNxehP9YrpaBAKC4urvaD1FdWrlyZM++lYKed2KdJE5ZfeinzL764xu3tfs45tPjmG37acUc+atnSxCACVb2TpkccQdcJE1hyzjksHDSoxjamSsfbbqP1mjUVfuWuW7uWkgEDWDBkSGqNbbcdu++yC03PP5/3W7SgtEWLKovn0nclzIoVbVm5sj2vvPI6jRplvjecq+/FqSYadNdqewM9DvS+0PHJoHfGlHkedP/Q8WTQPaqqCzob9A5QAe0K+hmoVGVL48aN1anMlClTsm1CRc45R7VxY9Vly6rfxm+/qQ4cuH5AsWlT1ZKSyNWTvpOzzlJt1Ej1s8+qb2N12W23eMH+VDt1ql57M2aoNmyoeuaZSYvm3Hcl4K677BV8/XV27p+r7yVdAL9ohjQjF7ZMDrFGWUSaqExVdRcDTwXP8z6wDmiVRrudbHH++Za0eNSo6tWfPdvm5EaMWJ/AuboZRxJx1VXmEXr11elrMyqXX26fzz5b83i4ALvuCn//u81dvvVW+uzMIB5Nx0knmRTIqUBHEdqLEH8RqR2fEniz7g38pEpJkrrPAAcBiLA9tvD021p/Gqf22XFHW5Jw990mlFEpK4Obb4Y99oDFi6GwcP08XXXzViaiTRsYNMjm7mLXF9Y2t98OHTrAn/+cvjavvtoCNgwYUNnTtQ7gAumkk4wJpCqVFpGqMkeEASKULyKdhIUKWgjcC5xbVd2gzhhgWxFmY847p6pGihDv1AWGDIFly+Dxx6OV/+wzW4bxj39Ar17wl79ULpPuXuSll5qnaHmPLhO8/76lFjv//MprGmtC8+bW454zx9ZP1jFcIJ10kkknHVSZhIlg+Nyo0L4C50WtG5xfA5yUXkudnKFHD9h5Z+stnXxyYu9MVesZDh5sZR580GKndu6cvryViSgqgosvNoF8+23Yd9/0tZ2I4cNhww1tzWO6+ctfbHnINdfA8cfXqTyYLpBOOvFIOk5uI2K9yI8+gjcS5Dpdvtz+oP/tb7b8YtYsOPVUq5vOvJVVMXiwLZe49NLaX0u4ZAmMH2/PG7vGMV0MH24904ED69TaSBdIJ524QDq5z0knwSabxA8c8OyzFmf0xRdtTeLkybbeMdM0b24OO2++abbUJiNH2jBxbS4t2XpruPZai2b01FPpaTMD+Sc32sj8sVwgnXTgAunkPk2bmtPIs89amLQDD7RQdGecYZFxttrKosf8/e/rvVWzwZlnwrbbwkUXwQEH1I4QrF5tXr29e9u9apNBgyygwvnnpyekXgbyTzZoAC1bukA66cEF0qkbnHuuDfn172+9tM6dLSj35ZdbEuNddsm2heYte911MHdu7QnBuHEW4m7w4PS3HUtBAdxzj/X8rrqqZm19+qktH8lA/kmPpuOkCxdIp26w5ZYWZHzWLJsT++UX61Fed50JU65wwAE291nuNJROIVC1YebddrNedCbo2hXOOQfuvNOyoKTKTz/Bv/9tw+Br19q51attXrmW8JRXTrpwgXTqDuHlDI0awQsvZM+WRFx//Xo7051CavJkW34xZEhqsVZryvXXw2abwdln29xnFL791vJttm0L//xnZU/ixx+3+K+14ADkPUgnXbhAOnWDkhL4v/9bf5zuBf/poDx4eGmpHZeV2bBiumwcPtyE6oQT0tNeVDbe2NZETpuWPAfm0qVw4YUmjNddBwcfbGm3CmJWlImYx+8xx6RdzVwgnXThAunUDYYNq5xAOd0L/mtKPBvXrDEhqCkLFsBzz5mzUnVyWNaUvn1tTeo//2kiGMtnn9lQbPv2JuRHH2293QkTYOHCyj1IVRs2f+45GzJOtISnGrhAOunCBdKpG7zzTu0v+K8p8WwEE4moQ5OJuOMOG1Y+55yatVNdRCzk35o1cPbZdBo82HrGH39sa047doQxYyxwwSefWOi9nYN0r4nWoi5ZYv9+TZpA9+4W5q68910Diops6jMNTTl5jgukUzfI1IL/mhDPxvvug5Urbb6tuvz4ow3d9utnSY2zxXbbwRVXwHPP0WLmTNhvPxPBJ56wJSGLFpnXayrLT7p0Meefk06ydZfdusEXX9TIzPJgAT/+WKNmHMcF0nFqlTPOsOHJq66yHmZ1GDPGvHYzsbQjGSedBCIImCAOGmSCdtttth61Omy4oS3ZefhhmDHD1l5OmFBtEzfZxD59mNWpKS6QjlObiFivapttrAeYaremtNSGVw84wNZ+ZpubblrvcFNYaPZtuml62v7rX2H6dBuuPfZY85pdtSrlCDwebi6LiPREZD4iCxGpPPku0huRmYhMR2QaIvsH57dGZAoi8xCZg8jgUJ2hiCwJ6kxHpFemHscF0nFqmxYt4NFHbc6tf//UljZMnGg9tFzoPZZ76ZavZ6wNT+IOHSzIwiWXwOjRsOee9uwpBF5wgcwSIg2Bu4DDgZ2BfojsHFNqMrAbqp2AM4D7gvOlwIWo7gTsDZwXU/c2VDsFW6WkFbWFC6TjZIK99rJlD088YfOSUbn9dmjXzkLLZZtMeRIXFsINN8DLL8M339g7SyECjwtk1ugKLER1EaprsPSDFb+4qivR338hNocgNaFqCaofBvs/Y2kNqzlmnz5cIB0nU/zjH3DIIdYjmjs3efkPP7SweoMGpTfnY3XJtCdxjx6WDLo8KMJvv5mTUBLyRiAzEPw9llZQEAyNlm/9Q5e3Ar4KHS8mnsiJ9EHkY+B5rBcZe70dsDvwXujswGBodgwiLWv+JNFwgXScTNGgAYwdCxtsYI47q1dXXX74cCv7t79lxr5khLx0i6dMqX1P4pISeOyx9UPS69aZw9Krr1ZZbeON7bPeC2QGgr/H8i2UotoltI0OXY4X3qnyfILq06juCBwFVDReZANgAjAE1fII+SOBDkAnoAS4pYaPERkXSMfJJK1bm0jOnm0RZxLx9dc2b3naaTaHmY/EG9JVhcMOM8enBHO5DRuaSNZbgSwttfi2I0fa+0lntKaasRjYOnTcBogTVSJA9Q2gAyKtABBphInjOFSfCpVbhmoZquuAe7Gh3IzgAuk4maZnTxPHkSPh6afjlxk50pxhzj8/s7blEokCLzRvbhGFTjrJ1pjGod5G03n1Vdh9d4toVD70vGaNzXHPnp1d22Aq0BGR9ogUAicAEyuUENkOCQwX6QwUAt8F5+4H5qF6a0yd1qGjPkDGHtQF0nGywb/+ZYvk//Y3+PLLitd+/dUE8ogjbMlDvpIoOMSPP1rv8rHH7B3OmlWpar3L6LFwoeU+7dHDwgQVFlbsQX/5Jey6K5x1lg1NZwPVUmAg8BLmZDMe1TmIDEBkQFDqGGA2ItMxj9e+gdPOfsDJwEFxlnPchMgsRGYC3YELMvVILpCOkw0KC20Ide1aW/8Xjov26KPmvVmLKaHqNA0amLPOq6+aWOy1l3m4hqg3PcgVK+Diiy1i0eTJNrTas2flcoWF8Mc/WsCF7baDa65J2LuuVVQnobo9qh1QvT44NwrVUcH+jajuEizX2AfVt4Lzb6EqqO5aaTmH6smo/jG4diSqGfsF4ALpONliu+1g1KiKjhaq5pzzhz/AQQdl175cp3t362XuvbdFLDrtNIs4RD0QyLIyuP9+G0H4z3/sR9Qnn1jg+6lT43sTN2gA8+aZ5+/QoVb3vvtqHgc4j3GBdJxs8te/WrDv666Dp56yMGszZthSkEzmfKyrbLEFvPKKhfIbO9YSPM+bV7cF8s037TnOPNN+RE2daj3k1sFUXFVxiTt0gPHj4X//s8wqZ51l36kXXlg/JJuF5SF1FRdIx8k2I0bYH8JTToGZM6FxYxNOJxoNG9qQ4ksv2dB0ly50X/Iwjb8vQQ+oA0JQLlhTp9rynwMOgOXLbaj9rbdsnjVV9t3XRPKJJyxcX69ecOihFsovC8tD6ioukI6TbTbYAO688/fhQcrKbG7NSY0ePUwA9tiDY589mVc5qHpCkOke1lVXWa9xn30sKfjQoTB/viXGrskogojFtJ03z4LJf/ihecCOHm3LQ+6/3wLOR6WkhB0gC8lIs0dB8iKO49Q6zzxjPaGyMptLGjYM7ror21bVPbbcEl57jbmH/52dX73TlqnffbfFtC0qgo02Wr9tuGH844ceMsHKxL/Bo49WDD345puwxx7pvUdhoTl8nXqqzdvOmGHnf/vNhmRbtbJg+m3b2md4a9sWNtvMxHbYMDbIs06VC6TjZJvyIODlzhTlQcCvvDK7+R/rKgUFNG1UyhoKKKTUfnA0a2YxbX/+GZYtgwULzEP0559tCDIeo0db0PRttkm/jT/+aGthx4xZf65hQztOt0CW8+uv1jMNU1AAhx8O335r115+ef1IRjmNG9v8Z+xypDwgr34NOE5Okqkg4PlCSQnbTH7AxBHs3X71lUXfee01mDbNPEK//trEYO1a8+j54gs4/nho1MjqlZba2sLJk9Nr3/PPm5fygw+uTx0GtZMdJUy871mDBtZznjQJ5syxHwzff29D1c8+a0P/559v4p1KFpp6gguk42SbTAcBr+8MG4ZoCj84CgqgZUsTxokT16fzAutlHnKIOc8sWVIzu77/3hyxjjjC7tenjwlUVDtrSpTvmYjZtttucOSRMHAgXHCBPbsLpOM4Gacqt30ndd55hwZrq/GDI14Pq1Ejy0k5cSLsuCPcfHNFAY3Ks8/CLrvYnOOVV1ov9tNPM/vDqLrfs3jvJU9wgXQcp37x0UesXaMIyrBrUxCCRD2stWtt+LF7d0tZ1qkTFBdHs+Xbb6FfPwsTt/nmtpTj2mttXq+u/DBKFBM3D3CBdByn3tGokU2tpRQsoCrB2nZb60VOnGhOPd27W7D0quKePvmkhYibMMFEcepUE9e6Rui9fAAJPJrqJxkVSBF6ijBfhIUiXBrnuohwR3B9pgidk9UVYagIS0SYHmy9Ytt1HCe/GDfOdOz22815ddy46PXatbOpwbj1/vIXS3Z95ZW2CH+HHeD223lkbCl7tilBul/NoVvO5ou9joPjjjMP2A8+sPLlzj+p3K+6dtZSPXMHziNUNSMbaEPQT0G3BS0EnQG6c0yZXqAvgAro3qDvJasLOhT0olRsady4sTqVmTJlSrZNyDn8ncQnl9/Lww+rNmtWsRvYrJmdT2u9BQtUe/ZUBZ0lf9QJHKVliK6isf5KoX54/L9V167Nvp1prddMNUOakQubqGbGM0mEfYChqhwWHF9mAs2/Q2XuAYpVeTQ4ng90A9olqivCUGClKjdHtaVJkyb666+/puGp6hfFxcV069Yt22bkFP5O4pPL76VdO1uxEUtBAWy/feJ6n3xSMalKpHqq7DT/GW5fN4g2mJdrGUJ3pvBOwYHpv1/W6zVH9Ze8CRKcyUABWwFfhY4XA3tFKLNVhLoDRTgFmAZcqMoPsTcXoT/QH6CgQCiOOsmeR6xcudLfSwz+TuKTy+/lyy8PBCr/DS8tVTbd9JuE9ebO3bRa9SbMO4pDmcQZPEABZaylEX0Zz5ulB9TK/XKlXl6Qqa4q6HGg94WOTwa9M6bM86D7h44ng+5RVV3QzYMh2Aag14OOSWaLD7HGJ5eHzbKFv5P45PJ7adu24jBi+da2be3U67LVUl1FkwqVfqGpdmlTklN2pqdefg2xZtJJZzGwdei4DbA0YpmEdVVZpkqZKuuAe4GuabbbcZw6xPXXV3YladbMztdGvXE7DkOouE6wAWU8vEPVC/4zbWc66+UNmVJi0ALQRaDtQ442u8SU+XOMk877yeqCtg7VvwD0sWS2eA8yPrncK8gW/k7ik+vv5eGHrecjYp/JHFFqVK9Tp/hds06dcsvONNTLtx5kxpx0AIIlGLcDDYExqlwvwgATakaJIMAIoCe23uZ0VaYlqhuc/y/QCVDgc+BsVapYnOROOonIZceLbOHvJD7+XuJT39+LiKxS1ebZtiNTZDSbhyqTgEkx50aF9hU4L2rd4PzJaTbTcRzHcTySjuM4jpMmRHoiMh+RhYhUCgaDSG9EZiIyHZFpiOyftK5IESKvILIg+GyZkWfBBdJxHMdJByINgbuAw4GdgX6I7BxTajKwG6qdgDOA+yLUvRSYjGrHoH5l4a0lXCAdx3GcdNAVWIjqIlTXAI8BvSuUUF3JeseX5pjvSLK6vYGHgv2HgKNq7QlicIF0HMdxItEKCoKh0fKtf+hyokAvFRHpg8jHwPNYLzJZ3c1RNcdL+9wsHc8ShYw66eQKv/32m4rI6mzbkYMUAHGCUeU1/k7i4+8lPvX9vTRFtUuCa/HC7VReJqH6NPA0IgcAw4BDItfNMHkpkMCHmvgfOW8RkWn+Xiri7yQ+/l7ik+fvJUowmPWovoFIB0RaJam7DJHWqJYg0hpYnl6zE+NDrI7jOE46mAp0RKQ9IoXACcDECiVEtkNEgv3OQCHwXZK6E4FTg/1TgWdr+Tl+J197kI7jOE46US1FZCDwEkFAF1TnIDIguD4KOAY4BZG1wGqgb+C0E7+ucQMwHpG/AV8Cx2XqkTIaSSdXEJH+qjo623bkGv5eKuPvJD7+XuLj76V+kZcC6TiO4zjJ8DlIx3Ecx4mDC6TjOI7jxCGvBFJEeorIfBFZKPHiBOYpIvK5iMwSkekiMi3b9mQLERkjIstFZHboXJGIvCIiC4LPjMWBzBUSvJehIrIk+M5MF5Fe2bQx04jI1iIyRUTmicgcERkcnM/770t9Im8EUuLE+pPKcQLzme6q2imP13ABPIilWgtzKTBZsxAHMod4kMrvBeC24DvTSVUrZdqp55QCF6rqTsDewHnB3xP/vtQj8kYgCWL9qeoiTRQn0MlrVPUN4PuY01mLA5krJHgveY2qlqjqh8H+z8A8LDRa3n9f6hP5JJDR4gTmJwq8LCIfSMXYig5srkEcSM1wHMg6wEARmRkMwebtUKKItAN2B97Dvy/1inwSyJyM9Zcj7KeqnbHh5/PEYiQ6TlWMBDoAnYAS4JasWpMlRGQDYAIwRFVXZNseJ73kk0CmFicwj1DVpcHncuBpbDjaMZaJxX9EMhwHMpdR1WWqWqaq64B7ycPvjIg0wsRxnKo+FZz270s9Ip8EcirQUUTaS6I4gXmIiDQXkQ3L94FDgdlV18orshYHMpcpF4GAPuTZd0Ysnuj9wDxVvTV0yb8v9Yi8iqQTuKLfThDrT1Wvz65F2UdEtsV6jWCxeR/J1/ciIo8C3YBWwDLgauAZYDywDUEcSFXNK4eVBO+lGza8qsDnwNnlc2/5gIjsD7wJzALWBaf/ic1D5vX3pT6RVwLpOI7jOFHJpyFWx3Ecx4mMC6TjOI7jxMEF0nEcx3Hi4ALpOI7jOHFwgXQcx3GcOLhAOk4dR0RURI7Nth2OU99wgXScGiAiDwYCFbu9m23bHMepGQXZNsBx6gGvAifHnFuTDUMcx0kf3oN0nJrzm6p+HbN9D78Pfw4UkedFZJWIfCEiJ4Uri8gfReRVEVktIt8HvdIWMWVODZJa/yYiy0TkwRgbikTkCRH5RUQWxd7DcZzUcYF0nNrnGixGZydgNDBWRLoAiEgz4EVgJRbwuw+wLzCmvLKInA3cAzwA7Ar0AubE3OMqLO7nbsDjwBgRaVtrT+Q4eYCHmnOcGhD05E4Cfo25dJeqXiIiCtynqmeF6rwKfK2qJ4nIWcDNQJsg8S4i0g2YAnRU1YUishh4WFXjZqcP7nGDql4WHBcAK4D+qvpw+p7WcfILn4N0nJrzBhCbaPrH0P47MdfeAf4c7O8EzCwXx4C3sQDYO4vICiyx9+QkNsws31HVUhH5Bk/W6zg1wgXScWrOKlVdWM26QuLE3Ur8RN/xWBunrk+hOE4N8P9AjlP77B3neF6wPxfYrTwnZ8C+2P/Neaq6DFgCHFzrVjqOUwHvQTpOzWksIlvEnCtT1W+C/aNFZCpQDByLid1ewbVxmBPPWBG5CmiJOeQ8FeqVXg/cJiLLgOeBZsDBqnpLbT2Q4zgukI6TDg4BYpMFLwHaBPtDgWOAO4BvgNNVdSqAqq4SkcOwRN7vY84+zwKDyxtS1ZEisga4ELgR+B6YVEvP4jhOgHuxOk4tEniYHqeqT2bbFsdxUsPnIB3HcRwnDi6QjuM4jhMHH2J1HMdxnDh4D9JxHMdx4uAC6TiO4zhxcIF0HMdxnDi4QDqO4zhOHFwgHcdxHCcO/w/PifgHVc6qJwAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# extra code – this cell plots performance scheduling\n",
"\n",
"plt.plot(history.epoch, history.history[\"lr\"], \"bo-\")\n",
"plt.xlabel(\"Epoch\")\n",
"plt.ylabel(\"Learning Rate\", color='b')\n",
"plt.tick_params('y', colors='b')\n",
"plt.gca().set_xlim(0, n_epochs - 1)\n",
"plt.grid(True)\n",
"\n",
"ax2 = plt.gca().twinx()\n",
"ax2.plot(history.epoch, history.history[\"val_loss\"], \"r^-\")\n",
"ax2.set_ylabel('Validation Loss', color='r')\n",
"ax2.tick_params('y', colors='r')\n",
"\n",
"plt.title(\"Reduce LR on Plateau\", fontsize=14)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### tf.keras schedulers"
]
},
{
"cell_type": "code",
"execution_count": 86,
"metadata": {},
"outputs": [],
"source": [
"import math\n",
"\n",
"batch_size = 32\n",
"n_epochs = 25\n",
"n_steps = n_epochs * math.ceil(len(X_train) / batch_size)\n",
"scheduled_learning_rate = tf.keras.optimizers.schedules.ExponentialDecay(\n",
" initial_learning_rate=0.01, decay_steps=n_steps, decay_rate=0.1)\n",
"optimizer = tf.keras.optimizers.SGD(learning_rate=scheduled_learning_rate)"
]
},
{
"cell_type": "code",
"execution_count": 87,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1719/1719 [==============================] - 2s 864us/step - loss: 0.6808 - accuracy: 0.7683 - val_loss: 0.4806 - val_accuracy: 0.8268\n",
"Epoch 2/10\n",
"1719/1719 [==============================] - 1s 812us/step - loss: 0.4686 - accuracy: 0.8359 - val_loss: 0.4420 - val_accuracy: 0.8408\n",
"Epoch 3/10\n",
"1719/1719 [==============================] - 1s 809us/step - loss: 0.4221 - accuracy: 0.8494 - val_loss: 0.4108 - val_accuracy: 0.8530\n",
"Epoch 4/10\n",
"1719/1719 [==============================] - 1s 828us/step - loss: 0.3976 - accuracy: 0.8592 - val_loss: 0.3867 - val_accuracy: 0.8582\n",
"Epoch 5/10\n",
"1719/1719 [==============================] - 1s 825us/step - loss: 0.3775 - accuracy: 0.8655 - val_loss: 0.3784 - val_accuracy: 0.8620\n",
"Epoch 6/10\n",
"1719/1719 [==============================] - 1s 817us/step - loss: 0.3633 - accuracy: 0.8705 - val_loss: 0.3796 - val_accuracy: 0.8624\n",
"Epoch 7/10\n",
"1719/1719 [==============================] - 1s 843us/step - loss: 0.3518 - accuracy: 0.8737 - val_loss: 0.3662 - val_accuracy: 0.8662\n",
"Epoch 8/10\n",
"1719/1719 [==============================] - 1s 805us/step - loss: 0.3422 - accuracy: 0.8779 - val_loss: 0.3707 - val_accuracy: 0.8628\n",
"Epoch 9/10\n",
"1719/1719 [==============================] - 1s 821us/step - loss: 0.3339 - accuracy: 0.8809 - val_loss: 0.3475 - val_accuracy: 0.8696\n",
"Epoch 10/10\n",
"1719/1719 [==============================] - 1s 829us/step - loss: 0.3266 - accuracy: 0.8826 - val_loss: 0.3473 - val_accuracy: 0.8710\n"
]
}
],
"source": [
"# extra code – build and train the model\n",
"model = build_and_train_model(optimizer)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For piecewise constant scheduling, try this:"
]
},
{
"cell_type": "code",
"execution_count": 88,
"metadata": {},
"outputs": [],
"source": [
"# extra code – shows how to use PiecewiseConstantDecay\n",
"scheduled_learning_rate = tf.keras.optimizers.schedules.PiecewiseConstantDecay(\n",
" boundaries=[5. * n_steps_per_epoch, 15. * n_steps_per_epoch],\n",
" values=[0.01, 0.005, 0.001])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1Cycle scheduling"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `ExponentialLearningRate` custom callback updates the learning rate during training, at the end of each batch. It multiplies it by a constant `factor`. It also saves the learning rate and loss at each batch. Since `logs[\"loss\"]` is actually the mean loss since the start of the epoch, and we want to save the batch loss instead, we must compute the mean times the number of batches since the beginning of the epoch to get the total loss so far, then we subtract the total loss at the previous batch to get the current batch's loss."
]
},
{
"cell_type": "code",
"execution_count": 89,
"metadata": {},
"outputs": [],
"source": [
"K = tf.keras.backend\n",
"\n",
"class ExponentialLearningRate(tf.keras.callbacks.Callback):\n",
" def __init__(self, factor):\n",
" self.factor = factor\n",
" self.rates = []\n",
" self.losses = []\n",
"\n",
" def on_epoch_begin(self, epoch, logs=None):\n",
" self.sum_of_epoch_losses = 0\n",
"\n",
" def on_batch_end(self, batch, logs=None):\n",
" mean_epoch_loss = logs[\"loss\"] # the epoch's mean loss so far \n",
" new_sum_of_epoch_losses = mean_epoch_loss * (batch + 1)\n",
" batch_loss = new_sum_of_epoch_losses - self.sum_of_epoch_losses\n",
" self.sum_of_epoch_losses = new_sum_of_epoch_losses\n",
" self.rates.append(K.get_value(self.model.optimizer.learning_rate))\n",
" self.losses.append(batch_loss)\n",
" K.set_value(self.model.optimizer.learning_rate,\n",
" self.model.optimizer.learning_rate * self.factor)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `find_learning_rate()` function trains the model using the `ExponentialLearningRate` callback, and it returns the learning rates and corresponding batch losses. At the end, it restores the model and its optimizer to their initial state."
]
},
{
"cell_type": "code",
"execution_count": 90,
"metadata": {},
"outputs": [],
"source": [
"def find_learning_rate(model, X, y, epochs=1, batch_size=32, min_rate=1e-4,\n",
" max_rate=1):\n",
" init_weights = model.get_weights()\n",
" iterations = math.ceil(len(X) / batch_size) * epochs\n",
" factor = (max_rate / min_rate) ** (1 / iterations)\n",
" init_lr = K.get_value(model.optimizer.learning_rate)\n",
" K.set_value(model.optimizer.learning_rate, min_rate)\n",
" exp_lr = ExponentialLearningRate(factor)\n",
" history = model.fit(X, y, epochs=epochs, batch_size=batch_size,\n",
" callbacks=[exp_lr])\n",
" K.set_value(model.optimizer.learning_rate, init_lr)\n",
" model.set_weights(init_weights)\n",
" return exp_lr.rates, exp_lr.losses"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `plot_lr_vs_loss()` function plots the learning rates vs the losses. The optimal learning rate to use as the maximum learning rate in 1cycle is near the bottom of the curve."
]
},
{
"cell_type": "code",
"execution_count": 91,
"metadata": {},
"outputs": [],
"source": [
"def plot_lr_vs_loss(rates, losses):\n",
" plt.plot(rates, losses, \"b\")\n",
" plt.gca().set_xscale('log')\n",
" max_loss = losses[0] + min(losses)\n",
" plt.hlines(min(losses), min(rates), max(rates), color=\"k\")\n",
" plt.axis([min(rates), max(rates), 0, max_loss])\n",
" plt.xlabel(\"Learning rate\")\n",
" plt.ylabel(\"Loss\")\n",
" plt.grid()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's build a simple Fashion MNIST model and compile it:"
]
},
{
"cell_type": "code",
"execution_count": 92,
"metadata": {},
"outputs": [],
"source": [
"model = build_model()\n",
"model.compile(loss=\"sparse_categorical_crossentropy\",\n",
" optimizer=tf.keras.optimizers.SGD(learning_rate=0.001),\n",
" metrics=[\"accuracy\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's find the optimal max learning rate for 1cycle:"
]
},
{
"cell_type": "code",
"execution_count": 93,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"430/430 [==============================] - 1s 1ms/step - loss: 1.7725 - accuracy: 0.4122\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABAZUlEQVR4nO2dd5gc1bG331qlVRaKIKEECmQQQQiTBDY52aRLtMGAjAEb30sONsaYZAwPl2CCsT6wScYYc4UQwWANiCBAAiQQEkpIKKEcUFhJqz3fHzWH7untSavZ2Zndep9nnk6ne870zvZvqupUHXHOYRiGYRieiobugGEYhlFamDAYhmEYKZgwGIZhGCmYMBiGYRgpmDAYhmEYKZgwGIZhGCk0b+gObC2dOnVyAwYMaOhuNBrWrVtH27ZtG7objQa7n4Wl3O7nzJmwejW0bw+DBgX7P/kEunSBpUuhVy/Ydtvi923ixInLnHPd4o6VvTD06NGDCRMmNHQ3Gg2JRILhw4c3dDcaDXY/C0u53c/jj4eXX4b994d//zvY37YtnHEG3H8/XHIJXH998fsmInPTHTNXkmEYRj3h84ejecTOQfPkz/KamuL2KRdMGAzDMOoJ/9CPPvxragJh2LKluH3KBRMGwzCMeiKTxdCsma6bMBiGYTQhvKUQFYaaGhBRcTBXkmEYRhPCC0L04e8cVFToq0lbDCJSKSIfisgkEZkiIjfHtBERuU9EZorIZBHZu1j9MwzDKDS5WAylKAzFHK66ETjcObdWRFoA74jIK8658aE2xwADk6/9gYeSS8MwjLIjU4zBXEmAU9YmN1skX9HJIE4C/ppsOx7oJCLbFauPhmEYhSRuVJIXiVJ2JRU1wU1EmgETgQHAg865DyJNegHzQtvzk/sWRa4zAhgB0K1bNxKJRH11ucmxdu1au58FxO5nYSm3+7ly5V5AJ1avXk0i8QngRWI4c+d+hXPbM3fuYhKJmQ3Yy9oUVRicc1uAvUSkE/AvEdnNOfd5qInEnRZznUeBRwEGDx7syikTstQpt8zSUsfuZ2Ept/vZoYMu27fv+F2/vYWwww79adUKevbcnuHDt2+YDqahQUYlOedWAQng6Mih+UDv0Pb2wMLi9MowDKOwxI1K8usipetKKuaopG5JSwERaQ38AJgWaTYK+HFydNIwYLVzbhGGYRhlSNyopHCMoVSDz8V0JW0HPJGMM1QAzznnRovIxQDOuYeBMcCxwExgPXB+EftnGIZRUOJGJZWDxVA0YXDOTQaGxOx/OLTugEuL1SfDMIz6JNuopFLNY7DMZ8MwjHoim8VQqq4kEwbDMIx6IlOMoZRdSSYMhmEY9UTcqCRzJRmGYTRhzJVkGIZhpJBtuKq5kgzDMJoY2RLczJVkGIbRxCjXBDcTBsMwjHqiXBPcTBgMwzDqiUwJbhZ8NgzDaILEWQwWfDYMw2jCxMUYLPhsGIbRhMklwc1cSYZhGE2IbBaDuZIMwzCaGNliDOZKMgzDaGLEjUqykhiGYRhNGBuVZBiGYaRgo5IMwzCMFDKNSjJXUpnw6quwcmVD98IwjMaCVVctccaNg8ceS398zRo45hg4/vji9ckwjMaNzcdQQsyeDUcfDUuWBPsOPRQuuggWLIg/Z9UqXb73Xr13zzCMJkK2BDezGIrIY4/Ba6/BHXcE+/wf4y9/CfatWQObN+u6FwbDMIxCYcHnEmLhQl2OHq3Ldev0jwAwcWLQrmNHOOUUXTdhMAyj0OSS4GaupCLxySe6nDEDVq+GadOCP4YPLvvtl17S5erVwfnffqvLhQvhhhtKU9ENwyh9siW4mSupSGzcCF98AYMH6/bs2TB1qq7vvDOsWKHr69ennhe2GGbP1uUhh8BttwXnG4Zh5EMmi8FcSYCI9BaRsSIyVUSmiMjlMW2Gi8hqEfk0+fpNtuuuXt0iZfubb6C6Go44QrdnzYJFi3R9zz0DYfBWgScsDAsWqHDMmqXbURGJY9Qo6NsXNmzI3tYwjKaBTe2ZnWrgCufczsAw4FIR2SWm3Tjn3F7J1++yXXTJksqUG+sf8HvvrctZs2D5cmjRAnr3VpHo0iU11hA+D2DZMnVDedauzf7h3nsPvv4avvoqdf/rr6cfCWUYRuMmblSSuZJCOOcWOec+Tq5/C0wFem39dWHp0mDbP+D79IGuXdUttGKFikHnznpsxQq49dbgnC1bUmMMS5fC4sXBdtS6iGPePF1+/XWwb/16OOooHTqbjbffhsmTs7czDKN8yMViaNLCEEZE+gFDgA9iDh8gIpNE5BUR2TWX6911F9xyi/6y9w/4Tp1gxx3hyy9VCDp3DoQB4IPQO69YoYKy3XZqWRRKGD77TJdh6yPMDTfAm2/q+qGHqqvLMIzGQ7kmuDUv9huKSDvgn8CvnHNrIoc/Bvo659aKyLHAi8DAmGuMAEbo1j7cfbeuvf76UpYsqQTaM23aeLbffnteemk7Bg5cS/PmjkWLFgCqNeE/xujRHzFjRj9atmxDhw7NmTx5BatXrwd2BODcc+Gppxax556refrpPjz22ARatkz9a86YsT/QmnHj5tKr19dMntyRpUsrgUF07FhFIjE+pX1NDdx223Buuw2OPPIbYFsAEokEX3zRgfXrm7Hvvpnrc3zxRQcmTerImWfOy9guH9auXUsikSjY9Zo6dj8LS7ndz+rqg4FmVFdvIZEYB8CXX7YH9mHKlM9YtGgbNm7sQSLxboP2sxbOuaK9gBbAa8D/5Nh+DtA1c5t9nOpx6mvpUudeeinYPukk5958s3Y7cO6uu3Q5bJhze+zh3IknOnfllfFtwbl33nHuxhudmzfPOeecq652rnlzPXb66c61a6fru++uy/79tV1NjXMjRzq3YIH2L+7aNTXB+pYtLiPNmmm7b7/N3C4fxo4dW7iLGXY/C0y53c/WrfV/tLIy2PfBB7pv9GjnLr/cuY4dG6ZvwASX5rlazFFJAvwFmOqcuydNm22T7RCRoaira3ku1+/RI3W7Y0d1zzRrpttRV1KYq67S5eLF0K1b4Erq0ydIjAtz773w+9/DJZcE51VX6/qrrwbBau9KWrRIH/XTpsFPfwq77ZbqqgoTDoKffHL6dvPnB+v+fQzDKC2siF52DgTOBQ4PDUc9VkQuFpGLk21OBT4XkUnAfcAZSWXLyjnnpG63aAHt28OuyShFly4qFpn4/e9ThWHbbaGyUo8NGwavvAK9esHzz+u+xYs1fuDjC127apmNMJ07Q1UV3H8//Oc/um/lSnjqqfg+TJkSrP/f/8G//lW7zYwZOsLKf6E+/TTz5zIMo2HINiqpyQefnXPvOOfEObeHC4ajjnHOPeycezjZ5gHn3K7OuT2dc8Occ1lL2nXvXsWf/6yJaHfdVfv4zjvrsnNnfdC3agU33RQc79JFl+edB2edpUHrmTN1mGnXrkEtpbPP1tFFBxwQnPvhhzBoEBx7rG6fdFLt9//JT3R5+eXw5JNq2bRrBw89pPt32AGOPDJo74PR++2ny+Uhe+nrr+Hjj/kupuJ5773UXySGYZQGmSyGUg4+l33mc6dOm7nwQmjZEg47rPbxvn11uWkTtG6tv97/+7+D4wMGpLbbYYfgWGVl4CLq2lWXp59e+z180typpwb7fv1rXR5zTLBv/HgYPhyOOy61mutrrwVJeK+/rsvHHlMLx7uSxo3TPu6zDzzySOr7P/mkWiRxjB2rSX+eKVM0O9wwjPrHpvYsAXr2rL1v33112a5dsK9Dh2C9f39d9uuny8su03yCf/4ztTqrtyxOOCH+vSsrU4Xpd7/TP/hBB8HA0Liq7t1TxcJft0cP2H57FYpmzVSwevRQYbjrLrj22vj3veYaGDIE/v731P1PPgnf+x4cfrgKEei1dtsNfvGL+GsZhlE4tmwJHvrmSmpAunevve/UU9VP/8tfBvvCAWUvCN5iaN0adt9dA7/hB7q3GCorNZv66ad1+/JkYY9mzdRNdeut8OKLuq+iQq8Xjid07qxi4WnePOiTdyn94AfQpo1+niVL4Oqr088T0bOnTi40fryKg5+MaORIeP99XZ80SZfLlunyrbfir2UYRuGYNk2X/fplTnDz4xBLiUYlDH4EUhgR+OEPNRgdx847a5uBtbIlUvHCAOpuOvNMdTN5C2LTJl1ef33tWEOnTsH6NtukuqvCeGE491xdeoshzFFHwd/+Flyza1etC1VTA2ecoZMRrV8P74aGRXfurIH1Aw/U7bj7ZBhGYfFld3y8MBqI9iUxwvtKhaInuNU3H32kI4uyceed6ms/6yzNON5++8ztvcsnTLNmml0NQZA6jm22CdY7d9YvxNVX1x4Ke/rp6kLydZ569NAYgadXLx0OC+oqeu01HXnlA+yed98NhAp0pJSPeYCargsXqrVRXa2jnKLXMAxj65gwAdq2Df63nNP/+ajFAPo/WUo/2BqdMPiYQjauvjpYz6UURZs28ft799ZlOKAdJTxM1udS3Hln7XYiGlz29OgRBLYhVXx8PKVFCxWt9u2D0h2+3Mett6qQjBuX+j7Tp6vIbN6slsTNN6vZ60uVG4ax9Xz2GeyxR/DAj7MY/DGzGMqMt99W/306mjXTUtstW6ZvE3Zjha2HbERjJuHRRPfcow/yI4/UL9gOOwSxhI8/1uUVV8DQoUEJ8ijz5gUiEp7DwjCMrWf9ev3RFnYX+ZgCpLqSSi0A3ahiDPXBwQcHmdHpqKwM/sDZSJd9HUd0lFXYndWpk45I8u/rR1eBCkOXLhoM32239NefNSu4pp+cyDCMwlBdrULgXcZRiyHqSkpHTQ3813/BO+/UX1+jmDAUmXwsBu+mArUMxoxJ3zYc0J47VyvFgib1pWP27OALOW2ajmI67LDhOc1YN306nHiiJgOCTlS0ww42UZFheKqrddRhVBjiLIaaGq24cNZZtas5f/MNPPdc+qHy9YEJQ5HJRxjCAfFbbsns6jnwwFRrxAsDaPwhjtmzdTgs6PSlf/yjrr/8spbZCE9QNGOGWjBXX63njBmj82Xvt59+0ceP10mKwmXHDaMps2WLCkN05FG64PPvfgfPPKNDzcP46geZBrgUGhOGIpMpFhElPEQ2m6CcfLJ+gbxrKCwMt98ef86sWYEwTJoUDK+76ipNmttrL53PAuCNNzQ7+667dC7s6dN1/6pVam34WerCxf1AR0f9+tepxQHD/UqXn2EY5U46iyFd8Dnd0FU/EZkJQyMk3a/2TISHs+ZqafjkubD1cOml8T7M11+Hzz/XEUpr16oLyjNwoA5zHTYMEgkVjS5dNLHvyy+11pPPpUgkdPgr1J7GdNQoHfl0zTWp+2tq4MYbNSfDMBoj6WIMcRZDdXUgDNFkN5+Y6svzFAMThiIxYwY5+e7Tka0yrOf883UZDYZXVOgX7OJkHduf/SzIdTjzzPD76M4//1lHLPXooQUEJ07UobSnnx7kbhx9tLq7rr5aLQqItxigdnB71arAr2oYjRFvMUQtgbDF4Ks3b9wYCEg6i6GYQ1pNGIpEjx6w0051Pz9d5naUE0+EJ57QDOwoXboEcYthw4JM686dtQjf3/8Ov/71F5x2mloe/ftrFvbChRpz2Htv/YVz1llBnx56KDUW8cEHajVcfLGWC/EF/N54A154IWjn/abelWUYjY1sweeKikAYqqqyu5IgNXG1PrE8hhJn3DjN5s4VEfjxj9Mfv/JKdUude67GIUaN0tFPfj6LRGIVV1wRtA8PmR00SJc/+YkGw08+Wes0HXWUZmGDXm/UqOCccI2q007TobR77hmYx2YxGI2VqMUQF2MIC0O6ZDf/vwL6Q6tPn/rrs8eEocQ56KDUontbS6tWwcxzRx2lQWTvGoqjV69g3Rcc3HHHwH8KWsbDC0OU8CRCLVtq8Pr88wP3lQmD0Vjxo5KiLqLwcNVWrXS9qipoFw0yh/9HFi4sjjCYK6mJM3Bg5uS8OGGA1LouF12kyzvuCNb9qKhPPlG31ezZmoE9aZJWj/Vf9uXLixtUM4xi4X88RbOb07mSvCCsW5d6naVLA9FYvbp+++wxi8HISNiVlK7Q4J57po6kOOMM/SIffrgm6/Ttq/GKXXcNigCGWb689pzdhlHueFdS69a6XVWly3SuJJ8cun596nWWL9f/oTlzaie/1RdmMRgZCZcMzzUAfvjhqdaFF5ddd40fi33rrcGvqQ0bgppPhlHOeGHwBTj9gz+dxeCPRy2GjRuDitHhgR71iQmDkZFoafBcCVsAu+6qy3R1m+6/X8uIg+Y87LUX/OlPdXtfwygVohaDtwSyWQxRYdi8OchjMovBKBkee0ynOs2HcJlyXwo905wPvgy5n2Do7rth5Uqd37rUKk8aRi744LP/X/DCEGcxbNyY3pW0eXOQsGoWg1EyXHCBDk2tK95iaNUqmJ1uv/00MW7ZMrjpJs2m3rhR59sGrbt0002aD3HTTfpP9sYbmqdhwWqj1KmpUQFo1qy2Kylfi6G6Wuesb9483mL45htNPo0ml2YiKj5RTBiMeqd5aIjDI4/A449r0b0779SkuwED9J/l/vvVSjj6aP2nevRRPefWWzVT+4gjtHDfhx9quzvuMJEwShP/vYxzJeUbY9i8WeN77drFC8Nf/qL5QQ89lHv/jjsu83ETBqPeWLy49pzVrVtrglx4iOyAAbq86iq1Kn71K93euBH+5390yOxzzwXt//1vuOEGuO46GD26Xj+CYdSJsDDkajH4UUtxrqQWLbTeWpwrKZwXkY6XXoIHHgi2/Xulw4TBqDe6d689C10cAwcG62+8oXkOnksuUTeW/6XUurUKw8qVuj1tWuq1nNOyxSNH1i5GZhjFIheLIZrgls1iCE/fCzBliv5gCk/847nvvtQpi088EX7xi2DbhMEoecKVYA88UP2yo0frQ3/HHVMnIbr0UnVD+QmCXnpJq8B6Xn5ZYyIXXGDDXo2Gww+YyBZ8rqjQh35dXEn7768zu/kh4GGL4csvMxftLBlhEJHeIjJWRKaKyBQRuTymjYjIfSIyU0Qmi8jexeqf0XCIwIgROm+D/3Ifd1wwMVFYGI47Tv/pJkzQ7ffe01FPo0apZXHzzUFbK9BnNBTeYsgWfAZ1J6VLcHMu1WJ4/XWtOQaBgHirOWwxbNyo56WbwyE8f3wcxcx8rgaucM59LCLtgYki8m/n3BehNscAA5Ov/YGHkkujkfPII+mPheezPuAA/WfbsgUuvFBN8YcegpNOCtr8+Mfw17/qiKfp0zVjOzx81jDqm1yDz1BbGNatC3IgvOXhLQaA55/XZd++OoeKt4zDFoN/8K9fH1+yv2QsBufcIufcx8n1b4GpQK9Is5OAvzplPNBJRLbDaNJ4YTjtNBWCN9/U2MOvf60BtaFDU9uffbYuFy9Wq+Oww2pfc+HC/Ib3GUY+hIWhRQtd5mIxdO+ux7/6So/5X/wtWgQC4/F1zGbMSL0epApDGC9K2YShQWoliUg/YAjwQeRQL2BeaHt+ct+iyPkjgBEA3bp1I5FI1FdXmxxr164tyfv5xBNt6NGjikRC/6tOO00L882eDb16DQC0kNM222yiouJ9RA7hlVcWA9vy4YcwZsw4mjev4brr9uDMM7/mqqv2pLJyC6+8Mq5e+12q97NcKZf7uWhRJTCMGTOmkkgspmXLg5g+fRGJxCymTdsOGMz48e8xa9YmnBvKnDlr2bixO/37L2fJki4899xnHHjgctatawYczNy5s/jyy86ApkC/+WaC5cv3Bjp8957Tp88lkVBFWbBgN6Ar//nPeHr1qgKGJ897i+bNHevXH5z5AzjnivoC2gETgZNjjr0MHBTafhPYJ9P1Bg0a5IzCMXbs2IbuQt7cd59z4NzZZzv3wQe6r3Nn5wYN0v3+tc02qdtQ/30rx/tZypTL/ZwxQ79fTz6p2z16ODdihK4//LAeW7BAt3ff3bmjjtJ9N9ygy9tv12PLlun2vfc6N2BA8L1dvty5wYNTv8tXXBG8v7/e5Mm67dusWeNcTY1zIs4BE1ya52pRLQYRaQH8E3jKOfdCTJP5QO/Q9vbAwmL0zShfLrxQA81XXQUdkj+gunTR+AKoyb1gQTDE1dO8uWZa77573WtCGUYc4eAzaIwrWkQv7EpasULXe/TQmNgXychr2JU0dGgwGm/58to5Df76kOpKCg/brqpSd2y2odzFHJUkwF+Aqc65e9I0GwX8ODk6aRiw2jm3KE1bwwDU93rLLYEoAHTtGhxLJLQUuGf58iBres894d57i9lboykQjjGAfg8zBZ99ImjHjhoX8z9q/HVatNB52P/4R91evrx2FnQ4bhAWhvAIpHAiXSaKmcdwIHAucLiIfJp8HSsiF4tIcop6xgCzgZnAn4FLitg/oxHRpYsud9xRM6ufeUaF4/TTNW9i222DtmXgsjbKjKgwtGkTX10V9Bd8WBi23TYYah22GNq0CWZzzGYxhLOow3kRVVXZh6pCEYPPzrl3gIwGu/rCuLQ4PTIaM/6fITwt6rJlgWkfLgu+caP+I37ve3DPPZolahhbQ5zFEDcfA6jF4B/WHTvqyKQ4YYAgGXTBAhWYsODEWQzr1tUWhlKzGAyjaPikuOuuC/a1aBH8M4aFYd48+M1vYNYsLaVhGFtLOPMZMlsMvl4SBMKwbp22jwqDt4TnzNFleFbFdDGGsGWxYYMJg9GEufdeHd+dbuL0sDB88YVWfIXa5QgMoy7kEnwOWwyeDh2C2dqWLq0tDJ066Xlz5+p279BQHX/9OXNSs6jrYjHYnM9Go6Rdu6Bqaxz+n8+zcaOa6WPHwosv6nSkffqkxiIMI1cyBZ+jFoMfKAGBxQAqDF5EvDBUVOhsbnHCUFWlohCuFFBXYTCLwWiStGihBfnCZbt//nN1AfzoR3DooXDjjQ3XP6O8yRR8jloM24VqO3TsGPxoWbKktsXg2/is/agr6ZtvUvthFoNh5ImvT//xx1qCIDqsb/z4humXUf5kCj5HLYawMLRoEVgMS5ZA27bBfk+nTkF9pKjF0DzyRI/GGHIdlWQWg9HkGTJEK7Oecgr87/8G+7/4oniTrxuNi7gYQ9x8DJAqDJDqSvIWQ/iB37FjENyOWgzRGQ1tVJJhbCUtW8IvfwmDBum2c/DRRw3bJ6M8iRuVtGmT7s/kSgK1Eior07uSOnUK1vv2DdbjHvoWYzCMAvGLX8CVV+o/4O236z/y1VfDJ5/o8Y0btYLrnXcGbgGjaTJkCJxwQu39ca4k0F/1mVxJfn+XLprEli7G4NlxRzjrLNhnH712eMgqqDCEhWD0aL1uNizGYBgRLrtMlz176pzTY8bAXXfBn/6k/tovv4Snn9bX8cfDrrs2bH+NhuPTT/UVJS74DKm1i7zFsM02tc/v0kXrJ4VLYni8xdChg1oWTz0F11+vcYdswvDSS/rKhlkMhpGGH/1Il34SoXXr4O9/Tx35scgqeRkxpBOGOIshroBj587ZLYbwlLitW+t7RstkRGsl5YpZDIaRhr59tTJr+BfWGWfoXLseX+PGMMJEg8/hWdyiFgPAu+8GWc2g61OnZo4xhBPj/PV9lVaPF4bwqKhc2GqLIVlK2zAaHSJwyCHB9rXX6vDAD0LTS5kwGHHEBZ9BH9RRiwG0Tpef4xxysxhatgz2+eunE4ZWrfLrf17CICK/FJFTQtt/ATaIyJciMjjDqYZRllx1lS67dNFA9Hnn6XazZvqPacJgxJEp+BwdrhqHjzFkshjihCEcWK6sVPdnnDBUZHny52sx/BJYCiAihwCnA2cBnwJ353ktwyh5hgyB116DV1/VbT+UdcsWrbdkwmDEkSn4HGcxROncWUXBTy4VFgY/70g2Ydhmm/QWQ3hkUxz5CkMvYE5y/QTgH86554DfAsPyvJZhlAVHHgn77qvrYXPfhKFpk2kWtEzB51wtBgi+X2Fh8OdncyV16hQIQ2UlvPEGDByox8KJcXHkKwxrAF9+7Ah0TmaAzUBl7BmG0YjwFgNohqoJQ9Ml02ifbMHnbK4cP+LIj4ALC4N/uJ9zTrAvnSvJD1dt1Qq+/32NZUDhheF14M/J2MIA4JXk/l2Br/K8lmGUHd4EP/54tRg++UTzGzxjx8Lhh2v9JaNxkymDOFvwOdsc41GLIVwSo29fvc4FFwT74oSheXMVrw0bAleSj08UWhguBd4FugKnOue84bI38Eye1zKMsmTNGnjhBTjzTN2+7Tb9FXjbbSoKY8fCs882bB+N+icXiyFd8DlXiyHOlRS+nifOleTFYOXKYN0X5ctWTj6vPAbn3BrgFzH7b8rnOoZRzrRvr8sjjtAJ2i+6SMsaLF6seQ7TpsGHHzZsH42688QT+ov85z/P3C5sMTiXagVkCz5nsxj8d2zFCm3rXVLp8Nf/9lu1as8+W92e77yjwuCrsPpcBn/9dOQ7XHWX8LBUETlCRJ4UketEJEvXDaPxcdRRuly8WGvWPPaYzjM9cWLgTjDKi/POg0suyd4uLAzRqqbpYgx+as3KLBHZsDBErYU4vCUA6mp68MHAbbRyZfB+vqBeuH0c+bqS/gIMARCR7YH/AzqjLqbf53ktwyh7eveG55+H6dO1Zk3btrDfflqaYNq0hu6dUZ+EhSHqVqquVneRtwwqKtSds369ikPUFRSlXbvgurkIg7cYILh22L3kXUk+D+eYYzJfL19h2BnwYbXTgA+cc8cC5wJn5nktw2gUnHJKMFIEYOhQXZo7qXETFoPw+sqVWnQxWnnXz8mQizC0aBE8zPMVBm8d+H01NcG1DjhA3V7h6T/jyFcYmgGbkuvfB8Yk12cBPWLPMIwmxqBBmoQ0ciTMnRv8xy5bVrvImVFa+EzjXAhbDJs2BetjxqRue3y9olyEAQJ3Ui7C0KJF7UB3WCzqtSQG8DnwcxE5GBWGZD4ovYBleV7LMBolFRWa4/DOO3DFFXt+t79bt2AcuVGa5DJXgSedK8lbCkOGpLbPx2KAwJ2ULR4Rvj40jDBcA1wEJIBnnHOfJfefCJjhbBhJzj9flytXtsQ5mD1btz/7LL9fpUbhWb0aPv88/tjSpblfJ50w+MDzCy+ktm/Tpm4WQ9euufXHC0HUlRTelyt5CYNz7m0087mrc+6noUOPABkHd4nISBFZIiKxfxIRGS4iq0Xk0+TrN/n0zTBKieuug8svh2bNHI8/rjNted57r8G6ZaDDjHffPf5YWBiyjSpLF2OIK3wHKgZ1sRjyFYaGsBhwzm1BK6ruJiK7ikilc26Oc25JllMfB47O0macc26v5Ot3+fbNMEoFERgwADZvrvhuoh/P++83TJ8Mxc/jHbXcpkzRshGesEVQVVU7mJzOYkgnDGFXUi6/4PO1GHz/+vUL3s9T32W3m4vIXcBKYBLwGbBSRP6QbV6GpLWxIlMbw2hM+LIDEyfq8mc/0zIaM2c2XJ+MgPXrU7ejZUx8Mpj/hX/99anH8xUGH3yuqqofV9KcObrcYw9dhnMV6tti+ANwDnAxMAgYiLqQzgVuz/NacRwgIpNE5BURsZl0jbLGC0N1tbqVHn5YrQgThtLAJ3t5vOvG44XBjyR77LHU4/m6kvINPnurIldh8HhhCL9HvsKQ79SeZwE/dc6NCe2bJSJLgceAK/O8XpiPgb7OubUicizwIio8tRCREcAIgG7dupFIJLbibY0wa9eutftZIFasaAnoMKRWraaRSHxDu3Y7MXHiNiQSjcOf9O67Xais3MI++6wqyvsV5vs5HICxYz+gV69gvstPP+0G7Mq++65gwoTOJBIf0Lv3BlatagEcyObNm0kk3v2u/ZQpvQENHk2YMJlmzdQh8uWXfYAdeO+9t2jZMqjNvWbNTqxY0YlNmypYuXIZicT0jL2cP38XoDsrVkwnkViY8+f66qsEc+em7ps3bwaJxIIcrpHEOZfzC9gADI7ZvxOwIYfz+wGf5/hec9Agd8Z2gwYNckbhGDt2bEN3odGwZYtzbdpsdh06OLdihe77/e+dA+fWrQva1dQ0TP8Kwd57O3fkkcV7v0J8PzXFy7lPP03d/+STuv+OO1KPL1ig2507p7a/5ZbgWv/6V7D/t7/VfVu2pLb/2c+c697dufbtnfvVr7L388QT9TrPPpvb57rjDufOPDN1X8+eeo1HHqndHpjg0jxX83UlTUJncYtyefJYnRGRbUU0gVxEhqJurjxGFRtGaVFRAU8/PZ5ly3Q2LVBXEsDbb8Ott8Kdd2q7WbNqBzfLgaqqzOWnS5lojMEPM/W+fe9KiktWg8wxhoqK2hVU83Ul+etnK3jnueYaePrp1H2+eF59u5KuBsaIyBHA+4ADDgB6Ahmrb4jIM6hd01VE5gM3AS0AnHMPA6eiyXPVqGVyRlLVDKNs6dixOsXXfPDBuozWqhkwQIPTl16qmbPXXFO8Pm4NVVWZy0+XGuEnSjTG4GMDfupMLwx+f/RplCnGEJet3KZNEK/IRRj8NfPNQQjTuzd88EH2Mt9R8i27/baIDEKL5u0ECPAPtDTGr4B3MpybsZaSc+4B4IF8+mMY5UbPnvpPGmcdPPII3w1tPfvs7JOplAIbN5aXxfDtt8F61GLIVxgyWQxxwhAWg1yE4bLL4K230udc5IL/Di1alN95dcljWOicu8E5d4pz7mTn3I3AOuCUfK9lGE2RJ5+Eww6Dk07S7d69YVhkxvTx44vfr7pQbsKwalWwns5iiLqS0mWqb9gQCECuFoMnFyvg1FNVjLp1y942HYceqsuePfM7L29hMAxj6zjzTPjPf4L5ow84ICiH7CmXJLhyizGsXBmsp4sxRC0GH2OIWgwbNgRzHtSHxVAIfvhDzc84++z8zjNhMIwGwpc+7tMHBg8O9n/veyoML7wQPJxKlXKzGMLCkG+MIUq+whA3Z0IxGDIk+4xxUUwYDKOB8KUL+vSBnXbS9eHD1YJ4/32d5+Guuxqqd9mpqdGHYDkJw+rVwXq+MYYoGzboNJpQ2sJQF3IKPovIqCxNOhSgL4bRpNhrLx3Guv/+Ojn7Sy/BgQeqm8lTyiN+fN9KuY9RwmIQtRiyDVeNcyW1aaNDQcPiWEqupLqS66ikbPkEy4GvtrIvhtGk2G47nXbRc/zxugwHotONoS8FvCBUVelDM193RUMQFoY4i6FZsyAwnG1U0vr10Lmzto8KQ/OYJ2ujsxicc+fXd0cMw1B69YJjj9V8hj/+UauBlmKVkugMZvkmURWCiy/Wh/Ntt+XW3otBs2bxMYYWLYL5mXNxJbVuHS8M5W4xWIzBMEqQl1+G/fbT9bfeSh1/XyqEXUgNFWd45BG4PY/ynV4YunbVIcHhYHT4gV5Zmd1iyFcYwhZDqeeomDAYRonSIzSL+pQpDdePdETnKygHvDDU1MCkSWpxeKqrAxdQZWUgfOnceVsjDPnmFRQbEwbDKFG6dAnWP/ssfbuGohQshnzxD3M/U1s4xhN+oLdqFXy+fF1J1dWZhWFrMpmLhQmDYZQo4RyGUheGfEcmzZsHX39d2P7kwvr1+oDeay/dDk+5mk0Y1q7VALsfvZSvxdCzJzz6aOqos1LFhMEwSpTwmPsnnoDHH4d771U3yB13ZJ64fv58eOON+u3f1riS+vSBvn0L259cWL9eH+bvv68WWVjQwq6k8BDUqCtp9WqNN/iZ2HIVBoCLLsp/4p2GIN/qqoZhFInrrtMA6bPPajXW85NjAysq9NikSfDMM/HnHnCAisOWLflX1syVcnQleYuhslJzSNIlpoVjDFFX0sqVgVvIC0O6IHa5YhaDYZQohx6qRd+OPhrGjQtq7T+QrEH8xhtBNdYo8+frMvzAKjSFCD6vKNAs8Fu2xO+/4AJ4881g2wsDZP6lnynGsGpV4ObL12IoF0wYDKMMOOggLb53yCEwY4buW7ZMR9XMnp3+vG++qft7Pvus+tQXL44/XgiLoUsXrTa7tUST1UD7NHIkvPZasM9nK0Pqwx/SC0PUlbRypQmDYRglxMkn19732GPpZ3/LJgw1NSowcTz4oC6nTYs/XihXUl1jIeG8Aj8BThhfYjs6B0M6YYgOV/WfKc6VZMJgGEbJECcMt9+umbyTJ9c+lk0Ybr9d6/0vjJlr3j9405W6SDdRTZh167JPWVrXh2jYfRTNYobAjRYWjUzCkKsryYTBMIySondvuPHGoNzzww8H4+J9Elz4l3Q2YXjxRV3Om1f7mL9ONHj96KM6XWQ2i8E5aNcOfvzjzH2o60M07OKJsxi8MORqMZgwBJgwGEaZccstMGECnHYanHNOMC7eD18ND3PNJgzNmukyLnjrhcGP2/dceaUGvbMFn/2D9amnapeTCFMIYcjHYvB1iuIS0+JcSRZjMAyjLNhxR3juOWjbVovIVVQEwrBkSdDuj3+EDz9Mfx1vDcRNCORdQNGH3rff6sMxm8UQ/qX+6afp+9AYLIZNm1Sov/99EwbDMEqAigod3eOFIZr4NmxYenHwwrBmTe1j/ld++KHvH7a5CEP4YT12bPr+17cwhI/VZVRSLsIA8Pzzar2ZMBiGURJ07x5YCl4YRo3SDF/n0o/88a6kXIXB5x2sWKH7M1kc4V/q772Xvu91TcDL1ZXk+1FTk58w5ONKCtMYhMEynw2jEdCtG7z6qo7Z93GFffbR+jwDB2pMIg4vDHFlvb0rKfzQ98LgLYY2bXQZl0fgf6l37QrvvhtM5hONWdR1BrhsFoMfruqP+Qd9rsNVN27UPkcthuXLAyHt0KG2MKQroldOmMVgGI2Abt30AX700SoMImpFAOy7b3phyNeVFBYGH8ht1y7+F7t/IB92mPbJWzTRX+CFEIZsFkPYamrXTpfhshdQ22Lw+6LCsGBBIDrbbFNbGCB+BrdywoTBMBoBLVsG6zNmqFD4h9OwYToc9a9/rX2ef+i99VbtDOpMwrBunf5y7tBBA+BxD2Zvhey0ky69JVMoYQg/sP2DOowXhupqfY+LL4bddoPTT9f9meZq9sJQVVVbGJYu1c/SsqWKQpwwmMVgGEaDE36ov/UWbLttsH3RRfC978G119Y+zz/QX389tQQ1BMIQ50oCLZudSRi8FeKv60trRIUhU9b0rFnpy1SHr7M8Zlb6cJ2o6dNh0SIYMSKobtqqlbrLvGsr6koCFZTw+3TrpsspUzSXRMSEYasQkZEiskREPk9zXETkPhGZKSKTRWTvYvXNMMqde+6BXXbR9QULUoWhdWv40Y/0wRgtWhd9oN99tz4M//GPIGEuzmIAmDsX2revLQxbtsAJJ8BPf6rbAwbo0gtD1ELIZDHsvLMOAY0j/MCOK0EeFoa339blkCHBPm8VhEcfRS2GjRtTLQYvcp99FiQZRmMmYMKQD48DR2c4fgwwMPkaATxUhD4ZRqNg2DD45JMgmLzddqnHvWh88UXq/mjQ9sorVWROPz14IKYThsWL4y2GV16B0aODbS8MdXEl+T7EtfHXadYsvt7TypVBPOHtt/XX/Z57BsfrIgz+Pi5erPEFCO5h+J6bMOSIc+5tIFOR3ZOAvzplPNBJRLbL0N4wjBAtWwYPvvB80QC77qrLqDDEuYB8mQxPOlcSqMXQrl2qwETLa3Tvru6WdK6kXGIMixbV3uev07Nneouhd29d/+gj6NdP++vJRRiqqvR9DjgArr8e7rorON9bDGeeCTffDH/4Q3AsHPMpR0opdt4LCH+l5if31fpKiMgI1KqgW7duJBKJYvSvSbB27Vq7nwWk2Pfz4IN78vHHg5g9ex6JxKzv9tfUQGXlwbz22iIGDZoZ6t8hRH8fRpPhvvpqIYnEdACmTt2TTp3asmpVy+T5C1m/vjlLl7YlkfgIgAkT+gL9vzv/rbcSdOw4jMmTV5FITGPmzLbAft8d/+ablSQSk9J8ouEA9O8PI0bM4oQTgvv58cddgd1o3341Cxe2JpEIkiWqq4V16w6lbdsVQGcWLKihX791JBITv2szZ862wE4kEuPZbrsqNm06mEWLFpBIzGbGjC7A7rz77gSWLx9Ehw6bOeKIz5g8GTp0OJA1a1qwadMSEglV2kMOgQ8/7AzsAcD8+Z+TSKQpW1sOOOeK9gL6AZ+nOfYycFBo+01gn2zXHDRokDMKx9ixYxu6C42KYt/PzZudu/lm5+bNq31sv/2c+8EPgu1Nm5zTEHPm1znnBOf07+/ccccFx666yrmf/MS5Pn2CNpdfnnq+c87tv79zRxyh6x9+mHp8r72cW7Ys/vO0ahW0Gzgw9X4++6zuP/lk5yoqnNuyJThvyRI9duGFwfkHHZR67aef1v1Tp+p28+bOXXedrr/yih577z3nhgxx7oQTgvMOOkiP/exnqdd7663gvcrh3wiY4NI8V0tpVNJ8oHdoe3sgphiwYRjpaN4cfvMb2H772sd22UUDyr5gnncjHXaYJsGlw7uStmxRN9Huuwejd+KCz8uWqXsnTI8eQYzBu278aJ5PP9WRQiNH1nZDdegQrEev6V1JvXqpRRQONvv13qEnio83eMKuJOdSE9PCrqQ1a/QzerxbzruSPOE2XbpQ1pSSMIwCfpwcnTQMWO2ci/EsGoZRF3bZRX31zZtr7SI/E9wZZ8C996Y/zwefFy3Sh2ffvsFDMC74vGyZPqzDdOsWBIj9A/3VV+GUU4I2F1wAffqkZlGHH+Y+Yc8TjjH49/V4YejTJ9gXfnBDqjB4sYwOV92wQYfl9usXnLfDDsGxMD6jGkwYckZEngHeBwaLyHwRuUBELhaRi5NNxgCzgZnAn4FLitU3w2gK+F+6AIcfDkOH6nrbtsHY/s6da5/nhWHuXF2GhcEHn6uqgofr8uXB9Txduuh+54IHeqtWwcM5zPjxwXo4UB0NUvvRQl6EwgHoOIshkzD4a0Uthjlz9FhYGLw1Fp3cqDEJQ9GCz865M7Mcd8ClReqOYTQ5wsIQZscdgwdZixbwwx+mjkzyv4zjhMFbDKBWw4IFMHWqZjt/+mkwfLZLF33Ir1sXPOx95nCU8BzTYevBJ8ytWKEWyIEH6rZPOguX9fDC0LOnlv2oqantSgonsUWFwc/Z4EdxhYXhhBPgiCN0JFKYsPDECV45UUquJMMw6pF+/eCf/9QRNKC/6q+4QnMg/MP1/PPhX/+CSaFBQu+9B7/6leYnVFRkFoZddtFl1646dHa33fSYF57ly1OFIe4BGhWG889XEfATEE2dqg/6ceN0u2PHoK3HC8M22wSCkM5iqKoK3FD+Wt5t9cEHuuzbNzivfXvNFPc5DZ6wxVDulNJwVcMw6pmTT4Yf/EB//e+6a1BEr0MHfZj6YO8ee+iv/5/+VCu2/u//6v7rr9cHrH/ItmwZrIcnCIqW0g4Lg3cJtWxZOxGsefNAGLZs0bb9++vy6afhn//sxfDhqef4IHDY5+9rJ3XqpA/yaAAZgof/7NmBxbCHjjalc2ftny8+GBaGdMRZP+WKCYNhNDE6dAjmiQ4THWXTs2eQ09CjB1x1lVoOEPj1a2qCB+5ee6W+R5h0FkP4Yd6vnz6gvTB4C6BNm+CX/AMPDKw1Dal/r6jF4AvceYsh6krq21dHY736Kuy/v2ZGe3ebiH7+OXP0s3vXUibqOq9EKdKIPophGIXGP5DnzFG3k48ZPPQQ/P73MHx4bRfK3XfD1Ven7vPB6DPOgInJHLNWrYKM6XvuUfdVjx6ZhQF0Dukw/v3DIrNkSWAR+GznqMUAcMwxOkLr/fdVJMJt/GinnXeufV5jx4TBMIy0/Oc/+oq6STp3hhtu0F/JPj7h+clPav/C9hbDihXw4IO63rJlIAw77KC//MP5Dl4Y2rZNFYY1a1JHG3lhCFsMYWFIF2MAda1VVam7bO9I2U5f+ygaS8jEZZfFlzcvN8yVZBhGWvr311cm9tlH/fR+fH/ckNe4fWFh8A/vHj00yD1zZuqMa2FhgKA+UlWVXkektsXgK8x6iyHqSoIgEA+Bm8zjXUP5WAz3359721LGLAbDMLaa/v3h+OPVHSNS+3jcjGYtWwb7fZzAP8zPOivVlRS8jypJz57w1Vc6akhE29TFYhCBMWO0ON7++6ce8wHscJJcU8GEwTCMgjBqlE6Ik4nwr+8WLbQMxk03qdUBcGkyk2n+/FRXkh9+OnToCnbYQYfBbrttkKTXpk1gMTinwuArzGaKMYDGGa68svb+ESP8e2b+TI0RcyUZhlEQ4iyFMCtXasB5zz21HEdFhWYR//a3QZvtt9cZ5156KSiz0aYNXHihzkx3+unzGTmyT638h9atAyFZs0ZHPkUthjhXUiZOPTWYxa6pYRaDYRhFoVMnfYBPmKCTCqWjY0dNZgu7krp31wBx586baNMmGB3lCVsMflRTLq4kIx4TBsMwikqHDqk5D3HHN2wIah/l8kAPWww+0S6X4apGPOZKMgyjpPCB6Fde0fyHXIK/YYvhzTd16YebnnCCVoaNzmpnpMeEwTCMksIPTR09Gk47LbeM4tat4Y03tCDg7Nl6nq/TNHiwJt0ZuWOuJMMwSopwOY3DD8/tHD+kdfZsrXz6t79lD4Yb6TFhMAyjpAgLQ67JZV4YTjhBXVDlXva6oTFhMAyjpAhnOedS1RSCEhyDB9cesWTkjwmDYRglRdhiiE4Rmg4/IilcQ8moOyYMhmGUFGGLITpfQzqWL9eln3bT2DpMGAzDKCmicznkgp+BzYShMJT9cNV58+YxPDqlk1FnVq1aRafojC1GnbH7mT9ahiIBUOt/O939nDXrT8Au/PKXp1NZuaTWcSM/yl4YDMNoXIhA69Zf06PH6zmfs8suv2PZsoNo1cpEoRCIK/MqUYMHD3ZffvllQ3ej0ZBIJMwCKyB2PwuL3c/CISITnXP7xh2zGINhGIaRggmDYRiGkYIJg2EYhpFCUYVBRI4WkS9FZKaIXBtzfLiIrBaRT5Ov3xSzf4ZhGEYRRyWJSDPgQeAIYD7wkYiMcs59EWk6zjl3fLH6ZRiGYaRSTIthKDDTOTfbObcJeBY4qYjvbxiGYeRAMYWhFzAvtD0/uS/KASIySUReEZFdi9M1wzAMw1PMBLe46ujRJIqPgb7OubUicizwIjCw1oVERgAjALp160YikShsT5swa9eutftZQOx+Fha7n8WhmMIwHwjXPtweWBhu4JxbE1ofIyJ/EpGuzrllkXaPAo+CJrhZwkvhsASiwmL3s7DY/SwOxXQlfQQMFJH+ItISOAMYFW4gItuK6LxLIjI02b/lReyjYRhGk6doFoNzrlpELgNeA5oBI51zU0Tk4uTxh4FTgZ+LSDWwATjDlXvNDsMwjDKjqEX0nHNjgDGRfQ+H1h8AHihmnwzDMIxULPPZMAzDSMGEwTAMw0jBhMEwDMNIwYTBMAzDSMGEwTAMw0jBhMEwDMNIwYTBMAzDSMGEwTAMw0jBhMEwDMNIwYTBMAzDSMGEwTAMw0jBhMEwDMNIwYTBMAzDSMGEwTAMw0jBhMEwDMNIwYTBMAzDSMGEwTAMw0jBhMEwDMNIwYTBMAzDSMGEwTAMw0jBhMEwDMNIwYTBMAzDSMGEwTAMw0jBhMEwDMNIwYTBMAzDSMGEwTAMw0ihqMIgIkeLyJciMlNEro05LiJyX/L4ZBHZu5j9MwzDMIooDCLSDHgQOAbYBThTRHaJNDsGGJh8jQAeKlb/DMMwDKWYFsNQYKZzbrZzbhPwLHBSpM1JwF+dMh7oJCLbFbGPhmEYTZ7mRXyvXsC80PZ8YP8c2vQCFoUbicgI1KIA2Cginxe2q7XoCKwuwrnZ2qY7ns/+6L7odldgWdaebh12PwuL3c/CUoz7mUu7+r6ffdO+s3OuKC/gNOCx0Pa5wP2RNi8DB4W23wT2yXLdCUXo+6PFODdb23TH89kf3RezbffT7qfdz3o+N5d2xbif6V7FdCXNB3qHtrcHFtahTUPwUpHOzdY23fF89kf3bc1nqyt2PwuL3c/CUoz7mUu7BrufklSRekdEmgPTge8DC4CPgLOcc1NCbY4DLgOORd1M9znnhma57gTn3L711vEmht3PwmL3s7DY/SwORYsxOOeqReQy4DWgGTDSOTdFRC5OHn8YGIOKwkxgPXB+Dpd+tJ663FSx+1lY7H4WFrufRaBoFoNhGIZRHljms2EYhpGCCYNhGIaRggmDYRiGkUKjFwYRaSsiE0Xk+IbuS7kjIjuLyMMi8ryI/Lyh+1PuiMgPReTPIvJ/InJkQ/en3BGRHUTkLyLyfEP3pdwpWWEQkZEisiSa1ZytEF8M1wDP1U8vy4dC3E/n3FTn3MXA6UCTHjJYoPv5onPuIuA84L/qsbslT4Hu52zn3AX129OmQcmOShKRQ4C1aO2k3ZL7mqG5EEegyXAfAWeiw19vj1zip8AeaAp9JbDMOTe6OL0vPQpxP51zS0TkROBa4AHn3NPF6n+pUaj7mTzvbuAp59zHRep+yVHg+/m8c+7UYvW9MVLMWkl54Zx7W0T6RXZ/V4gPQESeBU5yzt0O1HIVichhQFu0musGERnjnKup356XJoW4n8nrjAJGicjLQJMVhgJ9PwW4A3ilKYsCFO77aRSGkhWGNORSiO87nHM3AIjIeajF0CRFIQN53U8RGQ6cDLRCkxGNVPK6n8AvgB8AHUVkQDLJ0wjI9/vZBbgVGCIi1yUFxKgD5SYMErMvqy/MOfd44bvSKMjrfjrnEkCivjrTCMj3ft4H3Fd/3Sl78r2fy4GL6687TYeSDT6noVSL7JUrdj8Li93PwmL3s4EoN2H4CBgoIv1FpCVwBjCqgftUztj9LCx2PwuL3c8GomSFQUSeAd4HBovIfBG5wDlXjVZffQ2YCjwXrs5qpMfuZ2Gx+1lY7H6WFiU7XNUwDMNoGErWYjAMwzAaBhMGwzAMIwUTBsMwDCMFEwbDMAwjBRMGwzAMIwUTBsMwDCMFEwbD2EpE5LfRctGGUc5YHoNRFojI40BX51zJVdUUkXZAq2StnpJERBxwmnPOJrExsmIWg2GkIVmGISvOubUNIQoiUpGcs8AwCooJg9EoEJFdRORlEfk2ORPYMyKybej4fiLyuogsE5E1IvKOiBwQuYYTkUtF5AURWQfc5t1EInKGiMxKXv9FEekaOi/FlSQij4vIaBG5XEQWiMhKEfl/ItIm1KatiPxVRNaKyGIRuS55zuMZPuN5yfbHJt9vE7Bzts8mInOSq/9IfsY5oWMniE59WyUiX4nIrbkKotF4MWEwyh4R2Q54G/gcndzlB0A7dEIh/x1vD/wNODjZ5lNgTPgBn+QmdK6J3YEHk/v6oVNv/gg4EhiC1v3PxMHAbsm++HMvDx2/Gzg0uf9wYM/kOdmoBG4EfoZOQDU3h8+2X3J5EbCd3xaRo4CngAeAXdFZD08FbsuhH0ZjxjlnL3uV/At4HBid5tjvgDcj+7ZBa/cPTXOOAIuAc0L7HHB/pN1vgSqgY2jfDejMYuE2n0f6Og9oHtr3Z+CN5Ho79Nf+GaHjbYGVwOMZ7sF5yT7uk+Vepftsp0bavQ38OrLvh+gUm9LQf3N7NdzLLAajMbAPcEjSzbJWRNYSzPy1I4CIdBeRR0RkuoisBr4FugN9IteaEHP9uc651aHthclzM/GF0+qgcefsCLQAPvQHnXPrUIsnG9WoRfAdeXy2KPsAN0Tu29OoSG2b+VSjMVNuM7gZRhwVwMvAlTHHFieXTwA9gP8G5gAbgTeBqD99Xcw1Nke2HdndsJnOkdC+fNnonNsS2ZfrZ4tSAdwM/CPm2NI69M1oJJgwGI2Bj4HT0V/20Qey5yDgl865lwFEpAfqb28IZqLCMRT4KtmfNmhMYlYdrpfLZ9sMREcwfQzs5JybWYf3NBoxJgxGOdFBRPaK7FuFBokvAv4uIneiv3Z3QMXiCufct8B04BwR+QB1lfwB9fMXHefcWhEZCdwpIsvQeMCN6C/4ulgRuXy2OcD3ReQt1OpYicZmRovIXOA51E21GxqXuboO/TAaCRZjMMqJg4FPIq8/OucWAgcCNcCrwBRULDYmX6AjbtoBE4FngZHow7KhuBIYh05VORaYjMY3qupwrVw+2xXAYWjs5RMA59xrwHHJ/R8mX9cCX9ehD0YjwjKfDaMEEJFW6NDTu5xzdzd0f4ymjbmSDKMBEJEhwM7or/T2wDXJ5d8bsl+GASYMhtGQ/A8wmGAI6iHOufkN2iPDwFxJhmEYRgQLPhuGYRgpmDAYhmEYKZgwGIZhGCmYMBiGYRgpmDAYhmEYKZgwGIZhGCn8f6VhC2DS56ADAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"batch_size = 128\n",
"rates, losses = find_learning_rate(model, X_train, y_train, epochs=1,\n",
" batch_size=batch_size)\n",
"plot_lr_vs_loss(rates, losses)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Looks like the max learning rate to use for 1cycle is around 10–1."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `OneCycleScheduler` custom callback updates the learning rate at the beginning of each batch. It applies the logic described in the book: increase the learning rate linearly during about half of training, then reduce it linearly back to the initial learning rate, and lastly reduce it down to close to zero linearly for the very last part of training."
]
},
{
"cell_type": "code",
"execution_count": 94,
"metadata": {},
"outputs": [],
"source": [
"class OneCycleScheduler(tf.keras.callbacks.Callback):\n",
" def __init__(self, iterations, max_lr=1e-3, start_lr=None,\n",
" last_iterations=None, last_lr=None):\n",
" self.iterations = iterations\n",
" self.max_lr = max_lr\n",
" self.start_lr = start_lr or max_lr / 10\n",
" self.last_iterations = last_iterations or iterations // 10 + 1\n",
" self.half_iteration = (iterations - self.last_iterations) // 2\n",
" self.last_lr = last_lr or self.start_lr / 1000\n",
" self.iteration = 0\n",
"\n",
" def _interpolate(self, iter1, iter2, lr1, lr2):\n",
" return (lr2 - lr1) * (self.iteration - iter1) / (iter2 - iter1) + lr1\n",
"\n",
" def on_batch_begin(self, batch, logs):\n",
" if self.iteration < self.half_iteration:\n",
" lr = self._interpolate(0, self.half_iteration, self.start_lr,\n",
" self.max_lr)\n",
" elif self.iteration < 2 * self.half_iteration:\n",
" lr = self._interpolate(self.half_iteration, 2 * self.half_iteration,\n",
" self.max_lr, self.start_lr)\n",
" else:\n",
" lr = self._interpolate(2 * self.half_iteration, self.iterations,\n",
" self.start_lr, self.last_lr)\n",
" self.iteration += 1\n",
" K.set_value(self.model.optimizer.learning_rate, lr)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's build and compile a simple Fashion MNIST model, then train it using the `OneCycleScheduler` callback:"
]
},
{
"cell_type": "code",
"execution_count": 95,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/25\n",
"430/430 [==============================] - 1s 2ms/step - loss: 0.9502 - accuracy: 0.6913 - val_loss: 0.6003 - val_accuracy: 0.7874\n",
"Epoch 2/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.5695 - accuracy: 0.8025 - val_loss: 0.4918 - val_accuracy: 0.8248\n",
"Epoch 3/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.4954 - accuracy: 0.8252 - val_loss: 0.4762 - val_accuracy: 0.8264\n",
"Epoch 4/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.4515 - accuracy: 0.8402 - val_loss: 0.4261 - val_accuracy: 0.8478\n",
"Epoch 5/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.4225 - accuracy: 0.8492 - val_loss: 0.4066 - val_accuracy: 0.8486\n",
"Epoch 6/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.3958 - accuracy: 0.8571 - val_loss: 0.4787 - val_accuracy: 0.8224\n",
"Epoch 7/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.3787 - accuracy: 0.8626 - val_loss: 0.3917 - val_accuracy: 0.8566\n",
"Epoch 8/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.3630 - accuracy: 0.8683 - val_loss: 0.4719 - val_accuracy: 0.8296\n",
"Epoch 9/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.3512 - accuracy: 0.8724 - val_loss: 0.3673 - val_accuracy: 0.8652\n",
"Epoch 10/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.3360 - accuracy: 0.8766 - val_loss: 0.4957 - val_accuracy: 0.8466\n",
"Epoch 11/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.3287 - accuracy: 0.8786 - val_loss: 0.4187 - val_accuracy: 0.8370\n",
"Epoch 12/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.3173 - accuracy: 0.8815 - val_loss: 0.3425 - val_accuracy: 0.8728\n",
"Epoch 13/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.2961 - accuracy: 0.8910 - val_loss: 0.3217 - val_accuracy: 0.8792\n",
"Epoch 14/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.2818 - accuracy: 0.8958 - val_loss: 0.3734 - val_accuracy: 0.8692\n",
"Epoch 15/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.2675 - accuracy: 0.9003 - val_loss: 0.3261 - val_accuracy: 0.8844\n",
"Epoch 16/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.2558 - accuracy: 0.9055 - val_loss: 0.3205 - val_accuracy: 0.8820\n",
"Epoch 17/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.2464 - accuracy: 0.9091 - val_loss: 0.3089 - val_accuracy: 0.8894\n",
"Epoch 18/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.2368 - accuracy: 0.9115 - val_loss: 0.3130 - val_accuracy: 0.8870\n",
"Epoch 19/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.2292 - accuracy: 0.9145 - val_loss: 0.3078 - val_accuracy: 0.8854\n",
"Epoch 20/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.2205 - accuracy: 0.9186 - val_loss: 0.3092 - val_accuracy: 0.8886\n",
"Epoch 21/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.2138 - accuracy: 0.9209 - val_loss: 0.3022 - val_accuracy: 0.8914\n",
"Epoch 22/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.2073 - accuracy: 0.9232 - val_loss: 0.3054 - val_accuracy: 0.8914\n",
"Epoch 23/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.2020 - accuracy: 0.9261 - val_loss: 0.3026 - val_accuracy: 0.8896\n",
"Epoch 24/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.1989 - accuracy: 0.9273 - val_loss: 0.3020 - val_accuracy: 0.8922\n",
"Epoch 25/25\n",
"430/430 [==============================] - 1s 1ms/step - loss: 0.1967 - accuracy: 0.9276 - val_loss: 0.3016 - val_accuracy: 0.8920\n"
]
}
],
"source": [
"model = build_model()\n",
"model.compile(loss=\"sparse_categorical_crossentropy\",\n",
" optimizer=tf.keras.optimizers.SGD(),\n",
" metrics=[\"accuracy\"])\n",
"n_epochs = 25\n",
"onecycle = OneCycleScheduler(math.ceil(len(X_train) / batch_size) * n_epochs,\n",
" max_lr=0.1)\n",
"history = model.fit(X_train, y_train, epochs=n_epochs, batch_size=batch_size,\n",
" validation_data=(X_valid, y_valid),\n",
" callbacks=[onecycle])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Avoiding Overfitting Through Regularization"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## $\\ell_1$ and $\\ell_2$ regularization"
]
},
{
"cell_type": "code",
"execution_count": 96,
"metadata": {},
"outputs": [],
"source": [
"layer = tf.keras.layers.Dense(100, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\",\n",
" kernel_regularizer=tf.keras.regularizers.l2(0.01))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or use `l1(0.1)` for ℓ1 regularization with a factor of 0.1, or `l1_l2(0.1, 0.01)` for both ℓ1 and ℓ2 regularization, with factors 0.1 and 0.01 respectively."
]
},
{
"cell_type": "code",
"execution_count": 97,
"metadata": {},
"outputs": [],
"source": [
"tf.random.set_seed(42) # extra code – for reproducibility"
]
},
{
"cell_type": "code",
"execution_count": 98,
"metadata": {},
"outputs": [],
"source": [
"from functools import partial\n",
"\n",
"RegularizedDense = partial(tf.keras.layers.Dense,\n",
" activation=\"relu\",\n",
" kernel_initializer=\"he_normal\",\n",
" kernel_regularizer=tf.keras.regularizers.l2(0.01))\n",
"\n",
"model = tf.keras.Sequential([\n",
" tf.keras.layers.Flatten(input_shape=[28, 28]),\n",
" RegularizedDense(100),\n",
" RegularizedDense(100),\n",
" RegularizedDense(10, activation=\"softmax\")\n",
"])"
]
},
{
"cell_type": "code",
"execution_count": 99,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/2\n",
"1719/1719 [==============================] - 2s 878us/step - loss: 3.1224 - accuracy: 0.7748 - val_loss: 1.8602 - val_accuracy: 0.8264\n",
"Epoch 2/2\n",
"1719/1719 [==============================] - 1s 814us/step - loss: 1.4263 - accuracy: 0.8159 - val_loss: 1.1269 - val_accuracy: 0.8182\n"
]
}
],
"source": [
"# extra code – compile and train the model\n",
"optimizer = tf.keras.optimizers.SGD(learning_rate=0.02)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
" metrics=[\"accuracy\"])\n",
"history = model.fit(X_train, y_train, epochs=2,\n",
" validation_data=(X_valid, y_valid))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Dropout"
]
},
{
"cell_type": "code",
"execution_count": 100,
"metadata": {},
"outputs": [],
"source": [
"tf.random.set_seed(42) # extra code – for reproducibility"
]
},
{
"cell_type": "code",
"execution_count": 101,
"metadata": {},
"outputs": [],
"source": [
"model = tf.keras.Sequential([\n",
" tf.keras.layers.Flatten(input_shape=[28, 28]),\n",
" tf.keras.layers.Dropout(rate=0.2),\n",
" tf.keras.layers.Dense(100, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\"),\n",
" tf.keras.layers.Dropout(rate=0.2),\n",
" tf.keras.layers.Dense(100, activation=\"relu\",\n",
" kernel_initializer=\"he_normal\"),\n",
" tf.keras.layers.Dropout(rate=0.2),\n",
" tf.keras.layers.Dense(10, activation=\"softmax\")\n",
"])"
]
},
{
"cell_type": "code",
"execution_count": 102,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.6703 - accuracy: 0.7536 - val_loss: 0.4498 - val_accuracy: 0.8342\n",
"Epoch 2/10\n",
"1719/1719 [==============================] - 2s 996us/step - loss: 0.5103 - accuracy: 0.8136 - val_loss: 0.4401 - val_accuracy: 0.8296\n",
"Epoch 3/10\n",
"1719/1719 [==============================] - 2s 998us/step - loss: 0.4712 - accuracy: 0.8263 - val_loss: 0.3806 - val_accuracy: 0.8554\n",
"Epoch 4/10\n",
"1719/1719 [==============================] - 2s 977us/step - loss: 0.4488 - accuracy: 0.8337 - val_loss: 0.3711 - val_accuracy: 0.8608\n",
"Epoch 5/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.4342 - accuracy: 0.8409 - val_loss: 0.3672 - val_accuracy: 0.8606\n",
"Epoch 6/10\n",
"1719/1719 [==============================] - 2s 983us/step - loss: 0.4245 - accuracy: 0.8427 - val_loss: 0.3706 - val_accuracy: 0.8600\n",
"Epoch 7/10\n",
"1719/1719 [==============================] - 2s 995us/step - loss: 0.4131 - accuracy: 0.8467 - val_loss: 0.3582 - val_accuracy: 0.8650\n",
"Epoch 8/10\n",
"1719/1719 [==============================] - 2s 959us/step - loss: 0.4074 - accuracy: 0.8484 - val_loss: 0.3478 - val_accuracy: 0.8708\n",
"Epoch 9/10\n",
"1719/1719 [==============================] - 2s 997us/step - loss: 0.4024 - accuracy: 0.8533 - val_loss: 0.3556 - val_accuracy: 0.8690\n",
"Epoch 10/10\n",
"1719/1719 [==============================] - 2s 998us/step - loss: 0.3903 - accuracy: 0.8552 - val_loss: 0.3453 - val_accuracy: 0.8732\n"
]
}
],
"source": [
"# extra code – compile and train the model\n",
"optimizer = tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.9)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
" metrics=[\"accuracy\"])\n",
"history = model.fit(X_train, y_train, epochs=10,\n",
" validation_data=(X_valid, y_valid))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The training accuracy looks like it's lower than the validation accuracy, but that's just because dropout is only active during training. If we evaluate the model on the training set after training (i.e., with dropout turned off), we get the \"real\" training accuracy, which is very slightly higher than the validation accuracy and the test accuracy:"
]
},
{
"cell_type": "code",
"execution_count": 103,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1719/1719 [==============================] - 1s 578us/step - loss: 0.3082 - accuracy: 0.8849\n"
]
},
{
"data": {
"text/plain": [
"[0.30816400051116943, 0.8849090933799744]"
]
},
"execution_count": 103,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.evaluate(X_train, y_train)"
]
},
{
"cell_type": "code",
"execution_count": 104,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"313/313 [==============================] - 0s 588us/step - loss: 0.3629 - accuracy: 0.8700\n"
]
},
{
"data": {
"text/plain": [
"[0.3628920316696167, 0.8700000047683716]"
]
},
"execution_count": 104,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.evaluate(X_test, y_test)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Note**: make sure to use `AlphaDropout` instead of `Dropout` if you want to build a self-normalizing neural net using SELU."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## MC Dropout"
]
},
{
"cell_type": "code",
"execution_count": 105,
"metadata": {},
"outputs": [],
"source": [
"tf.random.set_seed(42) # extra code – for reproducibility"
]
},
{
"cell_type": "code",
"execution_count": 106,
"metadata": {},
"outputs": [],
"source": [
"y_probas = np.stack([model(X_test, training=True)\n",
" for sample in range(100)])\n",
"y_proba = y_probas.mean(axis=0)"
]
},
{
"cell_type": "code",
"execution_count": 107,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[0. , 0. , 0. , 0. , 0. , 0.024, 0. , 0.132, 0. ,\n",
" 0.844]], dtype=float32)"
]
},
"execution_count": 107,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.predict(X_test[:1]).round(3)"
]
},
{
"cell_type": "code",
"execution_count": 108,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0. , 0. , 0. , 0. , 0. , 0.067, 0. , 0.209, 0.001,\n",
" 0.723], dtype=float32)"
]
},
"execution_count": 108,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_proba[0].round(3)"
]
},
{
"cell_type": "code",
"execution_count": 109,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0. , 0. , 0. , 0.001, 0. , 0.096, 0. , 0.162, 0.001,\n",
" 0.183], dtype=float32)"
]
},
"execution_count": 109,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_std = y_probas.std(axis=0)\n",
"y_std[0].round(3)"
]
},
{
"cell_type": "code",
"execution_count": 110,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8717"
]
},
"execution_count": 110,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_pred = y_proba.argmax(axis=1)\n",
"accuracy = (y_pred == y_test).sum() / len(y_test)\n",
"accuracy"
]
},
{
"cell_type": "code",
"execution_count": 111,
"metadata": {},
"outputs": [],
"source": [
"class MCDropout(tf.keras.layers.Dropout):\n",
" def call(self, inputs, training=None):\n",
" return super().call(inputs, training=True)"
]
},
{
"cell_type": "code",
"execution_count": 112,
"metadata": {},
"outputs": [],
"source": [
"# extra code – shows how to convert Dropout to MCDropout in a Sequential model\n",
"Dropout = tf.keras.layers.Dropout\n",
"mc_model = tf.keras.Sequential([\n",
" MCDropout(layer.rate) if isinstance(layer, Dropout) else layer\n",
" for layer in model.layers\n",
"])\n",
"mc_model.set_weights(model.get_weights())"
]
},
{
"cell_type": "code",
"execution_count": 113,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model: \"sequential_25\"\n",
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"flatten_22 (Flatten) (None, 784) 0 \n",
"_________________________________________________________________\n",
"mc_dropout (MCDropout) (None, 784) 0 \n",
"_________________________________________________________________\n",
"dense_89 (Dense) (None, 100) 78500 \n",
"_________________________________________________________________\n",
"mc_dropout_1 (MCDropout) (None, 100) 0 \n",
"_________________________________________________________________\n",
"dense_90 (Dense) (None, 100) 10100 \n",
"_________________________________________________________________\n",
"mc_dropout_2 (MCDropout) (None, 100) 0 \n",
"_________________________________________________________________\n",
"dense_91 (Dense) (None, 10) 1010 \n",
"=================================================================\n",
"Total params: 89,610\n",
"Trainable params: 89,610\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n"
]
}
],
"source": [
"mc_model.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can use the model with MC Dropout:"
]
},
{
"cell_type": "code",
"execution_count": 114,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[0. , 0. , 0. , 0. , 0. , 0.07, 0. , 0.17, 0. , 0.76]],\n",
" dtype=float32)"
]
},
"execution_count": 114,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# extra code – shows that the model works without retraining\n",
"tf.random.set_seed(42)\n",
"np.mean([mc_model.predict(X_test[:1])\n",
" for sample in range(100)], axis=0).round(2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Max norm"
]
},
{
"cell_type": "code",
"execution_count": 115,
"metadata": {},
"outputs": [],
"source": [
"dense = tf.keras.layers.Dense(\n",
" 100, activation=\"relu\", kernel_initializer=\"he_normal\",\n",
" kernel_constraint=tf.keras.constraints.max_norm(1.))"
]
},
{
"cell_type": "code",
"execution_count": 116,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1719/1719 [==============================] - 2s 1ms/step - loss: 0.5500 - accuracy: 0.8015 - val_loss: 0.4510 - val_accuracy: 0.8242\n",
"Epoch 2/10\n",
"1719/1719 [==============================] - 2s 960us/step - loss: 0.4089 - accuracy: 0.8499 - val_loss: 0.3956 - val_accuracy: 0.8504\n",
"Epoch 3/10\n",
"1719/1719 [==============================] - 2s 974us/step - loss: 0.3777 - accuracy: 0.8604 - val_loss: 0.3693 - val_accuracy: 0.8680\n",
"Epoch 4/10\n",
"1719/1719 [==============================] - 2s 943us/step - loss: 0.3581 - accuracy: 0.8690 - val_loss: 0.3517 - val_accuracy: 0.8716\n",
"Epoch 5/10\n",
"1719/1719 [==============================] - 2s 949us/step - loss: 0.3416 - accuracy: 0.8729 - val_loss: 0.3433 - val_accuracy: 0.8682\n",
"Epoch 6/10\n",
"1719/1719 [==============================] - 2s 951us/step - loss: 0.3368 - accuracy: 0.8756 - val_loss: 0.4045 - val_accuracy: 0.8582\n",
"Epoch 7/10\n",
"1719/1719 [==============================] - 2s 935us/step - loss: 0.3293 - accuracy: 0.8767 - val_loss: 0.4168 - val_accuracy: 0.8476\n",
"Epoch 8/10\n",
"1719/1719 [==============================] - 2s 951us/step - loss: 0.3258 - accuracy: 0.8779 - val_loss: 0.3570 - val_accuracy: 0.8674\n",
"Epoch 9/10\n",
"1719/1719 [==============================] - 2s 970us/step - loss: 0.3269 - accuracy: 0.8787 - val_loss: 0.3702 - val_accuracy: 0.8578\n",
"Epoch 10/10\n",
"1719/1719 [==============================] - 2s 948us/step - loss: 0.3169 - accuracy: 0.8809 - val_loss: 0.3907 - val_accuracy: 0.8578\n"
]
}
],
"source": [
"# extra code – shows how to apply max norm to every hidden layer in a model\n",
"\n",
"MaxNormDense = partial(tf.keras.layers.Dense,\n",
" activation=\"relu\", kernel_initializer=\"he_normal\",\n",
" kernel_constraint=tf.keras.constraints.max_norm(1.))\n",
"\n",
"tf.random.set_seed(42)\n",
"model = tf.keras.Sequential([\n",
" tf.keras.layers.Flatten(input_shape=[28, 28]),\n",
" MaxNormDense(100),\n",
" MaxNormDense(100),\n",
" tf.keras.layers.Dense(10, activation=\"softmax\")\n",
"])\n",
"optimizer = tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.9)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
" metrics=[\"accuracy\"])\n",
"history = model.fit(X_train, y_train, epochs=10,\n",
" validation_data=(X_valid, y_valid))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercises"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. to 7."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Glorot initialization and He initialization were designed to make the output standard deviation as close as possible to the input standard deviation, at least at the beginning of training. This reduces the vanishing/exploding gradients problem.\n",
"2. No, all weights should be sampled independently; they should not all have the same initial value. One important goal of sampling weights randomly is to break symmetry: if all the weights have the same initial value, even if that value is not zero, then symmetry is not broken (i.e., all neurons in a given layer are equivalent), and backpropagation will be unable to break it. Concretely, this means that all the neurons in any given layer will always have the same weights. It's like having just one neuron per layer, and much slower. It is virtually impossible for such a configuration to converge to a good solution.\n",
"3. It is perfectly fine to initialize the bias terms to zero. Some people like to initialize them just like weights, and that's OK too; it does not make much difference.\n",
"4. ReLU is usually a good default for the hidden layers, as it is fast and yields good results. Its ability to output precisely zero can also be useful in some cases (e.g., see Chapter 17). Moreover, it can sometimes benefit from optimized implementations as well as from hardware acceleration. The leaky ReLU variants of ReLU can improve the model's quality without hindering its speed too much compared to ReLU. For large neural nets and more complex problems, GLU, Swish and Mish can give you a slightly higher quality model, but they have a computational cost. The hyperbolic tangent (tanh) can be useful in the output layer if you need to output a number in a fixed range (by default between –1 and 1), but nowadays it is not used much in hidden layers, except in recurrent nets. The sigmoid activation function is also useful in the output layer when you need to estimate a probability (e.g., for binary classification), but it is rarely used in hidden layers (there are exceptions—for example, for the coding layer of variational autoencoders; see Chapter 17). The softplus activation function is useful in the output layer when you need to ensure that the output will always be positive. The softmax activation function is useful in the output layer to estimate probabilities for mutually exclusive classes, but it is rarely (if ever) used in hidden layers.\n",
"5. If you set the `momentum` hyperparameter too close to 1 (e.g., 0.99999) when using an `SGD` optimizer, then the algorithm will likely pick up a lot of speed, hopefully moving roughly toward the global minimum, but its momentum will carry it right past the minimum. Then it will slow down and come back, accelerate again, overshoot again, and so on. It may oscillate this way many times before converging, so overall it will take much longer to converge than with a smaller `momentum` value.\n",
"6. One way to produce a sparse model (i.e., with most weights equal to zero) is to train the model normally, then zero out tiny weights. For more sparsity, you can apply ℓ1 regularization during training, which pushes the optimizer toward sparsity. A third option is to use the TensorFlow Model Optimization Toolkit.\n",
"7. Yes, dropout does slow down training, in general roughly by a factor of two. However, it has no impact on inference speed since it is only turned on during training. MC Dropout is exactly like dropout during training, but it is still active during inference, so each inference is slowed down slightly. More importantly, when using MC Dropout you generally want to run inference 10 times or more to get better predictions. This means that making predictions is slowed down by a factor of 10 or more."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 8. Deep Learning on CIFAR10"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### a.\n",
"*Exercise: Build a DNN with 20 hidden layers of 100 neurons each (that's too many, but it's the point of this exercise). Use He initialization and the Swish activation function.*"
]
},
{
"cell_type": "code",
"execution_count": 117,
"metadata": {},
"outputs": [],
"source": [
"tf.random.set_seed(42)\n",
"\n",
"model = tf.keras.Sequential()\n",
"model.add(tf.keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
"for _ in range(20):\n",
" model.add(tf.keras.layers.Dense(100,\n",
" activation=\"swish\",\n",
" kernel_initializer=\"he_normal\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### b.\n",
"*Exercise: Using Nadam optimization and early stopping, train the network on the CIFAR10 dataset. You can load it with `tf.keras.datasets.cifar10.load_data()`. The dataset is composed of 60,000 32 × 32–pixel color images (50,000 for training, 10,000 for testing) with 10 classes, so you'll need a softmax output layer with 10 neurons. Remember to search for the right learning rate each time you change the model's architecture or hyperparameters.*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's add the output layer to the model:"
]
},
{
"cell_type": "code",
"execution_count": 118,
"metadata": {},
"outputs": [],
"source": [
"model.add(tf.keras.layers.Dense(10, activation=\"softmax\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's use a Nadam optimizer with a learning rate of 5e-5. I tried learning rates 1e-5, 3e-5, 1e-4, 3e-4, 1e-3, 3e-3 and 1e-2, and I compared their learning curves for 10 epochs each (using the TensorBoard callback, below). The learning rates 3e-5 and 1e-4 were pretty good, so I tried 5e-5, which turned out to be slightly better."
]
},
{
"cell_type": "code",
"execution_count": 119,
"metadata": {},
"outputs": [],
"source": [
"optimizer = tf.keras.optimizers.Nadam(learning_rate=5e-5)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\",\n",
" optimizer=optimizer,\n",
" metrics=[\"accuracy\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's load the CIFAR10 dataset. We also want to use early stopping, so we need a validation set. Let's use the first 5,000 images of the original training set as the validation set:"
]
},
{
"cell_type": "code",
"execution_count": 120,
"metadata": {},
"outputs": [],
"source": [
"cifar10 = tf.keras.datasets.cifar10.load_data()\n",
"(X_train_full, y_train_full), (X_test, y_test) = cifar10\n",
"\n",
"X_train = X_train_full[5000:]\n",
"y_train = y_train_full[5000:]\n",
"X_valid = X_train_full[:5000]\n",
"y_valid = y_train_full[:5000]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can create the callbacks we need and train the model:"
]
},
{
"cell_type": "code",
"execution_count": 121,
"metadata": {},
"outputs": [],
"source": [
"early_stopping_cb = tf.keras.callbacks.EarlyStopping(patience=20,\n",
" restore_best_weights=True)\n",
"model_checkpoint_cb = tf.keras.callbacks.ModelCheckpoint(\"my_cifar10_model\",\n",
" save_best_only=True)\n",
"run_index = 1 # increment every time you train the model\n",
"run_logdir = Path() / \"my_cifar10_logs\" / f\"run_{run_index:03d}\"\n",
"tensorboard_cb = tf.keras.callbacks.TensorBoard(run_logdir)\n",
"callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]"
]
},
{
"cell_type": "code",
"execution_count": 122,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%load_ext tensorboard\n",
"%tensorboard --logdir=./my_cifar10_logs"
]
},
{
"cell_type": "code",
"execution_count": 123,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/100\n",
"1404/1407 [============================>.] - ETA: 0s - loss: 4.0493 - accuracy: 0.1598INFO:tensorflow:Assets written to: my_cifar10_model/assets\n",
"1407/1407 [==============================] - 17s 10ms/step - loss: 4.0462 - accuracy: 0.1597 - val_loss: 2.1441 - val_accuracy: 0.2036\n",
"Epoch 2/100\n",
"1407/1407 [==============================] - ETA: 0s - loss: 2.0667 - accuracy: 0.2320INFO:tensorflow:Assets written to: my_cifar10_model/assets\n",
"1407/1407 [==============================] - 12s 9ms/step - loss: 2.0667 - accuracy: 0.2320 - val_loss: 2.0134 - val_accuracy: 0.2472\n",
"Epoch 3/100\n",
"1407/1407 [==============================] - ETA: 0s - loss: 1.9472 - accuracy: 0.2819INFO:tensorflow:Assets written to: my_cifar10_model/assets\n",
"1407/1407 [==============================] - 14s 10ms/step - loss: 1.9472 - accuracy: 0.2819 - val_loss: 1.9427 - val_accuracy: 0.2796\n",
"Epoch 4/100\n",
"1405/1407 [============================>.] - ETA: 0s - loss: 1.8636 - accuracy: 0.3182INFO:tensorflow:Assets written to: my_cifar10_model/assets\n",
"1407/1407 [==============================] - 14s 10ms/step - loss: 1.8637 - accuracy: 0.3182 - val_loss: 1.8934 - val_accuracy: 0.3222\n",
"Epoch 5/100\n",
"1402/1407 [============================>.] - ETA: 0s - loss: 1.7975 - accuracy: 0.3464INFO:tensorflow:Assets written to: my_cifar10_model/assets\n",
"1407/1407 [==============================] - 14s 10ms/step - loss: 1.7974 - accuracy: 0.3465 - val_loss: 1.8389 - val_accuracy: 0.3284\n",
"Epoch 6/100\n",
"1407/1407 [==============================] - 9s 7ms/step - loss: 1.7446 - accuracy: 0.3664 - val_loss: 2.0006 - val_accuracy: 0.3030\n",
"Epoch 7/100\n",
"1407/1407 [==============================] - ETA: 0s - loss: 1.6974 - accuracy: 0.3852INFO:tensorflow:Assets written to: my_cifar10_model/assets\n",
"1407/1407 [==============================] - 12s 8ms/step - loss: 1.6974 - accuracy: 0.3852 - val_loss: 1.7075 - val_accuracy: 0.3738\n",
"Epoch 8/100\n",
"1405/1407 [============================>.] - ETA: 0s - loss: 1.6605 - accuracy: 0.3984INFO:tensorflow:Assets written to: my_cifar10_model/assets\n",
"1407/1407 [==============================] - 12s 8ms/step - loss: 1.6604 - accuracy: 0.3984 - val_loss: 1.6788 - val_accuracy: 0.3836\n",
"Epoch 9/100\n",
"1405/1407 [============================>.] - ETA: 0s - loss: 1.6322 - accuracy: 0.4114INFO:tensorflow:Assets written to: my_cifar10_model/assets\n",
"1407/1407 [==============================] - 14s 10ms/step - loss: 1.6321 - accuracy: 0.4114 - val_loss: 1.6477 - val_accuracy: 0.4014\n",
"Epoch 10/100\n",
"1407/1407 [==============================] - 12s 8ms/step - loss: 1.6065 - accuracy: 0.4205 - val_loss: 1.6623 - val_accuracy: 0.3980\n",
"Epoch 11/100\n",
"1401/1407 [============================>.] - ETA: 0s - loss: 1.5843 - accuracy: 0.4287INFO:tensorflow:Assets written to: my_cifar10_model/assets\n",
"1407/1407 [==============================] - 14s 10ms/step - loss: 1.5845 - accuracy: 0.4285 - val_loss: 1.6032 - val_accuracy: 0.4198\n",
"Epoch 12/100\n",
"1407/1407 [==============================] - 9s 6ms/step - loss: 1.5634 - accuracy: 0.4367 - val_loss: 1.6063 - val_accuracy: 0.4258\n",
"Epoch 13/100\n",
"1401/1407 [============================>.] - ETA: 0s - loss: 1.5443 - accuracy: 0.4420INFO:tensorflow:Assets written to: my_cifar10_model/assets\n",
"<<47 more lines>>\n",
"1407/1407 [==============================] - 12s 8ms/step - loss: 1.3247 - accuracy: 0.5256 - val_loss: 1.5130 - val_accuracy: 0.4616\n",
"Epoch 33/100\n",
"1407/1407 [==============================] - 13s 9ms/step - loss: 1.3164 - accuracy: 0.5286 - val_loss: 1.5284 - val_accuracy: 0.4686\n",
"Epoch 34/100\n",
"1407/1407 [==============================] - 12s 9ms/step - loss: 1.3091 - accuracy: 0.5303 - val_loss: 1.5208 - val_accuracy: 0.4682\n",
"Epoch 35/100\n",
"1407/1407 [==============================] - 12s 8ms/step - loss: 1.3026 - accuracy: 0.5319 - val_loss: 1.5479 - val_accuracy: 0.4604\n",
"Epoch 36/100\n",
"1407/1407 [==============================] - 11s 8ms/step - loss: 1.2930 - accuracy: 0.5378 - val_loss: 1.5443 - val_accuracy: 0.4580\n",
"Epoch 37/100\n",
"1407/1407 [==============================] - 12s 8ms/step - loss: 1.2833 - accuracy: 0.5406 - val_loss: 1.5165 - val_accuracy: 0.4710\n",
"Epoch 38/100\n",
"1407/1407 [==============================] - 11s 8ms/step - loss: 1.2763 - accuracy: 0.5433 - val_loss: 1.5345 - val_accuracy: 0.4672\n",
"Epoch 39/100\n",
"1407/1407 [==============================] - 12s 9ms/step - loss: 1.2687 - accuracy: 0.5437 - val_loss: 1.5162 - val_accuracy: 0.4712\n",
"Epoch 40/100\n",
"1407/1407 [==============================] - 11s 7ms/step - loss: 1.2623 - accuracy: 0.5490 - val_loss: 1.5717 - val_accuracy: 0.4566\n",
"Epoch 41/100\n",
"1407/1407 [==============================] - 14s 10ms/step - loss: 1.2580 - accuracy: 0.5467 - val_loss: 1.5296 - val_accuracy: 0.4738\n",
"Epoch 42/100\n",
"1407/1407 [==============================] - 13s 9ms/step - loss: 1.2469 - accuracy: 0.5532 - val_loss: 1.5179 - val_accuracy: 0.4690\n",
"Epoch 43/100\n",
"1407/1407 [==============================] - 11s 8ms/step - loss: 1.2404 - accuracy: 0.5542 - val_loss: 1.5542 - val_accuracy: 0.4566\n",
"Epoch 44/100\n",
"1407/1407 [==============================] - 12s 8ms/step - loss: 1.2292 - accuracy: 0.5605 - val_loss: 1.5536 - val_accuracy: 0.4608\n",
"Epoch 45/100\n",
"1407/1407 [==============================] - 12s 9ms/step - loss: 1.2276 - accuracy: 0.5606 - val_loss: 1.5522 - val_accuracy: 0.4624\n",
"Epoch 46/100\n",
"1407/1407 [==============================] - 13s 9ms/step - loss: 1.2200 - accuracy: 0.5637 - val_loss: 1.5339 - val_accuracy: 0.4794\n",
"Epoch 47/100\n",
"1407/1407 [==============================] - 13s 9ms/step - loss: 1.2080 - accuracy: 0.5677 - val_loss: 1.5451 - val_accuracy: 0.4688\n",
"Epoch 48/100\n",
"1407/1407 [==============================] - 15s 10ms/step - loss: 1.2050 - accuracy: 0.5675 - val_loss: 1.5209 - val_accuracy: 0.4770\n",
"Epoch 49/100\n",
"1407/1407 [==============================] - 10s 7ms/step - loss: 1.1947 - accuracy: 0.5718 - val_loss: 1.5435 - val_accuracy: 0.4736\n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 123,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.fit(X_train, y_train, epochs=100,\n",
" validation_data=(X_valid, y_valid),\n",
" callbacks=callbacks)"
]
},
{
"cell_type": "code",
"execution_count": 124,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"157/157 [==============================] - 0s 2ms/step - loss: 1.5062 - accuracy: 0.4676\n"
]
},
{
"data": {
"text/plain": [
"[1.5061508417129517, 0.4675999879837036]"
]
},
"execution_count": 124,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.evaluate(X_valid, y_valid)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The model with the lowest validation loss gets about 46.8% accuracy on the validation set. It took 29 epochs to reach the lowest validation loss, with roughly 10 seconds per epoch on my laptop (without a GPU). Let's see if we can improve the model using Batch Normalization."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### c.\n",
"*Exercise: Now try adding Batch Normalization and compare the learning curves: Is it converging faster than before? Does it produce a better model? How does it affect training speed?*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The code below is very similar to the code above, with a few changes:\n",
"\n",
"* I added a BN layer after every Dense layer (before the activation function), except for the output layer.\n",
"* I changed the learning rate to 5e-4. I experimented with 1e-5, 3e-5, 5e-5, 1e-4, 3e-4, 5e-4, 1e-3 and 3e-3, and I chose the one with the best validation performance after 20 epochs.\n",
"* I renamed the run directories to run_bn_* and the model file name to `my_cifar10_bn_model`."
]
},
{
"cell_type": "code",
"execution_count": 125,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/100\n",
"1403/1407 [============================>.] - ETA: 0s - loss: 2.0377 - accuracy: 0.2523INFO:tensorflow:Assets written to: my_cifar10_bn_model/assets\n",
"1407/1407 [==============================] - 32s 18ms/step - loss: 2.0374 - accuracy: 0.2525 - val_loss: 1.8766 - val_accuracy: 0.3154\n",
"Epoch 2/100\n",
"1407/1407 [==============================] - 17s 12ms/step - loss: 1.7874 - accuracy: 0.3542 - val_loss: 1.8784 - val_accuracy: 0.3268\n",
"Epoch 3/100\n",
"1407/1407 [==============================] - 20s 15ms/step - loss: 1.6806 - accuracy: 0.3969 - val_loss: 1.9764 - val_accuracy: 0.3252\n",
"Epoch 4/100\n",
"1403/1407 [============================>.] - ETA: 0s - loss: 1.6111 - accuracy: 0.4229INFO:tensorflow:Assets written to: my_cifar10_bn_model/assets\n",
"1407/1407 [==============================] - 24s 17ms/step - loss: 1.6112 - accuracy: 0.4228 - val_loss: 1.7087 - val_accuracy: 0.3750\n",
"Epoch 5/100\n",
"1402/1407 [============================>.] - ETA: 0s - loss: 1.5520 - accuracy: 0.4478INFO:tensorflow:Assets written to: my_cifar10_bn_model/assets\n",
"1407/1407 [==============================] - 21s 15ms/step - loss: 1.5521 - accuracy: 0.4476 - val_loss: 1.6272 - val_accuracy: 0.4176\n",
"Epoch 6/100\n",
"1406/1407 [============================>.] - ETA: 0s - loss: 1.5030 - accuracy: 0.4659INFO:tensorflow:Assets written to: my_cifar10_bn_model/assets\n",
"1407/1407 [==============================] - 23s 16ms/step - loss: 1.5030 - accuracy: 0.4660 - val_loss: 1.5401 - val_accuracy: 0.4452\n",
"Epoch 7/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.4559 - accuracy: 0.4812 - val_loss: 1.6990 - val_accuracy: 0.3952\n",
"Epoch 8/100\n",
"1403/1407 [============================>.] - ETA: 0s - loss: 1.4169 - accuracy: 0.4987INFO:tensorflow:Assets written to: my_cifar10_bn_model/assets\n",
"1407/1407 [==============================] - 21s 15ms/step - loss: 1.4168 - accuracy: 0.4987 - val_loss: 1.5078 - val_accuracy: 0.4652\n",
"Epoch 9/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.3863 - accuracy: 0.5123 - val_loss: 1.5513 - val_accuracy: 0.4470\n",
"Epoch 10/100\n",
"1407/1407 [==============================] - 17s 12ms/step - loss: 1.3514 - accuracy: 0.5216 - val_loss: 1.5208 - val_accuracy: 0.4562\n",
"Epoch 11/100\n",
"1407/1407 [==============================] - 16s 12ms/step - loss: 1.3220 - accuracy: 0.5314 - val_loss: 1.7301 - val_accuracy: 0.4206\n",
"Epoch 12/100\n",
"1404/1407 [============================>.] - ETA: 0s - loss: 1.2933 - accuracy: 0.5410INFO:tensorflow:Assets written to: my_cifar10_bn_model/assets\n",
"1407/1407 [==============================] - 25s 18ms/step - loss: 1.2931 - accuracy: 0.5410 - val_loss: 1.4909 - val_accuracy: 0.4734\n",
"Epoch 13/100\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 1.2702 - accuracy: 0.5490 - val_loss: 1.5256 - val_accuracy: 0.4636\n",
"Epoch 14/100\n",
"1407/1407 [==============================] - 17s 12ms/step - loss: 1.2424 - accuracy: 0.5591 - val_loss: 1.5569 - val_accuracy: 0.4624\n",
"Epoch 15/100\n",
"<<12 more lines>>\n",
"Epoch 21/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.1174 - accuracy: 0.6066 - val_loss: 1.5241 - val_accuracy: 0.4828\n",
"Epoch 22/100\n",
"1407/1407 [==============================] - 18s 13ms/step - loss: 1.0978 - accuracy: 0.6128 - val_loss: 1.5313 - val_accuracy: 0.4772\n",
"Epoch 23/100\n",
"1407/1407 [==============================] - 17s 12ms/step - loss: 1.0844 - accuracy: 0.6198 - val_loss: 1.4993 - val_accuracy: 0.4924\n",
"Epoch 24/100\n",
"1407/1407 [==============================] - 17s 12ms/step - loss: 1.0677 - accuracy: 0.6244 - val_loss: 1.4622 - val_accuracy: 0.5078\n",
"Epoch 25/100\n",
"1407/1407 [==============================] - 18s 13ms/step - loss: 1.0571 - accuracy: 0.6297 - val_loss: 1.4917 - val_accuracy: 0.4990\n",
"Epoch 26/100\n",
"1407/1407 [==============================] - 19s 14ms/step - loss: 1.0395 - accuracy: 0.6327 - val_loss: 1.4888 - val_accuracy: 0.4896\n",
"Epoch 27/100\n",
"1407/1407 [==============================] - 18s 13ms/step - loss: 1.0298 - accuracy: 0.6370 - val_loss: 1.5358 - val_accuracy: 0.5024\n",
"Epoch 28/100\n",
"1407/1407 [==============================] - 18s 13ms/step - loss: 1.0150 - accuracy: 0.6444 - val_loss: 1.5219 - val_accuracy: 0.5030\n",
"Epoch 29/100\n",
"1407/1407 [==============================] - 16s 12ms/step - loss: 1.0100 - accuracy: 0.6456 - val_loss: 1.4933 - val_accuracy: 0.5098\n",
"Epoch 30/100\n",
"1407/1407 [==============================] - 20s 14ms/step - loss: 0.9956 - accuracy: 0.6492 - val_loss: 1.4756 - val_accuracy: 0.5012\n",
"Epoch 31/100\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 0.9787 - accuracy: 0.6576 - val_loss: 1.5181 - val_accuracy: 0.4936\n",
"Epoch 32/100\n",
"1407/1407 [==============================] - 17s 12ms/step - loss: 0.9710 - accuracy: 0.6565 - val_loss: 1.7510 - val_accuracy: 0.4568\n",
"Epoch 33/100\n",
"1407/1407 [==============================] - 20s 14ms/step - loss: 0.9613 - accuracy: 0.6628 - val_loss: 1.5576 - val_accuracy: 0.4910\n",
"Epoch 34/100\n",
"1407/1407 [==============================] - 19s 14ms/step - loss: 0.9530 - accuracy: 0.6651 - val_loss: 1.5087 - val_accuracy: 0.5046\n",
"Epoch 35/100\n",
"1407/1407 [==============================] - 19s 13ms/step - loss: 0.9388 - accuracy: 0.6701 - val_loss: 1.5534 - val_accuracy: 0.4950\n",
"Epoch 36/100\n",
"1407/1407 [==============================] - 17s 12ms/step - loss: 0.9331 - accuracy: 0.6743 - val_loss: 1.5033 - val_accuracy: 0.5046\n",
"Epoch 37/100\n",
"1407/1407 [==============================] - 19s 14ms/step - loss: 0.9144 - accuracy: 0.6808 - val_loss: 1.5679 - val_accuracy: 0.5028\n",
"157/157 [==============================] - 0s 2ms/step - loss: 1.4236 - accuracy: 0.5074\n"
]
},
{
"data": {
"text/plain": [
"[1.4236289262771606, 0.5073999762535095]"
]
},
"execution_count": 125,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tf.random.set_seed(42)\n",
"\n",
"model = tf.keras.Sequential()\n",
"model.add(tf.keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
"for _ in range(20):\n",
" model.add(tf.keras.layers.Dense(100, kernel_initializer=\"he_normal\"))\n",
" model.add(tf.keras.layers.BatchNormalization())\n",
" model.add(tf.keras.layers.Activation(\"swish\"))\n",
"\n",
"model.add(tf.keras.layers.Dense(10, activation=\"softmax\"))\n",
"\n",
"optimizer = tf.keras.optimizers.Nadam(learning_rate=5e-4)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\",\n",
" optimizer=optimizer,\n",
" metrics=[\"accuracy\"])\n",
"\n",
"early_stopping_cb = tf.keras.callbacks.EarlyStopping(patience=20,\n",
" restore_best_weights=True)\n",
"model_checkpoint_cb = tf.keras.callbacks.ModelCheckpoint(\"my_cifar10_bn_model\",\n",
" save_best_only=True)\n",
"run_index = 1 # increment every time you train the model\n",
"run_logdir = Path() / \"my_cifar10_logs\" / f\"run_bn_{run_index:03d}\"\n",
"tensorboard_cb = tf.keras.callbacks.TensorBoard(run_logdir)\n",
"callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]\n",
"\n",
"model.fit(X_train, y_train, epochs=100,\n",
" validation_data=(X_valid, y_valid),\n",
" callbacks=callbacks)\n",
"\n",
"model.evaluate(X_valid, y_valid)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* *Is the model converging faster than before?* Much faster! The previous model took 29 epochs to reach the lowest validation loss, while the new model achieved that same loss in just 12 epochs and continued to make progress until the 17th epoch. The BN layers stabilized training and allowed us to use a much larger learning rate, so convergence was faster.\n",
"* *Does BN produce a better model?* Yes! The final model is also much better, with 50.7% validation accuracy instead of 46.7%. It's still not a very good model, but at least it's much better than before (a Convolutional Neural Network would do much better, but that's a different topic, see chapter 14).\n",
"* *How does BN affect training speed?* Although the model converged much faster, each epoch took about 15s instead of 10s, because of the extra computations required by the BN layers. But overall the training time (wall time) to reach the best model was shortened by about 10%."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### d.\n",
"*Exercise: Try replacing Batch Normalization with SELU, and make the necessary adjustements to ensure the network self-normalizes (i.e., standardize the input features, use LeCun normal initialization, make sure the DNN contains only a sequence of dense layers, etc.).*"
]
},
{
"cell_type": "code",
"execution_count": 126,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/100\n",
"1403/1407 [============================>.] - ETA: 0s - loss: 1.9386 - accuracy: 0.3045INFO:tensorflow:Assets written to: my_cifar10_selu_model/assets\n",
"1407/1407 [==============================] - 20s 13ms/step - loss: 1.9385 - accuracy: 0.3046 - val_loss: 1.8175 - val_accuracy: 0.3510\n",
"Epoch 2/100\n",
"1405/1407 [============================>.] - ETA: 0s - loss: 1.7241 - accuracy: 0.3869INFO:tensorflow:Assets written to: my_cifar10_selu_model/assets\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 1.7241 - accuracy: 0.3869 - val_loss: 1.7677 - val_accuracy: 0.3614\n",
"Epoch 3/100\n",
"1407/1407 [==============================] - ETA: 0s - loss: 1.6272 - accuracy: 0.4263INFO:tensorflow:Assets written to: my_cifar10_selu_model/assets\n",
"1407/1407 [==============================] - 18s 13ms/step - loss: 1.6272 - accuracy: 0.4263 - val_loss: 1.6878 - val_accuracy: 0.4054\n",
"Epoch 4/100\n",
"1406/1407 [============================>.] - ETA: 0s - loss: 1.5644 - accuracy: 0.4492INFO:tensorflow:Assets written to: my_cifar10_selu_model/assets\n",
"1407/1407 [==============================] - 18s 13ms/step - loss: 1.5643 - accuracy: 0.4492 - val_loss: 1.6589 - val_accuracy: 0.4304\n",
"Epoch 5/100\n",
"1404/1407 [============================>.] - ETA: 0s - loss: 1.5080 - accuracy: 0.4712INFO:tensorflow:Assets written to: my_cifar10_selu_model/assets\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 1.5080 - accuracy: 0.4712 - val_loss: 1.5651 - val_accuracy: 0.4538\n",
"Epoch 6/100\n",
"1404/1407 [============================>.] - ETA: 0s - loss: 1.4611 - accuracy: 0.4873INFO:tensorflow:Assets written to: my_cifar10_selu_model/assets\n",
"1407/1407 [==============================] - 17s 12ms/step - loss: 1.4613 - accuracy: 0.4872 - val_loss: 1.5305 - val_accuracy: 0.4678\n",
"Epoch 7/100\n",
"1407/1407 [==============================] - 17s 12ms/step - loss: 1.4174 - accuracy: 0.5077 - val_loss: 1.5346 - val_accuracy: 0.4558\n",
"Epoch 8/100\n",
"1406/1407 [============================>.] - ETA: 0s - loss: 1.3781 - accuracy: 0.5175INFO:tensorflow:Assets written to: my_cifar10_selu_model/assets\n",
"1407/1407 [==============================] - 17s 12ms/step - loss: 1.3781 - accuracy: 0.5175 - val_loss: 1.4773 - val_accuracy: 0.4882\n",
"Epoch 9/100\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 1.3413 - accuracy: 0.5345 - val_loss: 1.5021 - val_accuracy: 0.4764\n",
"Epoch 10/100\n",
"1407/1407 [==============================] - 15s 10ms/step - loss: 1.3182 - accuracy: 0.5422 - val_loss: 1.5709 - val_accuracy: 0.4762\n",
"Epoch 11/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.2832 - accuracy: 0.5571 - val_loss: 1.5345 - val_accuracy: 0.4868\n",
"Epoch 12/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.2557 - accuracy: 0.5667 - val_loss: 1.5024 - val_accuracy: 0.4900\n",
"Epoch 13/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.2373 - accuracy: 0.5710 - val_loss: 1.5114 - val_accuracy: 0.5028\n",
"Epoch 14/100\n",
"1404/1407 [============================>.] - ETA: 0s - loss: 1.2071 - accuracy: 0.5846INFO:tensorflow:Assets written to: my_cifar10_selu_model/assets\n",
"1407/1407 [==============================] - 17s 12ms/step - loss: 1.2073 - accuracy: 0.5847 - val_loss: 1.4608 - val_accuracy: 0.5026\n",
"Epoch 15/100\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 1.1843 - accuracy: 0.5940 - val_loss: 1.4962 - val_accuracy: 0.5038\n",
"Epoch 16/100\n",
"1407/1407 [==============================] - 16s 12ms/step - loss: 1.1617 - accuracy: 0.6026 - val_loss: 1.5255 - val_accuracy: 0.5062\n",
"Epoch 17/100\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 1.1452 - accuracy: 0.6084 - val_loss: 1.5057 - val_accuracy: 0.5036\n",
"Epoch 18/100\n",
"1407/1407 [==============================] - 17s 12ms/step - loss: 1.1297 - accuracy: 0.6145 - val_loss: 1.5097 - val_accuracy: 0.5010\n",
"Epoch 19/100\n",
"1407/1407 [==============================] - 16s 12ms/step - loss: 1.1004 - accuracy: 0.6245 - val_loss: 1.5218 - val_accuracy: 0.5014\n",
"Epoch 20/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.0971 - accuracy: 0.6304 - val_loss: 1.5253 - val_accuracy: 0.5090\n",
"Epoch 21/100\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 1.0670 - accuracy: 0.6345 - val_loss: 1.5006 - val_accuracy: 0.5034\n",
"Epoch 22/100\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 1.0544 - accuracy: 0.6407 - val_loss: 1.5244 - val_accuracy: 0.5010\n",
"Epoch 23/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.0338 - accuracy: 0.6502 - val_loss: 1.5355 - val_accuracy: 0.5096\n",
"Epoch 24/100\n",
"1407/1407 [==============================] - 14s 10ms/step - loss: 1.0281 - accuracy: 0.6514 - val_loss: 1.5257 - val_accuracy: 0.5164\n",
"Epoch 25/100\n",
"1407/1407 [==============================] - 14s 10ms/step - loss: 1.4097 - accuracy: 0.6478 - val_loss: 1.8203 - val_accuracy: 0.3514\n",
"Epoch 26/100\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 1.3733 - accuracy: 0.5157 - val_loss: 1.5600 - val_accuracy: 0.4664\n",
"Epoch 27/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.2032 - accuracy: 0.5814 - val_loss: 1.5367 - val_accuracy: 0.4944\n",
"Epoch 28/100\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 1.1291 - accuracy: 0.6121 - val_loss: 1.5333 - val_accuracy: 0.4852\n",
"Epoch 29/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.0734 - accuracy: 0.6317 - val_loss: 1.5475 - val_accuracy: 0.5032\n",
"Epoch 30/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.0294 - accuracy: 0.6469 - val_loss: 1.5400 - val_accuracy: 0.5052\n",
"Epoch 31/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.0081 - accuracy: 0.6605 - val_loss: 1.5617 - val_accuracy: 0.4856\n",
"Epoch 32/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.0109 - accuracy: 0.6603 - val_loss: 1.5727 - val_accuracy: 0.5124\n",
"Epoch 33/100\n",
"1407/1407 [==============================] - 17s 12ms/step - loss: 0.9646 - accuracy: 0.6762 - val_loss: 1.5333 - val_accuracy: 0.5174\n",
"Epoch 34/100\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 0.9597 - accuracy: 0.6789 - val_loss: 1.5601 - val_accuracy: 0.5016\n",
"157/157 [==============================] - 0s 1ms/step - loss: 1.4608 - accuracy: 0.5026\n"
]
},
{
"data": {
"text/plain": [
"[1.4607702493667603, 0.5026000142097473]"
]
},
"execution_count": 126,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tf.random.set_seed(42)\n",
"\n",
"model = tf.keras.Sequential()\n",
"model.add(tf.keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
"for _ in range(20):\n",
" model.add(tf.keras.layers.Dense(100,\n",
" kernel_initializer=\"lecun_normal\",\n",
" activation=\"selu\"))\n",
"\n",
"model.add(tf.keras.layers.Dense(10, activation=\"softmax\"))\n",
"\n",
"optimizer = tf.keras.optimizers.Nadam(learning_rate=7e-4)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\",\n",
" optimizer=optimizer,\n",
" metrics=[\"accuracy\"])\n",
"\n",
"early_stopping_cb = tf.keras.callbacks.EarlyStopping(\n",
" patience=20, restore_best_weights=True)\n",
"model_checkpoint_cb = tf.keras.callbacks.ModelCheckpoint(\n",
" \"my_cifar10_selu_model\", save_best_only=True)\n",
"run_index = 1 # increment every time you train the model\n",
"run_logdir = Path() / \"my_cifar10_logs\" / f\"run_selu_{run_index:03d}\"\n",
"tensorboard_cb = tf.keras.callbacks.TensorBoard(run_logdir)\n",
"callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]\n",
"\n",
"X_means = X_train.mean(axis=0)\n",
"X_stds = X_train.std(axis=0)\n",
"X_train_scaled = (X_train - X_means) / X_stds\n",
"X_valid_scaled = (X_valid - X_means) / X_stds\n",
"X_test_scaled = (X_test - X_means) / X_stds\n",
"\n",
"model.fit(X_train_scaled, y_train, epochs=100,\n",
" validation_data=(X_valid_scaled, y_valid),\n",
" callbacks=callbacks)\n",
"\n",
"model.evaluate(X_valid_scaled, y_valid)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This model reached the first model's validation loss in just 8 epochs. After 14 epochs, it reached its lowest validation loss, with about 50.3% accuracy, which is better than the original model (46.7%), but not quite as good as the model using batch normalization (50.7%). Each epoch took only 9 seconds. So it's the fastest model to train so far."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### e.\n",
"*Exercise: Try regularizing the model with alpha dropout. Then, without retraining your model, see if you can achieve better accuracy using MC Dropout.*"
]
},
{
"cell_type": "code",
"execution_count": 127,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/100\n",
"1405/1407 [============================>.] - ETA: 0s - loss: 1.8953 - accuracy: 0.3240INFO:tensorflow:Assets written to: my_cifar10_alpha_dropout_model/assets\n",
"1407/1407 [==============================] - 18s 11ms/step - loss: 1.8950 - accuracy: 0.3239 - val_loss: 1.7556 - val_accuracy: 0.3812\n",
"Epoch 2/100\n",
"1403/1407 [============================>.] - ETA: 0s - loss: 1.6618 - accuracy: 0.4129INFO:tensorflow:Assets written to: my_cifar10_alpha_dropout_model/assets\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 1.6618 - accuracy: 0.4130 - val_loss: 1.6563 - val_accuracy: 0.4114\n",
"Epoch 3/100\n",
"1402/1407 [============================>.] - ETA: 0s - loss: 1.5772 - accuracy: 0.4431INFO:tensorflow:Assets written to: my_cifar10_alpha_dropout_model/assets\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 1.5770 - accuracy: 0.4432 - val_loss: 1.6507 - val_accuracy: 0.4232\n",
"Epoch 4/100\n",
"1406/1407 [============================>.] - ETA: 0s - loss: 1.5081 - accuracy: 0.4673INFO:tensorflow:Assets written to: my_cifar10_alpha_dropout_model/assets\n",
"1407/1407 [==============================] - 15s 10ms/step - loss: 1.5081 - accuracy: 0.4672 - val_loss: 1.5892 - val_accuracy: 0.4566\n",
"Epoch 5/100\n",
"1403/1407 [============================>.] - ETA: 0s - loss: 1.4560 - accuracy: 0.4902INFO:tensorflow:Assets written to: my_cifar10_alpha_dropout_model/assets\n",
"1407/1407 [==============================] - 14s 10ms/step - loss: 1.4561 - accuracy: 0.4902 - val_loss: 1.5382 - val_accuracy: 0.4696\n",
"Epoch 6/100\n",
"1401/1407 [============================>.] - ETA: 0s - loss: 1.4095 - accuracy: 0.5050INFO:tensorflow:Assets written to: my_cifar10_alpha_dropout_model/assets\n",
"1407/1407 [==============================] - 16s 11ms/step - loss: 1.4094 - accuracy: 0.5050 - val_loss: 1.5236 - val_accuracy: 0.4818\n",
"Epoch 7/100\n",
"1401/1407 [============================>.] - ETA: 0s - loss: 1.3634 - accuracy: 0.5234INFO:tensorflow:Assets written to: my_cifar10_alpha_dropout_model/assets\n",
"1407/1407 [==============================] - 14s 10ms/step - loss: 1.3636 - accuracy: 0.5232 - val_loss: 1.5139 - val_accuracy: 0.4840\n",
"Epoch 8/100\n",
"1405/1407 [============================>.] - ETA: 0s - loss: 1.3297 - accuracy: 0.5377INFO:tensorflow:Assets written to: my_cifar10_alpha_dropout_model/assets\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.3296 - accuracy: 0.5378 - val_loss: 1.4780 - val_accuracy: 0.4982\n",
"Epoch 9/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.2907 - accuracy: 0.5485 - val_loss: 1.5151 - val_accuracy: 0.4854\n",
"Epoch 10/100\n",
"1407/1407 [==============================] - 13s 10ms/step - loss: 1.2559 - accuracy: 0.5646 - val_loss: 1.4980 - val_accuracy: 0.4976\n",
"Epoch 11/100\n",
"1407/1407 [==============================] - 14s 10ms/step - loss: 1.2221 - accuracy: 0.5767 - val_loss: 1.5199 - val_accuracy: 0.4990\n",
"Epoch 12/100\n",
"1407/1407 [==============================] - 13s 9ms/step - loss: 1.1960 - accuracy: 0.5870 - val_loss: 1.5167 - val_accuracy: 0.5030\n",
"Epoch 13/100\n",
"1407/1407 [==============================] - 14s 10ms/step - loss: 1.1684 - accuracy: 0.5955 - val_loss: 1.5815 - val_accuracy: 0.5014\n",
"Epoch 14/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.1463 - accuracy: 0.6025 - val_loss: 1.5427 - val_accuracy: 0.5112\n",
"Epoch 15/100\n",
"1407/1407 [==============================] - 13s 9ms/step - loss: 1.1125 - accuracy: 0.6169 - val_loss: 1.5868 - val_accuracy: 0.5212\n",
"Epoch 16/100\n",
"1407/1407 [==============================] - 12s 8ms/step - loss: 1.0854 - accuracy: 0.6243 - val_loss: 1.6234 - val_accuracy: 0.5090\n",
"Epoch 17/100\n",
"1407/1407 [==============================] - 15s 11ms/step - loss: 1.0668 - accuracy: 0.6328 - val_loss: 1.6162 - val_accuracy: 0.5072\n",
"Epoch 18/100\n",
"1407/1407 [==============================] - 15s 10ms/step - loss: 1.0440 - accuracy: 0.6442 - val_loss: 1.5748 - val_accuracy: 0.5162\n",
"Epoch 19/100\n",
"1407/1407 [==============================] - 12s 9ms/step - loss: 1.0272 - accuracy: 0.6477 - val_loss: 1.6518 - val_accuracy: 0.5200\n",
"Epoch 20/100\n",
"1407/1407 [==============================] - 13s 10ms/step - loss: 1.0007 - accuracy: 0.6594 - val_loss: 1.6224 - val_accuracy: 0.5186\n",
"Epoch 21/100\n",
"1407/1407 [==============================] - 15s 10ms/step - loss: 0.9824 - accuracy: 0.6639 - val_loss: 1.6972 - val_accuracy: 0.5136\n",
"Epoch 22/100\n",
"1407/1407 [==============================] - 12s 9ms/step - loss: 0.9660 - accuracy: 0.6714 - val_loss: 1.7210 - val_accuracy: 0.5278\n",
"Epoch 23/100\n",
"1407/1407 [==============================] - 13s 10ms/step - loss: 0.9472 - accuracy: 0.6780 - val_loss: 1.6436 - val_accuracy: 0.5006\n",
"Epoch 24/100\n",
"1407/1407 [==============================] - 14s 10ms/step - loss: 0.9314 - accuracy: 0.6819 - val_loss: 1.7059 - val_accuracy: 0.5160\n",
"Epoch 25/100\n",
"1407/1407 [==============================] - 13s 9ms/step - loss: 0.9172 - accuracy: 0.6888 - val_loss: 1.6926 - val_accuracy: 0.5200\n",
"Epoch 26/100\n",
"1407/1407 [==============================] - 14s 10ms/step - loss: 0.8990 - accuracy: 0.6947 - val_loss: 1.7705 - val_accuracy: 0.5148\n",
"Epoch 27/100\n",
"1407/1407 [==============================] - 13s 9ms/step - loss: 0.8758 - accuracy: 0.7028 - val_loss: 1.7023 - val_accuracy: 0.5198\n",
"Epoch 28/100\n",
"1407/1407 [==============================] - 12s 8ms/step - loss: 0.8622 - accuracy: 0.7090 - val_loss: 1.7567 - val_accuracy: 0.5184\n",
"157/157 [==============================] - 0s 1ms/step - loss: 1.4780 - accuracy: 0.4982\n"
]
},
{
"data": {
"text/plain": [
"[1.4779616594314575, 0.498199999332428]"
]
},
"execution_count": 127,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tf.random.set_seed(42)\n",
"\n",
"model = tf.keras.Sequential()\n",
"model.add(tf.keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
"for _ in range(20):\n",
" model.add(tf.keras.layers.Dense(100,\n",
" kernel_initializer=\"lecun_normal\",\n",
" activation=\"selu\"))\n",
"\n",
"model.add(tf.keras.layers.AlphaDropout(rate=0.1))\n",
"model.add(tf.keras.layers.Dense(10, activation=\"softmax\"))\n",
"\n",
"optimizer = tf.keras.optimizers.Nadam(learning_rate=5e-4)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\",\n",
" optimizer=optimizer,\n",
" metrics=[\"accuracy\"])\n",
"\n",
"early_stopping_cb = tf.keras.callbacks.EarlyStopping(\n",
" patience=20, restore_best_weights=True)\n",
"model_checkpoint_cb = tf.keras.callbacks.ModelCheckpoint(\n",
" \"my_cifar10_alpha_dropout_model\", save_best_only=True)\n",
"run_index = 1 # increment every time you train the model\n",
"run_logdir = Path() / \"my_cifar10_logs\" / f\"run_alpha_dropout_{run_index:03d}\"\n",
"tensorboard_cb = tf.keras.callbacks.TensorBoard(run_logdir)\n",
"callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]\n",
"\n",
"X_means = X_train.mean(axis=0)\n",
"X_stds = X_train.std(axis=0)\n",
"X_train_scaled = (X_train - X_means) / X_stds\n",
"X_valid_scaled = (X_valid - X_means) / X_stds\n",
"X_test_scaled = (X_test - X_means) / X_stds\n",
"\n",
"model.fit(X_train_scaled, y_train, epochs=100,\n",
" validation_data=(X_valid_scaled, y_valid),\n",
" callbacks=callbacks)\n",
"\n",
"model.evaluate(X_valid_scaled, y_valid)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The model reaches 48.1% accuracy on the validation set. That's worse than without dropout (50.3%). With an extensive hyperparameter search, it might be possible to do better (I tried dropout rates of 5%, 10%, 20% and 40%, and learning rates 1e-4, 3e-4, 5e-4, and 1e-3), but probably not much better in this case."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's use MC Dropout now. We will need the `MCAlphaDropout` class we used earlier, so let's just copy it here for convenience:"
]
},
{
"cell_type": "code",
"execution_count": 128,
"metadata": {},
"outputs": [],
"source": [
"class MCAlphaDropout(tf.keras.layers.AlphaDropout):\n",
" def call(self, inputs):\n",
" return super().call(inputs, training=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's create a new model, identical to the one we just trained (with the same weights), but with `MCAlphaDropout` dropout layers instead of `AlphaDropout` layers:"
]
},
{
"cell_type": "code",
"execution_count": 129,
"metadata": {},
"outputs": [],
"source": [
"mc_model = tf.keras.Sequential([\n",
" (\n",
" MCAlphaDropout(layer.rate)\n",
" if isinstance(layer, tf.keras.layers.AlphaDropout)\n",
" else layer\n",
" )\n",
" for layer in model.layers\n",
"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then let's add a couple utility functions. The first will run the model many times (10 by default) and it will return the mean predicted class probabilities. The second will use these mean probabilities to predict the most likely class for each instance:"
]
},
{
"cell_type": "code",
"execution_count": 130,
"metadata": {},
"outputs": [],
"source": [
"def mc_dropout_predict_probas(mc_model, X, n_samples=10):\n",
" Y_probas = [mc_model.predict(X) for sample in range(n_samples)]\n",
" return np.mean(Y_probas, axis=0)\n",
"\n",
"def mc_dropout_predict_classes(mc_model, X, n_samples=10):\n",
" Y_probas = mc_dropout_predict_probas(mc_model, X, n_samples)\n",
" return Y_probas.argmax(axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's make predictions for all the instances in the validation set, and compute the accuracy:"
]
},
{
"cell_type": "code",
"execution_count": 131,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.4984"
]
},
"execution_count": 131,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tf.random.set_seed(42)\n",
"\n",
"y_pred = mc_dropout_predict_classes(mc_model, X_valid_scaled)\n",
"accuracy = (y_pred == y_valid[:, 0]).mean()\n",
"accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We get back to roughly the accuracy of the model without dropout in this case (about 50.3% accuracy).\n",
"\n",
"So the best model we got in this exercise is the Batch Normalization model."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### f.\n",
"*Exercise: Retrain your model using 1cycle scheduling and see if it improves training speed and model accuracy.*"
]
},
{
"cell_type": "code",
"execution_count": 132,
"metadata": {},
"outputs": [],
"source": [
"tf.random.set_seed(42)\n",
"\n",
"model = tf.keras.Sequential()\n",
"model.add(tf.keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
"for _ in range(20):\n",
" model.add(tf.keras.layers.Dense(100,\n",
" kernel_initializer=\"lecun_normal\",\n",
" activation=\"selu\"))\n",
"\n",
"model.add(tf.keras.layers.AlphaDropout(rate=0.1))\n",
"model.add(tf.keras.layers.Dense(10, activation=\"softmax\"))\n",
"\n",
"optimizer = tf.keras.optimizers.SGD()\n",
"model.compile(loss=\"sparse_categorical_crossentropy\",\n",
" optimizer=optimizer,\n",
" metrics=[\"accuracy\"])"
]
},
{
"cell_type": "code",
"execution_count": 133,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"352/352 [==============================] - 3s 8ms/step - loss: nan - accuracy: 0.1706\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEOCAYAAACKDawAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAu0klEQVR4nO3deXxU5bkH8N8TguwQE5IQdhDBIIoIolcqm0VxrVtdqN66VCrKtVbrVtuqvS612l4XXFFBiytaXBBQSwkoWhBEWYxBEFBZw04WAiTv/eOZ13NmMjOZCZMzkzm/7+eTz2xn5rxzkjznPc+7iTEGRESU/jKSXQAiIvIGAz4RkU8w4BMR+QQDPhGRTzDgExH5BAM+EZFPZCa7ANFkZWWZXr16JbsYaaO8vBytWrVKdjHSAo9lYiXzeJaUAJWVQHU1kJkJ9O+flGIkzOLFi7caY3LDvZbSAT8/Px+LFi1KdjHSRlFREYYPH57sYqQFHsvESubxHDYMKC4GSkuBrCygsYccEVkX6TWmdIjI14wBMnwSCX3yNYmIwqupAZo00fvpPvEAAz4R+ZoxDPhERL7gTukw4BMRpTGmdIiIfII1fCIin2AOn4jIJ9wpnXTHgE9EvsaUDhGRTzClQ0TkE+ylQ0TkE0zpEBH5BFM6REQ+wZQOEZFPMKVDROQTnB6ZiMgnmNIhIvIJNtoSEfkEc/hERD7BlA4RkU8wpUNE5BOs4RMR+QS7ZRIR+QRTOkREPlFTwxo+EZEv7NsHNG+e7FJ4gwGfiHytqooBn4jIFxjwiYh8wBhg716gRYtkl8QbDPhE5Fv79+stAz4RUZqrqtJbpnSIiNIcAz4RkU/s3au3TOkQEaU51vCJiHzCBvxmzZJbDq8w4BORbzHgNzARaSIiS0Rkutf7JiJyszl8BvyG8xsAxUnYLxFRkNAc/hVXJK8sXsj0cmci0hnAGQDuBXCjl/smIgrlTulUVQGZnkZE73n99R4GcAuANpE2EJGxAMYCQG5uLoqKijwpmB+UlZXxeCYIj2ViJet4LlqUA+AoLFu2CPv2lXm+f695FvBF5EwAW4wxi0VkeKTtjDHPAHgGAPr06WOGD4+4KcWpqKgIPJ6JwWOZWMk6nqWlejtkyCD06+f57j3nZQ5/CICzRWQtgFcBjBSRKR7un4goiG20ZT/8BDPG3G6M6WyM6Q7gYgD/NsZc6tX+iYhCsVsmEZFP+C3gJ6VN2hhTBKAoGfsmIrL8FvBZwyci32IOn4jIJ2wN/5BDklsOrzDgE5FvVVVpsBdJdkm8wYBPRL5VVeWf/D3AgE9EPrZ3LwM+EZEvVFX5p8EWYMAnIh9jSoeIyCcY8ImIfII5fCIin2AOn4jIJ5jSISLyCQZ8IiKfYA6fiMgn9u5lDp+IyBf27AHatUt2KbzDgE9EvrVrF9C2bbJL4R0GfCLypQMHgIoKBnwiorS3Z4/eMuATEaW53bv1lgGfiCjNMeATEfkEAz4RkU8w4BMR+YQN+OyHT0SU5ljDb2SM0R8iongx4DcyZ54JtGyZ7FIQUWO0ezcgArRqleySeCcz2QU4GDNmJLsERNRY7d4NtGkDZDTqam98Uvqr7tuXgexsYM0aYMcOYMqU8NsxrUNE8fLbPDpAigf8qqoM7NgBlJQAkyYBl10GfP+9vnbggLOdHSJNRBSr3bsZ8FNKdbUA0DPx6tX63MaNert+vbPdjh3O/TvvBF57zaMCElHK2bEDuPtuoLo6+nYM+CmmpqZ2wN+0SW/XrXO2cwf8CROA55/3qIBElHLefRe46y5g6dLo223f7q8++ECKB3x3Df/bb/W5zZv1du1aZzsb8Kuq9Jf4zTfelZGIUsuWLXpru12GU1GhJ4Sjj/amTKmiUQT87dudAB+thu9+rarKmzISUWopLdXbaG178+cD+/cDI0d6U6ZU0SgC/ooV+ssBnKBuG28BPSHU1Div1dQ4VwRuX3/NHj1E6S6WGv6cOUBmJvCTn3hTplSR0gHf5vCXLHGesymdrVuBrl31/tVXAxdc4DToAsDKlcGf9e67QGEh8MorDVhgIko6G/Cj1fAXLAAGDABat/amTKnCs4AvIs1FZKGIfCkiK0Tk7rreY2v4P/ygj7t2dWrx27YB3bsDTZro4w8+CO658803zlUBADz5pN5+8snBfhMiSmWxpHRKS4GOHb0pTyrxsoZfBWCkMaY/gGMAjBaRE6K9wQZ8a9Cg4Bp++/aavgGA8nKgqEiHSnfooIO0DjkEePppfc+sWbrdnDnAffcBe/cm8qsRUaqIJaWzYwdw6KHelCeVeBbwjSoLPGwa+ImaUXcH/Pz82jX8nJzgnPw77wB5ecCDDwJffqnPzZ6tDb7GaErnq6+AO+4Apk1L2FcjohQSWsOfMwcYNy54mx07gOxsb8uVCjydS0dEmgBYDKAXgMeNMQvCbDMWwFh9NPDH59u124OKii0oKzsM06d/hK1bh6C8/HsA3QAAmZk12LcvA61bl6FTp0W44YaOePjh3ti0qRSzZ28CcBT69fsexcVdAADPPrsZBQXFDfp9U01ZWRmKioqSXYy0wGOZWIk6npWVGaioGAoAKCnZiKKiEjz55GGYOrULBg78DJs2tcDxx29Defkw7Ny5BkVF6+r4xDRjjPH8B0AWgDkA+kXfbqBp0UInQT7nHGPeflvvz5yptw89ZCdINuaii/T2xBPNj04+2ZgTTjDm2Wf1tZISY+67z5jzzzcmK8uYfftMVI88YsykSdG3SaQZM4yprGy4z58zZ07DfbjP8FgmVqKO55o1Tkz4+c/1uauuch7n5hqzaZM+fvzxhOwy5QBYZCLE1KT00jHG7ARQBGB0Xdva6Y87dwb699f7//633ubkAC+/DDz0EPDii9ow++CDznsLCrTnjs3pdekC3H47MGYMsHNn3Q24f/878MwzcXyxg7B8OXD66cD48d7sjygd2f91wMnh79rlvLZzp3bjBpjDb1AikisiWYH7LQD8FMDX0d7Tps0BnH++3s/O1hx+Vpbm5QFttL3kEuCmm7SB9pprgBNPdN5vA/7mzToNaosW+vyoUbr9u+9G3ndlJfDdd9rf/803gc8/r9/3jpX945w3r2H3Q5TONmzQ23btnBz+zp16u3Wr9tyzvfkY8BtWAYA5IrIUwGcAPjTGTI/6hoLKH4N027baA6d/fyf45uTUscMCYN8+nW0zL895vk0bYPjw6AF/9Wq9MNywAbjiCr0yaEj2j9JdQyGi+BQHmuUGDXICvq3h28bcNWv01o+Ntl720llqjBlgjDnaGNPPGPPn2N6nt/aXY9M6QN0B3/az/fLL4IAP6GpZK1fqL7+8HPjVr4CBA7UXz9y5wFNP6XY1NfqH8/HHevKoj/vv155B0djLTPvHSUTx++orTf927Fg7pbNtm97agO/HGn7Kr3j1pz/p4KoxY/Tx2WcDjz6q99u3j/7eggK93bgRGDw4+DU7pHrBAr0CeP55TRcdc0zwgC2rogJYuDD+odjPPgv8/vd6/957I29nA77dF5duJIpfcTHQt69mBEJr+Ha6ZNbwU1hOjjaeNmumj92THWVlRX+vDfhA7Rp+v35A8+bAZ59p0D/qKG04HTtWXwvnyiu1T2883nxTb0WcQWLhuAM+Z/skil9NjRPw27TRGr4xta+a7TxbdcWPdJTyAT+UiI6avfHGutei7NRJG2cBIDc3+LWmTXUujYULNeXTv79eBk6Y4Myjba8gWrUC7rlHA/Frr8WX2ikp0VtjdL6fG24Iv5293AR0kjciis933+nVcWGhBvwDB7SWHzqqfu1avQKw07L4SaML+ABw6qnA3/5W93YtWwK33KL3beB3GzxYc/MbNgTPiy2ijairVukfRu/emoMfNEhPBnl5wCmnBE/BvH69pm/ctfjKSv3jOuoofTxtmtOl1M0YDfj2KsQu9kJEsbPdrPv3d1aysvNwuW3Z4s90DtBIA3487rwT+L//A37969qvnX66c9/dGAxot6527TTI27x99+7Af/6jl4gffgi89JLWIMaP14aiq6/W9JC1apUG8xEjnOfcE7wBerWQkaGzeHbtqvMArV6ttRUiit20afr/c9xxzkpW7oWS3Bjw60lEmiaiIA0lM1PTKB061H5t1Ci99ANqB3zrww+Bhx/W+z16BM/ds3y5jgl4/HGnveDzz/Xnlluc1Iw74G/frjV/y04GB2h7xWGHaQNyt27Av/4FfPQRcPzxwe8homCVlcCMGcA552gFyqZjI10t9+rlWdFSSlwBX0SuF5HzXY+fA1ApIiUi0ifhpWtgIppjf+212o26VkaG01bQvbveNmumDbslJc68+8XF+ke2ZAnw3HM64nfqVH1t6NDgz7SDQwCnbzCgtY7DDnMeP/YY8Prr2s5g2wISYckS4OKLNcdJlA6WL9f8/Smn6GP7/xypA8QRR3hTrlQTbw3/egClACAiQwFcCGAMgC8AxJBVTz0FBcCFF8a2bY8eenvEEdowtHKl/kHl5ekl5LHHau3eNvpOnao9BrKzgy8h3Wkd90CrVq2Cax7TpzuDw9as0auL4uKDT/dMn64nOfeqYUSN2datemuv5G0njVWrwm/PgB+bTgDWBu6fBWCqMeZ1AHcBiDq3fTqwNfzCQqBPHw3Cy5droy6gvX6WLQMWLXLec/31eutebOHPf9aAu2dPcEpn0yadBtp+ljHO2r1r1miaqG9fTfEczFKN9grDvW9ArzZC2xiIGgPbrdkOxqwr4PdpdPmIxIg34O8GYDs4jgIQmNUG+wE0T1ShUlX37trz59hjNchXV2sj7uGH6+tDh2qaZO9eHcB12GHAf/+3vtarl9NbZ/ZsTam0bQv89rfO57dpo8Ec0EVa3I3K776r4xEAPTF88YUzHUMkGzaEPzFECvjjxmkOlKixsd2a7ZV0ixZ6xWwDfmgvPVtJ85t4A/4HACYGcve9AMwMPH8kgDWJLFgqatFCa/TXXx98SWj/eEa75v586in9Y7NzAT3zDPD++87rF1+stzt26JiAKVM0Z9+/v3b3HD1aR+gOGKAnmqIiPdnYKcOPPVaHhkcaCLZ6tb7PLu3oZmvxofP2fPutpqM++ghYvLju40FUl5dfbriJB/fvB4YN0xRluBkw8/K0wtO8uXPlbPltLVsr3oB/HYD5ANoDuMAYY8eHHgvAF8uD9+ihjbYDBzrdNW3wz8gAnnhCaxm2Nm/l5gaP/H35ZeC88/R+Xh7wi184l6O2NnLiifrPcuSR+viss2pP7XDrreHL+dxz+g/x1FO1a/mRavgbN2o30aFDtTsq5/Whg2GM/l0PHFj3tvv3S9yVjHXrdHbZs87SGn5WVvBgKpvWOeIIpzfelCnBKVe/iSvgG2N2G2P+xxjzM2PMLNfzdxpj7kt88VJXRobWrt95BzjjDOf5ceP0jy/SXDjTpztr79org7omcbI18nPPDf6DPvts7Wd8ySXBI3gPHAAmT9ZazLJlTg2rsjIDTz6pgR0IDvjV1bVr/A88EL1cVD9ff+2PNZVDKxTRPPxwbwwaFF8bkm3fAoD582tPpmgD/pFHOv+P/frFdgJKV/F2y+zr7n4pIqNEZIqI3B5YvtBXMjO1dtE0jpEIZ5yhl6GA03BUURH9PTffrCeIU0/Vx3PnApMmab6/tFRPOk884QTymTP1/qOP6gnCrt87a1YHXHut87nuf8jS0uBRwk2bandQSqzKSk3TTZyY7JIk3v3363xTVl0jxjds0DEqW7YA8+drtHbPKVUXd8BfsqT2YKrmgVZFd8D3+6SE8aZ0ngMwAABEpDOAtwFkQ1M99yS2aOnP1vB37Ii+3ZgxGoztcPGhQ4HLL9fBWYCeMPbv13x9dbWmcfLzgUsv1bTQxIk6Kdy8ecETCrlr9HZxeEAvf08/PXi8AKBtC5dd5iwQD+jlcb9+weMJ3P7yF+2VRGr7dq3dp+NI6jlztLJh2YAvEn77RYv0avezz4BduzSPWdf/gtu6dfrZNl0TWsO3XTX79nUCfatWsX9+Ooo34BcCsE0wPwewwBhzOoDLAFySyIL5gQ347jl54tG1q3M/J0cD/dFH64jDX/1Ka+mjR2tgnzgR+OILJ3fUpYvW8KdM0bYEe3UwcKA2KHftWvvy+osvdPtjjnGuBqZNA1as0BHJobZsAe66S6egiNfChToSOt1SH6Fzs6eT7dv1d26nIbYB31ZUQtlj4a5sxFvD79jRaS8LreHb/fbsyRq+FW/AbwLAzhV5MoAZgfurAeSHfQdFlJOjPX7cvXfiYWv4gM61X1qqI3InTdJAC2jeH3DGAVxwgZ5oRozQgD9tmv7YWvvUqdqjqFMnnV62rMzZh11NCADee09v7YRVc+fWLt8TT+jJbONGPal07gx8+mls3+3993VqCTuSOZJNm5x5zxsD25U2HQP+tm1aEbBXezbgl5WF7x5sFyhx95WPFPDffrv2/8m6dfo/YDtNhNbwJ04Enn5ar0BZw1fxBvzlAMaJyEnQgG8bbjsB2JrIgvmBCPDII/EvqmJ17Kg5+sxMzZ2OGaOfd/nl+hygg8TKy4EPPgD69NmNRx7Rk8Jhh+nl8xdf6HbTA4tN2pGK9gThruUXF2tjdatW2m5w4ICT5w8N+JWVGvAzMjQgT5umn/W730X+PhUVOv6grMxJedi5y8OpqdF2DDu4rTFI5xq+/U72atEG/OpqJ7i7hVvHOVLAv+WW2qlBG/Dt6PTQ6dLz8zWVKaIBPzMzvva2dBRvwL8VwNUAigC8YoxZFnj+bABs4vNYZqbWxHv00D/kl14Crruu9nYtW2rD1VNPff5jILeXwTagzp+v00PYcQOdOumtO4//1Vf6OaNHa672yy81SA8cqCcR+48OaFlKS/XkYx1yiNbwI+X733tPp6G++urYAv6CBbqdHZswb17qB9J0reHv2+dcadkUjbuy8P77tQcK2pOfvUoEwgf8ffv05OGe6ri6WqcG6dbN6WMf6e8K0BSl+4rYr+LtljkPOtK2vTHG1R6PpwGMS2TBKDb/9V+1J2eL9X2h3BO32YC/fr02CF90kV4FFBYCp52mz9vc/I036q27pjZ1ql5qX+Jq2bnuOr20z8vTSeEAvRK47DKdk8imj1591UkxRQv4tvfR2rU69cSwYc7YhkTbskWDhnv66/pI1xq+O1DbE//27U7t+6KLanfzddf6MzNrkJUVPuCvXq0BfsMGp31g40a9wuzWzVm+NNqV8s03czAhUI/pkY0x1dAZMvuJyJEi0twYs9YYs6XON1PCvfqqLrwSrw4dnLmB7DiAxx93XrcB/4EHtH3ABuiePYHhw/X+yy/rVcZ552lPCVvT3r9fF5YZNSp4DqFzz3Uaqt95x7mdMgX44x+Da2i2y2i0gD9jhjMrop2ZdPnyGL58PSxcqDXK2bPr3jYaW8vdvv3g5kPyWk2Ntqncdlv4Fd/cJ7BNm3Sb8nJn2hGg9ohb98C+Ll0q0KFD+IBv244OHHB6ltkumd26aY5+/Xrgmmsil/+QQ5w58v0s3n74mSLyIIAdAL4EsAzADhH5a6rPi0+1nXii3s6fr7n8E1zT39mh58uXA3ffrWmje+/VhWR69NA8/ubNelXQvLnWrmwef9EiTfUMGxY8urhHD93PwIHOP64N0G3aaDe60DxspIC/Z4+mmK66StNZr72mzzfUP/WKFXr71VcH9znuBbV379ZAecop4VdmAvQkfNZZB7fPeFRU1J42e9EiDayjRmkFINz4DHfA37jR6V7pnv3V3Z0XCK7h9+hRjuzs8AHfveSnneHVHfABrVhE6v5Jjnhr+H8FcCmAawD0BnA4NJVzGYD7E1s0amjXXquNqIWF4ReAeeABHUwzdKgG+9//Xq8KMjKc6R5sjf2kk7QmtnOnU9MfOlSHuzdvrlcCBQXaRmC7fJaWOnMBbd+uAb9jR2dx6V69NFVTXa2plOefd8r2+edaQz7pJO1nbWuP0QL+/v31OkwAnEAfS8DfsqX2SktVVVpedx572zbtg/7hh3rM5szR7rLu8RHvvacN7om+Gvjf/w1emMcaMEDHTlg7d2qbTWamM9FfuHSUfU5Ea/g2cLsD/ubNwYP93DX8goK9yM4O3w/fHfDtidG28TAvH594A/4YAFcZY14wxqwO/EwG8CsAv0h46ahBDRmiC7VEcsstegk/d25wLh7Qy2jACfj20n3dOt2+Xz8d2i6igb5LF2daiI4dNSB27apXF4D+A5eW6iIyffvqcyNGaGqguFivPq66ytm/nQ9l4MDgeYvc3e62bAF++lOtcb71lqauQqePiJWt4RcXO3nkSHr3dtZOAPRE1qqVnjBDA74NgGvWaGrrhx80dWKVlOgxSPS8Rn/6k55k3J+7b592g125Un9v99yj33vbNh21bXtDhQv4NsD37Kk5d/vY3S4EOGtFAME1/Kys/RFr+EuW6NxOgBPw163Tbph+72YZr3gDfjton/tQqwFkHXRpqNGwQdYGfDsIbPVqzd/b6SMAnULCXhEA2j5QUeEMqho8WP+Bt27VgP/SSzpw7Kab9HV3l88NGzRoL1qk+8zLc04+QPC4gU8/1Zz7/Pna1lFerlcCS5YAf/1r5O9mR8NaNTUa6LOz9flI66RaNoja1Mj48XqSePppfc2mHkID/qxZwd+3qsrZl91u5crgNJcxehVW37aLjz927tuToU2b/PGPTgNs165OP/doNfwrrtDj+9Zb+ti9vkOTJjrX1LJA377du3XW1+7dgREjtoQN+Dt36knnrLN00kJ3Soe1+/jFG/C/hK56Feo3gdfIJ2y+f8AAvbUB/623NLDahl1AG3dffNF57G7IXbVKJ4ErLdVafm6uBoCJE/Vk0rmzLvBuXXCBbr98uZOGctfw3TVo2y3wu+80LQLo+8aN01lGw9Waly3TwGZPNoDWKisqdL9AcIohGhusbY29pkbLZxvE3QF/xgw9mTVr5qTEVq1yRjTbYNynT3Ctec8evXKYNEnL1a+fkybbtk2DdrilLG2wdJ9MbVncK6HZgF9QoO06TZtGDviHHKJpn/x84KGH9PnsbC37p58Cb76pFQI7/cKuXdpbbM0aICdnH7Kz9STgTr0tWKAntSFD9G+BAf/gxBvwbwHwSxFZKSIviMhkESmB5vWjDKmhdHPCCfpPd9xx+jgvT//h//lPfezuKnroocEzgtqA16KFpj7sP+6GDc7i04DWhIcOddI+gAaApUu1lmsDX6SAby//33rLyQ0/95zTtTI0cL/xhtO1zy4tCTgNhPY7/fADUFLSBmefHX2U74QJ2pV12zYNVrt26QnHTpq3fr0TZO3t+PFai//+++BRxuFmnqysdALgmjW62M6KFXoCMEZ/F/fcE34+ensFM2WK0y5h9+FuQJ43T/P37dvr7yMnxwn469c7jeXbtulrLVsCJ5/svD87W0/izZoBP/uZvr5li5Zv9+7gNhfbwO8e+/HJJ9pmdPzx2h6wcqWzEhwDfvzq0w+/N4CpAFoDaBu4fyrC1/wpjbnn8snI0Dx9ebnm4CMtCg84Nfy+ffV97s+xU9paY8cGP66p0UBXUaH5YkCD6THH6D5373ZqxTZw2bTFkUdqkLezKLqnigA0UHbtqnn/ykrneRvwBw/W8paUANdcMxDvvht8MnIfC0BHPdseNnYK7T179Aqmc2ftteJuUxgwQHtBAVobdi9cH9r2cOCAfudxgdEvs2drA/AJJ+gqbJ995izg7R4QB2jA3L5dZ1+trnbaRmzAd8/t9MYb2oXXfqf27Z1JyS65ROddWr/eCfhA8PKBofPo5OXpd6ms1O/gft3+Pt0pKzs5X+vW2rng66/1iq283L/LFB6M+vTD32CMucMYc74x5jxjzB8AlAM4P/HFo8akSxe9dadzwrEB3+bejz7aec1dwwe0LeC773QulVC2YVRE88a33abBzNa6bcA3Rrt9jhqlj2+9Va9Giot1da/XX9ca7+rVOqbg1FM1qNl8su0R0qOHpivcvYUWL9bUjfuKoFmz2mU97TTnfrt2GqyXLAmuuZ92mjZ+9++v4wpKSpyG782bg3vqLF6std2PPtLHtgH05pv1dsWKyAHfzq46cqSeYBYu1KuPSPPXu7vWumv49nM//jg44LtXgwvtZpuXp/ux5a0r4K9a5bQTHXGEnijsnDqFheHLS5HFHfCJIrE1dXeDbTjt2jm1Q0C7Ydp0Sbhg2aVLcMC0bICwbHdOm9ZxpyaOOEI/46ijtLfJ4Yc7/fivukrv19Q4C9QDTkrFti20bKnpKG14NejcWU8WPXpofn/jRg2k7qsDq39/Xf2pc2dNGw0YoAF97VrtadSkiTNK+LzzNJXxn//oVVBOjtaKy8udzwt3AhRxTrYbN0YO+PZElp2tPaFqavTEERrwberN3ebiDvg2WNspLcIF/FD5+fpdbPuJO6XTubOmj9xz8Kxd6/yebYC3I6wZ8OPHgE8J07OnBp26Ar6INuS61wCePFn71Lvzv25Nm9aeDdGOFLZs8Ni5U2vD7oDfp48OcFq6VANdYaHm1+2UDpMn63aFhU6N8tFHNQ309NPOycwGwby8KgwZEtzN8OOPnZrrcccBF17ofN9OnTRf/v33OuLYTjG9e7eeiLZvd1ZistNefP21ltumQdx91MMF/J499btlZel3t4HTBvyaGp3byLZ9HHqo7qtZM21UtycIy/4u3FddOTkahKdOda58QgO+e3RtqLw8rcHbhXjcNfzMTM3Lz5ypv4/167WrqC2vDfCzZum+QtN/VDcGfEqY8eN1EFHogtGx6NFDA0e0hjj7uS1aaF45dG5zW8OfNUtrq5WVzuyfobXOMWP0NjdX0w6PPaa3vXtr4GzSRHsH2Tx/aMDv0qXix8A2YYKWZeZMp7/+uHF6wgA0JRI6S6Od/wXQIOgOfLbnE6ABPz9fv4+7QTrcADDb9bWgQHPftmHWBvynngJeeMHpBZOdre0Zv/iFBnA75bV10kl66+6Vk5OjKaELL9S0V2am01ffBnw7AV84tp3l3//WOZRC54Gyo7GvuAL4+9/1ORvwc3Odk0/nzhxZWx8xBXwReSfaD4BHGric1Ajk5ESuoSdCfr4G4lGjtIYcygb8224DfvlLvX/88Xob2sB37rlaq1661Olp1KGDBqumTXVKg3/8w1lXwOai3QH/ppv0SuXaazVVNGmSc3XTrp0Gt6wsp23DrXNnLaf7M6327Z332IC/aVPtUag2wNq2DDtgraBAG20BHZhkA77trmnZnlPPPhs8jxKgtf4hQ/S+PT6Ac4ytESP0aurAgeArsOnTnfYFN3dlYMKE2o269qTWurU2egPB3VDt+Im62okovMwYt6trbr9tANYcZFmIoureXVMVL70UfqoBdzD64AN9fOed2svGBi83G/CeeUa3c88lZHvLFBfrazbouAN+VpYzAvnKK4Nn0mzbVmug559fO6Bb992n+Xp3jd469lhN//TurVc9b77p1LTPPVdPTFdeqT2KBg/WK5ZLL9XX3Y2sI0c6s0QuXaptCXZOG7tClIh+3zfe0MD90Ud6wiosdNZOsEInThs61FntzB3wba+kUO7eW+FWwrrzTr0yuv12LTsQfMK84gpNzYWeeCg2MQV8Y8wVDV0Qoro88IA29tmJ3UK5+/oDwA03aDBdsiT65x59tNMQGKqwUN9vrxBs8OvZszxou7FjNQhddJE+tu0J0WYyFQmuPbuNHKmDlXr00BTT/v3OaNqHHtLnqqv1WPTsqf3tLRvwu3TRK48ZM7Tn0jffAH/4gxPw3cerSRNNs8yc6QR8wGnPsK67TtNAn3yi4xvc7TWhSwyGY2v4kdJ+Z56pP4C2aWzb5kzJYUU6gVLdYq3hHzQR6QLgRQAdANQAeMYYw1QQxSw3N3pD3aGHaorl5JM1x21nAz1Y7vTRkCHae6aiovYwXXf7Q6R1XGM1frwuBJOZ6fRSsTV1G6ibNNGyhAZAG/B79tT71dV6QjNGrxyscCdOW+5I5c/O1q6fVVWau3dPjhbaqB6OXXA89EQSzqef1j1vEcXHs4AP4ACAm4wxn4tIGwCLReRDY8xBTjhL5LAplmgDvw6GiLYL2OkP3NwB/2Cnac7IcBo/bcC3I2bdwdg9R5FlJxTr2FF7ALVr57RpuMc8hGv0tJ9dV/mbNXNOHm3bakNsLAF/wACdfsHOvBlN06ZckjDRPAv4xpiNADYG7u8RkWLoWrgM+JQW6spP11fXrlqb//57DcShKY5Q9sQzerSmoGbO1JRPQYG2gyxbFnmdgVgDvluXLlrbjyXgZ2Y6vW/Ie17W8H8kIt0BDABQa8E4ERkLYCwA5ObmoihcVYrqpaysjMczQSIfy+EAgAULihLabTA//3hs2NACzZvvRVHRf6Ju27w58NxzrdC1a/mPVyJ2BK5dhrJt2/BXKXv2ZAL4CSorN6KoqKT2BmG0anUUgBwsWzYXTZrUb+J+/m16Q4zH66yJSGsAcwHca4z5Z7RtW7ZsaQa7OyzTQdm5cyey2L0hISIdy7lziwAAw4YNT+j+vvzyIezcOQgtW67FccddntDPdjOmCebNm41Ond5Ar14TYnrPN9/8BqWlI3DiiefUe7/820ycuXPnLjbGDAr3mqc1/MAyiG8CeKmuYE/UGOXnz8KOHWH/1w5Kt24vIiNjH9q1W1r3xgdBpBrt289DVtYXMb+nW7d/ID///YYrFCWMZzV8EREALwDYboy5IZb39OnTx5SUxHZZSXUrKirCcI5YSQgey8Ti8UwcEYlYw/dyaoUh0LVvR4rIF4Gf0z3cPxGRr3nZS+djAJz9gogoSTh5GhGRTzDgExH5BAM+EZFPMOATEfkEAz4RkU8w4BMR+QQDPhGRTzDgExH5BAM+EZFPMOATEfkEAz4RkU8w4BMR+QQDPhGRTzDgExH5BAM+EZFPMOATEfkEAz4RkU8w4BMR+QQDPhGRTzDgExH5BAM+EZFPMOATEfkEAz4RkU8w4BMR+QQDPhGRTzDgExH5BAM+EZFPMOATEfkEAz4RkU8w4BMR+QQDPhGRTzDgExH5BAM+EZFPMOATEfkEAz4RkU94FvBF5HkR2SIiy73aJxERObys4U8GMNrD/RERkYtnAd8YMw/Adq/2R0REwTKTXYBQIjIWwFgAyM3NRVFRUXILlEbKysp4PBOExzKxeDy9IcYY73Ym0h3AdGNMv1i279OnjykpKWnYQvlIUVERhg8fnuxipAUey8Ti8UwcEVlsjBkU7jX20iEi8gkGfCIin/CyW+YrAD4F0EdEfhCRq7zaNxERedhoa4y5xKt9ERFRbUzpEBH5BAM+EZFPMOATEfkEAz4RkU8w4BMR+QQDPhGRTzDgExH5BAM+EZFPMOATEfkEAz4RkU8w4BMR+QQDPhGRTzDgExH5BAM+EZFPMOATEfkEAz4RkU8w4BMR+QQDPhGRTzDgExH5BAM+EZFPMOATEfkEAz4RkU8w4BMR+QQDPhGRTzDgExH5BAM+EZFPMOATEfkEAz4RkU8w4BMR+QQDPhGRTzDgExH5BAM+EZFPMOATEfkEAz4RkU8w4BMR+YSnAV9ERotIiYisEpHbvNw3EZHfeRbwRaQJgMcBnAagL4BLRKSvV/snIvI7L2v4gwGsMsZ8a4zZB+BVAD/zcP9ERL6W6eG+OgH43vX4BwDHh24kImMBjA08rBKR5Q1crnYAdjXwe+vaLtrrkV4LfT7cdqHPtQewNWpJD15jPJ71ec6LYxmpHIl+L49nYt+XjOPpftwt4p6NMZ78APg5gGddjy8D8Fgd71nkQbmeaej31rVdtNcjvRb6fLjtwmzD4xnDcYrlOS+OJY9n6hzPeN6XjOMZa/m8TOn8AKCL63FnABs83H8k73rw3rq2i/Z6pNdCnw+33cF8t/pqjMfzYJ5raDyeiVXffcbzvmQcz5jKJ4GzQ4MTkUwAKwGcDGA9gM8AjDHGrIjynkXGmEGeFNAHeDwTh8cysXg8veFZDt8Yc0BExgN4H0ATAM9HC/YBzzR8yXyFxzNxeCwTi8fTA57V8ImIKLk40paIyCcY8ImIfIIBn4jIJxptwBeRViKyWETOTHZZGjsRKRSRp0TkDREZl+zyNHYico6ITBSRt0XklGSXp7ETkZ4i8pyIvJHssjR2ngd8EXleRLaEjqCtx8RqtwJ4vWFK2Xgk4ngaY4qNMdcAuBCAr7vGJeh4vmWMuRrA5QAuasDiprwEHc9vjTFXNWxJ/cHzXjoiMhRAGYAXjTH9As81gfbRHwUdoPUZgEug3TfvD/mIKwEcDR2K3RzAVmPMdG9Kn3oScTyNMVtE5GwAtwGYYIx52avyp5pEHc/A+/4G4CVjzOceFT/lJPh4vmGMucCrsqcjL+fSAQAYY+aJSPeQp3+cWA0ARORVAD8zxtwPoFbKRkRGAGgFnXWzUkRmGGNqGrbkqSkRxzPwOe8AeEdE3gPg24CfoL9PAfAXADP9HOyBxP19UmJ4HvAjiGliNcsYcwcAiMjl0Bq+L4N9FHEdTxEZDuA8AM0AzGjIgjVScR1PAP8D4KcA2olIL2PMUw1ZuEYo3r/PHAD3AhggIrcHTgxUD6kS8CXMc3XmmowxkxNflLQQ1/E0xhQBKGqowqSBeI/nowAebbjiNHrxHs9tAK5puOL4R6r00knVidUaKx7PxOLxTCwezyRJlYD/GYDDRaSHiBwC4GIA7yS5TI0Zj2di8XgmFo9nkiSjW+YrAD4F0EdEfhCRq4wxBwDYidWKAbwew8RqBB7PROPxTCwez9TCydOIiHwiVVI6RETUwBjwiYh8ggGfiMgnGPCJiHyCAZ+IyCcY8ImIfIIBnygCEbkrdFpfosaM/fApqURkMoD2xpiUmyVRRFoDaBaYyyUliYgB8HNjDBcHoTqxhk++ExjOXydjTFkygr2IZATmjCdKKAZ8Smki0ldE3hORPYGVk14RkQ6u148TkQ9EZKuI7BaRj0Xkv0I+w4jIdSLyTxEpB3CfTdeIyMUisjrw+W+JSHvX+4JSOiIyWUSmi8hvRGS9iOwQkUki0tK1TSsReVFEykRks4jcHnjP5Cjf8fLA9qcH9rcPQGFd301E1gbuTg18x7Wu184SXQJ0r4isEZF7Yz3RUfpiwKeUJSIFAOYBWA5dNOOnAFpDF2qxf7ttAPwDwEmBbb4AMMMduAPuhM71fxSAxwPPdYcuQXgugFMADIDOux7NSQD6Bcpi3/sb1+t/AzAs8PxIAP0D76lLcwB/APBr6MI+62L4bscFbq8GUGAfi8ipAF4CMAHAkdBV4i4AcF8M5aB0ZozhD3+S9gNgMoDpEV77M4DZIc8dCp07fXCE9wiAjQAudT1nADwWst1dAPYCaOd67g7oSkzubZaHlPV7AJmu5yYC+Ffgfmto7fxi1+utAOwAMDnKMbg8UMaBdRyrSN/tgpDt5gH4Y8hz50CXGpRk/875k7wf1vAplQ0EMDSQ7igTkTI4KyUdBgAikiciT4vIShHZBWAPgDwAXUM+a1GYz19njNnlerwh8N5ovjI622O49xwGoCmAhfZFY0w59AqlLgegNfgfxfHdQg0EcEfIcXsZevLpEP2tlM5SZcUronAyALwH4HdhXtscuH0BQD6A3wJYC6AKwGwAofnq8jCfsT/ksUHdac5o7xHXc/GqMsZUhzwX63cLlQHgbgBTw7xWWo+yUZpgwKdU9jmAC6E18dBAa/0EwPXGmPcAQETyofnsZFgFPSEMBrAmUJ6W0Jz/6np8XizfbT+A0B49nwM4whizqh77pDTGgE+poK2IHBPy3E5o4+rVAF4TkQegtdOe0JPATcaYPQBWArhURBZAUxZ/hebRPWeMKROR5wE8ICJbofn2P0Br3PWp9cfy3dYCOFlE5kKvEnZA2z6mi8g6AK9D00X9oO0et9SjHJQmmMOnVHASgCUhPw8ZYzYAGAKgBsAsACugJ4GqwA+gPVBaA1gM4FUAz0ODYLL8DsBH0CX75gBYCm0/2FuPz4rlu90EYAS0bWMJABhj3gdwRuD5hYGf2wB8V48yUBrhSFuiBiQizaBdLB80xvwt2eUhf2NKhyiBRGQAgEJorboNgFsDt68ls1xEAAM+UUO4EUAfOF0thxpjfkhqiYjAlA4RkW+w0ZaIyCcY8ImIfIIBn4jIJxjwiYh8ggGfiMgnGPCJiHzi/wEx1+eWoFC05AAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"batch_size = 128\n",
"rates, losses = find_learning_rate(model, X_train_scaled, y_train, epochs=1,\n",
" batch_size=batch_size)\n",
"plot_lr_vs_loss(rates, losses)"
]
},
{
"cell_type": "code",
"execution_count": 134,
"metadata": {},
"outputs": [],
"source": [
"tf.random.set_seed(42)\n",
"\n",
"model = tf.keras.Sequential()\n",
"model.add(tf.keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
"for _ in range(20):\n",
" model.add(tf.keras.layers.Dense(100,\n",
" kernel_initializer=\"lecun_normal\",\n",
" activation=\"selu\"))\n",
"\n",
"model.add(tf.keras.layers.AlphaDropout(rate=0.1))\n",
"model.add(tf.keras.layers.Dense(10, activation=\"softmax\"))\n",
"\n",
"optimizer = tf.keras.optimizers.SGD(learning_rate=2e-2)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\",\n",
" optimizer=optimizer,\n",
" metrics=[\"accuracy\"])"
]
},
{
"cell_type": "code",
"execution_count": 135,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/15\n",
"352/352 [==============================] - 3s 9ms/step - loss: 2.0559 - accuracy: 0.2839 - val_loss: 1.7917 - val_accuracy: 0.3768\n",
"Epoch 2/15\n",
"352/352 [==============================] - 3s 8ms/step - loss: 1.7596 - accuracy: 0.3797 - val_loss: 1.6566 - val_accuracy: 0.4258\n",
"Epoch 3/15\n",
"352/352 [==============================] - 3s 8ms/step - loss: 1.6199 - accuracy: 0.4247 - val_loss: 1.6395 - val_accuracy: 0.4260\n",
"Epoch 4/15\n",
"352/352 [==============================] - 3s 9ms/step - loss: 1.5451 - accuracy: 0.4524 - val_loss: 1.6202 - val_accuracy: 0.4408\n",
"Epoch 5/15\n",
"352/352 [==============================] - 3s 8ms/step - loss: 1.4952 - accuracy: 0.4691 - val_loss: 1.5981 - val_accuracy: 0.4488\n",
"Epoch 6/15\n",
"352/352 [==============================] - 3s 9ms/step - loss: 1.4541 - accuracy: 0.4842 - val_loss: 1.5720 - val_accuracy: 0.4490\n",
"Epoch 7/15\n",
"352/352 [==============================] - 3s 9ms/step - loss: 1.4171 - accuracy: 0.4967 - val_loss: 1.6035 - val_accuracy: 0.4470\n",
"Epoch 8/15\n",
"352/352 [==============================] - 3s 9ms/step - loss: 1.3497 - accuracy: 0.5194 - val_loss: 1.4918 - val_accuracy: 0.4864\n",
"Epoch 9/15\n",
"352/352 [==============================] - 3s 9ms/step - loss: 1.2788 - accuracy: 0.5459 - val_loss: 1.5597 - val_accuracy: 0.4672\n",
"Epoch 10/15\n",
"352/352 [==============================] - 3s 9ms/step - loss: 1.2070 - accuracy: 0.5707 - val_loss: 1.5845 - val_accuracy: 0.4864\n",
"Epoch 11/15\n",
"352/352 [==============================] - 3s 10ms/step - loss: 1.1433 - accuracy: 0.5926 - val_loss: 1.5293 - val_accuracy: 0.4998\n",
"Epoch 12/15\n",
"352/352 [==============================] - 3s 9ms/step - loss: 1.0745 - accuracy: 0.6182 - val_loss: 1.5118 - val_accuracy: 0.5072\n",
"Epoch 13/15\n",
"352/352 [==============================] - 3s 10ms/step - loss: 1.0030 - accuracy: 0.6413 - val_loss: 1.5388 - val_accuracy: 0.5204\n",
"Epoch 14/15\n",
"352/352 [==============================] - 3s 10ms/step - loss: 0.9388 - accuracy: 0.6654 - val_loss: 1.5547 - val_accuracy: 0.5210\n",
"Epoch 15/15\n",
"352/352 [==============================] - 3s 9ms/step - loss: 0.8989 - accuracy: 0.6805 - val_loss: 1.5835 - val_accuracy: 0.5242\n"
]
}
],
"source": [
"n_epochs = 15\n",
"n_iterations = math.ceil(len(X_train_scaled) / batch_size) * n_epochs\n",
"onecycle = OneCycleScheduler(n_iterations, max_lr=0.05)\n",
"history = model.fit(X_train_scaled, y_train, epochs=n_epochs, batch_size=batch_size,\n",
" validation_data=(X_valid_scaled, y_valid),\n",
" callbacks=[onecycle])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One cycle allowed us to train the model in just 15 epochs, each taking only 2 seconds (thanks to the larger batch size). This is several times faster than the fastest model we trained so far. Moreover, we improved the model's performance (from 50.7% to 52.0%)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
},
"nav_menu": {
"height": "360px",
"width": "416px"
},
"toc": {
"navigate_menu": true,
"number_sections": true,
"sideBar": true,
"threshold": 6,
"toc_cell": false,
"toc_section_display": "block",
"toc_window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 4
}