{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c869bafb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:45.145563Z",
     "start_time": "2023-03-14T00:56:42.879564Z"
    }
   },
   "source": [
    "# Exercise 1 for the class EE-618 Theory and Methods of Reinforcement Learning taught at EPFL in Spring 2023 by Prof. Volkan Cevher"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8921d51d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from typing import List\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "os.environ['KMP_DUPLICATE_LIB_OK'] = 'True'\n",
    "import sys\n",
    "\n",
    "sys.path.insert(0, \"src/\")\n",
    "from environment import GridWorldEnvironment\n",
    "from MDPsolver import MDPsolver\n",
    "from plot import plot_log_lines, plot_lines\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "284733c0",
   "metadata": {},
   "source": [
    "# Dynamic Programming exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24373bdd",
   "metadata": {},
   "source": [
    "We will make use of the operators introduced in the slides of lectures 1 and 2. \n",
    "$(T V)(s)$, $\\mathcal{G}$ and $T^{\\pi}$\n",
    "\n",
    "Before diving into the implementation of Value Iteration or Policy Iteration, we instantiate an environment using the library in the foilder `src`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56a7da04",
   "metadata": {},
   "source": [
    "The gridworld environment is instantiated via the class `GridWorldEnvironment`. It takes 4 input values:\n",
    "\n",
    "- `reward_mode` : integer between 0 and 3 for different reward profiles\n",
    "- `size`: Gridworld size\n",
    "- `prop`: probability assigned to the event that the agent do not follow the chosen action but another one selected uniformely at random.\n",
    "- `gamma`: the discount factor of the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b61c70d2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:45.302563Z",
     "start_time": "2023-03-14T00:56:45.148567Z"
    }
   },
   "outputs": [],
   "source": [
    "reward_mode = 2 # put an image to show which is the reward\n",
    "size = 10 \n",
    "prop = 0\n",
    "gamma=0.99 # rename discount_factor\n",
    "gridworld = GridWorldEnvironment(reward_mode, size, prop=0, gamma=gamma)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a204d2f",
   "metadata": {},
   "source": [
    "Below, we illustrate how to access the gridworld environment, i.e. how to access the reward, the transition matrix and the discount factor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6bea31a5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:45.429564Z",
     "start_time": "2023-03-14T00:56:45.303598Z"
    }
   },
   "outputs": [],
   "source": [
    "size = 3\n",
    "gridworld_example = GridWorldEnvironment(reward_mode, size, prop=0, gamma=gamma)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "875607fb",
   "metadata": {},
   "source": [
    "***Interface with a Gridworld instance***\n",
    "- `print(gridworld.n_states)` #return the number of states\n",
    "- `print(gridworld.n_actions)` #return the number of actions\n",
    "- `print(gridworld_example.r)` #return a matrix where each element indicates the reward corresponding to each (state, action) pair.\n",
    "- `print(gridworld.gamma)` # return the discount factor\n",
    "- `print(gridworld.sparseT[1])` #Input: action, Return: a vector indicates the transition probability p(s'|s a), where s is current state of the environment\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88945c56",
   "metadata": {},
   "source": [
    "<img src=\"src/vis_gridworld.png\" alt=\"fishy\" class=\"bg-primary\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99553e09",
   "metadata": {},
   "source": [
    "Then we set up the solver object that will implement the routine needed to evaluate the value of the produced policies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4a8aea9b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:45.570566Z",
     "start_time": "2023-03-14T00:56:45.431568Z"
    }
   },
   "outputs": [],
   "source": [
    "solver = MDPsolver(gridworld)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d5bd704",
   "metadata": {},
   "source": [
    "We will use this solver to compute the optimal value function to measure the suboptimality of the policies produced by value iteration or policy iteration.\n",
    "\n",
    "To access the optimal value function use `solver.v`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "48b77426",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:45.681600Z",
     "start_time": "2023-03-14T00:56:45.571564Z"
    }
   },
   "outputs": [],
   "source": [
    "solver.value_iteration()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b20dbd17",
   "metadata": {},
   "source": [
    "## Ex 1: Value Iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7d9a1d4",
   "metadata": {},
   "source": [
    "### 1.1 Implement value iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "03b8030d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:45.809598Z",
     "start_time": "2023-03-14T00:56:45.682600Z"
    }
   },
   "outputs": [],
   "source": [
    "def value_iteration(env, tol=1e-10):\n",
    "    \"\"\"Inplementation of value iteration, note that the implementation is based on Q-value iteration mentioned in the lecture.\n",
    "    Args:\n",
    "        env: environment\n",
    "        tol: a scalar to dermerminate whether the algorithm convergences\n",
    "    Returns:\n",
    "        policies: policy  at each iteration (list)\n",
    "        vs: value functions at each iteration (list)\n",
    "    \"\"\"\n",
    "    policies = []\n",
    "    vs = []\n",
    "    v = np.zeros(env.n_states) # initialize value function\n",
    "    q = np.zeros((env.n_states, env.n_actions)) #initialize Q-value\n",
    "    \n",
    "    while True:\n",
    "        v_old = np.copy(v) # save a copy of value function for the convergence criterion at the step\n",
    "        for a in range(env.n_actions):\n",
    "            q[:, a] = ??? # calculate Q-value\n",
    "        v = ??? # update value function\n",
    "        policies.append(???) # obtain policy\n",
    "        vs.append(v)\n",
    "        if np.linalg.norm(v - v_old) < tol: # convergence criterion\n",
    "            break\n",
    "    return policies, vs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a39f557a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:45.936564Z",
     "start_time": "2023-03-14T00:56:45.810566Z"
    }
   },
   "outputs": [],
   "source": [
    "policies, vs = value_iteration(gridworld)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88fd23d2",
   "metadata": {},
   "source": [
    "### 1.2: Evaluate the extracted policies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db59f231",
   "metadata": {},
   "source": [
    "To evaluate a policy (we can simply compute the occupancy measure and consider the inner product with the reward move it to LP notebook). The occupancy measure can be compute via the method `solver.mu_policy` that takes two input:\n",
    "- The policy of which we need to compute the occupancy measure.\n",
    "- `stochastic`: A flag to be set to True if you pass a stochastic policy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4407d9b2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:46.078567Z",
     "start_time": "2023-03-14T00:56:45.937566Z"
    }
   },
   "outputs": [],
   "source": [
    "def evaluate_policy_sequence(policies, env, tol=1e-10):\n",
    "    \"\"\"Inplementation of policy evaluation through iteratively applying policy value iteration \n",
    "    Args:\n",
    "        policies: a list of policies obtained by section 1.1\n",
    "        env: environment\n",
    "        tol: a scalar to dermerminate whether the policy evaluation convergences\n",
    "    Returns:\n",
    "        values: a list of value function for each policy\n",
    "    \"\"\"\n",
    "    values = []\n",
    "    for pi in policies:\n",
    "        v = np.zeros(env.n_states) # initialize value function\n",
    "        q = np.zeros((env.n_states, env.n_actions)) #initialize Q-value\n",
    "        while True:\n",
    "            v_old = np.copy(v) # save a copy of value function for the convergence criterion at the step\n",
    "            for a in range(env.n_actions):\n",
    "                q[:, a] = ??? #calculate Q-value\n",
    "            for s in range(env.n_states):\n",
    "                action_taken = pi[s] # obtain the action determined by the policy\n",
    "                v[s] = q[s,???] #calculate value function by $v(s) = max_a Q(s,a)$\n",
    "            if np.linalg.norm(v - v_old) < tol: # convergence criterion\n",
    "                break\n",
    "        values.append(v)\n",
    "    return values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "87b7f040",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:48.241600Z",
     "start_time": "2023-03-14T00:56:46.083566Z"
    }
   },
   "outputs": [],
   "source": [
    "values = evaluate_policy_sequence(policies, gridworld)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a331466c",
   "metadata": {},
   "source": [
    "### 1.3: Plot the results "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fcf9844",
   "metadata": {},
   "source": [
    "Plot using the method ```plot_lines(list_to_plot, list_name, axis_label, folder, title, x_axis = None, show = False) ```\n",
    "\n",
    "Compute both the iterates $V_t$ produced by value iteration and the value function achieved by the extracted policies $V^{\\pi_t}$ with $\\pi_t = \\mathcal{G}(V_t)$. What do you observe ?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56bd6079",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "03a72635",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:48.384121Z",
     "start_time": "2023-03-14T00:56:48.244565Z"
    }
   },
   "outputs": [],
   "source": [
    "def compute_subopt(values, v_star):\n",
    "    \"\"\"\n",
    "    Args:\n",
    "        values: a list of value function.\n",
    "        v_star: the optimal value function obtained by MDP.solver\n",
    "    Returns:\n",
    "        subopts: an array indicates the suboptimality.\n",
    "    \"\"\"\n",
    "    subopts = []\n",
    "    for v in values:\n",
    "        i = np.argmax(np.abs(v - v_star))\n",
    "        subopts.append(-v[i] + v_star[i])\n",
    "    return np.array(subopts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "714046c3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:48.969703Z",
     "start_time": "2023-03-14T00:56:48.386122Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgEAAAFgCAYAAAAipqoDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA9/klEQVR4nO3deVhUZf8/8PcwzAwwMywmaipouWtuSKaGuVeibSZq2e6TWallrvVU6uNWFplp+tU0t8wEcXnsV1am9USSW6JGarkvmWKAs8EMM3N+f8AcGIFBYOCcGd6v6/KSOedw+NwMcH/OvSoEQRBAREREtU6A1AEQERGRNJgEEBER1VJMAoiIiGopJgFERES1FJMAIiKiWopJABERUS0VKHUANS0z0+jV+0VEhCA72+LVe0qFZZEffykHwLLIlb+UxV/KAXi/LJGR+jLPsSWgigIDlVKH4DUsi/z4SzkAlkWu/KUs/lIOoGbLwiSAiIiolmISQEREVEsxCSAiIqqlmAQQERHVUkwCiIiIaikmAURERLUUkwAiIqJaikkAERFRLeXzKwY6nU7MmDEDJ06cgFqtxuzZs9GkSROpwyIiIpI9n28J2LlzJ2w2GzZu3IiJEyfinXfekTokIiIin+DzLQEHDx5Ez549AQCdOnXCb7/9JnFERETkjwRBQK7VjhyTDdfNNlw3WWEw22B3Cl79OjFtG6BBqMar9yyLzycBJpMJOp1OfK1UKmG32xEYWHrRIiJCvL4us6fNGXwNyyI//lIOgGWRK38pS2XL4XA4kWOyIttgRbYxD1kGK3KMecgy5CHbaEW2IQ9ZRityDHmw2Z1ejrqkfcev4uPJfav96wB+kATodDqYzWbxtdPpLDMBAOD1XaYiI/Ve35lQKiyL/PhLOQCWRa78pSyllcOa78B1k1V8cs8pfHLPMVlxvdjTvNGSD0/P8soABUK1ajSsq0WYVo0wnQZhWjXCdWqEajVQBXq3Z719q3pefU88JUc+nwTExMRg9+7diI+PR3p6Olq2bCl1SEREVE0EQYA5z15QuRdW4tfNNtgcwOVMI66bbOLxPJvD4700aiXCtWo0uMVVuasLK3dN4ccF/+uCVQhQKGqohEBkREiNJWY+nwQMGDAAP//8M0aMGAFBEDB37lypQyIiogpyOgUYLLaCSrywYnc9sbteu57e7Q7PTfL6EBXqhgUjTKdGuOvJXVdYuRer7IPUPl8FVpnPfwcCAgLwn//8R+owiIioFHaHs/DpvLAJXmyeL/y/8JzBbIPgoU3e1SQfVU+LMK0G4bqiyj1Mq0bTxhEQ8u0I1aoRqPT5iW81xueTACIiqnm2fEdRc7zJhmxTsUreXFTZm3LzPd5HHRiAMJ0azRqFiU/t4bqiJvnwwiZ5bTlN8v4ytqGmMQkgIiKR1eZAjtmKHKOrSd7VLG8t9rENFqvd432CNUqEaTVoHKl162N3PcGHF74O1iihqMH+dnLHJICIqBbIsxXObzdZkW2yIsdY0CSfm+/E1X/MYgVf3mA6bVAgIvQaNL1V7/a0Hq4vGjEfptNAo/LuVGyqHkwCiIh82I2Vu2sgXdGxmxsprwsuGExXUIkXNMe7BtKF6zWFTfVqqLy8zgpJi0kAEZEMFW+WzxEr9qJK/mYrd32ICpHhwUVP7Tf0uTeLvgV2q42D6WopJgFERDXINaCuoHIvemLPMblX9rlWz5V7qIfK3fXxzYyUj4wIRmam5/598l9MAoiIvMBeuPRsjsm9gs8pXsEbreUOqNMFq3BLaDDC9SUr9jCdGhE6DafBkdcwCSAi8sDhdMJgznerzG1OAX9dMSLHZEN2YYVf3lQ414C62woH1IXrb+hzLxwt7+0laIk8YRJARLWSIAgw5ea7VeQ3Pslnm8pfxCZIrUS4rnAqnF4jPrVH6N0H1ak5Wp5kiEkAEfkd14j5HGPhdDiTtbCiL95Ub4XdUXbtHqgMQIRejeaNwor62Qub6Js2DofC4USYVo1gDf+Mku/iTy8R+QxxCVqxUi+a8170JO95UF2AQoEwnRpR9XRis3yEq7+92JO8NiiwzEVsuDod+QsmAUQkuVKb5l1P8eLTvA1Gs83jlq/FB9VFiE/vBZW86yk+NESNgACuUEcEMAkgompmy3cg22TF3wYrzl7MFp/as4tV8jkmzzvDqVUBiNBpcGud8IKndX1Rv3t44Yh5LmRDVHFMAoioUpyCAKMlv+BJvbBCL+0p3pxX9pQ4hQIIK9wZzu2p3VXB6zWI0BX0u3N9eSLvYxJARCVY8x1ulbvYB3/D07vDWXbjfLCmYNR8kwZ6ROg0aFhfD41SUewJXoNQrQrKAE6JI5IKkwCiWkQQBBhzb3h6N7j3vWcbPC9o4xpY56rcw/UFlXrxj8N1agSp3f+8cDAdkfwwCSDyE3aHs6giNxb9yzG5f+xpWlywRokIfVDBgjalVu4cWEfkT5gEEPmAXKvdrWLPNuYh22RDtiFPfIo3WMpesa6o710nNsdHlFLJ3/j0TkT+jb/xRBIqPrguy5hX+L9V/N+Ym49rObked4pTBwYgXK9Bw7pacWCdq4IvPgeefe9EdCMmAUTVxLWwTVHfex6ybuiLzzFZPQ6u04cU7PHu9uR+w78QjpwnokpiEkBUCa6579mGgkq94Cnehixjnthkb/CwsI1CAbeR8zdW7K4n+EYNwzmYjoiqDZMAohu4+t+zjHnFKvmCp/YsQ0F/vKe574FKBSL0GrSICkcdfVF/u+vjOvogTo0jIllgEkC1hiAIsFjtyDZYC5vl88QKvviAO0/rzmvUStTRFz7B6zWI0AeJT+91Cv/XBavYPE9EPoFJAPkF19rz2caCp/X8P6/h/F/XxcrdVenb8stemlYbFIhbQoPEit1VqUeEFjTN1wkN4o5xRORX+BeNZE8oHEEvNtEbi5rlXR9nGa0e157Xh6jQoE4I6hR7chcr+tAgROg00Ki57jwR1S5MAkhSTrGCzyus2Euv6Mta4EYBIFSrRuNILeqEFlXwTRqFI1BwFlbw3FiGiKg0TAKo2rie4F0D7LKKDbbLKpwu52kFOwWAUJ0aUfX0YtN88YreNdAuUFlygB2XqCUiKh+TAKoU1xr0BZX7DU/xhqKneU8VfJhOjej6+sIKvbAfPrTo4zCdutQKnoiIvINJAJUgCALMeXbxab2gaT5PbJ4vrw/e1UQfVU+HCH0Q6hR7gndV8qzgiYikxySgFsq1FlXwWYai/ndTnh1//2NGVjmj6ENDVGgUqS2o3PVBqBNaMIK+TmGFX1YTPRERyQuTAD9jy3eIlXtWsab64s30nubB64JVaBARUvDkHqq5oaIvGEWvCmQFT0TkD5gE+BC7wymuWlf8Sb54ZW/KLXsnuRBNIOqEBomVuquZ3jVNrsVtt8B4PbcGS0RERFJiEiAzFzNN+PsfS7EKvqiyv24qey16tSoAdfRBiK6vK6rkQ4PcBtqVt9BNkDoQHE9PRFR7MAmQkYyzWUj8Ir3EcWVA4Vr0jcMKK/aiyt1V2WuDuJMcERFVDJMAGfn7HwsAoGeHW9H+9lvEwXZhWjUCAljBExGRdzEJkBGD2QYA6N6uAVo3iZA4GiIi8ncc5i0jRktBEhCqVUscCRER1QZMAmTEYCkY2c8kgIiIagKTABkxmG0IUCgQEsReGiIiqn4+UdsYjUZMnjwZJpMJ+fn5mDZtGjp37oz09HTMmTMHSqUScXFxGDt2rNShVonBYoM+RIUAjvInIqIa4BMtAatWrUK3bt3w2WefYd68efjPf/4DAJg+fToSExOxYcMGHD58GBkZGRJHWjVGi41dAUREVGN8oiXgmWeegVpdUDk6HA5oNBqYTCbYbDZER0cDAOLi4pCWloZ27dpJGWql5dsdyLU6EBqikjoUIiKqJWSXBCQnJ2PNmjVux+bOnYsOHTogMzMTkydPxhtvvAGTyQSdTideo9VqceHChZoO12sM5oJBgXq2BBARUQ2RXRKQkJCAhISEEsdPnDiB1157DVOmTEHXrl1hMplgNpvF82azGaGhoeXePyIiBIGBSq/GHBmpr/I9cvLsAID6t+i8cr/KkvJre5u/lMVfygGwLHLlL2Xxl3IANVcW2SUBpTl58iReeeUVfPjhh2jdujUAQKfTQaVS4fz584iKikJqaupNDQzMzrZ4NbbISD0yM6u+4v75SzkAAFUAvHK/yvBWWeTAX8riL+UAWBa58pey+Es5AO+XxVNC4RNJQGJiImw2G+bMmQOgIAFYunQpZs6ciUmTJsHhcCAuLg4dO3aUONLKE7sDOCaAiIhqiE8kAUuXLi31eKdOnZCUlFTD0VQPg2u1wBCOCSAioprhE1MEawPXvgGcIkhERDWFSYBMGNkSQERENYxJgEy49g3gmAAiIqopTAJkwmC2IUithFrl3emLREREZWESIBMGi41dAUREVKOYBMiAUxBgsuRzUCAREdUoJgEyYMmzw+EUOB6AiIhqFJMAGeD0QCIikgKTABlwTQ/Uc0wAERHVICYBMuCaHhjGlgAiIqpBTAJkwNUdwDEBRERUk5gEyIA4JoDdAUREVIOYBMiAOCaA3QFERFSDmATIgGtMQCi7A4iIqAYxCZABg8WGAIUC2mAmAUREVHOYBMiAwWyDPkSFAIVC6lCIiKgWYRIgA0aLjWsEEBFRjWMSILF8uwO5VgdCtewKICKimsUkQGJG16BAzgwgIqIaxiRAYgYL1wggIiJpMAmQGFcLJCIiqTAJkJjB7FojgC0BRERUs5gESMy1WiDHBBARUU1jEiAxA5MAIiKSCJMAiXFMABERSYVJgMSK9g1gSwAREdUsJgESM5ptCFIroVYppQ6FiIhqGSYBEjNYbGwFICIiSTAJkJBTEGC05EPPJYOJiEgCTAIkZMmzw+EU2BJARESSYBIgIa4RQEREUmISIKGi6YFMAoiIqOYxCZBQ0fRAjgkgIqKaxyRAQq6WAHYHEBGRFJgESMjIbYSJiEhCTAIk5OoO0LMlgIiIJMAkQEJidwDHBBARkQSYBEjIYLFBoQC0wUwCiIio5jEJkJDRbIM+RI0AhULqUIiIqBZiEiAhgyWfgwKJiEgyPpUEnDp1Cl26dIHVagUApKenIyEhASNGjMDixYsljq5i8u0O5FrtCOW+AUREJBGfSQJMJhPeffddqNVFT87Tp09HYmIiNmzYgMOHDyMjI0PCCCvGKC4UxJYAIiKShk8kAYIg4K233sJrr72G4OBgAAVJgc1mQ3R0NBQKBeLi4pCWliZxpDfPYOGSwUREJK1AqQO4UXJyMtasWeN2rGHDhoiPj0fr1q3FYyaTCTqdTnyt1Wpx4cKFcu8fERGCwECl9wIGEBmpr/DnnLtmAQDcWk9Xqc+vLnKKpar8pSz+Ug6AZZErfymLv5QDqLmyyC4JSEhIQEJCgtuxAQMGICUlBSkpKcjMzMRzzz2HZcuWwWw2i9eYzWaEhoaWe//sbItX442M1CMz01jhz7t4+ToAQCkIlfr86lDZssiRv5TFX8oBsCxy5S9l8ZdyAN4vi6eEQnZJQGm+++478eO+ffvi008/hUajgUqlwvnz5xEVFYXU1FSMHTtWwigrRtxBkKsFEhGRRHwiCSjLzJkzMWnSJDgcDsTFxaFjx45Sh3TTDNw3gIiIJOZzScCuXbvEjzt16oSkpCQJo6k8g7lwdgCnCBIRkUR8YnaAPzJydgAREUmMSYBEDGYbNGolNCrvzlQgIiK6WUwCJGKw2Lh7IBERSYpJgAQEQYDRko9QzgwgIiIJMQmQgMVqh8MpcGYAERFJikmABMQ1ApgEEBGRhKolCcjNza2O2/oNVxLA6YFERCSlCiUB/fr1w9q1az1es3jxYvTt27dKQfk77iBIRERy4HGxoIsXL8JkMomvL126hNOnT+P48eOlXp+fn4+0tDS2BJRDXC2QAwOJiEhCHpOAw4cPY+LEiVAoFAAAhUKBjRs3YuPGjWV+jiAIuPvuu70bpZ/hmAAiIpIDj0nAoEGD8PvvvyMrKwuCIGDr1q1o3bo12rRpU+r1KpUK9erVw8iRI6slWH9hELsDOCaAiIikU+7eAZMnTxY/3rdvH4YMGYKnnnqqWoPyd0YzuwOIiEh6FdpAqPjmPVR5BosNCgWgDWZLABERSadSuwhu3boV27dvx/Hjx2E0GhEREYGOHTvi0UcfRa9evbwdo98xmG3Qh6gRUDjWgoiISAoVSgJsNhteeOEF/PLLLxAEAWFhYYiOjobBYMC3336L7777Dg8//DDmzZtXXfH6BYMlH7eEaqQOg4iIarkKJQFLlixBWloaBgwYgClTpiAqKko8d+nSJbzzzjvYunUr2rVrhyeeeMLrwfqDfLsTuVY79CF6qUMhIqJarkKLBW3fvh1t27bFwoUL3RIAAGjUqBEWLFiAFi1a4IsvvvBqkP7EWLhGQBgHBRIRkcQqlARcu3YN3bp1Q0BA6Z8WGBiI7t274+LFi14Jzh+5FgriGgFERCS1CiUBLVq0wG+//ebxmpMnT6Jp06ZVicmvGcyFawRw3wAiIpJYhZKAyZMn49ChQ5g9ezaMRmOJ85988gn27duHN954w2sB+hsjWwKIiEgmKjQwMCUlBU2bNsX69euxZcsWtGrVCvXr10deXh4yMjKQmZkJrVZbYnaAQqHA5s2bvRq4r+K+AUREJBcVSgL++9//ih+bzWb8+uuvJa4xmUw4duyY2zEF58OLjGbuIEhERPJQoSSgrN0D6eZddy0ZzH0DiIhIYhUaE0BVJ44JYHcAERFJrFLLBh84cAApKSk4ceIEcnNzER4ejhYtWuDBBx9EbGyst2P0KwaLDRq1EhqVUupQiIiolqtwEpCYmIgVK1ZAEAQAQHBwMM6ePYtDhw4hOTkZo0ePxoQJE7weqL8wWvLZFUBERLJQoe6Ar776Cp988gmaN2+OZcuW4cCBAzh06BAOHz6MTz/9FK1atcLy5cuxc+fO6orXpwmCAIPZxkGBREQkCxVKAtauXYvIyEisXbsWvXr1gk6nAwCo1Wr06NEDn376KerWrYt169ZVS7C+zmK1w+EUuEYAERHJQoWSgBMnTqBPnz6IiIgo9XydOnXQp0+fElMEqYDBzDUCiIhIPqpldkB+fn513NbnGS1cMpiIiOSjQklAq1atsHv3buTk5JR6PisrC7t27UKrVq28EZvfcbUEsDuAiIjkoEJJwFNPPYXMzEyMGjUK+/btg91uB1CwSuCPP/6IZ555Bv/88w+eeOKJagnW14lLBjMJICIiGajQFMH4+HgcPXoUq1atwtNPP42AgACo1Wrk5eUBKBj9/uyzz2Lw4MHVEqyv45gAIiKSkwqvEzB16lT069cPmzdvxvHjx2E2m6HVatG6dWsMGTKEiwV5II4J4DoBREQkA5VaMTA2NpaVfSWIYwLYEkBERDJQqSTAZDJh586dOH78OCwWC8LCwnDHHXegT58+UKtZwZXFYLFBoQB0QWwJICIi6VU4Cdi4cSPeffdd5ObmiksHAwXbBdepUwdz585Fr169vBqkvzBY8qEPUSMggFsrExGR9CqUBHzzzTeYPn066tatizFjxqBDhw7QarW4evUqDh06hA0bNuDll1/GunXr0Llz5+qK2WcZzTbUCdVIHQYRERGACiYBK1asQEREBJKSktCwYUO3c/369cMjjzyC4cOH48MPP8SaNWu8Gqivy7c7YbHa0SREL3UoREREACq4TsAff/yB++67r0QC4NKsWTPce++9OHLkiFeCc3E4HJg9ezZGjBiBIUOGYPfu3QCA9PR0JCQkYMSIEVi8eLFXv6a3GS2cHkhERPJSoZaAiIgIOJ1Oj9cEBwdDq9VWKagbbdu2DXa7HV988QWuXLmCr7/+GgAwffp0LFq0CFFRURg9ejQyMjLQrl07r35tb3FND9RzeiAREclEhVoChg0bhu3bt5e5QdD58+fx5ZdfYsiQIV4JziU1NRUNGjTA6NGj8eabb6Jv374wmUyw2WyIjo6GQqFAXFwc0tLSvPp1vcm1WmAYWwKIiEgmPLYErF271u11SEgItFotEhIS8NBDD6Fz586oW7cuDAYDjh49im3btiEiIgJt27atdEDJycklxhNERERAo9Fg2bJl2L9/P15//XUkJiaKWxkDgFarxYULF8q9f0RECAIDlZWOrzSRkeX38wtnswEADeuH3tT1UpFzbBXlL2Xxl3IALItc+UtZ/KUcQM2VxWMSMHfuXCgUCnEqYPGPU1JSkJKSAoWiYLqb67jBYMCECRNw//33VyqghIQEJCQkuB2bMGECevfuDYVCga5du+Ls2bPQ6XQwm83iNWazGaGhoeXePzvbUqm4yhIZqUdmprHc6y5dMQAAFA7nTV0vhZstiy/wl7L4SzkAlkWu/KUs/lIOwPtl8ZRQeEwC5s2b57UgqqJLly748ccfcd999+H48eO49dZbodPpoFKpcP78eURFRSE1NRVjx46VOtQyGc2FYwK4jTAREcmExyTgkUceqak4PBo2bBimT5+OYcOGQRAEzJw5EwAwc+ZMTJo0CQ6HA3FxcejYsaPEkZZNHBPAHQSJiEgmKrVscE1Tq9Wltkp06tQJSUlJEkRUca4kQM8kgIiIZKJCSUDXrl1v6jqFQoG9e/dWKiB/ZTDboFEpoVF7d1AiERFRZVUoCSg+Gr+4vLw85OTkwOl0omXLloiKivJKcP7EaMnnGgFERCQrFUoCdu3aVeY5o9GIpUuXIiUlBQsWLKhyYP5EEAQYzDY0beA/01eIiMj3VWixIE/0ej2mTJmC5s2b47333vPWbf1CrtUOh1PgeAAiIpIVryUBLp07d8aBAwe8fVufdt3s2jeA3QFERCQfXk8Cjh07Ji4gRAWK9g1gSwAREclHhcYEfP/996UeFwQBFosFP/zwA/bs2YMBAwZ4JTh/YTBzB0EiIpKfCiUBL7/8ssenfEEQUK9ePUyaNKnKgfkTcRthtgQQEZGMeC0JUKvVuP3229GrVy+oVOz7Lk4cE8ApgkREJCMVSgLGjRtX4pjVasXff/+NunXrQqvVei0wfyKOCWB3ABERychNDQzctWsXXn/9dRw/ftzteGJiIrp164b7778fXbt2xauvvors7OxqCdSXuZYM5pgAIiKSk3JbAt5++20kJycDAHr37o3WrVsDAD744AN88sknUCgU6NGjBwDg22+/xcmTJ7F582ao1azwXIxmGxQKQBfE7gAiIpIPjy0Bu3btQlJSEtq0aYMVK1agd+/eAIArV67g008/hUKhwKxZs7By5UqsXLkSixYtwsmTJ7F27dqaiN1nXLfkQx+sQkAAp04SEZF8eEwCNm3ahPDwcKxduxZ33303NBoNAGDHjh2w2+2Ijo7G0KFDxev79euHmJgY7Nixo3qj9jFGs43jAYiISHY8JgFHjhxB7969S2wctGfPHigUCvTt27fE53Ts2BHnzp3zbpQ+zO5wwmK1c3ogERHJjsck4Pr166hfv77bMafTiYMHDwIAunfvXuJzAgMDkZ+f78UQfZtrZgAHBRIRkdx4TAL0en2J0f5HjhyByWRCYGAg7rzzzhKfc/bsWURERHg3Sh/mWi2Q2wgTEZHceEwC2rdvjz179sDpdIrHvvzySwAFrQDBwcFu12dmZiI1NRXt27evhlB9k4GrBRIRkUx5TAKGDRuGixcv4rXXXsP+/fuxfv16bNy4EQqFAiNHjnS7NisrC6+++iry8vLw4IMPVmvQvoT7BhARkVx5XCegX79+GDlyJNavX49vvvkGQMH+AI8//jh69eolXjdmzBikpaXBarXi/vvvR//+/as3ah8ijglgSwAREclMuYsFvfXWW7jvvvuwe/du2O123H333eJ6AS6nT5+GVqvF6NGj8eKLL1ZXrD5JHBOg5ZgAIiKSl5vaO6Br167o2rVrmec3b95cYhohFeCYACIikqub2jugPEwAysYkgIiI5MorSQCVzWjOh0alhEatlDoUIiIiN0wCqpnBYuMaAUREJEtMAqqRIAgwmG2cHkhERLLEJKAa5VrtcDgFjgcgIiJZYhJQjQzivgHsDiAiIvlhElCNivYNYEsAERHJD5OAaiQuGcwkgIiIZIhJQDUyWrhaIBERyReTgGrkGhMQxpYAIiKSISYB1cggtgQwCSAiIvlhElCNOCaAiIjkjElANTKabVAoAF0wxwQQEZH8MAmoRgZLPvTBKgQEKKQOhYiIqAQmAdXIaLFxPAAREckWk4BqYnc4Yc6zczwAERHJFpOAamIsnB7IHQSJiEiuAqUO4GYYjUZMmDABubm5UKlUeO+99xAZGYn09HTMmTMHSqUScXFxGDt2rNShisSZAewOICIimfKJloDNmzejZcuWWL9+PeLj47Fy5UoAwPTp05GYmIgNGzbg8OHDyMjIkDjSIq7VAtkdQEREcuUTSUDLli1hNpsBACaTCYGBgTCZTLDZbIiOjoZCoUBcXBzS0tIkjrTIdbYEEBGRzMmuOyA5ORlr1qxxO/b222/j559/Rnx8PK5fv47169fDZDJBp9OJ12i1Wly4cKHc+0dEhCAwUOnVmCMj9SWOORVXAACNG4SWel6ufCnW8vhLWfylHADLIlf+UhZ/KQdQc2WRXRKQkJCAhIQEt2Njx47Fv/71L4wYMQLHjx/HuHHjsGHDBrF1AADMZjNCQ0PLvX92tsWr8UZG6pGZaSxx/HLhMcHuKPW8HJVVFl/kL2Xxl3IALItc+UtZ/KUcgPfL4imh8InugNDQUOj1BYW45ZZbYDabodPpoFKpcP78eQiCgNTUVMTGxkocaREjuwOIiEjmZNcSUJpXXnkFb775Jj7//HPY7XbMmjULADBz5kxMmjQJDocDcXFx6Nixo8SRFrnOgYFERCRzPpEE1K9fH5988kmJ4506dUJSUpIEEZXPaM6HWhUAjdq74w+IiIi8xSe6A3yRwWJjKwAREckak4BqIAgCjBYbxwMQEZGsMQmoBrlWO+wOgS0BREQka0wCqoGB+wYQEZEPYBJQDbhvABER+QImAdWA+wYQEZEvYBJQDcTuAC27A4iISL6YBFQDsTuALQFERCRjTAKqgYHdAURE5AOYBFQD7htARES+gElANTBY8qEAoAvmmAAiIpIvJgHVwGC2QReiQkCAQupQiIiIysQkoBoYuW8AERH5ACYBXmZ3OGHOs3M8ABERyR6TAC8zcslgIiLyEUwCvIxrBBARka9gEuBlriWD9ewOICIimWMS4GWuhYLCmAQQEZHMMQnwMoOZYwKIiMg3MAnwMi4ZTEREvoJJgJe5lgzmmAAiIpI7JgFe5tpGOJTdAUREJHNMArzMYLFBrQpAkDpQ6lCIiIg8YhLgZQYzlwwmIiLfwCTAiwRBgNFig55JABER+QAmAV6Ua3XA7hA4HoCIiHwCkwAvcq0WyM2DiIjIFzAJ8KLrZiYBRETkO5gEeJG4bwDHBBARkQ9gEuBFXCOAiIh8CZMALzKyO4CIiHwIkwAv4r4BRETkS5gEeJGB+wYQEZEPYRLgRQZLPhQAdMFcMpiIiOSPSYAXGS026EJUUAbw20pERPLH2sqLuG8AERH5EiYBXmJ3OGHOs0PP6YFEROQjmAR4idG1RgAHBRIRkY9gEuAlRk4PJCIiHyPbJOC7777DxIkTxdfp6elISEjAiBEjsHjxYvH44sWLMXToUIwYMQJHjhyRIlQARWsEcHogERH5ClnOZZs9ezZSU1PRpk0b8dj06dOxaNEiREVFYfTo0cjIyAAA7Nu3D8nJybh8+TLGjRuHlJQUSWJ2rRHAJYOJiMhXyDIJiImJQf/+/bFx40YAgMlkgs1mQ3R0NAAgLi4OaWlpUKvViIuLg0KhQMOGDeFwOJCVlYU6derUeMwGs2vfALYEEBGRb5A0CUhOTsaaNWvcjs2dOxfx8fHYu3eveMxkMkGn04mvtVotLly4AI1Gg/DwcLfjRqPRYxIQERGCwECl9woBIDJSD0fhx9GNwxEZqffq/WuSL8d+I38pi7+UA2BZ5MpfyuIv5QBqriySJgEJCQlISEgo9zqdTgez2Sy+NpvNCA0NhUqlKnFcr/f8jcvOtlQ+4FJERuqRmWnE39dMAACHzY7MTKNXv0ZNcZXFH/hLWfylHADLIlf+UhZ/KQfg/bJ4SihkOzCwOJ1OB5VKhfPnz0MQBKSmpiI2NhYxMTFITU2F0+nEX3/9BafTKUlXAFC8O4BjAoiIyDfIckxAaWbOnIlJkybB4XAgLi4OHTt2BADExsZi+PDhcDqdePvttyWLz2CxQR0YAI3Ku10NRERE1UW2ScBdd92Fu+66S3zdqVMnJCUllbhu3LhxGDduXE2GViqjxQZ9iBoKhULqUIiIiG6KT3QHyJ0gCDCY87laIBER+RQmAV6Qa3XA7nByPAAREfkUJgFeYORqgURE5IOYBHiBgfsGEBFRFSxe/CFee63mx7cxCfACcXogWwKIiKgSjh//HW3btqvxryvb2QG+pKglgGMCiEh+UlN/xH//uwW//54Bs9mE0NBQtGnTDoMHP4RHHhlc6fuOHTsa6em/4uuvd5e7UFt1OnYsAydO2NGqVUev3/urr7Zj7dpPceXK39Dp9FiyZAWioqK9dn+73Y7+/eNgt9uRnv4rVq9egebNm2P16i+89jU8YUuAFxjNHBNARPK0YMF8TJs2EWfOnEbPnr0wfPhIxMbehSNHDmPatIl46623pA6xSvbsScULLzyLkydPev3eZ8+ewTvvzEJOTg4eeWQo7r13IBo0uLXUa7/7bgfi4mKRmPhuufd97725iIuLxZIlCxEQEID/+79VAIDly1dj27Yd+Pzzz71aDk/YEuAFHBNARHL0668HkJKShN69+2LGjLkIDCz6k28ymTB+/AtISkpC585d0bNnb+kCrYKcnGw4nc5qufeff56A0+nEkCEJGD36JY/XNm/eEgBw+rTnZOSPP45j+/atiIysh2eeeR4BAQH4559rCAnRok2bdlAoFAgLq7klkNkS4AUGC8cEEJH87NmTCgAYMmSYWwIAFCzHPmZMwUC0H3/cXeOx+YL8/IK/7cU3qitLdHQTqNUanDrlOQlYsOA9OJ1OjB07ASEhIQAKko3mzVtIstgckwAvMJptUADQBbNhhYjkw263Ayj76bRjx0748MMPMXz44+KxsWNHIy4uFkaj+5Po5ct/IS4uFq+/PrHEfa5c+Ruvvz4JAwb0RHx8P8yY8W9cunSxxHXXrl3De+/NxZAhg9C7dzcMGTII7703F9euXXO7bs6cGYiLi0V2dhZmzXoL99/fGwMH9sXrr0/E6dOn3K6bO3cmAGDevHmIi4vF5ct/efye3GwMQ4c+IN77o48+QFxcLFauXFbmfZVKJW677XaYTEZcvXql1Gu++eYrHD16GF26dEW/fgPE43/+eQItW7byGHd1YRLgBQaLDdpgFZQB/HYSkXzceWfB0usff7wQCxbMx2+/HYHD4RDPazRBGDhwIFq0qFoFNHHiWJw7dwYPPzwU7dt3wM6d32DMmOfw999/i9dcunQRzz03Etu2bUZ0dBM8+uhwREc3wbZtmzFq1BOlJg0TJ47HwYMHMGjQg4iN7Yo9e1Lx0kuj8OeffwAAevbsjZ49ewEA4uLi8Oyzz0OnK3uAYkViGDbsMfHeXbt2x7PPPo/Onbt4/D40b94CAEptDbBYLFi6dBECAwPx2mtT3M6dPPknmjVr4fHe1YWPrl5gMNsQptNIHQYRkZu77+6Jhx8eiq1bNyElJQkpKUnQarXo0KET7rzzLvTu3c8r+9bfemsjLFy4FBpNwd/BrVtT8P7787Bs2WJMnz4bADB//hxkZf2DqVPfxAMPPCx+7pYtm5CY+A7mz5+DhQuXut3XYLiO1as3iM3xP/zwPd58cyoWLnwfixcvxz339IbJZMRPP/2Inj17YtCgRz3GWZEYhg17HDqdHj/99CO6deuOYcMeL+OuRYqPC+je/W63c6tXr8C1a5kYOfJpNGnS1O2cw+HAuXNnce1aJjSaIK+8JzeLj65VZHc4Yc6zc3ogEcnSpEnTMH/+h7jrrh4IDAyE2WxGWtrP+OijDzBs2ENITEys8sC6F154WUwAAODhhx9FdHQT/PjjLthsNly9egUHD+5Hx46d3SpfAHjkkaFo06YtDh7cX6Ip/+mnR7n1x/fu3Q8dOnRCevqvZTa5l6WyMVREs2bNAcCtywIAzp8/h+TkDahXrz6eeeZfJT5v9OiX8P333+KRR+KxbNniSn/9ymBLQBUZCqcHclAgkW9I2nUS+49frdI9lEoFHA7BSxGVdGfrehjWt7nX7tejRxx69IiDxWLB4cO/4sCB/fj55//h4sULWL58OUymPLz00vhK3VuhUOCOOzqUON6mTTucP38O586dxZUrBd0CHTt2LvUe7dt3xLFjv+PkyT9w660NxeOdOsWUet8jR9Jx6tSfqFev/k3H+ccfJyoVQ0WUNUPgo48SkZ+fj3HjJiA4OLjE591770Dce+/ASn3NqmJLQBXlGK0AAD2nBxKRzIWEhKB79ziMGzcBGzZsxtSpb0KhUCAlZSPy8vIqdU+9PhQqVcmW0JAQLQAgN9cCi8UMoGBGQmnq1o0EgBIxREbWK3HtLbfcAqBgimNFVDaGiggNDUW9evVx7txZcVBmaur/8Msve3DnnXehT5/+lb53dWFLQBXlmAqSAHYHEPmGYX2bV/kpOzKy5uZxV5bZbMKoUU8iOroJ5s//sMR5hUKBBx54GD///ANSU1Nx9eoVREc3EaepCYJ7S0dZlaPZbIIgCCWmt127lgkACA0Ng8Fw3e3YjVwzEcLCwt2OW615CAoKKvXam5m2V5xrOl5FY6io5s1bYs+en3Dx4gU0bNgIixZ9AJVKhQkTppT/yRJgS0AVXXclAewOICIZ0Wp1MJlMOHBgH7Ky/vF4bUBAgPiE7Xqqz821uF1T2uh9oGBQm6up3cVut+PEiWMIDg5BVFQ0mjcvmH1w+HB6qfdIT/8VCoUCTZve5nb82LHfS1ybkXEUSqUSrVq1AYCbnltf2RgqqvgMgQ0b1uHSpYsYMeIJREc3qdJ9qwuTgCoSkwB2BxCRzDz66DDYbDa8+ebUEvPggYI9Bfbs2YN77ukNrbagmTw6uimAooWGAMBqtWLDhnVlfp1Vq5a7TT1cv34Nrl69goEDB0GpVKJBgwaIiYnFiRPHsHXrJrfP3b59K44ePYyYmNgSffwrVy6D2VzU7P/DD9/j118PoGfPXggNDQMAKJUFDdquhX3KUtkYKso1LmDv3j347LPVqF+/AZ5+elSV7lmd2B1QReKYALYEEJHMPPXUczh16iR++OF7jBjxMLp27Y6oqGjY7Xb8/vtvOHr0MG6//XZMnPi6+DmDBj2ILVuSsXDh+8jIOIqwsHCkpv4IrVZX6qA2tVqDEyeO4/nnn0Zs7J34888/sH//XjRp0hTPP1+01O7kyW/g5Zefx/vvv4MfftiFZs1a4PTpk9i/fy/q1o3ElCn/LnHvv/66iGefHYkePXoiM/MKfvrpR0RG1sPYsa+J10RGFvTlb9iwAZcvZyIhYYTYv3+jysRQUa6WgK++2g4AePPNmSW6NOSELQFVxDEBRCRXSqUSs2e/izlz3sNdd3XH8eO/Izn5C3z55VbYbDa88MJYbNmyBREREeLntGjREu+9txCtW7fFrl078e23XyM2tisWLlwKpVJZ4muo1SosXLgUYWFhSElJwp9//oGHHhqCJUtWuO0sGBUVjRUr1uKBBx7B2bNnsHlzEi5evIChQ0dg1ar1aNSocYl7z5r1Llq3bov/9//+iyNHDuP++wdh+fLVaNCggXhNp04xGDIkAdevX8fmzUk4c+Z0md+PysRQUY0bR4mVfteu3dGrV98q37M6KYQbR3/4OW8P5lmyLQMHjl3BktfuQZDatxtWfGGw083yl7L4SzkAlkWu5FiWOXNm4Ouvv8SqVetvejVDOZajsrxdFk+LD7EloIpyTFaoAwOgUZXMkImIiOSMSUAV5Rit0IeoJdn9iYiIqCqYBFSBIAi4brIiVMvxAERE5HuYBFRBns2BfLuTqwUSEXnRv/89A6mpB6q8uyGVj0lAFRgs3DeAiIh8F5OAKhA3D2JLABER+SAmAVVgMBesUMU1AoiIyBcxCagCY2F3AFcLJCIiX8QkoAo4JoCIiHwZk4Aq4JgAIiLyZUwCqiBCr0GoVo26YfLdHIKIiKgsvr3YvcQGdW+KJwa1Q3aWWepQiIiIKowtAVUUqOS3kIiIfBNrMCIiolqKSQAREVEtxSSAiIiolmISQEREVEsxCSAiIqqlZJsEfPfdd5g4caL4Oi0tDcOHD8fIkSMxfvx45ObmAgAWL16MoUOHYsSIEThy5IhU4RIREfkcWa4TMHv2bKSmpqJNmzbisRkzZmD9+vWoW7cuEhMTkZycjC5dumDfvn1ITk7G5cuXMW7cOKSkpEgYORERke+QZUtATEwMZsyY4XZs3bp1qFu3LgDAbrdDo9Hg4MGDiIuLg0KhQMOGDeFwOJCVlSVBxERERL5H0paA5ORkrFmzxu3Y3LlzER8fj71797odr1evHoCCboK9e/fi1VdfxcqVKxEeHi5eo9VqYTQaUadOnTK/ZkRECAIDld4rBIDISL1X7ycllkV+/KUcAMsiV/5SFn8pB1BzZZE0CUhISEBCQsJNX7969Wrs2LEDK1asgEajgU6ng9lctGSv2WyGXu/5G5edbal0vKWJjNQjM9Po1XtKhWWRH38pB8CyyJW/lMVfygF4vyyeEgpZdgeUZunSpThw4ABWr14tPunHxMQgNTUVTqcTf/31F5xOp8dWACIiIioiy4GBN7p27Ro+/vhjtG3bFs8//zwAYODAgXj88ccRGxuL4cOHw+l04u233y73XtXRxMImKHnyl7L4SzkAlkWu/KUs/lIOoObKohAEQaiRr0RERESy4jPdAURERORdTAKIiIhqKSYBREREtRSTACIiolqKSQAREVEt5RNTBOXA6XRixowZOHHiBNRqNWbPno0mTZqI53ft2oWPP/4YgYGBePTRRzFs2DAJoy1bfn4+3njjDVy6dAk2mw0vvvgi+vXrJ55ftWoVNm3aJK63MHPmTNx+++1ShVuuhx9+WFwgqnHjxpg3b554zlfeEwDYvHkztmzZAgCwWq04duwYfv75Z4SGhgLwjffl8OHDeP/997Fu3TqcO3cO06ZNg0KhQIsWLTB9+nQEBBQ9c5T3+yS14mU5duwYZs2aBaVSCbVajXfffVdcwtzF08+h1IqXJSMjA2PGjEHTpk0BAI899hji4+PFa33pfZkwYQKuXbsGALh06RI6duyIBQsWuF0vx/eltL/BzZs3l+73RaCb8s033whTp04VBEEQDh06JIwZM0Y8Z7PZhP79+ws5OTmC1WoVhgwZIly9elWqUD3atGmTMHv2bEEQBCErK0vo1auX2/mJEycKR48elSCyisvLyxMeeuihUs/50ntyoxkzZghffPGF2zG5vy/Lly8XBg8eLCQkJAiCIAgvvPCC8MsvvwiCIAhvvfWW8O2337pd7+n3SWo3lmXkyJHC77//LgiCIGzYsEGYO3eu2/Wefg6ldmNZkpKShJUrV5Z5vS+9Ly45OTnCgw8+KFy5csXtuFzfl9L+Bkv5+8LugJt08OBB9OzZEwDQqVMn/Pbbb+K5U6dOITo6GmFhYVCr1ejSpQsOHDggVage3X///XjllVfE10ql+z4KGRkZWL58OR577DEsW7aspsOrkOPHjyM3NxfPPfccnnrqKaSnp4vnfOk9Ke7o0aM4efIkhg8f7nZc7u9LdHQ0Fi1aJL7OyMhA165dAQD33HMP9uzZ43a9p98nqd1Ylg8++EDc0dThcECj0bhd7+nnUGo3luW3337DDz/8gJEjR+KNN96AyWRyu96X3heXRYsW4YknnhD3l3GR6/tS2t9gKX9fmATcJJPJBJ1OJ75WKpWw2+3iueJ7Fmi12hK/XHKh1Wqh0+lgMpkwfvx4vPrqq27nBw0ahBkzZmDNmjU4ePAgdu/eLU2gNyEoKAijRo3CypUrMXPmTEyaNMkn35Pili1bhpdffrnEcbm/L/fddx8CA4t6FwVBgEKhAFC0sVdxnn6fpHZjWVyVy6+//orPPvsMzzzzjNv1nn4OpXZjWTp06IApU6Zg/fr1iIqKwscff+x2vS+9LwDwzz//IC0tDUOGDClxvVzfl9L+Bkv5+8Ik4CbduFmR0+kUfyArs5GRlC5fvoynnnoKDz30EB544AHxuCAIePrpp1GnTh2o1Wr06tULv//+u4SRenbbbbfhwQcfhEKhwG233Ybw8HBkZmYC8L33BAAMBgNOnz6Nbt26uR33tfcFgFt/ptlsFsc2uHj6fZKjr776CtOnT8fy5ctL7E/i6edQbgYMGIA77rhD/PjGnyNfe1927NiBwYMHl2jRBOT9vtz4N1jK3xcmATcpJiYG//vf/wAA6enpaNmypXiuWbNmOHfuHHJycmCz2XDgwAF07txZqlA9unbtGp577jlMnjwZQ4cOdTtnMpkwePBgmM1mCIKAvXv3in8w5GjTpk145513AABXrlyByWRCZGQkAN96T1z279+PHj16lDjua+8LALRt21bcDvx///sfYmNj3c57+n2Sm23btuGzzz7DunXrEBUVVeK8p59DuRk1ahSOHDkCAEhLS0O7du3czvvS+wIUlOGee+4p9Zxc35fS/gZL+fsi3xRPZgYMGICff/4ZI0aMgCAImDt3LrZv3w6LxYLhw4dj2rRpGDVqFARBwKOPPor69etLHXKp/u///g8GgwFLlizBkiVLABRs6Zybm4vhw4djwoQJeOqpp6BWq9G9e3f06tVL4ojLNnToULz++ut47LHHoFAoMHfuXHz99dc+9564nDlzBo0bNxZfF//58qX3BQCmTp2Kt956Cx988AFuv/123HfffQCAKVOm4NVXXy3190mOHA4H5syZg1tvvRXjxo0DANx5550YP368WJbSfg7l+vQ8Y8YMzJo1CyqVCnXr1sWsWbMA+N774nLmzJkSiZnc35fS/gb/+9//xuzZsyX5feEGQkRERLUUuwOIiIhqKSYBREREtRSTACIiolqKSQAREVEtxSSAiIiolmISQOTnLl68iFatWuHJJ58Uj1mtVnz66acSRlUkNTVVnLsOAHv37kWrVq0wZ84cCaMiqh2YBBDVQk888YQ4R1lKn3/+OUaNGoWrV6+Kxxo1aoSxY8eKa6UTUfWRfuUEIqpx//zzj9QhACg9jsaNG4sL8xBR9WJLABERUS3FJICoFnGND7h06RKMRiNatWqFadOmiedNJhPef/999O/fH3fccQd69uyJ6dOnl3hinzZtGlq1aoUjR44gPj4e7du3F5c0BYDdu3fjX//6F7p164Z27dqhW7dueOmll3Ds2DHxHk8++SQWL14MAHj55ZfRqlUrAGWPCThz5gwmTZqEHj164I477kD//v0xf/78EjuuuWK7fv06pk+fjrvvvhvt27fHkCFD8M0333jvm0nkB9gdQFSLhIaGYuzYsVizZg2sVitGjx6NNm3aAACMRiMef/xx/PHHH+jevTvuvfdeXLx4EUlJSfjpp5/wxRdflNiz/cUXX0T79u1x9913IyQkBAqFAp999hlmzZqF6OhoDB48GCqVCkePHsX333+PX375BTt27EC9evXwyCOPAAD27duH+Ph43H777WXGffjwYTzzzDPIy8tDnz59EBUVhfT0dKxcuRK7d+/Ghg0bEB4e7vY5zz77LHJycjBw4EBYLBZs374dr7zyCj777LMSG7QQ1VZMAohqkdDQUIwbNw5btmyBwWBw63v/4IMP8Mcff+Dtt9/GyJEjxePff/89XnrpJcyZMwcLFy50u19MTAwWLVokvrbZbFiwYAGaNm2KLVu2ICQkRDw3Y8YMbNiwAbt378bw4cMxZMgQXLp0Cfv27cOgQYPQv3//UmN2OByYMmUKbDYbli1b5rZr3Pvvv49PPvkE8+fPL7GpilKpxJdffinG0L17d0yaNAlJSUlMAogKsTuAiGC327F161a0aNHCLQEAgH79+iEmJgbfffcdTCaT2znXbmcuDocDs2bNwpw5c9wSAADo2rUrgIoPSjx06BDOnj2LQYMGldg2dvz48ahfvz62b98Om83mdm7kyJFuMbh2Xjx79myFvj6RP2NLABHhzJkzsFgscDgcbk/2LlarFQ6HAydOnECXLl3E440aNXK7Ljg4GPHx8eI9T506hfPnz+PPP/9EWloaAMDpdFYoNtc4gjvvvLPEObVajfbt22Pnzp04ffo0WrduLZ677bbb3K7V6/UAUCJZIKrNmAQQEQwGAwDg9OnT4mC90ly/ft3tdVBQUIlr9u/fj3nz5iEjIwMAoNFo0Lp1a7Rr1w6XL19GRXcvd7U+6HS6Us+7xink5ua6HVer1W6vFQoFAFT46xP5MyYBRAStVgsAeOihhzB//vxK3+fSpUt4/vnnodFoMGvWLHTp0gVNmzaFUqnEV199hZ07d1Y6tuILChXnSmBuHBhIROXjmAAiwm233Qa1Wo2MjIxSn5RXr16NJUuWIDs72+N9du7cidzcXIwfPx7Dhg1Ds2bNoFQqAQCnTp0C4P4k7no698Q1e+HgwYMlzjmdThw8eBAhISEluiaIqHxMAohqIZVKBbvdLr7WaDSIj4/HyZMnsWrVKrdr9+7di/nz5yMlJQVhYWEe76vRaAAA165dczt+/PhxrF27FgDcvm5gYEFjpKd++i5duqBJkyb49ttv8eOPP7qd++ijj3D58mUMHDiwRPM/EZWP3QFEtVC9evVw9uxZTJo0CXFxcXj44YcxdepUHDp0CO+++y6+//57dOjQAVeuXMG3336LwMBAzJ07FwEBnp8b+vTpg8TERCxbtgynT59GdHQ0zp07h927d4sD83JycsTr69evDwBYunQpjh07hrFjx5a4Z0BAAN555x2MGjUKY8aMQZ8+fRAdHY1Dhw4hPT0dzZo1w5QpU7z3zSGqRdgSQFQLTZ48GS1atMCOHTuwbds2AECdOnWQlJSE5557DleuXMG6detw4MAB9O3bF0lJSbjrrrvKvW/9+vWxatUqdOvWDb/88gs+//xznDlzBk8++SS+/vprhIeH46effhK7BOLj4zFw4EBcuHABn3/+OS5dulTqfWNiYrBp0ybEx8fj0KFDWL9+PXJycvDiiy8iOTmZ4wGIKkkhcKgsERFRrcSWACIiolqKSQAREVEtxSSAiIiolmISQEREVEsxCSAiIqqlmAQQERHVUkwCiIiIaikmAURERLUUkwAiIqJaikkAERFRLfX/AXou4ARkzayAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x396 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_lines([compute_subopt(vs, solver.v)], [r\"Subopt of $V^t$\"], [\"Iteration\", \"Subopt\"], \"figs\", \"VI.pdf\", show = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49ddd875",
   "metadata": {},
   "source": [
    "**Question** : \n",
    "- Add easier (warm up) questions.\n",
    "- Why the iterates of Value Iterations are an upper bound to $V^\\star$ ? How is this possible considering that $V^\\star \\triangleq \\max_{\\pi \\in \\Pi} V^{\\pi}$ ?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "706ce2d7",
   "metadata": {},
   "source": [
    "**Answers**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3febb059",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "514d0662",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:49.239768Z",
     "start_time": "2023-03-14T00:56:48.970703Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAFgCAYAAAC8KtBSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA9PklEQVR4nO3daWAT1d4G8GeSNF2SNi1SUIGy71DZLPsOAgUEKhYoAgpXBVEvXGRRr8C9bAIKclUUFFTKWlmFV0FZBFksi6yFspRFqSwFWtqkbdIk834oCQ0thbRpZpI+vy80M5PJ/zBNnzmznBFEURRBREREXkshdQFERERUshj2REREXo5hT0RE5OUY9kRERF6OYU9EROTlGPZEREReTiV1ASUlJSXDpesLCQlAamqmS9cpFbZFnrylLd7SDoBtkSNvaQfg+raEhgY+dB579o9JpVJKXYLLsC3y5C1t8ZZ2AGyLHHlLOwD3toVhT0RE5OUY9kRERF6OYU9EROTlGPZERERejmFPRETk5Rj2REREXo5hT0RE5OUY9kRERF6OYU9EROTlGPZERERejmH/GDIyTdh15C+Ioih1KURERE5j2D+GvSevYd7KP3Ah+a7UpRARETmNYf8Y/HxyH1ZwOz1b4kqIiIicx7B/DEEaXwDAXb1J4kqIiIicx7B/DMFaNQCGPREReSaG/WPQ2cLeYJS4EiIiIucx7B+DTpMb9mns2RMRkQdi2D8GH5USWn8fpBsY9kRE5HlUUhfgKUKCfHHnLq/GJyLp7d27Gz/8sAGnTyfAYNAjKCgIdevWR69efdCmTfsir/fNN1/DsWN/4KefdiEwMNCFFTvnzJkEZGRkICKihcvX/eOPm7Fs2VLcuHEdWm0gFi78GpUqhbn0M1566UVcvnypwHmvvPIqRox43aWf9zjYs39MIYF+MGSbkWO2Sl0KEZVi8+fPwaRJ43Dp0kW0bdseAwYMRrNmzXHixHFMmjQOs2fPkLrEYtm/fy9ef/0VXL580eXrvnz5Ej78cBrS0tLQr19/PPdcDzz55FMFLvvLL1vRpk0zfPzx7Eeud+7cmWjTphkWLlwAAJg586N70xdg06at2LhxK/z8/DBp0gd46aVhrmuQE9izf0whgX4AgHSDCU/o/CSuhohKoz/+OIx16+LQoUMnTJ06EyrV/T/her0eb7/9OjZv3oBWrVqjbdsO0hVaDGlpqbBaS6ZTdf78WVitVkRFvYjXXnuj0GVr1KgFALh48UKhy507l4jNmzciNLQcXn75VQBAauodCIKAZ55pjICAAFy9+heys7MRHt4Ivr5+uHbtb1y4cA5RUb1d07DHwJ79YwoJunevPc/bE5FE9u/fCwCIiop2CHoA0Gq1GDnyLQDA7t273F6bJ8jJyQEABAcHP3LZsLDKUKt9kZRUeNjPnz8XVqsVb745FgEBAQCACxfO4emnK9hfnz9/Fn5+fqhYsRIAID7+AC5eTCpGS5zHsH9Mtp79XT1vvyMiaZjNZgAP720+80wj/Pe/H2LAgBj7tDfffA1t2jRDenq6w7LXrv2NNm2a4d13x+Vbz40b1/Huu++ga9e2iIzsjKlT30dy8tV8y926dQtz585EVFRPdOjQAlFRPTF37kzcunXLYbkZM6aiTZtmSE29g2nTPkD37h3Qo0cnvPvuOIfQmzFjKmbO/A8A4H//m4c2bZrh2rW/C/0/edwa+vfvnW/dS5Yseuh6lUolqlatBr0+Azdv3ihwmW3bfsTJk8fRtGkEOnfuap9+4cIFVK9e0/76/PlzqFq1OhQKBY4ePYJFiz7HTz9tQZ8+faDX6wttn6tIGvY5OTkYN24cBg4ciJiYGCQlJeHKlSsYNGgQYmJiMGXKFPvhnLi4OERFRSE6Ohq7drl/r7UMe/ZEJLFnn20OAPj88wWYP38OTp06AYvFYp/v6+uHTp26oGbN2sX6nHHj3sSVK5fQt29/NGwYju3bt2HkyOG4fv26fZnk5KsYPnwwNm1aj7CwynjhhQEIC6uMTZvWY8SIlwrcORg37m0cOXIYPXs+j2bNIrB//1688cYInD9/DgDQtm0HtG2be4FhRERLvPLKq9BqH36hoDM1REcPyrfuxo2bFvr/UKNGbmAX1LvPzMzEF198CpVKhX/9a4LDvKSk8/b3Ark9fdtpgcaNm6J69RqYN+8zbNq0CVqtttAaXEXSc/a7d++G2WzG6tWrsW/fPnzyySfIycnBmDFj0Lx5c0yePBk7duxAo0aNEBsbi3Xr1sFoNCImJgatW7eGWq12W622nn0ae/ZEJJHWrduib9/+2LhxLdati8O6dXHQaDQID2+EZ59tjg4dOqNcufLF/pynnqqABQu+gK9vbidn48Z1+OijWVi06DNMmTIdADBnzgzcuXMbEyf+G71797W/d8OGtfj44w8xZ84MLFjwhcN609Pv4ttvV9kPo//66w78+98TsWDBR/jss8Vo164D9PoM/PbbbrRo0RLR0TEojDM1REfHQKsNfOx1A47n7Vu2bO0w79tvv8atWykYPHgYKleuYp9utVpx6VISBg++fyFecvJV1K/f0P76+vVreOqppx/5+a4kac++atWqsFgssFqt0Ov1UKlUSEhIQEREBACgXbt22L9/P06cOIHGjRtDrVYjMDAQYWFhSExMdGuttnP2vNeeiKT0zjuTMGfOJ2jevBVUKhUMBgMOHNiH//1vHqKj++DLLz8r9gVur78+2h70ANC37wsIC6uM3bt3wmQy4ebNGzhy5BCeeaaxQ8gCQL9+/VG3bj0cOXIo3yH4YcNGOJwv79ChM8LDG+HYsT8eeqj8YYpagzOqV68BAPnOr//55xV8//0qlCtXHi+//A+HecnJV5GVleXQs69WrQZWrFiGAwf24ebNGyhT5gkIglDkuopC0p59QEAAkpOT0aNHD6SmpuLLL7/EoUOH7P8JGo0GGRkZ0Ov1Dvd8ajSaR57nCAkJgEqldFmtGZm5IZ+VY0VoqHT3n7qKN7TBhm2RH3e2Y+nmBOw7nuy2zyuK1s9UwPDe9V22vj59eqBPnx4wGAw4fPgwDhw4gJ07d+LKlStYvvxb+PmpMH78eACAWn3/z3ze7WI0auzzbdPVahUEQUCHDq3yHTlt3LgRNm26goyMFNy4kRugLVs2L3BbR0Q8izNnTuPmzb8QHl4bfn4+AIBOndrmW75p08Y4ceIYbt1KRv36NRB47yiqRuP70N+j0NBAnDx5yKkaADzWuvNq0aIJAODPPy85LP/eewuQk5OD999/D2Fh5R6orT7Onj3rMO3LLz+3/3z06FFUqPCUfX3u+q5IGvbffvst2rRpg3HjxuHatWsYNmyY/WpJADAYDAgKCoJWq4XBYHCY/qgBH1JTM11aa9myWqiUAm7eyURKSoZL1+1uoaGBHt8GG7ZFftzdjqxMEywWsUTWrVQKLll3VqapxP5P6tVrgnr1mmD48DewZcsmzJkzA7GxsRg48GX4+fnBZDLbl81bw507uX9TTSazfbrJZEZgYBDu3jUCcDxlqVDkhn9ycgquX79ln1ZQuzQaHQDg5s1UpKRkIDs79++6UqnJt7y/f+C99d5ESkoGMjJyBy8zGIwFrtv2+3XtmnM1AHjkuvNToFy58khKSsK1a6lQqVTYu3cP9uzZg2efbY6mTVs7vV2Dg5/ElSt/oUePSHzyyXyUKeO6w/mF7ThIGvZBQUHw8cnd49PpdDCbzahXrx7i4+PRvHlz7NmzBy1atEB4eDg++eQTGI1GmEwmJCUloVatWm6tVRAE6DRqPgyHSGaiO9VAdKcaJbJuOe2AGQx6jBgxBGFhlTFnzif55guCgN69+2LXru04ePB33Lx5A2Fhle1HSkVRBHD/0HF2dsEjghoMeoiimO8w861bKQCAoCAd0tPvOkx7UEZG7v+ZThfsMN1ozIafn1+Byz7O7XB52W5rc7YGZ9WoUQv79/+Gq1f/wtNPV8Cnn86Dj48Pxo6d8Og3F0Cr1WLp0uUA3Pv7Jek5+5dffhkJCQmIiYnBsGHDMHbsWEyePBmffvopBgwYgJycHHTr1g2hoaEYMmSIw3J5zye5i07ri7t6070vDRGR+2g0Wuj1ehw+fBB37twudFmFQoEnnngCAOwdqsxMx6OdBV0tDwAWiwXnzjkehjabzTh79gz8/QNQqVIYatTIPSx+/PixAtdx7NgfEAQBVapUdZh+5szpfMsmJJyEUqlE7dp1AeCxz2UXtQZn5b0if9WqWCQnX8XAgS8hLKxysdbrbpKGvUajwYIFC7By5Up8//336N27N6pWrYrly5djzZo1mDVrFpTK3PPu0dHRWLduHdavX49u3bpJUq9Oo4bFKsKQbX70wkRELvbCC9EwmUz4978n5ruPHMgdM//w4YNo164DNJrcW7rCwqoAgMMty0ajEatWxT70c775ZrHDLX0rVnyHmzdvoEePnlAqlXjyySfRpEkznD17Bhs3rnV47+bNG3Hy5HE0adIs350BS5YsgsFw/3qrX3/dgT/+OIy2bdsjKCj3sLtSmXvAOe8p3YIUtQZn2a7Ij4/fj+XLv0X58k9i2LARxVqnFDhcrhN02nv32uuN0Pr7SFwNEZU2Q4cOR1LSBfz66w4MHNgXEREtUalSGMxmM06fPoWTJ4+jcuUqGDfuXft7evZ8Hhs2fI+ZM2ciPv4wdLpg7N27GxqNFv7+/vk+Q632xdmziXj11WFo1uxZnD9/DocOxaNy5Sp49dX7Q8yOH/8eRo9+FR999CF+/XUnqleviYsXL+DQoXiULRuKCRPez7fuv/++ildeGYxWrdoiJeUGfvttN0JDy+HNN/9lXyY0NBRA7u1+6enpePHFgShbNrTA/4+i1OAsW8/+xx83AwD+/e//5DsV4Qk4gp4T7M+15+13RCQBpVKJ6dNnY8aMuWjevCUSE0/j++9XY8uWjTCZTHj99TexdOlyhISE2N9Ts2YtzJ27AA0aNMDOndvx888/oVmzCCxY8IX9yGlearUPFiz4AjqdDuvWxeH8+XPo0ycKCxd+7XBhdKVKYfj662Xo3bsfLl++hPXr43D16l/o338gvvlmBSpUqJhv3dOmzUadOvXwf//3A06cOI7u3Xti8eJv8eSTT9qXadSoCaKiXkR6+l2sXx+HS5ce/kCcotTgrIoVK9nDPSKiJdq371TsdUpBEL30BLSrL3oIDQ3E978kYtnWs3i1Vz20bPDko98kU3K66Ki42Bb58ZZ2AGyLq8yYMRU//bQF33yzotij+3GbFL6+h2HP3gnBmtzD+Gm8Ip+IiDwIw94JOm3uYfy7eh7GJyIiz8Gwd4LtnD0fhkNERJ6EYe+EIFvY82E4RESP7f33p2Lv3sPFPl9PRcewd4JKqYDW34c9eyIi8igMeyfptGqk8Zw9ERF5EIa9k4I1amQZzTDlWB69MBERkQww7J0UdO/2Ox7KJyIiT8Gwd1KwllfkExGRZ2HYO0nHK/KJiMjDMOydZH8YDnv2RETkIRj2TrI/DIdX5BMRkYdg2DvJNmRuOsfHJyIiD8Gwd5LO9jAc9uyJiMhDMOyd5O+rhFql4Dl7IiLyGAx7JwmCgCCNmlfjExGRx2DYF0Gw1hfphhxYRVHqUoiIiB6JYV8EOo0aVlGEPjNH6lKIiIgeiWFfBDqOokdERB6EYV8EHEWPiIg8CcO+CDiKHhEReRKGfRHYHoaTxp49ERF5AIZ9EdgG1rnLgXWIiMgDMOyLgBfoERGRJ2HYF0FggA8E8AI9IiLyDAz7IlAqFAjUqNmzJyIij8CwLyKdRo00hj0REXkAhn0R6bRqGE0WZJvMUpdCRERUKIZ9EdkH1mHvnoiIZI5hX0TBWt5+R0REnoFhX0RB7NkTEZGHYNgX0f2ePW+/IyIieWPYFxHP2RMRkadg2BeRfRQ9nrMnIiKZY9gXka1nn2bgYXwiIpI3hn0R+alV8FUrkc6ePRERyRzDvhg4ih4REXkChn0xBGvUyMg0wWoVpS6FiIjooRj2xRCk9YUoAumZ7N0TEZF8MeyLIVjDK/KJiEj+GPbFYL/9jlfkExGRjDHsi0Gn4fj4REQkfwz7YgjW2u61Z9gTEZF8MeyLwf4wHI6PT0REMsawLwb7w3DYsyciIhlj2BeDNsAHCkHgOXsiIpI1hn0xKAQBQRofXo1PRESyxrAvJp3GF3f1JogiR9EjIiJ5YtgXk06rhslsRbbJInUpREREBWLYF5P9Ube8Ip+IiGSKYV9MuntX5KfzinwiIpIphn0x3e/ZM+yJiEieGPbFFGwfH59hT0RE8qSSuoBFixZh586dyMnJwaBBgxAREYFJkyZBEATUrFkTU6ZMgUKhQFxcHFavXg2VSoVRo0ahY8eOUpcOIO/4+DxnT0RE8iRpzz4+Ph5Hjx7FqlWrEBsbi+vXr2PWrFkYM2YMVq5cCVEUsWPHDqSkpCA2NharV6/GkiVLMG/ePJhM8uhJ69izJyIimZM07Pfu3YtatWph9OjRGDlyJDp06ICEhAREREQAANq1a4f9+/fjxIkTaNy4MdRqNQIDAxEWFobExEQpS7fTcXx8IiKSOUkP46empuLvv//Gl19+iatXr2LUqFEQRRGCIAAANBoNMjIyoNfrERgYaH+fRqOBXq8vdN0hIQFQqZQurTc0NLDA6Ro/FQxGy0Pny5En1foobIv8eEs7ALZFjrylHYD72iJp2AcHB6NatWpQq9WoVq0afH19cf36dft8g8GAoKAgaLVaGAwGh+l5w78gqamZLq01NDQQKSkZBc4LDFDjVlrWQ+fLTWFt8TRsi/x4SzsAtkWOvKUdgOvbUtiOg6SH8Zs2bYrffvsNoijixo0byMrKQsuWLREfHw8A2LNnD5o1a4bw8HAcOXIERqMRGRkZSEpKQq1ataQs3UGwVg19Vg7MFqvUpRAREeUjac++Y8eOOHToEPr37w9RFDF58mRUrFgRH3zwAebNm4dq1aqhW7duUCqVGDJkCGJiYiCKIsaOHQtfX18pS3dge659usGEMkF+EldDRETkSPJb7yZMmJBv2vLly/NNi46ORnR0tDtKclre59oz7ImISG44qI4L2G+/4yh6REQkQwx7F7Dffsfn2hMRkQwx7F3A9jAc9uyJiEiOGPYucL9nz7AnIiL5Ydi7gO0CPT7TnoiI5Ihh7wIaPxWUCoE9eyIikiWGvQsIggCdVs1z9kREJEsMexfRadS4azBCFEWpSyEiInLAsHcRncYXZouITKNZ6lKIiIgcMOxdxDawThoP5RMRkcww7F3EdvtdOq/IJyIimWHYu4htYJ00XpFPREQyw7B3kWANx8cnIiJ5Yti7SJCW4+MTEZE8MexdJFhz/zG3REREcsKwd5EgHsYnIiKZYti7iI9KAY2fij17IiKSHYa9C+m0vrjLW++IiEhmGPYupNOoYcg2I8dslboUIiIiO4a9C+l4RT4REckQw96FeEU+ERHJEcPehew9e16RT0REMlIiYZ+VlVUSq5U92/j47NkTEZGcOBX2nTt3xrJlywpd5rPPPkOnTp2KVZSnso2PzyvyiYhITlSFzbx69Sr0er39dXJyMi5evIjExMQCl8/JycGBAwfYs2fPnoiIZKTQsD9+/DjGjRsHQRAAAIIgYM2aNVizZs1D3yOKIlq3bu3aKj1EMM/ZExGRDBUa9j179sTp06dx584diKKIjRs3ok6dOqhbt26By/v4+KBcuXIYPHhwiRQrd/6+KqiUCqTxMD4REclIoWEPAOPHj7f/fPDgQURFRWHo0KElWpSnEgQBwVo1D+MTEZGsPDLs89q5c2dJ1eE1dBo1Ll/PgFUUobh3+oOIiEhKToW9zcaNG7F582YkJiYiIyMDISEheOaZZ/DCCy+gffv2rq7Ro+i0vrBY02HIykFggFrqcoiIiJwLe5PJhNdffx2///47RFGETqdDWFgY0tPT8fPPP+OXX35B3759MWvWrJKqV/Z0eR51y7AnIiI5cCrsFy5ciAMHDqBr166YMGECKlWqZJ+XnJyMDz/8EBs3bkT9+vXx0ksvubxYT3B/fHwTKkpcCxEREeDkoDqbN29GvXr1sGDBAoegB4AKFSpg/vz5qFmzJlavXu3SIj2JrWfPK/KJiEgunAr7W7duoUWLFlAoCn6bSqVCy5YtcfXqVZcU54lso+il84p8IiKSCafCvmbNmjh16lShy1y4cAFVqlQpTk0e7X7PnmFPRETy4FTYjx8/HkePHsX06dORkZGRb/5XX32FgwcP4r333nNZgZ4m2DY+Pp9pT0REMuHUBXrr1q1DlSpVsGLFCmzYsAG1a9dG+fLlkZ2djYSEBKSkpECj0eS7Gl8QBKxfv96lhctVYIAPBHDIXCIikg+nwv6HH36w/2wwGPDHH3/kW0av1+PMmTMO04RSNLiMSqmANsCHo+gREZFsOBX2D3vaHTnSadS4nZ4tdRlEREQAnDxnT49Hp/VFltECY45F6lKIiIiKNlzu4cOHsW7dOpw9exZZWVkIDg5GzZo18fzzz6NZs2aurtHjBOd5rn25YH+JqyEiotLO6bD/+OOP8fXXX0MURQCAv78/Ll++jKNHj+L777/Ha6+9hrFjx7q8UE8SdG8UvXQ9w56IiKTn1GH8H3/8EV999RVq1KiBRYsW4fDhwzh69CiOHz+OpUuXonbt2li8eDG2b99eUvV6hGBN7u13HEWPiIjkwKmwX7ZsGUJDQ7Fs2TK0b98eWq0WAKBWq9GqVSssXboUZcuWRWxsbIkU6ynyjo9PREQkNafC/uzZs+jYsSNCQkIKnF+mTBl07Ngx3613pY39yXccWIeIiGSgRK7Gz8nJKYnVegzb+PgcWIeIiOTAqbCvXbs2du3ahbS0tALn37lzBzt37kTt2rVdUZvH0ml4GJ+IiOTDqbAfOnQoUlJSMGLECBw8eBBmsxlA7qh5u3fvxssvv4zbt2+X2mfZ2/iplVD7KHiBHhERyYJTt95FRkbi5MmT+OabbzBs2DAoFAqo1WpkZ+eOFieKIl555RX06tWrRIr1FIIgIFjjy549ERHJgtP32U+cOBGdO3fG+vXrkZiYCIPBAI1Ggzp16iAqKoqD6twTpFUjKfkurFYRCkXpeTYAERHJT5FG0GvWrBlD/RGCNWqIIpCRlWM/h09ERCSFIoW9Xq/H9u3bkZiYiMzMTOh0OjRo0AAdO3aEWs1gAwCdxnZFvpFhT0REknI67NesWYPZs2cjKyvLPmQukHueukyZMpg5cybat2/v0iI9EQfWISIiuXAq7Ldt24YpU6agbNmyGDlyJMLDw6HRaHDz5k0cPXoUq1atwujRoxEbG4vGjRuXVM0ewdab5xX5REQkNafC/uuvv0ZISAji4uLw9NNPO8zr3Lkz+vXrhwEDBuCTTz7Bd99959JCPY1tYJ109uyJiEhiTt1nf+7cOXTr1i1f0NtUr14dzz33HE6cOOGS4jzZ/Z49w56IiKTlVNiHhITAarUWuoy/vz80Gs1jr/P27dto3749kpKScOXKFQwaNAgxMTGYMmWK/bPi4uIQFRWF6Oho7Nq1y5mSJRPMc/ZERCQTToV9dHQ0Nm/e/NAH3fz555/YsmULoqKiHmt9OTk5mDx5Mvz8/AAAs2bNwpgxY7By5UqIoogdO3YgJSUFsbGxWL16NZYsWYJ58+bBZJJ/gAYGqCEIuVfjExERSanQc/bLli1zeB0QEACNRoMXX3wRffr0QePGjVG2bFmkp6fj5MmT2LRpE0JCQlCvXr3H+vDZs2dj4MCBWLx4MQAgISEBERERAIB27dph3759UCgUaNy4MdRqNdRqNcLCwpCYmIjw8PCitNdtFAoBQQFq9uyJiEhyhYb9zJkzIQiC/Ra7vD+vW7cO69atgyDkjg5nm56eno6xY8eie/fuhX7w+vXrUaZMGbRt29Ye9qIo2ten0WiQkZEBvV6PwMBA+/s0Gg30en1R2up2Oo0aN1KzpC6DiIhKuULDftasWSX2wbYdhQMHDuDMmTOYOHEi7ty5Y59vMBgQFBQErVYLg8HgMD1v+D9MSEgAVCqlS2sODX305zosXyYAf97UQxvkD3/fIo1fVGKcbYucsS3y4y3tANgWOfKWdgDua0uhCdSvX78S++AVK1bYfx4yZAimTp2KuXPnIj4+Hs2bN8eePXvQokULhIeH45NPPoHRaITJZEJSUhJq1ar1yPWnpma6tN7Q0ECkpGQ49R5/de7ORtLl2yhfJsCl9RRHUdoiV2yL/HhLOwC2RY68pR2A69tS2I6DrLqbEydOxAcffIB58+ahWrVq6NatG5RKJYYMGYKYmBiIooixY8fC19dX6lIfS97n2ssp7ImIqHRxKuxtF889iiAIiI+Pf+z1xsbG2n9evnx5vvnR0dGIjo5+7PXJRfC9gXU4ih4REUnJqbDXarUFTs/OzkZaWhqsVitq1aqFSpUquaQ4T5e3Z09ERCQVp8J+586dD52XkZGBL774AuvWrcP8+fOLXZg3sD8Mh6PoERGRhJwaVKcwgYGBmDBhAmrUqIG5c+e6arUe7X7PnofxiYhIOi4Le5vGjRvj8OHDrl6tR7r/THv27ImISDouD/szZ87YB8Yp7XzVSviplXwYDhERScqpc/Y7duwocLooisjMzMSvv/6K/fv3o2vXri4pzhvotL5I52F8IiKSkFNhP3r06EJ77aIooly5cnjnnXeKXZi30GnUuHknExarFUqFyw+kEBERPZLLwl6tVqNatWpo3749fHx8XFKcNwjWqiECSDfkICTQMwYDIiIi7+JU2L/11lv5phmNRly/fh1ly5Z16jn2pUVQnivyGfZERCSFxzquvHPnTrz77rtITEx0mP7xxx+jRYsW6N69OyIiIjBmzBikpqaWSKGeyjaKHq/IJyIiqTyyZz958mR8//33AIAOHTqgTp06AIB58+bhq6++giAIaNWqFQDg559/xoULF7B+/Xqo1eoSLNtzcBQ9IiKSWqE9+507dyIuLg5169bF119/jQ4dOgAAbty4gaVLl0IQBEybNg1LlizBkiVL8Omnn+LChQtYtmyZO2r3CPdH0eMV+UREJI1Cw37t2rUIDg7GsmXL0Lp1a/vT5rZu3Qqz2YywsDD079/fvnznzp3RpEkTbN26tWSr9iC2gXXS2LMnIiKJFBr2J06cQIcOHfI9AGf//v0QBAGdOnXK955nnnkGV65ccW2VHszWs0/nOXsiIpJIoWF/9+5dlC9f3mGa1WrFkSNHAAAtW7bM9x6VSoWcnBwXlujZtP4+UCoEpHFgHSIikkihYR8YGJjv6voTJ05Ar9dDpVLh2Wefzfeey5cvIyQkxLVVejCFICBIo+bV+EREJJlCw75hw4bYv38/rFarfdqWLVsA5Pbq/f39HZZPSUnB3r170bBhwxIo1XPpNGrcNZggiqLUpRARUSlUaNhHR0fj6tWr+Ne//oVDhw5hxYoVWLNmDQRBwODBgx2WvXPnDsaMGYPs7Gw8//zzJVq0p9Fp1MgxW5FltEhdChERlUKF3mffuXNnDB48GCtWrMC2bdsA5I5/HxMTg/bt29uXGzlyJA4cOACj0Yju3bujS5cuJVu1h9HZBtYxGBHg59SghURERMX2yOT54IMP0K1bN+zatQtmsxmtW7e2329vc/HiRWg0Grz22msYNWpUSdXqsewD6+hNeOoJDilMRETu9VjdzIiICERERDx0/vr16/Pdnkf3Bd+7/Y5X5BMRkRRc8sxVBn3hgu4NrMN77YmISAp8wLob3O/ZM+yJiMj9GPZukPecPRERkbsx7N3A/jAcnrMnIiIJMOzdwEelRICvio+5JSIiSTDs3USn5ZC5REQkDYa9m+g0auizcmC2WB+9MBERkQsx7N0k+N4oeuk8lE9ERG7GsHeToHtX5KfxUD4REbkZw95NgvOMj09ERORODHs34b32REQkFYa9m9y/155hT0RE7sWwd5P7PXsexiciIvdi2LvJ/Wfas2dPRETuxbB3E42fCiqlwKvxiYjI7Rj2biIIAnQaNdJ5NT4REbkZw96NdFpf3DWYIIqi1KUQEVEpwrB3I51GDbNFhCHbLHUpRERUijDs3ch+kR6vyCciIjdi2LuR/fY7XpFPRERuxLB3I/vAOrwin4iI3Ihh70bs2RMRkRQY9m5kexhOGs/ZExGRGzHs3cjWs+cz7YmIyJ0Y9m50/5n27NkTEZH7MOzdSKVUQOvvw3P2RETkVgx7N9Np1bwan4iI3Iph72Y6jRqZRjNMORapSyEiolKCYe9mOk3uFfm8SI+IiNyFYe9mtoF10hj2RETkJgx7NwvWcBQ9IiJyL4a9mwXZhszlc+2JiMhNGPZuFqyxPfmOPXsiInIPhr2b6dizJyIiN2PYu5mOPXsiInIzhr2b+fsq4aNS8Gp8IiJyG5VUH5yTk4P33nsPycnJMJlMGDVqFGrUqIFJkyZBEATUrFkTU6ZMgUKhQFxcHFavXg2VSoVRo0ahY8eOUpVdbIIgQKdR8z57IiJyG8nC/ocffkBwcDDmzp2L1NRU9OvXD3Xq1MGYMWPQvHlzTJ48GTt27ECjRo0QGxuLdevWwWg0IiYmBq1bt4ZarZaq9GIL1vri0rV0WEURCkGQuhwiIvJykoV99+7d0a1bN/trpVKJhIQEREREAADatWuHffv2QaFQoHHjxlCr1VCr1QgLC0NiYiLCw8OlKr3YdBo1LFYR+qwcBAV47k4LERF5BsnCXqPRAAD0ej3efvttjBkzBrNnz4Zwr6er0WiQkZEBvV6PwMBAh/fp9fpHrj8kJAAqldKlNYeGBj56ocfwZKgWOJcChY/KZet0llSfWxLYFvnxlnYAbIsceUs7APe1RbKwB4Br165h9OjRiImJQe/evTF37lz7PIPBgKCgIGi1WhgMBofpecP/YVJTM11aa2hoIFJSMlyyLvW9yyIvX02F1sf910i6si1SY1vkx1vaAbAtcuQt7QBc35bCdhwkuxr/1q1bGD58OMaPH4/+/fsDAOrVq4f4+HgAwJ49e9CsWTOEh4fjyJEjMBqNyMjIQFJSEmrVqiVV2S6h0/L2OyIich/JevZffvkl0tPTsXDhQixcuBAA8P7772P69OmYN28eqlWrhm7dukGpVGLIkCGIiYmBKIoYO3YsfH19pSrbJXT3xsdP03NgHSIiKnmCKIqi1EWUBFcf5nHl4ZZbaVmYtOh3hASq8Z/hEQjw83HJeh8XD4PJk7e0xVvaAbAtcuQt7QBKyWH80qxssD96t66C2+lGLP/lnNTlEBGRl2PYS6RXq8qo9nQQfk+4gd9PX5e6HCIi8mIMe4koFQq82rsefH2UiN12DrfvZktdEhEReSmGvYTKhwQgpktNZBnN+HrLaVitXnn5BBERSYxhL7E24U+hSa1QnP0rDdsO/il1OURE5IUY9hITBAHDuteGTqvG+j0XceW6d1xlSkRE8sGwl4HAADVGRNaFxSpi8eYEGHMsUpdERERehGEvEw2qPYEuTSvi2u1MrN2VJHU5RETkRRj2MtK/Q3VUKKvBjj+u4kTSbanLISIiL8GwlxG1jxKv9q4HlVLA0h/PID2TY+cTEVHxMexlJqx8IKLaVUe6wYTvfkqEl45mTEREbsSwl6HnIiqhTlgwjp6/hT3H/5a6HCIi8nAMexlSCAL+0aseAnxVWLXjPG7cyZS6JCIi8mAMe5kqE+SHod1rw5RjxeLNp2G2WKUuiYiIPBTDXsYi6pZHy/pP4tK1dGzed1nqcoiIyEMx7GVucNdaeCLID1sOXMaFq3elLoeIiDwQw17mAvxUeLV3PQDA4s0JyDKaJa6IiIg8DcPeA9SqFIzIFpVx6242Vm4/J3U5RETkYRj2HqJPm6qo/GQg9p28jsOJN6Uuh4iIPAjD3kOolAq81rse1CoFvtuaiNQMo9QlERGRh2DYe5CnntBgQOeaMGSbseT/TsPK0fWIiOgxMOw9TIdGTyO8+hM4fTkV2w9flbocIiLyAAx7DyMIAl6JrIvAAB+s/TUJV2/qpS6JiIhkjmHvgXQaNV6JrAuzxYrFmxOQY7ZIXRIREckYw95DNapRFh0aV8DVFAPW7b4odTlERCRjDHsPNqBjDZQvE4CfD/2FhMt3pC6HiIhkimHvwXzVSrzWux6UCgFLtpyGPitH6pKIiEiGVFIXQMVT9akg9GlTFev3XMR7i39HgJ8KSoUAhUKAUhAgKASH1wqFAH8/H5jNlvvTH5hve6144HVByxe0/geXUwgFTLP9LDww7RGfqRAECIIg9X87EZFHYdh7gcgWlXEjNROJV1JhyrHAahVhsYqwiiKsVuT+fO+1NxAE2MNfpVRAQO5r4TF2IISH7NQUtFORb2dDeGBHRSFAIQBKhSLftELX/5CdqgyTFXfvZuZbXsjz2kepgK9aKfUmICIPw7D3AgqFgBE96z1yOVHMDfwyZbS4cTP9/k5B3n/F+68tlns7DPemFbi89f78gtbx4PusYuGfaS1ofQUsL1pFmK0ilAoFjDnmfO/PybHaP/vBz/TkfR4BQK9WVdCvXTWpSyEiD8KwL0UEIbdXq/ZRwk/tHZs+NDQQKSkZTr1HFB+yQyECFovVcZ4Ixx2WAt/38J2aB3dgHrYeqyjC19cHeoPx3mda88y7f3Tm8vV0bN5/GZXKadGsTrkS+l8lIm/jHX/xiZxg2+lRKgAfqYvJ43F2XJJvGTD9u8NY8uMZVAjV4KknNG6qjog8Ga/GJ/IgFcpq8EpkHRhNFny+4RSyTWapSyIiD8CwJ/IwEXXLo0uzivj7lgHf/pQI0ZMvQiAit2DYE3mg6I41UKOiDgfP3MSOI3wgEhEVjmFP5IFUSgVG9WmAoAAfrNl5AReu3pW6JCKSMYY9kYcKCfTFyD4NIIrAwo0ncddgkrokIpIphj2RB6tTOQQvdKiGNL0JizadgsVqlbokIpIhhj2Rh+seEYYmtUKR+Gca1u/hExCJKD+GPZGHEwQBwyPronyIP376/U/8cS5F6pKISGYY9kReIMBPhdH9GkKtUmDJ/53GjTuZUpdERDLCsCfyEhXLaTGsRx1kGS34bMNJGE0WqUsiIplg2BN5kZb1n0SnJhWQnGLAsm0ccIeIcjHsibzMgE41Ue3pIBxIuIFdR5OlLoeIZIBhT+RlfFQKvNG3AbT+Pli1/TySkjngDlFpx7An8kJlgvwwsk99WEURCzeeQnomB9whKs0Y9kReql6VMujXthpSM4xY/EMCrFaevycqrRj2RF4ssmVlNKpRFqcvp2LDbxxwh6i0YtgTeTGFIOAfveoiNNgP/3fgCo6e54A7RKURw57IywX4+WB0v4bwUSnw9ZYzuJnKAXeIShuGPVEpEFY+EEO71UaW0YzPN5yCMYcD7hCVJgx7olKidcOn0L7R0/jrph7Lt53lgDtEpQjDnqgUielSE1WeDMS+U9ex+/jfUpdDRG7CsCcqRXxUSrzRrwE0fiqs/OUcLl1Ll7okInIDhj1RKVNW54/Xn68Pi0XEwg0nkcEBd4i8nkrqAojI/RpUewJ92lTFxr2XMPHLA1D7KF22boVCcPkAPkqFAB+lAiqVAirlvZ+VuT+r7k23T1Plma5UwEcp3JuWu4wy7/vt77u/fO603Nc+fmpkZpvhoxKgVCqgEASXtovIXTwm7K1WK6ZOnYqzZ89CrVZj+vTpqFy5stRlEXmsXq2rICMzBwmX77h0vUqlAhaL1WXrEwFYLFaYzBZkGs3IsVhhNlthkWBEQKVCuL+TkXcHo6Cdjrw7HKo8OxP35vmoHF/nLlfQex82Lfe1UiFA4E4IPYLHhP327dthMpmwZs0aHDt2DB9++CG++OILqcsi8lgKQcDg52q5fL2hoYFISclw+XofZBVFWCxWmC2ifQfAbLEixyLafzYXON8Ki23avek5ltx15dxb3rasUqWEIdPk8H6zRbSvx2yxIjM7x+H97r7JQQAKOJqR/whHgL8PRIvVvkPy4BEO+05IAUdI7k8T7r03zxER+/ruHxFRqXgURG48JuyPHDmCtm3bAgAaNWqEU6dOSVwREUlJIQhQqJTwUQH+JfQZRdlxsVitMJtFmK22HYk8OxD35uXd0TBbxTzL3X+P4w5JAet7YGfl/s5N7jKZRrPDDo+7KRWC4w5FgTsQ906xKHJ/xmPsIPj6qmA0mt3QgpJXq3IZdGn8tFs+y2PCXq/XQ6vV2l8rlUqYzWaoVAU3ISQkACqV685DArlffG/BtsiTt7TFW9oBeEdbRFHMPcJhtiDHdoTDbM33s21+wfOsyLHce/+9nQ77zw9Zl8NrixXGHAsMWfffW9ofznT5Wjqiu9SCUlHyR0E8Juy1Wi0MBoP9tdVqfWjQA0Cqi4cEddehSXdgW+TJW9riLe0AvK8taan3/4YqASgVABQKwEeaG7OsVvHeUYzHP/rwxBNa3L6tL+HK3COsYjDuuLAthe2YekzYN2nSBLt27UJkZCSOHTuGWrVcf66RiIjcR6EQ4KtQAk7cDRIc6IucbO+4XdTHxUefC+MxYd+1a1fs27cPAwcOhCiKmDlzptQlEREReQSPCXuFQoH//ve/UpdBRETkcTiCHhERkZdj2BMREXk5hj0REZGXY9gTERF5OYY9ERGRl2PYExEReTmGPRERkZdj2BMREXk5hj0REZGXE0TR3U9fJiIiIndiz56IiMjLMeyJiIi8HMOeiIjIyzHsiYiIvBzDnoiIyMsx7ImIiLycSuoC5MZqtWLq1Kk4e/Ys1Go1pk+fjsqVK9vn79y5E59//jlUKhVeeOEFREdHS1jtw+Xk5OC9995DcnIyTCYTRo0ahc6dO9vnf/PNN1i7di3KlCkDAPjPf/6DatWqSVXuI/Xt2xeBgYEAgIoVK2LWrFn2eZ6yTQBg/fr12LBhAwDAaDTizJkz2LdvH4KCggB4znY5fvw4PvroI8TGxuLKlSuYNGkSBEFAzZo1MWXKFCgU9/sRj/pOSSlvO86cOYNp06ZBqVRCrVZj9uzZKFu2rMPyhf0eSi1vWxISEjBy5EhUqVIFADBo0CBERkbal5XzNgEc2zJ27FjcunULAJCcnIxnnnkG8+fPd1hejtuloL/BNWrUkO67IpKDbdu2iRMnThRFURSPHj0qjhw50j7PZDKJXbp0EdPS0kSj0ShGRUWJN2/elKrUQq1du1acPn26KIqieOfOHbF9+/YO88eNGyeePHlSgsqcl52dLfbp06fAeZ60TR40depUcfXq1Q7TPGG7LF68WOzVq5f44osviqIoiq+//rr4+++/i6Ioih988IH4888/Oyxf2HdKSg+2Y/DgweLp06dFURTFVatWiTNnznRYvrDfQ6k92Ja4uDhxyZIlD11erttEFPO3xSYtLU18/vnnxRs3bjhMl+t2KehvsJTfFR7Gf8CRI0fQtm1bAECjRo1w6tQp+7ykpCSEhYVBp9NBrVajadOmOHz4sFSlFqp79+745z//aX+tVCod5ickJGDx4sUYNGgQFi1a5O7ynJKYmIisrCwMHz4cQ4cOxbFjx+zzPGmb5HXy5ElcuHABAwYMcJjuCdslLCwMn376qf11QkICIiIiAADt2rXD/v37HZYv7DslpQfbMW/ePNStWxcAYLFY4Ovr67B8Yb+HUnuwLadOncKvv/6KwYMH47333oNer3dYXq7bBMjfFptPP/0UL730EsqVK+cwXa7bpaC/wVJ+Vxj2D9Dr9dBqtfbXSqUSZrPZPs92qAgANBpNvi+RXGg0Gmi1Wuj1erz99tsYM2aMw/yePXti6tSp+O6773DkyBHs2rVLmkIfg5+fH0aMGIElS5bgP//5D9555x2P3CZ5LVq0CKNHj8433RO2S7du3aBS3T8DKIoiBEEAkPv/n5GR4bB8Yd8pKT3YDluI/PHHH1i+fDlefvllh+UL+z2U2oNtCQ8Px4QJE7BixQpUqlQJn3/+ucPyct0mQP62AMDt27dx4MABREVF5VtertuloL/BUn5XGPYP0Gq1MBgM9tdWq9X+i/fgPIPB4BA0cnPt2jUMHToUffr0Qe/eve3TRVHEsGHDUKZMGajVarRv3x6nT5+WsNLCVa1aFc8//zwEQUDVqlURHByMlJQUAJ63TQAgPT0dFy9eRIsWLRyme9p2scl7ztFgMNivP7Ap7DslNz/++COmTJmCxYsX26+bsCns91BuunbtigYNGth/fvD3yJO2CQBs3boVvXr1yneEEpD3dnnwb7CU3xWG/QOaNGmCPXv2AACOHTuGWrVq2edVr14dV65cQVpaGkwmEw4fPozGjRtLVWqhbt26heHDh2P8+PHo37+/wzy9Xo9evXrBYDBAFEXEx8fb/zDI0dq1a/Hhhx8CAG7cuAG9Xo/Q0FAAnrVNbA4dOoRWrVrlm+5p28WmXr16iI+PBwDs2bMHzZo1c5hf2HdKTjZt2oTly5cjNjYWlSpVyje/sN9DuRkxYgROnDgBADhw4ADq16/vMN9TtonNgQMH0K5duwLnyXW7FPQ3WMrvinx35STStWtX7Nu3DwMHDoQoipg5cyY2b96MzMxMDBgwAJMmTcKIESMgiiJeeOEFlC9fXuqSC/Tll18iPT0dCxcuxMKFCwEAL774IrKysjBgwACMHTsWQ4cOhVqtRsuWLdG+fXuJK364/v37491338WgQYMgCAJmzpyJn376yeO2ic2lS5dQsWJF++u8v1+etF1sJk6ciA8++ADz5s1DtWrV0K1bNwDAhAkTMGbMmAK/U3JjsVgwY8YMPPXUU3jrrbcAAM8++yzefvttezsK+j2Ua2946tSpmDZtGnx8fFC2bFlMmzYNgGdtk7wuXbqUbwdM7tuloL/B77//PqZPny7Jd4VPvSMiIvJyPIxPRETk5Rj2REREXo5hT0RE5OUY9kRERF6OYU9EROTlGPZEXuLq1auoXbs2hgwZYp9mNBqxdOlSCau6b+/evfZ7vwEgPj4etWvXxowZMySsiqh0YNgTebGXXnrJfo+vlFauXIkRI0bg5s2b9mkVKlTAm2++aR8LnIhKjvQjDxBRibl9+7bUJQAouI6KFSvaB7AhopLFnj0REZGXY9gTeSHb+fvk5GRkZGSgdu3amDRpkn2+Xq/HRx99hC5duqBBgwZo27YtpkyZkq8HPmnSJNSuXRsnTpxAZGQkGjZsaB/KEwB27dqFf/zjH2jRogXq16+PFi1a4I033sCZM2fs6xgyZAg+++wzAMDo0aNRu3ZtAA8/Z3/p0iW88847aNWqFRo0aIAuXbpgzpw5+Z4QZqvt7t27mDJlClq3bo2GDRsiKioK27Ztc91/JpEX4GF8Ii8UFBSEN998E9999x2MRiNee+01+7PaMzIyEBMTg3PnzqFly5Z47rnncPXqVcTFxeG3337D6tWr8z0zfNSoUWjYsCFat26NgIAACIKA5cuXY9q0aQgLC0OvXr3g4+ODkydPYseOHfj999+xdetWlCtXDv369QMAHDx4EJGRkahWrdpD6z5+/DhefvllZGdno2PHjqhUqRKOHTuGJUuWYNeuXVi1ahWCg4Md3vPKK68gLS0NPXr0QGZmJjZv3ox//vOfWL58eb4HjRCVVgx7Ii8UFBSEt956Cxs2bEB6errDufF58+bh3LlzmDx5MgYPHmyfvmPHDrzxxhuYMWMGFixY4LC+Jk2a4NNPP7W/NplMmD9/PqpUqYINGzYgICDAPm/q1KlYtWoVdu3ahQEDBiAqKgrJyck4ePAgevbsiS5duhRYs8ViwYQJE2AymbBo0SKHp5x99NFH+OqrrzBnzpx8DwdRKpXYsmWLvYaWLVvinXfeQVxcHMOe6B4exicqRcxmMzZu3IiaNWs6BD0AdO7cGU2aNMEvv/wCvV7vMM/2dC4bi8WCadOmYcaMGQ5BDwAREREAnL848OjRo7h8+TJ69uyZ73Gmb7/9NsqXL4/NmzfDZDI5zBs8eLBDDbYnBV6+fNmpzyfyZuzZE5Uily5dQmZmJiwWi0NP3cZoNMJiseDs2bNo2rSpfXqFChUclvP390dkZKR9nUlJSfjzzz9x/vx5HDhwAABgtVqdqs12nv/ZZ5/NN0+tVqNhw4bYvn07Ll68iDp16tjnVa1a1WHZwMBAAMi3U0BUmjHsiUqR9PR0AMDFixftF80V5O7duw6v/fz88i1z6NAhzJo1CwkJCQAAX19f1KlTB/Xr18e1a9fg7NOzbUcTtFptgfNt1xFkZWU5TFer1Q6vBUEAAKc/n8ibMeyJShGNRgMA6NOnD+bMmVPk9SQnJ+PVV1+Fr68vpk2bhqZNm6JKlSpQKpX48ccfsX379iLXlnfgnbxsOyoPXqBHRI/Gc/ZEpUjVqlWhVquRkJBQYM/322+/xcKFC5GamlroerZv346srCy8/fbbiI6ORvXq1aFUKgEASUlJABx71rbedmFsdwscOXIk3zyr1YojR44gICAg3ykFIno0hj2RF/Px8YHZbLa/9vX1RWRkJC5cuIBvvvnGYdn4+HjMmTMH69atg06nK3S9vr6+AIBbt245TE9MTMSyZcsAwOFzVarcg4iFnUdv2rQpKleujJ9//hm7d+92mPe///0P165dQ48ePfIdtieiR+NhfCIvVq5cOVy+fBnvvPMO2rRpg759+2LixIk4evQoZs+ejR07diA8PBw3btzAzz//DJVKhZkzZ0KhKLwf0LFjR3z88cdYtGgRLl68iLCwMFy5cgW7du2yXyCXlpZmX758+fIAgC+++AJnzpzBm2++mW+dCoUCH374IUaMGIGRI0eiY8eOCAsLw9GjR3Hs2DFUr14dEyZMcN1/DlEpwp49kRcbP348atasia1bt2LTpk0AgDJlyiAuLg7Dhw/HjRs3EBsbi8OHD6NTp06Ii4tD8+bNH7ne8uXL45tvvkGLFi3w+++/Y+XKlbh06RKGDBmCn376CcHBwfjtt9/sh/IjIyPRo0cP/PXXX1i5ciWSk5MLXG+TJk2wdu1aREZG4ujRo1ixYgXS0tIwatQofP/99zxfT1REgshLVomIiLwae/ZERERejmFPRETk5Rj2REREXo5hT0RE5OUY9kRERF6OYU9EROTlGPZERERejmFPRETk5Rj2REREXo5hT0RE5OX+H0Kq8ienyjtrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x396 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_lines([compute_subopt(values, solver.v)], [\"Subopt of $V^{\\pi_t}$\"], [\"Iteration\", \"Subopt\"], \"figs\", \"VI.pdf\", show = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a424696f",
   "metadata": {},
   "source": [
    "To better appreciate the difference in the convergence plot use log scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bdd6b924",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:49.747688Z",
     "start_time": "2023-03-14T00:56:49.240768Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAFnCAYAAAChL+DqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABLv0lEQVR4nO3dd3xUZd428Gtqykx6ISEVUukt9GTpImABROAVsbGiq+iuC4LuuuojgoioD+rCgyi4iIUgnUVBBFGK9CQEQgohgYSQQuqkTaa8f0wyyZACk0yYdn33wycz55y55wduznWf+9znHIFWq9WCiIiIbJbQ3AUQERFR52LYExER2TiGPRERkY1j2BMREdk4hj0REZGNY9gTERHZOLG5C+gshYUVJm3Pw8MZJSVVJm2TiMgScP9mO3x8XFpcbjVhn5ycjI0bN0IsFuPVV1+Ft7f3Pf1+sVh0T7+PiOhe4f7N9lnNMH5tbS3eeustjBo1CgkJCeYuh4iIyGpYTdgPGjQIGRkZ2LBhA3r06GHucoiIiKyG1YR9UlISevfujfXr12Pz5s3mLoeIiMhqWETYJyYmYu7cuQAAjUaDN998E7NmzcLcuXORnZ0NAKisrMQ//vEPvPvuu5g4caI5yyUiIrIqZp+gt379euzevRtOTk4AgIMHD0KpVGLLli1ISEjAihUrsHbtWgwfPhzDhw83c7VERETWx+xhHxwcjE8//RSLFy8GAJw9exZxcXEAgP79+yM5Obld7Xp4OJt8hmlrlzQQEVk77t9sm9nDfuLEicjJydG/VygUkMvl+vcikQgqlQpisXGlmvqaUR8fF5Nfu09EZAm4f7MdrXXaLOKcfVNyuRyVlZX69xqNxuigJyIiokYWF/YDBw7Eb7/9BgBISEhAZGSkmSsiIiKybhZ3yDxhwgQcO3YMs2fPhlarxfLly81dEhERkVUTaLVarbmL6AymPv/Ec1pEZKu4f7MdVnPOnoiIiEyLYU9ERGTjGPZEREQ2jmFPRERk4xj2RERENo5hT0REZOMY9kRERDaOYU9ERGRin332v/j7318ydxl6DHsiIiITu3z5Enr27GXuMvR4B727xDtMEZGlOHr0CHbv3oFLly6islIBV1dX9OjRCw888DBiY0cZ3V7D/m3BgvlISDiHH388DBcX8z3yNiXlIioqKjBkyDCTt71v3x5s2rQB+fk3IZe7YM2aLxAUFGyy9lUqFcaPj4VKpdIvCw3tjs2b4032HW3hHfSIiGzAxx+vxGuvLcTVq5mIixuFWbPmICZmKJKSEvHaawvx/vvLzF1ihxw/fhTPPfc0srIyTd52VtZVrFixFKWlpZg2bQbuu28S/Pz8W9z2559/QmxsDD788P07tvvBB8sRGxuDNWtWQygU4v/+byMA4PPPv8KuXT9hzZovTPr3aA+LexAOERG17Ny5M9i2LR6jR4/F228vN3j8t0KhwMsvP4c9e3ZgxIiRiIsbbb5CO6C0tAQajaZT2k5PT4VGo8H06Y9i/vwX2tw2PFz3xNXMzIw2t0tLu4w9e3bCx8cXTz31LIRCIW7dKoKzsww9evSCQCAwWf0dwSN7IiIrcfz4UQDA9OkzDYIeAORyOZ5/Xjch7MiRw/e8NmtQV1cHAHB3d7/jtsHBIZBKHXDlStth//HHH0Cj0WDBglfg7OwMQNepCA+PsJigBxj2RERWo+E8cGtHm/369cc776zArFmP6ZctWDAfsbExqKgwnHOUl3cDsbExeP31hc3ayc+/iddfX4QJE+IwefI4vP32P5Gbm9Nsu6KiInzwwXJMnz4Fo0cPw/TpU/DBB8tRVFRksN2yZW8jNjYGJSXFWLr0X7j//tGYNGksXn99ITIzrxhst3z5/wAAPvnkI8TGxiAv70ab/yZ3W8OMGQ82a/vLL9e12q5IJEK3bt2hUFSgoCC/xW3279+HCxcSMWjQEIwbN0G/PD09FZGRUW3Wfa8x7ImIrMTgwUMBAP/+92p8/PFKJCcnQa1W69c7ODhi7NjxiIjoWNAsXLgA2dlXMXXqDPTp0xcHD+7H888/g5s3b+q3yc3NwTPPzMGuXdsRHByCRx6ZheDgEOzatR3z5j3eYudg4cKXcfbsGUyZ8hBiYobg+PGjeOGFeUhPTwMAxMWNRlycboLhkCHD8fTTz0Iub32ioDE1zJz5/5q1PWDAoDb/HcLDIwCgxaP7qqoqrF37KcRiMf7+98UG6zIy0hEWFtFm2/caz9kTEVmJkSPjMHXqDOzc+QO2bYvHtm3xkMlk6Nu3PwYPHorRo8fB17dLh7/H3z8Aq1evhYODAwBg585tWLXqPaxb9xneeutdAMDKlctQXHwLS5a8gQcfnKr/7I4dP+DDD1dg5cplWL16rUG75eVl+Oqr7/TD6L/++gveeGMJVq9ehc8++xx/+tNoKBQV+P33Ixg2bDhmznwMbTGmhpkzH4Nc7nLXbQOG5+2HDx9psO6rr75AUVEh5sx5EiEhoQbr1Go1srOzUFRUCAcHR7Ne2dCAYU9EVi3+UAZOXy4wdxltGhzti5ljw03S1qJFr2HEiFhs2xaPs2dPobKyEidOHMOJE8ewZs0nmD37ccyf/wKEwvYP3D733Iv6oAeAqVMfQXz8tzhy5BCUSiVKS0tw9uxp9Os3wCBkAWDatBnYt283zp49jby8G/D376pf9+ST8wzOl48ePQ59+/ZHQsI5FBTkG9VRKSjIb1cNxggL0/03a3qqAQCuXcvG1q3fwde3C5566s/NPjd//gtYu/ZTbNnyDR5+eDoWLXq9Xd9vSgx7IiIrM2JELEaMiEVVVRUSE8/hzJnTOHbsN+TkXMfmzV9Bo9HghRdeblfbAoEAvXv3bba8R49euHYtG9nZWcjP1w3n9+s3oMU2+vTph5SUS8jISDMI2v79B7bYblJSAq5cSTcq7NPSUttVgzFam5H/yScfoq6uDi+99AqcnJyafe6++ybhvvsmtes7OwvDnois2syx4SY7arY2zs7OGD48FsOHx2LBgr9h795dWLlyGbZt24JnnpkPR0dHo9t0cXGFRCJp4btkAIDq6ipUVVUC0F0B0BJvbx8AQE1NjcFyHx/fZtt6eXkB0F06aIz21mAMV1dX+Pp2QXZ2FlQqFcRiMY4e/Q1//HEcgwcPxZgx49vd9r3GCXpERFagslKB2bOnYfHiv7W4XiAQ4MEHp2Lw4KGora3VzyBvuPzr9pulthaClZWKZtsCQFFRIQDA1dVNf4lZw7LbNcz8d3NzN1heW9v8Oxu2vZvL4Zpqbw3GCg+PhFKpRE7OdSiVSnz66UeQSCR45ZXFd/6wBWHYExFZAZlMDoVCgTNnTqG4+Fab2wqFQv0Rc8NRenV1lcE2Lc2WB3STyxqGyBuoVCqkpqbAyckZQUHBCA/XzfZPTExosY2EhHMQCAQIDe1msDwl5VKzbS9evACRSISoqB4AcNfXpre3BmM1nZH/3XdfIzc3B7NnP47g4JAOtXuvMeyJiKzEI4/MhFKpxBtvLGl2HTmgu2f+mTOn8Kc/jYZMphveDg4OBdB4Qx4AqK2txXfffd3q92zc+LnBJX3ffPMfFBTkY9KkKRCJRPDz88PAgTFITU3Bzp0/GHx2z56duHAhEQMHxjQ7B//ll+tQWdk4XP/rr7/g3LkziIsbBVdXNwCASKQ7u9xwA5zWtLcGYzWctz958jg2b/4KXbr44ckn53WoTXPgOXsiIivxxBPP4MqVDPz66y+YPXsqhgwZjqCgYKhUKly6lIwLFxIREhKKhQsbZ39PmfIQduzYitWrV+HixQtwc3PH0aNHIJPJW5xcJpU6IDX1Mp599knExAxGenoaTp8+iZCQUDz7bOMtZl999R948cVnsWrVCvz66yGEhUUgMzMDp0+fhLe3DxYv/meztm/cyMHTT8/BiBFxKCzMx++/H4GPjy8WLPi7fhsfH9259p07t6G8vByPPjpbf/79du2pwVgNR/b79u0BALzxxv+0ay6EufHInojISohEIrz77vtYtuwDDB06HJcvX8LWrd9j796dUCqVeO65BdiwYTM8PDz0n4mIiMQHH6xGdHRPHDp0EAcO/IiYmCFYvXotRCJRs++QSiVYvXot3NzcsG1bPNLT0/Dww9OxZs0XBteLBwUF44svNuHBB6chK+sqtm+PR07OdcyYMRsbN36DgIDAZm0vXfo+oqN74r//3Y2kpETcf/8UfP75V/Dz89Nv07//QEyf/ijKy8uwfXs8rl5t/YE47anBWIGBQfpwHzJkOEaNGtvhNs2Bj7i9S3zELRHZqs7evy1b9jZ+/HEvNm78psN396O28RG3REREdophT0REZOMY9kRERDaO5+zvEs/ZE5Gt4v7NdvCcPRERkZ1i2BMREdk4hj0REZGNY9gTERHZOIY9ERGRjWPYExER2TiGPRERkY1j2BMREdk4hj0REZGNY9gTERHZOIY9ERGRjWPYExER2TiGPRERkY1j2BMREdk4sbkLICIi4xw9egS7d+/ApUsXUVmpgKurK3r06IUHHngYsbGj2t3uggXzkZBwDj/+eBguLi0/KvVeSEm5iIqKCgwZMszkbe/btwebNm1Afv5NyOUuWLPmCwQFBZv0Ox5//FFkZV1tcd3TTz+LefOeM+n33Q0e2RMRWZGPP16J115biKtXMxEXNwqzZs1BTMxQJCUl4rXXFuL995eZu8QOOX78KJ577mlkZWWavO2srKtYsWIpSktLMW3aDNx33yT4+fm3uO3PP/+E2NgYfPjh+3ds94MPliM2NgZr1qwGACxfvqp++Wrs2vUTdu78CY6OjnjttX/h8cefNN1fyAg8sicishLnzp3Btm3xGD16LN5+eznE4sZduEKhwMsvP4c9e3ZgxIiRiIsbbb5CO6C0tAQajaZT2k5PT4VGo8H06Y9i/vwX2tw2PDwSAJCZmdHmdmlpl7Fnz074+PjiqaeeBQCUlBRDIBCgX78BcHZ2Rk7OddTU1KBv3/5wcHBEXt4NZGSk3dP/RjyyJyKyEsePHwUATJ8+0yDoAUAul+P5518CABw5cvie12YN6urqAADu7u533DY4OARSqQOuXGk77D/++ANoNBosWPAKnJ2dAQAZGWno2jVA/z49PRWOjo4IDAwCAJw8eQKZmVc68DcxHsOeiMhKqFQqAK0fbfbr1x/vvLMCs2Y9pl+2YMF8xMbGoKKiwmDbvLwbiI2NweuvL2zWTn7+Tbz++iJMmBCHyZPH4e23/4nc3Jxm2xUVFeGDD5Zj+vQpGD16GKZPn4IPPliOoqIig+2WLXsbsbExKCkpxtKl/8L994/GpElj8frrCw1Cb9myt7F8+f8AAD755CPExsYgL+9Gm/8md1vDjBkPNmv7yy/XtdquSCRCt27doVBUoKAgv8Vt9u/fhwsXEjFo0BCMGzdBvzwjIwNhYRH69+npaejWLQxCoRDnz5/FunX/xo8/7sVTTz0GhULR5t/PVBj2RERWYvDgoQCAf/97NT7+eCWSk5OgVqv16x0cHDF27HhERER16HsWLlyA7OyrmDp1Bvr06YuDB/fj+eefwc2bN/Xb5Obm4Jln5mDXru0IDg7BI4/MQnBwCHbt2o558x5vsXOwcOHLOHv2DKZMeQgxMUNw/PhRvPDCPKSnpwEA4uJGIy5ON8FwyJDhePrpZyGXtz5R0JgaZs78f83aHjBgUJv/DuHhusBu6ei+qqoKa9d+CrFYjL//fbHBuitX0vWfBXRH+g2nBQYMGISwsHB89NFn+OqrbyGXy9uswVR4zp6IyEqMHBmHqVNnYOfOH7BtWzy2bYuHTCZD3779MXjwUIwePQ6+vl06/D3+/gFYvXotHBwcAAA7d27DqlXvYd26z/DWW+8CAFauXIbi4ltYsuQNPPjgVP1nd+z4AR9+uAIrVy7D6tVrDdotLy/DV199px9G//XXX/DGG0uwevUqfPbZ5/jTn0ZDoajA778fwbBhwzFz5mNoizE1zJz5GORyl7tuGzA8bz98+EiDdV999QWKigoxZ86TCAkJ1S/XaDS4evUK5sxpnIiXm5uDXr366N/fvJkHf/+ud/x+U2LYE5FV256xF+cLLpi7jDYN8O2D6eEPmKStRYtew4gRsdi2LR5nz55CZWUlTpw4hhMnjmHNmk8we/bjmD//BQiF7R+4fe65F/VBDwBTpz6C+PhvceTIISiVSpSWluDs2dPo12+AQcgCwLRpM7Bv326cPXsaeXk3DELtySfnGZwvHz16HPr27Y+EhHMoKMg3qqNSUJDfrhqMERYWDgDNzq9fu5aNrVu/g69vFzz11J8N1uXm5qC6utrgyL5793B8880mREZGIywsHJ6eXhAIBO2qqb0Y9kREVmbEiFiMGBGLqqoqJCaew5kzp3Hs2G/IybmOzZu/gkajwQsvvNyutgUCAXr37ttseY8evXDtWjays7OQn68bzu/Xb0CLbfTp0w8pKZeQkZFmELT9+w9ssd2kpARcuZJuVNinpaW2qwZjtDYj/5NPPkRdXR1eeukVODk5GawLCgrG0aNnDJYtXbpC/zo5OQne3j7tqqcjGPZEZNWmhz9gsqNma+Ps7Izhw2MxfHgsFiz4G/bu3YWVK5dh27YteOaZ+XB0dDS6TRcXV0gkkha+SwYAqK6uQlVVJQC0er65IcxqamoMlvv4+Dbb1svLCwCMnqjW3hqM4erqCl/fLsjOzoJKpYJYLMbRo7/hjz+OY/DgoRgzZrzRbYaGdsfNm3l44olZeOutZfrRg87GCXpERFagslKB2bOnYfHiv7W4XiAQ4MEHp2Lw4KGora3VzyBvGC7WarUG27cWgpWVimbbAkBRUSEAwNXVTX9JWcOy2zXM/HdzczdYXlvb/Dsbtr2by+Gaam8NxgoPj4RSqUROznUolUp8+ulHkEgkeOWVxXf+cAvkcjk2bNiMTZu23LOgBxj2RERWQSaTQ6FQ4MyZUyguvtXmtkKhUH/E3HCUXl1dZbBNS7PlAUCtVuuHyBuoVCqkpqbAyckZQUHBCA/XzfZPTExosY2EhHMQCAQIDe1msDwl5VKzbS9evACRSISoqB4AcNfnsttbg7Gazsj/7ruvkZubg9mzH0dwcEiH2r3XGPZERFbikUdmQqlU4o03ljS7jhzQ3TP/zJlT+NOfRkMm0w1vBweHAmi8IQ8A1NbW4rvvvm71ezZu/Nzgkr5vvvkPCgryMWnSFIhEIvj5+WHgwBikpqZg584fDD67Z89OXLiQiIEDY5qdg//yy3WorGwcrv/1119w7twZxMWNgqurGwBAJNKdXW64AU5r2luDsRrO2588eRybN3+FLl388OST8zrUpjnwnD0RkZV44olncOVKBn799RfMnj0VQ4YMR1BQMFQqFS5dSsaFC4kICQnFwoWv6z8zZcpD2LFjK1avXoWLFy/Azc0dR48egUwmbza5DACkUgekpl7Gs88+iZiYwUhPT8Pp0ycREhKKZ59tvMXsq6/+Ay+++CxWrVqBX389hLCwCGRmZuD06ZPw9vbB4sX/bNb2jRs5ePrpORgxIg6Fhfn4/fcj8PHxxYIFf9dv4+OjO9e+c+c2lJeX49FHZ7c6oa09NRir4ch+3749AIA33vifds2FMDce2RMRWQmRSIR3330fy5Z9gKFDh+Py5UvYuvV77N27E0qlEs89twAbNmyGh4eH/jMREZH44IPViI7uiUOHDuLAgR8REzMEq1evhUgkavYdUqkEq1evhZubG7Zti0d6ehoefng61qz5wuBJeEFBwfjii0148MFpyMq6iu3b45GTcx0zZszGxo3fICAgsFnbS5e+j+jonvjvf3cjKSkR998/BZ9//hX8/Pz02/TvPxDTpz+K8vIybN8ej6tXW38gTntqMFZgYJA+3IcMGY5Ro8Z2uE1zEGhbmolhAw6lnDRpe94ervDQesNRbH09OiKitvj4uKCwsOLOG7bTsmVv48cf92Ljxm86fHc/apuPT8t3HLTZYfz/S/rK5G0KBUJ0cw1BD89I9PCKQLBLIIQCDo4QEZFls9mwnxY+xaTtaSUqnM+9hMyyLFwpu4q9V/fDWeyEKM8I9PCMQLRHJLycPO7cEBER0T1ms2E/PniUSdvz8XHBBP9xqKyrQmpJBi4Xp+HSrTScL0jC+YIkAEAXZx9Ee0aih2cEIty7c8ifiIgsgs2eszf1+aeWzmlptVoUVBchpTgNl4vTkFZyBbVqJQDdkH93t/ohf89IBLkEcMifiCxSZ5+zp3untXP2DPu7dDe/DCqNClfLruFycRpSitNxrSIHWuj+eWViZ0R5hqOHZySiPSPg6cghfyKyDAx728Gw76D2/DIo6iqRWpyBy8XpSClOQ0ltqX5dF2df3bl+zwhEuIfBUezQekNERJ2IYW87GPYd1NFfBq1Wi4KqQqTUB39a6RUo64f8RQIRuruF6M/3c8ifiO4lhr3tYNh3kKl/GXRD/tn68L9ekds45C9xRrRHhD78PRzdTfa9RES3Y9jbDoZ9B3X2L0PjkH8aLhWnobS2TL/Oz9kX0Z4R6OEZiXD37hzyJyKTYtjbDoZ9B93LXwatVov8qkKDWf5Kje6hEA1D/g2z/ANdunLIn4g6hGFvOxj2HWTOX4Y6/ZC/LvyvV9xoNuTfMMufQ/5EZCyGve1g2HeQJf0yKJSVSC1J15/vNxjyl3VBD4/6Wf4eYXAQSc1YKRFZA0vav1HHMOw7yFJ/GXRD/gX64E9vMuQvFojQ3S1Ud9TvFYFAOYf8iag5S92/kfEY9h1kLb8MuiH/LINZ/g3kEhmiPRtn+bs7uJmxUiKyFNayf6M7Y9h3kLX+MlQoFUgtyag/35/efMi/ySx/DvkT2Sdr3b9Rcwz7DrKFXwatVoubVQVIKU6rH/LPRF3TIX/3bvrwD5D7c8ifyE7Ywv6NdBj2HWSLvwx1GhUyS7MaZ/krbujXccifyH7Y4v7NXjHsO8gefhkqlAqkFjfO8i9TluvX+cu61F/eF4kI926QcsifyGbYw/7NXjDsO8jefhm0Wi3yKvP1T/BLLzUc8g9z76YP/wC5H4f8iayYve3fbBnDvoPs/ZehTl2HK2VZuFyc3mzI30Uirx/y1/3hkD+RdbH3/ZstsfqwP3HiBHbs2IGamhq88MILiI6ObnN7hn3nqlAq9I/uvVychjJl479NV5lfk3v5c8ifyNJx/2Y7Wgt78T2uo92qq6vx/vvvIyUlBUePHr1j2FPncpHKMdhvAAb7DdAP+TfM8s8ozcSN6zdx6PrvEAvFCHfrpg//rhzyJyK656zmyB4AqqqqsHTpUixatAheXl5tbssje/NpGPJvuLY/p+mQv1SOaA/dDP9ozwi4ObiasVIiArh/syVWP4xfUlKCVatWYcGCBfD397/j9gx7y1GurKg/168b9i+/bci/4Ql+Ye7dIBVJzFgpkX3i/s12WHTYJyYmYtWqVfj666+h0Wjw9ttvIzU1FVKpFO+++y5CQkKwePFiFBcXw93dHePHj8f999/fZpsMe8uk1Wpxo/Km/qg/ozQTdRoVAOiH/Ht46cK/q8wPAoHAzBUT2T7u32yHxYb9+vXrsXv3bjg5OSE+Ph4HDhzAoUOHsGLFCiQkJGDdunVYu3at0e2qVGqIxaJOqJhMSalS4nLRFSTlpyDxZgqyS3P069wcXRHl1R0SkRgCgRBCCCAQ6P4IIQDqfzYuE+pfC+qXC297DTRdJjRY3/JndPMLdMuEzdYbfEa/DSCA7rMN2whvq6txma4GoUCo/4z+vaDhfZPXMFzXuI3u79+0zaZtE5F9M/sEveDgYHz66adYvHgxAODs2bOIi4sDAPTv3x/JycntarekpMpkNQLs+XYmf1Eg/LsGYmLXCSirrah/fK9ust+p3ARzl2cTmnY+hE06Mk07SML6DpBhp0Ok7zA0bG/wvklnRKRvTwjRbZ2Vxs+Jmn8ezdsTCYT12za8brJcKGqyTNR8/W3LRQ3tCJtup1vfsM7eR5C4f7MdFjsbf+LEicjJaTyaUygUkMvl+vcikQgqlQpisdlLpXvAzcEFQ/wGYojfQGi1WpQrFdBCA61WCy20+p+aJu9x23v9/1r4DABotRqDNqAFNNDqlze2gSZtNNQA/TYa/baaJm3coY4myzX1bTZ8TqOvq+lyTeNPbcP3avTb67/z9u0M2mnYvvG1tslnW/petVYNlUYDdZM2dK917Td8xlY0dgDqOwnCJu+FTToMty1rWC4WNiwXQSwQQVj/s+ky0W3btLS9n6wLfJ29zf3PQTbI4hJULpejsrJS/16j0TDo7ZRAIICbQ8u9VDI/fYejvuOgf63vEKjrOx6GHQX9azTdVve6YZ1a3466yWsN1Le912gatlXrP9/atuom26g16ibbqKHWNL7WL69fplQp9d/TdJvO4Cx2worYNyES8hQkmZbFpejAgQNx+PBhTJ48GQkJCYiMjDR3SUTUAt3QvQgi2F8wNXR0bu8sqDTqJsvUhssa3rey7lxBIq6UZaGgugj+si7m/iuSjbG4sJ8wYQKOHTuG2bNnQ6vVYvny5eYuiYjIQGd0dLTQ4kpZFnIrbjDsyeQsIuwDAwMRHx8PABAKhXjnnXfMXBER0b0VINfdPyRHkYcYDDBzNWRreE0OEZEFCKwP+1xFnpkrIVvEsCcisgDOEmd4OLgb3F6ayFQY9kREFiLQxR/lygpUKBXmLoVsDMOeiMhCBMi7AgCP7snkGPZERBYigOftqZMw7ImILETDJL2cCoY9mRbDnojIQng7eUEqkiKXw/hkYgx7IiILIRQIESDzx82qAv2jn4lMgWFPRGRBAlz8odFqcLMy39ylkA1h2BMRWZDAJnfSIzIVhj0RkQUJrL/8LreC5+3JdBj2REQWpKvcHwIIeK09mRTDnojIgjiIpPBx8kKuIg9ardbc5ZCNYNgTEVmYALk/qlTVKK0tM3cpZCMY9kREFibQhbfNJdPqlLCvrq7ujGaJiOxCAO+kRyZmVNiPGzcOmzZtanObzz77DGPHju1QUURE9kw/I59H9mQi4rZW5uTkQKFofNRibm4uMjMzcfny5Ra3r6urw4kTJ3hkT0TUAe4ObnAWO/GBOGQybYZ9YmIiFi5cCIFAAAAQCATYsmULtmzZ0upntFotRo4cadoqiYjsiEAgQIDcHxmlV1GjqoWj2MHcJZGVazPsp0yZgkuXLqG4uBharRY7d+5EdHQ0evTo0eL2EokEvr6+mDNnTqcUS0RkLwJduiK9NBM3Km+iu1uIucshK9dm2APAq6++qn996tQpTJ8+HU888USnFkVEZO8Cmpy3Z9hTR90x7Js6dOhQZ9VBRERN8B75ZEpGhX2DnTt3Ys+ePbh8+TIqKirg4eGBfv364ZFHHsGoUaNMXSMRkd3xk3WBUCBELi+/IxMwKuyVSiWee+45/PHHH9BqtXBzc0NwcDDKy8tx4MAB/Pzzz5g6dSree++9zqqXiMguSIRi+Dn7IrcyDxqtBkIB74FG7WdU2K9ZswYnTpzAhAkTsHjxYgQFBenX5ebmYsWKFdi5cyd69eqFxx9/3OTFEhHZkwB5V9yovImi6lvwdfYxdzlkxYzqKu7Zswc9e/bE6tWrDYIeAAICAvDxxx8jIiIC33//vUmLJCKyR4EuPG9PpmFU2BcVFWHYsGEQClv+mFgsxvDhw5GTk2OS4oiI7FnDbXN5cx3qKKPCPiIiAsnJyW1uk5GRgdDQ0I7UREREaLxtbk4Fb5tLHWNU2L/66qs4f/483n33XVRUVDRbv379epw6dQr/+Mc/TFYgEZG9cpHK4SZ14ZE9dZhRE/S2bduG0NBQfPPNN9ixYweioqLQpUsX1NTU4OLFiygsLIRMJms2G18gEGD79u0mLZyIyB4EyLviUnEqKuuqIJM4m7scslJGhf3u3bv1rysrK3Hu3Llm2ygUCqSkpBgsa7i3PhERGSdA7o9LxanIVeQh0iPM3OWQlTIq7Ft72h0REXWOwCaT9Bj21F68SwMRkQULdOEkPeq4dt0u98yZM9i2bRtSU1NRXV0Nd3d3RERE4KGHHkJMTIypayQisls+Tt6QCMXIVTDsqf2MDvsPP/wQX3zxBbRaLQDAyckJWVlZOH/+PLZu3Yr58+fjlVdeMXmhRET2SCQUwV/mhxuKPKg1aoiEInOXRFbIqGH8ffv2Yf369QgPD8e6detw5swZnD9/HomJidiwYQOioqLw+eef4+DBg51VLxGR3QmU+0OlVSO/qtDcpZCVMirsN23aBB8fH2zatAmjRo2CXC4HAEilUowYMQIbNmyAt7c3vv76604plojIHjU82z6HQ/nUTkaFfWpqKsaMGQMPD48W13t6emLMmDHNLr0jIqL200/SY9hTO3XKbPy6urrOaJaIyC4FyP0AgM+2p3YzKuyjoqJw+PBhlJaWtri+uLgYhw4dQlRUlClqIyIiAE5iJ3g5eiBHcUM/OZrIGEaF/RNPPIHCwkLMmzcPp06dgkqlAqC7a96RI0fw1FNP4datW3yWPRGRiQXIu0JRV4lyZfPnkhDdiVGX3k2ePBkXLlzAxo0b8eSTT0IoFEIqlaKmpgYAoNVq8fTTT+OBBx7olGKJiOxVoNwfSUUXkaPIg5uDq7nLIStj9HX2S5Yswbhx47B9+3ZcvnwZlZWVkMlkiI6OxvTp03lTHSKiThBQP0kvV3EDvbx4qpSM06476MXExDDUiYjuoab3yCcyVrvCXqFQ4ODBg7h8+TKqqqrg5uaG3r17Y8yYMZBKpaaukYjI7nk6esBR5IAchj21g9Fhv2XLFrz//vuorq42mBUqEAjg6emJ5cuXY9SoUSYtkojI3gkFQnSV++NqWTaU6jpIRRJzl0RWxKiw379/P9566y14e3vj+eefR9++fSGTyVBQUIDz58/ju+++w4svvoivv/4aAwYM6KyaiYjsUqC8KzLLspBXeRMhrkHmLoesiFFh/8UXX8DDwwPx8fHo2rWrwbpx48Zh2rRpmDVrFv73f/8X//nPf0xaKBGRvWt63p5hT8Yw6jr7tLQ0TJw4sVnQNwgLC8N9992HpKQkkxRHRESNAlx0Yc/z9mQso8Lew8MDGo2mzW2cnJwgk8k6VBQRETXXVeYHAQTIqeA98sk4RoX9zJkzsWfPnlYfdHPt2jXs3bsX06dPN0lxRETUSCqSwtfZBzcq83jbXDJKm+fsN23aZPDe2dkZMpkMjz76KB5++GEMGDAA3t7eKC8vx4ULF7Br1y54eHigZ8+enVo0EZG9CpT742xBAYprSuDl5GnucshKCLRtdA+jo6MhEAj0Pcimr/UNCAQA0OwyPHM/5raw0LT3j/bxcTF5m0RExtqfdQi7M3/C/D5Pop9PL5O0yf2b7fDxcWlxeZtH9u+9916nFENERO0TIG+YpHfDZGFPtq/NsJ82bdq9qoOIiO5CoP4e+ZyRT3fPqAl6RERkXm5SV8glMuRyRj4Zwaib6gwZMuSuthMIBDh58mS7CiIiotYJBAIEyP2RWpKBalUNnMSO5i6JrIBRYS+Xy1tcXlNTg9LSUmg0GkRGRiIoiHd2IiLqLA1hf0NxE2HuoeYuh6yAUWF/6NChVtdVVFRg7dq12LZtGz7++OMOF0ZERC0LlOvO2+cobjDs6a6Y7Jy9i4sLFi9ejPDwcHzwwQemapaIiG7TOEmP5+3p7ph8gt6AAQNw5swZUzdLRET1ujj7QCQQ8R75dNdMHvYpKSn6G+0QEZHpiYVi+Ml8cUNxExpt288rIQKMPGf/yy+/tLhcq9WiqqoKv/76K44fP44JEyaYpDgiImpZoLwrchV5KKgqgp/M19zlkIUzKuxffPHFNo/atVotfH19sWjRog4X1lE7f880aXtymQNqa+sgFgnr/wggFgkhEQshEgohEQuarGtcLxYLIREJIRIJIGmyjqMfRNQRgXJ/nITuvD3Dnu7EZGEvlUrRvXt3jBo1ChKJxCTFdcTuY1nmLqFNIqEAYrEQ4vqfug6BEBKRwLDDIBZCLGzoNAjqt2neodB3Ppp1LIQQiwVN2m/yuRY+KxSyE0JkDQL0M/LzMKhLf/MWQxbPqLB/6aWXmi2rra3FzZs34e3tbVHPsV/y2IC72u5unxLp6uaEW8WVUKk0UGm0UKk0qFNroFZrUKfWQqXWNP5RNb7XbaNFnVpj8Fndem2TbTSorlWhQqWBSqNrQ2OGR1gKBDDsTIgNRyta7JSIDTsQBh2OJh0agxGRpu3c9h36durblUpEEHIkhMhAgIvuHvm8bS7djbsK+0OHDuHnn3/Gk08+iejoaP3yDz/8EJs3b0ZNTQ2EQiEmTJiAt956Cx4eHp1W8N2KCjZtDbqnQt3bO1VpNE07Ddr6jkV9p8Ggg6Ft0vGo72xomm/XrONxWxtNOx51TTosyjo1qmpUBtveaw4SERykIjjW/2z62lEigqNU3Hy5VAQHSf1P/WtxfVu60y9E1koukcHdwQ05vG0u3YU7hv2bb76JrVu3AgBGjx6tD/uPPvoI69evh0AgwIgRIwAABw4cQEZGBrZv3w6pVNqJZdsHoVAAqVAEqURk7lIMaLVaqDVa1N02QtHsteq2TkrT9c1GSLRNtqnfrn5drVKNmjo1apVq1NapUVFWhxql6q5HZVojEQsNOgP6joR+mbiFZa10OOo7EZyPQfdSgNwfF29dhkJZCbnUckZWyfK0GfaHDh1CfHw8evbsiYULFyImJgYAkJ+fjw0bNkAgEGDp0qWYMWMGAN1s/RdffBGbNm3Cn//8586vnsxCIBDoh9vNRavVdTb0nYAmHYIapRq1daoWlhm+blymQmlFLWrr1B0etRAJBY2jEE07CQbLdKMQTlIRZE4SyBzFkDlKGl87SSAVC9lpoDsKlHfFxVuXkaO4gWjPCHOXQxaszbD/4Ycf4O7ujk2bNhncF/+nn36CSqVCSEiIPugBYNy4cRg4cCB++uknhj11KoFAdy5fKhEBzqZrV6XWtNgpqFEadh5a7kioDJYpqutwq6wGSpXx10GLRULInMSQOzZ2AHQdguYdg6bbOEpF7CTYkYZn2+cq8hj21KY2wz4pKQmjR49u9gCc48ePQyAQYOzYsc0+069fP/zwww+mrZLoHmmYHChzNN0VJRqN9raRBF3nobpWjcqaOlTWqFBZXdf8dbUKpYpa3CiqxN2ONwgFgiYdgvqf9a/lt3USmm7j7CDmlRhWKLA+7HN421y6gzbDvqysDF26dDFYptFocPbsWQDA8OHDmzcoFqOurs6EJRJZN6FQACcHMZwcjLr4RU+j0aKqVqXvAFTV1EFR/7rS4Gd9Z6H+dWFpNdSau+smCAA4O4rhXH9KQe4sgZtMCne5A1zrf7rJpHCTS+Euc4CD1LLmkdgrH2dvSIQSzsinO2pz7+Pi4oKSkhKDZUlJSVAoFJBIJBg8eHCzz2RlZVnEbHwiWyEUCiB3kkDuJAGM+NXSarWoUaoNOwQtjiIYLrtRVHnHUw+OUlF9+DvAXS5tsUPgKpdC7iThZZOdSCgQIkDuj+sVuVBpVBAL29ehJNvX5v8z+vTpg+PHj0Oj0UBYf5nS3r17AeiO6p2cnAy2LywsxNGjRxEXF9dJ5RLR3RIIGkcUvN2M+2ytUo2yylqUKpQoq1SiTFGLskolSut/lil0ywpKqts8xSASCuAqk942SqDrJBh0DGRSSMS8FLI9AuT+yCq/hpuVBfqn4RHdrs2wnzlzJl588UX8/e9/x5w5c5CWloYtW7ZAIBBgzpw5BtsWFxfjb3/7G2pqavDQQw91atFE1LkcpCL4Sp3h69H27Ee1RoPyyjqUG3QEalHa0CGorEWZQomcwkpk3axosy2Zoxjucgd4uTnC280R3m5O8HZzhJebI3zcnSBzFHPyYQsCm0zSY9hTa9oM+3HjxmHOnDn45ptvsH//fgC6ocHHHnsMo0aN0m/3/PPP48SJE6itrcX999+P8ePHd27VRGQRREIhPFwc4OHigBC4tLqdVqtFda3KYKSgVKHUdRLqOwRllUoUV9Qit6iyxTYcpCJdJ8C1viPg3tgp8HJztNvOQONtc29gKAaZuRqyVHc8wfOvf/0LEydOxOHDh6FSqTBy5EiMHj3aYJvMzEzIZDLMnz8ff/nLXzqrViKyUgKBAM6OEjg7StDVu+2bv1TV1KGorKbJn2rcavI6t7DlzoCjVKQfDWj44+XmBJ/6ToGzCa+wsCQBcj8A4LPtqU13NZtjyJAhGDJkSKvrt2/f3uzyPCKi9nB2lCDYUYLgLs1HCrRa3ZUJRaWN4V9UVlPfGahGYVk1cgoVLbarm7ugOyXg7+WMrl4y+Hs7w99TZtVXFziKHeHt5IVcxQ1otVq7HN2gOzPJ1M17GfQnTpzA3r17sWzZsnv2nURkGQQCge7+AH4ShPi13BmorFHpOgH1HYJbZTUorB8dyC+pwvWC5p0BL1cH+HvJ6v846/54y+DqbB23/Q6U+yOhMBllynK4Oxg5G5PsglVdp5GdnY1Lly6htrbW3KUQkQUSCBovUwz1c222XqvVoqxSibyiSty4VYW8W5XIu1WFG7cqkXy1GMlXiw22lztJ6sNfhq71HQB/L2d4ujpa1CWFAfVhn1Nxg2FPLbKqsA8JCcG8efOwaNEic5dCRFZIIBDAXe4Ad7kDeoR6GqyrqlEhr7gSeUWGnYCM3DKk55QZbCuVCOHvWX8awEuGQG8ZQvxc4OHiYJZh9KbPtu/t3eOefz9ZPqsKeyKizuLsKEZYVzeEdTU8Mq5TqZFfXI0b9R2Aph2B7HzDywldnSUI8XNFqJ8LQv1c7lkHILA+7HN521xqhcWEfWJiIlatWoWvv/4aGo0Gb7/9NlJTUyGVSvHuu+8iJCTE3CUSkR2SiEUI9JUj0NdwbpJGo0VReQ3yiipxrUCB7JsVyL5ZjguZt3Ah85Z+u4YOQEh9ByC0EzoAno7ucBI7IbkoBStO/a/RnxeLRVCp1EZ9xkXqgge634cQ1yCjv4/uPYFW29Gngnfc+vXrsXv3bjg5OSE+Ph4HDhzAoUOHsGLFCiQkJGDdunVYu3atUW2qVGqIxdY7w5aIrFOZohZXcsqQnlOCKzllyMgpRWFJtcE27nIHhAW6ITzQHWGB7ggPdIe3u2OHOgAbzm3Bkat/dLT8u1atqoEAAkwIi8Psvg9BLm37kkoyL4sI+/379yMqKgqLFy9GfHw83nvvPfTt2xdTpkwBAMTFxeH33383qs3Cwrbv1mUsHx8Xk7dJRPahvEqJazcrcPVmhX4E4Fa54URjF2cJuvu7IjLYHVFBHgjxk0MkvDe3EG7P/i2tJANbUnfiZlUB5BIZpoVPwVC/Qbz0z8x8fFq+uZVFDONPnDgROTk5+vcKhcLgcj6RSASVSgWx2CLKJSIyiquzFL27e6F3dy/9soYOQFb9n+yb5Ui8cguJV3SnABylIoQHuiEqyB1RwR4I9XOBWGQ5zw+I9AjH60P+hkPXf8ePVw/i65R4HL9xCrOipiGg/ha+ZDksMj3lcjkqKxvvkqXRaBj0RGRTWuoAlFTUIvVaCVKvlyL1WimSM4uRnKm7HNBBIkJ4gCsigz0QFeSObv6uZn94kFgoxn0hYxDTpT9+SN+DxMJkrDi9GqMDR2JKtwlwFDuatT5qZJEJOnDgQBw+fBiTJ09GQkICIiMjzV0SEVGn83BxwLBefhjWS3cL3FJFLdLqgz/1eikuZpXgYpbuseMSsRBhXV0RHeyBqGB3dO/qComZ5il5Onpgfp8nkFyUgq1pu3Do+u84m5+IRyIexEDfvhzatwAWGfYTJkzAsWPHMHv2bGi1WixfvtzcJRER3XPucgcM6dEFQ3p0AaAb+k+71hD+Jbh8rRSXr5UCAMQiISIC3dAv3Bv9I7zh6+7URsudo7d3D0R6hOPn7MM4cO1XbLj4DY7fOIWZUVPRxdnnntdDjSxigl5n4AQ9IrJ1iuq6xiP/ayW41uRWwAHeMn3wd/d3hVDY+tF1Z+zfCqqKsDVtFy4Vp0IsEGFi6FjcHzoOQoHlzDuwRa1N0GPY3yWGPRFZulJFLRIzipCYcQsXs4pRp9IA0F3r3zfcG/3DvdEr1LPZg386a/+m1WqRWJiMrem7UVpbht5e0Xiq1/+Dk/jejzrYC4Z9BzHsicia1NapcSmrGAnpRUi8cgvllUoAuuH+nqEe6B/hjX5h3vBwcej0/VtVXRU2XPwWKcVp8HX2xnN9noKfzLfTvs+eMew7iGFPRNZKo9Xial45EtKLkJBRhNzCxqudQv1cENs/AH1CPeDTief5NVoNdl35EQevHYGjyBFP9ZqNPt49O+377BXDvoMY9kRkKwpLq5GQUYSE9CKkXS+FWqOLgaggd4zo44eYKF84OXTO/O3TN8/jm8tbodKoMaXbfbg/dCxn65sQw76DGPZEZIuqauqQnqfA/hNX9TP7pRIhBkX6IraPH6JCPEz+ON9rFTn4PGkTSmpL0d+nD+b2mAlHsYNJv8NeMew7iGFPRLaqYf9WVFqN48k3cSw5D4WlNQAAL1cHDO/tj5F9/NDFw9lk31mhVOCL5K+RUXoVXWV+eK7vk/B28rrzB6lNDPsOYtgTka26ff+m1WqRnlOGYxfycPpyAWqUuifihQe6IbaPP2KifOHs2PFhfrVGjR/S9+C33ONwFjthXu/HEe0Z0eF27RnDvoMY9kRkq9rav9Uq1TiXVohjyXlIySqBFrq79w2K8sHEwcEI8Ws5XIxx/MYpbEndAbVWg2nhUzA2KI7n8duJYd9BDHsislV3u38rLq/RDfNfyEN+/WN7e4Z6YNLQEPQM9ehQQGeWZeOLC5tQpqxAbNehmB01nYHfDgz7DmLYE5GtMnb/ptVqcTGrGD/+cQ0p2bp79Qf7ynH/sGAMjvZt96N5S2vLsDZxI3IUNzApdDwe6H5fu9qxZwz7DmLYE5Gt6sj+7WpeOX46eQ1nUgug1QLebo6YOCQYsX38m92p726UKyvw4Zl/o6imGI9FPYKRAUPbVZe9Yth3EMOeiGyVKfZvBSVV2H/6Oo4m5aFOpYHcSYKxAwMwdlAgXJ2lxrVVVYgPz65BZV0Vnuv7JG++YwSGfQcx7InIVply/1ZepcShszn45WwOKmtUkIqFiO3rj4lDgo26Q9/VsmtYfX4dAOCvA55DN7dgk9Rn6xj2HcSwJyJb1Rn7t1qlGr8n3cD+U9dxq7wGYpEAE2KC8MCI0Lu+O9+FoktYl/QfyCTOWDjoBfjyMbl3xLDvIIY9Edmqzty/qTUanLpUgO2/XcGt8lq4yqR45E/dMbKv/13dme9Y7kl8m7oN3o6eWBjzIlylHb/Uz5Yx7DuIYU9Etupe7N+UdWr8dOoa9v2RDWWdBiFdXPD/xkcgMsj9jp/dm3kAP2YdRLBLIP464DneWrcNDPsOYtgTka26l/u34vIa/HDkCv64mA8AGNLDF4+ODoeXm2Orn9Fqtfjm8g84kXcaPb2i8HyfpyASGj/T3x4w7DuIYU9Etsoc+7eM3DJ8dzANV/MqIBELMWloMCYNDWn1cj21Ro3/u/AVLt1KxTD/GDwe/ShvutMChn0HMeyJyFaZa/+m0WpxIvkmfjhyBWUKJTxcHPDo6DAM7dmlxSCvUdVi9fl1uFaRw5vutIJh30EMeyKyVebev9UoVfjviWzsP3UdKrUGPUI88OcHesLDpfm5+QqlAqvO/htF1bfwWPQjGNmVN91pqrWwb989DYmIiEzEUSrGI6PCsOzZoegX5oWU7BK8teEUEjOKmm3rIpXjxX7zIJM444e03SipKb33BVshhj0REVkEH3cnvDyjL+ZMiESNUo3VPyTh+1/SUafSGGzn6+yNqWFToNTUYeeVfWaq1row7ImIyGIIBAKMGxSIN54YBD9PZxw4fR3LN59FfkmVwXbD/AchxCUIZ/ITkF6SaaZqrQfDnoiILE5wFxe89dRgxPbxR/bNCry98TROJN/UrxcKhHg08mEAwNb0XVBr1OYq1Sow7ImIyCI5SEV4ZkoPzH+oJwQA1u+9hC/3XkKNUgUA6OYWjGH+MchV5OHYjZPmLdbCMeyJiMiiDevph7efHoxQPxccS76J//nqDLJv6q4eeDhsEhxFjtiTuR+KukozV2q5GPZERGTxfD2c8Y+5g3D/kGDkF1dh2ddn8Ov5XLhKXTC523hUqaqxN/OAucu0WAx7IiKyCmKREDPHhuNvj/aDk4MYm/an4r8nsjA6cCT8nH1xNPcPXK+4Ye4yLRLDnoiIrErfMC/84/FB8HR1wLYjmdh1NBuPRDwILbTYmrYTNnqvuA5h2BMRkdXp4umM1+YMhK+7E/Yez0JSghD9vHvhSlkWzuQnmLs8i8OwJyIiq+Tt5oQlcwbC30t3PT7yekEiFGNHxn9Ro6o1d3kWhWFPRERWy8PFAUvmDESwrxx/nC+HV20vlCnLsT/7kLlLsygMeyIismquzlK8+tgAdO/qiquJvpBoZPjl2m8oqCo0d2kWg2FPRERWT+YowcJZ/REZ4AXFlQiotWpsTdtt7rIsBsOeiIhsgpODGK/M7Ido9x5Ql3viUnEqzt+8aO6yLALDnoiIbIaDRIS/PtIP4doR0GoF+CppG6prleYuy+wY9kREZFMkYiH+9lAsPOsioBIr8PnhI+YuyewY9kREZHPEIiHmDh4DALhUmoJjF/LMXJF5MeyJiMgmRXh2g0wsg9izAF/vv4ycQoW5SzIbhj0REdkkoUCIAb69AbESKqdbWLMjGdW1KnOXZRYMeyIisln9ffsAALpFV+JmcRX+89Nlu7x3PsOeiIhsVqR7GJzETqhyzEFYgCtOpRTg8Plcc5d1zzHsiYjIZomEIvT17onS2jI8OM4dcicJvv8lHVfzys1d2j3FsCciIpvW36c3ACCzKg3zH+oJtVqLtTuTUVlTZ+bK7h2GPRER2bRoz0hIRVIkFF5Ar1BPPDAiFEVlNfhyb4rdnL9n2BMRkU2TiiTo5RWNwupbuFF5Ew/HdkOPEA8kZBThp1PXzF3ePcGwJyIimzegfig/oTAZQqEA8x/qBTe5FNt+zUTa9VLzFncPMOyJiMjm9fKKhlggQkLBBQCAm0yKvzys6wCs33MJdSqNOcvrdAx7IiKyeY5iR0R7RuJG5U39c+4jg9wxPiYQt8pr8HvSDTNX2LkY9kREZBcabrCTWNj42NtJw0IglQix93gW6lRqc5XW6Rj2RERkF/p494BQIMT5wgv6ZW4yKcYNDESpQolfz9vu0T3DnoiI7IJcIkOEe3dkl19HSU2pfvn9Q4PhIBXhv39ko7bONo/uGfZERGQ3+vs0H8p3cZZiQkwQyiuVOHzONm+ly7AnIiK70c+nFwQQIKHJUD4ATBwSBCcHMfb9kW2TT8Zj2BMRkd1wc3BFN7dgZJReRYWy8fn2MkcJJg4OgqK6DofO5Zixws7BsCciIrvS36cPtNAiqeiiwfIJg4MgcxTjp5PXUFVjW0f3DHsiIrIr/RrupleQbLDcyUGM+4cGo7JGhZ/PXDdHaZ2GYU9ERHbF28kTQfKuSC3JQFVdtcG6cYMCIXeS4MDpazb1VDyGPRER2Z3+vn2g1qqRfCvFYLmjVIzJw0JQXavGfht6SA7DnoiI7E7/Jg/Gud2YgQFwk0nx85kcVFQp73VpnYJhT0REdsdP1gVdnH1x6VYqatWGge4gEWHy8BDUKtX46aRtHN0z7ImIyC4N8OmNOk0dUm6lNls3un9XeLg44JdzOSirtP6je4Y9ERHZpX6+rQ/lS8QiPDA8BMo6DfadyL7XpZkcw56IiOxSkDwALhI5MsuyWlwf168rvFwdcPh8rtXPzGfYExGRXRIIBJCKJNBotS2uF4uEiOvXFSq1BhevFt/j6kyLYU9ERNSKPt29AADJmQx7IiIimxTi5wIXZwkuXL0FbSsjANaAYU9ERNQKoUCA3t08UaZQ4nqB4s4fsFAMeyIiojb0bhjKt+Lz9gx7IiKiNvTq5gkBgAtXbpm7lHZj2BMREbXB1VmKUH8XZOSWobrWOh99y7AnIiK6g97dvKDWaJGSXWLuUtqFYU9ERHQHfcJ05+0vZFrnUD7DnoiI6A66+btA5ihGcqZ1XoLHsCciIroDkVCInqGeuFVei7xbVeYux2gMeyIiorvQcDc9axzKZ9gTERHdhd7dPQEAyQx7IiIi2+Qud0Cwrxyp10tRq1SbuxyjWE3Ynzt3DkuWLMGSJUtQXl5u7nKIiMgO9e7uBZVai8vXrOsSPKsJ+/j4eLzzzjuYMWMG9u3bZ+5yiIjIDvXRD+Vb161zrSbs1Wo1HBwc4OPjg8LCQnOXQ0REdigswA2OUpHVTdKzmrB3cnKCUqlEYWEhvL29zV0OERHZIbFIdwleQWk18kus5xI8iwj7xMREzJ07FwCg0Wjw5ptvYtasWZg7dy6ys7MBADNnzsSbb76J77//Hg899JA5yyUiIjvWMCvfmh6MIzZ3AevXr8fu3bvh5OQEADh48CCUSiW2bNmChIQErFixAmvXrkXv3r2xYsUKM1dLRET2rk+3xkfejo8JMnM1d8fsYR8cHIxPP/0UixcvBgCcPXsWcXFxAID+/fsjOTm5Xe16eDhDLBaZrE4A8PFxMWl7RESWwl73b0KREAKt1qi/v4+PC4K6uODytVK4uTtDKjFt1nQGs4f9xIkTkZOTo3+vUCggl8v170UiEVQqFcRi40otMfG5FB8fFxQWVpi0TSIiS2DP+zeNWgONVmv0379HsDuu51fg2Pnr6F1/pG8JWuu0WMQ5+6bkcjkqKyv17zUajdFBT0RE1JkanoJnLZfgWVzYDxw4EL/99hsAICEhAZGRkWauiIiIyFBkoBukEqHVXIJncYfMEyZMwLFjxzB79mxotVosX77c3CUREREZkIhFiA72QNKVWygqq4a3m5O5S2qTRYR9YGAg4uPjAQBCoRDvvPOOmSsiIiJqW5/uXki6cgvJmcUYPSDA3OW0yeKG8YmIiKxBw61zrWEon2FPRETUDr4ezpA5ipFfUm3uUu6IYU9ERNROAoHA3CXcFYY9ERGRjWPYExER2TiGPRERkY1j2BMREdk4hj0REZGNY9gTERHZOIY9ERGRjWPYExER2TiGPRERkY1j2BMREdk4hj0REZGNY9gTERF1gFarNXcJdyTQWkOVRERE1G48siciIrJxDHsiIiIbx7AnIiKycQx7IiIiG8ewJyIisnEMeyIiIhvHsCciIrJxDPt2OnfuHJYsWYIlS5agvLzc3OUQEZnUiRMn8M9//tPcZZCJMOzbKT4+Hu+88w5mzJiBffv2mbscIiKTyc7OxqVLl1BbW2vuUshEGPbtpFar4eDgAB8fHxQWFpq7HCIikwkJCcG8efPMXQaZEMO+nZycnKBUKlFYWAhvb29zl0NERNQqhn0LEhMTMXfuXACARqPBm2++iVmzZmHu3LnIzs4GAMycORNvvvkmvv/+ezz00EPmLJeI6K7dzf6NbI/Y3AVYmvXr12P37t1wcnICABw8eBBKpRJbtmxBQkICVqxYgbVr16J3795YsWKFmaslIrp7d7t/a7Bq1SpzlUomxiP72wQHB+PTTz/Vvz979izi4uIAAP3790dycrK5SiMi6hDu3+wXw/42EydOhFjcOOChUCggl8v170UiEVQqlTlKIyLqEO7f7BfD/g7kcjkqKyv17zUajcEvCxGRteL+zX4w7O9g4MCB+O233wAACQkJiIyMNHNFRESmwf2b/WAX7g4mTJiAY8eOYfbs2dBqtVi+fLm5SyIiMgnu3+yHQKvVas1dBBEREXUeDuMTERHZOIY9ERGRjWPYExER2TiGPRERkY1j2BMREdk4hj0REZGNY9gTERHZOIY9kY3IyclBVFSU/vGlAFBbW4sNGzaYsapGR48eRVJSkv79yZMnERUVhWXLlpmxKiL7wLAnsmGPP/441qxZY+4y8O2332LevHkoKCjQLwsICMCCBQv0T10jos7D2+US2bBbt26ZuwQALdcRGBiIl156yQzVENkfHtkTERHZOIY9kQ1qOH+fm5uLiooKREVF4bXXXtOvVygUWLVqFcaPH4/evXsjLi4Ob731VrMj8Ndeew1RUVFISkrC5MmT0adPH/1DUwDg8OHD+POf/4xhw4ahV69eGDZsGF544QWkpKTo25g7dy4+++wzAMCLL76IqKgoAK2fs7969SoWLVqEESNGoHfv3hg/fjxWrlyJioqKFmsrKyvDW2+9hZEjR6JPnz6YPn069u/fb7p/TCIbwGF8Ihvk6uqKBQsW4D//+Q9qa2sxf/589OjRAwBQUVGBxx57DGlpaRg+fDjuu+8+5OTkID4+Hr///ju+//57+Pr6GrT3l7/8BX369MHIkSPh7OwMgUCAzZs3Y+nSpQgODsYDDzwAiUSCCxcu4JdffsEff/yBn376Cb6+vpg2bRoA4NSpU5g8eTK6d+/eat2JiYl46qmnUFNTgzFjxiAoKAgJCQn48ssvcfjwYXz33Xdwd3c3+MzTTz+N0tJSTJo0CVVVVdizZw/++te/YvPmzYiJiTHtPyyRlWLYE9kgV1dXvPTSS9ixYwfKy8sNzo1/9NFHSEtLw5tvvok5c+bol//yyy944YUXsGzZMqxevdqgvYEDB+LTTz/Vv1cqlfj4448RGhqKHTt2wNnZWb/u7bffxnfffYfDhw9j1qxZmD59OnJzc3Hq1ClMmTIF48ePb7FmtVqNxYsXQ6lUYt26dfjTn/6kX7dq1SqsX78eK1eubPYYVpFIhL179+prGD58OBYtWoT4+HiGPVE9DuMT2RGVSoWdO3ciIiLCIOgBYNy4cRg4cCB+/vlnKBQKg3UTJ040eK9Wq7F06VIsW7bMIOgBYMiQIQCMnxx4/vx5ZGVlYcqUKQZBDwAvv/wyunTpgj179kCpVBqsmzNnjkENo0aNAgBkZWUZ9f1EtoxH9kR25OrVq6iqqoJarTY4Um9QW1sLtVqN1NRUDBo0SL88ICDAYDsnJydMnjxZ3+aVK1dw7do1pKen48SJEwAAjUZjVG0N5/kHDx7cbJ1UKkWfPn1w8OBBZGZmIjo6Wr+uW7duBtu6uLgAQLNOAZE9Y9gT2ZHy8nIAQGZmpn7SXEvKysoM3js6Ojbb5vTp03jvvfdw8eJFAICDgwOio6PRq1cv5OXl6Sfx3a2G0QS5XN7i+oZ5BNXV1QbLpVKpwXuBQAAARn8/kS1j2BPZEZlMBgB4+OGHsXLlyna3k5ubi2effRYODg5YunQpBg0ahNDQUIhEIuzbtw8HDx5sd21Nb7zTVENH5fYJekR0ZzxnT2RHunXrBqlUiosXL7Z45PvVV19hzZo1KCkpabOdgwcPorq6Gi+//DJmzpyJsLAwiEQiAMCVK1cAGB5ZNxxtt6XhaoGzZ882W6fRaHD27Fk4Ozs3O6VARHfGsCeyYRKJBCqVSv/ewcEBkydPRkZGBjZu3Giw7cmTJ7Fy5Ups27YNbm5ubbbr4OAAACgqKjJYfvnyZWzatAkADL5XLNYNIrZ1Hn3QoEEICQnBgQMHcOTIEYN1n3zyCfLy8jBp0qRmw/ZEdGccxieyYb6+vsjKysKiRYsQGxuLqVOnYsmSJTh//jzef/99/PLLL+jbty/y8/Nx4MABiMViLF++HEJh28cBY8aMwYcffoh169YhMzMTwcHByM7OxuHDh/UT5EpLS/Xbd+nSBQCwdu1apKSkYMGCBc3aFAqFWLFiBebNm4fnn38eY8aMQXBwMM6fP4+EhASEhYVh8eLFpvvHIbIjPLInsmGvvvoqIiIi8NNPP2HXrl0AAE9PT8THx+OZZ55Bfn4+vv76a5w5cwZjx45FfHw8hg4desd2u3Tpgo0bN2LYsGH4448/8O233+Lq1auYO3cufvzxR7i7u+P333/XD+VPnjwZkyZNwvXr1/Htt98iNze3xXYHDhyIH374AZMnT8b58+fxzTffoLS0FH/5y1+wdetWnq8naieBllNWiYiIbBqP7ImIiGwcw56IiMjGMeyJiIhsHMOeiIjIxjHsiYiIbBzDnoiIyMYx7ImIiGwcw56IiMjGMeyJiIhsHMOeiIjIxv1/wMkWd75Isn4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x396 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_log_lines([-compute_subopt(vs, solver.v), compute_subopt(values, solver.v)], \n",
    "               [r\"Subopt of $V^t$\", \"Subopt of $V^{\\pi_t}$\"], \n",
    "               [\"Iteration\", \"Subopt\"], \"figs\", \"VI.pdf\", show = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9855f75c",
   "metadata": {},
   "source": [
    "# Ex 2: Policy Iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5f785a42",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:49.875724Z",
     "start_time": "2023-03-14T00:56:49.749686Z"
    }
   },
   "outputs": [],
   "source": [
    "def evaluate_policy(pi, env, tol=1e-10):\n",
    "    \"\"\"Implementation of policy evaluation through iteratively applying using a certain policy \n",
    "    Args:\n",
    "        pi: a list of policies obtained by section 1.1\n",
    "        env: environment\n",
    "        tol: a scalar to dermerminate whether the policy evaluation convergences\n",
    "    Returns:\n",
    "        v: an array with the values of the actions chosen\n",
    "        q: an array with the q values    \n",
    "    \"\"\"\n",
    "    v = np.zeros(env.n_states)\n",
    "    q = np.zeros((env.n_states, env.n_actions))\n",
    "    while True:\n",
    "        v_old = np.copy(v)\n",
    "        for a in range(env.n_actions):\n",
    "            q[:, a] = ???\n",
    "        for s in range(env.n_states):\n",
    "            v[s] = ???\n",
    "        if np.linalg.norm(v - v_old) < tol:\n",
    "            break\n",
    "    return v, q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "8028d769",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:49.987685Z",
     "start_time": "2023-03-14T00:56:49.877686Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_greedy_policy(q):\n",
    "    \"\"\"Implementation of a greedy approach to choose policies (policy improvement)\n",
    "    Args:\n",
    "        q: q values obtained from evaluating the policies\n",
    "    Returns:\n",
    "        policy: greedy policy (list)\n",
    "    \"\"\"\n",
    "    policy = []\n",
    "    for s in range(q.shape[0]):\n",
    "        policy.append(???)\n",
    "    return policy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "713b45cf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:50.126686Z",
     "start_time": "2023-03-14T00:56:49.988685Z"
    }
   },
   "outputs": [],
   "source": [
    "def policy_iteration(env, tol=1e-20):\n",
    "    \"\"\"Implementation of policy iteration\n",
    "    Args:\n",
    "        env: environment\n",
    "        tol: a scalar to dermerminate whether the algorithm convergences\n",
    "    Returns:\n",
    "        vs: value functions at each iteration (list)\n",
    "    \"\"\"\n",
    "    vs = []\n",
    "    v = np.zeros(env.n_states)\n",
    "    q = np.zeros((env.n_states, env.n_actions))\n",
    "    pi = np.zeros(env.n_states, dtype=int)\n",
    "    while True:\n",
    "        v_old = np.copy(v)\n",
    "        v, q = ???\n",
    "        pi = ???\n",
    "        vs.append(v)\n",
    "        if np.linalg.norm(v - v_old) < tol:\n",
    "            break\n",
    "    return vs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "00c7756a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:52.159688Z",
     "start_time": "2023-03-14T00:56:50.128686Z"
    }
   },
   "outputs": [],
   "source": [
    "values_pi=policy_iteration(gridworld)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "cd691fb1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:52.398689Z",
     "start_time": "2023-03-14T00:56:52.160690Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAFgCAYAAAC8KtBSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA8zElEQVR4nO3deUAU9f8/8OfsLsuxuwgq4MWiqOCR5IGoeWvmlZlkaJhp+qk0q49+zKPD45OpqWX6qa9p5pF4YphHv7LyyDtUwgvFAxUEORVll2uB3d8fxCqBwLILwy7Pxz+5M7MzL96Sz3nPvOc9gsFgMICIiIhslkTsAoiIiKhqMeyJiIhsHMOeiIjIxjHsiYiIbBzDnoiIyMYx7ImIiGycTOwCqkpqqsai+3N1dUJ6epZF91kbsR3NxzY0H9vQfGxD81m6Dd3cVE9cx559BclkUrFLsAlsR/OxDc3HNjQf29B81dmGDHsiIiIbx7AnIiKycQx7IiIiG8ewJyIisnEMeyIiIhvHsCciIrJxDHsiIiIbx7AnIiKycQx7IiIiG8ewJyIisnEM+wrQZufhcMQdGAwGsUshIiIyGcO+Ak5cTMTyrX/hSmy62KUQERGZjGFfAfXrOAIAbiVmiFwJERGR6Rj2FeDloQQAxCVrRa6EiIjIdAz7CqhXxwFKRzvEJWvELoWIiMhkDPsKEAQB3o3rIDk9G9m5+WKXQ0REZBKGfQV5N64DALiTwkv5RERkXRj2FdT877CP5aV8IiKyMjKxC7AWRT173rcnIrEdP34Ee/f+iMuXo5CZqYWzszNat26L558fjh49eld6v++88ybOnfsLv/xyGCqVyoIVm+bKlShoNBoEBHS1+L5//nkfNm1aj+TkJCiVKqxa9R08PdUWPcarr76M27dvlbru9dffwMSJb1n0eBXBnn0FNXZXQS6TcEQ+EYnqyy+XYvbs6bh16yZ69uyNUaPGwN+/Cy5cOI/Zs6djyZKFYpdolpMnj+Ott17H7ds3Lb7v27dv4bPPFuDBgwcYMWIknntuMBo0aFjqtr//vh89evjjiy+WlLvfZcsWoUcPf6xatRIAsGjR538vX4k9e/Zj9+79cHBwwOzZc/Dqq+Ms9wOZgD37CpJKBHi6K3E7SYO8fD3sZDxPIqLq9ddfZxEWFoo+ffph/vxFkMke/ROu1Wrx3ntvYd++H/HMM93Rs2cf8Qo1w4MH6dDr9VWy7+vXr0Kv1yMw8GW8+ebbZW7booUPAODmzRtlbnftWjT27dsNNzd3jB//BgAgPf0+BEHA0093gJOTE+Lj7yAnJwd+fu1hb++AxMS7uHHjGgIDh1nmB6sAJpYJ1B4qFOgNSEhj756Iqt/Jk8cBAIGBQcWCHgCUSiUmTXoXAHDkyOFqr80a5OXlAQBcXFzK3Vat9oJcbo+YmLLD/ssvl0Gv1+Odd6bByckJAHDjxjU0atTY+Pn69atwcHBAkyaeAIDw8FO4eTPGjJ/EdAx7E6g5uQ4RiSg/v/DR3yf1Np9+uj0++eQzjBoVbFz2zjtvokcPf2g0xccbJSbeRY8e/vjgg+kl9pOcnIQPPngfAwb0xJAh/TF//kdISIgvsV1aWhqWLVuEwMCh6NOnKwIDh2LZskVIS0srtt3ChfPRo4c/0tPvY8GCORg0qA8GD+6HDz6YXiz0Fi6cj0WL/gsA+N//lqNHD38kJt4ts00qWsPIkcNK7HvdujVP3K9UKkWzZt7QajVISUkudZtff/0ZFy+eR6dOAejff4Bx+Y0bN9C8eUvj5+vXr6FZs+aQSCSIjIzAmjX/h19++QnDhw+HVls9eSJq2Ofl5WH69OkYPXo0goODERMTg9jYWLzyyisIDg7GvHnzjJdzQkNDERgYiKCgIBw+LM5Zq1eDwgErsUkcpEdE1a9z5y4AgP/7v5X48suluHTpAgoKCozr7e0d0K/fs2jZ0tes40yf/g5iY2/hxRdHol07Pxw48CsmTZqApKQk4zZxcXGYMGEM9uzZBbXaCy+9NApqtRf27NmFiRNfLfXkYPr09xARcRZDh74Af/8AnDx5HG+/PRHXr18DAPTs2Qc9exYOMAwI6IbXX38DSuWTBwomJMRXuIagoFdK7LtDh05ltkOLFoWBXVrvPisrC9988xVkMhn+85+ZxdbFxFw3fhco7OkX3Rbo0KETmjdvgeXLv8aePXugVCrLrMFSRL1nf+TIEeTn52P79u04ceIEVqxYgby8PEydOhVdunTB3LlzcfDgQbRv3x4hISEICwtDbm4ugoOD0b17d8jl8mqtt3F9JaQSgSPyiUgU3bv3xIsvjsTu3T8gLCwUYWGhUCgU8PNrj86du6BPn/5wd/cw+zgNGzbGypXfwN7eHgCwe3cYPv98Mdas+Rrz5n0KAJgzZw7u37+HWbM+xrBhLxq/++OPP+CLLz7D0qULsXLlN8X2m5HxEBs3bjNeRv/jj4P4+ONZWLnyc3z99bfo1asPtFoNjh07gq5duyEoKBhlWbp0YYVrCAoKhlKpqvC+geL37bt1615s3caN3yEtLRVjxoyDl1dT43K9Xo9bt2IwZsyjgXgJCfFo27ad8XNSUiIaNmxU7vEtSdSefbNmzVBQUAC9Xg+tVguZTIaoqCgEBAQAAHr16oWTJ0/iwoUL6NChA+RyOVQqFdRqNaKjo6u9XjuZBA3rKXAnVQu9nq+7JaLq9/77s7F06Qp06fIMZDIZMjMzcerUCfzvf8sRFDQcq1d/bfYAt7femmIMegB48cWXoFZ74ciRQ9DpdEhJScaff/6Jp5/uUCxkAWDEiJFo3boNIiLOlLgEP27cxGL3y/v06Q8/v/Y4d+6vJ14qf5KUlGRERJwxuQZTNG/eAgBK3F+Pi4vFzp3b4O7ugfHj/1VsXUJCPLKzs4v17L29W2DLlk04deoEUlKSUbduPQiCUOm6KkPUnr2TkxMSEhIwePBgpKenY/Xq1Thz5oyxERQKBTQaDbRabbFnPhUKRbn3OVxdnSCTSS1ar5ubCj5erohP1UIHAZ5u4j2Has3c2G5mYxs+sn5fFE6cTxC7jDJ1f7oxJgxra7H9DR8+GMOHD0ZmZibOnj2LU6dO4dChQ4iNjcXmzRvh4CDDjBkzAAByeeE/8/XrK+Hs/Oj3JjdXYVxf9Pskl8sgCAL69HmmxJXTDh3aY8+eWGg0qUhOLgzQbt26lPq7GBDQGVeuXEZKyh34+fnCwcEOANCvX88S23fq1AEXLpxDWloC2rZtAZXKAQCgUNiX+Xt+8eIZk2oAUOF9F+natSMAIC7uVrHtP/xwJfLy8vDRRx9CrXYv9h03t7a4evVqsWWrV/+f8c+RkZFo3LihcX/V9f+yqGG/ceNG9OjRA9OnT0diYiLGjRtnHC0JAJmZmXB2doZSqURmZmax5eVN+JCenmXRWt3cVEhN1cDDpfCXJfJKEhw4vNFkRe1Ilcc2LC47S4eCAtOutEmlgsnfMUd2lq7K/s7atOmINm06YsKEt/HTT3uwdOlChISEYPTo8XBwcIBOVzioLy1Ni9zcR73J+/cL/03V6fKNtel0+VCpnPHwYS6A3GLHkUgKwz8hIRVJSWnGZaX9XApF4SRkKSnpSE3VICen8N91qVRRYntHR9Xf+01BaqoGGk0OACAzM7fMNktMNK0GABXe92M/NdzdPRATE4PExHTIZDIcP34UR48eRefOXdCpU3eT/15dXBogNvYOBg8eghUrvkTdupa7nF/WiYOoYe/s7Aw7u8Izvjp16iA/Px9t2rRBeHg4unTpgqNHj6Jr167w8/PDihUrkJubC51Oh5iYGPj4+IhSs5dHYWPGJWvQrW0DUWogokeC+rVAUL8WJn3HGk+YMjO1mDhxLNRqLyxduqLEekEQMGzYizh8+ABOn/4TKSnJUKu9jFdKDYbiJzc5OTlPPI7BYChxmTktLRUA4OxcBxkZD4st+6eikf916rgUW56bmwMHB4dSt63I43CPK3qszdQaTNWihQ9OnjyG+Pg7aNSoMb76ajns7OwwbdrM8r9cCqVSifXrNwOo3t9DUfum48ePR1RUFIKDgzFu3DhMmzYNc+fOxVdffYVRo0YhLy8PAwcOhJubG8aOHVtsu8fvJ1UnT3c+fkdE1U+hUEKr1eLs2dO4f/9emdtKJBLUq1cPAIwdquzs4lc7SxstDwAFBQW4dq34Zej8/HxcvXoFjo5O8PRUo0WLwsvi58+fK3Uf5879BUEQ0LRps2LLr1y5XGLbqKiLkEql8PVtDQAVvpdd2RpM9fiI/G3bQpCQEI/Ro1+FWu1l1n6rm6hhr1AosHLlSmzduhU7d+7EsGHD0KxZM2zevBk7duzA4sWLIZUW3ncPCgpCWFgYdu3ahYEDB4pWs6O9DO6ujohL1pQ4UyYiqkovvRQEnU6Hjz+eVeI5cqBwzvyzZ0+jV68+UCgKOyZqdVMAjybkAYDc3Fxs2xbyxONs2PBtsUf6tmz5HikpyRg8eCikUikaNGiALl264OrVK9i9+4di3923bzcuXjyPjh39SzwZsG7dGmRmPuoo/fHHQfz111n07Nkbzs6Fl92l0sILzo/f0i1NgwYN0LGjv8k1mKpoRH54+Els3rwRHh4NMG7cRLP2KQZOl1sJXh4qnIlOwb2HOajv4ih2OURUS7z22gTExNzAH38cxOjRLyIgoBs8PdXIz8/H5cuXcPHieXh5NcX06R8YvzN06Av48cedWLnyc0RFXUSdOi44fvwIFAolHB1L/vsll9vj6tVovPHGOPj7d8b169dw5kw4vLya4o03Hk0x+8knn+CVV4Lx+eef4Y8/DqF585a4efMGzpwJR/36bpg586MS+757Nx6vvz4GzzzTE6mpyTh27Ajc3Nzxzjv/MW7j5uYGoPBxv4yMDLz88mjUr+9WanvMmPEhpkx5w6QaTFXUs//5530AgI8//m+JWxHWgEPMKqFoJr1YXsonomoklUrx6adLsHDhMnTp0g3R0Zexc+d2/PTTbuh0Orz11jtYv34zXF1djd9p2dIHy5atRKtWbXDo0AH89tsv8PcPwMqV3xivnD5OLrfDypXfoE6dOggLC8X169cwfHggVq36rtjA6KZNm+K77zZh2LARuH37FnbtCkV8/B2MHDkaGzZsQePGTUrse8GCJWjVqg3+3//biwsXzmPQoKH49tuNaNDg0fin9u07IjDwZWRkPMSuXaG4devJL8Tx9FSbXIOpmjTxNIZ7QEA39O7dz+x9ikEw2Oi1aEsPenh8IMWlm/ewPPQ8hj3TFCN6eVv0OLbOGgdG1TRsQ/OxDc1nShsuXDgfv/zyEzZs2GL27H62xNK/h2WNxmfPvhLUj43IJyIiqukY9pXgrJDDVWWPWIY9ERFZAYZ9JandlXig1SEjUyd2KURERGVi2FcSL+UTEVXMRx/Nx/HjZ3m/XkQM+0oyvu6WYU9ERDUcw76S+PgdERFZC4Z9JdVzdoDCQcbL+EREVOMx7CtJEASoPVRISc9Gdm6+2OUQERE9EcPeDF4cpEdERFaAYW+Govv2fAMeERHVZAx7M/DxOyIisgYMezM0qOsEuZ2EI/KJiKhGY9ibQSIR4OmuxN20TOTlF5T/BSIiIhEw7M2k9lBBbzAgPjVT7FKIiIhKxbA3E0fkExFRTcewNxNH5BMRUU3HsDdT4/pKSCUC58gnIqIai2FvJjuZBI3qKxCfooVebxC7HCIiohIY9hag9lBCl69H4v0ssUshIiIqgWFvARykR0RENRnD3gKKZtKLTWLYExFRzcOwtwBPdyUEsGdPREQ1E8PeAhztZXB3dURcshYGAwfpERFRzcKwtxCvBipk5ebj3sMcsUshIiIqhmFvIcb79ryUT0RENQzD3kKKZtLjG/CIiKimYdhbCN9tT0RENRXD3kKcneRwVdkz7ImIqMZh2FuQl4cKD7Q6PMzUiV0KERGREcPegh69AY+9eyIiqjkY9hbE+/ZERFQTMewtyIvT5hIRUQ3EsLegus72UDjIEMfH74iIqAZh2FuQIAhQe6iQ8iAbWTn5YpdDREQEgGFvcUWX8u+k8FI+ERHVDAx7C1M34Ex6RERUszDsLcyLI/KJiKiGYdhbmIerE+R2EoY9ERHVGAx7C5NIBKjdVbibloW8/AKxyyEiImLYVwW1hxJ6gwHxqZlil0JERMSwrwp8tz0REdUkDPsq8GiQHkfkExGR+Bj2VaBRfQWkEoGD9IiIqEZg2FcBO5kEjesrcCdFiwK9XuxyiIiolmPYVxG1hwp5+Xok3csSuxQiIqrlGPZV5NG77XnfnoiIxMWwryJeDTgin4iIagaGfRXxdFdCAKfNJSIi8THsq4iDXAb3uk6IS9bCYDCIXQ4REdViDPsq5OWhRFZuPtIe5ohdChER1WIM+ypUNLlObBIv5RMRkXgY9lWoaNrcuBSGPRERiYdhX4X4+B0REdUEMrELWLNmDQ4dOoS8vDy88sorCAgIwOzZsyEIAlq2bIl58+ZBIpEgNDQU27dvh0wmw+TJk9G3b1+xSy+XykkOV5U9H78jIiJRidqzDw8PR2RkJLZt24aQkBAkJSVh8eLFmDp1KrZu3QqDwYCDBw8iNTUVISEh2L59O9atW4fly5dDp9OJWXqFeXmo8FCrw0NtrtilEBFRLSVq2B8/fhw+Pj6YMmUKJk2ahD59+iAqKgoBAQEAgF69euHkyZO4cOECOnToALlcDpVKBbVajejoaDFLr7CiS/mxvJRPREQiEfUyfnp6Ou7evYvVq1cjPj4ekydPhsFggCAIAACFQgGNRgOtVguVSmX8nkKhgFZbdni6ujpBJpNatF43N1X5G/1DOx937D1xG/e0ukp93xaxHczHNjQf29B8bEPzVVcbihr2Li4u8Pb2hlwuh7e3N+zt7ZGUlGRcn5mZCWdnZyiVSmRmZhZb/nj4lyY93bIvoHFzUyE11fR7766OhU185WYaUlMbWrQma1TZdqRH2IbmYxuaj21oPku3YVknDqJexu/UqROOHTsGg8GA5ORkZGdno1u3bggPDwcAHD16FP7+/vDz80NERARyc3Oh0WgQExMDHx8fMUuvMFeVPZSOdhykR0REohG1Z9+3b1+cOXMGI0eOhMFgwNy5c9GkSRPMmTMHy5cvh7e3NwYOHAipVIqxY8ciODgYBoMB06ZNg729vZilV5ggCFB7KHH5djqycvLh5CD6AxBERFTLiJ48M2fOLLFs8+bNJZYFBQUhKCioOkqyOLWHCpdvp+NOiga+alexyyEiolqGk+pUA+O0uRyRT0REImDYVwPj43ecI5+IiETAsK8GHnWdYG8n5Rz5REQkCoZ9NZAIAjzdlUhMy4Iur0DscoiIqJZh2FcTLw8V9AYDEtIyy9+YiIjIghj21YT37YmISCwM+2pifLc9J9chIqJqxrCvJo3dFJBKBD5+R0RE1Y5hX01kUgkauykQn6pFgV4vdjlERFSLMOyrkdpDhbx8PRLvWfYlPURERGVh2FcjL963JyIiETDsq1HRiPw43rcnIqJqxLCvRp7uSgjg43dERFS9GPbVyEEug0ddJ8SlaGEwGMQuh4iIagmGfTVTeyiRnZuP1Ic5YpdCRES1BMO+mnk1+HuQHi/lExFRNWHYVzO18d32DHsiIqoeDPtq9ujxO47IJyKi6sGwr2ZKRzvUdbbns/ZERFRtGPYi8PJQ4WGmDg+0uWKXQkREtQDDXgR8Ax4REVUnhr0IjO+25317IiKqBlUS9tnZ2VWxW5vBOfKJiKg6mRT2/fv3x6ZNm8rc5uuvv0a/fv3MKsrWuarsoXS0Y9gTEVG1kJW1Mj4+Hlrto0vNCQkJuHnzJqKjo0vdPi8vD6dOnWLPvhyCIMDLQ4mo2+nIysmDk4Od2CUREZENKzPsz58/j+nTp0MQBACFIbVjxw7s2LHjid8xGAzo3r27Zau0QWoPFaJupyMuWYtWXq5il0NERDaszLAfOnQoLl++jPv378NgMGD37t1o1aoVWrduXer2dnZ2cHd3x5gxY6qkWFvy+Ih8hj0REVWlMsMeAGbMmGH88+nTpxEYGIjXXnutSouqDYrmyOeIfCIiqmrlhv3jDh06VFV11Druro6wl0s5SI+IiKqcSWFfZPfu3di3bx+io6Oh0Wjg6uqKp59+Gi+99BJ69+5t6RptkkQQ4OmuxM2EDOjyCiC3k4pdEhER2SiTHr3T6XR4/fXX8cEHH+DEiRPIz8+HWq2GwWDAb7/9hkmTJuGDDz6oqlptjpeHCnqDAfGpmWKXQkRENsyknv2qVatw6tQpDBgwADNnzoSnp6dxXUJCAj777DPs3r0bbdu2xauvvmrxYm3No5n0NPBu5CxyNUREZKtM6tnv27cPbdq0wcqVK4sFPQA0btwYX375JVq2bInt27dbtEhbxZn0iIioOpgU9mlpaejatSskktK/JpPJ0K1bN8THx1ukOFvXqL4CUonAsCcioiplUti3bNkSly5dKnObGzduoGnTpubUVGvIpBI0cVPiTkom8gv0YpdDREQ2yqSwnzFjBiIjI/Hpp59CoynZG127di1Onz6NDz/80GIF2jq1hxL5BXok3csSuxQiIrJRJg3QCwsLQ9OmTbFlyxb8+OOP8PX1hYeHB3JychAVFYXU1FQoFAosXry42PcEQcCuXbssWritKJxJLxGxyRo0cVeKXQ4REdkgk8J+7969xj9nZmbir7/+KrGNVqvFlStXii0rmlufSno0SE+L7u1ELoaIiGySSWH/pLfdUeV5uishgCPyiYio6ph0z54sz14uRYN6TohL0UBvMIhdDhER2aBKTZd79uxZhIWF4erVq8jOzoaLiwtatmyJF154Af7+/pau0eapPVRIvJeFtAfZcHd1ErscIiKyMSaH/RdffIHvvvsOhr97oY6Ojrh9+zYiIyOxc+dOvPnmm5g2bZrFC7Vlag8lwi8nIy5Zy7AnIiKLM+ky/s8//4y1a9eiRYsWWLNmDc6ePYvIyEicP38e69evh6+vL7799lscOHCgquq1SUWD9GJ5356IiKqASWG/adMmuLm5YdOmTejduzeUysJHxeRyOZ555hmsX78e9evXR0hISJUUa6vUDHsiIqpCJoX91atX0bdvX7i6upa6vm7duujbt2+JR++obEpHO9RztkdcslbsUoiIyAZVyWj8vLy8qtitTVN7qJCRqcMDba7YpRARkY0xKex9fX1x+PBhPHjwoNT19+/fx6FDh+Dr62uJ2moVvgGPiIiqiklh/9prryE1NRUTJ07E6dOnkZ+fD6Bw1rwjR45g/PjxuHfvHt9lXwnG+/ZJDHsiIrIskx69GzJkCC5evIgNGzZg3LhxkEgkkMvlyMnJAQAYDAa8/vrreP7556ukWFum9igc7Mj79kREZGkmP2c/a9Ys9O/fH7t27UJ0dDQyMzOhUCjQqlUrBAYGclKdSnJV2UPlZMcR+UREZHGVmkHP39+foW5hgiBA7aFC1K37yMzJg8LBTuySiIjIRlQq7LVaLQ4cOIDo6GhkZWWhTp06eOqpp9C3b1/I5XJL11hrqD2UiLp1H3HJWrT2Kv3xRiIiIlOZHPY7duzAkiVLkJ2dbZwyFyjsmdatWxeLFi1C7969LVpkbfH4iHyGPRERWYpJYf/rr79i3rx5qF+/PiZNmgQ/Pz8oFAqkpKQgMjIS27Ztw5QpUxASEoIOHTpUVc02i4/fERFRVTAp7L/77ju4uroiNDQUjRo1Krauf//+GDFiBEaNGoUVK1bg+++/t2ihtYGbqyMc5FLEckQ+ERFZkEnP2V+7dg0DBw4sEfRFmjdvjueeew4XLlywSHG1jUQQ4OmuROK9TOTmFYhdDhER2QiTwt7V1RV6vb7MbRwdHaFQKCq8z3v37qF3796IiYlBbGwsXnnlFQQHB2PevHnGY4WGhiIwMBBBQUE4fPiwKSVbHbWHCgYDEJ/K3j0REVmGSWEfFBSEffv2PfFFN3Fxcfjpp58QGBhYof3l5eVh7ty5cHBwAAAsXrwYU6dOxdatW2EwGHDw4EGkpqYiJCQE27dvx7p167B8+XLodDpTyrYqj+7bM+yJiMgyyrxnv2nTpmKfnZycoFAo8PLLL2P48OHo0KED6tevj4yMDFy8eBF79uyBq6sr2rRpU6GDL1myBKNHj8a3334LAIiKikJAQAAAoFevXjhx4gQkEgk6dOgAuVwOuVwOtVqN6Oho+Pn5VebnrfGKZtLjtLlERGQpZYb9okWLIAiC8RG7x/8cFhaGsLAwCIIAAMblGRkZmDZtGgYNGlTmgXft2oW6deuiZ8+exrA3GAzG/SkUCmg0Gmi1WqhUKuP3FAoFtFrb7fU2qq+ATCpwRD4REVlMmWG/ePHiKjtw0YnCqVOncOXKFcyaNQv37983rs/MzISzszOUSiUyMzOLLX88/J/E1dUJMpnUojW7uZV/XEvwauiMuCQNXOsqIJNWyVuIRVVd7WjL2IbmYxuaj21ovupqwzLDfsSIEVV24C1bthj/PHbsWMyfPx/Lli1DeHg4unTpgqNHj6Jr167w8/PDihUrkJubC51Oh5iYGPj4+JS7//T0LIvW6+amQmpq9fS2G9dzQkz8Q1yMTkYTd2W1HLO6VGc72iq2ofnYhuZjG5rP0m1Y1olDpabLrSqzZs3CnDlzsHz5cnh7e2PgwIGQSqUYO3YsgoODYTAYMG3aNNjb24tdapUqfN1tImKTNTYX9kREVP1MCvuiwXPlEQQB4eHhFd5vSEiI8c+bN28usT4oKAhBQUEV3p+1M77bPlmD7u0ailwNERFZO5PCXqksvZeZk5ODBw8eQK/Xw8fHB56enhYprrbydFNCAB+/IyIiyzAp7A8dOvTEdRqNBt988w3CwsLw5Zdfml1YbWYvl6JBPSfcSdFAbzBA8vcTCkRERJVhsaHeKpUKM2fORIsWLbBs2TJL7bbW8vJQITu3AKkPssUuhYiIrJzFn+vq0KEDzp49a+nd1jpqzqRHREQWYvGwv3LlinFiHKo8r79n0uPkOkREZC6T7tkfPHiw1OUGgwFZWVn4448/cPLkSQwYMMAixdVmno+NyCciIjKHSWE/ZcqUMnvtBoMB7u7ueP/9980urLZTOtqhnrMD4pI0xaYRJiIiMpXFwl4ul8Pb2xu9e/eGnZ2dRYqr7dQeSkReT8MDrQ6uKtueSIiIiKqOSWH/7rvvlliWm5uLpKQk1K9f36T32FP5vBqoEHk9DXHJGoY9ERFVWoUG6B06dAgffPABoqOjiy3/4osv0LVrVwwaNAgBAQGYOnUq0tPTq6TQ2kjN+/ZERGQB5fbs586di507dwIA+vTpg1atWgEAli9fjrVr10IQBDzzzDMAgN9++w03btzArl27IJfLq7Ds2sGLj98REZEFlNmzP3ToEEJDQ9G6dWt899136NOnDwAgOTkZ69evhyAIWLBgAdatW4d169bhq6++wo0bN7Bp06bqqN3muSjlUDnZ8fE7IiIyS5lh/8MPP8DFxQWbNm1C9+7djW+b279/P/Lz86FWqzFy5Ejj9v3790fHjh2xf//+qq26lhAEAV4eKqQ9zEFmTp7Y5RARkZUqM+wvXLiAPn36lHgBzsmTJyEIAvr161fiO08//TRiY2MtW2UtZpxJL4m9eyIiqpwyw/7hw4fw8PAotkyv1yMiIgIA0K1btxLfkclkyMtjL9RS1H/PpBfL+/ZERFRJZYa9SqUqMbr+woUL0Gq1kMlk6Ny5c4nv3L59G66urpatshYzDtJLYc+eiIgqp8ywb9euHU6ePAm9Xm9c9tNPPwEo7NU7OjoW2z41NRXHjx9Hu3btqqDU2snN1REOcilH5BMRUaWVGfZBQUGIj4/Hf/7zH5w5cwZbtmzBjh07IAgCxowZU2zb+/fvY+rUqcjJycELL7xQpUXXJhJBgNpdicR7mcjNKxC7HCIiskJlPmffv39/jBkzBlu2bMGvv/4KoHD+++DgYPTu3du43aRJk3Dq1Cnk5uZi0KBBePbZZ6u26lpG7aHCtfiHiE/RonnjOmKXQ0REVqbcSXXmzJmDgQMH4vDhw8jPz0f37t2Nz9sXuXnzJhQKBd58801Mnjy5qmqttR69217DsCciIpNVaG78gIAABAQEPHH9rl27SjyeR5bj1aBo2lzetyciItNVaG788jDoq1bDek6QSSWcI5+IiCrFImFPVUsmlaCxmwIJqVrkF+jL/wIREdFjGPZWwstDhfwCAxLvZYldChERWRmGvZXw+nsmPb4Uh4iITMWwtxLGd9tzjnwiIjIRw95KNHFXQhDYsyciItMx7K2EvZ0UDespEJeihd5gELscIiKyIgx7K6L2UCJHV4DU9GyxSyEiIivCsLciaveiyXV4KZ+IiCqOYW9FHo3I50x6RERUcQx7K6Ju8GiOfCIioopi2FsRhYMd6tdxQGyyBgYO0iMiogpi2FsZtYcKmqw8PNDqxC6FiIisBMPeyqj/vm/PQXpERFRRDHsr4+XB+/ZERGQahr2V4bS5RERkKoa9lXFRyuHsZMfH74iIqMIY9lZGEASoPVS4l5EDbXae2OUQEZEVYNhbIa+/n7e/w/v2RERUAQx7K2S8b89L+UREVAEMeyukNk6by549ERGVj2FvhdxcHOFoL+Wz9kREVCEMeyskEQR4uquQdD8LuboCscshIqIajmFvpdQeShgMwJ1U3rcnIqKyMeytFGfSIyKiimLYWymGPRERVRTD3ko1qOcEmVSC2CRexiciorIx7K2UTCpBEzcFEtK0yC/Qi10OERHVYAx7K6b2UCG/wIC7aZlil0JERDUYw96KFU2by5fiEBFRWRj2VqxoJj1OrkNERGVh2FuxJm5KCAJH5BMRUdkY9lbM3k6KhvUUiEvRQm8wiF0OERHVUAx7K+floUSurgCp6dlil0JERDUUw97KPXrdLS/lExFR6Rj2Vo5hT0RE5ZGJdeC8vDx8+OGHSEhIgE6nw+TJk9GiRQvMnj0bgiCgZcuWmDdvHiQSCUJDQ7F9+3bIZDJMnjwZffv2FavsGufRu+35+B0REZVOtLDfu3cvXFxcsGzZMqSnp2PEiBFo1aoVpk6dii5dumDu3Lk4ePAg2rdvj5CQEISFhSE3NxfBwcHo3r075HK5WKXXKAoHO9Sv44C4ZA0MBgMEQRC7JCIiqmFEC/tBgwZh4MCBxs9SqRRRUVEICAgAAPTq1QsnTpyARCJBhw4dIJfLIZfLoVarER0dDT8/P7FKr3G8PFSIuJaKdE0u6jo7iF0OERHVMKKFvUKhAABotVq89957mDp1KpYsWWLsmSoUCmg0Gmi1WqhUqmLf02rLv2Tt6uoEmUxq0Zrd3FTlbySC1t71EHEtFQ9zCuDbvGbW+Lia2o7WhG1oPrah+diG5quuNhQt7AEgMTERU6ZMQXBwMIYNG4Zly5YZ12VmZsLZ2RlKpRKZmZnFlj8e/k+Snp5l0Vrd3FRITa2Zg+DqqwpvaVy8loJm7gqRqylbTW5Ha8E2NB/b0HxsQ/NZug3LOnEQbTR+WloaJkyYgBkzZmDkyJEAgDZt2iA8PBwAcPToUfj7+8PPzw8RERHIzc2FRqNBTEwMfHx8xCq7RioakR9zNwMGTq5DRET/IFrPfvXq1cjIyMCqVauwatUqAMBHH32ETz/9FMuXL4e3tzcGDhwIqVSKsWPHIjg4GAaDAdOmTYO9vb1YZddILkp7NKznhIs372HN3ii8NrAVnBxEvWhDREQ1iGCw0a6gpS8v1fRLVmkPs/Ht3su4kfAQ9es4YNLwp+DdyFnsskqo6e1oDdiG5mMbmo9taL5acRmfLKt+HUfMGtMBzz/jhXsPc7B4cwR+CY/lnPlERMSwtyVSiQSBvZrj/dHtoXSyw87DMVgReh4PM3Vil0ZERCJi2Nug1k3r4r8TAuDXvB4u3bqPeetPI+rWfbHLIiIikTDsbZSzkxzvjfTD6H4tkJmdhy92nMPOP24gv0AvdmlERFTNGPY2TCIIeC5AjQ/HdoK7iyN++TMOn235C6kP+DpcIqLahGFfCzRr6Ix5r3dG17YeuHk3A/M3nMbpK8lil0VERNWEYV9LONrL8MbzbTBxaGvo9cDqPVHY+Es0cvMKxC6NiIiqGGdeqUUEQUD3dg3h3cgZa/ZE4ej5u7iR8BCTXmiLJu5KscsjIqIqwp59LdSwngIfvdYJz3ZqgrtpmViw6SwO/xXPqXaJiGwUw76WspNJETzAB+++1A5ymQQhv13Dqh8vITMnT+zSiIjIwhj2tVyHlm7474QA+Hi6IOJaKuavP43r8Q/ELouIiCyIYU+o6+yAma90wPAezXBfk4slWyKx7+Rt6PW8rE9EZAsY9gQAkEgEDO/RDDNf6YA6Sjl+PHoTn2+PRLomV+zSiIjITAx7KsZX7Yr/TghA+xb1ER33APPWn8b5G2lil0VERGZg2FMJSkc7vPtSO4wZ4IMcXT5W/nAB2w9e51S7RERWimFPpRIEAf07NcHHr/mjQV0n/HbmDhaGRCA5PUvs0oiIyEQMeyqT2kOFueP90b1dA8QmaTB/wxmcupQkdllERGQChj2Vy0Euw8ShbfDmsDYQAKz96TLW/XQZObp8sUsjIqIK4HS5VGFd2zaAdyNnrN4ThROXknDjbgYmvdAWXg1UYpdGRERlYM+eTOLu6oQPx3bCwABPJN/PwsKQs/j97B1OtUtEVIMx7MlkMqkEo/q1xNSXn4ajvQzbDlzHV2EXocnSiV0aERGVgmFPlebXvB7+OyEArb1cce5GGuZvOIOrcelil0VERP/AsCezuCjtMX1Ue7zU2xsPtTos3RaJ3cduokDPZ/KJiGoKhj2ZTSIRMLRbU8we0xF1VQ7Ye+I2lm2NxP2MHLFLIyIiMOzJglo0qYP/TuiMTr5uuBb/EPPWn0bktVSxyyIiqvUY9mRRTg52ePvFp/DaQF/o8vX4atdFbPntGvLyC8QujYio1mLYk8UJgoA+HRpjzjh/NKqvwMG/4vHppggk3ssUuzQiolpJMNjoA9KpqRqL7s/NTWXxfdYGuXkF2H7wOo6cuwuZVIK6dRxg0OshCAIEQYBEwN9/BgT8/d+/lxVbJwiQ4B+fjesLl0keW1fqPvDoMwSU/D6Kf7/c/T3++R+1lfZ9yT+2K3W9AABlHEcAXF2dkPEw+8nHw6PPMLktS//8pLqtFf9/Nh/b0HyWbkM3tydPcMYZ9KhK2dtJMW5QK7RpWhd7j99CXoEeBXoDDAYD9AYDDAbA8Nh/9eV8tskzUytW5kkRSi5/0kmJUHg28cSTs8f3i3JPWIq+/+QTK0cHOXS6PJR3YvXPE9HyTyj/ecySPzuAUo9RYt/lHLPc2ss6eQUqdOwnnQgCAvR6/t9oTRj2VC06t3JH51buZp/JFgW+KScIpZ0wFH3+5wmH8TNKX1/m/v9RV5n7/3v7J61H0X7xj+PoDXBykkObmVvy5yrteDAAZdX9z/2Xub+in7HsdtAXNmCJ/Ze2X73BgAK9oeTP/djfQ2l18eRPfI3qKzBhSGt4N3IWuxSqAIY9WZWiXgms+BKyuXj59JHyT/5Knrjp9QbUq6dAWpr20YlJmfsw8QSwAidQQNkngvq/T3yKTq5KO2bRz13aSReeUGvRz1t0Ivj4iVzRfko7USu+zoC8fD2ibt/H4s0RGNHLG4O6qI1XLahmYtgTkdWq7MlfvTqO0POtjWa5m56DZVvO4oc/YhB16z7+9XwbuKrsxS6LnoCj8YmIyGRP+7jhkwkBeLp5PVyJTce89adx7kaa2GXREzDsiYioUlROcrw30g9jBvggR1eA//1wAVt+57waNRHDnoiIKk0QBPTv1ARzxvmjYT0nHIyIx4LvI3A3jfNq1CQMeyIiMpunuxJzx3dG7/aNEJ+qxScbz+DIuQTY6FQuVodhT0REFlE0r8aUEU9BJpXg+/1XsWr3JWTm5IldWq3H0fhERGRRnXzd0bSBM9bui0LE1VTcSszAm8PawsfTRezSai327ImIyOLq1XHAzOCOeLFHM6RrcrFk61/YfewmCvR6sUurlRj2RERUJSQSAS/0aIZZwR1RV2WPvSduY+nWSNx7mCN2abUOw56IiKqUj6cL5k8IgH8rd1yPf4h560/jbHSK2GXVKgx7IiKqcgoHO0we3hbjB7dCvl6PVbsvYeMv0cjV8Zn86sABekREVC0EQUCvpxuhZZM6WLMnCkfP38X1+Ad464W2UHs8+fWsZD727ImIqFo1rKfAR6/5Y4C/JxLvZeHTTWfx+9k7fCa/CjHsiYio2tnJJHjl2ZaY+rIfHO1l2HbgOlb+cAEZWTqxS7NJDHsiIhKNX/P6+O+EALRp6ooLMfcwb/1pRN2+L3ZZNodhT0REonJR2uM/o9rj5b7Noc3Kw/Lt57DzjxvIL+Az+ZbCsCciItFJBAGDu3jhw7Gd4ObiiF/+jMPizRFISc8SuzSbwLAnIqIao1lDZ8x7vTOeeaoBbiVqMG/DGZy6lCR2WVaPYU9ERDWKo70M/3q+Dd4Y1gYCgLU/XcbafVHIzs0XuzSrxefsiYioRurWtgGaN3LGmr2XcSoqGTEJGXhreFs0a+gsdmlWhz17IiKqsdxdnfDBqx0xpKsXUh9kY1FIBH7+MxZ6PpNvEoY9ERHVaDKpBCP7NMf00e2hdLLDD3/EYPmOc3igzRW7NKvBsCciIqvQpmldfDIhAE83r4fLt9Mxd91pnLuRJnZZVoFhT0REVkPlJMd7I/0wZoAPcnQF+N8PF7D192vIy+cLdcrCsCciIqsiCAL6d2qCOeP80bCeEw5ExOPTTRG4m5Ypdmk1ltWEvV6vx9y5czFq1CiMHTsWsbGxYpdEREQi8nRXYu74zujdvhHupGjxycYzOHIugS/UKYXVhP2BAweg0+mwY8cOTJ8+HZ999pnYJRERkcjs7aQYN6gVpox4CjKpBN/vv4pvdl9CZk6e2KXVKFbznH1ERAR69uwJAGjfvj0uXbokckVERFRTdPJ1R9MGzli7Lwpnr6Yi5m4GmjeuI3ZZZfLxqotnOzSqlmNZTdhrtVoolUrjZ6lUivz8fMhkpf8Irq5OkMmkFq3BzU1l0f3VVmxH87ENzcc2NF9Na0M3NxWW/rs3Qg9cw47fr+JsdIrYJZXpdmIGgp71gVQiVPmxrCbslUolMjMfDb7Q6/VPDHoASLfwyxPc3FRITdVYdJ+1EdvRfGxD87ENzVeT2/DZDo3Qo607dHk1+6156iYuuH9Pa7H9lXXyZTVh37FjRxw+fBhDhgzBuXPn4OPjI3ZJRERUQznIZXCQi11F2ewsfPW5LFYT9gMGDMCJEycwevRoGAwGLFq0SOySiIiIrILVhL1EIsEnn3widhlERERWx2oevSMiIqLKYdgTERHZOIY9ERGRjWPYExER2TiGPRERkY1j2BMREdk4hj0REZGNY9gTERHZOIY9ERGRjRMMBoNB7CKIiIio6rBnT0REZOMY9kRERDaOYU9ERGTjGPZEREQ2jmFPRERk4xj2RERENo5hXw69Xo+5c+di1KhRGDt2LGJjY8Uuyerk5eVhxowZCA4OxsiRI3Hw4EGxS7Ja9+7dQ+/evRETEyN2KVZpzZo1GDVqFAIDA7Fz506xy7FKeXl5mD59OkaPHo3g4GD+Lpro/PnzGDt2LAAgNjYWr7zyCoKDgzFv3jzo9foqOy7DvhwHDhyATqfDjh07MH36dHz22Wdil2R19u7dCxcXF2zduhVr167FggULxC7JKuXl5WHu3LlwcHAQuxSrFB4ejsjISGzbtg0hISFISkoSuySrdOTIEeTn52P79u2YMmUKVqxYIXZJVmPt2rX4+OOPkZubCwBYvHgxpk6diq1bt8JgMFRpR4hhX46IiAj07NkTANC+fXtcunRJ5Iqsz6BBg/Dvf//b+FkqlYpYjfVasmQJRo8eDXd3d7FLsUrHjx+Hj48PpkyZgkmTJqFPnz5il2SVmjVrhoKCAuj1emi1WshkMrFLshpqtRpfffWV8XNUVBQCAgIAAL169cLJkyer7Nj8WyqHVquFUqk0fpZKpcjPz+cvuAkUCgWAwrZ87733MHXqVHELskK7du1C3bp10bNnT3z77bdil2OV0tPTcffuXaxevRrx8fGYPHky9u/fD0EQxC7Nqjg5OSEhIQGDBw9Geno6Vq9eLXZJVmPgwIGIj483fjYYDMbfP4VCAY1GU2XHZs++HEqlEpmZmcbPer2eQV8JiYmJeO211zB8+HAMGzZM7HKsTlhYGE6ePImxY8fiypUrmDVrFlJTU8Uuy6q4uLigR48ekMvl8Pb2hr29Pe7fvy92WVZn48aN6NGjB3799Vfs2bMHs2fPNl6WJtNIJI8iODMzE87OzlV3rCrbs43o2LEjjh49CgA4d+4cfHx8RK7I+qSlpWHChAmYMWMGRo4cKXY5VmnLli3YvHkzQkJC0Lp1ayxZsgRubm5il2VVOnXqhGPHjsFgMCA5ORnZ2dlwcXERuyyr4+zsDJVKBQCoU6cO8vPzUVBQIHJV1qlNmzYIDw8HABw9ehT+/v5Vdix2UcsxYMAAnDhxAqNHj4bBYMCiRYvELsnqrF69GhkZGVi1ahVWrVoFoHCgCgeaUXXq27cvzpw5g5EjR8JgMGDu3LkcP1IJ48ePx4cffojg4GDk5eVh2rRpcHJyErssqzRr1izMmTMHy5cvh7e3NwYOHFhlx+Jb74iIiGwcL+MTERHZOIY9ERGRjWPYExER2TiGPRERkY1j2BMREdk4hj2RjYiPj4evr6/xJRsAkJubi/Xr14tY1SPHjx/HhQsXjJ/Dw8Ph6+uLhQsXilgVUe3AsCeyYa+++qpxbgMxbd26FRMnTkRKSopxWePGjfHOO+8Y3z1BRFWHk+oQ2bB79+6JXQKA0uto0qQJ3n33XRGqIap92LMnIiKycQx7IhtUdP8+ISEBGo0Gvr6+mD17tnG9VqvF559/jmeffRZPPfUUevbsiXnz5pXogc+ePRu+vr64cOEChgwZgnbt2hmnjgaAw4cP41//+he6du2Ktm3bomvXrnj77bdx5coV4z7Gjh2Lr7/+GgAwZcoU+Pr6AnjyPftbt27h/fffxzPPPIOnnnoKzz77LJYuXVrijWBFtT18+BDz5s1D9+7d0a5dOwQGBuLXX3+1XGMS2QBexieyQc7OznjnnXfw/fffIzc3F2+++SZat24NANBoNAgODsa1a9fQrVs3PPfcc4iPj0doaCiOHTuG7du3w93dvdj+Jk+ejHbt2qF79+5wcnKCIAjYvHkzFixYALVajeeffx52dna4ePEiDh48iD///BP79++Hu7s7RowYAQA4ffo0hgwZAm9v7yfWff78eYwfPx45OTno27cvPD09ce7cOaxbtw6HDx/Gtm3bSry85vXXX8eDBw8wePBgZGVlYd++ffj3v/+NzZs3V+mLRYisCcOeyAY5Ozvj3XffxY8//oiMjIxi98aXL1+Oa9euYe7cuRgzZoxx+cGDB/H2229j4cKFWLlyZbH9dezYEV999ZXxs06nw5dffommTZvixx9/LPYilPnz52Pbtm04fPgwRo0ahcDAQCQkJOD06dMYOnQonn322VJrLigowMyZM6HT6bBmzRr06tXLuO7zzz/H2rVrsXTp0hIvo5JKpfjpp5+MNXTr1g3vv/8+QkNDGfZEf+NlfKJaJD8/H7t370bLli2LBT0A9O/fHx07dsTvv/8OrVZbbN0/38ZVUFCABQsWYOHChSXeeBYQEADA9MGBkZGRuH37NoYOHVos6AHgvffeg4eHB/bt2wedTlds3ZgxY4rV0Lt3bwDA7du3TTo+kS1jz56oFrl16xaysrJQUFBQrKdeJDc3FwUFBbh69So6depkXN64ceNi2zk6OmLIkCHGfcbExCAuLg7Xr1/HqVOnAAB6vd6k2oru83fu3LnEOrlcjnbt2uHAgQO4efMmWrVqZVzXrFmzYtsWvWv9nycFRLUZw56oFsnIyAAA3Lx50zhorjQPHz4s9tnBwaHENmfOnMHixYsRFRUFALC3t0erVq3Qtm1bJCYmwtS3ZxddTVAqlaWuLxpHkJ2dXWy5XC4v9lkQBAAw+fhEtoxhT1SLKBQKAMDw4cOxdOnSSu8nISEBb7zxBuzt7bFgwQJ06tQJTZs2hVQqxc8//4wDBw5UurbHJ955XNGJyj8H6BFR+XjPnqgWadasGeRyOaKiokrt+W7cuBGrVq1Cenp6mfs5cOAAsrOz8d577yEoKAjNmzeHVCoFAMTExAAo3rMu6m2XpehpgYiIiBLr9Ho9IiIi4OTkVOKWAhGVj2FPZMPs7OyQn59v/Gxvb48hQ4bgxo0b2LBhQ7Ftw8PDsXTpUoSFhaFOnTpl7tfe3h4AkJaWVmx5dHQ0Nm3aBADFjiuTFV5ELOs+eqdOneDl5YXffvsNR44cKbbuf//7HxITEzF48OASl+2JqHy8jE9kw9zd3XH79m28//776NGjB1588UXMmjULkZGRWLJkCQ4ePAg/Pz8kJyfjt99+g0wmw6JFiyCRlN0P6Nu3L7744gusWbMGN2/ehFqtRmxsLA4fPmwcIPfgwQPj9h4eHgCAb775BleuXME777xTYp8SiQSfffYZJk6ciEmTJqFv375Qq9WIjIzEuXPn0Lx5c8ycOdNyjUNUi7BnT2TDZsyYgZYtW2L//v3Ys2cPAKBu3boIDQ3FhAkTkJycjJCQEJw9exb9+vVDaGgounTpUu5+PTw8sGHDBnTt2hV//vkntm7dilu3bmHs2LH45Zdf4OLigmPHjhkv5Q8ZMgSDBw/GnTt3sHXrViQkJJS6344dO+KHH37AkCFDEBkZiS1btuDBgweYPHkydu7cyfv1RJUkGDhklYiIyKaxZ09ERGTjGPZEREQ2jmFPRERk4xj2RERENo5hT0REZOMY9kRERDaOYU9ERGTjGPZEREQ2jmFPRERk4xj2RERENu7/A0f9499JC1BxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x396 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_lines([compute_subopt(values_pi, solver.v)], [\"Subopt of $V^{\\pi_t}$\"], [\"Iteration\", \"Subopt\"], \"figs\", \"VI.pdf\", show = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0f59f72",
   "metadata": {},
   "source": [
    "### Compare with Value Iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "ffe5e4ea",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:52.968686Z",
     "start_time": "2023-03-14T00:56:52.399687Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAFoCAYAAABQeVI/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABCJ0lEQVR4nO3dd3hUZf7+8feUJKQBgYQWem+hi1SB0IsNXXRV1oId7AqiX3F3UWRdXEVX+CEiiqKCFAUXqaH3FnoLkISEFmp6mfL7I2QgJkBCJkwyc7+8uJJT5plPxmvOfcpznmOw2+12RERExG0ZXV2AiIiIFC+FvYiIiJtT2IuIiLg5hb2IiIibU9iLiIi4OYW9iIiIm1PYi4iIuDmzqwsoqL179zJ9+nTMZjNvvfUWwcHBN1w/ISHJqe8fFOTHxYupTm1TRKQk0PbNfYSEBOY7v9Qc2WdkZPD+++/TrVs3IiMjb/v7m82m2/6eIiK3g7Zv7q/UhH3btm2Jiorim2++oUmTJq4uR0REpNQoNWG/e/dumjdvztSpU/nhhx9cXY6IiEipUSLCfteuXQwdOhQAm83GmDFjeOihhxg6dCgxMTEApKSk8M477/DBBx/Qt29fV5YrIiJSqri8g97UqVNZsGABvr6+ACxfvpzMzExmzZpFZGQk48ePZ/LkyXTs2JGOHTu6uFoREZHSx+VhX7NmTb744gtGjhwJwPbt2+natSsArVq1Yu/evbfUblCQn9M7nVyvl6OISGmn7Zt7c3nY9+3bl7i4OMd0cnIyAQEBjmmTyYTFYsFsLlypzr6NJCQk0Om384mIlATavrmPUnPrXUBAACkpKY5pm81W6KAXERGRq0pc2Ldp04Y1a9YAEBkZScOGDV1ckYiISOlW4g6Ze/fuzfr163n44Yex2+2MGzfO1SWJiIiUaga73W53dRHFwdnXn3RNS0TclbZv7qPUXLMXERER51LYi4iIuDmFvYiIiJtT2IuIiLg5hb2IiIibU9iLiIi4OYW9iIgLvP76CLp0acfatatuuJ7VauXuu/vQu3dX0tLSCtz+iBHP0qVLO5KSXHdL3YMP3k2XLu3yzF+2bDHx8XH5vOL2io2NISJiea55Xbq044knHnFRRcVHYS8i4gL9+g0CYMWKZTdcb+vWzVy8eIHu3Xs6ng5amk2a9Dn/+Mf/kZqacvOVi9GRI4d5/PGH2bNnV675Tz75DPfdN9hFVRWfEjeCnoiIJ+jWrTt+fv6sX7+G9PR0ypQpk+96y5b9AUD//oNuZ3nF5uLFC64uAYCkpESysrLyzB827DkXVFP8dGQvIuICPj5l6NGjJ2lpaWzYsC7fddLT01mzZjVVq1ajdeu2t7lCcScKexERF8k5Wl+xYkm+y9etW01aWir9+g3EYDAAsHt3JO+88xb33tuX7t070K9fD1599UV27Nh2w/fasWMbXbq0Y+LET/Ise/vtt+nSpR1HjhzKNf/QoYOMHv0GAwb0JDy8M0888Qi//jqHWx1l/cEH7+aPP34H4MknH+XBB+92LLPb7fz66xyeeupRwsM7069fD0aNeo3Dhw/mamPRooV06dKOiIjlvP76CMLDO/HAA4McfQCOHYti7Nj3GDx4IN27d6BPn2688MJTrFq1wtHGtGlTePnl5wH45Zef6NKlnePzy++afXJyMpMmTWTIkHvp3r0Dd9/dh3/84/+IjY3Jt7Zt27bw44/f8/DD9xMe3okhQ+7lu++mYbVab+lzcwadxhcRcZGWLVtTtWooGzduIDU1BT8//1zLly5djMFgoF+/gQCsXbuK//u/UZQvH0TXrj3w8/Pj+PGjbNq0gZ07t/P11zNo0KCRU2rbuHE97777FmazF9269SAoKIjNmzcyYcJ4Dh06xKhR7xa6zSFD/sqiRb8TFXWYe+8dTK1atR3LPvjgfZYsWUSdOnW5777BpKWls3LlMp5/fhj//vdntG17R662Pvvs3wQHB/Pggw9x8mQ8oaHV2b9/Ly+99BxeXt507x5O+fJBxMWdcHxu//rXp3Tu3JXWrdty+vQp/vjjd5o2bc6dd3akatVq+dZ8+fIlXnhhGLGxMTRrFkbXrt04eTKeFSuWsmHDWv7zny9p1qx5rtdMnvwFsbHR9OjRi8DAQJYvX8LUqZOx2+088cTThf7cnEFhLyLiItlBPoDp06eybt0a+vTp71h2+fIltmzZSMuWrQkNrQ5kh0hAQADTp8+kQoWKjnVnzvyOyZO/ICJiuVPCPj09nQ8//Dt+fv5MnfqdIwiff/4lxowZzcKF87nrrm507NilUO0OGfIIR44cJirqMPfd94Cj1oiI5SxZsojevfvx7rt/x2zOjqahQ5/g6af/xgcfvM/s2b/h5eXlaMtsNjNp0rRcfR2+/noKFouFadN+oHbtOo75K1Ys4/33R7Ns2WI6d+5KmzbZdwj88cfvNGsWdsPr9JMmfU5sbAyPPz6MZ555wTF/48Z1jBz5GmPHjmHmzF8wmUyOZfHxJ5g+/UeqV68BwIMPPsxf/zqYBQvmK+xFRHLMjohi68Gzri7jhu5oXIkh4fWL3E6/fgOZPn0qy5cvzRX2ERHLsVgsjlP9NpuN554bgbe3V66gBxzX853V+W3dutVcunSRF198JdcRr9Fo5PnnR7Bq1Qr+97+FhQ776/n9998AePnlNxxBD1CtWij33fcAM2Z8w9atm+nU6er7dejQKU+nxoceeoSBA+/JFfQAbdrkfD4XC1VXVlYWy5cvoUqVqnl2CDp27EK3buGsWrWCXbt2OnYgALp1C3cEPUDVqtWoXbsuUVGHycjIwMfHp1B1OIPCXkTEhUJDqxMW1pKtWzeRlJREYGD2I0qXLVtMmTLZnfggO2i7desBwOnTpzh27Cjx8XFERx9zXG+22WxOqenQoYNXfh5g2rQpeZabTCaiog475b0ADh8+gLe3D/Pmzc6zLDY2GoAjRw7lCvv8TrvfeWdHAM6fP0dU1BHi4+OIjY1m9+5IAGy2wl0zj42NISMjgxYtWmE05u3i1qJFK1atWkFU1JFcYV+jRs086wYEBADZOxAKexERYEh4faccNZcW/fsPYs+eXaxeHcGgQfdy+vQp9uzZRZ8+/XNdxz96NIrPPvs3O3duB7JPZdeuXZfGjZty4kTsLXec+7Pk5OyBeFasWHrddRITE53yXgBJSUlYrVamT59a4PfLLzDPnDnNZ5/9m3Xr1mC32zEajdSoUZMWLVpx+PChQn8+KSnJAPj7B+S7PDg4BICMjPRc8729va/bprP+HxWWwl5ExMXCw3vz2WcTiIhYxqBB97Js2RLsdjsDBlztrZ6amsJrrw0nOTmZ4cNf5Y477qRWrdp4eXmxb99eli1bfMP3yOnNn1/Y/HlkvpzBeyZOnJynY1xx8PX1w8/Pj3nz/nfLbdjtdt566xWio48zdOiT3HVXd+rUqYuPTxkuXDjPwoW/FrrNnB2tc+cS8l2elJS9A1K2bLlbrvt20a13IiIuFhAQQNeud7F9+1aSkpJYuXI5lStXyXVqePv2rVy4cJ4HHhjCX//6GPXrN3B0WIuJOQ7c+KgxZ920tNQ8y06cOJFrul69BgAcPLg/z7qJiZeZOPETlixZVMi/MlvOTse16tdvQELCWc6fP5dn2fr1a/nqq0kcOXLjywZRUUc4duwod93Vg2effZHGjZvi45N9TT86+niB6vizWrVq4+3tw4ED+8jMzMyzfNeunQDUqVP3pm25msJeRKQE6NdvEFarld9+m8vhwwdz3VsP4O2dfdr6woXzuV53+vRpx+lvi8Vy3fZDQ2tgMpnYsWMb6elXTztv2LCOffv25Vr3rrt64O/vz8yZM/LcSz5p0uf88stPxMXl3kEoKJMp+4TytaPX9e8/CLvdzqeffpxr/rlz5/jkk/H88MO3+Pn53bBdH5/sU+d/7qSYmHiZSZM+B3J/PjkdAfMbRS+Hl5cXvXr14dy5hDx9FzZt2sCKFUupXr0GYWEtb1hbSaDT+CIiJUD79h2oWDGYb7/9Gsg7PG6LFq2oWrUaS5Ys4vLlS9Sv35CzZ8+wdu1qfHy8MRgMJCZevm77QUFBdO3ajVWrInjmmb/RsWMXTp6MZ9261bRt25bt27c71g0MDGTUqPf4xz/e5amnHuWuu3oQHBzMzp3bOXBgP02aNOWvfx16S39nSEj2de7//vcz2rVrz1NPPcuAAXezbt0aVq2K4OjRh7jzzo5YLFZWrlzG5cuXef75EY7bD6+nevWaNGnSjF27dvLii08TFtaSy5cvsXbtKjIzMylTpgyXL1+6po5KAKxcuQw/P1/69RtE3br18rT74ouvsGfPLmbO/I7IyB00b96CkyfjWb9+DX5+frz33tgCnSVwNR3Zi4iUACaTiT59+pOenk6LFq1y3boF2dfRP/30S7p168GhQweZO3cWhw8fpG/f/nz77c/Ur9+AXbt2kpqa9zR9jtGjx/Dggw+RmHiZOXNmcfr0KT744GP69OmTZ93w8F58+eVU2ra9g02bNjBnzmxSU1N54omn+eyzSTc90r6ewYOHcMcdd3Lw4AHmzJlFamoqBoOBDz74F6+88iZlypRh4cJfiYhYRu3adRk3bgKPPfbETds1Go2MH/8JAwbczalTJ5kz52ciI3fSoUMnpk37njvu6MCJE7GOkfaqVKnKM8+8gMFgYO7c2Rw4sC/fdsuXL8+UKd/y8MOPceHCeebNm83Bg/vp128g06b9kGdAnZLKYHdV18BilpDg3Mc6hoQEOr1NEZGSQNs39xESEpjvfB3Zi4iIuDmFvYiIiJtT2IuIiLg5hb2IiIibU9iLiIi4OYW9iIiIm1PYi4iIuDmFvYiIiJtT2IuIiLg5hb2IiIibU9iLiIi4OYW9iIiIm1PYi4iIuDmFvYiIi0ybNoUuXdrl+det250MGNCTESOeZcmSRbfc/o4d2+jSpR0TJ37imPfhh3+nS5d2HDlyyBl/QoGdOnWSLl3aMWLEs7nmZ2Rk8NNPP9zWWq5ny5ZNuR51m9/nV1qZXV2AiIin69q1G/XrN3RMW61WLl68yMqVyxg7dgwxMdE8++yLTnqv7lSpUpUKFSo6pb2iGjHiWU6ciOGvf33MpXXMnz+HTz4Zz7hxE2jSJHte1arVePLJZ2jWLMyltTmDwl5ExMW6du3OgAF355n/yCNDeeqpR5k58zvuued+qlSpWuT3uuuu7tx1V/cit+MsFy9ecHUJAFy4cD7PvKpVqzFs2HMuqMb5dBpfRKSEqlGjJl27dsdqtbJ580ZXlyOlmMJeRKQECw4OASAx8bJjns1mY/78OTz55COEh3emb99uvPrqi2zduumm7V3vmv3atat46aXn6NevOwMH9uSVV14kMnIHAOnp6fTp043Bgwdit9vztDlu3D/o0qUdJ07EFvjvyrmGf/r0KZKTk+nSpR0ffvh3x/KUlGQmT/6CIUPupUePjtx3X38mTPgoz5mAnL/nwIF9PPbYXwgP78Tzzz/lqHP9+rW88cbLDBrUi27d7mTQoF6MHv1Grr9/xIhnmT59KgDvvPMmXbq0A65/zT42NoZ//vM97r67D927d2DIkHuZNGkiycnJ+daWmJjIhAkfcc89fQkP78SwYUNZtWpFgT8rZ1DYi4iUYPHxJ4CroW+z2Xj//Xf45JPxpKSkMHDgPXTt2p2DB/fz+usvMW/eL4V+jylTpjB69JtERx+ne/ee9OzZh8OHD/LKKy+wZcsmypQpQ/fu4Zw9e4Zdu3bmem1GRgarV0fQrFkYNWrULPB7BgQE8uSTzxAQEIC3tzdPPvkMXbt2ByA5OZkXXhjGzJnfUbVqKH/5y8M0bx7GggXzeeaZxzl37lye9kaNep3Q0Orce+8DtGnTDoPBwNy5sxg16jXi4mLp1asvQ4Y8Qu3adVm7djXDhz/raGfAgLtp1aoNAD179ubJJ5+5bt379u1l2LDHWL58Cc2bh/HAA0MICqrAjz9+z3PPPZFrpyzHa68NZ9OmDYSH96J3734cP36U9957O89nWZx0zV5EpIQ6eHA/69atwcfHhw4dOgOwdOkfrFy5nPbtO/Lhhx/j6+sLQHx8HC+++DQTJ07gzjs7EhpavUDvERsbw+eff06tWrX5/PP/R8WKwQAMGfIITz75CF9++Rnt2/9Mv34DWbRoIcuXL3EEI2QfOaekpNC374BC/W2BgYEMG/Ycf/zxO8nJSbmujU+Z8iXHjh3l9ddHMXjwXxzz161bzdtvv8HEiRMYO3Z8rvbCwlrw4Yf/dkxnZmby1VeTqFGjJt98M9PxOQFMmDCeX3+dw/r1a7j33sEMGHA3p06dJDJyBz179r1unwar1crYsWPIzMzk448/o0OHTo5lkyd/wcyZ3/HllxMZPXpMrtcZjUa+/362o4a2bdvzz3/+HwsWzKdly9aF+txulcJeREqceVG/s/PsHleXcUOtK4UxuP4gp7S1du0qTp066Zi2WCycOBHLhg3rsFqtvPLKmwQFBQHwxx+/A/DGG6NyBVhoaHX+9ren+PTTj1m8+H8F7li2cuVyLBYLjz/+tCPoAapXr8Hw4a+SkZGOxWKhdeu2VKlSlZUrl/Pqq29hNmfHx9KlizCbzfTs2buIn8LVv33x4v9Rp07dXEEP0KVLN8LCWrJmzUpSUpLx9w9wLOvevWeudW02GyNH/h/BwcG5PieA1q3b8uuvcwrdOXDv3t3ExcXSt++AXEEPMGzYcyxd+gfLli3mjTfextvb27HsgQeG5KqhY8fsHbfCXPYoKoW9iIiLrV27mrVrVzumzWYz5cqVp337Oxk8eAjt23dwLDty5DAhIZXyPXJv0aIVAFFRRwr83jnrNm+e9/ay++57INd0nz79mTHjG7Zs2USnTl1ITLzM5s0b6dixM+XKlS/we95IbGwMaWmp2Gw2pk2bkmd5ZmYmVquVo0ejHH8vZPecv1aZMmUcOyCxsTFERx8nPj6O48ePsm3bFiB7h6Awcq7zX3tmI4e3tzeNGzdl7dpVxMRE06DB1Vspa9SolWvdgIDsnZSsrMxCvX9RKOxFpMQZXH+Q046aS4N33nk/31vv8pOSknzde+RzrutnZKQX+L2TkhIB8Pf3v+m6/foNZMaMb1i+fAmdOnUhImIZWVlZhT6FfyPJyUkAxMREOzrN5ScxMTHXtI+PT551IiN38Pnn/+Hw4YMAeHv7UL9+Axo1asLZs2fy7Wx4IykpKQC5zihc63qfv7e3V65pg8EAQCHfvkgU9iIipYifnz/nziXkuywnuMuWLVfg9nx9/QBITU3Nc3SekZGOl5c3RmN2X+6aNWvRrFkYa9euJjMzk4iI5QQGlqVTp6638JfcuJ6+fQfw3nv/vOV2Tp8+xZtvvoy3tw8jR75LixatqFGjJiaTiRUrlrJ27apCt+nnl71D5MzP/3ZRb3wRkVKkQYOGJCcncexYVJ5lOb2769SpW+D26tWrD8D+/fvyLPvsswn06tWFkyfjHfP69RtIWloqa9euZvfuSMLDe+W6Pl1YOUe5OWrWrIW3tzeHDh3M98h79uwf+fbbr7l8+dIN212zZhXp6ekMG/Yc99xzP7Vr18FkMgEQHX38pnXkJ+fU/O7deXvR22w2du+OxNfXzymDHzmbwl5EpBTp3z/78sbEiZ+QlpbmmH/yZDzTp0/FbDbTq1ffArfXu3c/jEYjM2Z8kytA4+PjiIhYRrVqoVSrFuqY37NnH7y8vJg8+XMsFkuRT+GbTGYsFotj2sfHh/Dw3kRHH+Pnn2fmWnfHjm18+eVE/ve/hQQGlr1huzk7IH/uhBcVdYRffvkZINf75nQ4zMrKum6bLVq0onr1GqxevZKNG9flWjZt2hTOnj1T5J2f4qLT+CIipUi/fgNZv34Nq1ZF8PjjD9OhQyfS0tJYu3Y1KSnJvPbayALfdgdQq1ZtRowYweeff84TTzxC585dsdvtrFixlMzMTEaPfj/X+mXLlqVz566sWhVBtWqhuTrJ3YqQkBDi4mL55z/f44477qR//0EMH/4qe/fu5ssvP2PdutU0bdqcs2fPsGbNSkwmM6NHv+e4tHA9nTt35f/9v//y/ffTiYmJJjS0OnFx2Xc45Fxzv3bnJiSkEgAzZkzjyJFD+d5rbzQaeffdv/P66y8xatTrdO7clWrVqrN372727dtD7dp1GD78lSJ9HsVFR/YiIqWIwWDgn/8cz6uvvomfnz+//76A9evX0rx5GBMnTs5zu1pBDB8+nH/84yMqVarM4sX/Y+nSP2jSpBlffPEVzZo1z7N+jx7Zvdyd0THvhRdeok6duqxcudzxhL+goCC++upbHn74MRISzjJnzs/s3h1J585dmTJlOm3atLtpuyEhlfjss0m0adOO7du3Mn/+L5w4EcuDDz7EzJlzKFeuHJs3b3RcKggP7014eG/i4+OYP/8Xzpw5lW+7YWEt+frrGYSH92bPnt3Mn/8LiYmXefzxYXz11Xcl8no9gMFe2O6IpURCQpJT2wsJCXR6myIiJUFht29TpnzJ999P5+ef51O9eo1irEwKKyQkMN/5OrIXEZECO3v2DL///htt2rRT0JciumYvIiI3tXTpH/z00/ecOBFLRkYGTz31rKtLkkJQ2IuIyE2FhFTizJkz+PsH8PLLb+Q7ipyUXLpmX0C6Zi8i7krbN/eha/YiIiIeSmEvIiLi5hT2IiIibk5hLyIi4uYU9iIiIm5OYS8iIuLmFPYiIiJuTmEvIiLi5hT2IiIibk5hLyIi4ubcdmz8mMQTTm0v3bs83nZ/jAbtH4mISOnitmH/8bYvnN6mnymAVpWa0apScxoG1cPL6LYfn4iIuBG3TausU7Wd2p7BK5OUcglsOLWZDac242XwpmmFRrSp3JymFRvj5+Xr1PcTERFxFrd96t22PSed2p7Ry8yaHTFsjztMsnccpqAzGMukAWDASJ3AOtxRNYyw4KYElSnv1PcWESlOeuqd+7jeU+/cNuyL6xG3drud+IQUdhxJYFvMMU5bjmUHv3+iY93KZarSrmoYrUKaU9W/MgaDwam1iIg4k8LefSjsi+h6X4ZLyRnsijrH1qMxRCUdhnJnMAZewGDM/ljLmsvTunJz2lQOo265WurgJyIljsLefSjsi6ggX4aMLCv7oy+w7Ug8exIOkOF3ClP5BAwmKwA+Bl+aVWzCHVXDaFyhId4mL6fWKCJyKxT27kNhX0SF/TLY7HaOn0xkx5EzbDt5gEvGWEzlz2LwzgDAiJl6gfW4M7QlYSFNCPDyd2q9IiIFpbB3Hwr7Iirql+HspTQiDyewOfYQcRlHMZY/g9E3JXuh3UA13+q0D21B60rNCfat6KSqRURuTmHvPhT2ReTML0NKehZ7jp1nc9RRDicewhp4GmPAJXL68QWZg2lTJYx2VcOoERCqDn4iUqwU9u5DYV9ExfVlsFhtHDlxic1Rsew6u5+0MvEYy55zdPDzNQTQvGJTOlRvSYPydTEZTU6vQUQ8m8LefSjsi+h2fBnsdjvx51LYdvgkW0/u45w9OruDn9kCgBlv6gc2pFONljQLbkQZc5lirUdEPIPC3n0o7IvIFV+Gy8kZ7Iw6y8bo/cSmR2Xf1ueTDoDBbiTUtzYdq7ekdeXmlPPJ/3+wiMjNKOzdh8K+iFz9ZcjIsrL/+AU2HDvIwUuHsAScwuh3pR47VPSqStsqzekQ2pLK/pVcVqeIlD6u3r6J8yjsi6gkfRlsdjvHTyWy6cgxIs/uI8nrBMbAi44Ofv6G8jSv2JQuNVtRu1xNDeQjIjdUkrZvUjQK+yIqyV+GhEtpbD58gq3xezhri8ZQNgGDyQaAl92XBmUb0bVWa5oEN9CT+kQkj5K8fZPCUdgXUWn5MqSmZ7Hz6BnWR+8hJi0Ke+BpDF5ZABjtZqqXqUunGq1oW7WZntQnIkDp2b7JzSnsi6g0fhksVhuHTlxkbdQ+Dlw6QKbfSceT+rAbCDZX544qYXSu1UpP6hPxYKVx+yb5U9gXUWn/MmQ/rS+ZdUcOE3l2H5dNsRgDrj6pL4Bgwio2pXudNoQGVtVAPiIepLRv3+QqhX0RuduX4XJKJpsOHWfzyd2cthyHgPOOgXy8bYE0CGxIt7ptaBJcTx38RNycu23fPJnCvojc+cuQmWVl5/FTrDu+i+jUI1gDzjie1Gey+VC9TF261GxNu9CmeJu8XVytiDibO2/fPI3Cvog85ctgs9s5evIiq47s5sClg6SXiXc8qQ+biUqmGrSr2oK76rQi0CfAtcWKiFN4yvbNEyjsi8hTvwxnL6aw6tB+IhP2cskYi8HxpD4IpAphFZoSXr8tVQNDXFuoiNwyT92+uSOFfRHpywCp6RbWHznC5vjdnMo6ht3v6kA+PtbyNAhsRI96bWkUXEsd/ERKEW3f3EepD/uNGzcyf/580tPTefHFF2ncuPEN11fYFy+L1cbu6JOsid7JsZTDWPwSMBizB/IxWf2o4VOPzrVacWeNpnpSn0gJp+2b+yj1YR8REUGPHj04cOAAGzZs4Omnn77h+gr728dutxN99hIRR3Zy4NIBUr3jHU/qw+pFJVMt2lUJo0f91vh560l9IiWNtm/uo9SHPUBqaipjx47lzTffpGLFijdcV2HvOheT0lhxaDeRZ/dywRiDwTv7SX3YjJS1V6V5hab0atiOyoFBri1URABt39xJqQ/7ixcvMmHCBEaMGEHVqlVvur7CvmTIyLSw/ughNsXt4mTWMexlsgfysduhjKUiDQIbEV6vLY0q13BxpSKeS9s391Giw37Xrl1MmDCB77//HpvNxt///ncOHTqEt7c3H3zwAbVq1WLkyJFcuHCB8uXL06tXL/r163fDNhX2JY/NbmdXbCxrju/keMoRMsskODr4mbICqeFTn841W3Fn7Ya6zi9yG2n75j5KbNhPnTqVBQsW4Ovry+zZs1m6dCkRERGMHz+eyMhIpkyZwuTJkwvdrsK+5Is5f44Vh7Ov86eYTzqe1IfFh0qG2rStGkZ4w5b4efu4tlARN6ftm/sosWG/ZMkSGjVqxMiRI5k9ezYfffQRLVq0YODAgQB07dqVtWvXFrpdi8WK2ayjw9LiUkoKC3ZuYfOJSBJs0WDOdCyz2w1gB7hyGsBuwHDN7475ZK+XvezqfAMGx3pX5xj+9POa+QYDhj8t+/N8x9w/zXNM5/lpzH6FwYgBA8Zr5hsN2dNGjNnTRiNGjBiNBkwGI8Yr/0wGAwZjzu9Xfhqv/jRdM228Zjr7nynXOiaTEbPBhNFowJyzzGTCbLjy02jEy2TCy2TGy2TCbDJiNpnxNpsxm0yYDAaMRg2jLFJauPzh5n379iUuLs4xnZycTEDA1ZHZTCYTFosFs7lwpV68mOq0GkF7vrdD/0bt6N+oHVlWC+ui9rMpfhcJGaexX/mPK/+y907t2A0588h3OYarr8SQPU3Osivz7Nf8fu18pw8T8Ke3dgd225Udqpydr5wdL7vRMZ29U5Q9bbAbyd5tMl6ze2S8skOUvYOTM200mLKnDKbsJY4dHNM1OzomTI7p7B0Uk9GMyWB07MCYjSbMRvOV382OnRiz0Yz3lR0Zb7MXXsYrP81mfEwmvExe+Jizl3vCTo22b+7jekf2Lg/7PwsICCAlJcUxbbPZCh30Urp5mcz0aNSCHo1auLQOm82GzW7Hep2fNmzYbHasNis2mw2r3Y7NfmWZzYaNK9M2e/br7Hbs1yy32m1Yr/y0O37Pad+KzW7DarNjtduu1HLltfbs11w7bfvTtON9yP5pv3Y5NuzX/LRzZTn2K9M5v9uyf78yL3vN7HYg97ycHa+cedk7Y1d+z1mWs0NlsBVthypnv8zmnP/PN30727U7MEbH79k7L9k7KEa7EQMmjJgwGIyYMGWfYcGEyWC+ZockewfFbDQ7fubskHgZzTSuVIv2tRvcnj9MPEqJS9E2bdqwcuVKBgwYQGRkJA0bNnR1SeKhsk+ng9mky0HFwWazYbFZsdisZFmtWKx2sqxZWKxWsnLm2axkWq1YrFlYbDbHPIvNitVmxWKzYLFasVxpy2q/uix7Z+rqz+ydJOuVfznzrFi5unNlI+dn9rwru0tXdmJsV3ZgbGCwYTNYwHBlR8eQPa9AOzB2wJr/ok2XvGhbcywmDzibILdXiQv73r17s379eh5++GHsdjvjxo1zdUkiUgyMRiPeRiPeeLm6FKexWK2kZ2WRYckkw2Ihw5JFptVCRpaFTGsmmVYrWdasa35asFgtZFktbLqwlixjCmcvplG1or+r/xRxMy7voFdc1BtfREqT/1v1Hy5YzjCsxhu0bVTptr63tm/u43rX7HWuSESkBPC+cvfQyXMpN1lTpPAU9iIiJYC3V/bmOF5hL8VAYS8iUgKYTUYwwMlzzr1tWAQU9iIiJYLBkD0M1OkLqVhtt+m+QvEYCnsRkRLEYrWRcCnd1WWIm1HYi4iUEIYrN+qrk544W7GEfVpaWnE0KyLiEdRJT5ytUGHfs2dPZsyYccN1/vvf/xIeHl6kokREPFHOAHynFPbiZDccQS8uLo7k5GTHdHx8PMeOHePgwYP5rp+VlcXGjRt1ZC8icisM4G026shenO6GYb9r1y7eeOMNx3Ukg8HArFmzmDVr1nVfY7fb6dy5s3OrFBHxEFUr+hN/LgWbzY7R6OzHL4qnumHYDxw4kP3793PhwgXsdju//vorjRs3pkmTJvmu7+XlRaVKlXj00UeLpVgREXdXLdifmDNJJFxOo3KQn6vLETdx0wfhvPXWW47ft2zZwuDBg/nb3/5WrEWJiHiqasHZAX8yIUVhL05TqKfeRUREFFcdIiJC9pE9wMnzKbQmxMXViLu4pUfc/vrrryxcuJCDBw+SlJREUFAQLVu25IEHHqBbt27OrlFExGOEXgl7ddITZypU2GdmZvLcc8+xadMm7HY75cqVo2bNmiQmJrJ06VKWLVvGfffdx0cffVRc9YqIuLXgcr54mY0aWEecqlBhP2nSJDZu3Ejv3r0ZOXIkNWrUcCyLj49n/Pjx/PrrrzRr1ozHHnvM6cWKiLg7o9FA1Yp+nDqfqh754jSFGlRn4cKFNG3alIkTJ+YKeoDQ0FA+/fRTGjRowM8//+zUIkVEPEm1YH+yLDbOXdaYJeIchQr7c+fO0aFDB4zG/F9mNpvp2LEjcXFxTilORMQTVat4pZOeHncrTlKosG/QoAF79+694TpRUVHUrl27KDWJiHi0q530km+ypkjBFCrs33rrLXbu3MkHH3xAUlJSnuVTp05ly5YtvPPOO04rUETE0zhuv9ORvThJoTrozZ07l9q1azNz5kzmz59Po0aNqFy5Munp6ezbt4+EhAT8/f3z9MY3GAzMmzfPqYWLiLirkPK+mE1GTp5Xj3xxjkKF/YIFCxy/p6SksGPHjjzrJCcnc+DAgVzzcsbWFxGRm7vaIz8Fm92OUdtQKaJChf31nnYnIiJFlTvQqwX7c+JsMucvpxNS3tdFNYm7KNQ1exERuT2qaSQ9caJbGi5327ZtzJ07l0OHDpGWlkb58uVp0KAB99xzD+3atXN2jSIiHifn9rtT51JoVT/YxdVIaVfosP/kk0/4+uuvsdvtAPj6+hIdHc3OnTv55ZdfePbZZ3nttdecXqiIiCcJDcnpka8jeym6Qp3GX7RoEVOnTqV+/fpMmTKFbdu2sXPnTnbt2sU333xDo0aN+Oqrr1i+fHlx1Ssi4hFCypfBbDLoNL44RaHCfsaMGYSEhDBjxgy6detGQEAAAN7e3nTq1IlvvvmG4OBgvv/++2IpVkTEU5iMRqpUuDJG/pUzqSK3qlBhf+jQIXr06EFQUFC+yytUqECPHj3y3HonIiKFVy3Yn4wsKxcup7u6FCnliqU3flZWVnE0KyLiURwj6WlwHSmiQoV9o0aNWLlyJZcuXcp3+YULF4iIiKBRo0bOqE1ExKOFathccZJChf3f/vY3EhISGDZsGFu2bMFisQDZo+atXr2aJ554gvPnz+tZ9iIiTlBND8QRJynUrXcDBgxgz549TJ8+nccffxyj0Yi3tzfp6dnXk+x2O08++SSDBg0qlmJFRDxJpSBfTEaDjuylyAp9n/2oUaPo2bMn8+bN4+DBg6SkpODv70/jxo0ZPHiwBtUREXESk9FIlYp+nDyfgt1u13NG5Jbd0gh67dq1U6iLiNwG1Sr6E5+QwoXEDCqWK+PqcqSUuqWwT05OZvny5Rw8eJDU1FTKlStH8+bN6dGjB97e3s6uUUTEY4UG+7OV7DHyFfZyqwod9rNmzeJf//oXaWlpjiFzIfsxthUqVGDcuHF069bNqUWKiHgqx+1351JoUa+ii6uR0qpQYb9kyRLef/99goODef7552nRogX+/v6cPXuWnTt38tNPPzF8+HC+//57WrduXVw1i4h4DN1rL85QqLD/+uuvCQoKYvbs2VSrVi3Xsp49e3L//ffz0EMP8dlnn/Hdd985tVAREU90tUe+wl5uXaHusz98+DB9+/bNE/Q56tWrR58+fdi9e7dTihMR8XRmk5HKFfw4eS4l16VTkcIoVNgHBQVhs9luuI6vry/+/v5FKkpERK6qFuxPeqaVi0kZri5FSqlChf2QIUNYuHDhdR90Exsby++//87gwYOdUpyIiEC1in6Anm0vt+6G1+xnzJiRa9rPzw9/f3/+8pe/cO+999K6dWuCg4NJTExkz549/PbbbwQFBdG0adNiLVpExJOEhmQ/TvzkuRSa11WPfCm8G4b9uHHjMBgMjutE1/4+d+5c5s6d6xjRKWd+YmIir732Gv369SvOukVEPEbOkX28juzlFt0w7D/66KPbVYeIiMe7Xge8yhX8snvk6/Y7uUU3DPv777//dtUhIiLXYTYZqRTk6+iRrzHypbAK1UFPRESKx83iu1qwP2kZVi4lZ96WesS9FGpQnfbt2xdoPYPBwObNm2+pIBERySs02J/thxI4eS6FoEAfV5cjpUyhwj4gICDf+enp6Vy6dAmbzUbDhg2pUaOGU4oTEZFsOcPmxp9LoVmdCi6uRkqbQoV9RETEdZclJSUxefJk5s6dy6efflrkwkRE5KprH4gjUlhOu2YfGBjIyJEjqV+/Pv/+97+d1ayIiACVg/wwGjRGvtwap3fQa926Ndu2bXN2syIiHs3LnLtHvkhhOD3sDxw4oNtCRESKQWiwP6kZFi6nqEe+FE6hrtmvWLEi3/l2u53U1FRWrVrFhg0b6N27t1OKExGRq6oG+8PhBOLPpVA+QD3ypeAKFfbDhw+/4VG73W6nUqVKvPnmm0UuTEREcgu9ppNes9rqkS8F57Sw9/b2pm7dunTr1g0vLy+nFCciIlepR77cqkKF/UsvvZRnXkZGBqdPnyY4OFjPsRcRKUZVKvhiMCjspfAK1EEvIiKC0aNHc/DgwVzzP/nkEzp06EC/fv1o3749r776KhcvXiyWQkVEPJ2X2USlID/1yJdCu+mR/ZgxY/jll18A6N69O40bNwbgP//5D1OnTsVgMNCpUycAli5dSlRUFPPmzcPb27sYyxYR8UzVKvqx80IqiSmZlFMnPSmgGx7ZR0REMHv2bJo0acLXX39N9+7dAThz5gzffPMNBoOBsWPHMm3aNKZNm8YXX3xBVFQUM2bMuB21i4h4nNAQXbeXwrth2M+ZM4fy5cszY8YMOnfujI9P9l7k4sWLsVgs1KxZkwcffNCxfs+ePWnTpg2LFy8u3qpFRDxUtYpXx8gXKagbhv3u3bvp3r17ngfgbNiwAYPBQHh4eJ7XtGzZkpiYGOdWKSIiwDU98s+nurgSKU1uGPaXL1+mcuXKuebZbDa2b98OQMeOHfO8xmw2k5WV5cQSRUQkR5UKftk98hOSXV2KlCI3DPvAwMA8vet3795NcnIyZrOZO+64I89roqOjCQoKcm6VIiICgLeXiZDyvsSrR74Uwg3DPiwsjA0bNmCz2Rzzfv/9dyD7qN7X1zfX+gkJCaxbt46wsLBiKFVERCB7JL2UdAtJqTqLKgVzw7AfMmQIcXFxvP7662zdupWZM2cya9YsDAYDjz76aK51L1y4wKuvvkp6ejr33HNPsRYtIuLJcq7bq5OeFNQN77Pv2bMnjz76KDNnzmTJkiVA9vj3jzzyCN26dXOs9/zzz7Nx40YyMjLo168fvXr1Kt6qRUQ82LXD5jappcumcnM3HVTnvffeo2/fvqxcuRKLxULnzp0d99vnOHbsGP7+/jz77LO88MILxVWriIhw9fY73WsvBVWgsfHbt29P+/btr7t83rx5eW7PExGR4lGloh8GFPZScAUaG/9mFPQiIkV1/ceH/5nPlR75J88r7KVgnBL2IiJye1UL9icpNYvE1ExXlyKlgMJeRKQUyumkd0qn8qUAFPYiIqVQtWA/QLffScEo7EVESqFrb78TuRmFvYhIKVS1or965EuBKexFREohHy8TFcuVUdhLgSjsRURKqdBgfxJTs0hSj3y5CYW9iEgppev2UlAKexGRUsoR9udTXVyJlHQKexGRUsoR9gk6spcbU9iLiJRSVStm32uvYXPlZhT2IiKlVBlvM8HqkS8FoLAXESnFqgX7czklk+S0LFeXIiWYwl5EpBTTs+2lIBT2IiKlmG6/k4JQ2IuIlGIKeykIhb2ISCmmHvlSEAp7EZFSzNfHTMWyPnrUrdxQqQv7jRs38u6777q6DBGREqNacACXkzNJSVePfMlfqQr7mJgY9u/fT0ZGhqtLEREpMaoFXzmVr6N7uY5SFfa1atVi2LBhri5DRKRE0e13cjOlKuxFRNyZHfstva5aSE7Y64E4kr8SE/a7du1i6NChANhsNsaMGcNDDz3E0KFDiYmJcXF1IiIlV86RfVxCMlkWa6H/ZWYV/jU2263tmIhrmF1dAMDUqVNZsGABvr6+ACxfvpzMzExmzZpFZGQk48ePZ/LkyY71J0yY4KpSRUSKhcFw66/19TFToawPB2Iu8tyE1c4r6gb8y5h5uGcDOjWvgqEoxcttUSLCvmbNmnzxxReMHDkSgO3bt9O1a1cAWrVqxd69ewvdZlCQH2azyal1hoQEOrU9EZEcXl7Zm+Nb3c481q8J63addGZJ12W32zkYc4Fp/zvA3uiLDH+wJUFly9yW95ZbUyLCvm/fvsTFxTmmk5OTCQgIcEybTCYsFgtmc8HLvXjRudeuQkICSUhIcmqbIiI5srIsALe8nWlVtwKt6la4pdfeyvbt3KU0vll0gM37TrPv2Hke69OQ9k0q39L7i/Ncb2exxFyzv1ZAQAApKVd7ldpstkIFvYiIFK/g8r68+dfWPNKrAZlZVv7fb/uY/OteklIzXV2a5KNEhn2bNm1Ys2YNAJGRkTRs2NDFFYmIyJ8ZDQZ6tavBP55qT73Qsmw9eJb3vt7MzsMJri5N/qREhn3v3r3x9vbm4Ycf5qOPPmL06NGuLklERK6jcgU/Rj/alr/0qEdqhpUv5u1h6sL9GtGvBDHY7Xa3vH/C2dfXdc1eRIrTpzsmE3XpOF+Gf3zb39uZ27f4cylM+30/0aeTKB/gzZMDmhBWt6JT2pabK1XX7EVEpHQKDfbnnaFtub9rHZJSs/h09i6+/eMAaRkWV5fm0RT2IiLiVGaTkbs71+G9x9tRPSSANbtO8Y9vt3LWyXdJScEp7EVEpFjUrBzImCfa0a99Tc5eTGPc99uJOa3Loa6gsBcRkWJjNhkZEl6fR3s3JCk1i/E/7mBf9AVXl+VxFPYiIlLseratzgv3NcdqtfHZ7F1s2nfa1SV5FIW9iIjcFu0aV+KNh1rh7WXiq4X7WbIl1tUleQyFvYiI3DaNagYx+tE2lA/wZlZEFLMjorC55x3gJYrCXkREbqvqlQJ4Z2hbqlb0Y/GWWL7+fT8Wq83VZbk1hb2IiNx2weV8Gf1YW+qFlmXTvjNMnLNb9+IXI4W9iIi4RICvF28+3JqW9Sqy7/gFPv5pJ4kpepBOcVDYi4iIy/h4mRjxQBhdW1Ql5nQS477frsF3ioHCXkREXMpkNPJE/8bc3ak2Zy+lMeHnSJLT9BAdZ1LYi4iIyxkMBu6/qy53d6rNucvpTPltLzabeuk7i8JeRERKjHu71sm+hh99kbmrj7q6HLehsBcRkRLDaDDwzN3NqFzBjz82x7LlwBlXl+QWFPYiIlKi+JUxM2JwGD7eJr5ZdIATZ5NdXVKpp7AXESkBDBhcXUKJEhrsz9MDm5KZZeO/83arw14RKexFRKREatsohEGdapNwKZ2vFuxTh70iUNiLiEiJdV/XOrSoV5G9xy8wb80xV5dTainsRUSkxDIaDDx7d1MqBfmyaFMMWw+edXVJpZLCXkRESjS/Ml68lNNh738HiEtQh73CUtiLiEiJFxoSwNMDm5CRZeW/c/eQkq4Oe4WhsBcRkVKhbaNKDOxYi7OX0vhqwX512CsEhb2IiJQa93etS1jdiuw5dp4F64+7upxSQ2EvIiKlhtFo4Nl7mlKxrA+LNsVw7nKaq0sqFRT2IiJSqviX8eL+u+pisdpZsC7a1eWUCgp7EREpdTo0rUJosD/r954i/lyKq8sp8RT2IiJS6hiNBgbfVRe7HX7VYDs3pbAXEZFSqVWDYOpVK8v2wwkcP5Xo6nJKNIW9iIiUSgaDgcHd6gEwd/VRF1dTsinsRUSk1GpSK4hmdSqwP/oiB6IvuLqcEkthLyIipdoD3eoCMGf1Mex2DbSTH4W9iIiUarWrlKVdoxCOn0pk55Fzri6nRFLYi4hIqXf/XXUxGGDemmMaRjcfCnsRESn1qlb0p3NYVU6eS2HjvtOuLqfEUdiLiIhbuLdzHcwmA7+tO47FanN1OSWKwl5ERNxCxXJl6NG6Oucup7M68qSryylRFPYiIuI2BnaqhY+3iYUboknPtLi6nBJDYS8iUoLo1rGiKevnTd87apCYksmybXGuLqfEUNiLiIhb6du+JgG+XizeHEtyWparyykRFPYiIuJWfH3MDOhQi7QMC39sinF1OSWCwl5ERNxOeJtQggJ9WL49Tkf3KOxFRMQNeXuZCG8TSpbFxp5j511djssp7EVExC21qBcMwJ6jCnuFvYiIuKXqIf4EBfqw59h5jx9CV2EvIiJuyWAwEFa3IinpFo6dSnR1OS6lsBcREbfVol5FQKfyFfYiIuK2mtQKwmQ0sNvDO+kp7EVExG35+phpWKM8MaeTuJyc4epyXEZhLyIibs1xKv/YBRdX4joKexERcWs5Ye/Jp/IV9iIi4taqVPAjuFwZ9h2/4LHPuVfYi4iIWzMYDLSoV5G0DAtH4y+7uhyXUNiLiIjb8/RT+Qp7ERFxe41qBmE2GT32fnuFvYiIuD0fLxONa5UnLiGFC4npri7ntlPYi4iIR2hRN+cWPM87ulfYi4iIRwjLuW7vgafyFfYiIuIRKgf5UbmCH/tjLpJl8axb8BT2IiLiMVrUrUhGppUjcZdcXcptpbAXERGP0cJDT+Ur7EVExGM0rFEeby+jx3XSU9iLiJQgduyuLsGteZmNNK1VgVPnUzl7Kc3V5dw2CnsRkRLAgMHVJXgMx1PwPOhUvsJeREQ8SpgH3m+vsBcREY9SsVwZQoP9ORBzkcwsq6vLuS0U9iIi4nHC6lUky2LjYOwlV5dyWyjsRUTE43ja0LkKexER8Tj1q5ejjLeJ3UfPYbe7/x0QCnsREfE4ZpORZnUqkHApnTMX3f8WPIW9iIh4pJxT+Z4wmp7CXkREPFJzR9ifc3ElxU9hLyIiHiko0IcKZX04cyHV1aUUO4W9iIh4LKPBM0YuVNiLiIi4OYW9iIiIm1PYi4iIuDmFvYiIiJtT2IuIiLg5hb2IiIibU9iLiIi4OYW9iIiIm1PYi4iIuDmFvYiIiJtT2IuIiLg5hb2IiIibM9jtdrurixAREZHioyN7ERERN6ewFxERcXMKexERETensBcREXFzCnsRERE3p7AXERFxcwp7ERERN6ewv0U7duxg1KhRjBo1isTERFeXIyLiVBs3buTdd991dRniJAr7WzR79mz++c9/8uCDD7Jo0SJXlyMi4jQxMTHs37+fjIwMV5ciTqKwv0VWqxUfHx9CQkJISEhwdTkiIk5Tq1Ythg0b5uoyxIkU9rfI19eXzMxMEhISCA4OdnU5IiIi16Wwz8euXbsYOnQoADabjTFjxvDQQw8xdOhQYmJiABgyZAhjxozh559/5p577nFluSIiBVaQ7Zu4H7OrCyhppk6dyoIFC/D19QVg+fLlZGZmMmvWLCIjIxk/fjyTJ0+mefPmjB8/3sXViogUXEG3bzkmTJjgqlLFyXRk/yc1a9bkiy++cExv376drl27AtCqVSv27t3rqtJERIpE2zfPpbD/k759+2I2Xz3hkZycTEBAgGPaZDJhsVhcUZqISJFo++a5FPY3ERAQQEpKimPaZrPl+rKIiJRW2r55DoX9TbRp04Y1a9YAEBkZScOGDV1ckYiIc2j75jm0C3cTvXv3Zv369Tz88MPY7XbGjRvn6pJERJxC2zfPYbDb7XZXFyEiIiLFR6fxRURE3JzCXkRExM0p7EVERNycwl5ERMTNKexFRETcnMJeRETEzSnsRURE3JzCXsRNxMXF0ahRI8fjSwEyMjL45ptvXFjVVevWrWP37t2O6c2bN9OoUSM+/PBDF1Yl4hkU9iJu7LHHHmPSpEmuLoMff/yRYcOGcfbsWce80NBQRowY4XjqmogUHw2XK+LGzp8/7+oSgPzrqF69Oi+99JILqhHxPDqyFxERcXMKexE3lHP9Pj4+nqSkJBo1asTbb7/tWJ6cnMyECRPo1asXzZs3p2vXrrz//vt5jsDffvttGjVqxO7duxkwYABhYWGOh6YArFy5kqeffpoOHTrQrFkzOnTowIsvvsiBAwccbQwdOpT//ve/AAwfPpxGjRoB179mf/z4cd588006depE8+bN6dWrFx9//DFJSUn51nb58mXef/99OnfuTFhYGIMHD2bJkiXO+zBF3IBO44u4obJlyzJixAi+++47MjIyePbZZ2nSpAkASUlJPPLIIxw+fJiOHTvSp08f4uLimD17NmvXruXnn3+mUqVKudp74YUXCAsLo3Pnzvj5+WEwGPjhhx8YO3YsNWvWZNCgQXh5ebFnzx5WrFjBpk2bWLx4MZUqVeL+++8HYMuWLQwYMIC6detet+5du3bxxBNPkJ6eTo8ePahRowaRkZFMmzaNlStX8tNPP1G+fPlcr3nyySe5dOkS/fv3JzU1lYULF/LKK6/www8/0K5dO+d+sCKllMJexA2VLVuWl156ifnz55OYmJjr2vh//vMfDh8+zJgxY3j00Ucd81esWMGLL77Ihx9+yMSJE3O116ZNG7744gvHdGZmJp9++im1a9dm/vz5+Pn5OZb9/e9/56effmLlypU89NBDDB48mPj4eLZs2cLAgQPp1atXvjVbrVZGjhxJZmYmU6ZM4a677nIsmzBhAlOnTuXjjz/O8xhWk8nE77//7qihY8eOvPnmm8yePVthL3KFTuOLeBCLxcKvv/5KgwYNcgU9QM+ePWnTpg3Lli0jOTk517K+ffvmmrZarYwdO5YPP/wwV9ADtG/fHih858CdO3cSHR3NwIEDcwU9wMsvv0zlypVZuHAhmZmZuZY9+uijuWro1q0bANHR0YV6fxF3piN7EQ9y/PhxUlNTsVqtuY7Uc2RkZGC1Wjl06BBt27Z1zA8NDc21nq+vLwMGDHC0efToUWJjYzly5AgbN24EwGazFaq2nOv8d9xxR55l3t7ehIWFsXz5co4dO0bjxo0dy+rUqZNr3cDAQIA8OwUinkxhL+JBEhMTATh27Jij01x+Ll++nGu6TJkyedbZunUrH330Efv27QPAx8eHxo0b06xZM06dOuXoxFdQOWcTAgIC8l2e048gLS0t13xvb+9c0waDAaDQ7y/izhT2Ih7E398fgHvvvZePP/74ltuJj4/nmWeewcfHh7Fjx9K2bVtq166NyWRi0aJFLF++/JZru3bgnWvl7Kj8uYOeiNycrtmLeJA6derg7e3Nvn378j3y/fbbb5k0aRIXL168YTvLly8nLS2Nl19+mSFDhlCvXj1MJhMAR48eBXIfWeccbd9Izt0C27dvz7PMZrOxfft2/Pz88lxSEJGbU9iLuDEvLy8sFotj2sfHhwEDBhAVFcX06dNzrbt582Y+/vhj5s6dS7ly5W7Yro+PDwDnzp3LNf/gwYPMmDEDINf7ms3ZJxFvdB29bdu21KpVi6VLl7J69epcyz7//HNOnTpF//7985y2F5Gb02l8ETdWqVIloqOjefPNN+nSpQv33Xcfo0aNYufOnfzrX/9ixYoVtGjRgjNnzrB06VLMZjPjxo3DaLzxcUCPHj345JNPmDJlCseOHaNmzZrExMSwcuVKRwe5S5cuOdavXLkyAJMnT+bAgQOMGDEiT5tGo5Hx48czbNgwnn/+eXr06EHNmjXZuXMnkZGR1KtXj5EjRzrvwxHxIDqyF3Fjb731Fg0aNGDx4sX89ttvAFSoUIHZs2fz1FNPcebMGb7//nu2bdtGeHg4s2fP5s4777xpu5UrV2b69Ol06NCBTZs28eOPP3L8+HGGDh3KH3/8Qfny5Vm7dq3jVP6AAQPo378/J06c4McffyQ+Pj7fdtu0acOcOXMYMGAAO3fuZObMmVy6dIkXXniBX375RdfrRW6Rwa4uqyIiIm5NR/YiIiJuTmEvIiLi5hT2IiIibk5hLyIi4uYU9iIiIm5OYS8iIuLmFPYiIiJuTmEvIiLi5hT2IiIibk5hLyIi4ub+P2GgJOF+zUfNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x396 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_log_lines([compute_subopt(values, solver.v), compute_subopt(values_pi, solver.v)], \n",
    "               [\"Value Iteration\", \"Policy Iteration\"], \n",
    "               [\"Iteration\", \"Subopt\"], \"figs\", \"VIvsPI.pdf\", show = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db0b2058",
   "metadata": {},
   "source": [
    "***Questions :*** \n",
    "- Which algorithm converges faster ?\n",
    "- Is it theoretically expected ? Answer listing the expected number of iterations needed for VI and PI to converge ?\n",
    "- Compare the run time with evaluation by fix point and solving linear system.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd941be6",
   "metadata": {},
   "source": [
    "***Answers***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1a4abde",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e597341b",
   "metadata": {},
   "source": [
    "***Theory Questions*** \n",
    "- Prove the policy improvement theorem.\n",
    "- Prove that the Bellman operator is monotone elementwise."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ce2eeb4",
   "metadata": {},
   "source": [
    "***Answers***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbb94b14",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b00d1641",
   "metadata": {},
   "source": [
    "# Ex 3: Modified Policy Iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69202e2f",
   "metadata": {},
   "source": [
    "In this cell you will code and run Modified Value Iteration, that follows this pseudocode:\n",
    "- $\\pi_{k+1} = \\mathcal{G}(V_k)$\n",
    "- $V_{k+1} = T^m_{\\pi_{k+1}}(V_k)$\n",
    "\n",
    "In the implementation, we will use also the variable $Q$ as in the previous implementation of VI and PI for convenience of implementation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dba68c07",
   "metadata": {},
   "source": [
    "First, write a function that applies the Bellman evaluation operator $m$ times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "39f4b211",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:53.095723Z",
     "start_time": "2023-03-14T00:56:52.969686Z"
    }
   },
   "outputs": [],
   "source": [
    "def policy_operator_m_times(pi, m, env, v, q, tol=1e-10):\n",
    "    \"\"\"Implementation of partial policy evaluation through applying m times the Bellman operator\n",
    "    Args:\n",
    "        pi: a policy\n",
    "        env: environment\n",
    "        tol: a scalar to dermerminate whether the policy evaluation convergences\n",
    "        v: initial value vector\n",
    "        q: initial state action value vector\n",
    "    Returns:\n",
    "        v: an array with the values of the actions chosen\n",
    "        q: an array with the q values    \n",
    "    \"\"\"\n",
    "    i = 0\n",
    "    while i < ???:\n",
    "        for a in range(env.n_actions):\n",
    "            q[:, a] = ???\n",
    "        for s in range(env.n_states):\n",
    "            v[s] = ???\n",
    "        i = i + 1\n",
    "    return v, q"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "740d6d8d",
   "metadata": {},
   "source": [
    "At this point, using the function `get_greedy_policy` implemented for PI you are ready to implement Modified Policy Iteration !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "26f61e63",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:56:53.220688Z",
     "start_time": "2023-03-14T00:56:53.097686Z"
    }
   },
   "outputs": [],
   "source": [
    "def modified_policy_iteration(env, m, tol=1e-10):\n",
    "    \"\"\"Implementation of modified policy iteration\n",
    "    Args:\n",
    "        env: environment\n",
    "        tol: a scalar to dermerminate whether the algorithm convergences\n",
    "    Returns:\n",
    "        vs: value functions at each iteration (list)\n",
    "    \"\"\"\n",
    "    vs = []\n",
    "    policies = []\n",
    "    v = np.zeros(env.n_states)\n",
    "    q = np.zeros((env.n_states, env.n_actions))\n",
    "    pi = np.zeros(env.n_states, dtype=int)\n",
    "    while True:\n",
    "        v_old = np.copy(v)\n",
    "        pi = get_greedy_policy(q)\n",
    "        v, q = ???\n",
    "        policies.append(???)\n",
    "        vs.append(v)\n",
    "        if np.linalg.norm(v - v_old) < tol:\n",
    "            break\n",
    "    return vs,policies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0c4c515",
   "metadata": {},
   "source": [
    "Now run modified policy iteration for different values of $m$ for example between $1$ and $10$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "fc31ccd2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:57:06.759601Z",
     "start_time": "2023-03-14T00:56:53.221722Z"
    }
   },
   "outputs": [],
   "source": [
    "m_values = [\"1\", \"2\", \"3\", \"5\", \"10\", \"20\"]\n",
    "to_plot = []\n",
    "for m in m_values:\n",
    "    m = int(m)\n",
    "    _, policies_mpi=modified_policy_iteration(gridworld, m)\n",
    "    values_mpi = evaluate_policy_sequence(policies_mpi, gridworld)\n",
    "    to_plot.append(compute_subopt(values_mpi, solver.v))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "6d3dd3c9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:57:07.313599Z",
     "start_time": "2023-03-14T00:57:06.761599Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAFoCAYAAABQeVI/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABKtklEQVR4nO3dd3hUZd4+8PucMyUzmUkjCYSE0Am9hNBFREAUyyoooIji4rrqoqv7Cuy6r+VFRVTU3dWfrIKAoCBRRMGKCIp0CCT0XhNKCmmTZOo5vz8mmRBSIDAzJ5m5P9eVK1NOJl8jM/d5ztMERVEUEBERUcAS1S6AiIiIfIthT0REFOAY9kRERAGOYU9ERBTgGPZEREQBjmFPREQU4Bj2REREAU6jdgFXa+/evViwYAE0Gg2mTp2K6OjoOo/PySn26u+PjDQiP7/Uq69JRMGHnyXkSzEx5hofbzQte5vNhpdeeglDhgxBenq633+/RiP5/XcSUeDhZwmpodGEfe/evXH06FHMnz8fnTp1UrscIiKiRqPRhP3u3bvRtWtXzJ07F59++qna5RARETUaDSLsMzIyMHHiRACALMt48cUXMW7cOEycOBGnTp0CAJSUlOD555/Hq6++ipEjR6pZLhERUaOi+gC9uXPnYuXKlTAYDACANWvWwG63Y9myZUhPT8esWbMwZ84cDBgwAAMGDFC5WiIiosZH9bBPTEzEe++9h2nTpgEA0tLSMHjwYABAz549sXfv3mt63chIo9cHwtQ2ypGIqD74WUL+pnrYjxw5EpmZmZ77FosFJpPJc1+SJDidTmg09SvV21NbYmLMXp/OR0TBh58l5EuNZuqdyWRCSUmJ574sy/UOeiIiIqrU4MI+OTkZ69evBwCkp6ejQ4cOKldERETUuDW4JvOIESOwceNGjB8/HoqiYObMmWqXRERE1KgJiqIoahfhC97uE2M/GxF5Az9LyJdq67NvcC17b/li3VGvvl50VCj6JUXDGKL16usSERH5WsCG/Q9bT3v9NVdvMWDKmO6Ijw71+msTERH5SsBext+SkXnlg+rh4JlCLF93FHqdhEdv74TeSbFefX0iCg68jE++VNtl/IANe1/02X+3/ijmf38AdoeM2we0xD2D20AUBa/+HiIKbAx78qVGM8++IevbqSn+d2IKYiMM+G7zKfz7y90osTrULouIiKhODPt6Sog14YVJKejaJgp7jufhlYU7kJljUbssIiKiWjHsr0FoiBbP3NsDtw9oieyCMry2KA3bD2arXRYREVGNGPbXSBQFjBnSFk/e3RUAMOfrvfjy12OQ5YAcAkFERI0Yw/46pXSMxf8+1BtNIw34fsspvPtFBixl7McnIqKGg2HvBfExJrzwcAq6t22CfScuYsbC7TiTzX58IiJqGBj2XmIM0eLpe7vjzoGtkFtoxWuLd2Dr/gtql0VERMSw9yZREHDPjW3wl3u6QRAEfLhyH1LXHYVLltUujYiIghjD3gd6J8XghYdS0DTKiB+3nsa7qezHJyIi9TDsfaR5dCheeCgFPdtFY//JfMxYuB2nL3DVLCIi8j+GvQ8ZQzSYMqYb/nBDa+QWWjFzcRq27DuvdllERBRkGPY+JgoC/nBDazw1phskScBHq/bj81+OsB+fiIj8hmHvJ73ax+B/H0pBXBMjVm8/g3eWZaCo1K52WUREFAQY9n4U1yQU//tQCnq1j8aBU/l4ZeF2nDrPfnwiIvIthr2fGfQa/GV0N9w9uDUuFtkw89M0bN7LfnwiIvIdhr0KREHAXYNa46l7u0MjCZj77X4sXXMEThf78YmIyPsY9irq2S4aLzzcB82jQ/HzjjN4Z1k6ikrYj09ERN7FsFdZsygj/jmxN3p3iMHB0wWY8cl2nDhXpHZZREQUQBj2DYBBr8GT93TF6BvbIL/Ihtc/3YmNe86pXRYREQUIhn0DIQgC7hjYCn+9rwd0GhEff3cAn60+zH58IiK6bgz7BqZ72yZ4YVIK4mNC8cvOTMxeuguF7McnIqLrwLBvgJpGuvvxU5JicDizEDMWbsfxs+zHJyKia8Owb6BCdBo8cXdX3HtTWxQU2zDrszT8nnFW7bKIiKgRYtg3YIIgYFT/lnh2bA/otRIW/HAQi1cfYj8+ERHVC8O+EejapgleeDgFCTGhWLczC28u3YVCi03tsoiIqJFg2DcSsZFG/HNiCvp2isXRzEL838LtOJZVqHZZRETUCDDsGxG9TsKf7+qCsUPbobDEjjeW7MR69uMTEdEVMOwbGUEQcGu/RPxtXE/otRIW/nAQi348CIeT/fhERFQzhn0j1aVVFF6c1AeJsSb8mn4Wby7difxi9uMTEVF1DPtGLCbCgH9M7I1+nZviWFYRZizcjqOZ7McnIqKqGPaNnF4r4bE7O2Pcze1QVOrux/91V5baZRERUQPCsA8AgiBgZN9EPDeuJwx6DRb9dAgLf2A/PhERuTHsA0inVlF4cVIKEpuasD7jLN5Ywn58IiJi2Aec6HADnn+wNwZ0aYrjZ4vwfwu34/CZArXLIiIiFTHsA5BOK+HROzrj/uHtYSl14K2lu7B2ZyYURVG7NCIiUgHDPkAJgoARKS3w3PieMIZo8Onqw1jw/UE4nC61SyMiIj9j2Ae4ji0j8eLDfdCymRkb9pzDrM924mKRVe2yiIjIjxj2QaBJeAj+MSEZA7s2w4lzxZixcDsOnc5XuywiIvIThn2Q0GklTL69EyaM6IASqxOzP0/HL2nsxyciCgYM+yAiCAKG9U7Ac+N7IjREg89+Poz53x2A3cF+fCKiQMawD0JJiZF4cVIftI4zY+Pe83j9s53IK2Q/PhFRoGLYB6mosBD8fUIybugWh1Pni/F/C7fj4Cn24xMRBSKGfRDTaiQ8MqojJt7SAWU2dz/+z9vPsB+fiCjAMOyDnCAIGJqcgKn394LJqMXSX45g3rf7YWM/PhFRwGDYEwCgQ4sIvDSpD9o0D8PmfRfw+qdpyC0oU7ssIiLyAoY9eUSa9Zj+QDJu7BGH0xcsmPHJDhw4eVHtsoiI6Dox7KkKrUbEpNs64aFbk9z9+MvS8dO20+zHJyJqxBj2VKObesZj+gPJCDPqsGztUXy0iv34RESNFcOeatUuIRwvTuqDtvFh2Lr/AmYuTkMO+/GJiBodhj3VqaIf/6Ze8TiTbcGMhdux7wT78YmIGhOGPV2RRhLx0MgkTLqtI2wOF95JTccPW0+xH5+IqJFg2NNVu7FHc0yfkIzwUB2+WHcM//1mH2x29uMTETV0DHuql7bNw/HSpD5olxCO7Qez8driHcjOL1W7LCIiqgPDnuot3KTHtPt7YWhyPDJzSvDKJzuw93ie2mUREVEtGPZ0TTSSiIm3JOGR8n78d7/IwPdb2I9PRNQQMezpugzu0Rx/n9AbESY9vvz1GOZ8vRdWu1PtsoiI6BIMe7pubZqH4cVJfdChRQR2HMrBa4vScIH9+EREDQbDnrwiPFSH58b3xLDeCcjKLcErC3dg9zH24xMRNQQMe/IajSRiwogOmHx7J9idMv79RQZWbTrJfnwiIpUx7MnrBnWLw/MTkxEZpseK9cfxwYq9KLOxH5+ISC0Me/KJVs3c/fgdEyOQdjgHry7agfMX2Y9PRKQGQQnQa6xrtqTV42jhikeEmUMAERAE4bKfECBU3BGEqo9fcstzzOW3Uf1nLz++/OnK25c/Xn6wWP3ZasdcemTlmZ4AQbzkdg3/DRAvq+/Se3X89ztdMpb/dgI/7zgDg16Dx+7sjB7tokEUrGJizMjJKVa7DApQMTHmGh8P2LD/0/c71S6BauT+5yag8f6z08CFG8PtGNahp9qlUCPEsCdfqi3sNX6uw2/a5x/x6uvJgginpENFTEmSgBCjDnq9BqJUNbqq3L4s05QrHFNTBCp1HKPU8ODlz9f4mld6vauqUfDcU2qJ70t/zqUAdoer2t+ksSmRTPilMBR5u7diTNc+kET2hhFRwxawYT80wcuXiq0FOLNuHRxnNcgztkaeMQEWwQ4L7GgSo0GX5DZo2zEGIQatd39vgLE5XDiXV6J2Gdcl/cRp7BctSLdFoWDXNkzqkQydRqd2WUREtQrYy/jevkxWcenNZi3F4S0/4eKWNDhyjcgJbYMCQzMAgAAZzeKN6JrSBi3bNYFWK3m1Bmo4th44jfWF55AvRSEGBZjctSPCDDVfPiO6FC/jky8FXZ+9r8L+UqWWAhze+APyd+yFvSAS2aY2sOijAACi4EKLNuHo2rs1ElpFQuSl3oBz/Hw+Vh7bh2xdU5gVCya2i0NCVDO1y6IGjmFPvsSwv05XeoMWF+Tg8O/foXDnUVhLY3DB1AZWrfuPrpFcaNMxGl2SW6Fp8zDPiH5q/PKLrVi6aysyDc2hhw33NDOge4u2apdFDRjDnnyJYX+d6vMGvZidhSPrv0Px7iyU2Zvhgrk1HFIIAECvc6JD93h07pmIqOhQr9ZI6nA4ZSzevBHHQmIhQsHNZhuGduyhdlnUQDHsyZcY9tfpWt+g2VnHcPy371G87yIsSgJyTIlwie5BfKFGJzolt0Kn7gkwhYV4tV7yL0VRsGL7DqQLoXBCg2RNPkb36MPuG6qGYU++xLC/Tt54g549vhfHf/sZJUdKUSS1QJ4xAYogAlAQHiajW9/2aN8ljiP6G7Hf9h3A+hIHygQD2ijZeLh3P2ilgJ30QteAYU++xLC/Tt58g8qyjDOHd+LU+l9RcsKFAl3LKiP6m0QL6DGgI1p3iOWI/kbowJksrDx7DoViOJrJuZjUtTvCQtllQ24Me/Ilhv118tUbVJZlnNizCac3bEJppgYXQ1pVjuiHE03jteg1sAtatI7iJeFGJLugEJ8dPIAcqQki5EKMa9MSLWNi1C6LGgCGPfkSw/46+eMN6nI5cTRtHTI37UTpBSNyja1h1ZoAABrBgYQ2oeg5oDOaxYdzRH8jUGa345Od23FaioVRKcVt0Wb0btNG7bJIZQx78iWG/XXy9xvUYbfi0NY1OLtlH0rzw5Eb2sozol8n2dAqKRLJA7sgkiP6GzSXy4WlO7diP2KghR0DdQ6M7NFT7bJIRQx78iWG/XVS8w1qLSvGgQ0/4fyO4yi1NEFeaOWIfoOuDO27xaFHv44c0d+Afbc7DZttoVAgoKsrD+P6DYDIqzNBiWFPvsSwv04N5Q1aUpyPA799jwvp51Bia4qLxnj3iH5FgdlYho4prdGtdzvoQziiv6HZeuwAfrgoww4dWtnPYWLKABj0/P8UbBrKZwkFJob9dWqIb9DCixewf+0PyN2fj2JXcxQamgIABEVGeJgV3Qd0RFK3ltBwRH+DcfRCFlJP5cAihCLOfgHju/ZCTDi7YoJJQ/wsocDBsL9ODf0NmnfhFPb/shoXD5ehSEioHNGvONCkiRM9B3dHm6Q4juhvAPKK87HwwFHkCRFo4szD7Qmt0bFFU7XLIj9p6J8l1Lgx7K9TY3qDXjhzGPvXrEPBSRcKNImVI/phQ0wzIOWmZMS3jOaIfhWVOaxYkJ6OTDSBWS7CQGMYhnRrr3ZZ5AeN6bOEGh+G/XVqrG/Q04fTcWjdZhRlScjXJ1aO6Ecp4hJ16HtzCqKbRahbZJByyS4szdiO/c4mCFGs6OxwYPTAXhy4F+Aa62cJNQ4M++vU2N+gsizj5L5tOPzbThRlG5AfkgC5YkS/WIyEdmb0vbkPwiLYf+xv3+1Pw8YSMzRwIbEoFxMHD4Rex3EWgaqxf5ZQw8awv06B9AZ1uZw4krYBRzfuh6UgDAUhzT0j+k3aQrTsHI0+N/WBwahXu9SgsfXEfnybK0CGiOZFZ3F/nwGI4lTKgBRInyXU8DDsr1OgvkGdDjv2bV6Lk9tOwGKJQlFIxYh+F8whRWjXMwG9b0jmiH4/OJJ9CktPFcGKEDQtycLt7XqgXUKk2mWRlwXqZwk1DAz76xQMb1C7rQwZv/6MzPRzKLLFolTnDhpRcSAitBid+rdD15SuHNHvQ9lFOVhwKBOFMCHadgH9wltiULcWapdFXhQMnyWknkYf9ps3b8aKFStgtVrx5JNPomPHjnUez7C/PmWlxdi5ZjXO7b2IIlccbBUj+uUyREWWodvgrmjfpT1H9PtAibUEH+/dj/NKBCJc+WjrCMU9N3TmwL0AEWyfJeRfjT7s165di6FDh+LAgQPYtGkTHn300TqPZ9h7T3F+DnauXosLR0tRiDg4K0b0yxZExziQPKw3WrRJVLnKwOJwOvDp3p044ohCqGJBdI4dk0akcOBeAAjmzxLyvUYf9gBQWlqKV155Bc899xyaNGlS57EMe9/Iyz6LnT/+itxTThRJcZUj+uUCxMYDfUYOREyzWJWrDAyyLGPlwTRsK4mATrEh4lwuHhranwP3Gjl+lpAvNfqwz8/Px+zZszFlyhTExcVd8XiGve+dP30Mu1ZvwsVzEoo1zSpH9CMPTVvr0e/WGxAewQFm1+v347vxY54eAhREns/E3cl90DY+XO2y6Brxs4R8qUGHfUZGBmbPno3FixdDlmW8/PLLOHToEHQ6HV599VW0bNkS06ZNw8WLFxEREYHhw4fj1ltvrfM1Gfb+deLgXuxdtxMFuQZYtDEAykf0i7mI72BGv5E3wWA0qlxl47Xv3FGkZtrhgBbRF0+hf/POGNitudpl0TXgZwn5UoMN+7lz52LlypUwGAxITU3F6tWrsXbtWsyaNQvp6en48MMPMWfOnHq/LsNeHbIs40jGTuzfcACFBWaUaSMAAJJsR5g2F4ldY9Bn+E3QarnbW32dyT+LRUdzUQIDokuykCjEY/SN7Thwr5HhZwn5UoMN+59++glJSUmYNm0aUlNT8frrr6N79+64/fbbAQCDBw/G77//Xu/XdTpd0Gg4mElNLpcLW3/5FXvWH0Z+cSTsmsrV+QRFBqBAUBT3d+Cy++7vADy3BaXq/So/J1T8M77ksfLvEC49DpXHXX5bgPu4iuMFeI4RargtiO4fEMtvC2L5dwEQRBFi+X1RFCGIgCiKECWx/DEBolRx3/1d0kieY0RJhEYjQZKk8uc0EDUirIoT31pcyEM4oh0XYNmfDZcswukS4JRFOFwinLIIuyxBdomQZcHzl7hekihAI4mQJBEaSYAkln+XRGikS26LQuUxkgitJEKSBGjE8u8VryEKECUBgpfq00gCNBoROo0ErUas+iVJ0Fz22KXHaSTxmmeWaDUidFoReq0ESeK0VGqYNGoXMHLkSGRmZnruWywWmEwmz31JkuB0OqHR1K/U/PxSr9UI8Gz8WrXv1Rfte/WF0+lA2rr1OJlxHjabe2U+BUJ5nl8aSMJlcV15WxFEz2Oe2K9I6fKfUyoT2z2G4GpVnluoTAHgKv9y1HhEqM4FTX87LmibAj0qd8uTyr9qek0RsvtLUeD+K8qV3xX384JSeV8oP7kSFdlzW1BkCIoLouIElEsfkwG5/GRMlgFFAWQFSvnjcCmQnQpsLgWKokCWAZR/d7kAlwy4XAJcLsDuEuF0ibDLIuxOATaXBjaXBrIsAkLDP3mXRKH8REKEtvxkQqcRodVWnlyEGnWw25xX94ICoNOI0Gkl6LVS+Xf3iYVeK0GvK39MI17ViYYgAHqthBCdBL1OgxCdBJ3m2k90qOGprWWvethfzmQyoaSkxHNfluV6Bz01PBqNFv1GDEO/Ef79vS6XC7Isw+V0wOWSocguuJxOyLIM2SXD6XJCkWW4nDJkufy2Sy7/7oIiy55jFVm+5HkXXC4ZsssF2SVXfpUf476tuH/GpbgfVxQosgK5PPQU+dLvgCKj/D7cYai4cxOXf3cB8iYFEV0uwmYIgSyIUASh8gsiZM9tofL5itsQIJefDMkQ4ax4TBQhe04D1G2higBCyr/cFEieExb3d+mS24JSXrUiV/sS5PLvigxRLj9BkeVLTk4AweGCVUmEKBnqVacCwOmUYXfKcDhl2J0uOBwV910osTqQ75ThcMiQ1R8eVSNBAEJ0EkI84S+hrostAtzHG/QahOg0MOo1CNFL5d810GvFWq/WhOgldG/bBBIX5vK7BpeiycnJWLduHUaNGoX09HR06NBB7ZKoEZMk96VwjhGoH7vdBrvDBrvNBpvdBqvdCofdCbvdCrvDAafLAafLBafTBYfshEtW3CdWigyXokAuP1lxQYECdwNfFipOMlB+4iF4vleckLhPUsTK+xDL77uPkSFe8l2AQ9BAhgiXKMIF6bpOUpILD+Le4Xd67494GZcsw+6QEREZirw8y1X9jKwocDhk2Bwu2Bwu2B0u2Bxy+fdLvuyuqzqZUBTAZnfBanfBanfC6ii/bXPB5nDCUuaA3WG7wmsosDvlq6q/Jk/f2x0920Vf88/TtWlwYT9ixAhs3LgR48ePh6IomDlzptolEQUdnU4PnU4PNLJNEG12G0pKSlBWZkGZtRRWqxU2uw02hwNOhwMO2QmHywWnLLtPUKAgTxOCE8aWcDhdPq1NEkUY9CLCQnWwlTbuk0+XLMNqd6HM5kSZreK7+6u2E4HjZ4uwPuMsCix1n0yQbzSIsE9ISEBqaioA9yCmGTNmqFwRETVGep0eep0eiIy66p9ZuXEdTgCQXb4N+0AiiSJCQ0SEhlz9SUuYUYf1GWdRdrXjFcir2HFCREFNq9W5b8gMe18y6N0DLBn26mDYE1FQEzXlrVO5YQ6gCxTG8qsApVaGvRoY9kQU3MqnaIrlMybIN9iyVxfDnogIgCADZc4ytcsIWEa9e4hYmY3dJWpg2FM1ubk5GDlyCFJTl6hdCpHfiIqCi9YCtcsIWCF6DQQApdaaF4si32LYUxWlpaV4/vmpVRY2IgoGggIUFOWoXUbAEgUBIXoJpWzZq8InYV9WxkthjdH58+cwZcpj2L9/r9qlEKmiMP+C2iUENINewz57ldQr7IcNG4ZFixbVecz777+Pm2+++bqKIv9LTV2Chx4aj2PHjqB37z5ql0OkitKCXLVLCGhGhr1q6lxUJzMzExZL5bKOWVlZOH78OA4ePFjj8Q6HA5s3b2bLvhFKTV2KZs2aYerU53HmzGmkpW1XuyQivysruKh2CQHN3bIvgawo3JrZz+oM+4yMDPzP//yPZ0ckQRCwbNkyLFu2rNafURQFgwYN8m6V5HNTpz6PlJS+kCQJZ86cVrscIlXYiwvULiGgGfQaKHCvz2/QN4gFXINGnX/t22+/Hfv378fFixehKAq+/vprdOzYEZ06darxeK1Wi9jYWEyYMMEnxZLv9Os3QO0SiFTnKuY21r5UOf3OybD3syv+tadOneq5vW3bNowePRoPPfSQT4tqKFLXHsX2g9kAAEkS4HI1vAU3+nSMxdib26ldBlFAUEpKoCgK93f3EUOIO3JKrU5EhalcTJCp16nV2rVrfVUHEZHqdFYnypxlMGqNapcSkCpa9qUcpOd313Qd5euvv8aqVatw8OBBFBcXIzIyEj169MCYMWMwZMgQb9eomrE3t/O0mmNizMjJ4SU+okBmsMrItxUy7H3EcMllfPKveoW93W7Hn//8Z2zZsgWKoiA8PByJiYkoKirC6tWr8fPPP+Puu+/G66+/7qt6iYh8xmBTkG8tQLwpTu1SAhJb9uqpV9h/8MEH2Lx5M0aMGIFp06ahRYsWnueysrIwa9YsfP311+jSpQsefPBBrxdLROQzogiDTUaBrVDtSgIWW/bqqdeiOqtWrULnzp3x73//u0rQA0B8fDzeffddtG/fHp9//rlXiyQi8jnJHfb5DHufYdirp15hn5ubi/79+0MUa/4xjUaDAQMGIDMz0yvFERH5iyBJCLErKCzJV7uUgGW8ZDQ++Ve9wr59+/bYu7fuddOPHj2KVq1aXU9NRER+J0juILIUcslcX2HLXj31CvupU6di165dePXVV1Fcw+ITc+fOxbZt2/D88897rUDyv1Gj7sSGDTswduwDapdC5DeCKAEAbEVs2fsKB+ipp14D9JYvX45WrVrhs88+w4oVK5CUlISmTZvCarVi3759yMnJQWhoaLXR+IIg4KuvvvJq4URE3iRI7rC3FxdyYR0fqVxBj9vc+lu9wn7lypWe2yUlJdi5c2e1YywWCw4cOFDlMb5piKihEyT3hU5NmZ0L6/iITitCFASU2hxqlxJ06hX2te12R0TU6JW37I1cWMdnBEGAQS+xZa+CevXZExEFqoo++4qFdcg3DNzTXhXXtFzujh07sHz5chw6dAhlZWWIiIhA+/btcddddyElJcXbNRIR+V55yz6EC+v4lDFEgwsXy9QuI+jUO+zffvttzJs3D4ri3gHOYDDg5MmT2LVrF7744gs89thjePbZZ71eKBGRL1UM0DNYGfa+ZNRrYHO44JJlSLWs2ULeV6+/9Pfff4+5c+eiXbt2+PDDD7Fjxw7s2rULGRkZmD9/PpKSkvDRRx9hzZo1vqqXiMirPOOHywfoGWwy8q0Me18xcES+KuoV9osWLUJMTAwWLVqEIUOGwGQyAQB0Oh0GDhyI+fPnIzo6GosXL/ZJsUREviIIIoSQEBhsClv2PsS59uqoV9gfOnQIQ4cORWRkZI3PR0VFYejQodWm3hERNQYasxlGm4J8W4HapQQsT8ueS+b6lU86TBwOzqEkosZHMpvLL+MXeMYlkXdxyVx11Cvsk5KSsG7dOhQUFNT4/MWLF7F27VokJSV5ozYiIr+STGaIsgJYbShzWtUuJyB5NsNh2PtVvcL+oYceQk5ODiZPnoxt27bB6XT/z7JYLPjtt98wadIk5OXlcS97ImqUJJMZQPkgPV7K9wm27NVRr6l3o0aNwp49e7BgwQI8/PDDEEUROp0OVqv7DFhRFDzyyCO44447fFIsEZEvSWb3oOOKQXrxpjiVKwo8HKCnjnrPs58+fTqGDRuGr776CgcPHkRJSQlCQ0PRsWNHjB49movqNGJ5ebmYP/8jbN68ERcv5iEsLBwpKX0xefKfER+foHZ5RD4nmcIAuOfacxU932DLXh3XtIJeSkoKQz3A5OXl4k9/ehjZ2RfQp08/DBt2C06fPomff/4RW7ZswocfLkCLFolql0nkU5K58jI+p9/5hqfPnqPx/eqawt5isWDNmjU4ePAgSktLER4ejq5du2Lo0KHQ6XTerpH8YP78j5CdfQFTpjyD8eMrx1ysXv0DZsx4Ae+//y7eeONdFSsk8j3JVHEZnwvr+Apb9uqod9gvW7YMb7zxBsrKyqpMTREEAVFRUZg5cyaGDBni1SLJ99av/xUREZEYO/aBKo/fcstt+PjjD7Ft2xbIsgyRy1tSAKto2YfYFOSxZe8TDHt11Cvsf/rpJ7z00kuIjo7G448/ju7duyM0NBTZ2dnYtWsXli5dir/85S9YvHgxevXq5auayctcLhcmTnwEGo2mxjDXanVwOBxwOBzQ6/UqVEjkHxWj8cPsIo4y7H3CqHfvQcABev5Vr7CfN28eIiMjkZqaiubNm1d5btiwYbjnnnswbtw4/Otf/8Inn3zi1ULJdyRJwtix99f43KlTJ3H69EnExycw6CngVbTszQ4B+Tb3wjqCZ/F88gatRoJGEtmy97N6XZM9fPgwRo4cWS3oK7Rt2xa33HILdu/e7ZXiSF2yLOOdd96ELMu466571C6HyOdEgwGQJBhtgN1l58I6PmLUSyjlRjh+Va+WfWRkJGRZrvMYg8GA0NDQ6yqqofjq6LfYlb0HACCJAlxyw1s+s1dsN4xu5/11DRRFwVtvzURa2jZ07Ni5Wl8+USASBAGSyYwQqx0AkG8rgFFrULmqwGMI0aLMymXV/aleLfuxY8di1apVtW50c/r0aXz77bcYPXq0V4ojdTidTrz++gysWvU1mjePx6xZb0Or1apdFpFfSGYztOVBxOl3vsGWvf/V2bJftGhRlftGoxGhoaG477778Ic//AG9evVCdHQ0ioqKsGfPHnzzzTeIjIxE586dfVq0v4xud4en1RwTY0ZOTrHKFfme1WrFCy9Mx+bNG5GQkIh//esDREfHqF0Wkd9IJhPETAckl4ICTr/zCYNeA6dLhsMpQ6vhDB9/qDPsZ86cCUEQPFPsLr29fPlyLF++3DN4peLxoqIiPPvss7j11lt9WTf5QFFREZ577mns378XHTok4e2330NkZJTaZRH5lcYz/Y7r4/uK8ZLpd1oN12bxhzrD/vXXX/dXHaQym82G6dOfwf79e9GzZzLeeOMdhIaa1C6LyO8uXUUvn5fxfcJwyfr4YaEMe3+oM+zvuYcjsIPFRx/9P+zZsxtdu3bH22//B3p9iNolEamicuc7Xsb3FS6s43/XtFwuBZa8vFx89dUXAICWLVvh009rXiPhwQcnca49BbyKln2EU4Nstux9gjvf+V+9wr5v375XdZwgCNi6des1FUT+t2/fXjgc7tHH3323stbjxo59gGFPAa+iZR/p1OEQF9bxCUP5Zjhl3AzHb+oV9iZTzX24VqsVBQUFkGUZHTp0QIsWLbxSHPnHjTfehA0bdqhdBlGD4GnZOzSwu6woc1o5197L2LL3v3qF/dq1a2t9rri4GHPmzMHy5cvx7rvcHY2IGqeKsA91uKeEFdgKGfZexj57//PaBEez2Yxp06ahXbt2eOutt7z1skREflWxza3R5l4tlNPvvM/TsudlfL/x+moGvXr1wo4dvCRMRI2LAvdaIVL5lFO91b3CG0fkex9b9v7n9bA/cOAAB7MQUaMlaDQQjUZoSivWx2fYe5tngB7D3m/q1Wf/yy+/1Pi4oigoLS3Fr7/+ik2bNmHEiBFeKY6ISA2S2QyltASAkZfxfYAD9PyvXmH/l7/8pc5Wu6IoiI2NxXPPPXfdhRERqUUymeHIyQEUAy/j+4BBLwFgy96fvBb2Op0Obdq0wZAhQ7hDGhE1apLZDMgyouQQXsb3AUkUoddKbNn7Ub3C/qmnnqr2mM1mw/nz5xEdHR0w+9gTUXCrWFgnRgnFCS6s4xMGvcSWvR9d1QC9tWvX4h//+AcOHjxY5fG3334b/fv3x6233oq+ffvimWeeQX5+vk8KJSLyl4rpd9EuPewuO8qcVpUrCjzGEC2n3vnRFVv2L774Ir74wr1u+k033YSOHTsCAN555x3MnTsXgiBg4MCBAIDVq1fj6NGj+Oqrr6DTcScjImqcPKvoudyfY1xYx/sMegnn81y8auIndbbs165di9TUVHTq1Anz5s3DTTfdBAC4cOEC5s+fD0EQ8Morr+Djjz/Gxx9/jPfeew9Hjx7FokWL/FE7EZFPaMxhAIBwh3sgGfvtvc+g10BWFNgdstqlBIU6w/7LL79EREQEFi1ahEGDBnk2Qfnxxx/hdDqRmJiIe++913P8sGHDkJycjB9//NG3VRMR+ZBYfhk/1Oa+X2AtUK+YAMXpd/5VZ9jv3r0bN910U7UNcDZt2gRBEHDzzTdX+5kePXrg1KlT3q2SiMiPNObKPe0Btux9gWHvX3X22RcWFqJp06ZVHpNlGWlpaQCAAQMGVH9BjcazXSo1LoWFBViwYC42bdqA3NxcNG/eHKNG3YmxYx+ARlOviRtEjVrFaHyd1f1ZxoV1vI9L5vpXnS17s9lcbXT97t27YbFYoNFo0KdPn2o/c/LkSURGRnq3SvK50tISPPnko/jyy2Vo3boNxowZi9BQEz744D94/vmpUBRF7RKJ/KZigJ6m1H0dnwvreJ8xhJvh+FOdzbVu3bph06ZNkGUZoug+L/j2228BuFv1BkPV0ak5OTnYsGEDBg8e7KNyyVcWL16IU6dO4q9/fQ733Tfe8/jLL/8Ta9b8hM2bN2LgwBtUrJDIfwS9HoJWC7mkFKEaIwp4Gd/r2LL3rzpb9mPHjkVmZib+9re/Yfv27fjss8+wbNkyCIKACRMmVDn24sWLeOaZZ2C1WnHXXXf5tGjyvnPnziI2tinuuefeKo8PH34LAGDv3t1qlEWkCkEQIJnMcBUXISIkHPnlC+uQ9zDs/avOlv2wYcMwYcIEfPbZZ/jpp58AuNe/f+CBBzBkyBDPcY8//jg2b94Mm82GW2+9FcOHD/dt1eR1L7/8Wo2Pnzp1EgAQFRXlx2qI1CeZzbBfOI9IfTiyLOdgdVlh0HCuvbdwgJ5/XXHU1QsvvICRI0di3bp1cDqdGDRokGe+fYXjx48jNDQUjz32GJ544glf1Up+oigKCgrysW7dL/j444/QtGkz3HLLKLXLIvIryWyGcvoUIkX3bKR8ayEMJoa9t7Bl719XNcS6b9++6Nu3b63Pf/XVV9Wm51HjNW/ef/HJJx8DAKKimuDdd99HWFiYylUR+VfFkrlN5BAA7ul3zU3N1CwpoLBl719emU8VqEGf88XnKN6xHQBwShLhcjW8lZ7MKX0Qc8mAOm9o1iwO998/EVlZmdiw4Tc8+eSf8Pbb7yEpqaNXfw9RQ+ZZMtfp/pgs4PQ7r2LL3r84eZqqufPOuz23N23agOnTn8Wrr76IRYuWcQ1rChoVc+3D7OVL5nL6nVdx6p1/MezrEHPfeE+rOSbGjJycYpUr8r+BA29A7959sGPHNmRlZSIhoYXaJRH5RUXL3mh33+f0O+/S6yQIYMveX65qi1sKbE6nE9u3b8X27VtqfL5ZszgAQEFBgR+rIlJXRcs+xOoCAORzfXyvEgUBIXoNw95P2LInAMD06X+D0WjEN9/8CEmSqjx39OgRCIKA5s2bq1Qdkf9VtOyFklKERnNhHV8w6jUcoOcnbNkTNBoNhgwZioKCfCxZsrjKcytWfImDB/djwIAbEBXVRKUKifyvomXvsli4sI6PGNiy9xu27AkA8OSTTyMjYxc+/PB97Nq1A23btsfhw4eQlrYNcXHxmDbtebVLJPIryeyeZeQqLubCOj5i1EvIsrkgKwpEDv71KbbsCQAQExOLuXM/wZ133oNjx44iNXUJMjNPY+zY+zFv3ieIjo5Ru0Qiv5JCTYAgwGUpRoQ+HABH5HubMUQLBYCVrXufY8uePJo0icb06f9UuwyiBkEQRUihJriKixGhjwDAhXW8zaB3jw8qtTlhDNGqXE1gY8ueiAhATd3xkskEp6UYkSHulj0X1vGuyoV1XCpXEvgY9kQU1ATU3lcsmc2QS0oQoXUP1uNlfO+qCPtSq0PlSgIfw56IqBaSyQwoCsJdOgBcWMfbKlbRY8ve9xj2RES1qJhrb3a4PyoZ9t7F9fH9h2FPRFSLirAXS60wagxcRc/LuPOd/zDsiYhqUbHNrau4CJEhEWzZexnD3n8Y9kREtaho2bssFkTow2F12VDmLFO5qsDBy/j+w7AnIqqFZ8nc8lX0AI7I9yaGvf8w7ImIalHZsq9cWIeX8r2He9r7D8OeiKgWnrAvrlxYJ58L63gNW/b+0+jCfvPmzfjnP7mkKxH53qWX8SvWxy/gZXyv0WlESKLAsPeDRhX2p06dwv79+2Gz2dQuhYiCgKjTQdDr4bJYKvvseRnfawRBgIF72vtFowr7li1bYvLkyWqXQURBRDKVb4YTEgGAffbeZmTY+wV3vSOPjz76AIsWza/xuWHDRuD//u91P1dEpD7JHAZ7ViZ0ota9sA7D3qsMeg0KSni11tcaTNhnZGRg9uzZWLx4MWRZxssvv4xDhw5Bp9Ph1VdfRcuWLdUuMeAdO3YEOp0OEyY8XO25Nm3aqlARkfokkxmKwwHFZkNkSATyyi6qXVJAMegl2B0ynC4ZGqlRXWxuVBpE2M+dOxcrV66EwWAAAKxZswZ2ux3Lli1Deno6Zs2ahTlz5niOnz17tlqlBrRjx46iVavWmDz5z2qXQtRgSOaKVfTcg/SyLOdQ5rTCoAlRubLAULGPfZnNCbNRp3I1gatBhH1iYiLee+89TJs2DQCQlpaGwYMHAwB69uyJvXv31vs1IyON0Ggkr9YZE2P26us1JBaLBefPn8OAAf0D+r+T6HI6nQQ4AK1WqvHfviW2CYoBmLUy4sKjsS8PEIwOxITHXPPv5HusUmS4+6TJEBqCmOhQlasJXA0i7EeOHInMzEzPfYvFAlP5mtQAIEkSnE4nNJqrLzc/v9SrNcbEmJGTU+zV12xIMjLSAQDx8S0D+r+T6HJ2u3t7VYfDVeO/fbukBwDknj6PELM7jI6fO4sQ+7UFdqB/ltSXoCgAgKxzhdAossrVNH61nUg2iLC/nMlkQklJiee+LMv1Cnqqv2PHjgAACgsL8cwzT+LgwQMAgJSUPnjssSeRmNhKxeqI1FMx1162WBARw4V1vM2zGY7VoXIlga1BjoZITk7G+vXrAQDp6eno0KGDyhUFvoqwX7JkEUJDQ3HXXXejc+eu+PXXtXjssUk4cuSQyhUSqaNiFT1ncZFnrj0X1vGeyp3vXCpXEtgaZHN5xIgR2LhxI8aPHw9FUTBz5kxV6ti09hiOH8wGAIiSCNnV8C4xtekYi4E3X/9IeVGU0KxZHJ5//iUkJ6d4Hl+9+gfMmPECXn99BubP/+y6fw9RY1PTZjica+89XDLXPxpM2CckJCA1NRUAIIoiZsyYoXJFweV//mc6gOnVHr/lltuwcuUKpKfvxOnTJ3k5n4LOpdvcRpYvrMO59t7j2QyHYe9TDSbsG6KBN7f1tJqDeVBNhw5JSE/fibNnzzLsKehI5YOFXZZi6CUdF9bxMrbs/YNhT3A6nThy5BBkWUGXLl2rPV+xF4FOxzmwFHxEoxEQRbiK3Sf7EfpwXLTmq1xV4GDY+wfDniDLMp54YjIMBiO+/fZnSFLl+gSKomDv3t2QJAnt2yepWCWROgRRdK+Pb3GHfWRIBM6WnOfCOl5SORqfYe9LDXI0PvmXTqfDoEGDUVxchE8/XVjluaVLP8WxY0cxYsStMJu5EAgFJ8lkrtKyBzhIz1sMIWzZ+wNb9gQAmDLlWezduxtz587Brl1paNeuAw4dOoBdu9LQqlVrPPXUs2qXSKQayWyG/WwWFKcTkfoIAEC+tQBxoU3VLSwAVE69Y9j7EsOeAABxcc0xb95izJv3X2zZshHp6TsRHR2D8eMfxKRJj1ZZ0ZAo2HgG6ZVYEBHClr03aSQRWo3IsPcxhj15xMTE4h//eFHtMoj8SriKYyRzGIDy6XeG8lX0rAW+KyrIGPUaXsb3MfbZExFdgWeuPRfW8QkDw97nGPZERFfguYxfXIwILqzjdRVhr5RvikPex7AnIrqCylX0uLCOLxhDNHC6FDicDW9J8kDBsCciuoJL18cH3NPvuBmO93BhHd9j2BMRXYHmkpY9AESEhMPqsqLMaVWzrIBhNmgBAGfzSlWuJHAx7ImIruDSAXoAPHPtOUjPOwZ0aQYAWLXxhMqVBC6GPRHRFYih7gF6Tk/Yc/qdN7VLCEfXNlE4eLoAB05x3wFfYNgTEV2BqNVCNBjgslgAwDMiny1777lncBsAwIrfj3NUvg8w7ImIroJkMl1yGb+8Zc+w95rWcWHo1T4aRzMLsffERbXLCTgMeyKiqyCZzXBZiqEoSuVmOLyM71V3V7Tu17N1720MeyKiqyCZzIDLBbmszBP2bNl7V4tYE1I6xuLk+WKkH81Vu5yAwrAnIroKl861D9HoYeDCOj5x9w2tIQjAivUnILN17zUMeyKiqyBdNtc+kgvr+ETz6FD079wUmTkWpB3KUbucgMGwp2pyc3MwcuQQpKYuqfH5H374Fo888gCGD78B99wzCu+99w5KS7kYBgW2aqvocWEdn7nrhtYQBQFf/34csszWvTcw7KmK0tJSPP/8VJSUlNT4/OLFC/Daay9DlhWMGTMO7dq1x7JlS/C3v02Bw+Hwc7VE/lPZsndPv+Pud77TNNKIQd2a4VxeKbYeuKB2OQGBYU8e58+fw5Qpj2H//r21PH8e8+b9F127dsfHHy/GE088hbfe+jcmTXoUe/fuxsqVX/m5YiL/qXUVPV7K94k7B7WCJAr4ZsMJuGRukHO9NGoXQA1DauoSzJv3IWw2K3r37oO0tO3Vjvnmm+VwuVyYOPERaDSV/3QmTnwEX3yxFKtWfYMxY8b5s2wiv/Fsc2spAgDPiPw8K+eE+0J0uAE39miOdbuyMHPxThj0UrVjBndvjn6dm6pQXePDsCcAQGrqUjRr1gxTpz6PM2dO1xj2GRm7AAC9eiVXeVyv16NLl+7Ytm0zLBYLTOUfikSBRDKHAQBcxe7L+IlhCRAg4OfTvyGlaS+EaPRqlheQ7hjYCmmHsnHiXFGNz5/LK0WfTrEQBcHPlTU+DHsCAEyd+jxSUvpCkiScOXO6xmOysjIRFdUERmNotefi4uIAAGfOnEKnTl18WiuRGi4fjR9visPwxCH4+fSv+PLISjzY6T41ywtIkWY93nnqhhoH6S384SA27T2PE2eL0DY+XIXqGhf22RMAoF+/AZCk6pfJLlVUVFhrqz20fKMQS/ngJaJAI4aEAJLkCXsAuKPNLWhhjsfmc9uxK3uPitUFLlEQoJHEal8pHWMBADsOZatcYePAln0d8rN+RmnBfgDAeVFskINEjBGdERk/wi+/y+l0QqvV1ficTud+3G63+6UWIn8TBMG9ZG5xZdhrRA0mdb4fs7b/G0sOfolWYS0QWb5JDvlWl1ZRCNFJ2HEwB2OHtoPAS/l1Ysuerpper4fTWfP0uoqQNxgM/iyJyK8kU9WwB4BmobEY0/4OlDrLsOhAKmSl4TUKApFWI6Jn+2jkFVlx8nzxlX8gyLFlX4fI+BGeVnNMjBk5OcH9D8psDqv1Mn1Jifvxisv5RIFIYzbDnnkGssMBUav1PH5D8/7Yl3cQe3IPYO2Z3zE8cYiKVQaPlKRYbNl3ATsOZaN1XJja5TRobNnTVWvRIhH5+Rdhs1VfMezcubMQRREtWrRQoTIi/7h8YZ0KgiBgQsf7YNaZsPLYjzhTfFaN8oJO19ZR0OskpB3M4S55V8Cwp6vWvXtPyLKMjIz0Ko/bbDbs27cHrVu3qXGkPlGgqJhrL1uqX+Uz60yY2GkcXIoLC/ctgd3F8Su+ptNK6NG2CbILynD6AgcH14VhT1ftlltuhSRJmD//oyoD8RYvXoCSkhLcddc9KlZH5HsVc+2dxTV36XVpkoSbEgbhfGk2Vhz9zp+lBa2UJI7Kvxrss6erlpjYCuPHP4jPPvsEf/zjBAwcOBgnTx7Hpk0b0K1bD9x5J8OeGqF6DOL2rKJXS9gDwB/ajsKh/KNYn7UZnZskoVt05+utkOrQrW0T6LQidhzMxugb23BUfi3Ysqd6efzxKXj22WkABHz55ec4fvwYxo17AG+99W/P9DuiQHX5wjo10UlaPNLlAWgECZ8e+AJF9uAe2Otreq2E7m2a4EJ+GbJyat7Ai9iypxqMGnUnRo26s8bnBEHAmDFjMWbMWD9XRaS+y7e5rU28KQ5/aDcKy4+swqcHvsAT3R9hi9OHUjrGYsehHOw4lI2EWM4Iqglb9kREV6m20fg1uSlhEDpGtse+vINYn7XZ16UFte5tm0CrEbHjUI7apTRYDHsioqtU2bKveWOWS4mCiImdxyJUa8SKo9/irOW8r8sLWiE6Dbq1aYKzuSXIyuWl/Jow7ImIrpIU6p5aeqXL+BUi9OGY0PFeOGQnFu5fCofs9GV5QS0lKQYAkHaQo/JrwrAnIrpKgkYD0Rh6VZfxK/SI6YpBzfsiy3IOq4796MPqgluPdtHQSAKn4NWCYU9EVA+Xb4ZzNca0vwuxxmj8cmY9dp8/4KPKgptBr0HX1k2QmeO+lO+S5Spfwb7CHkfjExHVg2QywZGTDUWWIYhX117SSzpM6nw/Zqf9P3ywbRFe7DcNWpEfv97WOykG6Udz8cK8rdWeaxIWglcf7Qe9ru6tvAMV/7UREdWDZDYDsgy5tNSzyM7VaBnWAkMSBmLdmQ3Ym3sAvWK7+bDK4JSSFIu9Jy6iqKTqUsU5BWXILbTiTLYF7RLCVapOXQx7IqJ68IzItxTXK+wBYEBcH6w7swFbzu1g2PuAXifhz3d1qfb47xlnseCHg8jMDd6wZ589EVE9eObaF9d/45V4UxzaRCZi/8VDKLRxZT1/qVhoJys7eKflMeyJiOpB41lY58pz7WtyU+sBkBUZ2y/s9GZZVIfm0aEQAGTmBO/OeAx7IqJ6qFxY59qCY1BiCjSChK3n0oJ+hLi/6LUSYiINyMyxBO3fnGFPRFQPV7MZTl3MehO6RXfG2ZLzOFOc5c3SqA4JMSaUWJ0osNivfHAAYtgTEdVDxaC82va0vxr941IAAJvP7fBKTXRlCTHu1Q+zgvRSPkfjk0deXi7mz/8ImzdvxMWLeQgLC0dKSl9MnvxnxMcnVDn2hx++RWrqEpw5cxpmcxhuvnk4Jk9+HEajUaXqifzjelv2ANApqgPMOhN2XNiF0e3v4Jx7P0iIcZ+kZeaUoGubJipX439s2RMAd9D/6U8P45tvvkLLlq1w3333o1Onzvj55x/xpz89jDNnTnuOXbx4AV577WXIsoIxY8ahXbv2WLZsCf72tylwOBwq/lcQ+Z5kDgNw9evj1/gaooS+zZJR6izDntz93iqN6hDPlj0RMH/+R8jOvoApU57B+PEPeh5fvfoHzJjxAt5//1288ca7OH/+PObN+y+6du2O99//CBqN+5/QvHn/xcKF87By5VcYM2acWv8ZRD4n6HQQtNp6rY9fk/7NUvDL6fXYem4HkmO7e6k6qk1spAEaSURmTnBOv2PLngAA69f/ioiISIwd+0CVx2+55TbExydg27YtkGUZ33yzHC6XCxMnPuIJegCYOPERhIaGYtWqb/xdOpFfCYJQvj7+tU29q9Dc1AyJ5gTsv3gYhbbrey26MkkU0TzaiLN5JZDl4BuRz7AnT3j/8Y+PQaxhrW+tVgeHwwGHw4GMjF0AgF69kqsco9fr0aVLdxw9ehiW62zxEDV0kqn+m+HUpH9cSvmc+11eqIquJCHGBIdTxoX8UrVL8TuGPUGSJIwdez9Gj76v2nOnTp3E6dMnER+fAL1ej6ysTERFNYHRGFrt2Li4OADAmTOnfF4zkZoksxmK3Q7ZZruu10lp2hMaQcKWczuCdv63P1UM0ssKwkv5DHuqlSzLeOedNyHLMu666x4AQFFRIUy1rAceGup+nC17CnSV6+Nf37/1UK0R3WK64FzJBZwuzvRGaVSHiul3wbiSHgfo1eGHMznYc9H9j0KSRLhcssoVVdctyoTbWsR4/XUVRcFbb81EWto2dOzY2dOX73Q6odXqavwZnc79uN0enItWUPCQzO4TW5elGNom1zeNq3+z3tiVvRtbzu1Ay7AW3iiPahF/yfS7YMOWPVXjdDrx+uszsGrV12jePB6zZr0NrVYLwN0373TWPL2uIuQNBoPfaiVSQ+WSudffb98pqgPCdGbsuJAOh4tTV30pwqRDaIgmKKffsWVfh9taxHhazTExZuTkBP4uVVarFS+8MB2bN29EQkIi/vWvDxAdXXnlwGwOq/UyfUmJ+/GKy/lEgapy57vr/0yomHO/5vRv2JN3gNPwfEgQBMTHmHDkTAFsDhf0WkntkvyGLXvyKCoqwtNPP47NmzeiQ4ckzJkzD82aNatyTIsWicjPvwibzVrt58+dOwtRFNGiBS9FUuOj4OoHyF26p7039GvWGwCwhcvn+lxCTCgUAGdzg+tSPsOeAAA2mw3Tpz+D/fv3omfPZLz33oeIjIyqdlz37j0hyzIyMtKr/fy+fXvQunWbGkfqEzVUwjX8jDdb9oB7zn1LcwvszzvEOfc+VrlsbnBdymfYEwDgo4/+H/bs2Y2uXbvj7bf/U+ul+FtuuRWSJGH+/I+qDMRbvHgBSkpKPKP2iQKZt0bjX6p/XG8oULDtPPe596VgnX7HPntCXl4uvvrqCwBAy5at8Omnn9R43IMPTkJiYiuMH/8gPvvsE/zxjxMwcOBgnDx5HJs2bUC3bj1w550Mewp8Gi+37AGgd9OeWH5kFbacT8PwxCEQhGu55kBXEh+k0+8Y9oR9+/Z6NrD57ruVtR43duwD0Ov1ePzxKYiNbYoVK77El19+jqioJhg37gE88shjnul3RIFMDA0FBMFrffaAe85995gu2Jm9G6eKz6BVWKLXXpsqGfQaNAkLCbrpdwx7wo033oQNG65+YJAgCBgzZizGjBnrw6qIGi5BFCGFmrzasgfcy+fuzN6NLefSGPY+FB8Tit3H8lBUakeYMTgaKOyzJyK6BpLZDKcXW/YA0DGyPcI5597ngrHfnmFPRHQNJJMJckkJFNl7K2u659z3RpmzDLu5z73PBOOyuQx7IqJrIJnNgKLAVeLdwOgXVz7n/jzn3PtKZcueYU9ERHXw5pK5l4oLbYqWYS1wIO8wCmyFXn1tcmvWxAhJFIJqkB7DnojoGngW1vHBLo/9m6VAgYLt57nPvS9oJBHNmhiRlVMCOUi2FmbYExFdg8pV9Ly/4l1K0x7QCBI2c597n0mIMcHmcCG3sPrS34GIU++IiK6BZKrY5vbqW/bHD+Vg97ZMdO+bUOdxxkvm3J8sOoPW4ZyG523x0e5BenNW7IUxpDIKtRoR993U1rMdbqBg2BMRXQMxxL2Vs2y9+pbhTyv2AQA69YyDVlf3jmudm3TEzuzdOGs5x7D3gS6to/DtppM4daH6mIt28eEMeyIiAnBdy9le+dK8JLCX1Zdax4Xhg/8ZUqWbZP/JfLybmlGP/Q8bD4Y9EREFJVEQqpy0iQG8HwFPHYmIiAIcw56IiCjAMeyJiIgCHMOeiIgowDHsiYiIAhzDnoiIKMAx7ImIiAIcw56IiCjAMeyJiIgCHMOeiIgowDHsiYgAAP5bKrU+u9YqAblSO/mboHCzZCIiooDGlj0REVGAY9gTEREFOIY9ERFRgGPYExERBTiGPRERUYBj2BMREQU4hj0REVGAY9hfo507d2L69OmYPn06ioqK1C6HiBqxzZs345///KfaZVAAY9hfo9TUVMyYMQP33nsvvv/+e7XLIaJG6tSpU9i/fz9sNpvapVAAY9hfI5fLBb1ej5iYGOTk5KhdDhE1Ui1btsTkyZPVLoMCHMP+GhkMBtjtduTk5CA6OlrtcoiIiGrFsK9BRkYGJk6cCACQZRkvvvgixo0bh4kTJ+LUqVMAgLFjx+LFF1/E559/jrvuukvNcomogbqazxIif9CoXUBDM3fuXKxcuRIGgwEAsGbNGtjtdixbtgzp6emYNWsW5syZg65du2LWrFkqV0tEDdXVfpZUmD17tlqlUhBgy/4yiYmJeO+99zz309LSMHjwYABAz549sXfvXrVKI6JGhJ8l1JAw7C8zcuRIaDSVFzwsFgtMJpPnviRJcDqdapRGRI0IP0uoIWHYX4HJZEJJSYnnvizLVd7ARERXg58lpCaG/RUkJydj/fr1AID09HR06NBB5YqIqDHiZwmpiaeVVzBixAhs3LgR48ePh6IomDlzptolEVEjxM8SUpOgKIqidhFERETkO7yMT0REFOAY9kRERAGOYU9ERBTgGPZEREQBjmFPREQU4Bj2REREAY5hT0REFOAY9kQBIjMzE0lJSZ4tVQHAZrNh/vz5KlZVacOGDdi9e7fn/tatW5GUlITXXntNxaqIggPDniiAPfjgg/jggw/ULgNLlizB5MmTkZ2d7XksPj4eU6ZM8ewER0S+w+VyiQJYXl6e2iUAqLmOhIQEPPXUUypUQxR82LInIiIKcAx7ogBU0X+flZWF4uJiJCUl4e9//7vneYvFgtmzZ2P48OHo2rUrBg8ejJdeeqlaC/zvf/87kpKSsHv3bowaNQrdunXzbOQCAOvWrcOjjz6K/v37o0uXLujfvz+efPJJHDhwwPMaEydOxPvvvw8A+Mtf/oKkpCQAtffZnzhxAs899xwGDhyIrl27Yvjw4XjzzTdRXFxcY22FhYV46aWXMGjQIHTr1g2jR4/GTz/95L0/JlEA4GV8ogAUFhaGKVOm4JNPPoHNZsNjjz2GTp06AQCKi4vxwAMP4PDhwxgwYABuueUWZGZmIjU1Fb///js+//xzxMbGVnm9J554At26dcOgQYNgNBohCAI+/fRTvPLKK0hMTMQdd9wBrVaLPXv24JdffsGWLVvw448/IjY2Fvfccw8AYNu2bRg1ahTatGlTa90ZGRmYNGkSrFYrhg4dihYtWiA9PR0ff/wx1q1bh6VLlyIiIqLKzzzyyCMoKCjAbbfdhtLSUqxatQp//etf8emnnyIlJcW7f1iiRophTxSAwsLC8NRTT2HFihUoKiqq0jf+zjvv4PDhw3jxxRcxYcIEz+O//PILnnzySbz22mv497//XeX1kpOT8d5773nu2+12vPvuu2jVqhVWrFgBo9Hoee7ll1/G0qVLsW7dOowbNw6jR49GVlYWtm3bhttvvx3Dhw+vsWaXy4Vp06bBbrfjww8/xI033uh5bvbs2Zg7dy7efPPNalvDSpKEb7/91lPDgAED8NxzzyE1NZVhT1SOl/GJgojT6cTXX3+N9u3bVwl6ABg2bBiSk5Px888/w2KxVHlu5MiRVe67XC688soreO2116oEPQD07dsXQP0HB+7atQsnT57E7bffXiXoAeDpp59G06ZNsWrVKtjt9irPTZgwoUoNQ4YMAQCcPHmyXr+fKJCxZU8URE6cOIHS0lK4XK4qLfUKNpsNLpcLhw4dQu/evT2Px8fHVznOYDBg1KhRntc8duwYTp8+jSNHjmDz5s0AAFmW61VbRT9/nz59qj2n0+nQrVs3rFmzBsePH0fHjh09z7Vu3brKsWazGQCqnRQQBTOGPVEQKSoqAgAcP37cM2iuJoWFhVXuh4SEVDtm+/bteP3117Fv3z4AgF6vR8eOHdGlSxecO3fOM4jvalVcTTCZTDU+XzGOoKysrMrjOp2uyn1BEACg3r+fKJAx7ImCSGhoKADgD3/4A958881rfp2srCz86U9/gl6vxyuvvILevXujVatWkCQJ33//PdasWXPNtV268M6lKk5ULh+gR0RXxj57oiDSunVr6HQ67Nu3r8aW78KFC/HBBx8gPz+/ztdZs2YNysrK8PTTT2Ps2LFo27YtJEkCABw7dgxA1ZZ1RWu7LhWzBdLS0qo9J8sy0tLSYDQaq3UpENGVMeyJAphWq4XT6fTc1+v1GDVqFI4ePYoFCxZUOXbr1q148803sXz5coSHh9f5unq9HgCQm5tb5fGDBw9i0aJFAFDl92o07ouIdfWj9+7dGy1btsTq1avx22+/VXnuP//5D86dO4fbbrut2mV7IroyXsYnCmCxsbE4efIknnvuOdxwww24++67MX36dOzatQtvvPEGfvnlF3Tv3h0XLlzA6tWrodFoMHPmTIhi3e2AoUOH4u2338aHH36I48ePIzExEadOncK6des8A+QKCgo8xzdt2hQAMGfOHBw4cABTpkyp9pqiKGLWrFmYPHkyHn/8cQwdOhSJiYnYtWsX0tPT0bZtW0ybNs17fxyiIMKWPVEAmzp1Ktq3b48ff/wR33zzDQAgKioKqamp+OMf/4gLFy5g8eLF2LFjB26++WakpqaiX79+V3zdpk2bYsGCBejfvz+2bNmCJUuW4MSJE5g4cSJ++OEHRERE4Pfff/dcyh81ahRuu+02nDlzBkuWLEFWVlaNr5ucnIwvv/wSo0aNwq5du/DZZ5+hoKAATzzxBL744gv21xNdI0HhkFUiIqKAxpY9ERFRgGPYExERBTiGPRERUYBj2BMREQU4hj0REVGAY9gTEREFOIY9ERFRgGPYExERBTiGPRERUYBj2BMREQW4/w+flLISiXr+jQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x396 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_log_lines(to_plot, \n",
    "           m_values, \n",
    "           [\"Iteration\", \"Subopt\"], \"figs\", \"MPI.pdf\", show = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f1064cc",
   "metadata": {},
   "source": [
    "**Questions** \n",
    "- How does the value of $m$ affects convergence ?\n",
    "- Prove that MPI converges at least linearly with parameter $\\gamma$. (prof outline )\n",
    "- Which is the cost per iteration of Modified Policy Iteration."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54dbec0f",
   "metadata": {},
   "source": [
    "***Answers***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "690a0128",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a59265d9",
   "metadata": {},
   "source": [
    "# Ex 4: Q-Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7252fd60",
   "metadata": {},
   "source": [
    "Now, we are ready to tackle the problem in the model free setting. For technical reason, we focus on the finite horizon setting in this part and we compare two exploration strategies.\n",
    "1. **$\\varepsilon$-greedy** (Watkins, Christopher John Cornish Hellaby. “Learning from Delayed Rewards.” PhD Thesis, King’s College, Cambridge United Kingdom, 1989.)\n",
    "  - Initialize $V_h(x) \\leftarrow H, Q_h(x,a) \\leftarrow H$ for all $(x,a,h)\\in \\mathcal{S} \\times \\mathcal{A} \\times [H], V_{H}(x) \\leftarrow 0$ for all $x \\in \\mathcal{S}$  \n",
    "  - For episode $k=1,\\dots,K$ do\n",
    "    - Receive $x_1$\n",
    "    - For step $h=1,\\dots,H$ do\n",
    "      - Sample $b_h \\sim \\text{Bernoulli}(\\varepsilon), \\hat{a}_h \\sim \\text{Uniform}(\\mathcal{A})$\n",
    "      - Take action $a_h \\leftarrow \\left[(1-b_h) \\left(\\underset{a' \\in \\mathcal{A}}{\\text{argmax}}{Q_h(x_h,a')}\\right) + b_h \\hat{a}_h \\right]$ and observe $x_{h+1}$\n",
    "      - $Q_h(x_h,a_h)\\leftarrow (1-\\alpha)Q_h(x_h,a_h) +\\alpha\\left[r_h(x_h,a_h) + V_{h+1}(x_{h+1})\\right]$\n",
    "      - $V_h(x_h)\\leftarrow \\min{\\left\\{H, \\underset{a' \\in \\mathcal{A}}{\\max}{Q_h(x_h,a')}\\right\\}}$\n",
    "2. **UCB bonuses** (Jin, Chi, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. “Is Q-Learning Provably Efficient?” In Advances in Neural Information Processing Systems, Vol. 31. Curran Associates, Inc., 2018. https://proceedings.neurips.cc/paper/2018/hash/d3b1fb02964aa64e257f9f26a31f72cf-Abstract.html.)\n",
    "  - Initialize $V_h(x) \\leftarrow H, Q_h(x,a) \\leftarrow H$ and $N_h(x,a) \\leftarrow 0$ for all $(x,a,h)\\in \\mathcal{S} \\times \\mathcal{A} \\times [H], V_{H}(x) \\leftarrow 0$ for all $x \\in \\mathcal{S}$  \n",
    "  - For episode $k=1,\\dots,K$ do\n",
    "    - Receive $x_1$\n",
    "    - For step $h=1,\\dots,H$ do\n",
    "      - Take action $a_h \\leftarrow \\underset{a' \\in \\mathcal{A}}{\\text{argmax}}{Q_h(x_h,a')}$ and observe $x_{h+1}$\n",
    "      - $t=N_h(x_h,a_h)\\leftarrow N_h(x_h,a_h) + 1, \\alpha_t = \\frac{H+1}{H+t}$ and $b_t \\leftarrow c\\sqrt{\\frac{H^3}{t}}$\n",
    "      - $Q_h(x_h,a_h)\\leftarrow (1-\\alpha_t)Q_h(x_h,a_h) +\\alpha_t\\left[r_h(x_h,a_h) + V_{h+1}(x_{h+1}) + b_t\\right]$\n",
    "      - $V_h(x_h)\\leftarrow \\min{\\left\\{H, \\underset{a' \\in \\mathcal{A}}{\\max}{Q_h(x_h,a')}\\right\\}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "184330cc",
   "metadata": {},
   "source": [
    "For other technical reason, we also need to rescale the reward between 0 and 1, as done in the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "5ff59bfb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:57:07.438599Z",
     "start_time": "2023-03-14T00:57:07.314600Z"
    }
   },
   "outputs": [],
   "source": [
    "reward_mode = 0\n",
    "size = 10\n",
    "gridworld = GridWorldEnvironment(reward_mode, size, prop=0, gamma=gamma)\n",
    "r_max = np.max(gridworld.r)\n",
    "r_min = np.min(gridworld.r)\n",
    "gridworld.r = (gridworld.r - r_min) / (r_max - r_min)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7545ef1",
   "metadata": {},
   "source": [
    "### Q learning with epsilon greedy "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17117c6d",
   "metadata": {},
   "source": [
    "In this subsection, we implement $Q$ Learning with the simplest possible exploration strategy: $\\epsilon$-greedy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "3a5b63f2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:57:07.549598Z",
     "start_time": "2023-03-14T00:57:07.439601Z"
    }
   },
   "outputs": [],
   "source": [
    "def argmax_with_random_tie_breaking(b):\n",
    "    return np.random.choice(np.where(b == b.max())[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b6e5f11f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:59:43.054854Z",
     "start_time": "2023-03-14T00:57:07.551597Z"
    }
   },
   "outputs": [],
   "source": [
    "def q_learning_epsilon_greedy(K: int = 100000, H: int = 20, epsilon: float = 0.1, alpha: float = 0.1) -> List[float]:\n",
    "    \"\"\"\n",
    "    Function implementing the epsilon-greedy Q-learning algorithm.\n",
    "\n",
    "    :param K: Number of episodes, positive int\n",
    "    :param H: Number of steps per episode, positive int\n",
    "    :param epsilon: Algorithm hyperparameter, exploration probability, float in [0, 1]\n",
    "    :param alpha: Algorithm hyperparameter, Q update weight, float in [0, 1]\n",
    "\n",
    "    :return: reward after each step, list of K * H floats\n",
    "    \"\"\"\n",
    "    # Initialize tabular records\n",
    "    rewards = []\n",
    "    Q = H * np.ones((H, gridworld.n_states, gridworld.n_actions))\n",
    "    V = H * np.ones((H + 1, gridworld.n_states))\n",
    "    V[H, :] = 0\n",
    "\n",
    "    for k in range(K):  # Episode loop\n",
    "        state = 99  # Initial state\n",
    "        for h in range(H):  # Step loop\n",
    "            explore = np.random.binomial(2, p=epsilon)\n",
    "            if explore:\n",
    "                # Exploration: With probability epsilon take a random action\n",
    "                a = np.random.choice(gridworld.n_actions)\n",
    "            else:\n",
    "                # Exploitation: With probability 1 - epsilon take one of the optimal actions for the current state\n",
    "                a = argmax_with_random_tie_breaking(Q[h, state, :])\n",
    "\n",
    "            # Get reward for action\n",
    "            rewards.append(gridworld.r[state, a])\n",
    "\n",
    "            # Get the new state according to the transition dynamics\n",
    "            new_state = np.random.choice(gridworld.n_states,\n",
    "                                         p=gridworld.T[a][state])\n",
    "\n",
    "            # Update Q according to the algorithm\n",
    "            Q[h, state, a] = ???\n",
    "\n",
    "            # Update V as the Q-value of the optimal actions for the current state\n",
    "            V[h, state] = np.min([np.max(Q[h, state, :]), H])\n",
    "\n",
    "            state = new_state\n",
    "    return rewards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "07817512",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T00:59:43.293855Z",
     "start_time": "2023-03-14T00:59:43.055853Z"
    }
   },
   "outputs": [],
   "source": [
    "reward_eps_greedy = q_learning_epsilon_greedy()  # You can play around with the arguments if you like\n",
    "cumulative_reward_eps_greedy = np.cumsum(reward_eps_greedy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2e809e2",
   "metadata": {},
   "source": [
    "### Q-Learning with bonuses"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c49d6483",
   "metadata": {},
   "source": [
    "In this subsection, you are asked to implement an optimistic version of  Q Learning suggested in (Jin et al., 2018)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "ff556a7b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T01:02:45.032943Z",
     "start_time": "2023-03-14T00:59:43.294856Z"
    }
   },
   "outputs": [],
   "source": [
    "def q_learning_ucb_bonuses(K: int = 100000, H: int = 20, c: float = 0.0001) -> List[float]:\n",
    "    \"\"\"\n",
    "    Function implementing the Q-learning with UCB bonuses algorithm.\n",
    "\n",
    "    :param K: Number of episodes, positive int\n",
    "    :param H: Number of steps per episode, positive int\n",
    "    :param c: Algorithm hyperparameter, constant which scales the bonuses, positive float\n",
    "\n",
    "    :return: reward after each step, list of K * H floats\n",
    "    \"\"\"\n",
    "\n",
    "    # Initialize tabular records\n",
    "    rewards = []\n",
    "    Q = H * np.ones((H, gridworld.n_states, gridworld.n_actions))\n",
    "    V = H * np.ones((H + 1, gridworld.n_states))\n",
    "    V[H, :] = 0\n",
    "    N = np.ones((H, gridworld.n_states, gridworld.n_actions))\n",
    "\n",
    "    for k in range(K):  # Episode loop\n",
    "        state = 99  # Initial state\n",
    "        for h in range(H):  # Step loop\n",
    "            # Take one of the optimal actions for the current state\n",
    "            a = argmax_with_random_tie_breaking(Q[h, state, :])\n",
    "            rewards.append(gridworld.r[state, a])\n",
    "\n",
    "            # Record that we visited this state-action pair (again)\n",
    "            N[h, state, a] += 1\n",
    "\n",
    "            # Get the new state according to the transition dynamics\n",
    "            new_state = np.random.choice(gridworld.n_states,\n",
    "                                         p=gridworld.T[a][state])\n",
    "\n",
    "            # Calculate the UCB bonus\n",
    "            bonus = c * np.sqrt(H ** 3 / N[h, state, a])\n",
    "\n",
    "            # Calculate the adaptive alpha according to the algorithm\n",
    "            alpha = (H + 1) / (H + N[h, state, a])\n",
    "\n",
    "            # Update Q according to the algorithm\n",
    "            Q[h, state, a] = ???\n",
    "\n",
    "            # Update V as the Q-value of the optimal actions for the current state\n",
    "            V[h, state] = np.max(Q[h, state, :])\n",
    "            V[h, state] = np.clip(V[h, state], 0, H)\n",
    "\n",
    "            state = new_state\n",
    "\n",
    "    return rewards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "ca3125a5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T01:02:45.254892Z",
     "start_time": "2023-03-14T01:02:45.033927Z"
    }
   },
   "outputs": [],
   "source": [
    "reward_UCB = q_learning_ucb_bonuses()  # You can play around with the arguments if you like\n",
    "cumulative_reward_UCB = np.cumsum(reward_UCB)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d470164",
   "metadata": {},
   "source": [
    "### Comparison of exploration strategies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "b234b79f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-14T01:02:48.908004Z",
     "start_time": "2023-03-14T01:02:45.256893Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgwAAAFgCAYAAADXWCGzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABs7klEQVR4nO3dd3xT9f748Vea0ZF00rJbdhktZZWhFBCRKSCyWzYoioBXriLCD0W+COpV7lVBHHhFRYYsEURRQASBssoulL1n6U5H0iTn90chlwqlKbSk4/18PLTJOZ9z8n73kOadz/mcz1EpiqIghBBCCHEfLs4OQAghhBDFnxQMQgghhMiXFAxCCCGEyJcUDEIIIYTIlxQMQgghhMiXFAxCCCGEyJfG2QEUZ/HxaYW6P19fD5KSMgp1n84iuRQ/pSUPkFyKq9KSS2nJAwo/l4AAzzzXSQ/DI6TRqJ0dQqGRXIqf0pIHSC7FVWnJpbTkAY82FykYhBBCCJEvKRiEEEIIkS8pGIQQQgiRLykYhBBCCJEvKRiEEEIIkS8pGIQQQgiRLykYhBBCCJEvKRiEEEIIkS8pGIQQQgiRLykYhBBCCJEvuZeEEEIIUQKdSTnPDUVHeVWlR/J6UjAIIYQQJYjFZuHnM7+z4cKfBHpV4o3wCY/kdaVgEEIIIUqI+IwEFsQu5nzaRQLcyzG25XCwPprXloJBCCGEKAH2XtvPkuOryLKaaFGxKQOCexHoF0B8fNojeX0pGIQQQohiLMtiYvnJn9h5dS+uah1D6w+gZaVmjzwOKRiEEEKIYupi2mW+jl3EjYybBHlWYURIFOU9ApwSixQMQgghRDGjKAp/XtrO6lPrsChWngxswzO1uqJxcd7HthQMQgghRDFiNKez8NgyjiQcw6DVM7TBAELK1XN2WDJxkxCF4eTJ40REhDNz5tvODkUIUYKdSDrFrN3/4UjCMer51mFKiwnFolgA6WEQQgghnM5qs/LL2Q38dn4zKpWKZ2p25alq7XBRFZ/v9VIwCCGEEE6UkJnEN0cXcyblPOXcfBkREkUN72rODusuTi8YDh48yIcffsjChQuZMGECN2/eBODy5cs0atSI//znP7zzzjvs27cPvV4PwLx589BqtUycOJGEhAT0ej3vv/8+fn5+HDhwgJkzZ6JWq4mIiGDcuHEAzJ07lz///BONRsOUKVMICwtzWs5CCCEEwL4bh1gct4JMSxbNyjcisl5v3DXuzg7rnpxaMMyfP581a9bg7p7zy/nPf/4DQEpKCkOHDmXy5MkAxMbG8tVXX+Hn52ffdsGCBQQHBzN+/HjWrVvHvHnzmDp1KtOmTWPOnDkEBgYyevRoYmNjAdi9ezfLly/n6tWrjB8/npUrVz7ibMuO8+fPsXz5Uvbu3UV8/A3Uag3+/v7Urh3M1KnT0el0993+8uVLfPnlPPbt20tWViaNGjVh3LgJTJo0gYCA8syd+yUA48aN5tq1q7z66hvMnv0eSUmJPP54G2bMeA+A9HQj3323gM2bNxIffwNvbx8iItoyatQL+Pr65XrNgrQ9deokX331GQcPHkBRFNq2fYLOnbvlavPNN1/x1VefM2nSVHr06JVr3bVrV+nXrycdO3bhrbdmPMivWAhRwpmtZlacXMv2K7vQuWgZVK8fj1UKR6VSOTu0PDm1YAgKCmLOnDm8/vrruZbPmTOHwYMHU758eWw2G+fPn+ett97i5s2b9O3bl759+xITE8Nzzz0HQNu2bZk3bx5GoxGz2UxQUBAAERERREdHo9PpiIiIQKVSUblyZaxWK4mJibkKEFE49u3by2uv/QNQePzxCJ54ogNZWZlcunSRU6dO5FssXLp0kRdfHElqagpt2rSjcuWqbN++lZdeeg6bzUZAQPlc7VNSUnjrrcm0adMODw891atXB8BoNPLSS6M4c+Y0zZq14IknnuTKlcusWfMjO3fu4PPPF+Dv7w9AWloaY8Y41vbkyeOMHTsas9nEE090wNPTi23btrBr145ccXXu3I3//vcLNmxYf1fB8Pvvv6Ioyl1FhhCibLhsvMrXsYu5ln6dKoZKjAwZREV9+fw3dDKnFgydO3fm0qVLuZYlJCQQHR1t713IyMhg8ODBjBgxAqvVytChQwkNDcVoNOLp6QmAXq8nLS0No9GIwWCw70uv13Px4kVcXV3x8fHJtTwtLS3fgsHX1wONRl1I2eYICPAs1P05071yWbDgC6xWC8uXLyckJKTA+3zzzU9ITk7i448/pkuXLgCYzWaGDx9OTEwMOp3G/ro6nYbMzAxGjBjBG2+8kWs/8+b9mzNnTvPWW28xaNAg+/JNmzbx0ksv8fnnH/Hxxx8DMH36dIfbTpjwESZTFv/973957LHHAEhMTGTIkCEkJCTg5qYlIMCTgIC6hIeHExMTg82WQYUKFez7/eOP3wkICKBr1w6o1fLvKy+SS/FUWnJxRh6KorDh9Fa+PbCSbGs2XWo/weDGvdGptQ+130eVi9PHMPzd+vXr6d69u/0Pqbu7O0OHDrWftmjVqhVxcXEYDAbS09MBSE9Px8vLK9eyO5drtdq7lt8uNu4nKSnjofNZ9scp9sTdAECtVmG1Kg+9z8LWvF55+j9Zu0DbBAR43nP+8ps3E9DrDXh7Vyjw/ObJycls2bKFRo2a0KxZ61zbP/fcS8TEjMJsttiXm80WAFq2bJOrrcVi4ccfV1OjRk06deqZa11YWAsaNmzEhg0bOHfuKq6ubqxe7VjbzMxM9uzZw+OPR1C7dugdbbUMGTKSadOmkJWVbV/eoUMX9uzZw/LlPzJgQE4hcuJEHKdOnWLAgEEkJj78v6875XVMSiLJpXgqLbk4I4/07AwWxa3gYPwR9BoPRjaMIiwghJTELCDrgfdb2Lncr/godgVDdHQ0Y8aMsT8/d+4cEyZM4Mcff8Rms7Fv3z6effZZEhMT2bJlC2FhYWzdupVmzZphMBjQarVcuHCBwMBAtm3bxrhx41Cr1XzwwQeMGjWKa9euYbPZ5HREERk//p+8++7/MWrUYFq2fBwPDw/Cw1vQqFGTfLc9fvwYNpuN+vXv7plo0CA0z2/jFStWzvX8woXzZGZmYLPZ+O9/v7irvdlsxmq1cvr0KQwGTzIyHGt7u+isW7f+Xe1CQ+8eRPvkk0/xn//8iw0bfrMXDL//vh6ALl3kdIQQZcWp5LN8E7uEJFMydXxqMqzBQHzdfJwdVoEVu4Lh7NmzBAYG2p/XqlWLHj160L9/f7RaLc888wx16tShatWqTJo0icjISLRaLbNnzwZyupdfe+01rFYrERERNGrUCIDw8HAGDBiAzWbjrbfeemT59H+ytv3be2mpzvOiKAqJiQlUrFiRY8eOcu7cWQCqV69hb3Py5HG2bv3zrm1HjXqBlJRkAPz8yt21Xq1W3zX48DZXV9dcz43GnN/x+fPnWLBgfp7xpqam2h870jYjI6dHwMNDf9d6Ly/vu5Z5eOhp0+YJNm78jYsXL1ClSlU2bvyNmjVrUadO3TxfSwhROtgUG+vPbeKXsxsB6F6jE52rP1ms5lYoCKcXDFWrVmXZsmX25+vWrburzfPPP8/zzz+fa5m7uzuffPLJXW0bN26ca3+3jR8/nvHjxxdCxCIvH330AStXLqNXr75MnjyNqlUD7xrkePLkiXt+MI8a9YL9gzgjI/2u9TnLHevCd3f3AHIGHr755v/dt+3JkyccbhsdvR3IuaLi7zIz7x1b167d2bjxNzZv3khYWGNu3oynf//IfHMQQpRsSVnJfHt0KSeTz+Dr6sPwkEhq+9TIf8NizOkFgygdkpIS+fHHFbRo8RivvfZGnu26detBt2497rmubt16qFQqjh6NvWvd2bNn8iwk/i4oqBo6nY7jx+NQFOWuy5SWLVtMRkYGzz7bt0Btg4ProlKpOHz44F2vGRd37J6xhIe3wN8/gO3b/8JoTMPFxYWOHbs4lIcQomQ6GB/LomPLSbdk0DgglKh6fdFrPZwd1kMrmf0iothJSkrEZrORkZGO1Wq9a73JlP+gnoCA8jRv3pK9e3cRHb3NvtxsNvPZZ3f3JuXF1dWVJ5/syLlzZ1i6dFGudfv27eXTTz9m3bq1eHp64erqSrdu3RxqW66cPy1bPkZMzB7+/HOTvV16upEFC768ZyxqtZqOHbtw7FgsGzf+TtOm4XddGiqEKB2yrdksO7GaLw9/i9lmZmDdZ3kudEipKBZAehhEIQkKqk5gYBBHjhxi8OB+NG/eEr3eQEpKMmfPniEwMIgpU6blu59XXnmN0aNH8MYbr9KmTTsCAiqwZ89OkpOTABy+DHHs2Fc4cuQQn376Edu2baFBg1Bu3LjO1q2bUas1TJ78Ji4uOfXypEmT2Ls3xqG2Eya8zpgxo+xzPwQEVGD79r9wccl7spWuXZ9myZKF3LhxneefH5NnOyFEyXUt/Tpfxy7msvEqlfQVGBkyiMqGis4Oq1BJD4MoFBqNho8//owePZ7FYrGwZs2PLF++hL17d1OunD9PP93Tof0EBVXns8/+y2OPtWbv3t2sXfsjlStX5eOPPwfA1dXNof34+vry5ZffMHDgYOLjb7BixVIOHTpA69Zt+OKLBTRtGm5v6+fn53DbKlWq8sUXC+jQoRMHD+5n3bo1BAcH8957/84zlpo1a1O9eg3c3Nxo1+5Jh+IXQpQMiqKw48pu3tvzCZeNV4mo3JLXw8eXumIBQKUoSvGbGKCYKOwrGkrTVRJFkYvNZuPKlctUrFgJjSZ359eVK5fp3/8ZevXqe98xEg+iqI+L0WikZ8/OtG//JG++WXRTQcu/r+JJcil+CiuPTEsmS+JWEXPjIO4adwbV60uT8g0LIULHlel5GETZpVKpGDFiEAEBAXz77VK02v/NfrZ48UKAXN/2S4pFi77FbDbRo8ezzg5FCFFIzqacZ0HsYhKykqjpXY3hDaIo5+7r7LCKlBQMothQqVT06tWHJUsWMmzYQFq1ehwXFzWHDx8kNvYwLVo8Rvv2HZwdpsPGjn2e+PgbXLlymWbNmtO4cVNnhySEeEg2xcbG81tYe/Y3FEWhS/UOdKv+FGqXwp3mvTiSgkEUK2PGjKdatWqsWbOaX375GavVQuXKVXjxxXEMHDi4WN/J7e+8vLyIiztK8+Yt853jQQhR/KWYUvn26FKOJ53CW+fF8JBIgn1rOTusR0YKBlGsuLi40L17L7p37+XsUB7au+/OdnYIQohCEpsQx3dHf8CYnU5oufoMqd8fg+7uWV9LMykYhBBCiDxk2yysOf0rf1z8C41KTd86PXmiausS1dtZWKRgEEIIIe7hRkY8X8cu5mLaZSp4BDAiZBCBnpXz37CUkoJBCCGE+JtdV2NYeuJHzFYzrSqF06/OM7hpXPPfsBSTgkEIIYS4JcuSxQ8nVrP72j7c1K6MaBBJeMUmzg6rWJCCQQghhAAupF7i69hFxGcmUM0rkJEhUfi7l3N2WMWGwwXDpUuXqFq1alHGIoQQQjxyNsXG5ovb+On0r1gVKx2DnqBHzc5lYm6FgnC4YBg6dCgNGzbk448/Lsp4hBBCiEcmzWzku6M/cDTxOJ46A8MaDKS+X7CzwyqWHC4Ybt68SWBgYFHGIoQQQjwycYkn+eboEtLMRur7BTO0wQC8dHnfS6Gsc7hgaN68OTt27MBsNqPT6YoyJiGEEKLIWGxWVp/6hY0XtuCicuHZ2k/zZGAbXFRyA+f7cbhg6NevH++88w6dO3embdu2VKlSBTe3e99qeOjQoYUWoBBCCFFYbmYm8NGmeZxMPIe/ezlGhkRRzUt6zx3hcMHwyiuv2B//8MMPebZTqVRSMAghhCh29l4/wJK4VWRZs2heoQkD6j6Lu+beX3zF3RwuGN59992ijEOUAr/8spZZs6YzYsTzjBr1Qp7tIiLCqVixEitWrM21PDMzk99//5UNG9Zz8eJ5UlJSKFfOn6ZNw4mMHELNmrlv8vLf/37BggXz79q/h4eeKlWq8NRTnRkwYBAajVw9LERZZrKaWX7iJ6Kv7kGn1jG2xTAaGEKcHVaJ4/Bf0meffbYo4xBl3NmzZ/h//28iFy6cp3r1mjz+eBsMBk/Onj3N+vXr2LBhPTNmvEebNk/ctW2bNu2oXTtnVLPNZiM93cjBg/v57LM5xMYeYdasDx5xNkKI4uJS2hW+jl3E9Yx4Aj2rMCIkitBqNYmPT3N2aCVOoX31slgsJCcns2XLFvr06VNYuxVlQHJyMq+8Mobk5GRef/3/0aNHr1w3djly5DD//Oc43nprMl9++Q116tTNtX2bNk/QrVuPXMsURWHSpH+ydetmYmL20KxZ80eSixCieFAUhS2XdvDjqZ+xKFaeDGxDz1pd0bpIj+ODcvg3Z7VamT17Nj///DOJiYlYrdY820rBIApi7tz/kJCQwOjRL9Gz5909WaGhDRk79h988MEsvv/+G6ZPz//0mEql4umne7Bjx18cOLBPCgYhyhBjdjrfH1vO4ZtHMWj1DKnfn1D/+s4Oq8RzuGD46quv+Prrr9HpdFSrVo3z58/j6+uLp6cnV65cISsri3LlyvHCC3mfuxbi7zIy0tm8eSMeHnr69YvMs12XLk+TlJRIkybNHN63Wp0zS5tWq33oOIUQJcOJpNN8e3QpyaYUgn1rM6zBAHxcvZ0dVqngcMGwdu1avL29WbNmDRUqVGDEiBGUK1eODz/8EJPJxDvvvMOKFSuoXbt2UcYrSpl9+2IwmUy0bNkMd3f3PNu5uroyfPhzDu9XURR++eVn1Gr1Pcc9CCFKF6vNyq/nNrL+3B+oVCp61uxCx2pPyNwKhahA95Lo0aMHFSpUAKBhw4asWbMGyPljPn36dPbu3cs333zD448/XjTRilInPv4GAIGBQQ+8j7/++pOrV6/Yn2dkpLN/fwxnzpxmwoSJ1KhR8yGjFEIUZ4lZSSyIXcKZlHOUc/NleEgUNb2rOTusUqdAoz/8/Pzsj6tXr87169dJTU3Fy8sLFxcXIiIi2LRpU6EHWZKtOvUz+28cBkDtosJqU5wc0d2alG9I79rdnfLaRmPOSGUPD48H3sdff23hr7+23LXc09OLtLQ0rFar/fSEEKJ0OXDjMN/HrSDTkknT8mFE1u2Dhzbv3krx4BwuGCpXrszZs2ftz4OCcr4RnjhxgvDw8JydaTQkJiYWcoiipLjzyob82+Z0E3p55ZxbTEt78EucpkyZlusqiczMTM6fP8d///s5X3zxKRcvXmDKlGkPvH8hRPFjtmaz8tRatl3eidZFS1S9PjxeqUWB/g6JgnG4YGjXrh0LFy5k5cqV9OrViwYNGuDm5saiRYsIDw8nNTWVjRs32k9ZOOrgwYN8+OGHLFy4kNjYWF588UWqV68OQGRkJN26dWPZsmUsXboUjUbDmDFjaN++PVlZWUycOJGEhAT0ej3vv/8+fn5+HDhwgJkzZ6JWq4mIiGDcuHEAzJ07lz///BONRsOUKVMICwsrUJwPqnft7vZv7wEBnqX62l+93gDkXGKbF5Mp61ZbPQCVK1cB4PLli/nu/9Kli1SsWCnfiZjc3d2pV68+s2Z9SP/+z/DLL2sZPHgYQUHVHUlDCFHMXTFe4+vYRVxNv05lfUVGhg6ikr5gnz2i4BwuGEaPHs2GDRuYOnUqVquV/v37079/f7799lv27NmDyWTCaDQyfvx4h198/vz5rFmzxj7Y7ejRo4wYMYKRI0fa28THx9sLFZPJRFRUFK1bt2bJkiUEBwczfvx41q1bx7x585g6dSrTpk1jzpw5BAYGMnr0aGJjYwHYvXs3y5cv5+rVq4wfP56VK1c6HKdwjLe3DwBJSXn3MsXHx+dq26hRE9zd3Tl06AAmUxaurveepjU7O5vnnx+GzWZl+fK1eHl55RuPVqslNDSMzZs3curUKSkYhCjhFEVh25VdrDy5hmybhbZVHqd37afRquVKqEfB4eGjvr6+rF69mgkTJti/nb/66qtERkZiNptxdXVlxIgRjB492uEXDwoKYs6cOfbnR44c4c8//2TQoEFMmTIFo9HIoUOHaNKkCTqdDk9PT4KCgoiLiyMmJoY2bdoA0LZtW6KjozEajZjNZoKCglCpVERERBAdHU1MTAwRERGoVCoqV66M1WqVUydFoG7demi1Wg4dOoDNZrtnm0OHDgDQoEHOtKw6nY4OHTqRlZXF0qWL8tz3r7/+TFpaKnXr1neoWLgtLS0VAIPB4PA2QojiJyM7g6+OfM/S46vQuegY3XAYA+r2kmLhEcqzh+HFF1+kS5cu9OrVC4ArV67g5eWVqyDQ6XRMmzaNadMe7Pxw586duXTpkv15WFgY/fr1IzQ0lM8++4xPP/2UevXq4en5v/uT6/V6jEYjRqPRvlyv15OWlobRaMz1waDX67l48SKurq74+PjkWp6WlpZrEOe9+Pp6oNEU7mC5gIDSc6/1u3PxpFOnTqxbt45ly767q7fpxo0bfP/9ArRaLZGR/ezbT578Olu2/MGCBfOpVq0Kffv2zXUecuvWrXzyyWw0Gg1vvPG6fTu93jXnVT3d7vl7PXjwIAcO7MPb25sOHdrkeXfVe+dSMpWWPEByKa6ckUtc/Gk+ifmamxmJ1A+ow/hWw/H3uP/f7/zIMSm4PAuG6Oho6tb93xS8HTp0YOzYsfYxAUWhY8eO9m+PHTt2ZMaMGYSHh5Oenm5vk56ejqenJwaDwb48PT0dLy+vXMvuXK7Vau+5j/wkJWUUVmpA6RrDkFcuzz8/nsOHjzB37lx+/fU3mjTJmV/hypXL7NixDZMpiwkTXsfLq/wd27sye/YcXn/9FaZOncpXX31NkybN0Gg0nDx5nAMH9qHVapkyZRqVK/9vDvj0dBMAv/yynhMnzthjsNlsnD17hh07/sJqtTJ+/D9JS8smLS27QLmUNKUlD5BciqtHnYtNsfHbuc2sO/s7AN1qdKRLtSdR0tXEpz94HHJM7r+/vORZMLi5ubFjxw769u2Lj48PiqJgNpsxGo35vuCDdv+OGjWKN998k7CwMKKjowkJCSEsLIyPPvoIk8mE2Wzm9OnTBAcH07RpU7Zs2UJYWBhbt26lWbNmGAwGtFotFy5cIDAwkG3btjFu3DjUajUffPABo0aN4tq1a9hstnx7F8SD8ff3Z/78b1m5chl//vkHv//+K1lZmZQr50/r1m3o23cgoaEN79quQYNQFi5cxk8/reKvv7awefNGjMY0/P0D6N79GSIjh1CtWvV7vubfL6vUaDT4+vrZX68gs0MKIYqHZFMK38Qu4WTyGXxcvRkREkVtnxrODqtMUymKcs+JASZPnsyPP/5Y4EtUVCoVR48edbj9pUuX+Oc//8myZcuIjY1lxowZaLVa/P39mTFjBgaDgWXLlvHDDz+gKAovvPACnTt3JjMzk0mTJhEfH49Wq2X27NkEBARw4MABZs2ahdVqJSIiggkTJgAwZ84ctm7dis1mY/LkyfZLQe+nsCtQqWqLp9KSS2nJAySX4upR5XL45lEWHltGenYGjfxDGFS/H3rtg8/V8ndyTO6/v7zkWTBkZmYyb948jh49itlsZu/evVSqVIkqVark+4ILFy588GiLESkY8ia5FD+lJQ+QXIqros4l25rN6tO/8Oel7WhcNPSp3Z02VR4r9LkV5Jjcf395yfOUhLu7O6+++qr9eb169ejdu3eRjmEQQghRNl1Lv8HXsYu4bLxKRY/yjAwdRBVDJWeHJe7g8DwM3333nUO9C0IIIYSjFEVh59W9LDuxGrMtm9aVW9C3Tk90ap2zQxN/43DB0KJFi6KMQwghRBmTaclkSdwqYm4cxF3jxqgGA2ha/tHMwisKrkA3nxJCCCEKw9mUCyyIXUxCViI1vKoxIiSScu5y9VpxJgWDEEKIR8am2Nh0YStrzqxHURQ6V3uSp2t0RO0id5Qt7qRgEEII8UikmNL47uhS4pJO4q3zZFiDSOr61XZ2WMJBUjAIIYQocrEJx/nu6FKM2emElqvH4Pr98dTJPV5KkgcqGK5cuUJcXBxZWVn4+PhQq1atAt/WWgghROlnsVlYc3o9my5uRaNS06dOD9pXjSj0uRVE0StQwXDp0iXefPNNdu7cmWu5SqWiVatWTJ8+ncDAwEINUAghRMl0I+MmC2IXcyHtEuU9/BkZMohAT7k8v6RyuGCIj48nMjKS+Ph4GjZsSNOmTSlfvjypqans3r2bHTt2MGTIEFatWiX3aRBCiDJu97V9LD2+CpPVTMuKzegf3As3jauzwxIPweGCYe7cucTHx/P2228zcODAu9YvX76cN998ky+++ILJkycXapBCCCFKhixLFstO/MSuazG4qV0Z1mAgLSo2dXZYohC4ONpwy5YttG7d+p7FAkC/fv1o3bo1mzZtKrTghBBClBwX0i7x/p5P2HUthmqegbzR/BUpFkoRh3sYbt68SdeuXe/bJjg4mD179jx0UEIIIUoORVHYfPEvVp/+Fati5amgdvSo2RmNi1yIV5o4fDT9/f05ceLEfdscP34cX1/fhw5KCCFEyZBmNrLw2DJiE+Lw1BoY2mAADcrVdXZYogg4fEqibdu27Nixg5UrV95z/dKlS4mOjqZdu3aFFpwQQojiKy7xJLN2/4fYhDjq+wUzucUEKRZKMYd7GMaPH8+mTZuYOnUqq1evJjw8HE9PT27cuMHevXuJjY2lXLlyjB07tijjFUII4WRWm5Wfz/7OhvN/olKp6FWrGx2C2uKicvg7qCiBHC4YAgICWLp0KVOnTmXXrl13jVVo2bIl//d//ycTOAkhRCl2MzORb2IXczb1Av5ufowIjaK6V5CzwxKPQIFGpAQGBvLtt99y/fp1jh49itFoRK/XU79+fSpVqlRUMQohhCgGdlyI4fPd35NlzSK8QmMG1u2Nu8bN2WGJR+SBhrBWqFDB3pNgtVq5dOkS6enp6PX6Qg1OCCGE85msZlacWMOOq7vRuWgZXL8/rSo2k+mdy5gCnXDas2cPr7zyClarFYC4uDg6dOhAly5dePzxx5k7d26RBCmEEMI5Lhuv8q89n7Dj6m6q+1Tljeb/4LFK4VIslEEOFwzR0dEMHz6c3377jatXrwIwdepUrl27RsuWLalSpQqffvopP/30U5EFK4QQ4tFQFIUtl3bwr71zuJZxg/ZVI5j51OtU0Jd3dmjCSRwuGL766iv0ej3Lly+natWqnD59miNHjhAREcE333zD6tWrqVmzJosXLy7KeIUQQhQxY3Y6Xx7+jmUnVuOq1vFi2HD6BvdEq9Y6OzThRA6PYThy5AjdunUjNDQUgM2bN6NSqeyzP+p0Otq0acOyZcuKJlIhhBBF7mTSGb45uoRkUwrBPrUYFjIQH1dvZ4cligGHCwaTyYSnp6f9+datWwFo3bq1fZnNZkOjkalAhRCipLHarKw/t4lfz21CpVLRo2ZnOlVrL3MrCDuHP92DgoI4ePAgANeuXWPfvn3Url2bihUrAmA2m9myZQuBgYFFE6kQQogikZSVzILYJZxOOYufmy8jQiKp6V3d2WGJYsbhgqFTp07MnTuXIUOGcPXqVaxWK3369AHgzz//5JNPPuHChQu8/fbbRRWrEEKIQnYg/giLji0nw5JJk4CGRNXrg4fWw9lhiWLI4YJhzJgxxMfHs3z5chRFoVu3bgwZMgSA/fv3ExcXx/Dhw+nfv3+RBSuEEKJwmK3ZrDr1M39djkbroiWybm9aV24pl0uKPKkURVEKsoHRaERRlFzjGS5duoSbmxv+/v6FHqAzxcenFer+AgI8C32fziK5FD+lJQ+QXIra1fTrfH1kEVfSr1FZX5ERIVFUNlTMd7vimMuDKC15QOHnEhDgmee6Ao9QNBgMdy2rWrVqQXcjhBDiEVMUhR1XdrP85Bqybdm0qfIYvWt3RyeXSwoHOP2ShoMHD/Lhhx+ycOFCjh07xowZM1Cr1eh0Ot5//338/f1555132Ldvn33q6Xnz5qHVapk4cSIJCQno9Xref/99/Pz8OHDgADNnzkStVhMREcG4ceMAmDt3Ln/++ScajYYpU6YQFhbmzLSFEOKRysjOZHHcCvbHH8ZD487wkEgaB4Q6OyxRgji1YJg/fz5r1qzB3d0dgJkzZ/Lmm29Sv359li5dyvz585k8eTKxsbF89dVX+Pn52bddsGABwcHBjB8/nnXr1jFv3jymTp3KtGnTmDNnDoGBgYwePZrY2FgAdu/ezfLly7l69Srjx49n5cqVTslZCCEetTMp51gQu4TErCRqeVdneEgkfm6+zg5LlDBOvcA2KCiIOXPm2J//+9//pn79+kDOTa1cXV2x2WycP3+et956i4EDB7JixQoAYmJiaNOmDQBt27YlOjoao9GI2WwmKCgIlUpFREQE0dHRxMTEEBERgUqlonLlylitVhITEx99wkII8QjZFBvrz23iP/s+Jykrma7Vn+IfTV6QYkE8EKf2MHTu3JlLly7Zn5cvnzNH+b59+/j+++9ZtGgRGRkZDB48mBEjRmC1Whk6dCihoaEYjUb7wEu9Xk9aWhpGozHXGAu9Xs/FixdxdXXFx8cn1/K0tLRcPRb34uvrgUajLsSM7z+gpKSRXIqf0pIHSC4PKzEzmXk7FxB74wTl3H0Z32o4DcoHP/R+S8txKS15wKPLxeljGP7ul19+4bPPPuPLL7/Ez8/PXiTcPm3RqlUr4uLiMBgMpKenA5Ceno6Xl1euZXcu12q1dy2/8yqPvCQlZRRqbjIyt3gqLbmUljxAcnlYh28e5ftjyzFmpxPmH8Kg+n0xqPQPHUdpOS6lJQ94tFdJFKs5P3/66Se+//57Fi5caJ8x8ty5c0RFRWG1WsnOzmbfvn2EhITQtGlTtmzZAuRMU92sWTMMBgNarZYLFy6gKArbtm0jPDycpk2bsm3bNmw2G1euXMFms+XbuyCEECVNts3CihNr+PzQN2RZTfQLfobRDYdi0OqdHZooBfLsYbg9lqCgVCoVR48eLfB2VquVmTNnUqlSJcaPHw9A8+bNefnll+nRowf9+/dHq9XyzDPPUKdOHapWrcqkSZOIjIxEq9Uye/ZsAKZPn85rr72G1WolIiKCRo0aARAeHs6AAQOw2Wy89dZbD5SbEEIUV9fTb/B17GIuGa9QwaM8I0OiqOpZ2dlhiVIkz4mbnnnmmbtm/Lp69SopKSl4enoSEhKCt7c3GRkZHDt2jJs3b1K5cmXq16/Pp59++kiCL2oycVPeJJfip7TkAZJLQSiKws5rMSw7sRqz1czjlZrTN/gZXNW6Qn+t0nJcSkseUEwmbvrpp59yPd+7dy8jR47kueee4+WXX0an+98/RpvNxldffcUnn3wi396FEOIRybRksfT4KvZeP4Cb2o2RIYNoVqGRs8MSpZTDgx4/+OADwsLCeO211+5a5+LiwujRo9m1axcfffQRTzzxRGHGKIQQ4m/Op17k6yOLuJmVSA2vIIaHROHvLmOzRNFxuGCIi4sjKirqvm1q167N3r17HzooIYQQ92ZTbGy6sJU1Z9ajKAqdqrWne41OqF0K9xJwIf7O4YKhXLly7Nu3L8/1FouF6OhoKlWqVCiBCSGEyC3VnMZ3R3/gWOIJvHSeDGswkHp+dZwdligjHL6ssnv37hw8eJA333zzrlkSr127xquvvsrJkyfp169foQcphBBl3bGEE8za9R+OJZ4gpFw9prSYIMWCeKQc7mEYO3Ys+/fvZ/ny5axcuZJKlSqh1+sxGo1cvXoVRVHo0qULI0eOLMp4hRCiTLHYLKw98xsbL2xBrVLTp3Z32ge2uesqNiGKmsMFg6urK9999x2rVq1i3bp1HD9+nOvXr+Pt7U2bNm149tln6dq1a1HGKoQQZUp8RgILYhdzPu0iAe7lGBkyiCCvqs4OS5RRBZoaWqVS0adPH/r06VNU8QghhAD2XNvP0uOryLKaaFmxGf2Dn8FN4+bssEQZVuB7SVgsFrZv305cXBwpKSm8/vrrHD9+HA8PD/t0zkIIIR5MlsXEshOr2XUtBle1jmENBtKiYlNnhyVEwQqGXbt2MWnSJK5fv46iKKhUKl5//XV+/fVX5s+fzz//+U9GjRpVVLEKIUSpdjHtMl/HLuJGxk2CPKswImQQ5T38nR2WEEABCoZjx44xevRo3NzceOGFFzhz5gwbNmwAoFGjRvj7+/Phhx9So0YNnnzyySILWAghShtFUfjz0nZWn1qHRbHSIbAtPWt1QeNS7G4oLMowhy+r/OSTT3B1dWXVqlW88sorBAf/777q7du3Z/ny5Xh7e7NgwYIiCVQIIUqjNLORzw8tYMXJNbhp3Hip0Sh61+kuxYIodhz+FxkTE0OXLl2oUqXKPdeXL1+erl278uuvvxZacEIIUZodTzzFt0eXkGJOo55vHYY2GIi3a943/xHCmRwuGEwmEx4eHvdto1arMZlMDx2UEEKUZlablXVnN/D7+c2oVCqeqdWVp4La4aJyuNNXiEfO4YKhVq1abN++HZvNhovL3f+os7Oz2bZtGzVq1CjUAIUQojRJyExkQewSzqaep5ybHyNCoqjhHeTssITIl8PlbL9+/Th58iRvvPEGSUlJudYlJCTw2muvcf78eXr37l3oQQohRGmw78Yh3t3zEWdTz9OsfCMmt/iHFAuixHC4hyEyMpL9+/ezZs0a1q5di6urKwBPPvkk165dw2az8dRTTzFo0KAiC1YIIUois9XMipNr2H5lNzoXLYPq9eOxSuEyvbMoUQo0DPdf//oX7du3Z8WKFRw9ehSLxYLRaKRZs2Y8++yz0rsghBB/cyH5Mh/unc+19OtUMVRiZMggKurLOzssIQqswNftdO3a9b73jMjMzMTd3f2hghJCiJJOURT+uhzNqlM/k22z0K5qa56t1Q2tWuvs0IR4IA6PYejQoQPffffdfdvMnTtXJm0SQpR56dkZzD/8HT+cWI2bxpUXw4bTP/gZKRZEiZZnD8OlS5cwGo3255cvX+bMmTPExcXds312djbR0dFkZmYWfpRCCFFCnEo+y4LYxSSbUqjjU5NX2zyPNV3t7LCEeGh5FgwHDx7k1VdftQ/KUalU/PDDD/zwww957kxRFFq3bl34UQohRDFnU2z8em4Tv57diEqlonuNTnSu/iR+Ht7Ep6c5OzwhHlqeBcPTTz/N0aNHSUxMRFEUVq9eTb169ahfv/4922u1WsqXLy9XSQghypykrGS+ObqEU8ln8XX1YURIFLV8qjs7LCEK1X0HPU6cONH+ePfu3fTu3ZuhQ4cWeVBCCFFSHIyP5ftjy8iwZNI4oCGD6vXBQ3v/WXGFKIkcvkrijz/+ACAxMRGz2UzFihXt69asWUOrVq0oX14uFRJClA3Z1mxWnVrH1ss70LpoGFi3NxGVW8rcCqLUKtDE5XPnzqVt27b89NNP9mXZ2dm88cYbPPnkkyxcuLDQAxRCiOLmWvp1PoiZy9bLO6ikr8Dr4S/TpkorKRZEqeZwD8PKlSuZO3cudevWJSwszL5cpVLx3nvv8c033zBr1iy8vb3p2bNnkQQrhBDOpCgKO67uZvmJNWTbsomo0oo+tXugk8slRRngcMGwaNEiatWqxfLly9HpdP/bgUZDz5496dKlCz169OCbb76RgkEIUepkZGey5PhK9t04hLvGnWENBtKkfENnhyXEI+NwwXDu3DkGDhyYq1i4k06no3379ixevLjQghNCiOLgTMp5voldTEJWEjW9qzMiJBI/N19nhyXEI+XwGAYPDw+uXr163zYJCQkFnhb64MGDDBkyBIDz588TGRlJVFQU06ZNw2azAbBs2TJ69+5N//792bx5MwBZWVmMHz+eqKgonn/+eRITEwE4cOAA/fr1Y+DAgcydO9f+OnPnzqVv374MHDiQQ4cOFShGIUTZZFNs/HbuD/6z7zMSs5LpWr0DrzR5QYoFUSY5XDA0b96cjRs35vlhGxcXx4YNG2jWrJnDLz5//nymTp2KyWQC4N133+WVV15h8eLFKIrCpk2biI+PZ+HChSxdupT//ve//Pvf/8ZsNrNkyRKCg4NZvHgxvXr1Yt68eQBMmzaN2bNns2TJEg4ePEhsbCyxsbHs3r2b5cuX8+9//5vp06c7HKMQomxKMaUy98BXrDmzHk+tgZebjKZ7zc6oXWTWRlE2OXxK4sUXX2Tz5s0MHTqUZ555hsaNG2MwGEhPT+fQoUOsXr0aRVEYN26cwy8eFBTEnDlzeP311wGIjY2lRYsWALRt25bt27fj4uJCkyZN0Ol06HQ6goKCiIuLIyYmhueee87edt68eRiNRsxmM0FBOfeXj4iIIDo6Gp1OR0REBCqVisqVK2O1WklMTMTPz8/hWIUQZceRm8dYeGwZxux0GvrXZ3C9/hh0emeHJYRTOVww1K1bl88++4ypU6fyww8/sGzZMvs6RVGoVKkSs2bNokGDBg6/eOfOnbl06VKu/dy+LEmv15OWlobRaMTT09PeRq/XYzQacy2/s63BYMjV9uLFi7i6uuLj45NreVpaWr4Fg6+vBxpN4X6bCAjwzL9RCSG5FD+lJQ9wTi7Z1mwWH/qJdSc2oXHRMKJJf7rUeeKhL5eU41L8lJY84NHlUqDbWz/22GNs2LCB/fv3c/z4cVJTU/Hw8CA4OJjmzZujVj/ch6uLy//OkKSnp+Pl5WXvxbhzuaenZ67l92vr5eWFVqu95z7yk5SU8VD5/F1AgCfx8aVjTnnJpfgpLXmAc3K5nhHPgtjFXEy7TAWPAEaEDCLQszI3bxrz3/g+5LgUP6UlDyj8XO5XfBSoYICcD/VmzZoVaKyCoxo0aMCuXbto2bIlW7dupVWrVoSFhfHRRx9hMpkwm82cPn2a4OBgmjZtypYtWwgLC2Pr1q00a9YMg8GAVqvlwoULBAYGsm3bNsaNG4dareaDDz5g1KhRXLt2DZvNJqcjhBBATs/m7mv7WHriR8xWM49Vak6/4GdwVd/7ijAhyqoCFwxbt25l1apVHDt2jNTUVKKjo1mzZg0XLlxg1KhRBb5K4k6TJk3izTff5N///jc1a9akc+fOqNVqhgwZQlRUFIqiMGHCBFxdXYmMjGTSpElERkai1WqZPXs2ANOnT+e1117DarUSERFBo0aNAAgPD2fAgAHYbDbeeuutB45RCFF6ZFmyWHr8R/Zc34+b2o0RIVGEV2js7LCEKJZUiqIojjZ+6623WL58OYqioFarsdlsHDt2jFmzZvHdd9/RqFEjvv76a/T60jE4qLC7rKQbrHgqLbmUljzg0eRyPvUiX8cu5mZmAtW8AhkZEoW/e7lCfx05LsVPackDHu0pCYcvq1y6dCnLli2jU6dO/P7777z44ov2dWPHjqVPnz4cPHiQBQsWPFy0QghRhGyKjY0XtjA7Zh43MxPoGPQErzZ9qUiKBSFKE4dPSSxdupS6devy8ccfA+QaNezt7c3MmTM5ceIEv/76a4EurRRCiEclzWzku6M/cDTxOF46T4Y2GEB9v2BnhyVEieBwD8PZs2dp06bNfdu0aNGCy5cvP3RQQghR2I4lnmDm7n9zNPE4DfzqMqXFBCkWhCgAh3sY3NzcSEhIuG+bGzdu4Obm9tBBCSFEYbHarKw98xsbLvyJWqXm2dpP82RgG1xUDn9fEkJQgB6GZs2asWHDhjzvJ3Hu3Dk2btxI06ZNCy04IYR4GDczE5i9bx4bLvxJgHs5Xm32Ek8FtZNiQYgH4HAPw9ixY9m2bRv9+vVj1KhRnD17FoDdu3dz+PBh5s+fT3Z2Ni+88EKRBSuEEI7ae20/S46vIstqonmFpgys2ws3jfSACvGgHC4YQkJCmDNnDm+88Qbvv/++ffmwYcNQFAWDwcCHH35on/dACCGcwWQ1s+zEanZe3YurWsfQ+gNoWanwJ5oToqwp0MRN7dq1Y/PmzWzatInY2FjS0tLw8PCgbt26dOzY0aHploUQoqhcTLvCgthFXM+IJ9CzCiNDoijvEeDssIQoFQo806ObmxtPP/00Tz/9dFHEI4QQBaYoCn9e2s7qU+uwKFaeDGxDz1pd0boU+E+cECIPeb6b4uLiHnin9erVe+BthRCiIIzmdL6PW8bhm8cwaPUMqd+fUP/6zg5LiFInz4KhV69eD3xL12PHjj1wQEII4agTSaf5JnYJKeZU6vrWZliDgXi7ejk7LCFKpSIpGIQQoihZbVZ+ObeR3879gUql4pmaXXmqmlwuKURRyrNgeO+99x5lHEII4ZCEzCS+ObqEMynnKOfmy4iQKGp4V3N2WEKUejIiSAhRYuy/cZhFcSvItGTStHwYUfX64K5xd3ZYQpQJeRYMD3oDKZVKxZw5cx44ICGE+Duz1czKk2vZdmUXOhctg+r15bFKzeW0qRCPUJ4Fw8aNGx9oh/IGFkIUpivGa3wdu4ir6depYqjEyJAoKuorODssIcqcPAuGTZs2Pco4hBAiF0VR2HZlJytPriXbZqFd1cd5ttbTaNVaZ4cmRJmUZ8FQpUqVRxmHEELYGU3pfHVkIQfij6DXeDAiZBCNAkKcHZYQZVqBBz3u3buXlStXcvz4cTIzM/Hx8aFOnTr07NmT8PDwoohRCFGGnEo+y3c7l5KQkURtnxoMbxCJr5uPs8MSoswrUMEwe/ZsvvrqKxRFAcDd3Z1z586xf/9+li9fzujRo5kwYUKRBCqEKN1sio3fzv3BurMbQAVP1+hIl+odZG4FIYoJhwuGX375hfnz51OnTh1ee+01mjVrhsFgwGw2s3fvXv71r3/x5Zdf0rBhQ5566qmijFkIUcokZSXz7dGlnEw+g6+rDxNaj6IcMrBRiOLE4dL9u+++IyAggO+++4527dphMBgA0Ol0PP7443z99df4+/uzcOHCIgtWCFH6HIqP5d3dH3Ey+QyNAkKZ3OIV6gXUdnZYQoi/cbiH4fjx4/To0QNfX997rvfz86N9+/asX7++0IITQpRe2dZsfjz9C1subUfromFA8LO0qdJKLs0Wopgq9Jkes7OzC3uXQohS5lr6Db6OXcRl41Uq6iswMiSKKoZKzg5LCHEfDhcMdevWZfPmzSQnJ+Pj43PX+sTERP744w/q1q1bmPEJIUoRRVGIvrqX5SdWY7Zl07pyS/rW6YFOrXN2aEKIfDg8hmHo0KHEx8czatQodu/ejcViAcBoNLJlyxaGDx9OQkICgwcPLrJghRAlV6YlkwWxi1kUtxy1i5pRoYOJqtdHigUhSgiHexi6devG4cOHWbBgAcOGDcPFxQWdTkdWVhaQ881hxIgRdO/evciCFUKUTGdTLrAgdjEJWYnU9K7G8AZRlHO/93goIUTxVKAxDJMmTaJDhw6sWrWKuLg40tPT0ev11KtXj969e8vETUKIXGyKjY0XtrD2zG8oikKXak/SrUZH1C5qZ4cmhCigAg96DA8Pp3Hjxmg0/9v08uXLMpW0ECKXFFMq3x39gbikk3jrvBgeMpBgX7lcUoiSqkAFw44dO3j//ffp3bs3w4YNA3JORXTt2pVq1aoxa9YsGjZs+FABrVq1ih9//BEAk8nEsWPHWLp0KS+++CLVq1cHIDIykm7durFs2TKWLl2KRqNhzJgxtG/fnqysLCZOnEhCQgJ6vZ73338fPz8/Dhw4wMyZM1Gr1URERDzw7buFEPmLTYjju6M/YMxOJ7RcfYbU749Bp3d2WEKIh+BwwbB3715Gjx6NVqvF3d3dvtxsNtOzZ0/Wr1/PoEGD+P777wkLC3vggHr37k3v3r0BmD59On369OHo0aOMGDGCkSNH2tvFx8ezcOFCVq5ciclkIioqitatW7NkyRKCg4MZP34869atY968eUydOpVp06YxZ84cAgMDGT16NLGxsYSEyM1shChMFpuFNafXs+niVjQqNX3r9OSJqq1lbgUhSgGHr5L49NNP0ev1/PTTT/Tv39++3NXVlXfeeYdVq1bh6urKJ598UiiBHT58mFOnTjFgwACOHDnCn3/+yaBBg5gyZQpGo5FDhw7RpEkTdDodnp6eBAUFERcXR0xMDG3atAGgbdu2REdHYzQaMZvNBAUFoVKpiIiIIDo6ulDiFELkuJERz+yYT9l0cSvlPfx5LXwc7QMjpFgQopRwuIfh2LFj9OjRg6CgoHuuDwoKolu3bqxdu7ZQAvviiy8YO3YsAGFhYfTr14/Q0FA+++wzPv30U+rVq4enp6e9vV6vx2g0YjQa7cv1ej1paWkYjUb7VNa3l1+8eDHfGHx9PdBoCndwVkCAZ/6NSgjJpfhxVh5bz+3iq5glZFlMPFH9MUY27Y+b1u2h9llajglILsVRackDHl0uDhcMVqsVk8l03zYqlcp+J8uHkZqaypkzZ2jVqhUAHTt2xMvLy/54xowZhIeHk56ebt8mPT0dT09PDAaDfXl6ejpeXl65lt25PD9JSRkPncudAgI8iY9PK9R9OovkUvw4I48sSxY/nFjN7mv7cFO7MrxBJM0rNiEtOZs0HnzW19JyTEByKY5KSx5Q+Lncr/hw+JREvXr12Lx5M4mJifdcn5yczObNmwtlpsc9e/bw+OOP25+PGjWKQ4cOARAdHU1ISAhhYWHExMRgMplIS0vj9OnTBAcH07RpU7Zs2QLA1q1b7XfV1Gq1XLhwAUVR2LZtm1wCKsRDupB6iff2fMzua/uo5hnIG81foXnFJs4OSwhRRBzuYRg2bBjjxo1j6NChvPTSSzRq1Mj+zf3QoUN8/vnn3Lhxg8mTJz90UGfPnqVq1ar252+//TYzZsxAq9Xi7+/PjBkzMBgMDBkyhKioKBRFYcKECbi6uhIZGcmkSZOIjIxEq9Uye/ZsIGcA5WuvvYbVaiUiIoJGjRo9dJxClEU2xcbmi9v46fSvWBUrHYOeoHvNTmhcCv3WNEKIYkSlFOAcwhdffMGcOXOwWq1370ilYty4cbz00kuFGqAzFXaXlXSDFU+lJZdHkUea2ch3x37gaMJxPHUGhtUfSP1ywYX+OqXlmIDkUhyVljzg0Z6SKNBXghdeeIHOnTvz66+/cvz4cVJTU/Hw8CA4OJju3btTs2bNhw5WCFE8xSWe5NujS0k1p1HfL5ihDQbgpSs9A8eEEPdX4D7E6tWrM2bMmKKIRQhRDFltVn4++zsbzv+JSqXi2dpP82RgG1xUDg+BEkKUAnLSUQiRp5uZiSyIXcy51Av4u5djZEgU1bwCnR2WEMIJpGAQQtxTzPUDLI5bRZY1i/AKjRlYtzfumoebW0EIUXJJwSCEyMVkNbPixE/suLoHnVrHkPr9aVmxmczYKEQZJwWDEMLuUtoVvo5dzPWMGwQaKjMidBAVPAKcHZYQohiQgkEIgaIobLm8gx9PrcNis9A+MIJnanVDK3MrCCFukb8GQpRxxux0vj+2nMM3j2LQ6hkSOoRQ//rODksIUczkWTDMnTv3gXaoUqnsN40SQhRvJ5NO883RpSSbUgj2rc2wBgPwcfV2dlhCiGKoQAXD7UFP95oc8vaNp6RgEKL4s9qs/HpuE+vPbUKlUtGjZhc6VXtC5lYQQuTJ4YLBZDLx7rvvAjn3lWjSpAk+Pj5kZGRw+PBhFixYgFqtZubMmUUbsRDioSRmJfFN7BJOp5zDz82XESFR1PSu5uywhBDFXJ4Fw1NPPZXr+fvvv4/FYmH58uUEBuaeuCUsLIyOHTvSu3dv1qxZI3eCFKKYOnDjMN/HrSDTkkmT8mFE1e2Dh9bd2WEJIUoAh/sf16xZQ6dOne4qFm4rX748HTt25Lfffiu04IQQhcNszWbp8R+Zf2QhFpuFqLp9GBUySIoFIYTDHL5KwmQyYbFY7tvGaDTec3yDEMJ5rhivsSB2MVfSr1FZX5GRoYOopK/g7LCEECWMwz0MISEh/P7775w+ffqe6/fv389vv/0mpyOEKCYURWHb5Z38a+8nXEm/RtsqjzExfLwUC0KIB+JwD8P48eMZPnw4/fv359lnnyU0NBS9Xk9aWhr79u1j7dq1aLVaXnnllSIMVwjhiIzsDBbHrWR//GE8NO6MCImiUUCos8MSQpRgDhcM4eHhzJs3j+nTp/P999/nmldeURRq167NrFmzCA4OLpJAhRCOOZNyjq+PLCbJlEwt7xqMCInE183H2WEJIUo4hwuGzMxM2rZty4YNGzh48CBxcXGkpqbi5eVFSEgIYWFhRRmnECIfNpuN9ec2se7sBhRFoVv1p+ha4ymZW0EIUSgcLhh69+5NixYtmD59Ok2aNKFJkyZFGZcQogCSTSnM2/IVsTdO4OPqzfAGkdTxrenssIQQpYjDBcOlS5do3759UcYihHgAh28eZeGxZaRnZxDmH8Kg+n0xaPXODksIUco4XDDUq1ePI0eOFGUsQogCyLZZWH1qHX9e2o7GRcPIpgNo6t001/giIYQoLA4XDP/85z+ZOHEiAwYMoEOHDlStWhVXV9d7tu3QoUOhBSiEuNv19Bt8HbuYS8YrVPQoz8jQQTSuEUx8fJqzQxNClFIOFwwjRowA4ObNmxw6dOiebW7ffOrYsWOFE50QIhdFUdh5dS/LTqzGbMvm8Uot6BfcE51a5+zQhBClnMMFw9ixY6WrUwgnyrRksfT4KvZeP4C7xo2R9fvTrEIjZ4clhCgjCjRxkxDCOc6lXmDBkcXczEqkhlcQI0KiKOfu5+ywhBBliMMFg6N27txJq1atCnu3QpRJNsXGpgtbWXNmPYqi0LnakzxdoyNqF7WzQxNClDEFKhgWLVrEzz//TGJiIlar1X6jKUVRsFgspKWlkZWVJWMYhCgEKaY0vju6lLikk3jrPBnWIJK6frWdHZYQooxyuGBYunQpM2bMAMDNzQ2TyYROlzPQymQyAeDt7U3//v2LIEwhypajCcf57ugPpGUbCSlXjyH1++OpMzg7LCFEGebwnLHLli3D3d2d5cuXc+DAARo3bkzPnj05ePAgGzdupF27dqSnp9OjR4+ijFeIUs1is7Dq1M98evC/ZFgy6VOnB2PCRkixIIRwOod7GM6ePUvnzp1p2LAhAI0bN2bjxo0AVK1alU8++YTOnTvz5Zdf8sknnzxUUL169cLT09O+7xdffJE33ngDlUpFnTp1mDZtGi4uLixbtoylS5ei0WgYM2YM7du3Jysri4kTJ5KQkIBer+f999/Hz8+PAwcOMHPmTNRqNREREYwbN+6hYhSisN3IuMmC2MVcSLtEeXd/RoRGEeRZ1dlhCSEEUICCwWq1UqFCBfvzGjVqcPnyZTIyMvDw8MDV1ZX27duzbdu2hwro9umNhQsX2pe9+OKLvPLKK7Rs2ZK33nqLTZs20bhxYxYuXMjKlSsxmUxERUXRunVrlixZQnBwMOPHj2fdunXMmzePqVOnMm3aNObMmUNgYCCjR48mNjaWkJCQh4pViMKy+9o+lh5fhclqpmXFZvQP7oWb5t4TowkhhDM4fEqiQoUKXL161f48KCgIRVE4ceKEfZmHhwfx8fEPFVBcXByZmZmMHDmSoUOHcuDAAWJjY2nRogUAbdu2ZceOHRw6dIgmTZqg0+nw9PQkKCiIuLg4YmJiaNOmjb1tdHQ0RqMRs9lMUFAQKpWKiIgIoqOjHypOIQpDlsXEd0d/4NujSwEY1mAgQxsMkGJBCFHsONzD8Pjjj7NmzRr7ZZP169dHrVazZs0aGjduTHZ2Ntu3b6dcuXIPFZCbmxujRo2iX79+nDt3jueff94+gySAXq8nLS0No9FoP21xe7nRaMy1/M62BoMhV9uLFy/mG4uvrwcaTeFevhYQ4Jl/oxJCcnk4ZxIv8PHu/3LVeINavtX4x2MjqehZ/qH2KcekeJJcip/Skgc8ulwcLhheeOEFfvvtN0aMGMHMmTPp3bs33bt3Z8mSJRw+fJjU1FQuXLjAsGHDHiqgGjVqUK1aNVQqFTVq1MDHx4fY2Fj7+vT0dLy8vDAYDKSnp+da7unpmWv5/dp6eXnlG0tSUsZD5fJ3AQGepWauf8nlwSmKwuZL21h96hesipUOQW3pWbML6iwN8VkPHocck+JJcil+SkseUPi53K/4cPiUROXKlVmxYgX9+/enevXqAEyZMoU2bdpw+PBhLl68SKdOnR56RsgVK1bw3nvvAXD9+nWMRiOtW7dm165dAGzdupXw8HDCwsKIiYnBZDKRlpbG6dOnCQ4OpmnTpmzZssXetlmzZhgMBrRaLRcuXEBRFLZt20Z4ePhDxSnEg0gzG/ns0AJWnlyLh8adsY1G0bt2dzQuhT6HmhBCFCqVcnv2pYeQlpaGVqvFzc3toQMym81MnjyZK1euoFKpeO211/D19eXNN98kOzubmjVr8s4776BWq1m2bBk//PADiqLwwgsv0LlzZzIzM5k0aRLx8fFotVpmz55NQEAABw4cYNasWVitViIiIpgwYUK+sRR2BSpVbfH0qHI5nniKb48uIcWcRj3fOgxtMBBv18LrSpRjUjxJLsVPackDHm0Pg8MFw5AhQ2jfvj3t2rWjVq1ahRZccSYFQ94kF8dZbVbWnd3A7+c3o1Kp6FmzCx2C2uKicriDzyFyTIonyaX4KS15wKMtGBzuBz1x4gR79uzhgw8+oHLlyvbioWXLlvYZH4UQuSVkJrIgdglnU8/j7+bHiNAoqnsFOTssIYQoMIcLhp07d3Lo0CG2bt3K9u3bWbJkCYsWLcLNzY1WrVrZC4g752oQoiyLuX6QJcdXkmnJIrxCYwbWfRZ3jbuzwxJCiAficMGgUqlo1KgRjRo1Yvz48aSmprJt2za2bdvG9u3b+fPPPwGoV68eP/74Y1HFK0SxZ7KaWXFiDTuu7kbnomVwvX60qhRuvzRYCCFKogcemu3l5UWLFi1QqVS4urry888/k5aWRlxcXGHGJ0SJctl4la+PLOJaxg2qGiozMiSKCvqHm1tBCCGKgwIVDJcvX2bPnj3s3buXPXv2cOHCBQDUajUNGjSgRYsWtGzZskgCFaI4UxSFrZejWXXqZyw2C+2rRvBM7W5o5XJJIUQp4fBfsyeffNI+NbRGo6F+/fqMHDmSFi1aEB4ejl6vL7IghSjOjNnpLDq2gkM3Y9FrPXgudDAN/Rs4OywhhChUDhcMV65cAXKmVe7VqxcRERE0b94815TLQpQ1J5PO8M3RJSSbUgj2qcWwkIH4uHo7OywhhCh0DhcMt+8jsXPnTtauXcvixYtxcXGhbt26tGjRghYtWtC8efNc93cQorSy2qz8em4T689tQqVS0aNmZzpVa1/ocysIIURx4XDBEBwcTHBwMEOHDkVRFA4fPszOnTvZs2cPK1eu5Ntvv7UXEKtWrSrKmIVwqqSsZBbELuZ0yjn83HwZERJJTe/qzg5LCCGK1AONyFKpVISFhREWFkabNm3Yvn07ixcv5sqVKxw7dqywYxSi2DgQf4RFx5aTYcmkSUBDour1wUPr4eywhBCiyBW4YDh//jzR0dFER0eza9cuUlJSUBSFoKAgBg8eTNu2bYsiTiGcymzNZtWpn/nrcjRaFy2RdXvTunJLmVtBCFFmOFwwTJ48mZ07d3Lt2jUURUGn09G8eXPatGlDu3btqFGjRlHGKYTTXDFeY0HsYq6kX6OyviIjQqKobKjo7LCEEOKRcrhg+PHHH6lcuTL9+/enXbt2PPbYY7i7yzS3ovRSFIXtV3ax4uRasm3ZtKnyGL1rd0en1jo7NCGEeOQcLhjWrl1LnTp1ijIWIYqNjOxMFsetYH/8YTw07gwPiaRxQKizwxJCCKdxuGC4XSycOnWKH3/8kbi4OFJSUlixYgWbN28mJSWFnj174uIil5WJku1MyjkWxC4hMSuJWt7VGR4SiZ+br7PDEkIIpyrQoMcvvviCTz75BKvVCmAf8LV7926++eYbfv/9dz7++GO0WumyFSWPTbHx+/nNrDu7AUVR6Fr9KbpW74DaRe3s0IQQwukc7g747bff+M9//kNYWBgLFixgxIgR9nUDBw7k8ccfZ/PmzSxevLhIAhWiKCWbUpizfz5rz/yGl86TfzQZTfeanaRYEEKIWxwuGBYsWEBQUBDffvstjz32WK57R1SrVo0vv/ySmjVryq2tRYkTc+Uws3b/hxPJpwnzD2Fyi1eo41vL2WEJIUSx4vApiePHjzNw4EB0Ot0916vVatq2bcsPP/xQaMEJUZSybRZ+OvULmy9tQ+OioX9wL9pWeUzmVhBCiHtwuGBQq9Wkp6fft01KSgpqtXThiuLvevoNFsQu5qLxClU8KzKsfiRVDJWcHZYQQhRbDp+SaNiwIX/88Qepqan3XH/z5k02bdpEaKhceiaKL0VRiL66l/f2fsJF4xUer9Scdzu9IcWCEELkw+GCYfTo0SQkJDBo0CB+//13bt68CcDly5dZv349gwYNIjU1NddgSCGKk0xLFt8cXcL3x5bhggsjQwYxqH4/3DSuzg5NCCGKPYdPSTz22GP83//9HzNmzOAf//gHkPNt7amnngLAxcWFSZMmyb0kRLF0LvUCC44s5mZWIjW8ghgeEoW/u5+zwxJCiBKjQPMw9OvXj7Zt2/LTTz8RGxtLWloaHh4e1K1bl549e1KtWrWiilOIB2JTbGy6sJU1Z9ajKAqdqrWnew25XFIIIQqqwHerrFChAqNHj85zfVpaGp6eng8VlBCFIcWUxsJjP3As8QReOk+GNRhIPT+Z3lwIIR6EQ2MYTCYTaWlp+bZbt24dXbt2feighHhYRxOO8+7u/3As8QQh5eoxpcUEKRaEEOIh3LeHYd++fXzwwQccOHAAgMqVKzNu3DieffbZXO2uXLnC22+/zV9//VVkgQrhCIvNwpoz69l0YStqlZo+tbvTPrCNzK0ghBAPKc+C4eDBgwwdOhSLxYJGo8HT05PLly8zZcoUUlNTGTZsGADLli3j3XffJTMzE4PBwIQJEx5Z8ELcKT4jga9jF3Eh7RLl3f0ZERpFkGdVZ4clhBClQp4Fw/z587FYLIwdO5YXX3wRrVbLkSNHmDBhAp9++imRkZG8//77LF68OGcwWadOTJ06lfLlyz/K+IUAYPe1ffxw/EeyrCZaVmxG/+BncNO4OTssIYQoNfIsGA4dOkTjxo0ZP368fVloaChTp07lhRde4M033+Snn37C39+ft99+23555cPKzs5mypQpXL58GbPZzJgxY6hYsSIvvvgi1atXByAyMpJu3bqxbNkyli5dikajYcyYMbRv356srCwmTpxIQkICer2e999/Hz8/Pw4cOMDMmTNRq9VEREQwbty4QolXOFeWxcSyE6vZdS0GV7WOYQ0G0qJiU2eHJYQQpU6eBUNycvI9BzA2bZrzx3jNmjU0b96cjz76iHLlyhVaQGvWrMHHx4cPPviApKQknn32WcaOHcuIESMYOXKkvV18fDwLFy5k5cqVmEwmoqKiaN26NUuWLCE4OJjx48ezbt065s2bx9SpU5k2bRpz5swhMDCQ0aNHExsbS0hISKHFLR69C2mXWHBkMTcybxLkWZURIVGU9/B3dlhCCFEq5VkwmM3me14eeXtZxYoVmT9/Pm5uhdvt26VLFzp37mx/rlarOXLkCGfPnmXTpk1Uq1aNKVOmcOjQIZo0aYJOp0On0xEUFERcXBwxMTE899xzALRt25Z58+ZhNBoxm80EBQUBEBERQXR0tBQMJZSiKGy+tI2fTv2CRbHSIbAtPWt1QeNS4KuEhRBCOOiB/8J26dKl0IsFwH7bbKPRyMsvv8wrr7yC2WymX79+hIaG8tlnn/Hpp59Sr169XAWNXq/HaDRiNBrty/V6PWlpaRiNRgwGQ662Fy9ezDcWX18PNJrCneAnIKD0zFHhjFxSs9KYt3sh+64ewcvVwLiWw2lc6eELv9JyXEpLHiC5FFelJZfSkgc8ulweuGC4/cFeFK5evcrYsWOJioqiR48epKam4uXlBUDHjh2ZMWMG4eHhue6emZ6ejqenJwaDwb48PT0dLy+vXMvuXJ6fpKSMQs0rIMCT+Pj857MoCZyRy/HEU3x7dAkp5jTq+dZhaIOBeGsePo7SclxKSx4guRRXpSWX0pIHFH4u9ys+HL751KNy8+ZNRo4cycSJE+nbty8Ao0aN4tChQwD2UwlhYWHExMTYJ5U6ffo0wcHBNG3alC1btgCwdetWmjVrhsFgQKvVcuHCBRRFYdu2bYSHhzstR1EwVpuVNafXM+fAfNKy0+lVqxtjG4/C27X0fEMQQoji7r49DLt372bu3LkFWqdSqRg7duwDB/T555+TmprKvHnzmDdvHgBvvPEGs2bNQqvV4u/vz4wZMzAYDAwZMoSoqCgURWHChAm4uroSGRnJpEmTiIyMRKvVMnv2bACmT5/Oa6+9htVqJSIigkaNGj1wjOLRSchMZEHsEs6mnqecmx8jQ6Oo7hXk7LCEEKLMUSmKotxrRb169R5shyoVx44de6igiovC7rKSbrCC2XfjEIvjVpBpyaJZ+UZE1uuNu8a90F+ntByX0pIHSC7FVWnJpbTkAY/2lESePQzvvvtuoQUgREGYrWZWnFzD9iu70bloGVyvH60qhcv0zkII4UR5Fgx/v1+EEI/CZeNVvj6yiGsZN6hiqMTIkEFU1MvsoUII4Wxy4booFhRFYevlaFad+hmLzcITVVvTq1Y3tGqts0MTQgiBFAyiGDBmp7Po2AoO3YxFr/XgudDBNPRv4OywhBBC3EEKBuFUJ5NO883RpSSbUgj2qcWwkIH4uHo7OywhhBB/IwWDcAqrzcqv5zax/twmVCoVPWp2oVO1J3BRFbupQYQQQiAFg3CCxKwkvoldwumUc/i5+TIiJJKa3tWdHZYQQoj7kIJBPFIHbhxmUdwKMiyZNAloSFS9vnhoC39uBSGEEIVLCgbxSJit2aw8tZZtl3eiddESVbcPj1duIXMrCCHEfSiKQqbJQmKqifjkTOKTM0lMM5FsNJGYZqJh7QB6tHo0s99KwSCK3BXjNRbELuZK+jUq6ysyMnQQlfQVnB2WEEI4XbbFSmKqietJmSSkZJJkNJFsNJOUZuJmShZJaVmYs2333FalgsoBhnuuKwpSMIgioygK267sYuXJNWTbLLSt8hjP1u6OTuZWEEKUAYqikGW2kpphJiEli4TULBJTTdxMyeRmchbXkjJIMZrz3N7grqWinwc+Blf8PF0J8HEnwMcdPy83fAw6vA06KlbwfmTTXEvBIIpERnYGi+JWciD+MB4ad0aERNEoINTZYQkhRKGw2my3PvyzSLnVK5CSbiLlVu9AQmoWyUYzFmsevQOAn5cr9av54uvpSgVfd/y93fH1dMXX0xUfgyuuOvWjTSofUjCIQnc6+RwLYheTZEqmlncNRoRE4uvm4+ywhBDCIVabjRSjmRtJmSSnm0jLyM4ZM3CrdyA5LWf8wL1v3ZhTDHgbdFQN0OOl1+HpoaWclxt+Xm74ebni7+1OOS83tJqSdRm5FAyi0NgUG7+d28wv5zagKArdanSkS7UnUbsUrypZCFF2WW02ElIyOX05haQ0E4mpWSSmmUhIySIxLYuktJzegry4qFR4G3TUquyNv48b/t5u+BhyegS8DTp89Dk/NeqSVQw4QgoGUSiSspL59uhSTiafwcfVmxEhUdT2qeHssIQQZYiiKKRlZpOWbiYh9daVBLcKgqQ0E/FJmdxMycKWR9eARq3Cx+BKcKAP3nod5X1zThF4eejw0uvw83LFz9MNF5eyeXWXFAzioR2MP8KiYytIt2TQKCCUQfX6otd6ODssIUQpY7HaSL519cCN5EwSUrLs4wUSb40ZMGVb89zey0NLjUqeVAwwYHBV4+vphp+nK35ebpTzcsVLr5NLve9DCgbxwMxWMytP/XxrbgUNA+v2JqJyS3nDCSEeSLbFSmJazsDBZGPOaYLrSZn/m38g1ZRn74Cnh5byvu74e7vhrdfhe6sQ8PHMucLAz9PNPogwIMDzkV1ZUJpIwSAeyIXky3y4dz7X0q9TWV+RESFRVDZUdHZYQohiSlEUMkwWbiTlfPAnpd26xPBW78DN5ExSM7Lz3N5br6NmZS/8vd0o5+1GeZ+c4sDPyw1fT1d0WhkrVdSkYBAFoigKWy7vYPWpdWTbLLSr+ji9aj0tcysIUcbZFAVjRjYJqVnEJ+eMFbiZklMIJKTmnDrIMt/7dIHaRUU5Lzcq++sp5+2Gtz7n0kI/T9ecXgMfd1ylIHA6KRiEw9LMRr4/towjCXF46vSMqjeYhv4NnB2WEKKI2RSFFKOZlCwrpy8k3rqSwGS/yiA+OYtkowmr7d6nC/RuGvy93fHzcqX8rYmH/Lxujx1ww9ugw0VOZRZ7UjAIh8QmHGfhsR9IMxup51uHV9qMxGqUil+I0sCmKKSlm+3jBW7eGkwYf6t3ICElK89iAMDX05VqFT3xNbji65UzI6G/txsB3u6U83bD3VU+akoDOYrivsxWM6tP/8KWSztQq9Q8W/tpngxsg5+7N/FGGTQkRElgzrbaewRu3pqi+HZRkHjrFILFeu+CwMtDS1AFT/y93agUYMBN44Kfl2vOwEIvN3wNOrQa+fJQFkjBIPJ0PvUi3x79gesZN6ior8DwBpEEelZ2dlhCiDsoioIxM9t+miDh1tiBG0mZ3EjOJDE1i/QsS57be7hqCCxvwM/TjQBfd8rful+Bj0F319gBubqgbJOCQdwl22bht3Ob+O38ZmyKjfZVI+hZq6sMbBTCCczZVpJuTTx0+5bGSakmEtOybl1umJXn3AM6rQvlvNyoVtEz5wZGt8YN+HvlXGlw56WGQuRHCgaRy/nUi3x/bDlX0q/h6+rDkPr9qetX29lhCVFqWW02bqZkEZ+UaT9VEJ+cyfWknN6BtPtcauiqU9vnHvDzdMPHU2cfSFje1x1vmYhIFCIpGAQAWZYsfj77O39e3I6CQkTllvSq/TTuGjdnhyZEiZZlsnA1IZ3UdDPxyTnjBxJSs249z+RGUuY9BxRq1C6U83ajaoABPy9XfD1z5hvwvdVT4OvpisFdKwWBeGSkYCjjFEVhf/xhVp5cS7IphQD3ckTV60uwby1nhyZEsWe12UhNzyYl3fS/guBWD0F8Ss4ERcbMvHsI3F01BFXwpKKfBxV8c64oKOflRoBPzj0Myuo9C0TxJAVDGXYh7RKrTv7MyeQzaFRqulbvQOdqT6KVsQqiDFMUhSxzzlUFaRnZGDNz/kvLMJOans3NlExS0s32qw7yusWxVpMzfqBOoA96Nw3eep19lkJ/75zTBW46tfQQiBJDCoYy6Fr6DX49t5GY6wdRUAgtV58+dXpQ3sPf2aEJUSgsVhvmbCumbBvpWdmkZ2aTnmUh02Qhy2wlPTMbY1Y2mVkWMkwW0rMspGdlk5FlwZiZTbbFdt/9q11ybnFcu4q3/bbG3vqc8QMB3u74+7jhpc+ZjEiuLBClRZkqGGw2G2+//TbHjx9Hp9PxzjvvUK1aNWeHVeQURSHZlMKJpNPsvXGAownHAahqqMyztZ+mnl8dJ0coyhJFUbBYFSxWG9lWGxaLjexb/5ktNrItVi4mZHI9Po0ssxWzxXrHeitZZismszXnwz8753nWrUIg69by+00ylBcPVw0ebhqq+Ovx9NDhY8i5pbHBXYvBXYunhw5PDy3lvNzw9JCxA6LsKVMFw8aNGzGbzfzwww8cOHCA9957j88++8zZYT0Uq81KpiWLDEsm6dkZGLONpJmNJJtSSMhKIj4jgesZNzBmp9u3qeFVjaeC2tIoILRM/tFTFAXl9k8F+4cR5DxXFFBuPc5p//fnt7fP+Z/98R37VOz7+t96+7p7LbvXays5M/DduZ2NnOWKomBTcufiec1ISkrGrX3cbqOg2MBqy3lsUxRstjv+U3LOw9sU7lh258+c5VabDYtNwWrNWW612rDalJz/7nx8q43VZrMXBTn/5X5cmFQqcNNpcNOp8dLrqODrjqtOjU6jxlWnxsNNg95Ng4erFg83De6uGjxcNRjctbi75Tz2cNXIeAEh8lGmCoaYmBjatGkDQOPGjTly5Mgje+2/Th3ltw1bsNgs3PpIIOfPpu3Wz9tL7vipst16nPNTUd36UMOGorJiU1lAdf+uUxQVaqsHbtlV0Jn9cc2qhOmSFz8fNbFW2ZvT5NYHDLc+rLA//t8HoX3NreVqtQsWizX39ree/H1fd37wYs/2bx+yd+z7Xh/CuV9HuTvmXPHe/WHPnTGKIqEC1GoVahcX1C4qNBoXNGoVOo0ad1cVWrXLrWUuaNWqnJ+anP9uP9Zp1Gg1Lvh4u2PJtuCmzfnQv91Op1HjemuZh2tOkaDVuJTJwleIR61MFQxGoxGDwWB/rlarsVgsaDT3/jX4+nqgKaQpT2N3niRJfRYc2F3Ot0sVoMr5qbjk/LSpUJRby2w6sLmj2NRg0aJYNSgWLWTrUCw6FLMbitkNzB6oFBdQqbj9J1WlSgNUuKjI+Xp268cdTwGVfZm9jf2hyt4up01Ow//t/2/b31ro4nK7be793bl9rm3/th9VrlhvbXNHjPfe3x3b/G17leoe7V1Ud23nco99uKhyv4b9563DpgJcXFT217n97fX2di63t3FR/S2eW8eFO7ZxUeW0V/0vZvvzW6/r4oL9mNr3ceuni4sLapdb+3FxQa1S3XqsQu1yj8cqFWp1zs/byzXqnA91tYvKXhRo1CrU6ls/XVzkG/p9BAR4OjuEQlNaciktecCjy6VMFQwGg4H09P91zdtstjyLBYCkpIxCe+0XWz1DtuZpbiYYc/5gq1xwUbmg4n9/pFW4oM75yw/k/2H8v+f3/pAsSqVpIFdpyaXI81BsYAWbFWxA3hcLPrzSckxAcimOSkseUPi53K/4KFMFQ9OmTdm8eTPdunXjwIEDBAcHP9LXr+zri9ZSpn7lQgghSoky9enVsWNHtm/fzsCBA1EUhVmzZjk7JCGEEKJEKFMFg4uLC//3f//n7DCEEEKIEscl/yZCCCGEKOukYBBCCCFEvqRgEEIIIUS+pGAQQgghRL6kYBBCCCFEvqRgEEIIIUS+pGAQQgghRL6kYBBCCCFEvqRgEEIIIUS+VIqiyF1/hRBCCHFf0sMghBBCiHxJwSCEEEKIfEnBIIQQQoh8ScEghBBCiHxJwSCEEEKIfEnBIIQQQoh8aZwdQGlhs9l4++23OX78ODqdjnfeeYdq1arZ1//xxx98+umnaDQa+vTpQ//+/fPdxlnyi+vnn3/m22+/Ra1WExwczNtvv42Liwu9evXC09MTgKpVq/Luu+86KwUg/zwWLFjAihUr8PPzA2D69OlUr169xB2T+Ph4/vnPf9rbHjt2jFdffZXIyMhid0xuO3jwIB9++CELFy7MtbwkvU9uyyuXkvI+uVNeuZSk98pt98qlpL1XsrOzmTJlCpcvX8ZsNjNmzBg6dOhgX//I3y+KKBS//fabMmnSJEVRFGX//v3Kiy++aF9nNpuVp556SklOTlZMJpPSu3dv5caNG/fdxpnuF1dmZqbSoUMHJSMjQ1EURZkwYYKyceNGJSsrS3nmmWecEW6e8vv9vvrqq8rhw4cLtI2zOBrXvn37lCFDhigWi6VYHhNFUZQvv/xS6d69u9KvX79cy0va+0RR8s6lJL1PbssrF0UpWe8VRbl/LreVhPfKihUrlHfeeUdRFEVJTExU2rVrZ1/njPeLnJIoJDExMbRp0waAxo0bc+TIEfu606dPExQUhLe3NzqdjmbNmrF37977buNM94tLp9OxdOlS3N3dAbBYLLi6uhIXF0dmZiYjR45k6NChHDhwwBmh55Lf7zc2NpYvv/ySyMhIvvjiC4e2cRZH4lIUhRkzZvD222+jVquL5TEBCAoKYs6cOXctL2nvE8g7l5L0Prktr1ygZL1X4P65QMl5r3Tp0oV//OMf9udqtdr+2BnvFzklUUiMRiMGg8H+XK1WY7FY0Gg0GI1Ge1cXgF6vx2g03ncbZ7pfXC4uLvj7+wOwcOFCMjIyaN26NSdOnGDUqFH069ePc+fO8fzzz7N+/Xqn5pLf7/fpp58mKioKg8HAuHHj2Lx5c4k8Jrf98ccf1KlTh5o1awLg5uZW7I4JQOfOnbl06dJdy0va+wTyzqUkvU9uyysXKFnvFbh/LlBy3it6vR7IeW+8/PLLvPLKK/Z1zni/OP/IlhIGg4H09HT7c5vNZj9Af1+Xnp6Op6fnfbdxpvzistlsfPDBB5w9e5Y5c+agUqmoUaMG1apVsz/28fEhPj6eSpUqOSMF4P55KIrCsGHD7G+4du3acfTo0RJ7TADWrFnD0KFD7c+L4zG5n5L2PslPSXmf5KekvVccUZLeK1evXmXs2LFERUXRo0cP+3JnvF/klEQhadq0KVu3bgXgwIEDBAcH29fVqlWL8+fPk5ycjNlsZu/evTRp0uS+2zhTfnG99dZbmEwm5s2bZ+9yXbFiBe+99x4A169fx2g0EhAQ8GgD/5v75WE0GunevTvp6ekoisKuXbsIDQ0tsccEcrqNmzZtan9eHI/J/ZS090l+Ssr7JD8l7b3iiJLyXrl58yYjR45k4sSJ9O3bN9c6Z7xfSkY5WAJ07NiR7du3M3DgQBRFYdasWaxdu5aMjAwGDBjAG2+8wahRo1AUhT59+lChQoV7blMc3C+X0NBQVqxYQXh4OMOGDQNg6NCh9O3bl8mTJxMZGYlKpWLWrFlO/7aR3zGZMGECQ4cORafT8dhjj9GuXTtsNluJOyYDBgwgMTERvV6PSqWyb1Mcj8m9lNT3yb2UxPdJXkrqe+VeSup75fPPPyc1NZV58+Yxb948APr160dmZqZT3i9yt0ohhBBC5EtOSQghhBAiX1IwCCGEECJfUjAIIYQQIl9SMAghhBAiX1IwCCGEEKXMwYMHGTJkyH3brFq1in79+tG7d28+/fTTfPcpBYMQwu7SpUvUrVs31x8ak8nE119/7cSo/mfbtm0cOnTI/nzXrl3UrVuXmTNnOjEqIYqX+fPnM3XqVEwmU55tLly4wJIlS1i4cCErVqwgOzub7Ozs++5XCgYhxH0NHjzYfg24My1evJhRo0Zx48YN+7IqVaowbtw4+9z5Qoi776Vx/PhxhgwZwpAhQxg/fjxpaWns2LGD0NBQJk2axODBg2natClarfa++3X+zBRCiGItISHB2SEA946jatWqjB8/3gnRCFF8/f1eGm+++SazZs2idu3aLF++nK+++go3Nzf27t3LkiVLMJlMREZGsmLFCry8vPLcrxQMQgghRCl2+vRppk+fDkB2djY1atQgLCyMFi1aYDAYMBgM1KpVi3PnzhEWFpbnfuSUhBDinm6PZ7h8+TJpaWnUrVuXN954w77eaDTy4Ycf8tRTTxEaGkqbNm2YNm3aXT0Bb7zxBnXr1uXQoUN069aNhg0b2qetBdi8eTPPPfccrVq1IiQkhFatWvHSSy9x7Ngx+z6GDBnC3LlzARg7dix169YF8h7DcPbsWV577TUef/xxQkNDeeqpp/jXv/5FWlraPWNLSUlh2rRptG7dmoYNG9K7d29+++23wvtlCuFENWrU4P3332fhwoVMnDiRdu3a0bRpU3bv3o3JZCIjI8N+u+z7kR4GIcQ9eXl5MW7cOL799ltMJhOjR4+mfv36AKSlpREVFcWJEyd47LHH6NSpE5cuXWLZsmX89ddfLF26lPLly+fa35gxY2jYsCGtW7fGw8MDlUrF999/z4wZMwgKCqJ79+5otVoOHz7Mpk2b2LlzJ+vXr6d8+fI8++yzAOzevZtu3brZb0t8LwcPHmT48OFkZWXRvn17AgMDOXDgAP/973/ZvHkzS5YswcfHJ9c2I0aMIDk5ma5du5KRkcHatWv5xz/+wffff094eHjh/mKFeMTefvttJk2ahNVqBWDmzJnUqFGDPn36EBkZiaIovPTSS3e9L+6iCCHELRcvXlSCg4OVwYMH25e1b99eadasWa52b7/9thIcHKx8//33uZZv3LhRCQ4OVl5++WX7skmTJinBwcHKuHHjcrU1mUxK06ZNlU6dOinp6em51k2bNk0JDg5Wli5dal/2ySefKMHBwcqGDRvsy3bu3KkEBwcr77zzjqIoimKxWJROnTopDRo0ULZs2ZJrnx988IESHBysTJ48+a7Y+vbtmyuGNWvWKMHBwcrEiRPv/wsTogyRUxJCiAKxWCysXr2aOnXqMGjQoFzrOnToQNOmTdmwYQNGozHXus6dO+d6brVamTFjBjNnzsTDwyPXuhYtWgAFH3C5f/9+zp07x9NPP03btm1zrXv55ZepUKECa9euxWw251o3aNCgXDG0a9cOgHPnzhXo9YUozeSUhBCiQM6ePUtGRgZWqzXXpVu3mUwmrFYrx48fp1mzZvblVapUydXO3d2dbt262fd5+vRpLly4wMmTJ4mOjgbAZrMVKLbb4x6aN29+1zqdTkfDhg3ZuHEjZ86coV69evZ1NWrUyNXW09MT4K7CQoiyTAoGIUSBpKamAnDmzBn7QMR7SUlJyfXczc3trjZ79uzh3XffJTY2FgBXV1fq1atHSEgIV69etQ+MdNTtXg2DwXDP9bfHVWRmZuZartPpcj1XqVQABX59IUozKRiEEAWi1+sBeOaZZ/jXv/71wPu5fPkyzz//PK6ursyYMYNmzZpRvXp11Go1v/zyCxs3bnzg2O6c3OlOt4udfAd3CSHuImMYhBAFUqNGDXQ6HbGxsff8Bv7NN98wb948kpKS7rufjRs3kpmZycsvv0z//v2pVasWarUayLluHHJ/w7/9rf9+bl/FERMTc9c6m81GTEwMHh4ed50eEULkTwoGIcR9abVaLBaL/bmrqyvdunXj1KlTLFiwIFfbXbt28a9//YuVK1fi7e193/26uroCcPPmzVzL4+Li+O677wByva5Gk9Mher9xBc2aNaNatWr8/vvvbNmyJde6Tz75hKtXr9K1a9e7TkEIIfInpySEEPdVvnx5zp07x2uvvUZERAS9evVi0qRJ7N+/n/fff59NmzYRFhbG9evX+f3339FoNMyaNQsXl/t/H2nfvj2zZ8/miy++4MyZMwQFBXH+/Hk2b95sH3SYnJxsb1+hQgUAPvvsM44dO8a4cePu2qeLiwvvvfceo0aN4sUXX6R9+/YEBQWxf/9+Dhw4QK1atXj99dcL75cjRBkiPQxCiPuaOHEiderUYf369fz0008A+Pn5sWzZMkaOHMn169dZuHAhe/fu5cknn2TZsmW0bNky3/1WqFCBBQsW0KpVK3bu3MnixYs5e/YsQ4YM4ddff8XHx4e//vrLflqiW7dudO3alYsXL7J48WIuX758z/02bdqUFStW0K1bN/bv38+iRYtITk5mzJgxLF++XMYvCPGAVIoMAxZCCCFEPqSHQQghhBD5koJBCCGEEPmSgkEIIYQQ+ZKCQQghhBD5koJBCCGEEPmSgkEIIYQQ+ZKCQQghhBD5koJBCCGEEPmSgkEIIYQQ+ZKCQQghhBD5+v+aTfC+Q12gyQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x396 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_lines(\n",
    "    [cumulative_reward_eps_greedy, cumulative_reward_UCB],\n",
    "    [r\"$\\epsilon$-greedy\", \"UCB\"],\n",
    "    [\"Iteration\", \"Reward collected so far\"],\n",
    "    \"figs\",\n",
    "    \"ucbvseps\",\n",
    "    show=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a1249fb",
   "metadata": {},
   "source": [
    "**Question:** You are given an environment with $N$ states linked in a chain, i.e. from every state except the two endpoints there are two possible actions going right or go back to the first state in the chain. For the endpoints the second action is to stay in the same state. The reward is only 1 if you are in the rightmost state and stay there, otherwise 0. What is the probability that the $\\varepsilon$-greedy algorithm will have a positive reward after $H$ steps in the first episode, if the chosen initial state is the leftmost state in the chain?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "598be158",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "426724d8",
   "metadata": {},
   "source": [
    "**Question:** Does the result match the theoretical result described in (Jin et al. 2018) ? Try to answer this question after reading their Appendix A and looking at their Table 1 for the algorithm they call UCB-H."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8de834a",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "author": "",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  },
  "vscode": {
   "interpreter": {
    "hash": "651c3b7b3f466eb030aa737f279c846be4db9cba49a2f229278cab5e41121ed5"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}