{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Environment 2\n",
    "## Policy Gradient algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "In this notebook, the goal of this environment is to implement 4 policy gradient algorithms: the regular policy gradient (REINFORCE), the policy gradient with a baseline subtracted (REINFORCE with baseline), the one-step actor-critic method, and the advantage actor-critic method.\n",
    "Our agent will play a game in continuous state space, for which we will use a basis function representation to limit ourselves to a finite amount of parameters.\n",
    "\n",
    "Let's start with a few imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from environments.environment2 import *\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.cm\n",
    "import matplotlib.colors\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import random\n",
    "from tqdm import tqdm\n",
    "from scipy.special import *\n",
    "\n",
    "# reasonable plot size\n",
    "plt.rcParams['figure.dpi'] = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The environment\n",
    "The environment is a 2D grid, with a goal in the center, and a \"U\"-shaped barrier. Our goal is to learn how to reach the objective in the center regardless of where we start. In particular, we will study how the agent learns to go around the barrier when starting at the bottom of the grid.\n",
    "\n",
    "- The grid is 10x10, going from $(-5, -5)$ to $(5, 5)$.\n",
    "- The objective is a circle of radius $0.5$ centered at $(0, 0)$, that must be reached for the agent to win.\n",
    "- The barrier is a \"U\"-shaped wall of width $1$ centered at $(0, 0)$.\n",
    "- The agent is not allowed to exit the grid, or enter the wall.\n",
    "\n",
    "Here is what the grid looks like:\n",
    "(the objective is in green, the walls are in red)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = UMaze()\n",
    "env.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Action and reward modeling\n",
    "We assume that the state space is continuous, i.e. that the agent can be at any allowed position in the grid. At each position, we can move a fixed distance $0.2$ horizontally, vertically, or diagonally. Hence, there are $8$ possible actions:\n",
    "- Action $0$: move by $0.2 \\cdot (0, 1)$.\n",
    "- Action $1$: move by $0.2 \\cdot (\\sqrt{2}/2, \\sqrt{2}/2)$.\n",
    "- Action $2$: move by $0.2 \\cdot (1, 0)$.\n",
    "- Action $3$: move by $0.2 \\cdot (\\sqrt{2}/2, -\\sqrt{2}/2)$.\n",
    "- Action $4$: move by $0.2 \\cdot (0, -1)$.\n",
    "- Action $5$: move by $0.2 \\cdot (-\\sqrt{2}/2, -\\sqrt{2}/2)$.\n",
    "- Action $6$: move by $0.2 \\cdot (-1, 0)$.\n",
    "- Action $7$: move by $0.2 \\cdot (-\\sqrt{2}/2, \\sqrt{2}/2)$.\n",
    "\n",
    "We grant a reward of $1$ when the agent reaches the objective. Actions that would cause the agent to hit a wall are ignored.\n",
    "\n",
    "_Note: the agent is not penalized when it makes a wrong move, otherwise it might become scared of walls and get stuck below the U._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Policy and value functions\n",
    "To model our policy function for the continous state space, we first need to give each state and each action a value. For that, we will use a grid of $N$ gaussian basis functions with a fixed standard deviation $\\sigma$. **We will study how these two parameters influence the speed and quality of learning.**\n",
    "\n",
    "Let's define the action value function $q(\\vec s, a)$ as follows:\n",
    "$$q(\\vec s, a) = \\sum_{i=1}^N w_{i,a} \\exp \\left(- \\frac{||\\vec s - \\vec x_i||^2}{2\\sigma^2}\\right)$$\n",
    "where $\\vec x_i$ denotes the position of the $i$-th basis function.\n",
    "For our policy function $\\pi$, we will use a softmax policy:\n",
    "$$\\pi(\\vec s, a) = \\frac{\\exp {q(\\vec s, a)}}{\\sum_{b=0}^7 \\exp {q(\\vec s, b)}}.$$\n",
    "Our goal is to learn the $N \\times 8$ weights $w_{i, a}$.\n",
    "\n",
    "For some algorithms, we will also need to learn a state value function $v(\\vec s)$. Once again, we will use basis functions:\n",
    "$$v(\\vec s) = \\sum_{i=1}^N \\theta_{i} \\exp \\left(- \\frac{||\\vec s - \\vec x_i||^2}{2\\sigma^2}\\right).$$\n",
    "For the value function, there are $N$ parameters to learn: the weights $\\theta_i$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Policy gradient (REINFORCE)\n",
    "The standard policy gradient only uses the policy function and the gradient of the log-policy $\\nabla_{w} \\ln \\pi(\\vec s, a)$. Here is the pseudocode:\n",
    "\n",
    "![Reinforce](assets/Reinforce.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Policy gradient with baseline subtracted (REINFORCE with baseline)\n",
    "We can subtract a baseline in the policy gradient algorithm. As seen in the lecture, that should improve the speed of learning. We will use the value function as the baseline, and we will also need its gradient to update it.\n",
    "\n",
    "![ReinforceWithBaseline](assets/ReinforceWithBaseline.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-step actor-critic\n",
    "In the previous two algorithms, we can only perform the update at the end of the episode. The disadvantages are that we need to wait longer before we can learn, and we also use more memory to store all the past states, actions and rewards. With the one-step actor-critic algorithm, the updates are instead performed in real time.\n",
    "\n",
    "![OneStepActorCritic](assets/OneStepActorCritic.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Actor-critic with eligibility traces\n",
    "We also look at the eligibility trace version of the actor-critic algorithm, also known as Advantage Actor-Critic.\n",
    "\n",
    "![EligibilityTraceActorCritic](assets/EligibilityTraceActorCritic.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation\n",
    "Your goal is to implement the algorithms explained above, and observe the following:\n",
    "- How the number of basis functions influences the speed of learning.\n",
    "- How the standard deviation of the basis functions influences the speed of learning.\n",
    "- How the choice of algorithm influences the speed of learning.\n",
    "\n",
    "Here are a few important notes:\n",
    "- In each episode, we pick the starting position at random.\n",
    "- The discount factor $\\gamma$ needs to be very close to $1$ due to the high number of steps per episode. $0.99$ is a good choice.\n",
    "- To avoid wasting too much time in the early episodes, we enforce a hard limit on the number of steps per episode. $300$ is a good limit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Core model functions\n",
    "Here are a few core functions that we will use in all 3 algorithms. The `ModelParams` class is used to store the discount factor $\\gamma$, the count and shape of the basis functions, and the weights $w_{i,a}$ and $\\theta_i$. Please complete the code as necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ModelParams:\n",
    "    def __init__(self, npoints=13, gamma=0.99, stdev=0.5):\n",
    "        self.action_weights = np.zeros((npoints * npoints, 8))\n",
    "        self.value_weights = np.zeros(npoints * npoints)\n",
    "        # need to be VERY careful here given the large number of iterations\n",
    "        # if gamma is too small we'll have a hard time updating the policy correctly!\n",
    "        self.gamma = gamma\n",
    "        self.stdev = stdev\n",
    "\n",
    "        self.npoints = npoints\n",
    "        positions = np.linspace(-6, 6, npoints)\n",
    "        x_pos, y_pos = np.meshgrid(positions, positions)\n",
    "        self.x_pos = x_pos.flatten()\n",
    "        self.y_pos = y_pos.flatten()\n",
    "\n",
    "        self.w_cache = None\n",
    "        self.w_cache_pos = None\n",
    "\n",
    "    def get_interpolation_weights(self, position: State) -> np.ndarray:\n",
    "        \"\"\"\n",
    "        This returns an array of size N, wit for each i: exp(-||s-x_i||^2/(2\\sigma^2))\n",
    "        \"\"\"\n",
    "        # We do a bit of caching to not compute this every time\n",
    "        if self.w_cache_pos == position:\n",
    "            return self.w_cache\n",
    "\n",
    "        w = np.exp(- ((self.x_pos - position[0]) ** 2 + (self.y_pos - position[1]) ** 2) / (2 * self.stdev ** 2))\n",
    "        self.w_cache = w\n",
    "        self.w_cache_pos = position\n",
    "        return w\n",
    "\n",
    "    def compute_value(self, position: State) -> float:\n",
    "        \"\"\"\n",
    "        This computes the v(s) function.\n",
    "        \"\"\"\n",
    "\n",
    "        # TO COMPLETE: multiply interpolation weights with self.value_weights\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    def get_action_values(self, position: State) -> np.ndarray:\n",
    "        \"\"\"\n",
    "        This computes [q(s,a) for all a] for a specific s.\n",
    "        \"\"\"\n",
    "\n",
    "        # TO COMPLETE: multiply interpolation weights with action weights\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    def get_action_probabilities(self, position: State) -> np.ndarray:\n",
    "        \"\"\"\n",
    "        This computes the [\\pi(s, a) for all a].\n",
    "        \"\"\"\n",
    "\n",
    "        # TO COMPLETE: compute the softmax\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    def get_action_probability(self, position: State, action: Action) -> float:\n",
    "        \"\"\"\n",
    "        This computes \\pi(s, a).\n",
    "        \"\"\"\n",
    "        return self.get_action_probabilities(position)[action]\n",
    "\n",
    "    def pick_action(self, position: State) -> Action:\n",
    "        \"\"\"\n",
    "        This randomly chooses an action following the policy.\n",
    "        \"\"\"\n",
    "        probs = self.get_action_probabilities(position)\n",
    "        # np.random.choice is a bit slow, so we code it by hand\n",
    "        r = random.random()\n",
    "        for i, p in enumerate(probs):\n",
    "            r -= p\n",
    "            if r < 0:\n",
    "                return i\n",
    "\n",
    "    def compute_action_log_grad(self, position: State, action: Action) -> np.ndarray:\n",
    "        \"\"\"\n",
    "        This computes \\nabla_w \\ln(\\pi(s,a))\n",
    "        \"\"\"\n",
    "        \n",
    "        # TO COMPLETE: compute the gradient\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    def compute_value_grad(self, position: State) -> np.ndarray:\n",
    "        \"\"\"\n",
    "        This computes \\nabla_\\theta v(s).\n",
    "        \"\"\"\n",
    "        \n",
    "        # TO COMPLETE: compute the gradient\n",
    "        raise NotImplementedError()\n",
    "\n",
    "def find_starting_pos(env: UMaze) -> State:\n",
    "    \"\"\"\n",
    "    This picks a random starting position inside the bounds of the environment.\n",
    "    \"\"\"\n",
    "    pos_candidate = None\n",
    "    while pos_candidate is None:\n",
    "        pos_candidate = (random.uniform(-5, 5), random.uniform(-5, 5))\n",
    "        # the two below are easier, and can be used for debugging\n",
    "        #pos_candidate = (0, 4)\n",
    "        #pos_candidate = (random.uniform(-1.5, 1.5), random.uniform(-1.5, 1.5))\n",
    "        if env.is_outside_of_area(*pos_candidate) or env.is_in_objective(*pos_candidate):\n",
    "            pos_candidate = None\n",
    "    return pos_candidate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plots and helpers\n",
    "Here are a few functions to provide nice visualization for the learned value and policy functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def render_values(params: ModelParams):\n",
    "    \"\"\"\n",
    "    Visualises the v(s) function for all possible states.\n",
    "    \"\"\"\n",
    "    xs = np.linspace(-6, 6, 101)\n",
    "    ys = np.linspace(-6, 6, 101)\n",
    "    X, Y = np.meshgrid(xs, ys, indexing=\"ij\")\n",
    "    values = np.zeros_like(X)\n",
    "    for i in range(101):\n",
    "        for j in range(101):\n",
    "            values[i, j] = params.compute_value((xs[i], ys[j]))\n",
    "    #plt.imshow(values, extent=[-6, 6, -6, 6])\n",
    "    cs = plt.contourf(X, Y, values, levels=10, cmap=\"jet\")\n",
    "    plt.xlabel(\"x\")\n",
    "    plt.ylabel(\"y\")\n",
    "    cbar = plt.colorbar(cs)\n",
    "    cbar.set_label(\"Value\")\n",
    "    plt.title(\"Computed values\")\n",
    "\n",
    "def render_actions(params: ModelParams):\n",
    "    \"\"\"\n",
    "    This visualises the most probable action in each state using a colored arrow. \n",
    "    Color of the arrow indicates probability.\n",
    "    Only shows a few discrete points.\n",
    "    \"\"\"\n",
    "    colormap = matplotlib.cm.get_cmap(\"jet\")\n",
    "\n",
    "    xs = np.linspace(-5, 5, 21)\n",
    "    ys = np.linspace(-5, 5, 21)\n",
    "    for x in xs:\n",
    "        for y in ys:\n",
    "            action_probabilities = params.get_action_probabilities((x, y))\n",
    "            action = np.argmax(action_probabilities)\n",
    "            max_probability = action_probabilities[action]\n",
    "            plt.plot(x, y, \"o\", color=\"black\", markersize=0.1)\n",
    "            plt.arrow(x, y, 0.3 * MOVES[action][0], 0.3 * MOVES[action][1], head_width=0.1, head_length=0.1, color=colormap(max_probability), linewidth=1)\n",
    "\n",
    "    plt.xlabel(\"x\")\n",
    "    plt.ylabel(\"y\")\n",
    "    cbar = plt.colorbar(matplotlib.cm.ScalarMappable(norm=matplotlib.colors.Normalize(vmin=0, vmax=1), cmap=colormap))\n",
    "    cbar.set_label(\"Probability\")\n",
    "    plt.title(\"Preferred actions\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a function that you can use to run a learning algorithm, and display a nice visualization of the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Markdown\n",
    "\n",
    "def run_algorithm(env: UMaze, params: ModelParams, algorithm, plot_values, num_episodes: int, bucket_size: int = 500, **kwargs):\n",
    "    \"\"\"\n",
    "    This function runs an algorithm for a specified number of episodes.\n",
    "    Its arguments are:\n",
    "    * env(UMaze): Instance of the environment\n",
    "    * params(ModelParam): Parameters given to the model.\n",
    "    * algorithm(function that accepts env and params as arguments and returns a Tuple[float, int]): Your algorithm.\n",
    "    * plot_values(bool): If True, it will plot the values using one of the helper functions above.\n",
    "    * num_episodes(int): Number of episodes to play.\n",
    "    * bucket_size(int): Specifies how big groups should be when computing averages, i.e. a size of 500 averages a reward for diplay over 500 episodes.\n",
    "    * Extra arguments specified like \"action_rate=0.1\" will be passed to the algorithm\n",
    "    \"\"\"\n",
    "    assert num_episodes % bucket_size == 0, \"num_episodes must be a multiple of bucket_size\"\n",
    "\n",
    "    num_buckets = num_episodes // bucket_size\n",
    "    rewards = np.zeros(num_buckets)\n",
    "    lengths = np.zeros(num_buckets)\n",
    "    for i in tqdm(range(num_episodes)):\n",
    "        random.seed(i) # set the seed to the episode id to ensure reproducibility\n",
    "        episode_reward, episode_length = algorithm(env, params, **kwargs)\n",
    "        rewards[i // bucket_size] += episode_reward / bucket_size\n",
    "        lengths[i // bucket_size] += episode_length / bucket_size\n",
    "    plot_points = (np.arange(num_buckets) + 1) * bucket_size\n",
    "\n",
    "    display(Markdown(\"#### %s (sigma=%.2f, npoints=%d^2)\" % (algorithm.__name__, params.stdev, params.npoints)))\n",
    "    plt.subplot(221)\n",
    "    plt.plot(plot_points, rewards)\n",
    "    plt.title(\"Fraction of episodes where the goal is reached before %d iterations\" % 300) # Max iterations is hardcoded to 300 by default\n",
    "    plt.xlabel(\"Episode\")\n",
    "    plt.ylabel(\"Fraction\")\n",
    "\n",
    "    plt.subplot(222)\n",
    "    plt.plot(plot_points, lengths)\n",
    "    plt.title(\"Episode length over time\")\n",
    "    plt.xlabel(\"Episode\")\n",
    "    plt.ylabel(\"Episode length\")\n",
    "\n",
    "    plt.subplot(223)\n",
    "    render_actions(params)\n",
    "\n",
    "    if plot_values:\n",
    "        plt.subplot(224)\n",
    "        render_values(params)\n",
    "\n",
    "    plt.subplots_adjust(wspace=0.2, hspace=0.3)\n",
    "    #plt.tight_layout()\n",
    "    plt.gcf().set_size_inches(12, 8)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### REINFORCE\n",
    "First, implement the reinforce algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'UMaze' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[0;31mNameError\u001B[0m                                 Traceback (most recent call last)",
      "Cell \u001B[0;32mIn [1], line 1\u001B[0m\n\u001B[0;32m----> 1\u001B[0m \u001B[39mdef\u001B[39;00m \u001B[39mreinforce\u001B[39m(env: UMaze, params: ModelParams, action_rate \u001B[39m=\u001B[39m \u001B[39m5e-2\u001B[39m, max_iter \u001B[39m=\u001B[39m \u001B[39m300\u001B[39m) \u001B[39m-\u001B[39m\u001B[39m>\u001B[39m Tuple[\u001B[39mfloat\u001B[39m, \u001B[39mint\u001B[39m]:\n\u001B[1;32m      2\u001B[0m     \u001B[39m\"\"\"\u001B[39;00m\n\u001B[1;32m      3\u001B[0m \u001B[39m    Run an episode and mutate the params in-place.\u001B[39;00m\n\u001B[1;32m      4\u001B[0m \u001B[39m    :return: tuple containing the (total) reward for the episode, and the number of steps\u001B[39;00m\n\u001B[1;32m      5\u001B[0m \u001B[39m    \"\"\"\u001B[39;00m\n\u001B[1;32m      6\u001B[0m     env\u001B[39m.\u001B[39mreset()\n",
      "\u001B[0;31mNameError\u001B[0m: name 'UMaze' is not defined"
     ]
    }
   ],
   "source": [
    "def reinforce(env: UMaze, params: ModelParams, action_rate = 5e-2, max_iter = 300) -> Tuple[float, int]:\n",
    "    \"\"\"\n",
    "    Run an episode and mutate the params in-place.\n",
    "    :return: tuple containing the (total) reward for the episode, and the number of steps\n",
    "    \"\"\"\n",
    "    env.reset()\n",
    "    # Pick random starting position\n",
    "    env._state = find_starting_pos(env)\n",
    "    states = [env._state]\n",
    "    actions = []\n",
    "    rewards = []\n",
    "    iteration = 0\n",
    "    while iteration < max_iter and not env.end:\n",
    "        # Compute action\n",
    "        action = params.pick_action(states[-1])\n",
    "        actions.append(action)\n",
    "        # Do action\n",
    "        state, reward = env.do_action(action)\n",
    "        states.append(state)\n",
    "        rewards.append(reward)\n",
    "        iteration += 1\n",
    "\n",
    "    rewards = np.array(rewards)\n",
    "    for t in range(len(actions)):\n",
    "        # TO COMPLETE: compute G\n",
    "        G = \"TODO\"\n",
    "        # TO COMPLETE: update action weights\n",
    "        params.action_weights += action_rate * \"TODO\"\n",
    "\n",
    "    return sum(rewards), iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you can perform a few experiments!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.25, npoints=9), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.50, npoints=9), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.75, npoints=9), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.00, npoints=9), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.50, npoints=9), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.25, npoints=13), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.50, npoints=13), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.75, npoints=13), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.00, npoints=13), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.50, npoints=13), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.15, npoints=19), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.25, npoints=19), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.50, npoints=19), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.75, npoints=19), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.00, npoints=19), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.50, npoints=19), reinforce, plot_values=False, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### REINFORCE with baseline\n",
    "Now, copy/paste and modify your `reinforce` function above to subtract a basline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reinforce_with_baseline(env: UMaze, params: ModelParams, action_rate = 5e-2, value_rate = 5e-2, max_iter = 300) -> Tuple[float, int]:\n",
    "    \"\"\"\n",
    "    Run an episode and mutate the params in-place.\n",
    "    :return: tuple containing the (total) reward for the episode, and the number of steps\n",
    "    \"\"\"\n",
    "    # TO COMPLETE: copy/paste reinforce() and subtract the baseline. Don't forget to update the value weights.\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you can perform a few experiments!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.25, npoints=9), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.50, npoints=9), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.75, npoints=9), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.00, npoints=9), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.50, npoints=9), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.25, npoints=13), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.50, npoints=13), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.75, npoints=13), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.00, npoints=13), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.50, npoints=13), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.15, npoints=19), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.25, npoints=19), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.50, npoints=19), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.75, npoints=19), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.00, npoints=19), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.50, npoints=19), reinforce_with_baseline, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-step actor-critic\n",
    "Now, implement the one-step actor-critic algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "def one_step_actor_critic(env: UMaze, params: ModelParams, action_rate = 5e-2, value_rate = 5e-2, max_iter = 300) -> Tuple[float, int]:\n",
    "    \"\"\"\n",
    "    Run an episode and mutate the params in-place.\n",
    "    :return: tuple containing the (total) reward for the episode, and the number of steps\n",
    "    \"\"\"\n",
    "    env.reset()\n",
    "    # Pick random starting position\n",
    "    env._state = find_starting_pos(env)\n",
    "    total_reward = 0\n",
    "    iteration = 0\n",
    "    I = 1\n",
    "    while iteration < max_iter and not env.end:\n",
    "        old_state = env._state\n",
    "        # Compute action\n",
    "        action = params.pick_action(old_state)\n",
    "        # Do action\n",
    "        state, reward = env.do_action(action)\n",
    "        total_reward += reward\n",
    "        # Update\n",
    "        new_value = 0 if env.end else params.compute_value(state)\n",
    "        old_value = params.compute_value(old_state)\n",
    "        # TO COMPLETE: \n",
    "        delta = \"TODO\"\n",
    "        params.value_weights += value_rate * \"TODO\"\n",
    "        params.action_weights += action_rate * I * \"TODO\"\n",
    "\n",
    "        I *= params.gamma\n",
    "        iteration += 1\n",
    "\n",
    "    return total_reward, iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you can perform a few experiments!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.25, npoints=9), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.50, npoints=9), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.75, npoints=9), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.00, npoints=9), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.50, npoints=9), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.25, npoints=13), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.50, npoints=13), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.75, npoints=13), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.00, npoints=13), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.50, npoints=13), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.15, npoints=19), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.25, npoints=19), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.50, npoints=19), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.75, npoints=19), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.00, npoints=19), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.50, npoints=19), one_step_actor_critic, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Actor-critic with eligibility traces\n",
    "Finally, implement the actor-critic algorithm with elgibility traces. Copy/paste and modify your `one_step_actor_critic` function above to use eligibility traces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "def actor_critic_with_eligibility_traces(env: UMaze, params: ModelParams, action_rate = 5e-2, value_rate = 5e-2, max_iter = 300, trace_decay=0.99) -> Tuple[float, int]:\n",
    "    \"\"\"\n",
    "    Run an episode and mutate the params in-place.\n",
    "    :return: tuple containing the (total) reward for the episode, and the number of steps\n",
    "    \"\"\"\n",
    "    # TO COMPLETE: copy/paste one_step_actor_critic() and add eligibility traces. Notice the additional trace_decay parameter.\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you can perform a few experiments!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.25, npoints=9), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.50, npoints=9), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.75, npoints=9), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.00, npoints=9), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.50, npoints=9), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.25, npoints=13), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.50, npoints=13), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.75, npoints=13), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.00, npoints=13), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.50, npoints=13), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.15, npoints=19), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.25, npoints=19), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.50, npoints=19), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=0.75, npoints=19), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.00, npoints=19), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "run_algorithm(UMaze(), ModelParams(stdev=1.50, npoints=19), actor_critic_with_eligibility_traces, plot_values=True, num_episodes=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questions\n",
    "Make sure you try running with various `npoints` and `stdev` values. Good values for `npoints` are $9$, $13$ and $19$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For each value of `npoints`: try running with various standard deviations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Find a value of `stdev` that works well.\n",
    "You can start with `stdev=0.5` and see what happens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What happens when `stdev` is too small?\n",
    "Can you explain this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What happens when `stdev` is too big?\n",
    "Can you explain this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Does the optimal value of `stdev` depend on `npoints`?\n",
    "Can you explain this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare the various algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How do REINFORCE and REINFORCE with baseline compare?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How do the two actor-critic algorithms compare?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How does REINFORCE with baseline compare to the actor-critic algorithms?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experiment with learning rates\n",
    "So far, we left the learning rates at their default value of `5e-2`. Algorithms with a value function are expected to be more stable than REINFORCE, and should allow higher learning rates. Experiment with the learning rates and see what happens.\n",
    "\n",
    "You can adjust the learning rates by passing extra arguments to `run_algorithm`, for example:\n",
    "```py\n",
    "run_algorithm(UMaze(), ModelParams(stdev=0.50, npoints=13), reinforce_with_baseline, plot_values=True, num_episodes=10000, value_rate=<insert value rate>, action_rate=<insert action rate>)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What happens when you lower the learning rates?\n",
    "Can you explain this?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TO COMPLETE: perform some experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What happens when you increase the learning rates? Try increasing the action rate and the value rate separately.\n",
    "Can you explain this?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TO COMPLETE: perform some experiments"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "e7370f93d1d0cde622a1f8e1c04877d8463912d04d973331ad4851f04de6915a"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
