{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-1 fw=all -->\n",
    "\n",
    "# Automatic Differentiation\n",
    "\n",
    "Recall from that section \n",
    "that derivatives drive all the optimization algorithms\n",
    "that we will use to train deep networks.\n",
    "While the calculations are straightforward,\n",
    "working them out by hand can be tedious and error-prone, \n",
    "and these issues only grow\n",
    "as our models become more complex.\n",
    "\n",
    "Fortunately all modern deep learning frameworks\n",
    "take this work off our plates\n",
    "by offering *automatic differentiation*\n",
    "(often shortened to *autograd*). \n",
    "As we pass data through each successive function,\n",
    "the framework builds a *computational graph* \n",
    "that tracks how each value depends on others.\n",
    "To calculate derivatives, \n",
    "automatic differentiation \n",
    "works backwards through this graph\n",
    "applying the chain rule. \n",
    "The computational algorithm for applying the chain rule\n",
    "in this fashion is called *backpropagation*.\n",
    "\n",
    "Autograd has a long history:\n",
    "the earliest references date back\n",
    "over half a century [@Wengert.1964],\n",
    "and reverse mode, the variant that powers\n",
    "modern backpropagation, was developed\n",
    "by @Linnainmaa.1970 .\n",
    "that section\n",
    "recounts this history in full.\n",
    "Before exploring methods, \n",
    "let's first master the autograd package."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-automatic-differentiation"
   },
   "source": [
    "import torch"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-2 fw=all -->\n",
    "\n",
    "## Mechanics\n",
    "\n",
    "We begin with the basic workflow (attach a gradient,\n",
    "record a computation, run the backward pass), first\n",
    "on a scalar-valued function, then on vector-valued ones.\n",
    "\n",
    "### A Simple Function\n",
    "\n",
    "Let's assume that we are interested\n",
    "in differentiating the function\n",
    "$y = 2\\mathbf{x}^{\\top}\\mathbf{x}$\n",
    "with respect to the column vector $\\mathbf{x}$.\n",
    "To start, we assign `x` an initial value."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-a-simple-function-1"
   },
   "source": [
    "x = torch.arange(4.0)\n",
    "x"
   ],
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "tensor([0., 1., 2., 3.])"
     },
     "metadata": {},
     "execution_count": 2
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-3 fw=mxnet, pytorch, tensorflow -->\n",
    "\n",
    "Before we calculate the gradient\n",
    "of $y$ with respect to $\\mathbf{x}$,\n",
    "we need a place to store it.\n",
    "In general, we avoid allocating new memory\n",
    "every time we take a derivative \n",
    "because deep learning requires \n",
    "successively computing derivatives\n",
    "with respect to the same parameters\n",
    "a great many times,\n",
    "and we might risk running out of memory.\n",
    "Note that the gradient of a scalar-valued function\n",
    "with respect to a vector $\\mathbf{x}$\n",
    "is vector-valued with \n",
    "the same shape as $\\mathbf{x}$."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-a-simple-function-2"
   },
   "source": [
    "# Can also create x = torch.arange(4.0, requires_grad=True)\n",
    "x.requires_grad_(True)\n",
    "x.grad  # The gradient is None by default"
   ],
   "execution_count": 3,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-4 fw=all -->\n",
    "\n",
    "We now calculate our function of `x` and assign the result to `y`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-a-simple-function-3"
   },
   "source": [
    "y = 2 * torch.dot(x, x)\n",
    "y"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "tensor(28., grad_fn=<MulBackward0>)"
     },
     "metadata": {},
     "execution_count": 4
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-5 fw=all -->\n",
    "\n",
    "Recording the operations gives the framework a *computational graph*,\n",
    "shown in the figure. Its nodes are operations and its\n",
    "edges carry intermediate values.\n",
    "\n",
    "![The computational graph for $y = 2\\mathbf{x}^\\top\\mathbf{x}$. The forward pass (black) flows from $\\mathbf{x}$ to $y$; reverse-mode automatic differentiation walks the same graph backward (blue), multiplying the local derivative at each node via the chain rule to accumulate $\\partial y / \\partial \\mathbf{x} = 4\\mathbf{x}$.](../img/autograd-comp-graph.svg)\n",
    "\n",
    "The forward pass evaluates the graph from $\\mathbf{x}$ to $y$; to obtain\n",
    "the gradient, automatic differentiation then traverses it in reverse,\n",
    "multiplying the local derivatives along the way. We unpack computational\n",
    "graphs and backpropagation in full in that section,\n",
    "and the underlying mathematics (both modes of automatic differentiation\n",
    "and their costs) is developed in\n",
    "that section; for now we\n",
    "simply use the resulting gradients."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-6 fw=pytorch -->\n",
    "\n",
    "We can now take the gradient of `y`\n",
    "with respect to `x` by calling \n",
    "its `backward` method.\n",
    "Next, we can access the gradient \n",
    "via `x`'s `grad` attribute."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-a-simple-function-4"
   },
   "source": [
    "y.backward()\n",
    "x.grad"
   ],
   "execution_count": 5,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "tensor([ 0.,  4.,  8., 12.])"
     },
     "metadata": {},
     "execution_count": 5
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-7 fw=all -->\n",
    "\n",
    "We already know that the gradient of the function $y = 2\\mathbf{x}^{\\top}\\mathbf{x}$\n",
    "with respect to $\\mathbf{x}$ should be $4\\mathbf{x}$.\n",
    "We can now verify that the automatic gradient computation\n",
    "and the expected result are identical."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-a-simple-function-5"
   },
   "source": [
    "x.grad == 4 * x"
   ],
   "execution_count": 6,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "tensor([True, True, True, True])"
     },
     "metadata": {},
     "execution_count": 6
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-8 fw=pytorch -->\n",
    "\n",
    "Now let's calculate \n",
    "another function of `x`\n",
    "and take its gradient.\n",
    "Note that PyTorch does not automatically \n",
    "reset the gradient buffer \n",
    "when we record a new gradient. \n",
    "Instead, the new gradient\n",
    "is added to the already-stored gradient.\n",
    "This behavior comes in handy\n",
    "when we want to optimize the sum \n",
    "of multiple objective functions.\n",
    "To reset the gradient buffer,\n",
    "we can call `x.grad.zero_()` as follows:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-a-simple-function-6"
   },
   "source": [
    "x.grad.zero_()  # Reset the gradient\n",
    "y = x.sum()\n",
    "y.backward()\n",
    "x.grad"
   ],
   "execution_count": 7,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "tensor([1., 1., 1., 1.])"
     },
     "metadata": {},
     "execution_count": 7
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-9 fw=all -->\n",
    "\n",
    "### Backward for Non-Scalar Variables\n",
    "\n",
    "When `y` is a vector, \n",
    "the most natural representation \n",
    "of the derivative of  `y`\n",
    "with respect to a vector `x` \n",
    "is a matrix called the *Jacobian*\n",
    "that contains the partial derivatives\n",
    "of each component of `y` \n",
    "with respect to each component of `x`.\n",
    "Likewise, for higher-order `y` and `x`,\n",
    "the result of differentiation could be an even higher-order tensor.\n",
    "\n",
    "While Jacobians do show up in some\n",
    "advanced machine learning techniques,\n",
    "more commonly we want to sum up \n",
    "the gradients of each component of `y`\n",
    "with respect to the full vector `x`,\n",
    "yielding a vector of the same shape as `x`.\n",
    "For example, we often have a vector \n",
    "representing the value of our loss function\n",
    "calculated separately for each example among\n",
    "a batch of training examples.\n",
    "Here, we just want to sum up the gradients\n",
    "computed individually for each example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-10 fw=pytorch -->\n",
    "\n",
    "Because deep learning frameworks vary \n",
    "in how they interpret gradients of\n",
    "non-scalar tensors,\n",
    "PyTorch takes some steps to avoid confusion.\n",
    "Invoking `backward` on a non-scalar elicits an error \n",
    "unless we tell PyTorch how to reduce the object to a scalar. \n",
    "More formally, we need to provide some vector $\\mathbf{v}$ \n",
    "such that `backward` will compute \n",
    "$\\mathbf{v}^\\top \\partial_{\\mathbf{x}} \\mathbf{y}$ \n",
    "rather than $\\partial_{\\mathbf{x}} \\mathbf{y}$. \n",
    "This argument is named `gradient`\n",
    "because the vector $\\mathbf{v}$ is the gradient arriving\n",
    "from the rest of a larger computation,\n",
    "as will become clear when we study\n",
    "backpropagation in that section. \n",
    "For a more detailed description, see the PyTorch documentation on the\n",
    "`gradient` argument to\n",
    "[`Tensor.backward`](https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-backward-for-non-scalar-variables"
   },
   "source": [
    "x.grad.zero_()\n",
    "y = x * x\n",
    "y.backward(gradient=torch.ones(len(y)))  # Equivalently: y.sum().backward()\n",
    "x.grad"
   ],
   "execution_count": 8,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "tensor([0., 2., 4., 6.])"
     },
     "metadata": {},
     "execution_count": 8
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-11 fw=all -->\n",
    "\n",
    "## Controlling the Graph\n",
    "\n",
    "Sometimes the graph the framework records\n",
    "is not the graph we want to differentiate.\n",
    "The next two subsections show how to prune it\n",
    "by detaching individual intermediate results, and how to\n",
    "switch recording off altogether.\n",
    "\n",
    "### Detaching Computation\n",
    "\n",
    "Sometimes, we wish to move some calculations\n",
    "outside of the recorded computational graph.\n",
    "For example, say that we use the input \n",
    "to create some auxiliary intermediate terms \n",
    "for which we do not want to compute a gradient. \n",
    "In this case, we need to *detach* \n",
    "the respective computational graph\n",
    "from the final result. \n",
    "The following toy example makes this clearer: \n",
    "suppose we have `z = x * y` and `y = x * x` \n",
    "but we want to focus on the *direct* influence of `x` on `z` \n",
    "rather than the influence conveyed via `y`. \n",
    "In this case, we can create a new variable `u`\n",
    "that takes the same value as `y` \n",
    "but whose *provenance* (how it was created)\n",
    "has been wiped out.\n",
    "Thus `u` has no ancestors in the graph\n",
    "and gradients do not flow through `u` to `x`.\n",
    "Now consider `z = x * u`.\n",
    "Because `u` is treated as a constant equal to $x^2$,\n",
    "the gradient is $\\partial z / \\partial x = u = x^2$.\n",
    "Had we *not* detached, so that `z = x * (x * x)` $= x^3$, we\n",
    "would instead have obtained $\\partial z / \\partial x = 3x^2$."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-detaching-computation-1"
   },
   "source": [
    "x.grad.zero_()\n",
    "y = x * x\n",
    "u = y.detach()\n",
    "z = u * x\n",
    "\n",
    "z.sum().backward()\n",
    "x.grad == u"
   ],
   "execution_count": 9,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "tensor([True, True, True, True])"
     },
     "metadata": {},
     "execution_count": 9
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-12 fw=all -->\n",
    "\n",
    "Note that while this procedure\n",
    "detaches `y`'s ancestors\n",
    "from the graph leading to `z`, \n",
    "the computational graph leading to `y` \n",
    "persists and thus we can calculate\n",
    "the gradient of `y` with respect to `x`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-detaching-computation-2"
   },
   "source": [
    "x.grad.zero_()\n",
    "y.sum().backward()\n",
    "x.grad == 2 * x"
   ],
   "execution_count": 10,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "tensor([True, True, True, True])"
     },
     "metadata": {},
     "execution_count": 10
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-13 fw=all -->\n",
    "\n",
    "### Turning Off Gradient Tracking\n",
    "\n",
    "Recording operations for a backward pass costs time and memory. When we\n",
    "only need a value (at prediction time, or while updating parameters by\n",
    "hand), we can skip the bookkeeping entirely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-14 fw=pytorch -->\n",
    "\n",
    "Wrap the computation in a `torch.no_grad()` block (or decorate a function\n",
    "with `@torch.no_grad()`). The result still shares data with `x`, but it is\n",
    "not attached to the graph, so no gradient can flow through it."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-turning-off-gradient-tracking-1"
   },
   "source": [
    "with torch.no_grad():\n",
    "    y = 2 * torch.dot(x, x)\n",
    "y.requires_grad  # False: y is detached from the graph"
   ],
   "execution_count": 11,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "False"
     },
     "metadata": {},
     "execution_count": 11
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-15 fw=all -->\n",
    "\n",
    "This untracked mode is the default for inference and evaluation\n",
    "throughout the rest of the book.\n",
    "\n",
    "## Beyond the Basics\n",
    "\n",
    "Automatic differentiation is more general\n",
    "than the fixed formulas we have differentiated so far:\n",
    "it handles arbitrary control flow, derivatives of derivatives,\n",
    "and even lets us choose the *direction*\n",
    "in which the graph is traversed.\n",
    "\n",
    "### Gradients and Python Control Flow\n",
    "\n",
    "So far we reviewed cases where the path from input to output \n",
    "was well defined via a function such as `z = x * x * x`.\n",
    "Programming offers us a lot more freedom in how we compute results. \n",
    "For instance, we can make them depend on auxiliary variables \n",
    "or condition choices on intermediate results. \n",
    "One benefit of using automatic differentiation\n",
    "is that even if building the computational graph of \n",
    "a function required passing through a maze of Python control flow\n",
    "(e.g., conditionals, loops, and arbitrary function calls),\n",
    "we can still calculate the gradient of the resulting variable.\n",
    "To illustrate this, consider the following code snippet where \n",
    "the number of iterations of the `while` loop\n",
    "and the evaluation of the `if` statement\n",
    "both depend on the value of the input `a`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-gradients-and-python-control-flow-1"
   },
   "source": [
    "def f(a):\n",
    "    b = a * 2\n",
    "    while b.norm() < 1000:\n",
    "        b = b * 2\n",
    "    if b.sum() > 0:\n",
    "        c = b\n",
    "    else:\n",
    "        c = 100 * b\n",
    "    return c"
   ],
   "execution_count": 12,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-16 fw=all -->\n",
    "\n",
    "Below, we call this function, passing in a random value, as input.\n",
    "Since the input is a random variable, \n",
    "we do not know what form \n",
    "the computational graph will take.\n",
    "However, whenever we execute `f(a)` \n",
    "on a specific input, we realize \n",
    "a specific computational graph\n",
    "and can subsequently run `backward`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-gradients-and-python-control-flow-2"
   },
   "source": [
    "a = torch.randn(size=(), requires_grad=True)\n",
    "d = f(a)\n",
    "d.backward()"
   ],
   "execution_count": 13,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-17 fw=all -->\n",
    "\n",
    "Even though our function `f` is, for demonstration purposes, a bit contrived,\n",
    "its dependence on the input is quite simple: \n",
    "it is a *linear* function of the scalar `a` \n",
    "with piecewise defined scale. \n",
    "As such, `f(a) / a` is a constant \n",
    "and, moreover, it needs to match \n",
    "the gradient of `f(a)` with respect to `a`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-gradients-and-python-control-flow-3"
   },
   "source": [
    "a.grad == d / a"
   ],
   "execution_count": 14,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "tensor(True)"
     },
     "metadata": {},
     "execution_count": 14
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-18 fw=all -->\n",
    "\n",
    "Dynamic control flow is very common in deep learning. \n",
    "For instance, when processing text, the computational graph\n",
    "depends on the length of the input. \n",
    "In these cases, automatic differentiation \n",
    "is necessary for statistical modeling \n",
    "since it is impossible to compute the gradient *a priori*. \n",
    "\n",
    "### Higher-Order Derivatives\n",
    "\n",
    "Occasionally we need the derivative of a derivative: the curvature of a\n",
    "function, or the Hessian--vector products (products of the matrix of\n",
    "second derivatives, the *Hessian*, with a vector) used by some optimizers.\n",
    "Autograd can differentiate *through* a gradient computation. Take\n",
    "$f(x) = x^3$, for which $f'(x) = 3x^2$ and $f''(x) = 6x$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-19 fw=pytorch -->\n",
    "\n",
    "Pass `create_graph=True` so the first gradient is itself a differentiable\n",
    "function of `x`, then differentiate again."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "autograd-higher-order-derivatives-1"
   },
   "source": [
    "x3 = torch.tensor(2.0, requires_grad=True)\n",
    "dy = torch.autograd.grad(x3 ** 3, x3, create_graph=True)[0]  # 3x^2 = 12\n",
    "d2y = torch.autograd.grad(dy, x3)[0]                          # 6x  = 12\n",
    "dy, d2y"
   ],
   "execution_count": 15,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "(tensor(12., grad_fn=<MulBackward0>), tensor(12.))"
     },
     "metadata": {},
     "execution_count": 15
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-20 fw=all -->\n",
    "\n",
    "### Forward versus Reverse Mode\n",
    "\n",
    "Automatic differentiation can traverse the computational graph in either\n",
    "direction. *Reverse mode*, the variant we have used so far and the engine\n",
    "behind backpropagation, sweeps from the output back to the inputs,\n",
    "yielding the gradient of a single scalar with respect to *all* inputs in\n",
    "one pass. *Forward mode* sweeps the other way, propagating derivatives\n",
    "from one input outward to every output.\n",
    "\n",
    "The choice is about cost, and a counting argument settles it.\n",
    "For a function with $n$ inputs and $m$ outputs,\n",
    "filling the full matrix of derivatives takes\n",
    "one reverse sweep per *output* ($m$ sweeps)\n",
    "or one forward sweep per *input* ($n$ sweeps),\n",
    "each sweep costing about as much as one evaluation of the function.\n",
    "A training loss is a single scalar ($m = 1$)\n",
    "depending on millions of parameters ($n$ huge),\n",
    "so reverse mode delivers the entire gradient\n",
    "for the price of roughly one extra forward pass.\n",
    "Forward mode wins in the opposite regime\n",
    "(few inputs, many outputs), and it is also the tool of choice\n",
    "for Hessian--vector products and per-input sensitivities,\n",
    "as in the Julia package ForwardDiff.jl\n",
    "[@Revels.Lubin.Papamarkou.2016].\n",
    "The exercises explore this trade-off further, and\n",
    "that section derives both modes\n",
    "and their costs in full.\n",
    "\n",
    "## Discussion\n",
    "\n",
    "Automatic differentiation frees practitioners\n",
    "from deriving gradients by hand,\n",
    "and it makes it practical to train models\n",
    "for which pen and paper gradient computations \n",
    "would be prohibitively time consuming.\n",
    "While we use autograd to *optimize* models\n",
    "(in a statistical sense),\n",
    "the *optimization* of autograd libraries themselves\n",
    "(in a computational sense)\n",
    "is a rich subject that matters to framework designers.\n",
    "Here, tools from compilers and graph manipulation \n",
    "are used to compute results \n",
    "quickly and with modest memory. \n",
    "\n",
    "For now, try to remember these basics:\n",
    "(i) attach gradients to those variables with respect to which we desire derivatives;\n",
    "(ii) record the computation of the target value;\n",
    "(iii) execute the backpropagation function; and\n",
    "(iv) access the resulting gradient.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Why is the second derivative much more expensive to compute than the first derivative?\n",
    "1. After running the function for backpropagation, immediately run it again and see what happens. Investigate.\n",
    "1. In the control flow example where we calculate the derivative of `d` with respect to `a`, what would happen if we changed the variable `a` to a random vector or a matrix? At this point, the result of the calculation `f(a)` is no longer a scalar. What happens to the result? How do we analyze this?\n",
    "1. Let $f(x) = \\sin(x)$. Plot the graph of $f$ and of its derivative $f'$. Do not exploit the fact that $f'(x) = \\cos(x)$ but rather use automatic differentiation to get the result. \n",
    "1. Let $f(x) = ((\\log x^2) \\cdot \\sin x) + x^{-1}$. Write out a dependency graph tracing results from $x$ to $f(x)$. \n",
    "1. Use the chain rule to compute the derivative $\\frac{df}{dx}$ of the aforementioned function, placing each term on the dependency graph that you constructed previously. \n",
    "1. Given the graph and the intermediate derivative results, you have a number of options when computing the gradient. Evaluate the result once sweeping from $x$ to $f$ (forward mode) and once from $f$ tracing back to $x$ (reverse mode). \n",
    "1. For the graph of exercise 5, count the operations that forward mode and reverse mode each perform, and the intermediate values each must store. How would the comparison change for a function with many inputs, or with many outputs?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=autograd-md-21 fw=pytorch -->\n",
    "\n",
    "[Discussions](https://d2l.discourse.group/t/35)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-pytorch",
   "display_name": "d2l (PyTorch)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
