{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=calculus-md-1 fw=all -->\n",
    "\n",
    "# Calculus\n",
    "\n",
    "For a long time, how to calculate \n",
    "the area of a circle remained a mystery.\n",
    "Then, in Ancient Greece, the mathematician Archimedes\n",
    "came up with the clever idea \n",
    "to inscribe a series of polygons \n",
    "with increasing numbers of vertices\n",
    "on the inside of a circle\n",
    "(the figure). \n",
    "For a polygon with $n$ vertices,\n",
    "we obtain $n$ triangles.\n",
    "The height of each triangle approaches the radius $r$ \n",
    "as we partition the circle more finely. \n",
    "At the same time, its base approaches $2 \\pi r/n$, \n",
    "since the ratio between arc and secant approaches 1 \n",
    "for a large number of vertices. \n",
    "Thus, the area of the polygon approaches\n",
    "$n \\cdot r \\cdot \\frac{1}{2} (2 \\pi r/n) = \\pi r^2$.\n",
    "\n",
    "![Finding the area of a circle as a limit procedure.](../img/polygon-circle.svg)\n",
    "\n",
    "This limiting procedure is at the root of both \n",
    "*differential calculus* and *integral calculus*. \n",
    "The former can tell us how to increase\n",
    "or decrease a function's value by\n",
    "manipulating its arguments. \n",
    "This comes in handy for the *optimization problems*\n",
    "that we face in deep learning,\n",
    "where we repeatedly update our parameters \n",
    "in order to decrease the loss function.\n",
    "Optimization addresses how to fit our models to training data,\n",
    "and calculus is its key prerequisite.\n",
    "However, do not forget that our ultimate goal\n",
    "is to perform well on *previously unseen* data.\n",
    "That problem is called *generalization*\n",
    "and will be a key focus of other chapters."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "calculus"
   },
   "source": [
    "%matplotlib inline\n",
    "from d2l import mxnet as d2l\n",
    "from matplotlib_inline import backend_inline\n",
    "from mxnet import np, npx\n",
    "npx.set_np()"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=calculus-md-2 fw=all -->\n",
    "\n",
    "## Derivatives and Differentiation\n",
    "\n",
    "Put simply, a *derivative* is the rate of change\n",
    "in a function with respect to changes in its arguments.\n",
    "Derivatives can tell us how rapidly a loss function\n",
    "would increase or decrease were we \n",
    "to *increase* or *decrease* each parameter\n",
    "by an infinitesimally small amount.\n",
    "Formally, for functions $f: \\mathbb{R} \\rightarrow \\mathbb{R}$,\n",
    "that map from scalars to scalars,\n",
    "the *derivative* of $f$ at a point $x$ is defined as\n",
    "\n",
    "$$f'(x) = \\lim_{h \\rightarrow 0} \\frac{f(x+h) - f(x)}{h}.$$\n",
    "\n",
    "This term on the right hand side is called a *limit* \n",
    "and it tells us what happens \n",
    "to the value of an expression\n",
    "as a specified variable \n",
    "approaches a particular value.\n",
    "This limit tells us what \n",
    "the ratio between the change in the function value \n",
    "$f(x + h) - f(x)$ and the perturbation $h$ \n",
    "converges to as we shrink $h$ to zero.\n",
    "Geometrically, the difference quotient is the slope of the *secant* line\n",
    "through the points $(x, f(x))$ and $(x+h, f(x+h))$;\n",
    "as $h \\rightarrow 0$, the secant pivots into the *tangent* line at $x$,\n",
    "whose slope is the derivative (the figure).\n",
    "\n",
    "![As $h \\to 0$, the secant line through $P$ and $Q$ pivots into the tangent line at $P$: the derivative is the slope of the tangent.](../img/calculus-secant-tangent.svg)\n",
    "\n",
    "When $f'(x)$ exists, $f$ is said \n",
    "to be *differentiable* at $x$;\n",
    "and when $f'(x)$ exists for all $x$\n",
    "on a set, e.g., the interval $[a,b]$, \n",
    "we say that $f$ is differentiable on this set.\n",
    "Not all functions are differentiable,\n",
    "including many that we wish to optimize,\n",
    "such as classification accuracy.\n",
    "However, because nearly every algorithm \n",
    "for training deep neural networks \n",
    "must compute the derivative of the loss,\n",
    "we often optimize a differentiable *surrogate* instead.\n",
    "\n",
    "\n",
    "We can interpret the derivative \n",
    "$f'(x)$\n",
    "as the *instantaneous* rate of change \n",
    "of $f(x)$ with respect to $x$.\n",
    "Let's develop some intuition with an example.\n",
    "Define $u = f(x) = 3x^2-4x$."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "calculus-derivatives-and-differentiation-1"
   },
   "source": [
    "def f(x):\n",
    "    return 3 * x ** 2 - 4 * x"
   ],
   "execution_count": 2,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=calculus-md-3 fw=all -->\n",
    "\n",
    "Setting $x=1$, we see that $\\frac{f(x+h) - f(x)}{h}$ approaches $2$\n",
    "as $h$ approaches $0$.\n",
    "While this experiment lacks \n",
    "the rigor of a mathematical proof,\n",
    "we can quickly see that indeed $f'(1) = 2$."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "calculus-derivatives-and-differentiation-2"
   },
   "source": [
    "for h in 10.0**np.arange(-1, -6, -1):\n",
    "    print(f'h={h:.5f}, numerical limit={(f(1+h)-f(1))/h:.5f}')"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "h=0.10000, numerical limit=2.30000\nh=0.01000, numerical limit=2.02999\nh=0.00100, numerical limit=2.00295\nh=0.00010, numerical limit=2.00033\nh=0.00001, numerical limit=2.00272\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=calculus-md-4 fw=all -->\n",
    "\n",
    "It is tempting to conclude that smaller $h$ is always better.\n",
    "Not so: in floating-point arithmetic,\n",
    "$f(x+h)$ and $f(x)$ become *nearly equal* numbers as $h$ shrinks,\n",
    "so their difference loses most of its significant digits\n",
    "to *cancellation*, and dividing by the tiny $h$\n",
    "amplifies whatever rounding noise remains.\n",
    "Continuing the sweep to much smaller $h$\n",
    "shows the approximation degrading and then failing outright:\n",
    "the error creeps into ever-higher digits, and once $h$ is so small\n",
    "that $1 + h$ rounds to exactly $1$,\n",
    "the quotient collapses to zero."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "calculus-derivatives-and-differentiation-3"
   },
   "source": [
    "for exp in range(-6, -17, -1):\n",
    "    h = 10.0 ** exp\n",
    "    print(f'h={h:.0e}, numerical limit={(f(1+h)-f(1))/h:.5f}')"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "h=1e-06, numerical limit=2.00000\nh=1e-07, numerical limit=2.00000\nh=1e-08, numerical limit=2.00000\nh=1e-09, numerical limit=2.00000\nh=1e-10, numerical limit=2.00000\nh=1e-11, numerical limit=2.00000\nh=1e-12, numerical limit=2.00018\nh=1e-13, numerical limit=1.99840\nh=1e-14, numerical limit=1.99840\nh=1e-15, numerical limit=2.22045\nh=1e-16, numerical limit=0.00000\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=calculus-md-5 fw=all -->\n",
    "\n",
    "The numerical limit is thus caught between two error sources:\n",
    "truncation error (from $h$ being too large)\n",
    "and cancellation (from $h$ being too small).\n",
    "This is one important reason why the automatic differentiation\n",
    "introduced in that section computes derivatives\n",
    "*analytically*, by applying differentiation rules,\n",
    "rather than by finite differences.\n",
    "\n",
    "There are several equivalent notational conventions for derivatives.\n",
    "Given $y = f(x)$, the following expressions are equivalent:\n",
    "\n",
    "$$f'(x) = y' = \\frac{dy}{dx} = \\frac{df}{dx} = \\frac{d}{dx} f(x) = Df(x) = D_x f(x),$$\n",
    "\n",
    "where the symbols $\\frac{d}{dx}$ and $D$ are *differentiation operators*.\n",
    "Below, we present the derivatives of some common functions:\n",
    "\n",
    "$$\\begin{aligned} \\frac{d}{dx} C & = 0 && \\textrm{for any constant $C$} \\\\ \\frac{d}{dx} x^n & = n x^{n-1} && \\textrm{for } x > 0 \\textrm{ if } n \\textrm{ is not an integer} \\\\ \\frac{d}{dx} e^x & = e^x \\\\ \\frac{d}{dx} \\ln x & = x^{-1}. \\end{aligned}$$\n",
    "\n",
    "Functions composed from differentiable functions \n",
    "are often themselves differentiable.\n",
    "The following rules cover compositions \n",
    "of any differentiable functions \n",
    "$f$ and $g$, and constant $C$.\n",
    "\n",
    "$$\\begin{aligned} \\frac{d}{dx} [C f(x)] & = C \\frac{d}{dx} f(x) && \\textrm{Constant multiple rule} \\\\ \\frac{d}{dx} [f(x) + g(x)] & = \\frac{d}{dx} f(x) + \\frac{d}{dx} g(x) && \\textrm{Sum rule} \\\\ \\frac{d}{dx} [f(x) g(x)] & = f(x) \\frac{d}{dx} g(x) + g(x) \\frac{d}{dx} f(x) && \\textrm{Product rule} \\\\ \\frac{d}{dx} \\frac{f(x)}{g(x)} & = \\frac{g(x) \\frac{d}{dx} f(x) - f(x) \\frac{d}{dx} g(x)}{g^2(x)} && \\textrm{Quotient rule} \\end{aligned}$$\n",
    "\n",
    "Using this, we can apply the rules \n",
    "to find the derivative of $3 x^2 - 4x$ via\n",
    "\n",
    "$$\\frac{d}{dx} [3 x^2 - 4x] = 3 \\frac{d}{dx} x^2 - 4 \\frac{d}{dx} x = 6x - 4.$$\n",
    "\n",
    "Plugging in $x = 1$ shows that, indeed, \n",
    "the derivative equals $2$ at this location. \n",
    "We state these derivatives and rules without proof for now;\n",
    "each follows from the limit definition,\n",
    "as shown in that section,\n",
    "which also collects a longer table of common derivatives.\n",
    "Note that derivatives tell us \n",
    "the *slope* of a function \n",
    "at a particular location.\n",
    "\n",
    "We can make that slope visible by plotting the function\n",
    "$u = f(x)$ together with its tangent line $y = 2x - 3$ at $x=1$,\n",
    "where the coefficient $2$ is the slope of the tangent line.\n",
    "We use `d2l.plot`, one of a handful of matplotlib helpers\n",
    "that we define at the end of this section\n",
    "and reuse throughout the book."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "calculus-visualization-utilities-5"
   },
   "source": [
    "x = np.arange(0, 3, 0.1)\n",
    "d2l.plot(x, [f(x), 2 * x - 3], 'x', 'f(x)', legend=['f(x)', 'Tangent line (x=1)'])"
   ],
   "execution_count": 5,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"243.529353pt\" height=\"183.35625pt\" viewBox=\"0 0 243.529353 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-19T14:04:09.569942</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 183.35625 \nL 243.529353 183.35625 \nL 243.529353 0 \nL 0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 40.603125 145.8 \nL 235.903125 145.8 \nL 235.903125 7.2 \nL 40.603125 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path d=\"M 49.480398 145.8 \nL 49.480398 7.2 \n\" clip-path=\"url(#p05207cdaef)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path id=\"m1981f76cb8\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m1981f76cb8\" x=\"49.480398\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <g transform=\"translate(46.299148 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path d=\"M 110.702966 145.8 \nL 110.702966 7.2 \n\" clip-path=\"url(#p05207cdaef)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#m1981f76cb8\" x=\"110.702966\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 1 -->\n      <g transform=\"translate(107.521716 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path d=\"M 171.925535 145.8 \nL 171.925535 7.2 \n\" clip-path=\"url(#p05207cdaef)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#m1981f76cb8\" x=\"171.925535\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 2 -->\n      <g transform=\"translate(168.744285 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-32\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_7\">\n      <path d=\"M 233.148103 145.8 \nL 233.148103 7.2 \n\" clip-path=\"url(#p05207cdaef)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#m1981f76cb8\" x=\"233.148103\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 3 -->\n      <g transform=\"translate(229.966853 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \nQ 3050 2419 3304 2112 \nQ 3559 1806 3559 1356 \nQ 3559 666 3084 287 \nQ 2609 -91 1734 -91 \nQ 1441 -91 1130 -33 \nQ 819 25 488 141 \nL 488 750 \nQ 750 597 1062 519 \nQ 1375 441 1716 441 \nQ 2309 441 2620 675 \nQ 2931 909 2931 1356 \nQ 2931 1769 2642 2001 \nQ 2353 2234 1838 2234 \nL 1294 2234 \nL 1294 2753 \nL 1863 2753 \nQ 2328 2753 2575 2939 \nQ 2822 3125 2822 3475 \nQ 2822 3834 2567 4026 \nQ 2313 4219 1838 4219 \nQ 1578 4219 1281 4162 \nQ 984 4106 628 3988 \nL 628 4550 \nQ 988 4650 1302 4700 \nQ 1616 4750 1894 4750 \nQ 2613 4750 3031 4423 \nQ 3450 4097 3450 3541 \nQ 3450 3153 3228 2886 \nQ 3006 2619 2597 2516 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-33\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_5\">\n     <!-- x -->\n     <g transform=\"translate(135.29375 174.076563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \nL 2247 1797 \nL 3578 0 \nL 2900 0 \nL 1881 1375 \nL 863 0 \nL 184 0 \nL 1544 1831 \nL 300 3500 \nL 978 3500 \nL 1906 2253 \nL 2834 3500 \nL 3513 3500 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-78\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_9\">\n      <path d=\"M 40.603125 116.769998 \nL 235.903125 116.769998 \n\" clip-path=\"url(#p05207cdaef)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_10\">\n      <defs>\n       <path id=\"m48336712b0\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m48336712b0\" x=\"40.603125\" y=\"116.769998\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 0 -->\n      <g transform=\"translate(27.240625 120.569217) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_11\">\n      <path d=\"M 40.603125 78.886661 \nL 235.903125 78.886661 \n\" clip-path=\"url(#p05207cdaef)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_12\">\n      <g>\n       <use xlink:href=\"#m48336712b0\" x=\"40.603125\" y=\"78.886661\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_7\">\n      <!-- 5 -->\n      <g transform=\"translate(27.240625 82.68588) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \nL 3169 4666 \nL 3169 4134 \nL 1269 4134 \nL 1269 2991 \nQ 1406 3038 1543 3061 \nQ 1681 3084 1819 3084 \nQ 2600 3084 3056 2656 \nQ 3513 2228 3513 1497 \nQ 3513 744 3044 326 \nQ 2575 -91 1722 -91 \nQ 1428 -91 1123 -41 \nQ 819 9 494 109 \nL 494 744 \nQ 775 591 1075 516 \nQ 1375 441 1709 441 \nQ 2250 441 2565 725 \nQ 2881 1009 2881 1497 \nQ 2881 1984 2565 2268 \nQ 2250 2553 1709 2553 \nQ 1456 2553 1204 2497 \nQ 953 2441 691 2322 \nL 691 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-35\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_13\">\n      <path d=\"M 40.603125 41.003325 \nL 235.903125 41.003325 \n\" clip-path=\"url(#p05207cdaef)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_14\">\n      <g>\n       <use xlink:href=\"#m48336712b0\" x=\"40.603125\" y=\"41.003325\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 10 -->\n      <g transform=\"translate(20.878125 44.802544) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_9\">\n     <!-- f(x) -->\n     <g transform=\"translate(14.798437 85.121094) rotate(-90) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \nL 2375 4384 \nL 1825 4384 \nQ 1516 4384 1395 4259 \nQ 1275 4134 1275 3809 \nL 1275 3500 \nL 2222 3500 \nL 2222 3053 \nL 1275 3053 \nL 1275 0 \nL 697 0 \nL 697 3053 \nL 147 3053 \nL 147 3500 \nL 697 3500 \nL 697 3744 \nQ 697 4328 969 4595 \nQ 1241 4863 1831 4863 \nL 2375 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \nQ 1566 4138 1362 3434 \nQ 1159 2731 1159 2009 \nQ 1159 1288 1364 580 \nQ 1569 -128 1984 -844 \nL 1484 -844 \nQ 1016 -109 783 600 \nQ 550 1309 550 2009 \nQ 550 2706 781 3412 \nQ 1013 4119 1484 4856 \nL 1984 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \nL 1013 4856 \nQ 1481 4119 1714 3412 \nQ 1947 2706 1947 2009 \nQ 1947 1309 1714 600 \nQ 1481 -109 1013 -844 \nL 513 -844 \nQ 928 -128 1133 580 \nQ 1338 1288 1338 2009 \nQ 1338 2731 1133 3434 \nQ 928 4138 513 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-66\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(35.205078 0)\"/>\n      <use xlink:href=\"#DejaVuSans-78\" transform=\"translate(74.21875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(133.398438 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_15\">\n    <path d=\"M 49.480398 116.769998 \nL 55.602655 119.573365 \nL 61.724912 121.922132 \nL 67.847169 123.816299 \nL 73.969426 125.255865 \nL 80.091682 126.240832 \nL 86.21394 126.771199 \nL 92.336195 126.846966 \nL 98.458453 126.468132 \nL 104.580712 125.634698 \nL 110.702966 124.346665 \nL 116.825225 122.604032 \nL 122.947483 120.406799 \nL 129.069741 117.754962 \nL 135.191992 114.648533 \nL 141.314251 111.087498 \nL 147.436509 107.071862 \nL 153.558767 102.601631 \nL 159.681025 97.676789 \nL 165.803276 92.297362 \nL 171.925535 86.463329 \nL 178.0478 80.174681 \nL 184.170051 73.431459 \nL 190.292302 66.233626 \nL 196.414568 58.581191 \nL 202.536819 50.474159 \nL 208.659085 41.91251 \nL 214.781336 32.896286 \nL 220.903587 23.425466 \nL 227.025852 13.5 \n\" clip-path=\"url(#p05207cdaef)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_16\">\n    <path d=\"M 49.480398 139.5 \nL 55.602655 137.984666 \nL 61.724912 136.469332 \nL 67.847169 134.954 \nL 73.969426 133.438667 \nL 80.091682 131.923333 \nL 86.21394 130.407999 \nL 92.336195 128.892666 \nL 98.458453 127.377332 \nL 104.580712 125.861998 \nL 110.702966 124.346665 \nL 116.825225 122.831332 \nL 122.947483 121.315998 \nL 129.069741 119.800664 \nL 135.191992 118.285332 \nL 141.314251 116.769998 \nL 147.436509 115.254664 \nL 153.558767 113.73933 \nL 159.681025 112.223997 \nL 165.803276 110.708665 \nL 171.925535 109.193331 \nL 178.0478 107.677995 \nL 184.170051 106.162663 \nL 190.292302 104.647331 \nL 196.414568 103.131995 \nL 202.536819 101.616663 \nL 208.659085 100.101328 \nL 214.781336 98.585996 \nL 220.903587 97.070664 \nL 227.025852 95.555328 \n\" clip-path=\"url(#p05207cdaef)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 40.603125 145.8 \nL 40.603125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 235.903125 145.8 \nL 235.903125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 40.603125 145.8 \nL 235.903125 145.8 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 40.603125 7.2 \nL 235.903125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 47.603125 44.55625 \nL 172.153125 44.55625 \nQ 174.153125 44.55625 174.153125 42.55625 \nL 174.153125 14.2 \nQ 174.153125 12.2 172.153125 12.2 \nL 47.603125 12.2 \nQ 45.603125 12.2 45.603125 14.2 \nL 45.603125 42.55625 \nQ 45.603125 44.55625 47.603125 44.55625 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_17\">\n     <path d=\"M 49.603125 20.298438 \nL 59.603125 20.298438 \nL 69.603125 20.298438 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_10\">\n     <!-- f(x) -->\n     <g transform=\"translate(77.603125 23.798438) scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-66\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(35.205078 0)\"/>\n      <use xlink:href=\"#DejaVuSans-78\" transform=\"translate(74.21875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(133.398438 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_18\">\n     <path d=\"M 49.603125 34.976562 \nL 59.603125 34.976562 \nL 69.603125 34.976562 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_11\">\n     <!-- Tangent line (x=1) -->\n     <g transform=\"translate(77.603125 38.476562) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-54\" d=\"M -19 4666 \nL 3928 4666 \nL 3928 4134 \nL 2272 4134 \nL 2272 0 \nL 1638 0 \nL 1638 4134 \nL -19 4134 \nL -19 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \nQ 2906 2416 2648 2759 \nQ 2391 3103 1925 3103 \nQ 1463 3103 1205 2759 \nQ 947 2416 947 1791 \nQ 947 1169 1205 825 \nQ 1463 481 1925 481 \nQ 2391 481 2648 825 \nQ 2906 1169 2906 1791 \nz\nM 3481 434 \nQ 3481 -459 3084 -895 \nQ 2688 -1331 1869 -1331 \nQ 1566 -1331 1297 -1286 \nQ 1028 -1241 775 -1147 \nL 775 -588 \nQ 1028 -725 1275 -790 \nQ 1522 -856 1778 -856 \nQ 2344 -856 2625 -561 \nQ 2906 -266 2906 331 \nL 2906 616 \nQ 2728 306 2450 153 \nQ 2172 0 1784 0 \nQ 1141 0 747 490 \nQ 353 981 353 1791 \nQ 353 2603 747 3093 \nQ 1141 3584 1784 3584 \nQ 2172 3584 2450 3431 \nQ 2728 3278 2906 2969 \nL 2906 3500 \nL 3481 3500 \nL 3481 434 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-3d\" d=\"M 678 2906 \nL 4684 2906 \nL 4684 2381 \nL 678 2381 \nL 678 2906 \nz\nM 678 1631 \nL 4684 1631 \nL 4684 1100 \nL 678 1100 \nL 678 1631 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-54\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(44.583984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(105.863281 0)\"/>\n      <use xlink:href=\"#DejaVuSans-67\" transform=\"translate(169.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(232.71875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(294.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(357.621094 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(396.830078 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(428.617188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(456.400391 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(484.183594 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(547.5625 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(609.085938 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(640.873047 0)\"/>\n      <use xlink:href=\"#DejaVuSans-78\" transform=\"translate(679.886719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-3d\" transform=\"translate(739.066406 0)\"/>\n      <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(822.855469 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(886.478516 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p05207cdaef\">\n   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=calculus-md-6 fw=all -->\n",
    "\n",
    "## Partial Derivatives and Gradients\n",
    "\n",
    "Thus far, we have been differentiating\n",
    "functions of just one variable.\n",
    "In deep learning, we also need to work\n",
    "with functions of *many* variables, typically functions that take\n",
    "the vectors and matrices from that section\n",
    "as their inputs.\n",
    "We briefly introduce notions of the derivative\n",
    "that apply to such *multivariate* functions.\n",
    "\n",
    "\n",
    "Let $y = f(x_1, x_2, \\ldots, x_n)$ be a function with $n$ variables. \n",
    "The *partial derivative* of $y$ \n",
    "with respect to its $i^\\textrm{th}$ parameter $x_i$ is\n",
    "\n",
    "$$ \\frac{\\partial y}{\\partial x_i} = \\lim_{h \\rightarrow 0} \\frac{f(x_1, \\ldots, x_{i-1}, x_i+h, x_{i+1}, \\ldots, x_n) - f(x_1, \\ldots, x_i, \\ldots, x_n)}{h}.$$\n",
    "\n",
    "\n",
    "To calculate $\\frac{\\partial y}{\\partial x_i}$, \n",
    "we can treat $x_1, \\ldots, x_{i-1}, x_{i+1}, \\ldots, x_n$ as constants \n",
    "and calculate the derivative of $y$ with respect to $x_i$.\n",
    "The following notational conventions for partial derivatives \n",
    "are all common and all mean the same thing:\n",
    "\n",
    "$$\\frac{\\partial y}{\\partial x_i} = \\frac{\\partial f}{\\partial x_i} = \\partial_{x_i} f = \\partial_i f = f_{x_i} = f_i = D_i f = D_{x_i} f.$$\n",
    "\n",
    "We can concatenate partial derivatives \n",
    "of a multivariate function \n",
    "with respect to all its variables \n",
    "to obtain a vector that is called\n",
    "the *gradient* of the function.\n",
    "Suppose that the input of function \n",
    "$f: \\mathbb{R}^n \\rightarrow \\mathbb{R}$ \n",
    "is an $n$-dimensional vector \n",
    "$\\mathbf{x} = [x_1, x_2, \\ldots, x_n]^\\top$ \n",
    "and the output is a scalar. \n",
    "The gradient of the function $f$ \n",
    "with respect to $\\mathbf{x}$ \n",
    "is a vector of $n$ partial derivatives:\n",
    "\n",
    "$$\\nabla_{\\mathbf{x}} f(\\mathbf{x}) = \\left[\\partial_{x_1} f(\\mathbf{x}), \\partial_{x_2} f(\\mathbf{x}), \\ldots,\n",
    "\\partial_{x_n} f(\\mathbf{x})\\right]^\\top.$$ \n",
    "\n",
    "When there is no ambiguity,\n",
    "$\\nabla_{\\mathbf{x}} f(\\mathbf{x})$ \n",
    "is typically replaced \n",
    "by $\\nabla f(\\mathbf{x})$.\n",
    "\n",
    "What if we move away from $\\mathbf{x}$\n",
    "in an arbitrary unit direction $\\mathbf{u}$,\n",
    "rather than along a coordinate axis?\n",
    "The rate of change of $f$ along $\\mathbf{u}$\n",
    "is called the *directional derivative*;\n",
    "for differentiable $f$ it equals the dot product\n",
    "$\\nabla f(\\mathbf{x})^\\top \\mathbf{u}$,\n",
    "an identity shown in that section.\n",
    "By the cosine formula for dot products\n",
    "from that section,\n",
    "this is largest when $\\mathbf{u}$ aligns with the gradient.\n",
    "\n",
    "Hence the gradient has a geometric interpretation:\n",
    "it points in the direction of *steepest ascent* of $f$,\n",
    "i.e., the direction in which the function grows fastest\n",
    "(a claim proved via the Cauchy--Schwarz inequality\n",
    "in that section).\n",
    "To *decrease* $f$ as quickly as possible\n",
    "we therefore take a small step in the opposite direction,\n",
    "along the *negative* gradient $-\\nabla f(\\mathbf{x})$.\n",
    "This is the idea behind *gradient descent*, which takes a small step along\n",
    "$-\\nabla f$. Later optimizers modify this direction using momentum,\n",
    "coordinatewise scaling, or curvature information, but the gradient remains\n",
    "the local signal from which their updates are constructed.\n",
    "the figure shows the picture to keep in mind:\n",
    "gradients are perpendicular to the *level sets* of $f$\n",
    "(the curves along which $f$ is constant)\n",
    "and point uphill, so $-\\nabla f$ points downhill.\n",
    "\n",
    "![Level sets of a function with its gradient field: $\\nabla f$ is perpendicular to the contours and points uphill, so $-\\nabla f$ is the direction of steepest descent.](../img/calculus-gradient-field.svg)\n",
    "\n",
    "The following rules for differentiating \n",
    "multivariate functions recur throughout the book.\n",
    "The first involves a *vector-valued* function\n",
    "$\\mathbf{u} = \\mathbf{A}\\mathbf{x}$:\n",
    "here we collect the partial derivatives\n",
    "$\\partial u_j / \\partial x_i$ into a matrix,\n",
    "and by convention we write\n",
    "$\\nabla_{\\mathbf{x}} \\mathbf{A} \\mathbf{x} = \\mathbf{A}^\\top$\n",
    "(the transpose of the *Jacobian*,\n",
    "developed in that section).\n",
    "\n",
    "* For $\\mathbf{A} \\in \\mathbb{R}^{m \\times n}$ we have $\\nabla_{\\mathbf{x}} \\mathbf{A} \\mathbf{x} = \\mathbf{A}^\\top$; for $\\mathbf{A} \\in \\mathbb{R}^{n \\times m}$ we have $\\nabla_{\\mathbf{x}} \\mathbf{x}^\\top \\mathbf{A}  = \\mathbf{A}$.\n",
    "* For square matrices $\\mathbf{A} \\in \\mathbb{R}^{n \\times n}$ we have that $\\nabla_{\\mathbf{x}} \\mathbf{x}^\\top \\mathbf{A} \\mathbf{x}  = (\\mathbf{A} + \\mathbf{A}^\\top)\\mathbf{x}$ and in particular\n",
    "$\\nabla_{\\mathbf{x}} \\|\\mathbf{x} \\|^2 = \\nabla_{\\mathbf{x}} \\mathbf{x}^\\top \\mathbf{x} = 2\\mathbf{x}$.\n",
    "\n",
    "Similarly, for any matrix $\\mathbf{X}$, \n",
    "we have $\\nabla_{\\mathbf{X}} \\|\\mathbf{X} \\|_\\textrm{F}^2 = 2\\mathbf{X}$. \n",
    "These identities (and many more) are derived\n",
    "in that section.\n",
    "\n",
    "\n",
    "## Chain Rule\n",
    "\n",
    "In deep learning, the gradients of concern\n",
    "are often difficult to calculate\n",
    "because we are working with \n",
    "deeply nested functions \n",
    "(of functions (of functions...)).\n",
    "Fortunately, the *chain rule* takes care of this. \n",
    "Returning to functions of a single variable,\n",
    "suppose that $y = f(g(x))$\n",
    "and that the underlying functions \n",
    "$y=f(u)$ and $u=g(x)$ \n",
    "are both differentiable.\n",
    "The chain rule states that \n",
    "\n",
    "\n",
    "$$\\frac{dy}{dx} = \\frac{dy}{du} \\frac{du}{dx}.$$\n",
    "\n",
    "\n",
    "Turning back to multivariate functions,\n",
    "suppose that $y = f(\\mathbf{u})$ has variables\n",
    "$u_1, u_2, \\ldots, u_m$, \n",
    "where each $u_i = g_i(\\mathbf{x})$ \n",
    "has variables $x_1, x_2, \\ldots, x_n$,\n",
    "i.e.,  $\\mathbf{u} = g(\\mathbf{x})$.\n",
    "Then the chain rule states that\n",
    "\n",
    "$$\\frac{\\partial y}{\\partial x_{i}} = \\frac{\\partial y}{\\partial u_{1}} \\frac{\\partial u_{1}}{\\partial x_{i}} + \\frac{\\partial y}{\\partial u_{2}} \\frac{\\partial u_{2}}{\\partial x_{i}} + \\ldots + \\frac{\\partial y}{\\partial u_{m}} \\frac{\\partial u_{m}}{\\partial x_{i}} \\ \\textrm{ and so } \\ \\nabla_{\\mathbf{x}} y =  \\mathbf{A} \\nabla_{\\mathbf{u}} y,$$\n",
    "\n",
    "where $\\mathbf{A} \\in \\mathbb{R}^{n \\times m}$ is the *matrix*\n",
    "whose entry $A_{ij} = \\partial u_j / \\partial x_i$ collects\n",
    "the derivatives of the components of $\\mathbf{u}$\n",
    "with respect to those of $\\mathbf{x}$.\n",
    "Reading off the sum on the left,\n",
    "$(\\nabla_{\\mathbf{x}} y)_i = \\sum_{j} A_{ij} (\\nabla_{\\mathbf{u}} y)_j$,\n",
    "which is exactly the matrix--vector product on the right.\n",
    "Thus, evaluating the gradient requires \n",
    "computing a matrix--vector product. \n",
    "This is one reason why linear algebra \n",
    "is a prerequisite for deep learning. \n",
    "\n",
    "### Plotting Utilities for This Book\n",
    "\n",
    "Earlier we plotted a function and its tangent line with `d2l.plot`.\n",
    "Here is the small set of matplotlib helpers behind it,\n",
    "which the book reuses whenever it draws a curve.\n",
    "The `#@save` comment is a special modifier that stores a function,\n",
    "class, or code block in the `d2l` package,\n",
    "so that later sections can invoke it\n",
    "(e.g., as `d2l.plot`) without repeating the code.\n",
    "`use_svg_display` requests crisp SVG output,\n",
    "`set_figsize` sets the figure size,\n",
    "and `set_axes` configures labels, ranges, and scales.\n",
    "You do not need to study these; skim and move on."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "calculus-visualization-utilities-1"
   },
   "source": [
    "def use_svg_display():\n",
    "    \"\"\"Use the svg format to display a plot in Jupyter.\"\"\"\n",
    "    backend_inline.set_matplotlib_formats('svg')"
   ],
   "execution_count": 6,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "calculus-visualization-utilities-2"
   },
   "source": [
    "def set_figsize(figsize=(3.5, 2.5)):\n",
    "    \"\"\"Set the figure size for matplotlib.\"\"\"\n",
    "    use_svg_display()\n",
    "    d2l.plt.rcParams['figure.figsize'] = figsize"
   ],
   "execution_count": 7,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "calculus-visualization-utilities-3"
   },
   "source": [
    "\n",
    "def set_axes(axes, xlabel, ylabel, xlim, ylim, xscale, yscale, legend):\n",
    "    \"\"\"Set the axes for matplotlib.\"\"\"\n",
    "    axes.set_xlabel(xlabel), axes.set_ylabel(ylabel)\n",
    "    axes.set_xscale(xscale), axes.set_yscale(yscale)\n",
    "    axes.set_xlim(xlim),     axes.set_ylim(ylim)\n",
    "    if legend:\n",
    "        axes.legend(legend)\n",
    "    axes.grid()"
   ],
   "execution_count": 8,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=calculus-md-7 fw=all -->\n",
    "\n",
    "Finally, `plot` overlays multiple curves;\n",
    "most of its body just aligns the shapes of its inputs."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "calculus-visualization-utilities-4"
   },
   "source": [
    "\n",
    "def plot(X, Y=None, xlabel=None, ylabel=None, legend=None, xlim=None,\n",
    "         ylim=None, xscale='linear', yscale='linear',\n",
    "         fmts=('-', 'm--', 'g-.', 'r:'), figsize=(3.5, 2.5), axes=None):\n",
    "    \"\"\"Plot data points.\"\"\"\n",
    "    legend = [] if legend is None else legend\n",
    "\n",
    "    def has_one_axis(X):  # True if X (tensor or list) has 1 axis\n",
    "        return (hasattr(X, \"ndim\") and X.ndim == 1 or isinstance(X, list)\n",
    "                and not hasattr(X[0], \"__len__\"))\n",
    "    \n",
    "    if has_one_axis(X): X = [X]\n",
    "    if Y is None:\n",
    "        X, Y = [[]] * len(X), X\n",
    "    elif has_one_axis(Y):\n",
    "        Y = [Y]\n",
    "    if len(X) != len(Y):\n",
    "        X = X * len(Y)\n",
    "        \n",
    "    set_figsize(figsize)\n",
    "    if axes is None:\n",
    "        axes = d2l.plt.gca()\n",
    "    axes.cla()\n",
    "    for x, y, fmt in zip(X, Y, fmts):\n",
    "        axes.plot(x,y,fmt) if len(x) else axes.plot(y,fmt)\n",
    "    set_axes(axes, xlabel, ylabel, xlim, ylim, xscale, yscale, legend)"
   ],
   "execution_count": 9,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=calculus-md-8 fw=all -->\n",
    "\n",
    "## Discussion\n",
    "\n",
    "Three ideas from this section recur throughout the book.\n",
    "First, and most important for what follows:\n",
    "from the viewpoint of optimization,\n",
    "the gradient tells us how to move the parameters of a model\n",
    "in order to lower the loss.\n",
    "Because $-\\nabla f$ points in the direction of steepest *descent*,\n",
    "each step of the optimization algorithms used throughout this book\n",
    "amounts to evaluating the gradient\n",
    "and taking a short step along $-\\nabla f$.\n",
    "Second, the composition rules for differentiation\n",
    "can be applied routinely, enabling\n",
    "us to compute gradients *automatically*;\n",
    "this task requires no creativity, so\n",
    "we can focus our cognitive powers elsewhere.\n",
    "Third, computing the derivatives of vector-valued functions \n",
    "requires us to multiply matrices as we trace \n",
    "the dependency graph of variables from output to input. \n",
    "Applying the chain rule backward through that graph\n",
    "is *backpropagation*, automated in that section\n",
    "and developed in that section.\n",
    "\n",
    "This section deliberately previews only the calculus we need\n",
    "to train models.\n",
    "For a thorough development (single- and multivariable calculus,\n",
    "the integral, and the matrix calculus behind automatic\n",
    "differentiation), see that section;\n",
    "@Deisenroth.Faisal.Ong.2020 give a complementary treatment\n",
    "aimed at machine learning.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. So far we took the rules for derivatives for granted. \n",
    "   Using the definition and limits prove the properties \n",
    "   for (i) $f(x) = c$, (ii) $f(x) = x^n$, (iii) $f(x) = e^x$ and (iv) $f(x) = \\log x$.\n",
    "1. In the same vein, prove the product, sum, and quotient rule from first principles. \n",
    "1. Prove that the constant multiple rule follows as a special case of the product rule. \n",
    "1. Calculate the derivative of $f(x) = x^x$. \n",
    "1. What does it mean that $f'(x) = 0$ for some $x$? \n",
    "   Give an example of a function $f$ \n",
    "   and a location $x$ for which this might hold. \n",
    "1. Plot the function $y = f(x) = x^3 - \\frac{1}{x}$ \n",
    "   and plot its tangent line at $x = 1$.\n",
    "1. Find the gradient of the function \n",
    "   $f(\\mathbf{x}) = 3x_1^2 + 5e^{x_2}$.\n",
    "1. What is the gradient of the function \n",
    "   $f(\\mathbf{x}) = \\|\\mathbf{x}\\|_2$? What happens for $\\mathbf{x} = \\mathbf{0}$?\n",
    "1. Can you write out the chain rule for the case \n",
    "   where $u = f(x, y, z)$ and $x = x(a, b)$, $y = y(a, b)$, and $z = z(a, b)$?\n",
    "1. Given a function $f(x)$ that is invertible, \n",
    "   compute the derivative of its inverse $f^{-1}(x)$. \n",
    "   Here we have that $f^{-1}(f(x)) = x$ and conversely $f(f^{-1}(y)) = y$. \n",
    "   Hint: use these properties in your derivation. \n",
    "1. Consider the function $f(\\mathbf{x}) = \\|\\mathbf{x}\\|_2^2$. \n",
    "   Starting from $\\mathbf{x} = [1, 1]^\\top$, take a single gradient-descent step \n",
    "   $\\mathbf{x} \\leftarrow \\mathbf{x} - \\eta \\nabla f(\\mathbf{x})$ \n",
    "   with learning rate $\\eta = 0.1$ and verify that $f$ decreases. \n",
    "   What happens if you instead pick $\\eta = 1$? And $\\eta = 2$? \n",
    "   What does this tell you about the role of the learning rate?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=calculus-md-9 fw=mxnet -->\n",
    "\n",
    "[Discussions](https://d2l.discourse.group/t/32)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-mxnet",
   "display_name": "d2l (MXNet)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
