{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-maximum-likelihood-md-1 fw=all -->\n",
    "\n",
    "# Maximum Likelihood\n",
    "\n",
    "Many standard machine-learning losses have a probabilistic interpretation. When\n",
    "we minimize a classifier's cross-entropy or a regressor's squared error under a\n",
    "Gaussian noise model, we are answering one question: *which parameters make the\n",
    "observed data most probable?* That is the **maximum likelihood** principle, the idea\n",
    "that turns a probabilistic model into a trainable objective. This section\n",
    "states the principle and proves the three equivalences that make it\n",
    "operational: likelihood becomes a *negative log-likelihood*, the negative\n",
    "log-likelihood *is* the cross-entropy to the data, and the Gaussian case *is*\n",
    "mean squared error. It then asks how good the resulting estimates are\n",
    "(consistency, asymptotic normality, and the Fisher information) and shows how\n",
    "putting a prior back in recovers $L_2$ regularization. The section ends where\n",
    "modern generative models begin: when latent variables make the likelihood\n",
    "intractable, a lower bound (the ELBO) and the EM algorithm make it tractable\n",
    "again. We keep the running coin-flip example throughout because it is small\n",
    "enough to solve by hand, which lets us check every claim against the answer we\n",
    "already know."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "maximum-likelihood-imports"
   },
   "source": [
    "%matplotlib inline\n",
    "from d2l import torch as d2l\n",
    "import numpy as onp\n",
    "import torch"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-maximum-likelihood-md-2 fw=all -->\n",
    "\n",
    "## The Maximum Likelihood Principle\n",
    "\n",
    "Suppose we have a model with parameters $\\boldsymbol{\\theta}$ and observed\n",
    "data $X$. The **likelihood** is the model probability or density of the observed\n",
    "data, viewed as a function of the parameter:\n",
    "\n",
    "$$\n",
    "L(\\boldsymbol{\\theta};X)=P(X\\mid\\boldsymbol{\\theta}).\n",
    "$$\n",
    "\n",
    "Maximum likelihood is the direct rule\n",
    "\n",
    "$$\n",
    "\\hat{\\boldsymbol{\\theta}}_{\\textrm{MLE}}\n",
    " = \\mathop{\\mathrm{argmax}}_{\\boldsymbol{\\theta}}\n",
    "   P(X \\mid \\boldsymbol{\\theta}).\n",
    "$$\n",
    "\n",
    "No prior is part of this definition. MLE asks which member of the chosen model\n",
    "family fits the observations best; it does not assign probabilities to the\n",
    "parameters themselves. Later, that section introduces the distinct\n",
    "**maximum a posteriori** rule\n",
    "$\\operatorname*{argmax}_{\\boldsymbol{\\theta}}P(\\boldsymbol{\\theta}\\mid X)$, which multiplies\n",
    "the likelihood by a prior density. A prior that is constant in the chosen\n",
    "parameterization makes the two optimizers coincide, but “flat” is not invariant\n",
    "under nonlinear reparameterization and is not the foundation of maximum\n",
    "likelihood.\n",
    "\n",
    "### A Worked Example: The Coin\n",
    "\n",
    "Take a single parameter $\\theta$, the probability of heads, so tails has\n",
    "probability $1-\\theta$. Because the flips are independent, their probabilities\n",
    "multiply: if $X$ has $n_H$ heads and $n_T$ tails,\n",
    "\n",
    "$$\n",
    "P(X \\mid \\theta) = \\theta^{n_H}(1-\\theta)^{n_T}.\n",
    "$$\n",
    "\n",
    "Flip $13$ coins and observe the sequence \"HHHTHTTHHHHHT\", which has $n_H=9$ and\n",
    "$n_T=4$, giving the likelihood $P(X \\mid \\theta) = \\theta^9(1-\\theta)^4$. We know\n",
    "the answer we *want*: asked \"$9$ of $13$ flips were heads, what is the bias?\",\n",
    "everyone replies $9/13$. The value of the maximum-likelihood machinery is that it\n",
    "*derives* that number from first principles, in a way that scales to models with\n",
    "billions of parameters where no intuition is available.\n",
    "\n",
    "The likelihood is a function of $\\theta$ alone, so we can plot it and read off\n",
    "its peak."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "maximum-likelihood-a-concrete-example"
   },
   "source": [
    "import numpy as onp\n",
    "theta = onp.arange(0, 1, 0.001)\n",
    "p = theta**9 * (1 - theta)**4.\n",
    "\n",
    "d2l.plot(theta, p, 'theta', 'likelihood')"
   ],
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"265.36875pt\" height=\"183.35625pt\" viewBox=\"0 0 265.36875 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-19T13:42:38.882556</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 183.35625 \nL 265.36875 183.35625 \nL 265.36875 0 \nL 0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 62.86875 145.8 \nL 258.16875 145.8 \nL 258.16875 7.2 \nL 62.86875 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path d=\"M 71.746023 145.8 \nL 71.746023 7.2 \n\" clip-path=\"url(#p7b26ad792f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path id=\"m6ea1f4e609\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m6ea1f4e609\" x=\"71.746023\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0.0 -->\n      <g transform=\"translate(63.79446 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \nL 1344 794 \nL 1344 0 \nL 684 0 \nL 684 794 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path d=\"M 107.290658 145.8 \nL 107.290658 7.2 \n\" clip-path=\"url(#p7b26ad792f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#m6ea1f4e609\" x=\"107.290658\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 0.2 -->\n      <g transform=\"translate(99.339096 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path d=\"M 142.835294 145.8 \nL 142.835294 7.2 \n\" clip-path=\"url(#p7b26ad792f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#m6ea1f4e609\" x=\"142.835294\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 0.4 -->\n      <g transform=\"translate(134.883731 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_7\">\n      <path d=\"M 178.379929 145.8 \nL 178.379929 7.2 \n\" clip-path=\"url(#p7b26ad792f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#m6ea1f4e609\" x=\"178.379929\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 0.6 -->\n      <g transform=\"translate(170.428367 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-36\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_9\">\n      <path d=\"M 213.924565 145.8 \nL 213.924565 7.2 \n\" clip-path=\"url(#p7b26ad792f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#m6ea1f4e609\" x=\"213.924565\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 0.8 -->\n      <g transform=\"translate(205.973002 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-38\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_11\">\n      <path d=\"M 249.4692 145.8 \nL 249.4692 7.2 \n\" clip-path=\"url(#p7b26ad792f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_12\">\n      <g>\n       <use xlink:href=\"#m6ea1f4e609\" x=\"249.4692\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 1.0 -->\n      <g transform=\"translate(241.517638 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- theta -->\n     <g transform=\"translate(147.289062 174.076563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(39.208984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(102.587891 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(164.111328 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(203.320312 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_13\">\n      <path d=\"M 62.86875 139.5 \nL 258.16875 139.5 \n\" clip-path=\"url(#p7b26ad792f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_14\">\n      <defs>\n       <path id=\"mf63599bc17\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#mf63599bc17\" x=\"62.86875\" y=\"139.5\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.0000 -->\n      <g transform=\"translate(20.878125 143.299219) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(159.033203 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(222.65625 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(286.279297 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_15\">\n      <path d=\"M 62.86875 101.02197 \nL 258.16875 101.02197 \n\" clip-path=\"url(#p7b26ad792f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_16\">\n      <g>\n       <use xlink:href=\"#mf63599bc17\" x=\"62.86875\" y=\"101.02197\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.0001 -->\n      <g transform=\"translate(20.878125 104.821189) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(159.033203 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(222.65625 0)\"/>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(286.279297 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_17\">\n      <path d=\"M 62.86875 62.54394 \nL 258.16875 62.54394 \n\" clip-path=\"url(#p7b26ad792f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_18\">\n      <g>\n       <use xlink:href=\"#mf63599bc17\" x=\"62.86875\" y=\"62.54394\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 0.0002 -->\n      <g transform=\"translate(20.878125 66.343159) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(159.033203 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(222.65625 0)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(286.279297 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_19\">\n      <path d=\"M 62.86875 24.06591 \nL 258.16875 24.06591 \n\" clip-path=\"url(#p7b26ad792f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_20\">\n      <g>\n       <use xlink:href=\"#mf63599bc17\" x=\"62.86875\" y=\"24.06591\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 0.0003 -->\n      <g transform=\"translate(20.878125 27.865129) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \nQ 3050 2419 3304 2112 \nQ 3559 1806 3559 1356 \nQ 3559 666 3084 287 \nQ 2609 -91 1734 -91 \nQ 1441 -91 1130 -33 \nQ 819 25 488 141 \nL 488 750 \nQ 750 597 1062 519 \nQ 1375 441 1716 441 \nQ 2309 441 2620 675 \nQ 2931 909 2931 1356 \nQ 2931 1769 2642 2001 \nQ 2353 2234 1838 2234 \nL 1294 2234 \nL 1294 2753 \nL 1863 2753 \nQ 2328 2753 2575 2939 \nQ 2822 3125 2822 3475 \nQ 2822 3834 2567 4026 \nQ 2313 4219 1838 4219 \nQ 1578 4219 1281 4162 \nQ 984 4106 628 3988 \nL 628 4550 \nQ 988 4650 1302 4700 \nQ 1616 4750 1894 4750 \nQ 2613 4750 3031 4423 \nQ 3450 4097 3450 3541 \nQ 3450 3153 3228 2886 \nQ 3006 2619 2597 2516 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(159.033203 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(222.65625 0)\"/>\n       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(286.279297 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_12\">\n     <!-- likelihood -->\n     <g transform=\"translate(14.798437 100.308594) rotate(-90) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6b\" d=\"M 581 4863 \nL 1159 4863 \nL 1159 1991 \nL 2875 3500 \nL 3609 3500 \nL 1753 1863 \nL 3688 0 \nL 2938 0 \nL 1159 1709 \nL 1159 0 \nL 581 0 \nL 581 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \nL 2906 4863 \nL 3481 4863 \nL 3481 0 \nL 2906 0 \nL 2906 525 \nQ 2725 213 2448 61 \nQ 2172 -91 1784 -91 \nQ 1150 -91 751 415 \nQ 353 922 353 1747 \nQ 353 2572 751 3078 \nQ 1150 3584 1784 3584 \nQ 2172 3584 2448 3432 \nQ 2725 3281 2906 2969 \nz\nM 947 1747 \nQ 947 1113 1208 752 \nQ 1469 391 1925 391 \nQ 2381 391 2643 752 \nQ 2906 1113 2906 1747 \nQ 2906 2381 2643 2742 \nQ 2381 3103 1925 3103 \nQ 1469 3103 1208 2742 \nQ 947 2381 947 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-6c\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(27.783203 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6b\" transform=\"translate(55.566406 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(109.851562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(171.375 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(199.158203 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(226.941406 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(290.320312 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(351.501953 0)\"/>\n      <use xlink:href=\"#DejaVuSans-64\" transform=\"translate(412.683594 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_21\">\n    <path d=\"M 71.746023 139.5 \nL 108.712444 139.389673 \nL 114.755032 139.1384 \nL 119.020388 138.755385 \nL 122.574852 138.219735 \nL 125.596146 137.545086 \nL 128.261993 136.731819 \nL 130.750118 135.745419 \nL 133.060519 134.596007 \nL 135.37092 133.186064 \nL 137.503598 131.622461 \nL 139.636277 129.778417 \nL 141.768955 127.626758 \nL 143.901633 125.142034 \nL 146.034311 122.301471 \nL 148.166989 119.085989 \nL 150.299667 115.48127 \nL 152.610069 111.127252 \nL 154.92047 106.305049 \nL 157.408594 100.599497 \nL 160.074442 93.928611 \nL 163.095736 85.74412 \nL 166.6502 75.433971 \nL 172.337341 58.119636 \nL 177.31359 43.251279 \nL 180.157161 35.402626 \nL 182.467562 29.606843 \nL 184.600241 24.856928 \nL 186.377472 21.423445 \nL 187.976981 18.797307 \nL 189.398766 16.868643 \nL 190.642829 15.51765 \nL 191.886891 14.499102 \nL 192.95323 13.902591 \nL 194.019569 13.570596 \nL 194.908185 13.50148 \nL 195.796801 13.625023 \nL 196.685417 13.944266 \nL 197.751756 14.588938 \nL 198.818095 15.521424 \nL 199.884434 16.742786 \nL 201.128496 18.5319 \nL 202.372558 20.709318 \nL 203.794344 23.663708 \nL 205.393852 27.561336 \nL 207.171084 32.566102 \nL 209.126039 38.822667 \nL 211.258717 46.433458 \nL 213.746842 56.147607 \nL 217.301305 71.032851 \nL 223.69934 97.968284 \nL 226.365187 108.119525 \nL 228.497865 115.403269 \nL 230.45282 121.285759 \nL 232.230052 125.901547 \nL 233.829561 129.428433 \nL 235.251346 132.058453 \nL 236.673132 134.221937 \nL 237.917194 135.748795 \nL 239.161256 136.956786 \nL 240.405318 137.874865 \nL 241.649381 138.538068 \nL 243.071166 139.03536 \nL 244.670675 139.340161 \nL 246.803353 139.482998 \nL 249.291477 139.5 \nL 249.291477 139.5 \n\" clip-path=\"url(#p7b26ad792f)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 62.86875 145.8 \nL 62.86875 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 258.16875 145.8 \nL 258.16875 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 62.86875 145.8 \nL 258.16875 145.8 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 62.86875 7.2 \nL 258.16875 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p7b26ad792f\">\n   <rect x=\"62.86875\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-maximum-likelihood-md-3 fw=all -->\n",
    "\n",
    "The curve peaks near the expected $9/13 \\approx 0.69$. To pin the maximum\n",
    "exactly we use calculus: at an interior maximum the derivative vanishes. Setting\n",
    "$\\frac{d}{d\\theta}P(X\\mid\\theta)=0$,\n",
    "\n",
    "$$\n",
    "0 = \\frac{d}{d\\theta}\\,\\theta^9(1-\\theta)^4\n",
    "  = 9\\theta^8(1-\\theta)^4 - 4\\theta^9(1-\\theta)^3\n",
    "  = \\theta^8(1-\\theta)^3(9-13\\theta).\n",
    "$$\n",
    "\n",
    "The roots $\\theta=0$ and $\\theta=1$ assign probability zero to a sequence that\n",
    "contains both heads and tails, so they are minima. The remaining root is the\n",
    "maximum likelihood estimate,\n",
    "\n",
    "$$\n",
    "\\hat\\theta = \\frac{9}{13},\n",
    "$$\n",
    "\n",
    "exactly the observed fraction of heads. The same calculation on the general\n",
    "likelihood the equation gives $\\hat\\theta = n_H/(n_H+n_T)$: the MLE\n",
    "of a coin's bias is *always* the empirical frequency of heads, our first sign\n",
    "that maximum likelihood recovers the \"obvious\" estimator.\n",
    "\n",
    "### The Negative Log-Likelihood\n",
    "\n",
    "The closed-form trick above does not extend to a real model: with billions of\n",
    "parameters there is no polynomial to factor, and we optimize numerically\n",
    "instead. But the likelihood itself is unusable in floating point. With $n$\n",
    "independent data points it is a product of $n$ probabilities,\n",
    "\n",
    "$$\n",
    "P(X\\mid\\boldsymbol{\\theta}) = \\prod_{i=1}^n p(x_i\\mid\\boldsymbol{\\theta}),\n",
    "$$\n",
    "\n",
    "each in $[0,1]$. A billion factors near $1/2$ produce roughly $(1/2)^{10^9}$,\n",
    "underflowing to zero in any floating-point format long before we can\n",
    "differentiate it.\n",
    "\n",
    "The logarithm rescues us, turning the product into a sum that stays in range:\n",
    "\n",
    "$$\n",
    "\\log\\!\\big((1/2)^{10^9}\\big) = 10^9 \\cdot \\log(1/2) \\approx -6.93\\times10^{8},\n",
    "$$\n",
    "\n",
    "which fits comfortably even in single precision. Since $x \\mapsto \\log x$ is\n",
    "strictly increasing, it preserves the location of the maximum, so maximizing the\n",
    "likelihood is the same as maximizing the *log-likelihood*\n",
    "$\\log P(X\\mid\\boldsymbol{\\theta})$. We use this exact reasoning for the naive\n",
    "Bayes classifier in that section. Finally, because we prefer to\n",
    "*minimize* losses, we flip the sign and define the **negative log-likelihood**\n",
    "(NLL):\n",
    "\n",
    "$$\n",
    "\\ell(\\boldsymbol{\\theta}) = -\\log P(X\\mid\\boldsymbol{\\theta})\n",
    "  = -\\sum_{i=1}^n \\log p(x_i\\mid\\boldsymbol{\\theta}).\n",
    "$$\n",
    "\n",
    "For the coin this reads\n",
    "$\\ell(\\theta) = -\\big(n_H\\log\\theta + n_T\\log(1-\\theta)\\big)$, which we can\n",
    "minimize by gradient descent even for billions of flips, no closed form\n",
    "required. The cell below does exactly this for almost nine million flips and\n",
    "confirms it recovers $n_H/(n_H+n_T)$."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "maximum-likelihood-numerical-optimization-and-the-negative-log-likelihood"
   },
   "source": [
    "# Set up our data\n",
    "n_H = 8675309\n",
    "n_T = 256245\n",
    "\n",
    "# Initialize our parameters\n",
    "theta = torch.tensor(0.5, requires_grad=True)\n",
    "\n",
    "# Perform gradient descent\n",
    "lr = 1e-9\n",
    "for iter in range(100):\n",
    "    loss = -(n_H * torch.log(theta) + n_T * torch.log(1 - theta))\n",
    "    loss.backward()\n",
    "    with torch.no_grad():\n",
    "        theta -= lr * theta.grad\n",
    "    theta.grad.zero_()\n",
    "\n",
    "# Check output\n",
    "theta, n_H / (n_H + n_T)"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "(tensor(0.9713, requires_grad=True), 0.9713101437890875)"
     },
     "metadata": {},
     "execution_count": 3
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-maximum-likelihood-md-4 fw=all -->\n",
    "\n",
    "The learning rate $10^{-9}$ looks absurd. It works only because this loss is a\n",
    "*sum* over nearly nine million flips, so its gradient carries that factor of\n",
    "$n$. Shrink $n_H$ and $n_T$ a thousandfold and the same learning rate barely\n",
    "moves $\\theta$; scale them up and the update diverges. Averaging the loss over\n",
    "examples instead of summing (as the minibatch losses in this book do) removes\n",
    "the coupling and returns sensible learning rates to sensible magnitudes. The\n",
    "lesson generalizes: a learning rate is always tuned *to a loss scale*, and\n",
    "silently changing one without the other is a classic way to break training.\n",
    "\n",
    "Beyond numerical stability, the log gives gradients an additive form. The\n",
    "product rule applied to $\\prod_i p(x_i\\mid\\boldsymbol{\\theta})$ produces $n$\n",
    "terms, each a product of $n-1$ surviving factors; the logarithm collapses this to\n",
    "a single sum,\n",
    "\n",
    "$$\n",
    "\\nabla_{\\boldsymbol{\\theta}}\\, \\ell(\\boldsymbol{\\theta})\n",
    "  = -\\sum_{i=1}^n \\frac{\\nabla_{\\boldsymbol{\\theta}}\\, p(x_i\\mid\\boldsymbol{\\theta})}{p(x_i\\mid\\boldsymbol{\\theta})},\n",
    "$$\n",
    "\n",
    "a chain-rule application term by term with no shared subproducts to track. This\n",
    "is why every minibatch loss in the book is an *average* of per-example NLLs:\n",
    "the gradient of a sum is the sum of gradients, so we can estimate\n",
    "$\\nabla\\ell$ from a random subset of the data.\n",
    "\n",
    "## Maximum Likelihood Is Minimizing a Loss\n",
    "\n",
    "We now make precise the claim from the introduction. \"Minimize the loss\" and \"do\n",
    "maximum likelihood\" are the same instruction in different words. The bridge is\n",
    "one rescaling.\n",
    "\n",
    "### NLL Is the Cross-Entropy to the Data\n",
    "\n",
    "Dividing the NLL the equation by the number of examples turns it into an\n",
    "average, the natural per-example loss, and that average has an\n",
    "information-theoretic name. Let $\\hat p_{\\textrm{data}}$ be the *empirical\n",
    "distribution*, which places mass $1/n$ on each *observation*; masses at repeated\n",
    "values stack, so an outcome $x$ that occurs $n_x$ times in the sample carries\n",
    "total mass $\\hat p_{\\textrm{data}}(x) = n_x/n$, its empirical frequency. (We work\n",
    "in natural logarithms, so the units are *nats*; switching to bits merely rescales\n",
    "every quantity by $\\ln 2$ and changes no $\\mathop{\\mathrm{argmin}}$.)\n",
    "\n",
    "**Proposition (discrete MLE = minimum cross-entropy).** *For discrete\n",
    "outcomes, maximizing the likelihood is the same as minimizing the\n",
    "cross-entropy from the empirical distribution $\\hat p_{\\textrm{data}}$ to the\n",
    "model $p_{\\boldsymbol{\\theta}}$:*\n",
    "\n",
    "$$\n",
    "\\mathop{\\mathrm{argmax}}_{\\boldsymbol{\\theta}} \\prod_{i=1}^n p(x_i\\mid\\boldsymbol{\\theta})\n",
    "  = \\mathop{\\mathrm{argmin}}_{\\boldsymbol{\\theta}}\\, \\textrm{CE}\\!\\left(\\hat p_{\\textrm{data}},\\, p_{\\boldsymbol{\\theta}}\\right),\n",
    "\\qquad\n",
    "\\textrm{CE}(\\hat p_{\\textrm{data}}, p_{\\boldsymbol{\\theta}}) = -\\!\\!\\sum_{x} \\hat p_{\\textrm{data}}(x)\\log p_{\\boldsymbol{\\theta}}(x).\n",
    "$$\n",
    "\n",
    "**Proof.** Take the average negative log-likelihood and regroup the sum over\n",
    "data points into a sum over *distinct outcomes*, pooling the observations with\n",
    "$x_i = x$. Each observation contributes mass $1/n$, so outcome $x$ receives\n",
    "total mass $n_x/n = \\hat p_{\\textrm{data}}(x)$. On the coin this regrouping is\n",
    "concrete: the nine head-flips pool into nine copies of $\\log p_{\\boldsymbol{\\theta}}(\\textrm{H})$,\n",
    "so heads enters the average with weight $\\hat p_{\\textrm{data}}(\\textrm{H}) = 9/13$,\n",
    "and tails with weight $4/13$. In general,\n",
    "\n",
    "$$\n",
    "\\frac{1}{n}\\,\\ell(\\boldsymbol{\\theta})\n",
    "  = -\\frac{1}{n}\\sum_{i=1}^n \\log p_{\\boldsymbol{\\theta}}(x_i)\n",
    "  = -\\sum_{x} \\frac{n_x}{n}\\,\\log p_{\\boldsymbol{\\theta}}(x)\n",
    "  = -\\sum_{x} \\hat p_{\\textrm{data}}(x)\\,\\log p_{\\boldsymbol{\\theta}}(x)\n",
    "  = \\textrm{CE}\\!\\left(\\hat p_{\\textrm{data}},\\, p_{\\boldsymbol{\\theta}}\\right),\n",
    "$$\n",
    "\n",
    "which is exactly the cross-entropy named in the proposition. Scaling by the\n",
    "positive constant $1/n$ and flipping the sign do not move the optimizer, so the\n",
    "$\\mathop{\\mathrm{argmax}}$ of the likelihood equals the $\\mathop{\\mathrm{argmin}}$\n",
    "of the cross-entropy. $\\blacksquare$\n",
    "\n",
    "This also explains *why* minimizing cross-entropy is the right thing to do.\n",
    "Cross-entropy decomposes as $\\textrm{CE}(P,Q)=H(P)+D_{\\textrm{KL}}(P\\|Q)$, the\n",
    "entropy of $P$ plus the Kullback--Leibler divergence from $P$ to $Q$ (see\n",
    "that section for that background). Setting\n",
    "$P=\\hat p_{\\textrm{data}}$ and $Q=p_{\\boldsymbol{\\theta}}$, the empirical entropy\n",
    "$H(\\hat p_{\\textrm{data}})$ is fixed by the data and does not depend on\n",
    "$\\boldsymbol{\\theta}$, so minimizing the cross-entropy is *identical* to\n",
    "minimizing the KL term alone:\n",
    "\n",
    "$$\n",
    "\\mathop{\\mathrm{argmin}}_{\\boldsymbol{\\theta}}\\, \\textrm{CE}\\!\\left(\\hat p_{\\textrm{data}},\\, p_{\\boldsymbol{\\theta}}\\right)\n",
    "  = \\mathop{\\mathrm{argmin}}_{\\boldsymbol{\\theta}}\\, D_{\\textrm{KL}}\\!\\left(\\hat p_{\\textrm{data}}\\,\\|\\,p_{\\boldsymbol{\\theta}}\\right).\n",
    "$$\n",
    "\n",
    "In other words, maximum likelihood is the *KL-projection* of the empirical\n",
    "distribution onto the model family: it picks the $p_{\\boldsymbol{\\theta}}$ that\n",
    "is KL-closest to the data. The projection reading even survives a\n",
    "*misspecified* model: when no $p_{\\boldsymbol{\\theta}}$ equals the\n",
    "data-generating distribution there is no true parameter to converge to, and the\n",
    "MLE instead converges to the parameter whose $p_{\\boldsymbol{\\theta}}$ is\n",
    "KL-closest to the data-generating distribution [@White.1982]. For this fixed **discrete empirical distribution**, the cross-entropy cannot\n",
    "fall below the constant $H(\\hat p_{\\textrm{data}})$; optimization changes only\n",
    "the KL gap, as the figure shows. This is an algebraic decomposition\n",
    "of the observed frequencies, not by itself a statement about irreducible\n",
    "population randomness. As a special\n",
    "case, the *categorical* NLL of a classifier with one-hot labels is precisely the\n",
    "softmax cross-entropy loss of that section.\n",
    "\n",
    "![For discrete outcomes, the per-example negative log-likelihood is the cross-entropy from the empirical frequencies to the model. It splits into the fixed empirical entropy $H(\\hat p_{\\textrm{data}})$ plus $D_{\\textrm{KL}}(\\hat p_{\\textrm{data}} \\| p_{\\boldsymbol{\\theta}})$, the part optimization can change. As the model improves, the KL slice shrinks and the cross-entropy approaches the empirical-entropy floor.](../img/mdl-prob-mle-kl.svg)\n",
    "\n",
    "### From Probabilities to Densities\n",
    "\n",
    "Everything so far was phrased for discrete outcomes, where $P(X\\mid\\boldsymbol{\\theta})$\n",
    "is a genuine probability. For continuous data we replace probabilities by\n",
    "densities $p$. The discrete empirical-cross-entropy identity above should not\n",
    "be copied literally: an atomic empirical measure is generally singular with\n",
    "respect to a continuous density, so that KL divergence is infinite. The\n",
    "sample objective remains the average negative log-density, while its population\n",
    "expectation is a cross-entropy and differs from\n",
    "$D_{\\textrm{KL}}(p_\\star\\|p_{\\boldsymbol{\\theta}})$ by the fixed differential\n",
    "entropy of the data-generating density when those quantities are finite. The\n",
    "remaining practical issue is that the probability of observing *any exact*\n",
    "real value is zero, so the naive probability likelihood is $0$ for every\n",
    "$\\boldsymbol{\\theta}$. The resolution is to ask for a match only to within a small\n",
    "tolerance $\\epsilon$, then watch the $\\epsilon$ cancel.\n",
    "\n",
    "For i.i.d. observations $x_1,\\ldots,x_n$, the probability that each lands in a\n",
    "window of width $\\epsilon$ is, to first order,\n",
    "\n",
    "$$\n",
    "P\\big(X_i \\in [x_i, x_i+\\epsilon]\\ \\forall i \\mid \\boldsymbol{\\theta}\\big)\n",
    "  \\approx \\epsilon^n \\prod_{i=1}^n p(x_i\\mid\\boldsymbol{\\theta}).\n",
    "$$\n",
    "\n",
    "Taking the negative log,\n",
    "\n",
    "$$\n",
    "-\\log P\\big(X_i \\in [x_i, x_i+\\epsilon]\\ \\forall i\\mid\\boldsymbol{\\theta}\\big)\n",
    "  \\approx -n\\log\\epsilon - \\sum_{i} \\log p(x_i\\mid\\boldsymbol{\\theta}).\n",
    "$$\n",
    "\n",
    "The tolerance enters only through the additive constant $-n\\log\\epsilon$, which is\n",
    "free of $\\boldsymbol{\\theta}$. Demanding four digits of precision or four hundred\n",
    "changes that constant but never the minimizer, so we drop it and minimize\n",
    "\n",
    "$$\n",
    "-\\sum_{i} \\log p(x_i\\mid\\boldsymbol{\\theta})\n",
    "$$\n",
    "\n",
    "exactly as in the discrete case. Maximum likelihood thus operates on continuous\n",
    "variables by swapping probabilities for densities and nothing more. The most\n",
    "important density to swap in is the Gaussian, which we do next.\n",
    "\n",
    "### Gaussian NLL Is Mean Squared Error\n",
    "\n",
    "The most-used regression loss falls out of the same principle by choosing a\n",
    "Gaussian noise model. Suppose each target is the model's prediction plus\n",
    "Gaussian noise of *fixed* variance, $y_i \\sim \\mathcal{N}(\\hat y_i, \\sigma^2)$\n",
    "with $\\hat y_i = f_{\\boldsymbol{\\theta}}(\\mathbf{x}_i)$.\n",
    "\n",
    "**Proposition (Gaussian NLL = MSE).** *For fixed-variance Gaussian targets, the\n",
    "negative log-likelihood equals the mean squared error up to a constant\n",
    "independent of $\\boldsymbol{\\theta}$:*\n",
    "\n",
    "$$\n",
    "-\\log \\prod_{i=1}^n \\mathcal{N}(y_i;\\,\\hat y_i,\\,\\sigma^2)\n",
    "  = \\frac{1}{2\\sigma^2}\\sum_{i=1}^n (y_i-\\hat y_i)^2 + \\frac{n}{2}\\log(2\\pi\\sigma^2).\n",
    "$$\n",
    "\n",
    "*Hence $\\mathop{\\mathrm{argmin}}_{\\boldsymbol{\\theta}}$ of this NLL is the\n",
    "$\\mathop{\\mathrm{argmin}}$ of $\\sum_i (y_i-\\hat y_i)^2$.*\n",
    "\n",
    "**Proof.** The Gaussian density the equation is\n",
    "$\\mathcal{N}(y;\\hat y,\\sigma^2) = (2\\pi\\sigma^2)^{-1/2}\\exp\\!\\big(-(y-\\hat y)^2/(2\\sigma^2)\\big)$.\n",
    "Take its negative log,\n",
    "\n",
    "$$\n",
    "-\\log \\mathcal{N}(y_i;\\hat y_i,\\sigma^2)\n",
    "  = \\frac{(y_i-\\hat y_i)^2}{2\\sigma^2} + \\frac{1}{2}\\log(2\\pi\\sigma^2),\n",
    "$$\n",
    "\n",
    "then sum over the $n$ independent examples, which is the equation.\n",
    "The second term is the same for every $\\boldsymbol{\\theta}$ (the variance is\n",
    "fixed), so it cannot affect the minimizer, and the leading factor $1/(2\\sigma^2)$\n",
    "is a positive constant. Dropping both leaves $\\sum_i (y_i-\\hat y_i)^2$, the\n",
    "sum of squared errors. $\\blacksquare$\n",
    "\n",
    "So squared-error regression is maximum likelihood under the assumption that\n",
    "the residuals are Gaussian, the assumption the central limit theorem makes\n",
    "plausible whenever the noise is a sum of many small independent effects\n",
    "(that section). The same recipe\n",
    "turns *every* common loss into the NLL of a chosen conditional\n",
    "$p(y\\mid\\mathbf x)$: pick the noise model and its negative log *is* the loss.\n",
    "\n",
    "| Noise model $p(y\\mid\\mathbf x)$ | Negative log-likelihood $-\\log p(y\\mid\\mathbf x)$ | Deep-learning loss |\n",
    "|---|---|---|\n",
    "| Gaussian, fixed variance | $\\tfrac{1}{2\\sigma^2}(y-\\hat y)^2 + \\textrm{const}$ | mean squared error (MSE) |\n",
    "| Bernoulli | $-\\,y\\log\\hat p - (1-y)\\log(1-\\hat p)$ | binary cross-entropy (BCE) |\n",
    "| Categorical (one-hot $y$) | $-\\sum_k y_k \\log \\hat p_k$ | softmax cross-entropy |\n",
    "| Laplace, fixed scale | $\\tfrac{1}{b}\\lvert y-\\hat y\\rvert + \\textrm{const}$ | mean absolute error (MAE) |\n",
    "\n",
    "Each row is one line of algebra (take the density, drop the\n",
    "$\\boldsymbol{\\theta}$-free constants) of exactly the kind we just did for the\n",
    "Gaussian. Picking a loss *is* picking a noise model.\n",
    "\n",
    "## Estimator Theory: Why Maximum Likelihood Works\n",
    "\n",
    "How good is $\\hat{\\boldsymbol{\\theta}}$? The first\n",
    "reassurance is the same KL picture. Drawing genuinely i.i.d. data from a true\n",
    "distribution $p_{\\boldsymbol{\\theta}^\\star}$, the average NLL is, by the law of\n",
    "large numbers (stated and proved in that section), an estimate\n",
    "of the *expected* cross-entropy\n",
    "$\\textrm{CE}(p_{\\boldsymbol{\\theta}^\\star}, p_{\\boldsymbol{\\theta}})$, and the\n",
    "decomposition $\\textrm{CE} = H + D_{\\textrm{KL}}$ shows that this population\n",
    "objective is minimized exactly at the truth\n",
    "$\\boldsymbol{\\theta}=\\boldsymbol{\\theta}^\\star$, where the KL term vanishes.\n",
    "\"Exactly at\" needs one hypothesis we have so far left implicit: the model must be\n",
    "**identifiable**, meaning distinct parameters give distinct distributions,\n",
    "$\\boldsymbol{\\theta}\\neq\\boldsymbol{\\theta}'\\Rightarrow p_{\\boldsymbol{\\theta}}\\neq p_{\\boldsymbol{\\theta}'}$;\n",
    "otherwise several parameters tie for the minimum and \"the\" true parameter is not\n",
    "even well defined. The hypothesis bites in deep learning: neural networks fail\n",
    "identifiability spectacularly, since permuting the hidden units of a layer\n",
    "(together with their weights) changes the parameter vector but not the function\n",
    "it computes, so for such models consistency can only ever be a statement about\n",
    "the fitted *distribution* $p_{\\hat{\\boldsymbol{\\theta}}}$, never about the\n",
    "parameter vector itself.\n",
    "\n",
    "Minimizing the empirical average therefore targets the right minimizer. One\n",
    "caveat: the law of large numbers makes the empirical objective converge to the\n",
    "population one at each *fixed* $\\boldsymbol{\\theta}$, and pointwise convergence\n",
    "of objectives does not by itself force their *minimizers* to converge. For\n",
    "that, the convergence must be *uniform* over the parameter space: the supremum\n",
    "over $\\boldsymbol{\\theta}$ of the gap between the empirical and population\n",
    "objectives must go to zero. With identifiability and uniform convergence in\n",
    "hand, the conclusion is a genuine theorem [@Wasserman.2013]: for a\n",
    "well-specified model the MLE is **consistent**, converging in probability to\n",
    "the true parameter,\n",
    "$\\hat{\\boldsymbol{\\theta}}\\xrightarrow{P}\\boldsymbol{\\theta}^\\star$, so the\n",
    "probability of an error larger than any fixed tolerance goes to zero. The coin\n",
    "made this concrete: $\\hat\\theta = n_H/(n_H+n_T) \\to \\theta^\\star$ directly by\n",
    "the law of large numbers, and the Bernoulli family is identifiable, since\n",
    "different biases give different distributions. Consistency, asymptotic\n",
    "unbiasedness, and efficiency are the estimator-quality notions that\n",
    "that section develops systematically; here they appear\n",
    "specialized to the MLE.\n",
    "\n",
    "Consistency says only *where* $\\hat{\\boldsymbol{\\theta}}$ lands; the sharper\n",
    "statement is *how fast* and *how tightly* it concentrates. Under mild regularity\n",
    "conditions the MLE is **asymptotically normal**: the rescaled error converges in\n",
    "distribution to a Gaussian,\n",
    "\n",
    "$$\n",
    "\\sqrt{n}\\,\\big(\\hat{\\boldsymbol{\\theta}} - \\boldsymbol{\\theta}^\\star\\big)\n",
    "  \\;\\xrightarrow{d}\\; \\mathcal{N}\\!\\big(\\mathbf{0},\\; I(\\boldsymbol{\\theta}^\\star)^{-1}\\big),\n",
    "$$\n",
    "\n",
    "where $I(\\boldsymbol{\\theta})$ is the **Fisher information** defined just below\n",
    "[@Bishop.2006; @Wasserman.2013]. Here and below, \"regularity conditions\"\n",
    "means one standard package of hypotheses that we state once and take on faith:\n",
    "the density is smooth enough in $\\boldsymbol{\\theta}$ that differentiation and\n",
    "integration can be interchanged (the tool of\n",
    "that section), the support of\n",
    "$p(x\\mid\\boldsymbol{\\theta})$ does not depend on $\\boldsymbol{\\theta}$, and the\n",
    "true parameter lies in the interior of the parameter space\n",
    "[@Wasserman.2013]. The display says two things. First, the error shrinks\n",
    "at the $1/\\sqrt{n}$ rate, so the standard error of each component falls like\n",
    "$1/\\sqrt n$: halving it costs four times the data. Second, the limiting\n",
    "variance is exactly $I(\\boldsymbol{\\theta}^\\star)^{-1}$, and the Cramér--Rao\n",
    "bound stated below (the equation) says *no* unbiased estimator\n",
    "can have smaller variance than $I(\\boldsymbol{\\theta}^\\star)^{-1}/n$. The MLE\n",
    "attains the Cramér--Rao floor in the limit: it is **asymptotically\n",
    "efficient**, asymptotically the best *unbiased* estimator. The qualifiers\n",
    "matter. At finite $n$ the MLE is generally only *asymptotically* unbiased, not\n",
    "exactly unbiased: the maximum-likelihood estimate of a Gaussian's variance\n",
    "divides the sum of squares by $n$ rather than $n-1$ and so systematically\n",
    "underestimates $\\sigma^2$. that section works out exactly this\n",
    "bias, identifying the $n$-divided estimator as the Gaussian MLE, when it\n",
    "derives the $n-1$ correction. Maximum likelihood is optimal *in the limit*,\n",
    "not a guarantee of unbiasedness at every sample size.\n",
    "\n",
    "**Watching the theorem happen.** Asymptotic normality is the section's central\n",
    "limit theorem, and like the CLT proper it can be watched. The cell below returns\n",
    "to the coin: it draws $20{,}000$ independent datasets of $n=400$ flips with true\n",
    "bias $\\theta^\\star=0.7$, computes the MLE $\\hat\\theta=n_H/n$ on each, and\n",
    "histograms the rescaled errors $\\sqrt n\\,(\\hat\\theta-\\theta^\\star)$ against the\n",
    "Gaussian that the equation names. One scope note: the\n",
    "coin's MLE is itself a sample mean, so for the coin the theorem reduces to the\n",
    "CLT proper; the demonstration illustrates the general statement on the one\n",
    "model where we can also verify it by hand. For the coin the Fisher information\n",
    "works out below to $I(\\theta)=1/(\\theta(1-\\theta))$, so the predicted limit is\n",
    "$\\mathcal N\\bigl(0,\\ \\theta^\\star(1-\\theta^\\star)\\bigr)$ with variance $0.21$,\n",
    "and the histogram lands on that *particular* Gaussian, center, width, and\n",
    "height. (One numerical nicety:\n",
    "$\\hat\\theta$ lives on the lattice $k/n$, so we offset the bin edges by half a\n",
    "lattice step to keep the discrete values off the edges.)"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mdl-maximum-likelihood-estimator-theory-why-maximum-likelihood-works"
   },
   "source": [
    "rng = onp.random.default_rng(0)\n",
    "theta_star, n, reps = 0.7, 400, 20000\n",
    "flips = rng.random((reps, n)) < theta_star             # 20,000 datasets at once\n",
    "z = onp.sqrt(n) * (flips.mean(axis=1) - theta_star)    # sqrt(n)(theta_hat - theta*)\n",
    "sigma2 = theta_star * (1 - theta_star)                 # = 1 / I(theta*)\n",
    "hist, edges = onp.histogram(z, bins=40, range=(-2.025, 1.975), density=True)\n",
    "x = onp.arange(-2, 2, 0.01)\n",
    "d2l.plot(x, [onp.interp(x, (edges[:-1] + edges[1:]) / 2, hist),\n",
    "             onp.exp(-x**2 / (2 * sigma2)) / onp.sqrt(2 * onp.pi * sigma2)],\n",
    "         xlabel='sqrt(n) * (theta_hat - theta*)', ylabel='density',\n",
    "         legend=['20,000 replications', 'N(0, theta*(1-theta*))'])"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"246.28125pt\" height=\"183.634375pt\" viewBox=\"0 0 246.28125 183.634375\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-19T13:42:39.206443</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 183.634375 \nL 246.28125 183.634375 \nL 246.28125 0 \nL 0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 43.78125 145.8 \nL 239.08125 145.8 \nL 239.08125 7.2 \nL 43.78125 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path d=\"M 52.658523 145.8 \nL 52.658523 7.2 \n\" clip-path=\"url(#pef14e19686)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path id=\"m3db612eeda\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m3db612eeda\" x=\"52.658523\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- −2 -->\n      <g transform=\"translate(45.287429 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \nL 4684 2272 \nL 4684 1741 \nL 678 1741 \nL 678 2272 \nz\n\" transform=\"scale(0.015625)\"/>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-2212\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(83.789062 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path d=\"M 97.15613 145.8 \nL 97.15613 7.2 \n\" clip-path=\"url(#pef14e19686)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#m3db612eeda\" x=\"97.15613\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- −1 -->\n      <g transform=\"translate(89.785037 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-2212\"/>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(83.789062 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path d=\"M 141.653738 145.8 \nL 141.653738 7.2 \n\" clip-path=\"url(#pef14e19686)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#m3db612eeda\" x=\"141.653738\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 0 -->\n      <g transform=\"translate(138.472488 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_7\">\n      <path d=\"M 186.151346 145.8 \nL 186.151346 7.2 \n\" clip-path=\"url(#pef14e19686)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#m3db612eeda\" x=\"186.151346\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 1 -->\n      <g transform=\"translate(182.970096 160.398438) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_9\">\n      <path d=\"M 230.648953 145.8 \nL 230.648953 7.2 \n\" clip-path=\"url(#pef14e19686)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#m3db612eeda\" x=\"230.648953\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 2 -->\n      <g transform=\"translate(227.467703 160.398438) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-32\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_6\">\n     <!-- sqrt(n) * (theta_hat - theta*) -->\n     <g transform=\"translate(70.352344 174.076563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-71\" d=\"M 947 1747 \nQ 947 1113 1208 752 \nQ 1469 391 1925 391 \nQ 2381 391 2643 752 \nQ 2906 1113 2906 1747 \nQ 2906 2381 2643 2742 \nQ 2381 3103 1925 3103 \nQ 1469 3103 1208 2742 \nQ 947 2381 947 1747 \nz\nM 2906 525 \nQ 2725 213 2448 61 \nQ 2172 -91 1784 -91 \nQ 1150 -91 751 415 \nQ 353 922 353 1747 \nQ 353 2572 751 3078 \nQ 1150 3584 1784 3584 \nQ 2172 3584 2448 3432 \nQ 2725 3281 2906 2969 \nL 2906 3500 \nL 3481 3500 \nL 3481 -1331 \nL 2906 -1331 \nL 2906 525 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \nQ 1566 4138 1362 3434 \nQ 1159 2731 1159 2009 \nQ 1159 1288 1364 580 \nQ 1569 -128 1984 -844 \nL 1484 -844 \nQ 1016 -109 783 600 \nQ 550 1309 550 2009 \nQ 550 2706 781 3412 \nQ 1013 4119 1484 4856 \nL 1984 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \nL 1013 4856 \nQ 1481 4119 1714 3412 \nQ 1947 2706 1947 2009 \nQ 1947 1309 1714 600 \nQ 1481 -109 1013 -844 \nL 513 -844 \nQ 928 -128 1133 580 \nQ 1338 1288 1338 2009 \nQ 1338 2731 1133 3434 \nQ 928 4138 513 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-2a\" d=\"M 3009 3897 \nL 1888 3291 \nL 3009 2681 \nL 2828 2375 \nL 1778 3009 \nL 1778 1831 \nL 1422 1831 \nL 1422 3009 \nL 372 2375 \nL 191 2681 \nL 1313 3291 \nL 191 3897 \nL 372 4206 \nL 1422 3572 \nL 1422 4750 \nL 1778 4750 \nL 1778 3572 \nL 2828 4206 \nL 3009 3897 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \nL 3263 -1509 \nL -63 -1509 \nL -63 -1063 \nL 3263 -1063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-2d\" d=\"M 313 2009 \nL 1997 2009 \nL 1997 1497 \nL 313 1497 \nL 313 2009 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use xlink:href=\"#DejaVuSans-71\" transform=\"translate(52.099609 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(115.576172 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(156.689453 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(195.898438 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(234.912109 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(298.291016 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(337.304688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2a\" transform=\"translate(369.091797 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(419.091797 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(450.878906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(489.892578 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(529.101562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(592.480469 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(654.003906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(693.212891 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(754.492188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(804.492188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(867.871094 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(929.150391 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(968.359375 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2d\" transform=\"translate(1000.146484 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(1036.230469 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(1068.017578 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(1107.226562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(1170.605469 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(1232.128906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(1271.337891 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2a\" transform=\"translate(1332.617188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(1382.617188 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_11\">\n      <path d=\"M 43.78125 139.5 \nL 239.08125 139.5 \n\" clip-path=\"url(#pef14e19686)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_12\">\n      <defs>\n       <path id=\"mcdd23a06ff\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#mcdd23a06ff\" x=\"43.78125\" y=\"139.5\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_7\">\n      <!-- 0.0 -->\n      <g transform=\"translate(20.878125 143.299219) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \nL 1344 794 \nL 1344 0 \nL 684 0 \nL 684 794 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_13\">\n      <path d=\"M 43.78125 110.660449 \nL 239.08125 110.660449 \n\" clip-path=\"url(#pef14e19686)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_14\">\n      <g>\n       <use xlink:href=\"#mcdd23a06ff\" x=\"43.78125\" y=\"110.660449\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.2 -->\n      <g transform=\"translate(20.878125 114.459667) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_15\">\n      <path d=\"M 43.78125 81.820897 \nL 239.08125 81.820897 \n\" clip-path=\"url(#pef14e19686)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_16\">\n      <g>\n       <use xlink:href=\"#mcdd23a06ff\" x=\"43.78125\" y=\"81.820897\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.4 -->\n      <g transform=\"translate(20.878125 85.620116) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_17\">\n      <path d=\"M 43.78125 52.981346 \nL 239.08125 52.981346 \n\" clip-path=\"url(#pef14e19686)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_18\">\n      <g>\n       <use xlink:href=\"#mcdd23a06ff\" x=\"43.78125\" y=\"52.981346\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 0.6 -->\n      <g transform=\"translate(20.878125 56.780565) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-36\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_19\">\n      <path d=\"M 43.78125 24.141794 \nL 239.08125 24.141794 \n\" clip-path=\"url(#pef14e19686)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_20\">\n      <g>\n       <use xlink:href=\"#mcdd23a06ff\" x=\"43.78125\" y=\"24.141794\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 0.8 -->\n      <g transform=\"translate(20.878125 27.941013) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-38\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_12\">\n     <!-- density -->\n     <g transform=\"translate(14.798438 94.832813) rotate(-90) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \nL 2906 4863 \nL 3481 4863 \nL 3481 0 \nL 2906 0 \nL 2906 525 \nQ 2725 213 2448 61 \nQ 2172 -91 1784 -91 \nQ 1150 -91 751 415 \nQ 353 922 353 1747 \nQ 353 2572 751 3078 \nQ 1150 3584 1784 3584 \nQ 2172 3584 2448 3432 \nQ 2725 3281 2906 2969 \nz\nM 947 1747 \nQ 947 1113 1208 752 \nQ 1469 391 1925 391 \nQ 2381 391 2643 752 \nQ 2906 1113 2906 1747 \nQ 2906 2381 2643 2742 \nQ 2381 3103 1925 3103 \nQ 1469 3103 1208 2742 \nQ 947 2381 947 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \nQ 1816 -950 1584 -1140 \nQ 1353 -1331 966 -1331 \nL 506 -1331 \nL 506 -850 \nL 844 -850 \nQ 1081 -850 1212 -737 \nQ 1344 -625 1503 -206 \nL 1606 56 \nL 191 3500 \nL 800 3500 \nL 1894 763 \nL 2988 3500 \nL 3597 3500 \nL 2059 -325 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-64\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(63.476562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(188.378906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(240.478516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(268.261719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-79\" transform=\"translate(307.470703 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_21\">\n    <path d=\"M 52.658523 139.5 \nL 63.337949 139.395457 \nL 74.907327 138.724937 \nL 80.247039 138.22385 \nL 82.026944 137.798466 \nL 85.141776 136.900835 \nL 90.036513 133.746509 \nL 93.596322 131.900778 \nL 94.041298 131.49342 \nL 98.491059 125.588521 \nL 102.940819 118.378634 \nL 107.39058 109.658274 \nL 111.840341 98.035935 \nL 118.070006 75.984493 \nL 124.299671 55.054189 \nL 129.194408 38.658904 \nL 130.08436 34.841268 \nL 133.644169 18.460403 \nL 134.089145 17.33566 \nL 142.54369 13.5 \nL 142.988666 13.853285 \nL 147.438427 22.768311 \nL 151.443212 27.894541 \nL 151.888188 28.885901 \nL 157.672877 47.55951 \nL 161.677661 61.380865 \nL 165.23747 74.22528 \nL 169.687231 86.943523 \nL 174.136992 103.922808 \nL 179.031728 114.474479 \nL 183.036513 123.685111 \nL 191.936035 134.110609 \nL 196.385795 135.866217 \nL 200.835556 136.781872 \nL 205.285317 138.595159 \nL 210.180054 139.233234 \nL 217.299671 139.355802 \nL 221.749432 139.463951 \nL 226.199193 139.5 \nL 230.203977 139.5 \nL 230.203977 139.5 \n\" clip-path=\"url(#pef14e19686)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_22\">\n    <path d=\"M 52.658523 139.490825 \nL 66.452781 139.360217 \nL 72.682446 139.088385 \nL 76.687231 138.715467 \nL 79.802063 138.23846 \nL 82.47192 137.63939 \nL 85.141776 136.80248 \nL 87.366657 135.87171 \nL 89.591537 134.677535 \nL 91.371441 133.496558 \nL 93.151346 132.083098 \nL 94.93125 130.406399 \nL 96.711154 128.435282 \nL 98.491059 126.139095 \nL 100.270963 123.488851 \nL 102.050867 120.458541 \nL 103.830772 117.026588 \nL 105.610676 113.177393 \nL 107.835556 107.767769 \nL 110.060437 101.6991 \nL 112.285317 95.002716 \nL 114.955173 86.227079 \nL 118.070006 75.187019 \nL 127.85948 39.640771 \nL 130.08436 32.629414 \nL 131.864264 27.630511 \nL 133.644169 23.2866 \nL 134.979097 20.514745 \nL 136.314025 18.197758 \nL 137.648953 16.36451 \nL 138.538906 15.422751 \nL 139.428858 14.711709 \nL 140.31881 14.235418 \nL 141.208762 13.996592 \nL 142.098714 13.996592 \nL 142.988666 14.235418 \nL 143.878618 14.711709 \nL 144.768571 15.422751 \nL 145.658523 16.36451 \nL 146.993451 18.197758 \nL 148.328379 20.514745 \nL 149.663307 23.2866 \nL 151.443212 27.630511 \nL 153.223116 32.629414 \nL 155.447996 39.640771 \nL 158.117853 48.885103 \nL 162.567614 65.310797 \nL 167.46235 83.147949 \nL 170.577183 93.592953 \nL 173.247039 101.6991 \nL 175.47192 107.767769 \nL 177.6968 113.177393 \nL 179.921681 117.923201 \nL 181.701585 121.25302 \nL 183.481489 124.186091 \nL 185.261394 126.745242 \nL 187.041298 128.957358 \nL 188.821202 130.851958 \nL 190.601106 132.459926 \nL 192.381011 133.812417 \nL 194.605891 135.190145 \nL 196.830772 136.272788 \nL 199.055652 137.112067 \nL 201.725508 137.862312 \nL 204.840341 138.467876 \nL 208.40015 138.908191 \nL 212.84991 139.217101 \nL 219.079575 139.407079 \nL 230.203977 139.48991 \nL 230.203977 139.48991 \n\" clip-path=\"url(#pef14e19686)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 43.78125 145.8 \nL 43.78125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 239.08125 145.8 \nL 239.08125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 43.78125 145.8 \nL 239.08125 145.8 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 43.78125 7.2 \nL 239.08125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 91.385938 92.678125 \nL 232.08125 92.678125 \nQ 234.08125 92.678125 234.08125 90.678125 \nL 234.08125 62.321875 \nQ 234.08125 60.321875 232.08125 60.321875 \nL 91.385938 60.321875 \nQ 89.385938 60.321875 89.385938 62.321875 \nL 89.385938 90.678125 \nQ 89.385938 92.678125 91.385938 92.678125 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_23\">\n     <path d=\"M 93.385938 68.420313 \nL 103.385938 68.420313 \nL 113.385938 68.420313 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_13\">\n     <!-- 20,000 replications -->\n     <g transform=\"translate(121.385938 71.920313) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-2c\" d=\"M 750 794 \nL 1409 794 \nL 1409 256 \nL 897 -744 \nL 494 -744 \nL 750 256 \nL 750 794 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \nL 1159 -1331 \nL 581 -1331 \nL 581 3500 \nL 1159 3500 \nL 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nz\nM 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \nL 3122 2828 \nQ 2878 2963 2633 3030 \nQ 2388 3097 2138 3097 \nQ 1578 3097 1268 2742 \nQ 959 2388 959 1747 \nQ 959 1106 1268 751 \nQ 1578 397 2138 397 \nQ 2388 397 2633 464 \nQ 2878 531 3122 666 \nL 3122 134 \nQ 2881 22 2623 -34 \nQ 2366 -91 2075 -91 \nQ 1284 -91 818 406 \nQ 353 903 353 1747 \nQ 353 2603 823 3093 \nQ 1294 3584 2113 3584 \nQ 2378 3584 2631 3529 \nQ 2884 3475 3122 3366 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-32\"/>\n      <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2c\" transform=\"translate(127.246094 0)\"/>\n      <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(159.033203 0)\"/>\n      <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(222.65625 0)\"/>\n      <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(286.279297 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(349.902344 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(381.689453 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(420.552734 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(482.076172 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(545.552734 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(573.335938 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(601.119141 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(656.099609 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(717.378906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(756.587891 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(784.371094 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(845.552734 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(908.931641 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_24\">\n     <path d=\"M 93.385938 83.098438 \nL 103.385938 83.098438 \nL 113.385938 83.098438 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_14\">\n     <!-- N(0, theta*(1-theta*)) -->\n     <g transform=\"translate(121.385938 86.598438) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-4e\" d=\"M 628 4666 \nL 1478 4666 \nL 3547 763 \nL 3547 4666 \nL 4159 4666 \nL 4159 0 \nL 3309 0 \nL 1241 3903 \nL 1241 0 \nL 628 0 \nL 628 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-4e\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(74.804688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(113.818359 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2c\" transform=\"translate(177.441406 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(209.228516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(241.015625 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(280.224609 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(343.603516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(405.126953 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(444.335938 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2a\" transform=\"translate(505.615234 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(555.615234 0)\"/>\n      <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(594.628906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2d\" transform=\"translate(658.251953 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(694.335938 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(733.544922 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(796.923828 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(858.447266 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(897.65625 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2a\" transform=\"translate(958.935547 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(1008.935547 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(1047.949219 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pef14e19686\">\n   <rect x=\"43.78125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-maximum-likelihood-md-5 fw=all -->\n",
    "\n",
    "### Fisher Information and the Score\n",
    "\n",
    "To make the equation quantitative we need the object\n",
    "$I(\\boldsymbol{\\theta})$ it contains. Define the **score** as the gradient of the\n",
    "log-likelihood with respect to the parameters,\n",
    "$s(x;\\boldsymbol{\\theta}) = \\nabla_{\\boldsymbol{\\theta}} \\log p(x\\mid\\boldsymbol{\\theta})$.\n",
    "The score is the per-example version of the NLL gradient we already met: setting\n",
    "$\\sum_i s(x_i;\\boldsymbol{\\theta}) = \\mathbf 0$ is exactly the first-order condition\n",
    "that locates $\\hat{\\boldsymbol{\\theta}}$. At the true parameter the score has\n",
    "mean zero: differentiating the normalization\n",
    "$\\int p(x\\mid\\boldsymbol{\\theta})\\,dx = 1$ under the integral sign gives\n",
    "\n",
    "$$\n",
    "\\mathbf 0\n",
    "  = \\nabla_{\\boldsymbol{\\theta}} \\int p(x\\mid\\boldsymbol{\\theta})\\,dx\n",
    "  = \\int \\nabla_{\\boldsymbol{\\theta}}\\, p(x\\mid\\boldsymbol{\\theta})\\,dx\n",
    "  = \\int s(x;\\boldsymbol{\\theta})\\, p(x\\mid\\boldsymbol{\\theta})\\,dx\n",
    "  = \\mathbb{E}_{\\boldsymbol{\\theta}}\\!\\big[\\,s(x;\\boldsymbol{\\theta})\\,\\big],\n",
    "$$\n",
    "\n",
    "exactly the computation of that section, which also\n",
    "grants the interchange of derivative and integral (the first of the regularity\n",
    "conditions named above); the third equality is the identity\n",
    "$\\nabla_{\\boldsymbol{\\theta}}\\, p = p\\,\\nabla_{\\boldsymbol{\\theta}} \\log p = p\\,s$.\n",
    "Because the score has mean zero, its variance equals its second moment, and\n",
    "that variance is the **Fisher information** [@Fisher.1928],\n",
    "\n",
    "$$\n",
    "I(\\boldsymbol{\\theta}) = \\operatorname{Var}_{\\boldsymbol{\\theta}}\\!\\big[\\,s(x;\\boldsymbol{\\theta})\\,\\big]\n",
    "  = \\mathbb{E}_{\\boldsymbol{\\theta}}\\!\\big[\\,s(x;\\boldsymbol{\\theta})\\,s(x;\\boldsymbol{\\theta})^\\top\\,\\big]\n",
    "  = -\\,\\mathbb{E}_{\\boldsymbol{\\theta}}\\!\\big[\\nabla^2_{\\boldsymbol{\\theta}} \\log p(x\\mid\\boldsymbol{\\theta})\\big].\n",
    "$$\n",
    "\n",
    "The middle equality is the mean-zero property at work:\n",
    "$\\operatorname{Var}[s]=\\mathbb E[ss^\\top]$ once $\\mathbb E[s]=\\mathbf 0$. The\n",
    "last equality is the *information identity*, and it follows by differentiating\n",
    "$\\mathbb{E}_{\\boldsymbol{\\theta}}[s]=\\mathbf 0$ once more, with the same\n",
    "interchange granted. The product rule splits the derivative into two terms,\n",
    "\n",
    "$$\n",
    "\\mathbf 0\n",
    "  = \\nabla_{\\boldsymbol{\\theta}} \\int s(x;\\boldsymbol{\\theta})\\,p(x\\mid\\boldsymbol{\\theta})\\,dx\n",
    "  = \\int \\nabla^2_{\\boldsymbol{\\theta}} \\log p(x\\mid\\boldsymbol{\\theta})\\; p(x\\mid\\boldsymbol{\\theta})\\,dx\n",
    "  + \\int s(x;\\boldsymbol{\\theta})\\,\\big(\\nabla_{\\boldsymbol{\\theta}}\\, p(x\\mid\\boldsymbol{\\theta})\\big)^\\top dx,\n",
    "$$\n",
    "\n",
    "and substituting $\\nabla_{\\boldsymbol{\\theta}}\\, p = p\\,s$ in the second term\n",
    "turns it into $\\mathbb{E}_{\\boldsymbol{\\theta}}[\\,s\\,s^\\top]$, so\n",
    "\n",
    "$$\n",
    "\\mathbb{E}_{\\boldsymbol{\\theta}}\\!\\big[\\nabla^2_{\\boldsymbol{\\theta}} \\log p(x\\mid\\boldsymbol{\\theta})\\big]\n",
    "  + \\mathbb{E}_{\\boldsymbol{\\theta}}\\!\\big[\\,s\\,s^\\top\\,\\big] = \\mathbf 0,\n",
    "$$\n",
    "\n",
    "which is the last equality of the equation rearranged. The identity\n",
    "says that the information is the expected *curvature* of the NLL. The reading\n",
    "is intuitive: a sharply peaked log-likelihood (large curvature, large $I$) pins\n",
    "the parameter down tightly, so the estimate has small variance; a flat one\n",
    "leaves $\\boldsymbol{\\theta}$ poorly determined. That is precisely what\n",
    "the equation says, $\\operatorname{Var}(\\hat{\\boldsymbol{\\theta}}) \\approx I(\\boldsymbol{\\theta}^\\star)^{-1}/n$.\n",
    "the figure draws the picture for a Gaussian's two\n",
    "parameters: the NLL is a bowl around the MLE, and the inverse of its curvature is\n",
    "the ellipse within which the estimate scatters.\n",
    "\n",
    "![Fisher information is the curvature of the negative log-likelihood. Contours show the NLL for a Gaussian's parameters $(\\mu, \\sigma)$ over $n=30$ draws from $\\mathcal{N}(1, 1.5^2)$; the dot marks the maximum-likelihood estimate, and the overlaid ellipse is the inverse-information covariance $I(\\hat\\mu,\\hat\\sigma)^{-1}/n$ that asymptotic normality predicts for the estimate's spread. Per observation $I = \\operatorname{diag}(1/\\sigma^2,\\ 2/\\sigma^2)$: the NLL is twice as curved in $\\sigma$ as in $\\mu$, so the data pin down the scale more tightly than the mean and the ellipse is correspondingly wider along the $\\mu$-axis.](../img/mdl-prob-mle-fisher.svg)\n",
    "\n",
    "The curvature intuition extends to a theorem constraining *every* estimator.\n",
    "The **Cramér--Rao bound** [@Cramer.1946; @Radhakrishna-Rao.1945] states that\n",
    "any *unbiased* estimator $\\tilde\\theta$\n",
    "built from $n$ i.i.d. observations obeys\n",
    "\n",
    "$$\n",
    "\\operatorname{Var}\\big(\\tilde\\theta\\big) \\;\\ge\\; \\frac{1}{n\\,I(\\theta)}\n",
    "$$\n",
    "\n",
    "(stated here for a scalar parameter; for vectors the statement reads\n",
    "$\\operatorname{Cov}(\\tilde{\\boldsymbol{\\theta}}) \\succeq I(\\boldsymbol{\\theta})^{-1}/n$\n",
    "in the positive-semidefinite order) [@Wasserman.2013]. Information caps\n",
    "precision: no cleverness in constructing the estimator can beat the curvature the\n",
    "model itself supplies, and comparing with the equation\n",
    "shows the MLE's limiting variance meets the Cramér--Rao floor exactly, which is\n",
    "precisely the asymptotic efficiency claimed above.\n",
    "\n",
    "Back to the coin. A single flip has log-likelihood\n",
    "$\\log p(x\\mid\\theta) = x\\log\\theta + (1-x)\\log(1-\\theta)$ for $x\\in\\{0,1\\}$, so\n",
    "the score and its derivative are\n",
    "\n",
    "$$\n",
    "s(x;\\theta) = \\frac{x}{\\theta} - \\frac{1-x}{1-\\theta},\n",
    "\\qquad\n",
    "\\frac{d}{d\\theta}\\, s(x;\\theta) = -\\frac{x}{\\theta^2} - \\frac{1-x}{(1-\\theta)^2}.\n",
    "$$\n",
    "\n",
    "Taking $-\\mathbb{E}[\\,\\cdot\\,]$ with $\\mathbb{E}[x]=\\theta$ collapses the two\n",
    "fractions to $\\tfrac{1}{\\theta} + \\tfrac{1}{1-\\theta}$, so the per-flip Fisher\n",
    "information and its inverse are\n",
    "\n",
    "$$\n",
    "I(\\theta) = \\frac{1}{\\theta} + \\frac{1}{1-\\theta} = \\frac{1}{\\theta(1-\\theta)},\n",
    "\\qquad\n",
    "\\frac{I(\\theta)^{-1}}{n} = \\frac{\\theta(1-\\theta)}{n}.\n",
    "$$\n",
    "\n",
    "Note that the per-flip information $1/(\\theta(1-\\theta))$ is *smallest* at\n",
    "$\\theta = 1/2$: fair coins are the hardest to pin down, each flip carrying the\n",
    "least information about the bias. The prediction of\n",
    "the equation is that the MLE $\\hat\\theta = n_H/n$ over\n",
    "$n$ flips has variance about $\\theta(1-\\theta)/n$, which, by\n",
    "the equation, is also the Cramér--Rao floor\n",
    "the equation for this problem. We can check that this is *exactly*\n",
    "right here ($\\hat\\theta$ is a scaled binomial count, whose variance is\n",
    "$\\theta(1-\\theta)/n$ on the nose) and confirm it by simulation: run the\n",
    "$n$-flip experiment many times and compare the spread of the resulting estimates\n",
    "to the floor."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "maximum-likelihood-fisher-information"
   },
   "source": [
    "# NumPy simulation of the coin MLE's variance.\n",
    "theta_star, n, trials = 0.3, 200, 20000\n",
    "rng = onp.random.default_rng(0)\n",
    "# Each trial: flip the coin n times; the MLE is the observed head fraction.\n",
    "heads = rng.binomial(n, theta_star, size=trials)\n",
    "theta_hats = heads / n\n",
    "\n",
    "empirical_var = theta_hats.var()\n",
    "cramer_rao = theta_star * (1 - theta_star) / n  # = I(theta)^-1 / n\n",
    "empirical_var, cramer_rao"
   ],
   "execution_count": 5,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "(np.float64(0.0010385371777499998), 0.00105)"
     },
     "metadata": {},
     "execution_count": 5
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-maximum-likelihood-md-6 fw=all -->\n",
    "\n",
    "The empirical variance of the maximum-likelihood estimates matches the\n",
    "Cramér--Rao floor $\\theta(1-\\theta)/n$ to two digits: the coin's MLE is as\n",
    "tight as the equation permits. This is asymptotic efficiency made\n",
    "concrete, and it is why the curvature of a loss surface (its Fisher\n",
    "information, or empirically its Hessian) tells us how trustworthy a fitted\n",
    "parameter is.\n",
    "\n",
    "## MAP Estimation: Priors as Regularizers\n",
    "\n",
    "We dropped the prior $P(\\boldsymbol{\\theta})$ by declaring it flat.\n",
    "Keeping it instead turns maximum likelihood into *maximum a posteriori* (MAP)\n",
    "estimation, and the prior reappears as a regularizer. Maximizing the full\n",
    "posterior the equation and taking negative logs,\n",
    "\n",
    "$$\n",
    "\\hat{\\boldsymbol{\\theta}}_{\\textrm{MAP}}\n",
    "  = \\mathop{\\mathrm{argmax}}_{\\boldsymbol{\\theta}}\\, P(X\\mid\\boldsymbol{\\theta})\\,P(\\boldsymbol{\\theta})\n",
    "  = \\mathop{\\mathrm{argmin}}_{\\boldsymbol{\\theta}}\\,\n",
    "    \\Big[\\underbrace{-\\textstyle\\sum_i \\log p(x_i\\mid\\boldsymbol{\\theta})}_{\\textrm{data fit (NLL)}}\n",
    "       \\;\\underbrace{-\\log P(\\boldsymbol{\\theta})}_{\\textrm{regularizer}}\\Big].\n",
    "$$\n",
    "\n",
    "The objective is the familiar data-fit term plus a penalty: the negative\n",
    "log-prior. A prior concentrated near small parameters penalizes large ones, which\n",
    "is precisely what a regularizer does.\n",
    "\n",
    "### Gaussian Priors Are Weight Decay\n",
    "\n",
    "**Proposition (Gaussian prior = $L_2$ / weight decay).** *A Gaussian prior\n",
    "$\\boldsymbol{\\theta}\\sim\\mathcal{N}(\\mathbf 0,\\tau^2 \\mathbf I)$ contributes the\n",
    "penalty $\\tfrac{1}{2\\tau^2}\\lVert\\boldsymbol{\\theta}\\rVert_2^2$ to the MAP\n",
    "objective. With Gaussian-noise data (variance $\\sigma^2$), MAP estimation is\n",
    "$L_2$-regularized least squares with weight-decay strength\n",
    "$\\lambda = \\sigma^2/\\tau^2$.*\n",
    "\n",
    "**Proof.** The density of $\\mathcal{N}(\\mathbf 0,\\tau^2\\mathbf I)$ is proportional\n",
    "to $\\exp\\!\\big(-\\lVert\\boldsymbol{\\theta}\\rVert_2^2/(2\\tau^2)\\big)$, so its\n",
    "negative log-prior is $\\tfrac{1}{2\\tau^2}\\lVert\\boldsymbol{\\theta}\\rVert_2^2$ plus\n",
    "a constant. Insert this and the Gaussian NLL the equation into\n",
    "the MAP objective the equation:\n",
    "\n",
    "$$\n",
    "\\hat{\\boldsymbol{\\theta}}_{\\textrm{MAP}}\n",
    "  = \\mathop{\\mathrm{argmin}}_{\\boldsymbol{\\theta}}\\;\n",
    "    \\frac{1}{2\\sigma^2}\\sum_{i}(y_i-\\hat y_i)^2\n",
    "      + \\frac{1}{2\\tau^2}\\lVert\\boldsymbol{\\theta}\\rVert_2^2 .\n",
    "$$\n",
    "\n",
    "Multiplying by the positive constant $2\\sigma^2$ leaves\n",
    "$\\sum_i(y_i-\\hat y_i)^2 + \\tfrac{\\sigma^2}{\\tau^2}\\lVert\\boldsymbol{\\theta}\\rVert_2^2$,\n",
    "which is ridge regression (that section) with\n",
    "$\\lambda=\\sigma^2/\\tau^2$. $\\blacksquare$\n",
    "\n",
    "This is the probabilistic origin of *weight decay*: the regularizer we add by\n",
    "hand in that section, and read as a norm constraint in\n",
    "that section, is the negative log of a Gaussian\n",
    "belief that weights should be small. The same construction with a *Laplace* prior\n",
    "$p(\\boldsymbol{\\theta})\\propto e^{-\\lVert\\boldsymbol{\\theta}\\rVert_1/b}$ gives the\n",
    "$L_1$ penalty $\\tfrac{1}{b}\\lVert\\boldsymbol{\\theta}\\rVert_1$, whose kink at the\n",
    "origin produces exact zeros, i.e. sparsity. the figure shows\n",
    "the tug-of-war: the NLL pulls toward the MLE, the log-prior pulls toward the\n",
    "prior mean, and the MAP estimate sits between them.\n",
    "\n",
    "![Maximum a posteriori balances data against prior. The negative log-likelihood (blue) is minimized at the MLE; adding a Gaussian log-prior centered at the prior mean (orange, dashed) yields the MAP objective (green), whose minimum is pulled from the MLE toward the prior mean. A tighter prior (smaller $\\tau$) pulls harder; as $\\tau\\to\\infty$ the prior flattens and MAP returns to the MLE.](../img/mdl-prob-map-prior.svg)\n",
    "\n",
    "The figure also reads off the two limits. As the prior flattens ($\\tau\\to\\infty$,\n",
    "or $b\\to\\infty$) the penalty vanishes and $\\hat{\\boldsymbol{\\theta}}_{\\textrm{MAP}}\n",
    "\\to \\hat{\\boldsymbol{\\theta}}_{\\textrm{MLE}}$, the flat-prior assumption\n",
    "we started with. And as the data grows the NLL term deepens faster than the fixed\n",
    "prior, so the data term dominates and MAP again approaches the MLE. A bookkeeping\n",
    "footnote makes the second limit quantitative: deep-learning code minimizes the\n",
    "*mean* per-example loss rather than the sum, and dividing the MAP objective by\n",
    "$n$ rescales the weight decay to $\\lambda = \\sigma^2/(n\\tau^2)$: the prior's\n",
    "pull on the averaged loss literally fades as $1/n$. Regularization matters most\n",
    "precisely when data is scarce.\n",
    "\n",
    "### A Beta Prior on the Coin\n",
    "\n",
    "Weight decay is MAP for regression; we now give the running coin its MAP\n",
    "treatment, completing the example. A prior on a coin's bias must\n",
    "live on $[0,1]$, and the natural family is the **Beta distribution**\n",
    "$p(\\theta)\\propto\\theta^{a-1}(1-\\theta)^{b-1}$ with shape parameters $a, b > 0$,\n",
    "the conjugate prior of that section: multiplying by the\n",
    "likelihood the equation just shifts the exponents by the observed\n",
    "counts, so the posterior is again a Beta (the equation),\n",
    "\n",
    "$$\n",
    "P(\\theta\\mid X) \\;\\propto\\; \\theta^{\\,n_H + a - 1}\\,(1-\\theta)^{\\,n_T + b - 1}.\n",
    "$$\n",
    "\n",
    "MAP asks for the mode of this density. That is the same single-variable\n",
    "maximization we solved for the likelihood itself: the posterior has the shape\n",
    "$\\theta^{\\alpha}(1-\\theta)^{\\beta}$ with $\\alpha = n_H+a-1$ and\n",
    "$\\beta = n_T+b-1$, and whenever both exponents are positive (e.g. $a, b \\ge 1$\n",
    "on our data) the maximum is interior. The answer is therefore the same ratio,\n",
    "\"head exponent over total exponent\":\n",
    "\n",
    "$$\n",
    "\\hat\\theta_{\\textrm{MAP}} \\;=\\; \\frac{n_H + a - 1}{n + a + b - 2},\n",
    "\\qquad n = n_H + n_T.\n",
    "$$\n",
    "\n",
    "The reading is **pseudo-counts**: the prior behaves exactly like $a-1$ phantom\n",
    "heads and $b-1$ phantom tails appended to the data before taking the empirical\n",
    "frequency. A $\\mathrm{Beta}(2,2)$ prior adds one phantom flip of each kind,\n",
    "turning our $9$-of-$13$ coin into $(9+1)/(13+2) = 2/3$, pulled from\n",
    "$9/13\\approx 0.692$ toward $1/2$; a $\\mathrm{Beta}(30,30)$ prior would all but\n",
    "pin the estimate near a fair coin until the data outweighed its $58$ phantom\n",
    "flips. And as $n$ grows, any fixed stock of phantom flips washes out:\n",
    "$\\hat\\theta_{\\textrm{MAP}}\\to\\hat\\theta_{\\textrm{MLE}}$, the data-swamps-the-prior\n",
    "limit we just met.\n",
    "\n",
    "Two subtleties in the equation are easy to get wrong. First, the\n",
    "*flat* prior $\\mathrm{Beta}(1,1)$, uniform on $[0,1]$, gives\n",
    "$\\hat\\theta_{\\textrm{MAP}} = n_H/n$: **MAP under a uniform prior\n",
    "is exactly the MLE**, zero phantom flips, no smoothing at all. In particular the\n",
    "\"add-one\" (Laplace) smoothing rule $(n_H+1)/(n+2)$ is *not* the uniform-prior\n",
    "MAP; read as pseudo-counts, it is the MAP under $\\mathrm{Beta}(2,2)$.\n",
    "\n",
    "Second, the mode is not the mean. The posterior mean is\n",
    "\n",
    "$$\n",
    "\\mathbb{E}[\\theta\\mid X] \\;=\\; \\frac{n_H + a}{n + a + b},\n",
    "$$\n",
    "\n",
    "which under the flat prior becomes $(n_H+1)/(n+2)$: **Laplace's rule of\n",
    "succession** [@Laplace.1814], precisely the add-one rule. So add-one\n",
    "smoothing *is* a Bayes\n",
    "estimate under the uniform prior, just a different summary: the posterior\n",
    "**mean**, not the posterior **mode**. (Numerically it coincides with the\n",
    "$\\mathrm{Beta}(2,2)$ mode, which is why the two are so often conflated.) Naive\n",
    "Bayes (that section) leans on exactly this smoothing to keep\n",
    "never-observed feature--class pairs from annihilating its products; the\n",
    "pseudo-count accounting built here is what justifies it.\n",
    "\n",
    "### The Posterior Mode Is Not the Posterior\n",
    "\n",
    "One caution before we move on: MAP is *not* the same as \"being Bayesian.\" MAP\n",
    "restores the prior but still reports a single point: the *mode* of the posterior\n",
    "$P(\\boldsymbol{\\theta}\\mid X)$, the same $\\mathop{\\mathrm{argmax}}$ as in\n",
    "the equation. Genuine Bayesian inference keeps the entire posterior\n",
    "*distribution* and *integrates* over it, predicting with the posterior-averaged\n",
    "$\\int p(x\\mid\\boldsymbol{\\theta})\\,P(\\boldsymbol{\\theta}\\mid X)\\,d\\boldsymbol{\\theta}$\n",
    "rather than plugging in one $\\hat{\\boldsymbol{\\theta}}_{\\textrm{MAP}}$. That\n",
    "average propagates parameter *uncertainty* into the prediction, where the mode\n",
    "discards it. The mode is not even reparameterization-invariant, since a\n",
    "nonlinear change of variables moves the peak of a density but not its integral.\n",
    "Maximum likelihood does not suffer this wobble: it is **equivariant** under\n",
    "reparameterization, meaning that if $\\hat\\theta$ is the MLE of $\\theta$, then $g(\\hat\\theta)$\n",
    "is the MLE of $g(\\theta)$ for any one-to-one $g$, so the MLE of the coin's\n",
    "log-odds is simply $\\log\\frac{9/13}{4/13} = \\log\\frac{9}{4}$\n",
    "[@Wasserman.2013]. The reason is that relabeling the parameter axis moves\n",
    "no maximum: likelihood values are attached to points, while densities carry a\n",
    "Jacobian. MAP inherits the prior density's Jacobian, and with it the\n",
    "dependence on how the parameter happens to be written down.\n",
    "Marginalizing over the posterior is its own subject, beyond our scope here\n",
    "[@Murphy.2022; @mackay2003information]; the point is only that MAP is one more\n",
    "*point estimate* (maximum likelihood with a penalty) and should not be mistaken\n",
    "for the full Bayesian treatment.\n",
    "\n",
    "## Latent Variables, EM, and the ELBO\n",
    "\n",
    "Every likelihood so far has been a product of terms we could log and\n",
    "differentiate directly. The most interesting generative models are not so\n",
    "obliging. They posit a **latent variable** $z$ (an unobserved cluster identity,\n",
    "a topic, the \"seed\" a generator decodes into an image) and define the joint\n",
    "distribution\n",
    "\n",
    "$$\n",
    "p(x, z; \\boldsymbol{\\theta}) = p(z; \\boldsymbol{\\theta})\\, p(x \\mid z; \\boldsymbol{\\theta}),\n",
    "$$\n",
    "\n",
    "of which we observe only $x$. The likelihood of an observation is then a\n",
    "*marginal*, obtained by summing (for continuous latents, integrating) the latent\n",
    "out:\n",
    "\n",
    "$$\n",
    "p(x;\\boldsymbol{\\theta}) = \\sum_{z} p(z;\\boldsymbol{\\theta})\\, p(x\\mid z;\\boldsymbol{\\theta}).\n",
    "$$\n",
    "\n",
    "### Why Latent Variables Break the Recipe\n",
    "\n",
    "The canonical example is the **Gaussian mixture model** (GMM): the latent\n",
    "$z \\in \\{1,\\ldots,K\\}$ picks a component with probability\n",
    "$P(z = k) = \\pi_k$, and the observation is then drawn from that component,\n",
    "$x \\mid z{=}k \\sim \\mathcal{N}(\\mu_k, \\sigma_k^2)$. The data's NLL is\n",
    "\n",
    "$$\n",
    "\\ell(\\boldsymbol{\\theta})\n",
    "  = -\\sum_{i=1}^n \\log\\Big(\\sum_{k=1}^K \\pi_k\\, \\mathcal{N}(x_i;\\, \\mu_k, \\sigma_k^2)\\Big),\n",
    "\\qquad \\boldsymbol{\\theta} = (\\pi_k, \\mu_k, \\sigma_k)_{k=1}^K.\n",
    "$$\n",
    "\n",
    "The trouble is the sum *inside* the logarithm. Until now every log reached an\n",
    "individual density and the NLL split into per-example, per-parameter\n",
    "pieces; here the log stops at the mixture, every parameter appears inside every\n",
    "term through the shared sum, and no closed form solves the first-order\n",
    "conditions. Contrast the *complete-data* problem: if an oracle revealed each\n",
    "$z_i$, the likelihood would factor by component, and maximum likelihood would\n",
    "collapse to the estimators we already own: the fraction of points in component\n",
    "$k$ for $\\pi_k$ (the coin), and the per-component sample mean and variance for\n",
    "$\\mu_k, \\sigma_k^2$ (the Gaussian). Plain gradient descent on\n",
    "$\\ell(\\boldsymbol{\\theta})$ remains possible, but the structure of the\n",
    "problem (easy if complete, hard only because $z$ is missing) suggests\n",
    "something better: *guess the missing labels probabilistically, then solve the\n",
    "easy problem.*\n",
    "\n",
    "### The Evidence Lower Bound\n",
    "\n",
    "Let $q(z)$ be any distribution over the latent with $q(z) > 0$ wherever\n",
    "$p(x,z;\\boldsymbol{\\theta}) > 0$, our probabilistic guess. Insert it into the\n",
    "marginal and apply Jensen's inequality ($\\log$ is concave, so\n",
    "$\\log \\mathbb{E}[\\cdot] \\ge \\mathbb{E}[\\log(\\cdot)]$; that section):\n",
    "\n",
    "$$\n",
    "\\log p(x;\\boldsymbol{\\theta})\n",
    "  = \\log \\sum_z q(z)\\, \\frac{p(x,z;\\boldsymbol{\\theta})}{q(z)}\n",
    "  \\;\\ge\\; \\sum_z q(z) \\log \\frac{p(x,z;\\boldsymbol{\\theta})}{q(z)}\n",
    "  \\;=:\\; \\mathcal{L}(q, \\boldsymbol{\\theta}).\n",
    "$$\n",
    "\n",
    "The right-hand side is the **evidence lower bound** ELBO. The name comes\n",
    "from calling the marginal log-likelihood $\\log p(x;\\boldsymbol{\\theta})$ the\n",
    "*evidence*: it plays for the latent $z$ exactly the role the denominator $P(X)$\n",
    "played for the parameters in Bayes' rule at the start of this section: the\n",
    "marginal probability of what we actually observed. Expanding the fraction splits\n",
    "the bound into two readable pieces,\n",
    "\n",
    "$$\n",
    "\\mathcal{L}(q,\\boldsymbol{\\theta})\n",
    "  = \\mathbb{E}_{q}\\big[\\log p(x,z;\\boldsymbol{\\theta})\\big] + H(q),\n",
    "$$\n",
    "\n",
    "the expected *complete-data* log-likelihood (the easy objective, averaged over\n",
    "the guess) plus the entropy of the guess, which does not involve\n",
    "$\\boldsymbol{\\theta}$ at all (that section). How much\n",
    "does the bound give away? Exactly a KL divergence:\n",
    "\n",
    "**Proposition (the ELBO gap is a KL).** *For every $q$ and every\n",
    "$\\boldsymbol{\\theta}$,*\n",
    "\n",
    "$$\n",
    "\\log p(x;\\boldsymbol{\\theta})\n",
    "  = \\mathcal{L}(q,\\boldsymbol{\\theta})\n",
    "  + D_{\\textrm{KL}}\\big(q(z)\\,\\|\\,p(z\\mid x;\\boldsymbol{\\theta})\\big).\n",
    "$$\n",
    "\n",
    "*Hence $\\mathcal{L}(q,\\boldsymbol{\\theta}) \\le \\log p(x;\\boldsymbol{\\theta})$,\n",
    "with equality if and only if $q$ is the posterior\n",
    "$p(z\\mid x;\\boldsymbol{\\theta})$.*\n",
    "\n",
    "**Proof.** Expand the KL term using Bayes' rule,\n",
    "$p(z\\mid x;\\boldsymbol{\\theta}) = p(x,z;\\boldsymbol{\\theta})/p(x;\\boldsymbol{\\theta})$:\n",
    "\n",
    "$$\n",
    "D_{\\textrm{KL}}\\big(q \\,\\|\\, p(z\\mid x;\\boldsymbol{\\theta})\\big)\n",
    "  = \\mathbb{E}_q\\!\\left[\\log \\frac{q(z)\\, p(x;\\boldsymbol{\\theta})}{p(x,z;\\boldsymbol{\\theta})}\\right]\n",
    "  = \\log p(x;\\boldsymbol{\\theta}) - \\mathcal{L}(q,\\boldsymbol{\\theta}),\n",
    "$$\n",
    "\n",
    "where $\\log p(x;\\boldsymbol{\\theta})$ exits the expectation because it does not\n",
    "involve $z$. Rearranging gives the equation; since the KL divergence\n",
    "is nonnegative and vanishes exactly when its arguments agree, the bound and its\n",
    "equality condition follow. $\\blacksquare$\n",
    "\n",
    "the figure draws the resulting geometry: for a fixed guess the\n",
    "ELBO is a curve running everywhere below the evidence, and choosing $q$ well\n",
    "pinches the two together at the current parameters.\n",
    "\n",
    "![The evidence and its lower bound. For a fixed guess $q$, the ELBO $\\mathcal{L}(q,\\theta)$ runs below the evidence $\\log p(x;\\theta)$ at every $\\theta$, and the gap between the curves is the KL divergence from $q$ to the posterior $p(z\\mid x;\\theta)$. The E-step of EM picks $q$ to be the posterior at the current iterate $\\theta^{(t)}$, closing the gap so the bound touches the evidence there; the M-step climbs the bound to its peak $\\theta^{(t+1)}$. Because the bound touches the evidence at $\\theta^{(t)}$ and never exceeds it anywhere, the evidence at $\\theta^{(t+1)}$ is at least what it was at $\\theta^{(t)}$.](../img/mdl-prob-elbo.svg)\n",
    "\n",
    "### Expectation--Maximization\n",
    "\n",
    "The proposition turns the intractable problem into **coordinate ascent** on\n",
    "$\\mathcal{L}(q, \\boldsymbol{\\theta})$: maximize over one of the two arguments\n",
    "while the other is held fixed, then swap, and repeat. For a dataset $X = \\{x_1,\\ldots,x_n\\}$ of\n",
    "independent observations the bound applies example by example and sums, with one\n",
    "guess $q_i$ per data point. The **expectation--maximization** (EM) algorithm\n",
    "alternates the two coordinates [@Dempster.Laird.Rubin.1977]:\n",
    "\n",
    "* **E-step.** Hold $\\boldsymbol{\\theta}^{(t)}$ fixed and maximize over the\n",
    "  guesses: by the equality condition of the equation, the best\n",
    "  $q_i$ is the posterior, $q_i^{(t)}(z) = p(z \\mid x_i; \\boldsymbol{\\theta}^{(t)})$.\n",
    "  This closes every gap, making the bound *tight*:\n",
    "  $\\sum_i \\mathcal{L}(q_i^{(t)}, \\boldsymbol{\\theta}^{(t)}) = \\log p(X;\\boldsymbol{\\theta}^{(t)})$.\n",
    "* **M-step.** Hold the guesses fixed and maximize over the parameters:\n",
    "  $\\boldsymbol{\\theta}^{(t+1)} = \\mathop{\\mathrm{argmax}}_{\\boldsymbol{\\theta}} \\sum_i \\mathbb{E}_{q_i^{(t)}}\\big[\\log p(x_i, z; \\boldsymbol{\\theta})\\big]$\n",
    "  (the entropy terms do not involve $\\boldsymbol{\\theta}$). This is the *easy*,\n",
    "  complete-data objective, with soft assignments standing in for the oracle's\n",
    "  labels.\n",
    "\n",
    "**Proposition (EM never decreases the likelihood).** *Each EM iteration\n",
    "satisfies $\\log p(X;\\boldsymbol{\\theta}^{(t+1)}) \\ge \\log p(X;\\boldsymbol{\\theta}^{(t)})$.*\n",
    "\n",
    "**Proof.** Chain three facts:\n",
    "\n",
    "$$\n",
    "\\log p(X;\\boldsymbol{\\theta}^{(t+1)})\n",
    "  \\;\\ge\\; \\sum_i \\mathcal{L}\\big(q_i^{(t)}, \\boldsymbol{\\theta}^{(t+1)}\\big)\n",
    "  \\;\\ge\\; \\sum_i \\mathcal{L}\\big(q_i^{(t)}, \\boldsymbol{\\theta}^{(t)}\\big)\n",
    "  \\;=\\; \\log p(X;\\boldsymbol{\\theta}^{(t)}).\n",
    "$$\n",
    "\n",
    "The first inequality is the ELBO bound the equation at the new\n",
    "parameters; the second is the definition of the M-step; the equality is the\n",
    "tightness the E-step bought. $\\blacksquare$\n",
    "\n",
    "For the Gaussian mixture both steps are closed-form. The E-step computes\n",
    "**responsibilities**, the posterior probability that point $i$ came from\n",
    "component $k$,\n",
    "\n",
    "$$\n",
    "r_{ik} = \\frac{\\pi_k\\, \\mathcal{N}(x_i;\\, \\mu_k, \\sigma_k^2)}\n",
    "              {\\sum_{j} \\pi_j\\, \\mathcal{N}(x_i;\\, \\mu_j, \\sigma_j^2)},\n",
    "$$\n",
    "\n",
    "and the M-step refits each component by responsibility-weighted counting and\n",
    "averaging: with effective counts $n_k = \\sum_i r_{ik}$,\n",
    "\n",
    "$$\n",
    "\\pi_k = \\frac{n_k}{n}, \\qquad\n",
    "\\mu_k = \\frac{1}{n_k}\\sum_{i} r_{ik}\\, x_i, \\qquad\n",
    "\\sigma_k^2 = \\frac{1}{n_k}\\sum_{i} r_{ik}\\,(x_i - \\mu_k)^2.\n",
    "$$\n",
    "\n",
    "These are the coin's counting estimator and the Gaussian's mean and variance\n",
    "estimators, softened: instead of voting for one component, each point spreads\n",
    "its single vote across components in proportion to its responsibilities\n",
    "(exercise 10 derives the $\\mu_k$ update). The cell below runs EM on a\n",
    "two-component mixture in plain NumPy and prints the log-likelihood as it climbs."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "maximum-likelihood-em-gmm"
   },
   "source": [
    "# EM for a two-component 1-D Gaussian mixture, in plain numpy.\n",
    "rng = onp.random.default_rng(42)\n",
    "# 300 points from N(-2, 0.8^2) and 200 from N(1.5, 0.6^2)\n",
    "x = onp.concatenate([rng.normal(-2.0, 0.8, 300), rng.normal(1.5, 0.6, 200)])\n",
    "\n",
    "def gauss(x, mu, sigma):\n",
    "    return onp.exp(-0.5 * ((x - mu) / sigma)**2) / (sigma * onp.sqrt(2 * onp.pi))\n",
    "\n",
    "# Deliberately poor start: equal weights, overlapping components\n",
    "pi, mu, sigma = onp.array([0.5, 0.5]), onp.array([-0.5, 0.5]), onp.array([1.0, 1.0])\n",
    "for step in range(31):\n",
    "    # E-step: responsibilities r[i, k], the posterior of component k given x_i\n",
    "    joint = pi * gauss(x[:, None], mu, sigma)\n",
    "    r = joint / joint.sum(axis=1, keepdims=True)\n",
    "    if step % 5 == 0:  # the log-likelihood never decreases\n",
    "        print(f'step {step:2d}, log-likelihood {onp.log(joint.sum(axis=1)).sum():.2f}')\n",
    "    # M-step: refit each component by responsibility-weighted averaging\n",
    "    n_k = r.sum(axis=0)\n",
    "    pi, mu = n_k / len(x), (r * x[:, None]).sum(axis=0) / n_k\n",
    "    sigma = onp.sqrt((r * (x[:, None] - mu)**2).sum(axis=0) / n_k)\n",
    "print(f'pi = {pi.round(3)}, mu = {mu.round(3)}, sigma = {sigma.round(3)}')"
   ],
   "execution_count": 6,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "step  0, log-likelihood -1289.53\nstep  5, log-likelihood -845.69\nstep 10, log-likelihood -845.45\nstep 15, log-likelihood -845.45\nstep 20, log-likelihood -845.45\nstep 25, log-likelihood -845.45\nstep 30, log-likelihood -845.45\npi = [0.594 0.406], mu = [-2.055  1.494], sigma = [0.715 0.625]\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-maximum-likelihood-md-7 fw=all -->\n",
    "\n",
    "The trace behaves exactly as the proposition demands: the log-likelihood rises\n",
    "monotonically, gains most of its ground in the first few iterations, and then\n",
    "flatlines at a stationary point; the recovered weights, means, and scales land\n",
    "near the generating values $(0.6, 0.4)$, $(-2, 1.5)$, $(0.8, 0.6)$, the small\n",
    "residual gaps being finite-sample error, not a failure of EM. Two cautions\n",
    "temper the success. EM is *ascent*, not global optimization: the mixture\n",
    "likelihood is multimodal (at minimum because relabeling the components permutes\n",
    "the parameters without changing the model, the same identifiability failure we\n",
    "met in the estimator theory above), and EM can stall at a poor local maximum, so\n",
    "practice restarts it from several initializations. And strictly speaking the GMM\n",
    "likelihood is *unbounded*: let one component collapse onto a single data point\n",
    "with $\\sigma_k \\to 0$. Monotonicity itself guarantees only nondecrease; because this objective is\n",
    "unbounded, the values may diverge rather than approach a finite limit. With a\n",
    "variance floor or prior making the relevant level set bounded, plus standard\n",
    "continuity and regularity conditions, EM limit points are stationary\n",
    "[@Wu.1983].\n",
    "\n",
    "The bound, not the closed-form M-step, is the part that scales. A *variational\n",
    "autoencoder* [@Kingma.Welling.2014] keeps the ELBO but replaces the\n",
    "per-point E-step with a neural\n",
    "network $q_{\\boldsymbol{\\phi}}(z\\mid x)$, an *amortized* guess trained jointly\n",
    "with the generator by stochastic gradient ascent on the ELBO, because for a deep\n",
    "decoder the true posterior is no longer available in closed form. Diffusion\n",
    "models walk further down the same road: their training objective is the ELBO of\n",
    "a deep chain of latent variables (that section).\n",
    "And the two divergences in this section are deliberately mirrored:\n",
    "that section recasts maximum likelihood itself as\n",
    "*forward*-KL minimization, while the ELBO gap the equation measures\n",
    "the guess against the posterior in the *reverse* orientation,\n",
    "$D_{\\textrm{KL}}(q\\,\\|\\,p)$, a distinction with real modelling consequences,\n",
    "taken up in detail there.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* The **maximum likelihood** principle directly picks the parameter that\n",
    "  maximizes the observed-data probability or density,\n",
    "  $\\mathop{\\mathrm{argmax}}_{\\boldsymbol{\\theta}}P(X\\mid\\boldsymbol{\\theta})$.\n",
    "  It does not require a prior; MAP is a separate posterior-mode rule that\n",
    "  coincides with MLE under a constant prior in the chosen parameterization.\n",
    "* We optimize the **negative log-likelihood** $-\\sum_i \\log p(x_i\\mid\\boldsymbol{\\theta})$:\n",
    "  the log fixes underflow and makes gradients an additive sum, while the sign\n",
    "  flip turns \"maximize probability\" into \"minimize a loss.\"\n",
    "* For **discrete outcomes**, the average NLL is the cross-entropy from the\n",
    "  empirical frequencies to the model and differs from the corresponding KL by\n",
    "  a fixed empirical entropy. For continuous data, use the average negative\n",
    "  log-density at the sample; the clean KL-projection statement is a population\n",
    "  statement under suitable integrability conditions. Categorical NLL is softmax cross-entropy,\n",
    "  **fixed-variance Gaussian NLL is mean squared error**, and a Laplace model gives\n",
    "  mean absolute error: picking a loss is picking a noise model.\n",
    "* The MLE is **consistent** (given identifiability and uniform convergence) and,\n",
    "  under the regularity conditions granted in the text, **asymptotically normal**\n",
    "  with limiting variance\n",
    "  the inverse **Fisher information** $I(\\boldsymbol{\\theta}^\\star)^{-1}/n$, so\n",
    "  it attains the Cramér--Rao bound the equation and is\n",
    "  *asymptotically efficient*. At finite $n$ it is generally only\n",
    "  asymptotically unbiased (that section).\n",
    "* **MAP** estimation restores the prior, adding its negative log as a penalty: a\n",
    "  Gaussian prior is exactly $L_2$ / weight decay ($\\lambda=\\sigma^2/\\tau^2$; for\n",
    "  the averaged loss, $\\sigma^2/(n\\tau^2)$), a Laplace prior is $L_1$ / sparsity,\n",
    "  and a Beta prior on the coin adds **pseudo-counts**, with MAP under the flat\n",
    "  Beta(1,1) prior being the plain MLE, while add-one smoothing is the posterior\n",
    "  *mean* (rule of succession). MAP $\\to$ MLE as the prior flattens or the data\n",
    "  grows, and it is still a *point estimate* (the posterior mode), not full\n",
    "  Bayesian inference, which integrates over the posterior.\n",
    "* When the model has **latent variables** the likelihood is a marginal with a\n",
    "  sum inside the log. The **ELBO** lower-bounds it for every guess $q$, with gap\n",
    "  exactly $D_{\\textrm{KL}}(q \\,\\|\\, p(z\\mid x;\\boldsymbol{\\theta}))$; **EM**\n",
    "  alternates closing the gap (E-step: $q$ = posterior) with climbing the bound\n",
    "  (M-step), and never decreases the likelihood. VAEs and diffusion models train\n",
    "  on the same bound with amortized guesses.\n",
    "* Maximum likelihood extends to continuous variables unchanged: probabilities\n",
    "  become densities, and the matching-tolerance $\\epsilon$ drops out as a constant.\n",
    "\n",
    "## Exercises\n",
    "1. A non-negative random variable has density $\\alpha e^{-\\alpha x}$ for some\n",
    "   $\\alpha>0$. From the single observation $x=3$, find the maximum likelihood\n",
    "   estimate of $\\alpha$. Generalize to a sample $\\{x_i\\}_{i=1}^N$ and show\n",
    "   $\\hat\\alpha = 1/\\bar x$.\n",
    "2. Given a sample $\\{x_i\\}_{i=1}^N$ from a Gaussian with unknown mean and\n",
    "   variance $1$, show the maximum likelihood estimate of the mean is the sample\n",
    "   average $\\bar x$. (*Hint:* minimize the Gaussian NLL of\n",
    "   that section.)\n",
    "3. Show directly that a fixed-variance Gaussian NLL equals the mean squared error\n",
    "   up to an additive constant, and identify the constant.\n",
    "4. For a $K$-class classifier with one-hot label $\\mathbf y$ and softmax\n",
    "   prediction $\\hat{\\mathbf p} = \\mathrm{softmax}(\\mathbf o)$, where the vector\n",
    "   $\\mathbf o$ collects the *logits*, the raw scores fed to the softmax\n",
    "   (that section), show the categorical NLL $-\\sum_k y_k\\log\\hat p_k$\n",
    "   is the cross-entropy, and that its gradient with respect to the logits is\n",
    "   $\\hat{\\mathbf p} - \\mathbf y$ (compare the equation).\n",
    "5. Show that a Gaussian prior on the regression weights recovers ridge\n",
    "   regression, and read off the weight-decay strength in terms of the noise and\n",
    "   prior variances.\n",
    "6. Show that MAP estimation with a Laplace prior gives an $L_1$ penalty, and\n",
    "   argue why its kink at the origin can force coefficients to be exactly zero.\n",
    "7. Prove that $\\hat{\\boldsymbol{\\theta}}_{\\textrm{MAP}} \\to \\hat{\\boldsymbol{\\theta}}_{\\textrm{MLE}}$\n",
    "   as the prior variance $\\tau^2\\to\\infty$, and explain why the same happens as\n",
    "   the dataset size $n\\to\\infty$ for fixed $\\tau$.\n",
    "8. For a single Gaussian observation with known variance $\\sigma^2$ and unknown\n",
    "   mean $\\mu$, compute the score $\\frac{d}{d\\mu}\\log p(x\\mid\\mu)$ and show the\n",
    "   Fisher information is $I(\\mu)=1/\\sigma^2$. Conclude that the sample-mean MLE\n",
    "   over $n$ observations has variance $\\sigma^2/n$, attaining the Cramér--Rao\n",
    "   bound the equation exactly. (*Hint:* use the equation.)\n",
    "9. Extend the asymptotic-normality demonstration in the text. First rerun it\n",
    "   with $n = 10$ in place of $400$: how many distinct values can $\\hat\\theta$\n",
    "   take, and what does the \"histogram\" become? Then restore $n = 400$ but move\n",
    "   $\\theta^\\star$ to $0.95$ and compare the match to the predicted Gaussian.\n",
    "   Explain why convergence is slower near the boundary, considering both the\n",
    "   skew of the binomial there and the shrinking variance\n",
    "   $\\theta^\\star(1-\\theta^\\star)$.\n",
    "10. Derive the GMM M-step for the means: holding the responsibilities $r_{ik}$\n",
    "    fixed, show that maximizing the expected complete-data log-likelihood\n",
    "    $\\sum_i \\sum_k r_{ik}\\big[\\log \\pi_k + \\log \\mathcal{N}(x_i;\\,\\mu_k,\\sigma_k^2)\\big]$\n",
    "    over $\\mu_k$ yields the responsibility-weighted mean of\n",
    "    the equation. Then derive the $\\pi_k$ update by maximizing over\n",
    "    the weights subject to $\\sum_k \\pi_k = 1$. (*Hint:* use a Lagrange\n",
    "    multiplier for the constraint.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-maximum-likelihood-md-8 fw=pytorch -->\n",
    "\n",
    "[Discussions](https://d2l.discourse.group/t/1096)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-pytorch",
   "display_name": "d2l (PyTorch)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
