{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-1 fw=all -->\n",
    "\n",
    "# Distributions\n",
    "\n",
    "A *distribution* assigns a probability to every outcome a random variable can\n",
    "take (that section). A working deep-learning practitioner\n",
    "does not need a zoo of hundreds: the fourteen distributions of this section\n",
    "cover almost everything, and they form a small family with a few generators and\n",
    "a handful of limiting arrows connecting them.\n",
    "The Bernoulli coin flip is the seed: sum $n$ of them and you get the binomial;\n",
    "let the trials become many and rare and the binomial collapses to the Poisson;\n",
    "let them become many and ordinary and the *central limit theorem* sends it to the\n",
    "Gaussian. The categorical generalizes the Bernoulli from two outcomes to $K$, and\n",
    "its softmax parameterization is the output layer of every classifier. At the end\n",
    "we will see that almost all of these (and more) are special cases of one form,\n",
    "the *exponential family*, which is the reason their maximum-likelihood losses are the\n",
    "convex objectives we minimize in practice and the reason each has a\n",
    "*conjugate prior* that makes Bayesian updating as simple as counting.\n",
    "\n",
    "the figure draws the picture: the distributions are\n",
    "nodes, the construction and limit relationships are arrows, and the whole\n",
    "picture sits inside the exponential-family envelope.\n",
    "\n",
    "![The distribution family. Solid arrows *construct*: Bernoulli is the seed; summing $n$ copies gives the Binomial; the many-and-rare limit ($n\\to\\infty$, $np\\to\\lambda$) gives the Poisson, whose waiting time between events is the Exponential; the many-and-ordinary limit (the central limit theorem) gives the Gaussian; the Categorical generalizes Bernoulli to $K$ outcomes and the Multinomial counts $n$ of them. Dashed arrows attach each likelihood to its *conjugate prior*: the Beta to the Bernoulli and Binomial, the Gamma to the Poisson, and the Dirichlet to the Categorical and Multinomial. Every node, the conjugate priors included, lies inside the exponential-family envelope.](../img/mdl-prob-family-tree.svg)\n",
    "\n",
    "For each distribution we keep to a tight template: its mass or density function,\n",
    "where it *arises* in machine learning, its mean and variance *derived* rather than\n",
    "asserted, and a compact teaching cell that evaluates the law and draws a\n",
    "sample. We treat the discrete distributions first, then the continuous ones, then\n",
    "unify them, using the following imports throughout."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "distributions-imports"
   },
   "source": [
    "%matplotlib inline\n",
    "from d2l import mxnet as d2l\n",
    "import numpy as np"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-2 fw=mxnet -->\n",
    "\n",
    "This section works in plain NumPy: every computation here is elementary array\n",
    "arithmetic, so nothing beyond NumPy is needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-3 fw=all -->\n",
    "\n",
    "## Discrete Distributions\n",
    "\n",
    "A discrete random variable takes values in a countable set, and its distribution\n",
    "is a *probability mass function* (pmf) $p_k = P(X=k)$ with $p_k\\ge0$ and\n",
    "$\\sum_k p_k=1$. Mean and variance are the sums\n",
    "$\\mathbb E[X]=\\sum_k k\\,p_k$ and\n",
    "$\\operatorname{Var}(X)=\\mathbb E[X^2]-\\mathbb E[X]^2$\n",
    "(that section). The five distributions here form a\n",
    "connected chain: each is built from, or is a limit of, the one\n",
    "before. the figure shows their shapes side by side so the\n",
    "shared bell-like silhouette of the count distributions is visible at a glance.\n",
    "\n",
    "![A gallery of discrete probability mass functions in one consistent style: $\\mathrm{Bernoulli}(0.3)$, a discrete uniform on $\\{1,\\dots,6\\}$, $\\mathrm{Binomial}(10,0.4)$, and $\\mathrm{Poisson}(4)$. The binomial and Poisson share the same right-skewed bell silhouette, foreshadowing their limiting relationship.](../img/mdl-prob-discrete-pmfs.svg)\n",
    "\n",
    "### Bernoulli\n",
    "\n",
    "The Bernoulli is the simplest non-trivial random variable: a single coin flip that\n",
    "comes up $1$ with probability $p$ and $0$ with probability $1-p$. We write\n",
    "$X\\sim\\mathrm{Bernoulli}(p)$, with pmf\n",
    "\n",
    "$$\n",
    "p_1 = P(X=1) = p, \\qquad p_0 = P(X=0) = 1-p .\n",
    "$$\n",
    "\n",
    "**Where it arises.** The Bernoulli is the output distribution of *every binary\n",
    "classifier*: a model emits $\\hat p$, and the negative log-likelihood\n",
    "$-y\\log\\hat p-(1-y)\\log(1-\\hat p)$ is exactly the *binary cross-entropy* loss,\n",
    "the maximum-likelihood view of binary classification developed in\n",
    "that section.\n",
    "\n",
    "**Mean and variance.** Because $X$ takes only the values $0$ and $1$, we have\n",
    "$X^2=X$, which makes both moments immediate:\n",
    "\n",
    "$$\n",
    "\\mu_X = \\mathbb E[X] = 1\\cdot p + 0\\cdot(1-p) = p,\n",
    "\\qquad\n",
    "\\sigma_X^2 = \\mathbb E[X^2]-\\mu_X^2 = p - p^2 = p(1-p).\n",
    "$$\n",
    "\n",
    "The variance $p(1-p)$ is largest at $p=\\tfrac12$ (the fairest, least predictable\n",
    "coin) and vanishes at $p\\in\\{0,1\\}$ (a certain outcome carries no randomness)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "distributions-bernoulli"
   },
   "source": [
    "import numpy as onp\n",
    "rng = onp.random.default_rng(0)\n",
    "p = 0.3\n",
    "sample = 1 * (rng.random((3, 3)) < p)          # 1 with prob p, else 0\n",
    "big = 1 * (rng.random(10000) < p)              # large sample for the mean\n",
    "print('pmf  P(0), P(1) =', (1 - p, p))\n",
    "print('mean of 10,000 draws =', float(big.mean()), ' (≈ p)')\n",
    "sample"
   ],
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "pmf  P(0), P(1) = (0.7, 0.3)\nmean of 10,000 draws = 0.3006  (≈ p)\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "array([[0, 1, 1],\n       [1, 0, 0],\n       [0, 0, 0]])"
     },
     "metadata": {},
     "execution_count": 2
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-4 fw=all -->\n",
    "\n",
    "### Categorical and Multinomial\n",
    "\n",
    "The *categorical* distribution generalizes the Bernoulli from two outcomes to $K$.\n",
    "It is parameterized by a probability vector $\\mathbf p=(p_1,\\dots,p_K)$ with\n",
    "$p_k\\ge0$ and $\\sum_k p_k=1$, and a draw lands on class $k$ with probability $p_k$,\n",
    "\n",
    "$$\n",
    "P(X=k) = p_k .\n",
    "$$\n",
    "\n",
    "**Where it arises.** Every *softmax* output layer turns a vector of scores\n",
    "(logits) $\\mathbf z$ into a categorical over classes or vocabulary tokens via\n",
    "\n",
    "$$\n",
    "p_k = \\frac{e^{z_k}}{\\sum_{j=1}^{K} e^{z_j}},\n",
    "$$\n",
    "\n",
    "and the negative log-likelihood of the categorical,\n",
    "$-\\sum_k y_k\\log p_k$ with $\\mathbf y$ the one-hot label, is exactly the softmax\n",
    "*cross-entropy* loss (that section, that section).\n",
    "The Bernoulli is the case $K=2$. To *sample* a class rather than score one, the\n",
    "*Gumbel-max trick* perturbs each logit with independent Gumbel noise (the Gumbel\n",
    "distribution is the location--scale law of the maxima of many independent draws\n",
    "[@Gumbel.1954]) and takes the argmax: an exact categorical sampler, and the\n",
    "basis of the Gumbel-softmax relaxation that lets gradients flow through discrete\n",
    "choices [@Jang.Gu.Poole.2017; @Maddison.Mnih.Teh.2017].\n",
    "\n",
    "The *multinomial* is its count version, the categorical analogue of how the\n",
    "binomial sums Bernoullis. Drawing $n$ independent categorical samples and counting\n",
    "how often each class occurs gives a vector $\\mathbf x=(x_1,\\dots,x_K)$ with\n",
    "$\\sum_k x_k=n$ and pmf\n",
    "\n",
    "$$\n",
    "P(\\mathbf x) = \\frac{n!}{x_1!\\cdots x_K!}\\,\\prod_{k=1}^{K} p_k^{x_k},\n",
    "$$\n",
    "\n",
    "the multinomial coefficient counting the orderings that yield those counts, times\n",
    "the probability $\\prod_k p_k^{x_k}$ of any one such ordering. For $K=2$ this is the\n",
    "binomial of the next section.\n",
    "\n",
    "The teaching cell turns a logit vector into a categorical via softmax, confirms it\n",
    "normalizes, and draws one sample."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "distributions-categorical"
   },
   "source": [
    "import numpy as onp\n",
    "rng = onp.random.default_rng(0)\n",
    "z = onp.array([2.0, 1.0, 0.1, -1.0])            # logits over K = 4 classes\n",
    "p_cat = onp.exp(z) / onp.exp(z).sum()            # softmax -> categorical\n",
    "draw = int(rng.choice(len(p_cat), p=p_cat))\n",
    "print('categorical p =', p_cat.round(3), ' sum =', float(p_cat.sum()))\n",
    "print('one sample -> class', draw)"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "categorical p = [0.638 0.235 0.095 0.032]  sum = 1.0\none sample -> class 0\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-5 fw=all -->\n",
    "\n",
    "### Discrete Uniform\n",
    "\n",
    "If a categorical assigns *equal* probability to each of its outcomes it is the\n",
    "*discrete uniform*. Supported on $\\{1,2,\\dots,n\\}$, every value is equally likely,\n",
    "$p_i=\\tfrac1n$, written $X\\sim U(n)$.\n",
    "\n",
    "**Where it arises.** It models a fair die, a uniformly chosen index, or a token\n",
    "sampled with no preference. It is also the maximum-entropy distribution on a\n",
    "finite set, a fact we take as given [@Cover.Thomas.1999] and place in\n",
    "context when we meet the exponential family below.\n",
    "\n",
    "**Mean and variance.** Both follow from the closed forms\n",
    "$\\sum_{i=1}^n i=\\tfrac{n(n+1)}2$ and $\\sum_{i=1}^n i^2=\\tfrac{n(n+1)(2n+1)}6$:\n",
    "\n",
    "$$\n",
    "\\mu_X = \\frac1n\\sum_{i=1}^n i = \\frac{n+1}{2},\n",
    "\\qquad\n",
    "\\sigma_X^2 = \\frac1n\\sum_{i=1}^n i^2 - \\mu_X^2 = \\frac{n^2-1}{12}.\n",
    "$$\n",
    "\n",
    "The mean is the midpoint, by symmetry; the variance grows like $n^2$, since a wider\n",
    "range of equally likely values is more spread out. The cell checks both formulas\n",
    "against a large sample of die rolls."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "distributions-discrete-uniform"
   },
   "source": [
    "rng = np.random.default_rng(0)\n",
    "n = 6\n",
    "sample = rng.integers(1, n + 1, 10000)           # 10,000 die rolls\n",
    "print('mean (n+1)/2  =', (n + 1) / 2, '   sample mean:', float(sample.mean()))\n",
    "print('var (n^2-1)/12 =', round((n**2 - 1) / 12, 3), ' sample var :',\n",
    "      float(sample.var().round(3)))"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "mean (n+1)/2  = 3.5    sample mean: 3.5045\nvar (n^2-1)/12 = 2.917  sample var : 2.937\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-6 fw=all -->\n",
    "\n",
    "### Binomial\n",
    "\n",
    "Sum $n$ independent Bernoulli flips and you count *successes*: this is the\n",
    "*binomial*. Writing each flip as $X_i\\sim\\mathrm{Bernoulli}(p)$ with $1$ for success,\n",
    "\n",
    "$$\n",
    "X = \\sum_{i=1}^{n} X_i \\sim \\mathrm{Binomial}(n,p).\n",
    "$$\n",
    "\n",
    "Exactly $k$ successes can occur in $\\binom nk$ orderings, each of probability\n",
    "$p^k(1-p)^{n-k}$, so\n",
    "\n",
    "$$\n",
    "P(X=k) = \\binom{n}{k}\\,p^k(1-p)^{n-k}, \\qquad k=0,1,\\dots,n.\n",
    "$$\n",
    "\n",
    "**Where it arises.** It models any \"how many out of $n$\" count with a fixed\n",
    "per-trial probability: clicks among impressions, correct predictions on a held-out\n",
    "batch, heads in a run of flips.\n",
    "\n",
    "**Mean and variance, via the sum.** Rather than wrestle with the binomial\n",
    "coefficient, use the representation the equation as a sum of\n",
    "Bernoullis. *Expectation is linear* (always, even for dependent terms), so\n",
    "\n",
    "$$\n",
    "\\mu_X = \\mathbb E\\Bigl[\\sum_i X_i\\Bigr] = \\sum_i \\mathbb E[X_i] = np .\n",
    "$$\n",
    "\n",
    "The $X_i$ are *independent*, and variance is additive over independent summands, so\n",
    "\n",
    "$$\n",
    "\\sigma_X^2 = \\operatorname{Var}\\Bigl(\\sum_i X_i\\Bigr) = \\sum_i \\operatorname{Var}(X_i) = np(1-p) .\n",
    "$$\n",
    "\n",
    "Two one-line sums replace a page of algebra: the payoff of seeing the binomial as\n",
    "*built from* Bernoullis. Successive pmf terms differ by the ratio\n",
    "\n",
    "$$\n",
    "\\frac{P(X=k+1)}{P(X=k)} = \\frac{n-k}{k+1}\\cdot\\frac{p}{1-p},\n",
    "$$\n",
    "\n",
    "so the whole pmf builds up from $P(X=0)=(1-p)^n$ by repeated multiplication, with\n",
    "never a factorial in sight."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "distributions-binomial"
   },
   "source": [
    "import numpy as onp\n",
    "rng = onp.random.default_rng(0)\n",
    "n, p = 10, 0.4\n",
    "pmf = onp.zeros(n + 1)\n",
    "pmf[0] = (1 - p)**n\n",
    "for k in range(n):                               # successive-ratio recursion\n",
    "    pmf[k + 1] = pmf[k] * (n - k) / (k + 1) * p / (1 - p)\n",
    "print('mean onp =', n * p, '  var onp(1-p) =', n * p * (1 - p))\n",
    "print('P(X=k):', pmf.round(3))\n",
    "rng.binomial(n, p, size=(3, 3))                  # sample counts of successes"
   ],
   "execution_count": 5,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "mean onp = 4.0   var onp(1-p) = 2.4\nP(X=k): [0.006 0.04  0.121 0.215 0.251 0.201 0.111 0.042 0.011 0.002 0.   ]\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "array([[5, 3, 1],\n       [1, 5, 6],\n       [4, 5, 4]])"
     },
     "metadata": {},
     "execution_count": 5
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-7 fw=all -->\n",
    "\n",
    "### Poisson\n",
    "\n",
    "What if the trials become *many* and each *rare*? Standing at a bus stop, the\n",
    "chance of an arrival in any tiny sub-interval is small, but there are many such\n",
    "intervals. Let $\\lambda$ be the expected number of arrivals per minute. Split the\n",
    "minute into $n$ slices, model each as $\\mathrm{Bernoulli}(\\lambda/n)$, and the\n",
    "count is $\\mathrm{Binomial}(n,\\lambda/n)$. Its mean is $n\\cdot\\lambda/n=\\lambda$\n",
    "for every $n$, and its variance\n",
    "$n\\cdot\\tfrac{\\lambda}{n}(1-\\tfrac{\\lambda}{n})\\to\\lambda$.\n",
    "The moments stabilize, which signals that a limiting distribution exists.\n",
    "\n",
    "**The limit, derived.** Take $\\mathrm{Binomial}(n,p_n)$ with $n\\to\\infty$ and\n",
    "$p_n\\to0$ such that $np_n\\to\\lambda$, the *law of rare events*. Writing\n",
    "$p_n=\\lambda/n$, the pmf at a fixed $k$ factors into four pieces:\n",
    "\n",
    "$$\n",
    "\\binom{n}{k}\\Bigl(\\frac\\lambda n\\Bigr)^k\\Bigl(1-\\frac\\lambda n\\Bigr)^{n-k}\n",
    "= \\underbrace{\\frac{n!}{(n-k)!\\,n^k}}_{\\to\\,1}\n",
    "  \\cdot \\frac{\\lambda^k}{k!}\n",
    "  \\cdot \\underbrace{\\Bigl(1-\\frac\\lambda n\\Bigr)^{n}}_{\\to\\,e^{-\\lambda}}\n",
    "  \\cdot \\underbrace{\\Bigl(1-\\frac\\lambda n\\Bigr)^{-k}}_{\\to\\,1} .\n",
    "$$\n",
    "\n",
    "The first factor is $\\tfrac{n(n-1)\\cdots(n-k+1)}{n^k}\\to1$, the third is the\n",
    "classic limit $(1-\\lambda/n)^n\\to e^{-\\lambda}$, and the last $\\to1$. Collecting\n",
    "the survivors gives the *Poisson* pmf:\n",
    "\n",
    "$$\n",
    "P(X=k) = \\frac{\\lambda^k e^{-\\lambda}}{k!}, \\qquad k=0,1,2,\\dots\n",
    "$$\n",
    "\n",
    "with $\\lambda>0$ the *rate*, the average count per unit time; we write\n",
    "$X\\sim\\mathrm{Poisson}(\\lambda)$.\n",
    "\n",
    "**Where it arises.** Counts of rare events with no fixed ceiling: click counts,\n",
    "photons on a sensor, mutations in a genome, requests hitting a server. Run over\n",
    "the whole timeline, the bus-stop slicing above is the *Poisson process*: the\n",
    "event counts in disjoint time windows are independent Poisson variables.\n",
    "\n",
    "**Mean and variance.** The binomial moments suggest the answer, but\n",
    "convergence in distribution alone does not automatically pass unbounded\n",
    "moments to the limit. The Poisson pmf verifies them directly:\n",
    "\n",
    "$$\n",
    "E[X]=\\sum_{k\\ge1}k e^{-\\lambda}\\frac{\\lambda^k}{k!}\n",
    "=\\lambda,\n",
    "\\qquad\n",
    "E[X(X-1)]=\\lambda^2.\n",
    "$$\n",
    "\n",
    "Therefore\n",
    "\n",
    "$$\n",
    "\\mu_X=\\lambda,\n",
    "\\qquad\n",
    "\\sigma_X^2=E[X(X-1)]+E[X]-E[X]^2=\\lambda.\n",
    "$$\n",
    "\n",
    "Mean equals variance is the Poisson fingerprint: over-dispersed count data (where\n",
    "the empirical variance exceeds the mean) is the standard sign that a plain Poisson\n",
    "model is too simple. The standard fix, a Poisson whose rate is *itself random*,\n",
    "reappears later in this section as the Gamma--Poisson mixture, better known as the\n",
    "*negative binomial*."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "distributions-poisson"
   },
   "source": [
    "import numpy as onp\n",
    "rng = onp.random.default_rng(0)\n",
    "lam = 4.0\n",
    "k = onp.arange(15)\n",
    "pmf = onp.empty_like(k, dtype=float)\n",
    "pmf[0] = onp.exp(-lam)\n",
    "for j in range(len(k) - 1):                    # P(j+1) / P(j) = lambda / (j+1)\n",
    "    pmf[j + 1] = pmf[j] * lam / (j + 1)\n",
    "print('mean = var = lambda =', lam)\n",
    "print('P(X=k):', pmf.round(3))\n",
    "rng.poisson(lam, size=(3, 3))                   # sample event counts"
   ],
   "execution_count": 6,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "mean = var = lambda = 4.0\nP(X=k): [0.018 0.073 0.147 0.195 0.195 0.156 0.104 0.06  0.03  0.013 0.005 0.002\n 0.001 0.    0.   ]\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "array([[2, 0, 7],\n       [3, 4, 4],\n       [6, 6, 4]])"
     },
     "metadata": {},
     "execution_count": 6
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-8 fw=all -->\n",
    "\n",
    "**Watching the limit happen.** The limit is visible in a plot. The cell below\n",
    "overlays the pmf of $\\mathrm{Binomial}(n,\\lambda/n)$ for\n",
    "growing $n$ on its $\\mathrm{Poisson}(\\lambda)$ limit, building every pmf by the\n",
    "same successive-ratio trick we used for the binomial (for the Poisson the ratio is\n",
    "simply $P(X=k+1)/P(X=k)=\\lambda/(k+1)$). Already at $n=20$ the curves are visibly\n",
    "closing in on each other, and at $n=100$ they coincide to plotting accuracy."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "distributions-binomial-to-poisson"
   },
   "source": [
    "lam, ks = 4.0, np.arange(15)\n",
    "\n",
    "def binomial_pmf(n, p):                # successive-ratio recursion, no factorials\n",
    "    pmf = [(1 - p)**n]\n",
    "    for k in range(len(ks) - 1):\n",
    "        pmf.append(pmf[-1] * max(n - k, 0) / (k + 1) * p / (1 - p))\n",
    "    return np.array(pmf)\n",
    "\n",
    "poisson_pmf = [np.exp(-lam)]\n",
    "for k in range(len(ks) - 1):\n",
    "    poisson_pmf.append(poisson_pmf[-1] * lam / (k + 1))\n",
    "d2l.plot(ks, [binomial_pmf(n, lam / n) for n in (5, 20, 100)]\n",
    "         + [np.array(poisson_pmf)], xlabel='k', ylabel='P(X=k)',\n",
    "         legend=[f'Binomial({n}, 4/{n})' for n in (5, 20, 100)] + ['Poisson(4)'])"
   ],
   "execution_count": 7,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"246.28125pt\" height=\"183.35625pt\" viewBox=\"0 0 246.28125 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-19T14:02:27.661494</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 183.35625 \nL 246.28125 183.35625 \nL 246.28125 0 \nL 0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 43.78125 145.8 \nL 239.08125 145.8 \nL 239.08125 7.2 \nL 43.78125 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path d=\"M 52.658523 145.8 \nL 52.658523 7.2 \n\" clip-path=\"url(#pedab15c7e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path id=\"m3cea282285\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m3cea282285\" x=\"52.658523\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <g transform=\"translate(49.477273 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path d=\"M 116.067614 145.8 \nL 116.067614 7.2 \n\" clip-path=\"url(#pedab15c7e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#m3cea282285\" x=\"116.067614\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 5 -->\n      <g transform=\"translate(112.886364 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \nL 3169 4666 \nL 3169 4134 \nL 1269 4134 \nL 1269 2991 \nQ 1406 3038 1543 3061 \nQ 1681 3084 1819 3084 \nQ 2600 3084 3056 2656 \nQ 3513 2228 3513 1497 \nQ 3513 744 3044 326 \nQ 2575 -91 1722 -91 \nQ 1428 -91 1123 -41 \nQ 819 9 494 109 \nL 494 744 \nQ 775 591 1075 516 \nQ 1375 441 1709 441 \nQ 2250 441 2565 725 \nQ 2881 1009 2881 1497 \nQ 2881 1984 2565 2268 \nQ 2250 2553 1709 2553 \nQ 1456 2553 1204 2497 \nQ 953 2441 691 2322 \nL 691 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-35\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path d=\"M 179.476705 145.8 \nL 179.476705 7.2 \n\" clip-path=\"url(#pedab15c7e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#m3cea282285\" x=\"179.476705\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 10 -->\n      <g transform=\"translate(173.114205 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_4\">\n     <!-- k -->\n     <g transform=\"translate(138.535938 174.076563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-6b\" d=\"M 581 4863 \nL 1159 4863 \nL 1159 1991 \nL 2875 3500 \nL 3609 3500 \nL 1753 1863 \nL 3688 0 \nL 2938 0 \nL 1159 1709 \nL 1159 0 \nL 581 0 \nL 581 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-6b\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_7\">\n      <path d=\"M 43.78125 139.5 \nL 239.08125 139.5 \n\" clip-path=\"url(#pedab15c7e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_8\">\n      <defs>\n       <path id=\"meb74e73a5d\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#meb74e73a5d\" x=\"43.78125\" y=\"139.5\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 0.0 -->\n      <g transform=\"translate(20.878125 143.299219) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \nL 1344 794 \nL 1344 0 \nL 684 0 \nL 684 794 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_9\">\n      <path d=\"M 43.78125 108.738281 \nL 239.08125 108.738281 \n\" clip-path=\"url(#pedab15c7e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#meb74e73a5d\" x=\"43.78125\" y=\"108.738281\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 0.1 -->\n      <g transform=\"translate(20.878125 112.5375) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_11\">\n      <path d=\"M 43.78125 77.976562 \nL 239.08125 77.976562 \n\" clip-path=\"url(#pedab15c7e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_12\">\n      <g>\n       <use xlink:href=\"#meb74e73a5d\" x=\"43.78125\" y=\"77.976562\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_7\">\n      <!-- 0.2 -->\n      <g transform=\"translate(20.878125 81.775781) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_13\">\n      <path d=\"M 43.78125 47.214844 \nL 239.08125 47.214844 \n\" clip-path=\"url(#pedab15c7e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_14\">\n      <g>\n       <use xlink:href=\"#meb74e73a5d\" x=\"43.78125\" y=\"47.214844\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.3 -->\n      <g transform=\"translate(20.878125 51.014063) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \nQ 3050 2419 3304 2112 \nQ 3559 1806 3559 1356 \nQ 3559 666 3084 287 \nQ 2609 -91 1734 -91 \nQ 1441 -91 1130 -33 \nQ 819 25 488 141 \nL 488 750 \nQ 750 597 1062 519 \nQ 1375 441 1716 441 \nQ 2309 441 2620 675 \nQ 2931 909 2931 1356 \nQ 2931 1769 2642 2001 \nQ 2353 2234 1838 2234 \nL 1294 2234 \nL 1294 2753 \nL 1863 2753 \nQ 2328 2753 2575 2939 \nQ 2822 3125 2822 3475 \nQ 2822 3834 2567 4026 \nQ 2313 4219 1838 4219 \nQ 1578 4219 1281 4162 \nQ 984 4106 628 3988 \nL 628 4550 \nQ 988 4650 1302 4700 \nQ 1616 4750 1894 4750 \nQ 2613 4750 3031 4423 \nQ 3450 4097 3450 3541 \nQ 3450 3153 3228 2886 \nQ 3006 2619 2597 2516 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_15\">\n      <path d=\"M 43.78125 16.453125 \nL 239.08125 16.453125 \n\" clip-path=\"url(#pedab15c7e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_16\">\n      <g>\n       <use xlink:href=\"#meb74e73a5d\" x=\"43.78125\" y=\"16.453125\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.4 -->\n      <g transform=\"translate(20.878125 20.252344) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_10\">\n     <!-- P(X=k) -->\n     <g transform=\"translate(14.798438 93.926563) rotate(-90) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \nL 1259 2394 \nL 2053 2394 \nQ 2494 2394 2734 2622 \nQ 2975 2850 2975 3272 \nQ 2975 3691 2734 3919 \nQ 2494 4147 2053 4147 \nL 1259 4147 \nz\nM 628 4666 \nL 2053 4666 \nQ 2838 4666 3239 4311 \nQ 3641 3956 3641 3272 \nQ 3641 2581 3239 2228 \nQ 2838 1875 2053 1875 \nL 1259 1875 \nL 1259 0 \nL 628 0 \nL 628 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \nQ 1566 4138 1362 3434 \nQ 1159 2731 1159 2009 \nQ 1159 1288 1364 580 \nQ 1569 -128 1984 -844 \nL 1484 -844 \nQ 1016 -109 783 600 \nQ 550 1309 550 2009 \nQ 550 2706 781 3412 \nQ 1013 4119 1484 4856 \nL 1984 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-58\" d=\"M 403 4666 \nL 1081 4666 \nL 2241 2931 \nL 3406 4666 \nL 4084 4666 \nL 2584 2425 \nL 4184 0 \nL 3506 0 \nL 2194 1984 \nL 872 0 \nL 191 0 \nL 1856 2491 \nL 403 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-3d\" d=\"M 678 2906 \nL 4684 2906 \nL 4684 2381 \nL 678 2381 \nL 678 2906 \nz\nM 678 1631 \nL 4684 1631 \nL 4684 1100 \nL 678 1100 \nL 678 1631 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \nL 1013 4856 \nQ 1481 4119 1714 3412 \nQ 1947 2706 1947 2009 \nQ 1947 1309 1714 600 \nQ 1481 -109 1013 -844 \nL 513 -844 \nQ 928 -128 1133 580 \nQ 1338 1288 1338 2009 \nQ 1338 2731 1133 3434 \nQ 928 4138 513 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-50\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(60.302734 0)\"/>\n      <use xlink:href=\"#DejaVuSans-58\" transform=\"translate(99.316406 0)\"/>\n      <use xlink:href=\"#DejaVuSans-3d\" transform=\"translate(167.822266 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6b\" transform=\"translate(251.611328 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(309.521484 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_17\">\n    <path d=\"M 52.658523 139.401563 \nL 65.340341 137.53125 \nL 78.022159 123.75 \nL 90.703977 76.5 \nL 103.385795 13.5 \nL 116.067614 38.7 \nL 128.749432 139.5 \nL 141.43125 139.5 \nL 154.113068 139.5 \nL 166.794886 139.5 \nL 179.476705 139.5 \nL 192.158523 139.5 \nL 204.840341 139.5 \nL 217.522159 139.5 \nL 230.203977 139.5 \n\" clip-path=\"url(#pedab15c7e5)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_18\">\n    <path d=\"M 52.658523 135.953415 \nL 65.340341 121.767076 \nL 78.022159 97.384307 \nL 90.703977 76.32646 \nL 103.385795 72.378114 \nL 116.067614 85.802491 \nL 128.749432 105.939057 \nL 141.43125 122.719528 \nL 154.113068 132.682933 \nL 166.794886 137.227644 \nL 179.476705 138.875102 \nL 192.158523 139.357978 \nL 204.840341 139.473371 \nL 217.522159 139.495903 \nL 230.203977 139.499488 \n\" clip-path=\"url(#pedab15c7e5)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"line2d_19\">\n    <path d=\"M 52.658523 134.3104 \nL 65.340341 117.876666 \nL 78.022159 94.901873 \nL 90.703977 78.796994 \nL 103.385795 78.164671 \nL 116.067614 90.431737 \nL 128.749432 107.128577 \nL 141.43125 121.387418 \nL 154.113068 130.726718 \nL 166.794886 135.763232 \nL 179.476705 138.083142 \nL 192.158523 139.01698 \nL 204.840341 139.350733 \nL 217.522159 139.457899 \nL 230.203977 139.489099 \n\" clip-path=\"url(#pedab15c7e5)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"line2d_20\">\n    <path d=\"M 52.658523 133.865795 \nL 65.340341 116.963179 \nL 78.022159 94.426357 \nL 90.703977 79.40181 \nL 103.385795 79.40181 \nL 116.067614 91.421448 \nL 128.749432 107.447632 \nL 141.43125 121.184361 \nL 154.113068 130.342181 \nL 166.794886 135.429858 \nL 179.476705 137.871943 \nL 192.158523 138.907979 \nL 204.840341 139.30266 \nL 217.522159 139.43928 \nL 230.203977 139.482651 \n\" clip-path=\"url(#pedab15c7e5)\" style=\"fill: none; stroke-dasharray: 1.5,2.475; stroke-dashoffset: 0; stroke: #ff0000; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 43.78125 145.8 \nL 43.78125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 239.08125 145.8 \nL 239.08125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 43.78125 145.8 \nL 239.08125 145.8 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 43.78125 7.2 \nL 239.08125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 94.495313 73.9125 \nL 232.08125 73.9125 \nQ 234.08125 73.9125 234.08125 71.9125 \nL 234.08125 14.2 \nQ 234.08125 12.2 232.08125 12.2 \nL 94.495313 12.2 \nQ 92.495313 12.2 92.495313 14.2 \nL 92.495313 71.9125 \nQ 92.495313 73.9125 94.495313 73.9125 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_21\">\n     <path d=\"M 96.495313 20.298438 \nL 106.495313 20.298438 \nL 116.495313 20.298438 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_11\">\n     <!-- Binomial(5, 4/5) -->\n     <g transform=\"translate(124.495313 23.798438) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \nL 1259 519 \nL 2272 519 \nQ 2781 519 3026 730 \nQ 3272 941 3272 1375 \nQ 3272 1813 3026 2020 \nQ 2781 2228 2272 2228 \nL 1259 2228 \nz\nM 1259 4147 \nL 1259 2741 \nL 2194 2741 \nQ 2656 2741 2882 2914 \nQ 3109 3088 3109 3444 \nQ 3109 3797 2882 3972 \nQ 2656 4147 2194 4147 \nL 1259 4147 \nz\nM 628 4666 \nL 2241 4666 \nQ 2963 4666 3353 4366 \nQ 3744 4066 3744 3513 \nQ 3744 3084 3544 2831 \nQ 3344 2578 2956 2516 \nQ 3422 2416 3680 2098 \nQ 3938 1781 3938 1306 \nQ 3938 681 3513 340 \nQ 3088 0 2303 0 \nL 628 0 \nL 628 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \nQ 3544 3216 3844 3400 \nQ 4144 3584 4550 3584 \nQ 5097 3584 5394 3201 \nQ 5691 2819 5691 2113 \nL 5691 0 \nL 5113 0 \nL 5113 2094 \nQ 5113 2597 4934 2840 \nQ 4756 3084 4391 3084 \nQ 3944 3084 3684 2787 \nQ 3425 2491 3425 1978 \nL 3425 0 \nL 2847 0 \nL 2847 2094 \nQ 2847 2600 2669 2842 \nQ 2491 3084 2119 3084 \nQ 1678 3084 1418 2786 \nQ 1159 2488 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1356 3278 1631 3431 \nQ 1906 3584 2284 3584 \nQ 2666 3584 2933 3390 \nQ 3200 3197 3328 2828 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-2c\" d=\"M 750 794 \nL 1409 794 \nL 1409 256 \nL 897 -744 \nL 494 -744 \nL 750 256 \nL 750 794 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \nL 2156 4666 \nL 531 -594 \nL 0 -594 \nL 1625 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-42\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(68.603516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(96.386719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(159.765625 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6d\" transform=\"translate(220.947266 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(318.359375 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(346.142578 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(407.421875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(435.205078 0)\"/>\n      <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(474.21875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2c\" transform=\"translate(537.841797 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(569.628906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(601.416016 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2f\" transform=\"translate(665.039062 0)\"/>\n      <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(698.730469 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(762.353516 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_22\">\n     <path d=\"M 96.495313 34.976562 \nL 106.495313 34.976562 \nL 116.495313 34.976562 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_12\">\n     <!-- Binomial(20, 4/20) -->\n     <g transform=\"translate(124.495313 38.476562) scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-42\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(68.603516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(96.386719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(159.765625 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6d\" transform=\"translate(220.947266 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(318.359375 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(346.142578 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(407.421875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(435.205078 0)\"/>\n      <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(474.21875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(537.841797 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2c\" transform=\"translate(601.464844 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(633.251953 0)\"/>\n      <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(665.039062 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2f\" transform=\"translate(728.662109 0)\"/>\n      <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(762.353516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(825.976562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(889.599609 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_23\">\n     <path d=\"M 96.495313 49.654687 \nL 106.495313 49.654687 \nL 116.495313 49.654687 \n\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_13\">\n     <!-- Binomial(100, 4/100) -->\n     <g transform=\"translate(124.495313 53.154687) scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-42\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(68.603516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(96.386719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(159.765625 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6d\" transform=\"translate(220.947266 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(318.359375 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(346.142578 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(407.421875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(435.205078 0)\"/>\n      <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(474.21875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(537.841797 0)\"/>\n      <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(601.464844 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2c\" transform=\"translate(665.087891 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(696.875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(728.662109 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2f\" transform=\"translate(792.285156 0)\"/>\n      <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(825.976562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(889.599609 0)\"/>\n      <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(953.222656 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(1016.845703 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_24\">\n     <path d=\"M 96.495313 64.332813 \nL 106.495313 64.332813 \nL 116.495313 64.332813 \n\" style=\"fill: none; stroke-dasharray: 1.5,2.475; stroke-dashoffset: 0; stroke: #ff0000; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_14\">\n     <!-- Poisson(4) -->\n     <g transform=\"translate(124.495313 67.832813) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-50\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(56.677734 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(117.859375 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(145.642578 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(197.742188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(249.841797 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(311.023438 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(374.402344 0)\"/>\n      <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(413.416016 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(477.039062 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pedab15c7e5\">\n   <rect x=\"43.78125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-9 fw=all -->\n",
    "\n",
    "## Continuous Distributions\n",
    "\n",
    "A continuous random variable is described by a *probability density* (pdf)\n",
    "$p(x)\\ge0$ with $\\int p(x)\\,dx=1$; probabilities are areas under it and\n",
    "expectations are integral averages (that section). The five\n",
    "laws below run from the structureless (uniform) to the universal (Gaussian) to the\n",
    "vector-valued (multivariate Gaussian), with the exponential covering waiting times\n",
    "on $[0,\\infty)$ and the Laplace the symmetric case with heavier-than-Gaussian\n",
    "tails.\n",
    "the figure overlays the three densities on the whole\n",
    "line so their tail behavior is directly comparable: the Gaussian's thin\n",
    "$e^{-x^2}$ tail, the Laplace's heavier $e^{-|x|}$ tail and sharp peak, and the\n",
    "uniform's flat plateau.\n",
    "\n",
    "![A gallery of continuous probability densities in one consistent style: a continuous uniform on the interval from $-2$ to $2$, a standard Gaussian $\\mathcal N(0,1)$, and a Laplace with matched variance. The Laplace has a sharper peak and heavier tails than the Gaussian; the uniform is a flat plateau with hard edges.](../img/mdl-prob-continuous-pdfs.svg)\n",
    "\n",
    "### Continuous Uniform\n",
    "\n",
    "Refine the discrete uniform (let the $n$ equally likely points fill an interval\n",
    "$[a,b]$) and you reach the *continuous uniform*, which picks a value in $[a,b]$\n",
    "with every value equally likely. The density is constant on the interval and zero\n",
    "outside, with the constant fixed by $\\int p=1$:\n",
    "\n",
    "$$\n",
    "p(x) = \\begin{cases}\\frac{1}{b-a} & x\\in[a,b],\\\\ 0 & \\text{otherwise,}\\end{cases}\n",
    "\\qquad X\\sim U(a,b).\n",
    "$$\n",
    "\n",
    "**Where it arises.** It is the raw material of randomness: the $U(0,1)$ generator\n",
    "underlies inverse-transform sampling (that section), Monte\n",
    "Carlo, dropout masks, and random weight initialization.\n",
    "\n",
    "**Mean and variance.** Direct integration, with the substitution $x\\mapsto x-\\tfrac{a+b}2$ centering the interval,\n",
    "\n",
    "$$\n",
    "\\mu_X = \\int_a^b \\frac{x}{b-a}\\,dx = \\frac{a+b}{2},\n",
    "\\qquad\n",
    "\\sigma_X^2 = \\int_a^b \\frac{(x-\\mu_X)^2}{b-a}\\,dx = \\frac{(b-a)^2}{12}.\n",
    "$$\n",
    "\n",
    "These are the continuous mirrors of the discrete-uniform formulas (with the range\n",
    "$b-a$ playing the role of $n$). The default generator samples $U(0,1)$; shifting\n",
    "and scaling gives any $U(a,b)$, and the cell checks the formulas against a large\n",
    "sample."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "distributions-continuous-uniform"
   },
   "source": [
    "rng = np.random.default_rng(0)\n",
    "a, b = 1.0, 3.0\n",
    "sample = (b - a) * rng.random(10000) + a         # shift-and-scale U(0,1)\n",
    "print('mean (a+b)/2   =', (a + b) / 2, '  sample mean:',\n",
    "      float(sample.mean().round(3)))\n",
    "print('var (b-a)^2/12 =', round((b - a)**2 / 12, 3), ' sample var :',\n",
    "      float(sample.var().round(3)))"
   ],
   "execution_count": 8,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "mean (a+b)/2   = 2.0   sample mean: 1.999\nvar (b-a)^2/12 = 0.333  sample var : 0.334\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-10 fw=all -->\n",
    "\n",
    "### Exponential\n",
    "\n",
    "If the Poisson counts how *many* rare events land in a window, the *exponential*\n",
    "models the *waiting time between* them. It is the canonical density on $[0,\\infty)$,\n",
    "\n",
    "$$\n",
    "p(x) = \\lambda e^{-\\lambda x}\\ (x\\ge0),\n",
    "\\qquad\n",
    "F(x) = 1 - e^{-\\lambda x},\n",
    "\\qquad X\\sim\\mathrm{Exp}(\\lambda),\n",
    "$$\n",
    "\n",
    "with rate $\\lambda$ matching the Poisson rate: more frequent events mean shorter\n",
    "waits.\n",
    "\n",
    "**Where it arises.** Inter-arrival times, time-to-failure in reliability, and\n",
    "survival analysis: anywhere a constant *hazard* (the instantaneous failure rate\n",
    "per unit time, here independent of the age already reached) is plausible.\n",
    "\n",
    "**Memorylessness.** The exponential is the *unique* continuous distribution with no\n",
    "memory: having already waited $s$, the chance of waiting $t$ longer is the same as\n",
    "starting fresh. Using $P(X>x)=e^{-\\lambda x}$,\n",
    "\n",
    "$$\n",
    "P(X>s+t \\mid X>s)\n",
    "= \\frac{P(X>s+t)}{P(X>s)}\n",
    "= \\frac{e^{-\\lambda(s+t)}}{e^{-\\lambda s}}\n",
    "= e^{-\\lambda t}\n",
    "= P(X>t).\n",
    "$$\n",
    "\n",
    "The waiting clock never \"ages\", exactly the property that makes the exponential\n",
    "the continuous-time partner of the Poisson. The converse, that memorylessness\n",
    "*forces* the exponential form, is Exercise 4.\n",
    "\n",
    "**Mean and variance.** With the standard integrals $\\int_0^\\infty u\\,e^{-u}\\,du=1$\n",
    "and $\\int_0^\\infty u^2 e^{-u}\\,du=2$ (integration by parts,\n",
    "that section), substitute $u=\\lambda x$:\n",
    "\n",
    "$$\n",
    "\\mu_X = \\int_0^\\infty x\\,\\lambda e^{-\\lambda x}\\,dx = \\frac1\\lambda,\n",
    "\\qquad\n",
    "\\mathbb E[X^2] = \\frac{2}{\\lambda^2}\n",
    "\\;\\Rightarrow\\;\n",
    "\\sigma_X^2 = \\frac{2}{\\lambda^2}-\\frac1{\\lambda^2} = \\frac1{\\lambda^2}.\n",
    "$$\n",
    "\n",
    "The teaching cell samples by the *inverse-transform* proposition of\n",
    "that section; this is the closed-form case promised\n",
    "there. Inverting $F(x)=1-e^{-\\lambda x}$ gives\n",
    "$F^{-1}(q)=-\\tfrac1\\lambda\\log(1-q)$, and since $1-U$ is again uniform when\n",
    "$U\\sim U(0,1)$, the sampler simplifies to\n",
    "$X=-\\tfrac1\\lambda\\log U\\sim\\mathrm{Exp}(\\lambda)$."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "distributions-exponential"
   },
   "source": [
    "import numpy as onp\n",
    "rng = onp.random.default_rng(0)\n",
    "lam = 0.5\n",
    "U = rng.random(100000)\n",
    "sample = -onp.log(U) / lam                        # inverse-transform sampler\n",
    "print('mean 1/lambda =', 1 / lam, '  sample mean =', float(sample.mean().round(3)))\n",
    "print('var 1/lambda^2 =', 1 / lam**2, ' sample var =', float(sample.var().round(3)))"
   ],
   "execution_count": 9,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "mean 1/lambda = 2.0   sample mean = 2.001\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "var 1/lambda^2 = 4.0  sample var = 3.984\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-11 fw=all -->\n",
    "\n",
    "### Gaussian\n",
    "\n",
    "The *Gaussian* (or normal) distribution is what sums of many small independent\n",
    "effects look like, and such sums are everywhere.\n",
    "Return to the binomial $X^{(n)}\\sim\\mathrm{Binomial}(n,p)$, but now hold $p$ fixed\n",
    "and send $n\\to\\infty$. Both mean $np$ and variance $np(1-p)$ blow up, so we\n",
    "standardize,\n",
    "\n",
    "$$\n",
    "Y^{(n)} = \\frac{X^{(n)} - np}{\\sqrt{np(1-p)}},\n",
    "$$\n",
    "\n",
    "which has mean $0$ and variance $1$ for every $n$. The *central limit theorem*\n",
    "(CLT) states that $Y^{(n)}$ converges in distribution to the *standard Gaussian*:\n",
    "for any interval $[a,b]$, $P(Y^{(n)}\\in[a,b])\\to P(\\mathcal N(0,1)\\in[a,b])$. This\n",
    "binomial case is the *de Moivre--Laplace theorem*, historically the first central\n",
    "limit theorem. The limiting density is\n",
    "\n",
    "$$\n",
    "p(x) = \\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\,\\exp\\!\\Bigl(-\\frac{(x-\\mu)^2}{2\\sigma^2}\\Bigr),\n",
    "\\qquad X\\sim\\mathcal N(\\mu,\\sigma^2).\n",
    "$$\n",
    "\n",
    "Nothing about coin flips was special: the *Lindeberg--Lévy* form of the CLT holds\n",
    "for any independent, identically distributed summands with finite variance (no\n",
    "higher moments needed), which is why the conditions are so easy to meet and the\n",
    "Gaussian is everywhere. We take the theorem as given; [@Wasserman.2013]\n",
    "gives the proof.\n",
    "\n",
    "**Watching universality happen.** Like the Poisson limit, this claim is\n",
    "checkable in a few lines. The cell below starts from about the least Gaussian\n",
    "summand available (a $U(0,1)$ draw, flat with hard edges and no tails), forms\n",
    "sums of $n$ of them, standardizes each sum by its mean $n/2$ and standard\n",
    "deviation $\\sqrt{n/12}$, and overlays the resulting histograms on the standard\n",
    "normal density. At $n=1$ we see the uniform's own plateau; at $n=2$ the\n",
    "triangular hat; by $n=8$ the curve is already hard to distinguish from the\n",
    "Gaussian, and at $n=32$ the two agree to the resolution of the plot. Nothing in\n",
    "the code knows about the Gaussian: the bell is manufactured by the summation\n",
    "itself, which is exactly the universality the CLT asserts."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mdl-distributions-gaussian-1"
   },
   "source": [
    "rng = np.random.default_rng(0)\n",
    "ns, x = (1, 2, 8, 32), np.arange(-4, 4, 0.01)\n",
    "curves = []\n",
    "for n in ns:                                   # standardized sums of n uniforms\n",
    "    s = (rng.random((100000, n)).sum(1) - n / 2) / np.sqrt(n / 12)\n",
    "    hist, edges = np.histogram(s, bins=80, range=(-4, 4), density=True)\n",
    "    curves.append(np.interp(x, (edges[:-1] + edges[1:]) / 2, hist))\n",
    "curves.append(np.exp(-x**2 / 2) / np.sqrt(2 * np.pi))     # N(0, 1) density\n",
    "d2l.plot(x, curves, xlabel='standardized sum', ylabel='density',\n",
    "         legend=[f'n = {n}' for n in ns] + ['N(0, 1)'])"
   ],
   "execution_count": 10,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"246.28125pt\" height=\"183.35625pt\" viewBox=\"0 0 246.28125 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-19T14:02:27.888902</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 183.35625 \nL 246.28125 183.35625 \nL 246.28125 0 \nL 0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 43.78125 145.8 \nL 239.08125 145.8 \nL 239.08125 7.2 \nL 43.78125 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path d=\"M 52.658523 145.8 \nL 52.658523 7.2 \n\" clip-path=\"url(#p2ddba0da8a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path id=\"mb346ce895f\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#mb346ce895f\" x=\"52.658523\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- −4 -->\n      <g transform=\"translate(45.287429 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \nL 4684 2272 \nL 4684 1741 \nL 678 1741 \nL 678 2272 \nz\n\" transform=\"scale(0.015625)\"/>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-2212\"/>\n       <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(83.789062 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path d=\"M 97.100439 145.8 \nL 97.100439 7.2 \n\" clip-path=\"url(#p2ddba0da8a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#mb346ce895f\" x=\"97.100439\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- −2 -->\n      <g transform=\"translate(89.729345 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-2212\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(83.789062 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path d=\"M 141.542355 145.8 \nL 141.542355 7.2 \n\" clip-path=\"url(#p2ddba0da8a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#mb346ce895f\" x=\"141.542355\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 0 -->\n      <g transform=\"translate(138.361105 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_7\">\n      <path d=\"M 185.984271 145.8 \nL 185.984271 7.2 \n\" clip-path=\"url(#p2ddba0da8a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#mb346ce895f\" x=\"185.984271\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 2 -->\n      <g transform=\"translate(182.803021 160.398438) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-32\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_9\">\n      <path d=\"M 230.426187 145.8 \nL 230.426187 7.2 \n\" clip-path=\"url(#p2ddba0da8a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#mb346ce895f\" x=\"230.426187\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 4 -->\n      <g transform=\"translate(227.244937 160.398438) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-34\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_6\">\n     <!-- standardized sum -->\n     <g transform=\"translate(96.75625 174.076563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \nL 2906 4863 \nL 3481 4863 \nL 3481 0 \nL 2906 0 \nL 2906 525 \nQ 2725 213 2448 61 \nQ 2172 -91 1784 -91 \nQ 1150 -91 751 415 \nQ 353 922 353 1747 \nQ 353 2572 751 3078 \nQ 1150 3584 1784 3584 \nQ 2172 3584 2448 3432 \nQ 2725 3281 2906 2969 \nz\nM 947 1747 \nQ 947 1113 1208 752 \nQ 1469 391 1925 391 \nQ 2381 391 2643 752 \nQ 2906 1113 2906 1747 \nQ 2906 2381 2643 2742 \nQ 2381 3103 1925 3103 \nQ 1469 3103 1208 2742 \nQ 947 2381 947 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-7a\" d=\"M 353 3500 \nL 3084 3500 \nL 3084 2975 \nL 922 459 \nL 3084 459 \nL 3084 0 \nL 275 0 \nL 275 525 \nL 2438 3041 \nL 353 3041 \nL 353 3500 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-75\" d=\"M 544 1381 \nL 544 3500 \nL 1119 3500 \nL 1119 1403 \nQ 1119 906 1312 657 \nQ 1506 409 1894 409 \nQ 2359 409 2629 706 \nQ 2900 1003 2900 1516 \nL 2900 3500 \nL 3475 3500 \nL 3475 0 \nL 2900 0 \nL 2900 538 \nQ 2691 219 2414 64 \nQ 2138 -91 1772 -91 \nQ 1169 -91 856 284 \nQ 544 659 544 1381 \nz\nM 1991 3584 \nL 1991 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \nQ 3544 3216 3844 3400 \nQ 4144 3584 4550 3584 \nQ 5097 3584 5394 3201 \nQ 5691 2819 5691 2113 \nL 5691 0 \nL 5113 0 \nL 5113 2094 \nQ 5113 2597 4934 2840 \nQ 4756 3084 4391 3084 \nQ 3944 3084 3684 2787 \nQ 3425 2491 3425 1978 \nL 3425 0 \nL 2847 0 \nL 2847 2094 \nQ 2847 2600 2669 2842 \nQ 2491 3084 2119 3084 \nQ 1678 3084 1418 2786 \nQ 1159 2488 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1356 3278 1631 3431 \nQ 1906 3584 2284 3584 \nQ 2666 3584 2933 3390 \nQ 3200 3197 3328 2828 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(52.099609 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(91.308594 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(152.587891 0)\"/>\n      <use xlink:href=\"#DejaVuSans-64\" transform=\"translate(215.966797 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(279.443359 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(340.722656 0)\"/>\n      <use xlink:href=\"#DejaVuSans-64\" transform=\"translate(380.085938 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(443.5625 0)\"/>\n      <use xlink:href=\"#DejaVuSans-7a\" transform=\"translate(471.345703 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(523.835938 0)\"/>\n      <use xlink:href=\"#DejaVuSans-64\" transform=\"translate(585.359375 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(648.835938 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(680.623047 0)\"/>\n      <use xlink:href=\"#DejaVuSans-75\" transform=\"translate(732.722656 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6d\" transform=\"translate(796.101562 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_11\">\n      <path d=\"M 43.78125 139.5 \nL 239.08125 139.5 \n\" clip-path=\"url(#p2ddba0da8a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_12\">\n      <defs>\n       <path id=\"m46b6ece08c\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m46b6ece08c\" x=\"43.78125\" y=\"139.5\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_7\">\n      <!-- 0.0 -->\n      <g transform=\"translate(20.878125 143.299219) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \nL 1344 794 \nL 1344 0 \nL 684 0 \nL 684 794 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_13\">\n      <path d=\"M 43.78125 108.549374 \nL 239.08125 108.549374 \n\" clip-path=\"url(#p2ddba0da8a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_14\">\n      <g>\n       <use xlink:href=\"#m46b6ece08c\" x=\"43.78125\" y=\"108.549374\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.1 -->\n      <g transform=\"translate(20.878125 112.348592) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_15\">\n      <path d=\"M 43.78125 77.598747 \nL 239.08125 77.598747 \n\" clip-path=\"url(#p2ddba0da8a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_16\">\n      <g>\n       <use xlink:href=\"#m46b6ece08c\" x=\"43.78125\" y=\"77.598747\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.2 -->\n      <g transform=\"translate(20.878125 81.397966) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_17\">\n      <path d=\"M 43.78125 46.648121 \nL 239.08125 46.648121 \n\" clip-path=\"url(#p2ddba0da8a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_18\">\n      <g>\n       <use xlink:href=\"#m46b6ece08c\" x=\"43.78125\" y=\"46.648121\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 0.3 -->\n      <g transform=\"translate(20.878125 50.44734) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \nQ 3050 2419 3304 2112 \nQ 3559 1806 3559 1356 \nQ 3559 666 3084 287 \nQ 2609 -91 1734 -91 \nQ 1441 -91 1130 -33 \nQ 819 25 488 141 \nL 488 750 \nQ 750 597 1062 519 \nQ 1375 441 1716 441 \nQ 2309 441 2620 675 \nQ 2931 909 2931 1356 \nQ 2931 1769 2642 2001 \nQ 2353 2234 1838 2234 \nL 1294 2234 \nL 1294 2753 \nL 1863 2753 \nQ 2328 2753 2575 2939 \nQ 2822 3125 2822 3475 \nQ 2822 3834 2567 4026 \nQ 2313 4219 1838 4219 \nQ 1578 4219 1281 4162 \nQ 984 4106 628 3988 \nL 628 4550 \nQ 988 4650 1302 4700 \nQ 1616 4750 1894 4750 \nQ 2613 4750 3031 4423 \nQ 3450 4097 3450 3541 \nQ 3450 3153 3228 2886 \nQ 3006 2619 2597 2516 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_19\">\n      <path d=\"M 43.78125 15.697494 \nL 239.08125 15.697494 \n\" clip-path=\"url(#p2ddba0da8a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_20\">\n      <g>\n       <use xlink:href=\"#m46b6ece08c\" x=\"43.78125\" y=\"15.697494\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 0.4 -->\n      <g transform=\"translate(20.878125 19.496713) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_12\">\n     <!-- density -->\n     <g transform=\"translate(14.798438 94.832813) rotate(-90) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \nQ 1816 -950 1584 -1140 \nQ 1353 -1331 966 -1331 \nL 506 -1331 \nL 506 -850 \nL 844 -850 \nQ 1081 -850 1212 -737 \nQ 1344 -625 1503 -206 \nL 1606 56 \nL 191 3500 \nL 800 3500 \nL 1894 763 \nL 2988 3500 \nL 3597 3500 \nL 2059 -325 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-64\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(63.476562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(188.378906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(240.478516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(268.261719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-79\" transform=\"translate(307.470703 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_21\">\n    <path d=\"M 52.658523 139.5 \nL 100.433582 139.5 \nL 102.655678 111.365881 \nL 104.877774 52.52874 \nL 107.09987 46.431466 \nL 109.321966 46.12196 \nL 113.766157 50.733604 \nL 115.988253 51.105011 \nL 118.210349 53.488209 \nL 120.432445 51.631172 \nL 124.210008 51.0493 \nL 124.876636 50.919307 \nL 127.098732 53.240604 \nL 129.320828 51.785925 \nL 131.542924 48.814665 \nL 133.765019 47.762343 \nL 135.987115 49.278924 \nL 138.209211 47.60759 \nL 140.431307 47.793294 \nL 142.653403 50.640752 \nL 144.875498 49.836035 \nL 147.097594 52.002579 \nL 149.31969 50.671702 \nL 151.763995 50.764554 \nL 153.763882 49.928887 \nL 155.985978 47.236183 \nL 158.208073 50.393147 \nL 160.430169 49.681282 \nL 162.652265 46.679071 \nL 165.540989 50.253869 \nL 167.096457 51.878777 \nL 169.318552 49.619381 \nL 171.540648 53.735814 \nL 173.762744 50.857406 \nL 175.98484 50.826455 \nL 178.206936 52.312085 \nL 180.651241 112.758659 \nL 182.651127 139.5 \nL 230.203977 139.5 \nL 230.203977 139.5 \n\" clip-path=\"url(#p2ddba0da8a)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_22\">\n    <path d=\"M 52.658523 139.5 \nL 85.101121 139.444289 \nL 87.101008 138.942889 \nL 89.989732 132.755859 \nL 93.767295 123.931835 \nL 96.65602 117.74171 \nL 98.878115 112.201548 \nL 100.655792 107.497052 \nL 102.877888 103.584893 \nL 105.544403 97.038836 \nL 107.544289 92.516949 \nL 111.766271 81.346868 \nL 113.766157 79.146279 \nL 118.654768 65.608475 \nL 120.654654 61.702506 \nL 125.098846 49.09322 \nL 127.098732 45.471997 \nL 129.543037 38.498821 \nL 133.987229 31.989904 \nL 135.987115 27.644436 \nL 138.209211 17.121223 \nL 140.653516 14.363522 \nL 142.653403 13.5 \nL 146.208756 26.592115 \nL 147.097594 29.687178 \nL 149.31969 31.049005 \nL 151.541786 39.374724 \nL 153.763882 40.303242 \nL 156.208187 46.846205 \nL 158.652492 55.444289 \nL 160.430169 60.42115 \nL 162.652265 62.556743 \nL 165.31878 68.697347 \nL 167.096457 73.451363 \nL 169.318552 75.803611 \nL 171.540648 83.200811 \nL 173.762744 85.955416 \nL 176.207049 94.469934 \nL 178.429145 99.016581 \nL 180.651241 104.996242 \nL 182.873337 109.233382 \nL 185.095433 114.646647 \nL 187.317528 118.571186 \nL 191.76172 129.914591 \nL 194.650444 135.878777 \nL 195.983702 138.850037 \nL 198.428007 139.5 \nL 230.203977 139.5 \nL 230.203977 139.5 \n\" clip-path=\"url(#p2ddba0da8a)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"line2d_23\">\n    <path d=\"M 52.658523 139.5 \nL 64.43563 139.400957 \nL 68.213193 139.082162 \nL 71.101918 138.980024 \nL 72.212966 138.880981 \nL 74.435062 138.469334 \nL 77.545996 137.667705 \nL 79.768092 137.064161 \nL 81.767978 136.367765 \nL 83.990074 135.420667 \nL 85.101121 134.844979 \nL 87.545427 133.099346 \nL 89.323103 132.133677 \nL 91.767409 129.295476 \nL 93.767295 126.036343 \nL 95.989391 125.169717 \nL 100.433582 115.01781 \nL 102.655678 112.479833 \nL 105.766612 105.460161 \nL 107.09987 102.730288 \nL 109.321966 94.249732 \nL 111.544061 90.907031 \nL 113.988367 79.554227 \nL 116.432672 72.800733 \nL 118.210349 68.962817 \nL 120.654654 59.352552 \nL 123.09896 52.750717 \nL 127.098732 39.96179 \nL 129.543037 33.963499 \nL 131.765133 30.103917 \nL 133.765019 24.950586 \nL 135.987115 24.579175 \nL 138.209211 18.605644 \nL 140.431307 18.45089 \nL 142.653403 15.851011 \nL 145.986546 20.307946 \nL 147.319804 22.081435 \nL 149.986319 27.216195 \nL 153.763882 35.102493 \nL 155.985978 38.29044 \nL 158.208073 47.204309 \nL 160.430169 50.516059 \nL 163.096684 60.395598 \nL 165.09657 66.985052 \nL 167.318666 76.459134 \nL 169.540762 82.021017 \nL 171.762858 89.508048 \nL 173.984953 95.224686 \nL 175.98484 102.049368 \nL 178.206936 105.670627 \nL 180.651241 112.585066 \nL 183.317756 116.977004 \nL 186.6509 123.28171 \nL 187.317528 124.399038 \nL 190.650672 127.992442 \nL 193.094977 130.595416 \nL 193.983816 131.474422 \nL 196.650331 133.322193 \nL 198.650217 134.820218 \nL 203.316618 137.243677 \nL 208.205229 138.308389 \nL 209.760696 138.590042 \nL 211.760582 139.150254 \nL 214.427097 139.131684 \nL 216.871402 139.270963 \nL 221.537804 139.314294 \nL 224.870947 139.5 \nL 230.203977 139.5 \nL 230.203977 139.5 \n\" clip-path=\"url(#p2ddba0da8a)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"line2d_24\">\n    <path d=\"M 52.658523 139.5 \nL 57.102714 139.391671 \nL 64.213421 139.184297 \nL 65.991098 139.144061 \nL 67.546565 139.066683 \nL 69.76866 138.713838 \nL 71.990756 138.713838 \nL 76.434948 137.871964 \nL 78.434834 137.0239 \nL 80.879139 136.572012 \nL 83.101235 135.593953 \nL 86.434379 133.328322 \nL 87.323217 132.79596 \nL 89.323103 132.350262 \nL 91.989618 129.106572 \nL 94.878343 124.535073 \nL 96.2116 122.430388 \nL 98.655906 119.740725 \nL 101.100211 116.25252 \nL 102.655678 113.717613 \nL 105.099984 106.174794 \nL 108.210918 98.675307 \nL 110.210804 93.289791 \nL 112.655109 86.000772 \nL 113.988367 82.060679 \nL 115.988253 77.937973 \nL 118.654768 68.367848 \nL 120.876864 59.107235 \nL 125.098846 44.297064 \nL 127.098732 40.982186 \nL 129.543037 32.62535 \nL 132.209552 27.202691 \nL 135.987115 18.759192 \nL 138.209211 15.911677 \nL 140.431307 17.490191 \nL 142.653403 16.22119 \nL 144.875498 21.111486 \nL 147.097594 19.625827 \nL 149.31969 21.637657 \nL 151.763995 27.688626 \nL 153.986091 35.265491 \nL 155.985978 39.527477 \nL 158.208073 47.822411 \nL 160.430169 51.134194 \nL 162.874474 61.642142 \nL 166.207618 71.735343 \nL 167.096457 74.285726 \nL 169.540762 85.369367 \nL 171.985067 91.420335 \nL 174.207163 97.598204 \nL 176.429259 102.940389 \nL 178.873564 109.418485 \nL 181.09566 114.308781 \nL 183.317756 118.326253 \nL 185.317642 121.622561 \nL 187.317528 125.243856 \nL 189.761834 127.645673 \nL 191.76172 129.273709 \nL 193.983816 131.808616 \nL 196.428121 133.452127 \nL 198.428007 135.092543 \nL 201.094522 136.151075 \nL 203.094408 137.042471 \nL 206.871971 138.101004 \nL 208.427438 138.466228 \nL 209.760696 138.670507 \nL 212.64942 138.803597 \nL 218.20466 139.25239 \nL 225.093157 139.357624 \nL 228.64851 139.447383 \nL 230.203977 139.438098 \nL 230.203977 139.438098 \n\" clip-path=\"url(#p2ddba0da8a)\" style=\"fill: none; stroke-dasharray: 1.5,2.475; stroke-dashoffset: 0; stroke: #ff0000; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 43.78125 145.8 \nL 43.78125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 239.08125 145.8 \nL 239.08125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 43.78125 145.8 \nL 239.08125 145.8 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 43.78125 7.2 \nL 239.08125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 50.78125 73.9125 \nL 116.579688 73.9125 \nQ 118.579688 73.9125 118.579688 71.9125 \nL 118.579688 14.2 \nQ 118.579688 12.2 116.579688 12.2 \nL 50.78125 12.2 \nQ 48.78125 12.2 48.78125 14.2 \nL 48.78125 71.9125 \nQ 48.78125 73.9125 50.78125 73.9125 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_25\">\n     <path d=\"M 52.78125 20.298438 \nL 62.78125 20.298438 \nL 72.78125 20.298438 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_13\">\n     <!-- n = 1 -->\n     <g transform=\"translate(80.78125 23.798438) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-3d\" d=\"M 678 2906 \nL 4684 2906 \nL 4684 2381 \nL 678 2381 \nL 678 2906 \nz\nM 678 1631 \nL 4684 1631 \nL 4684 1100 \nL 678 1100 \nL 678 1631 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-6e\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(63.378906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-3d\" transform=\"translate(95.166016 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(178.955078 0)\"/>\n      <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(210.742188 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_26\">\n     <path d=\"M 52.78125 34.976562 \nL 62.78125 34.976562 \nL 72.78125 34.976562 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_14\">\n     <!-- n = 2 -->\n     <g transform=\"translate(80.78125 38.476562) scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-6e\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(63.378906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-3d\" transform=\"translate(95.166016 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(178.955078 0)\"/>\n      <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(210.742188 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_27\">\n     <path d=\"M 52.78125 49.654687 \nL 62.78125 49.654687 \nL 72.78125 49.654687 \n\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_15\">\n     <!-- n = 8 -->\n     <g transform=\"translate(80.78125 53.154687) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-6e\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(63.378906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-3d\" transform=\"translate(95.166016 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(178.955078 0)\"/>\n      <use xlink:href=\"#DejaVuSans-38\" transform=\"translate(210.742188 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_28\">\n     <path d=\"M 52.78125 64.332813 \nL 62.78125 64.332813 \nL 72.78125 64.332813 \n\" style=\"fill: none; stroke-dasharray: 1.5,2.475; stroke-dashoffset: 0; stroke: #ff0000; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_16\">\n     <!-- n = 32 -->\n     <g transform=\"translate(80.78125 67.832813) scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-6e\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(63.378906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-3d\" transform=\"translate(95.166016 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(178.955078 0)\"/>\n      <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(210.742188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(274.365234 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p2ddba0da8a\">\n   <rect x=\"43.78125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-12 fw=all -->\n",
    "\n",
    "**The normalizer and the moments.** Why the constant $1/\\sqrt{2\\pi\\sigma^2}$?\n",
    "The identity it rests on, $\\int_{-\\infty}^\\infty e^{-x^2/2}\\,dx=\\sqrt{2\\pi}$, is\n",
    "the Gaussian integral computed in that section by squaring\n",
    "the integral and passing to polar coordinates (the equation).\n",
    "Substituting $x\\mapsto(x-\\mu)/\\sigma$ then shows the equation\n",
    "integrates to one. The parameters are the first two moments, and one integration\n",
    "by parts proves it: the mean is $\\mu$ by symmetry, and for the variance take\n",
    "$\\mu=0$, $\\sigma=1$ and integrate by parts with $u=x$, $dv=x\\,e^{-x^2/2}\\,dx$,\n",
    "\n",
    "$$\n",
    "\\int_{-\\infty}^{\\infty} x^2\\,\\frac{e^{-x^2/2}}{\\sqrt{2\\pi}}\\,dx\n",
    "= \\frac{1}{\\sqrt{2\\pi}}\\Bigl(\\bigl[-x\\,e^{-x^2/2}\\bigr]_{-\\infty}^{\\infty}\n",
    "+ \\int_{-\\infty}^{\\infty} e^{-x^2/2}\\,dx\\Bigr) = 0 + 1 = 1,\n",
    "$$\n",
    "\n",
    "so the standard Gaussian has variance $1$; shifting by $\\mu$ and scaling by\n",
    "$\\sigma$ gives mean $\\mu$ and variance $\\sigma^2$ in general. (The exponential\n",
    "family's moment property, the equation, will recover both by\n",
    "differentiation alone.)\n",
    "\n",
    "**Maximum entropy.** Among all distributions with a given mean and variance, the\n",
    "Gaussian has the largest entropy: it is the *most noncommittal* choice consistent\n",
    "with knowing only those two numbers [@Cover.Thomas.1999]. We prove this in\n",
    "that section as a short consequence of Gibbs'\n",
    "inequality, and place it in context when we meet the\n",
    "exponential family below.\n",
    "\n",
    "**Where it arises.** As the CLT limit it models aggregate noise; it is the noise\n",
    "model behind *regression*, where taking $y\\sim\\mathcal N(\\hat y,\\sigma^2)$ makes\n",
    "the negative log-likelihood equal to mean squared error\n",
    "(that section); and it is the default prior\n",
    "and latent distribution throughout deep generative modeling."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "distributions-gaussian"
   },
   "source": [
    "import numpy as onp\n",
    "rng = onp.random.default_rng(0)\n",
    "mu, sigma = 0.0, 1.0\n",
    "x = onp.arange(-4, 4, 0.01)\n",
    "p = onp.exp(-(x - mu)**2 / (2 * sigma**2)) / onp.sqrt(2 * onp.pi * sigma**2)\n",
    "print('total mass (≈1):', float((0.01 * p).sum().round(4)))\n",
    "rng.normal(mu, sigma, size=(3, 3))              # sample from N(mu, sigma^2)"
   ],
   "execution_count": 11,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "total mass (≈1): 0.9999\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "array([[ 0.12573022, -0.13210486,  0.64042265],\n       [ 0.10490012, -0.53566937,  0.36159505],\n       [ 1.30400005,  0.94708096, -0.70373524]])"
     },
     "metadata": {},
     "execution_count": 11
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-13 fw=all -->\n",
    "\n",
    "### Laplace\n",
    "\n",
    "The *Laplace* distribution is a Gaussian's sharper sibling: symmetric about a\n",
    "location $\\mu$, but with *exponential* decay on each side instead of Gaussian,\n",
    "\n",
    "$$\n",
    "p(x) = \\frac{1}{2b}\\,e^{-|x-\\mu|/b},\n",
    "\\qquad X\\sim\\mathrm{Laplace}(\\mu,b).\n",
    "$$\n",
    "\n",
    "The two-sided exponential gives it a sharp peak at $\\mu$ and *heavier tails* than\n",
    "the Gaussian, visible in the figure.\n",
    "\n",
    "**Where it arises.** It is the probabilistic shadow of the $L_1$ loss, just as the\n",
    "Gaussian is of MSE: the negative log-likelihood of a Laplace is $\\propto|y-\\hat y|$,\n",
    "the *mean absolute error*. As a *prior* on weights it produces the\n",
    "sparsity-inducing $L_1$ regularizer $\\propto\\|\\mathbf w\\|_1$, the engine of LASSO\n",
    "[@Tibshirani.1996].\n",
    "\n",
    "**Mean and variance.** By symmetry $\\mu_X=\\mu$. For the variance, center at $\\mu$\n",
    "and use $\\int_0^\\infty u^2 e^{-u}\\,du=2$ with $u=|x-\\mu|/b$:\n",
    "\n",
    "$$\n",
    "\\sigma_X^2 = \\int_{-\\infty}^\\infty \\frac{(x-\\mu)^2}{2b}e^{-|x-\\mu|/b}\\,dx\n",
    "= \\frac{2}{2b}\\int_0^\\infty x^2 e^{-x/b}\\,dx\n",
    "= \\frac{1}{b}\\cdot b^3\\!\\int_0^\\infty u^2 e^{-u}\\,du\n",
    "= 2b^2 .\n",
    "$$\n",
    "\n",
    "The *location estimator* makes the contrast concrete: the maximum-likelihood\n",
    "$\\hat\\mu$ under a Laplace minimizes $\\sum_i|x_i-\\mu|$, whose minimizer is the sample\n",
    "*median* (robust to outliers), whereas the Gaussian's is the mean. The cell\n",
    "matches a Laplace and a Gaussian on variance and confirms the Laplace's heavier\n",
    "tail by the fraction of mass beyond $3$ standard deviations."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "distributions-laplace"
   },
   "source": [
    "import numpy as onp\n",
    "rng = onp.random.default_rng(0)\n",
    "mu, b = 0.0, 1.0\n",
    "sigma = onp.sqrt(2) * b                            # matched-variance Gaussian sd\n",
    "U = (rng.random(200000) - 0.5) * (1 - 1e-7)       # open interval avoids log(0)\n",
    "lap = mu - b * onp.sign(U) * onp.log(1 - 2 * onp.abs(U))  # inverse transform\n",
    "gau = rng.normal(mu, sigma, 200000)\n",
    "print('Laplace var (2b^2):', float(lap.var().round(3)), ' Gaussian var:', float(gau.var().round(3)))\n",
    "print('P(|x| > 3sd): Laplace', float((onp.abs(lap) > 3 * sigma).mean()),\n",
    "      ' Gaussian', float((onp.abs(gau) > 3 * sigma).mean()))"
   ],
   "execution_count": 12,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Laplace var (2b^2): 1.998  Gaussian var: 2.005\nP(|x| > 3sd): Laplace 0.014575  Gaussian 0.00272\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-14 fw=all -->\n",
    "\n",
    "### Multivariate Gaussian\n",
    "\n",
    "Almost every latent-variable and noise model in deep learning uses a Gaussian over\n",
    "a *vector*. The *multivariate Gaussian* $\\mathcal N(\\boldsymbol\\mu,\n",
    "\\boldsymbol\\Sigma)$ on $\\mathbb R^d$ has a mean vector $\\boldsymbol\\mu$ and a\n",
    "symmetric positive-definite *covariance matrix* $\\boldsymbol\\Sigma$, with density\n",
    "\n",
    "$$\n",
    "p(\\mathbf x) = \\frac{1}{(2\\pi)^{d/2}|\\boldsymbol\\Sigma|^{1/2}}\n",
    "\\exp\\!\\Bigl(-\\tfrac12(\\mathbf x-\\boldsymbol\\mu)^\\top\\boldsymbol\\Sigma^{-1}(\\mathbf x-\\boldsymbol\\mu)\\Bigr).\n",
    "$$\n",
    "\n",
    "**The geometry is the eigendecomposition.** The exponent is a quadratic form in\n",
    "$\\boldsymbol\\Sigma^{-1}$, so its level sets (the contours of equal density) are\n",
    "*ellipsoids*. The covariance $\\boldsymbol\\Sigma$ is symmetric positive definite, so\n",
    "it has an orthonormal eigenbasis\n",
    "$\\boldsymbol\\Sigma=\\mathbf W\\boldsymbol\\Lambda\\mathbf W^\\top$ with all $\\lambda_i>0$\n",
    "by the spectral theorem (that section), and the contour\n",
    "ellipsoid's *axes point along the eigenvectors $\\mathbf w_i$ with half-lengths\n",
    "proportional to $\\sqrt{\\lambda_i}$*: the principal directions of spread are the\n",
    "eigenvectors, stretched by the standard deviations along them. The isotropic case\n",
    "$\\boldsymbol\\Sigma=\\sigma^2\\mathbf I$ gives spherical contours and factorizes into\n",
    "$d$ independent one-dimensional Gaussians. the figure draws\n",
    "this for a $2\\times2$ covariance.\n",
    "\n",
    "![Samples from a $2$-D Gaussian (blue points) with the elliptical density contours overlaid. The ellipse axes lie along the eigenvectors of the covariance $\\boldsymbol\\Sigma$ with half-lengths proportional to $\\sqrt{\\lambda_i}$ (the standard deviation along each principal direction), connecting the multivariate Gaussian directly to the spectral theorem.](../img/mdl-prob-mvn-contours.svg)\n",
    "\n",
    "It is the default vector noise and prior because it is *closed under linear maps and\n",
    "conditioning*. A linear transform $\\mathbf A\\mathbf x+\\mathbf b$ of a Gaussian is\n",
    "again Gaussian, with covariance $\\mathbf A\\boldsymbol\\Sigma\\mathbf A^\\top$; marginals\n",
    "(a special case, projecting onto a coordinate block) are therefore Gaussian too.\n",
    "*Conditioning* is closed in the same way. Partition the vector and its parameters\n",
    "into blocks $\\mathbf x=(\\mathbf x_1,\\mathbf x_2)$,\n",
    "\n",
    "$$\n",
    "\\boldsymbol\\mu = \\begin{pmatrix}\\boldsymbol\\mu_1\\\\\\boldsymbol\\mu_2\\end{pmatrix},\n",
    "\\qquad\n",
    "\\boldsymbol\\Sigma = \\begin{pmatrix}\\boldsymbol\\Sigma_{11} & \\boldsymbol\\Sigma_{12}\\\\\n",
    "\\boldsymbol\\Sigma_{21} & \\boldsymbol\\Sigma_{22}\\end{pmatrix},\n",
    "$$\n",
    "\n",
    "and the conditional $\\mathbf x_1\\mid\\mathbf x_2$ is again Gaussian. We state its\n",
    "mean and covariance as a given fact (the derivation is a block-matrix computation,\n",
    "carried out in [@Bishop.2006], §2.3; Exercise 10 puts the formulas to work):\n",
    "\n",
    "$$\n",
    "\\boldsymbol\\mu_{1\\mid 2} = \\boldsymbol\\mu_1 + \\boldsymbol\\Sigma_{12}\\boldsymbol\\Sigma_{22}^{-1}(\\mathbf x_2-\\boldsymbol\\mu_2),\n",
    "\\qquad\n",
    "\\boldsymbol\\Sigma_{1\\mid 2} = \\boldsymbol\\Sigma_{11} - \\boldsymbol\\Sigma_{12}\\boldsymbol\\Sigma_{22}^{-1}\\boldsymbol\\Sigma_{21}.\n",
    "$$\n",
    "\n",
    "The conditional mean is a *linear* function of the observed block (a linear\n",
    "regression of $\\mathbf x_1$ on $\\mathbf x_2$ read straight off the covariance),\n",
    "and the conditional covariance\n",
    "$\\boldsymbol\\Sigma_{11}-\\boldsymbol\\Sigma_{12}\\boldsymbol\\Sigma_{22}^{-1}\\boldsymbol\\Sigma_{21}$\n",
    "is the *Schur complement* of $\\boldsymbol\\Sigma_{22}$, the standard name for the\n",
    "block that remains when the variables in $\\mathbf x_2$ are eliminated; note that\n",
    "it does not depend on the observed value $\\mathbf x_2$. This pair is the engine\n",
    "of Gaussian process regression (that section): condition the joint Gaussian\n",
    "prior over function values on the observed points and read off the predictive\n",
    "mean and variance.\n",
    "\n",
    "**Sampling via Cholesky.** How do we draw from\n",
    "$\\mathcal N(\\boldsymbol\\mu,\\boldsymbol\\Sigma)$ given only a standard-normal\n",
    "generator? This is the application promised when the Cholesky factorization was\n",
    "introduced (that section): factor\n",
    "$\\boldsymbol\\Sigma=\\mathbf L\\mathbf L^\\top$ with $\\mathbf L$ lower triangular,\n",
    "draw $\\mathbf z\\sim\\mathcal N(\\mathbf 0,\\mathbf I_d)$ ($d$ independent standard\n",
    "normals), and set $\\mathbf x=\\boldsymbol\\mu+\\mathbf L\\mathbf z$. Then $\\mathbf x$\n",
    "is Gaussian (a linear map of a Gaussian), its mean is $\\boldsymbol\\mu$ because\n",
    "$\\mathbb E[\\mathbf z]=\\mathbf 0$, and one line verifies the covariance:\n",
    "\n",
    "$$\n",
    "\\mathbb E\\bigl[(\\mathbf x-\\boldsymbol\\mu)(\\mathbf x-\\boldsymbol\\mu)^\\top\\bigr]\n",
    "= \\mathbf L\\,\\mathbb E[\\mathbf z\\mathbf z^\\top]\\,\\mathbf L^\\top\n",
    "= \\mathbf L\\,\\mathbf I\\,\\mathbf L^\\top\n",
    "= \\boldsymbol\\Sigma .\n",
    "$$\n",
    "\n",
    "This recipe is what the library samplers run: $d$ standard normals plus one\n",
    "triangular matrix multiply. The cell draws samples through the library sampler,\n",
    "estimates the covariance, checks that its eigenvalues recover the $\\{1,3\\}$\n",
    "built into $\\boldsymbol\\Sigma$, then re-samples by the Cholesky recipe and\n",
    "confirms that it reproduces $\\boldsymbol\\Sigma$ as well."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "distributions-mvn"
   },
   "source": [
    "import numpy as onp\n",
    "rng = onp.random.default_rng(0)\n",
    "mu_v = onp.array([0., 0.])\n",
    "Sigma = onp.array([[2., 1.], [1., 2.]])\n",
    "sample = rng.multivariate_normal(mu_v, Sigma, size=5000)\n",
    "emp = onp.cov(sample.T)\n",
    "vals, vecs = onp.linalg.eigh(emp)\n",
    "print('empirical covariance:\\n', emp.round(2))\n",
    "print('eigenvalues (≈ 1, 3):', vals.round(2))   # Sigma has eigenvalues 1 and 3\n",
    "L = onp.linalg.cholesky(Sigma)                   # Sigma = L L^T\n",
    "chol = mu_v + rng.standard_normal((5000, 2)) @ L.T  # x = mu + L z\n",
    "print('Cholesky-recipe covariance:\\n', onp.cov(chol.T).round(2))"
   ],
   "execution_count": 13,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "empirical covariance:\n [[1.96 1.03]\n [1.03 2.06]]\neigenvalues (≈ 1, 3): [0.98 3.04]\nCholesky-recipe covariance:\n [[1.93 0.97]\n [0.97 2.  ]]\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-15 fw=all -->\n",
    "\n",
    "**Probability in high dimension.** One more Gaussian fact matters enormously in\n",
    "deep learning and defies low-dimensional intuition. For\n",
    "$\\mathbf x\\sim\\mathcal N(\\mathbf 0,\\mathbf I_d)$ the squared norm\n",
    "$\\|\\mathbf x\\|^2=\\sum_{i=1}^d x_i^2$ is a sum of $d$ independent terms with mean\n",
    "$1$, so $\\mathbb E\\,\\|\\mathbf x\\|^2=d$, and, by the same averaging that makes\n",
    "sample means reliable, the sum concentrates near $d$: virtually all the mass\n",
    "of a high-dimensional Gaussian lives in a *thin shell* of radius\n",
    "$\\approx\\sqrt d$, far from the origin where the density is pointwise largest.\n",
    "Likewise two independent draws $\\mathbf x,\\mathbf y$ are *nearly orthogonal*:\n",
    "their cosine has mean $0$ and standard deviation $\\approx1/\\sqrt d$, a\n",
    "probabilistic version of the concentration-of-angles proposition of\n",
    "that section\n",
    "[@Vershynin.2018]; both facts get their full quantitative treatment, with\n",
    "exponential tail bounds, in that section.\n",
    "Both matter in practice:\n",
    "weight-initialization schemes scale by $1/\\sqrt{d}$ precisely so that\n",
    "$\\|\\mathbf W\\mathbf x\\|$ stays comparable to $\\|\\mathbf x\\|$ layer after layer;\n",
    "cosine similarity between embeddings is informative because unrelated\n",
    "high-dimensional vectors sit near cosine $0$ by default; and nearest-neighbor\n",
    "search loses contrast because the distances between random points all\n",
    "concentrate around one common value. The check below watches both\n",
    "concentrations set in as $d$ grows."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mdl-distributions-multivariate-gaussian-1"
   },
   "source": [
    "rng = np.random.default_rng(0)\n",
    "for d in (3, 30, 300):\n",
    "    x, y = rng.standard_normal((2, 1000, d))\n",
    "    nx = np.linalg.norm(x, axis=1)\n",
    "    cos = (x * y).sum(1) / (nx * np.linalg.norm(y, axis=1))\n",
    "    print(f'd = {d:3d}:  ||x||/sqrt(d) = {(nx / np.sqrt(d)).mean():.3f}'\n",
    "          f' ± {(nx / np.sqrt(d)).std():.3f},   mean |cos(x,y)| = {np.abs(cos).mean():.3f}')"
   ],
   "execution_count": 14,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "d =   3:  ||x||/sqrt(d) = 0.920 ± 0.378,   mean |cos(x,y)| = 0.497\nd =  30:  ||x||/sqrt(d) = 0.992 ± 0.127,   mean |cos(x,y)| = 0.145\nd = 300:  ||x||/sqrt(d) = 1.000 ± 0.039,   mean |cos(x,y)| = 0.046\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-16 fw=all -->\n",
    "\n",
    "The norm ratio locks onto $1$ with a spread shrinking like $1/\\sqrt{2d}$, and\n",
    "the typical cosine collapses toward $0$ at the promised $1/\\sqrt d$ rate: in\n",
    "high dimension, Gaussian geometry is a thin shell of mutually near-orthogonal\n",
    "directions.\n",
    "\n",
    "## The Exponential Family\n",
    "\n",
    "Nearly every distribution above (Bernoulli, categorical, binomial, Poisson,\n",
    "exponential, Gaussian, Laplace with fixed location) is a special case of one\n",
    "form, the *exponential family*. A distribution\n",
    "belongs to it if its density (or mass) can be written\n",
    "\n",
    "$$\n",
    "p(\\mathbf x\\mid\\boldsymbol\\eta) = h(\\mathbf x)\\,\\exp\\!\\bigl(\\boldsymbol\\eta^\\top T(\\mathbf x) - A(\\boldsymbol\\eta)\\bigr).\n",
    "$$\n",
    "\n",
    "The four ingredients each play a clear role. The *base measure* $h(\\mathbf x)$ is a\n",
    "fixed background that does not depend on the parameters. The *natural parameters*\n",
    "$\\boldsymbol\\eta$ are the knobs. The *sufficient statistics* $T(\\mathbf x)$ are the\n",
    "only features of the data that matter: given $T(\\mathbf x)$, no other detail of\n",
    "$\\mathbf x$ affects the probability, which is exactly what \"sufficient\" means. The\n",
    "*log-partition function* (or cumulant function)\n",
    "\n",
    "$$\n",
    "A(\\boldsymbol\\eta) = \\log\\!\\int h(\\mathbf x)\\,\\exp\\!\\bigl(\\boldsymbol\\eta^\\top T(\\mathbf x)\\bigr)\\,d\\mathbf x\n",
    "$$\n",
    "\n",
    "is whatever it takes to make the equation integrate to one.\n",
    "\n",
    "### Recognizing Old Friends\n",
    "\n",
    "The form looks abstract until we match the pieces. The **Bernoulli**\n",
    "$p^x(1-p)^{1-x}$ rewrites as\n",
    "\n",
    "$$\n",
    "\\exp\\!\\Bigl(x\\log\\tfrac{p}{1-p} + \\log(1-p)\\Bigr),\n",
    "$$\n",
    "\n",
    "so $T(x)=x$, the natural parameter $\\eta=\\log\\tfrac{p}{1-p}$ is the *logit*, and\n",
    "$A(\\eta)=\\log(1+e^\\eta)$ is the *softplus*; the link between probabilities and\n",
    "logits is built into the family. The **Poisson** $\\tfrac{\\lambda^k e^{-\\lambda}}{k!}$\n",
    "has base measure $h(k)=1/k!$, statistic $T(k)=k$, parameter $\\eta=\\log\\lambda$, and\n",
    "$A(\\eta)=e^\\eta=\\lambda$. The **Gaussian** with unknown mean and variance is a\n",
    "two-parameter member: completing the exponent,\n",
    "\n",
    "$$\n",
    "\\frac{1}{\\sqrt{2\\pi}}\\exp\\!\\Bigl(\\underbrace{\\tfrac{\\mu}{\\sigma^2}}_{\\eta_1}\\,x\n",
    "+ \\underbrace{\\bigl(-\\tfrac{1}{2\\sigma^2}\\bigr)}_{\\eta_2}\\,x^2\n",
    "- \\underbrace{\\bigl(\\tfrac{\\mu^2}{2\\sigma^2}+\\log\\sigma\\bigr)}_{A(\\boldsymbol\\eta)}\\Bigr),\n",
    "$$\n",
    "\n",
    "so $T(x)=(x,x^2)^\\top$ with base measure $h(x)=1/\\sqrt{2\\pi}$. The second natural\n",
    "parameter is negative, $\\eta_2=-1/(2\\sigma^2)<0$, which is what forces the\n",
    "tail-suppressing $e^{-x^2}$ decay. The exact split between $h$, $\\eta$, and $T$ is\n",
    "not unique; what matters is that the distribution *fits the form at all*.\n",
    "\n",
    "Membership has edges. The conspicuous absentees from the list above\n",
    "are the two *uniforms*: when the endpoints are unknown parameters, the support of\n",
    "$U(a,b)$ (and likewise of $U(n)$) moves with those parameters, whereas every member\n",
    "of the equation has the fixed, parameter-free support of its base\n",
    "measure $h$. This is the classic counterexample. Heavy-tailed laws live outside\n",
    "the family for a different reason. The *Student-$t$* takes two lines to define:\n",
    "draw a random precision $1/\\sigma^2$ from a Gamma distribution (a two-parameter\n",
    "density on the positive half-line, defined in full among the conjugate priors\n",
    "below), then draw a Gaussian with that precision. Averaging the Gaussian over its\n",
    "random scale fattens the $e^{-x^2}$ tails into polynomial ones, and the Cauchy of\n",
    "that section is its one-degree-of-freedom extreme. The\n",
    "support here is all of $\\mathbb R$, fixed, yet the mixing destroys the exponential\n",
    "form: no fixed finite-dimensional sufficient statistic exists, the situation\n",
    "described by the *Pitman--Koopman--Darmois* theorem [@Bishop.2006]. The\n",
    "exclusion costs two conveniences: these laws admit no finite-dimensional\n",
    "conjugate prior, and their negative log-likelihoods are not convex in the\n",
    "parameters. Both conveniences are exponential-family privileges, as the rest of\n",
    "this section shows.\n",
    "\n",
    "### Where the Form Comes From: Maximum Entropy\n",
    "\n",
    "The exponential form is exactly what *maximizing\n",
    "entropy* produces. The two maximum-entropy facts cited earlier\n",
    "[@Cover.Thomas.1999] are the same\n",
    "statement seen twice: the discrete uniform is the maximum-entropy distribution on a\n",
    "finite set when we fix *nothing* but the support, and the Gaussian is the\n",
    "maximum-entropy distribution on the line when we fix the *mean and variance*. In\n",
    "general, maximizing the entropy *relative to the base measure* $h$,\n",
    "$H_h[p]=-\\int p\\log\\tfrac{p}{h}$, subject to fixing the expected\n",
    "sufficient statistics $\\mathbb E[T(\\mathbf x)]=\\boldsymbol\\tau$ has, by a Lagrange\n",
    "multiplier argument (that section), the solution\n",
    "\n",
    "$$\n",
    "p(\\mathbf x) \\propto h(\\mathbf x)\\,\\exp\\!\\bigl(\\boldsymbol\\eta^\\top T(\\mathbf x)\\bigr),\n",
    "$$\n",
    "\n",
    "which is precisely the equation: the multipliers $\\boldsymbol\\eta$\n",
    "are the natural parameters and $A(\\boldsymbol\\eta)$ is again the normalizer\n",
    "[@Murphy.2022; @Wainwright.Jordan.2008]. One subtlety: the base measure does\n",
    "real work here. With plain Shannon entropy, i.e. $h\\equiv1$, fixing the mean on\n",
    "$\\{0,1,2,\\dots\\}$ yields the *geometric* distribution; only entropy relative to\n",
    "$h(k)=1/k!$ yields the Poisson. For the uniform and the Gaussian above, $h$ is\n",
    "constant, so plain entropy was the right notion all along. So the exponential\n",
    "family is the *least-committal* family\n",
    "consistent with knowing a fixed set of averages: the uniform fixes none, the\n",
    "Bernoulli/categorical fix the class probabilities, and the Gaussian fixes the first\n",
    "two moments.\n",
    "\n",
    "### The Moment Property\n",
    "\n",
    "The single most useful fact about the equation is that $A$ is a moment\n",
    "generator: *differentiating the log-partition function returns the mean of the\n",
    "sufficient statistics.*\n",
    "\n",
    "**Proposition (mean of the sufficient statistics).**\n",
    "*For an exponential-family distribution the equation,*\n",
    "\n",
    "$$\n",
    "\\nabla_{\\boldsymbol\\eta} A(\\boldsymbol\\eta) = \\mathbb E[T(\\mathbf x)].\n",
    "$$\n",
    "\n",
    "**Proof.** Differentiate the equation under the integral sign;\n",
    "we grant the interchange of derivative and integral, which is valid on the\n",
    "interior of the natural-parameter domain [@Folland.1999].\n",
    "Writing $Z(\\boldsymbol\\eta)=e^{A(\\boldsymbol\\eta)}=\\int h(\\mathbf x)e^{\\boldsymbol\\eta^\\top T(\\mathbf x)}d\\mathbf x$,\n",
    "the derivative of the log is $\\nabla A = (\\nabla Z)/Z$, and\n",
    "\n",
    "$$\n",
    "\\nabla_{\\boldsymbol\\eta} Z\n",
    "= \\int h(\\mathbf x)\\,T(\\mathbf x)\\,e^{\\boldsymbol\\eta^\\top T(\\mathbf x)}\\,d\\mathbf x .\n",
    "$$\n",
    "\n",
    "Dividing by $Z=e^{A}$ pulls the $e^{-A}$ back inside, reconstituting the density\n",
    "the equation:\n",
    "\n",
    "$$\n",
    "\\nabla_{\\boldsymbol\\eta} A\n",
    "= \\frac{1}{Z}\\int h(\\mathbf x)\\,T(\\mathbf x)\\,e^{\\boldsymbol\\eta^\\top T(\\mathbf x)}\\,d\\mathbf x\n",
    "= \\int T(\\mathbf x)\\,\\underbrace{h(\\mathbf x)\\,e^{\\boldsymbol\\eta^\\top T(\\mathbf x) - A(\\boldsymbol\\eta)}}_{p(\\mathbf x\\mid\\boldsymbol\\eta)}\\,d\\mathbf x\n",
    "= \\mathbb E[T(\\mathbf x)] . \\quad\\blacksquare\n",
    "$$\n",
    "\n",
    "Differentiating once more (the same interchange) turns $\\nabla A=(\\nabla Z)/Z$\n",
    "into\n",
    "\n",
    "$$\n",
    "\\nabla^2 A\n",
    "= \\frac{\\nabla^2 Z}{Z} - \\frac{(\\nabla Z)(\\nabla Z)^\\top}{Z^2}\n",
    "= \\mathbb E[T\\,T^\\top] - \\mathbb E[T]\\,\\mathbb E[T]^\\top\n",
    "= \\operatorname{Cov}(T),\n",
    "$$\n",
    "\n",
    "so $A$ is *convex* (a covariance matrix is PSD, that section).\n",
    "Convexity pays off in fitting. For a sample $x_1,\\dots,x_n$ the negative\n",
    "log-likelihood is\n",
    "\n",
    "$$\n",
    "-\\sum_{i=1}^{n} \\log p(x_i\\mid\\boldsymbol\\eta)\n",
    "= n\\,A(\\boldsymbol\\eta) - \\boldsymbol\\eta^\\top \\sum_{i=1}^{n} T(x_i)\n",
    "- \\sum_{i=1}^{n} \\log h(x_i),\n",
    "$$\n",
    "\n",
    "convex in $\\boldsymbol\\eta$ because $A$ is. Setting its gradient to zero gives\n",
    "$n\\nabla A(\\boldsymbol\\eta)=\\sum_i T(x_i)$: by the moment property, maximum\n",
    "likelihood *matches the model's expected sufficient statistics to their empirical\n",
    "averages*, $\\mathbb E[T]=\\bar T$. Uniqueness of the optimum needs one more\n",
    "hypothesis, strict convexity: $\\operatorname{Cov}(T)\\succ0$, which holds when the\n",
    "representation is *minimal* (no linear combination of the sufficient statistics\n",
    "is constant). The section's own softmax parameterization the equation\n",
    "is *not* minimal: adding the same constant to every logit leaves every $p_k$\n",
    "unchanged, so the categorical negative log-likelihood is flat along the direction\n",
    "$\\mathbf 1$ and its optima form a line rather than a point, which is why\n",
    "implementations pin the shift (fixing one logit at zero, or subtracting the\n",
    "maximum). We verify the equation for the\n",
    "Bernoulli, where it should read $A'(\\eta)=\\sigma(\\eta)=\\mathbb E[x]=p$."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "distributions-exp-family"
   },
   "source": [
    "import numpy as onp\n",
    "eta = 0.7                                        # natural parameter (logit)\n",
    "A = lambda e: onp.log1p(onp.exp(e))                # Bernoulli log-partition (softplus)\n",
    "eps = 1e-6\n",
    "dA = (A(eta + eps) - A(eta - eps)) / (2 * eps)   # numerical dA/deta\n",
    "print('dA/deta      =', round(float(dA), 6))\n",
    "print('sigmoid(eta) = E[x] = p =', round(float(1 / (1 + onp.exp(-eta))), 6))"
   ],
   "execution_count": 15,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "dA/deta      = 0.668188\nsigmoid(eta) = E[x] = p = 0.668188\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-17 fw=all -->\n",
    "\n",
    "The derivative of the softplus is the sigmoid, and it lands exactly on the\n",
    "Bernoulli mean $p$. This is no coincidence: the sigmoid and softmax links that\n",
    "deep-learning classifiers use are *precisely* the maps from natural parameters to\n",
    "means in the Bernoulli and categorical families, and the convexity of $A$ is why\n",
    "their negative log-likelihoods are convex in the natural parameters. If those\n",
    "parameters are affine functions of fixed features, this gives a convex\n",
    "optimization problem. A neural network makes the natural parameters nonlinear\n",
    "functions of its weights, so the same loss is not generally convex in the\n",
    "weights. The exponential family still supplies the likelihood and the link\n",
    "functions used by standard deep-learning losses\n",
    "[@Bishop.2006; @Koller.Friedman.2009].\n",
    "\n",
    "## Conjugate Priors\n",
    "\n",
    "The family tree so far has one half missing. The arrows of\n",
    "the figure *build* distributions over data; but, as\n",
    "that section develops, fitting a model can also mean placing a *prior*\n",
    "over its parameters, and the parameters of the distributions above are themselves\n",
    "quantities to be inferred: the Bernoulli's $p$, the Poisson's $\\lambda$, the\n",
    "categorical's $\\mathbf p$. A *prior* over those parameters is again a distribution,\n",
    "and three new continuous laws, the **Beta**, the **Gamma**, and the\n",
    "**Dirichlet**, are exactly the priors that pair with the discrete\n",
    "distributions we have met. A prior is **conjugate** to a likelihood when the\n",
    "posterior belongs to the *same family* as the prior, so Bayesian updating just moves\n",
    "the parameters rather than changing the shape of the distribution. This closes the\n",
    "tree: a tier of prior nodes flanks the data distributions in\n",
    "the figure, joined to them by \"conjugate prior\" links.\n",
    "\n",
    "### Beta--Bernoulli: Counting with Pseudo-Counts\n",
    "\n",
    "The **Beta** distribution is a density on the unit interval $p\\in[0,1]$ (exactly\n",
    "the range of a probability) with two shape parameters $\\alpha,\\beta>0$,\n",
    "\n",
    "$$\n",
    "\\mathrm{Beta}(p\\mid\\alpha,\\beta) = \\frac{p^{\\alpha-1}(1-p)^{\\beta-1}}{B(\\alpha,\\beta)},\n",
    "$$\n",
    "\n",
    "where the normalizing constant is\n",
    "$B(\\alpha,\\beta)=\\Gamma(\\alpha)\\Gamma(\\beta)/\\Gamma(\\alpha+\\beta)$, with $\\Gamma$\n",
    "the Gamma function that the exercises of that section\n",
    "introduced (recall $\\Gamma(z+1)=z\\,\\Gamma(z)$ and $\\Gamma(n+1)=n!$). The mean\n",
    "takes one line from this normalizer:\n",
    "\n",
    "$$\n",
    "\\mathbb E[p]\n",
    "= \\frac{1}{B(\\alpha,\\beta)}\\int_0^1 p^{\\alpha}(1-p)^{\\beta-1}\\,dp\n",
    "= \\frac{B(\\alpha+1,\\beta)}{B(\\alpha,\\beta)}\n",
    "= \\frac{\\alpha}{\\alpha+\\beta},\n",
    "$$\n",
    "\n",
    "using $\\Gamma(\\alpha+1)=\\alpha\\,\\Gamma(\\alpha)$ to simplify the ratio; the same\n",
    "move one step further gives the variance\n",
    "$\\operatorname{Var}(p)=\\alpha\\beta/\\bigl((\\alpha+\\beta)^2(\\alpha+\\beta+1)\\bigr)$.\n",
    "The uniform prior on $[0,1]$ that that section will place\n",
    "on the coin's bias is the special case $\\alpha=\\beta=1$.\n",
    "\n",
    "Now take a Beta prior on a coin's bias $p$ and observe a Bernoulli/binomial sample\n",
    "with $x$ heads in $n$ flips. Multiplying the prior by the likelihood\n",
    "the equation,\n",
    "\n",
    "$$\n",
    "\\underbrace{p^{\\alpha-1}(1-p)^{\\beta-1}}_{\\text{prior}}\\cdot\n",
    "\\underbrace{p^{x}(1-p)^{n-x}}_{\\text{likelihood}}\n",
    "= p^{(\\alpha+x)-1}(1-p)^{(\\beta+n-x)-1},\n",
    "$$\n",
    "\n",
    "which, up to normalization, is again a Beta, now with parameters shifted by the\n",
    "data:\n",
    "\n",
    "$$\n",
    "p \\mid x \\;\\sim\\; \\mathrm{Beta}\\bigl(\\alpha+x,\\ \\beta+(n-x)\\bigr).\n",
    "$$\n",
    "\n",
    "This is the **pseudo-count** picture:\n",
    "$\\alpha$ and $\\beta$ act as *phantom* heads and tails seen\n",
    "before any real data, and observing $x$ real heads and $n-x$ real tails simply adds\n",
    "the real counts to the phantom ones. The posterior mean\n",
    "\n",
    "$$\n",
    "\\mathbb E[p\\mid x] = \\frac{\\alpha+x}{\\alpha+\\beta+n}\n",
    "$$\n",
    "\n",
    "is literally the heads-frequency with the phantom flips included; it interpolates\n",
    "between the prior mean $\\alpha/(\\alpha+\\beta)$ and the maximum-likelihood frequency\n",
    "$x/n$, and slides toward the data as $n$ grows, the same flattening of the prior\n",
    "that turns MAP into MLE in that section. With the uniform prior\n",
    "$\\alpha=\\beta=1$ it becomes *Laplace's rule of succession* $(x+1)/(n+2)$\n",
    "[@Laplace.1814]: estimate a\n",
    "probability by adding one phantom observation of each outcome. The cell verifies\n",
    "the update on a small sample."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "distributions-conjugate"
   },
   "source": [
    "alpha, beta = 2.0, 2.0                           # Beta prior pseudo-counts\n",
    "x, n = 7, 10                                     # 7 heads in 10 flips\n",
    "post_alpha, post_beta = alpha + x, beta + (n - x)  # Beta posterior update\n",
    "print('prior mean      =', round(alpha / (alpha + beta), 4))\n",
    "print('MLE  x/n        =', round(x / n, 4))\n",
    "print('posterior Beta  =', (post_alpha, post_beta))\n",
    "print('posterior mean  =', round(post_alpha / (post_alpha + post_beta), 4))"
   ],
   "execution_count": 16,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "prior mean      = 0.5\nMLE  x/n        = 0.7\nposterior Beta  = (9.0, 5.0)\nposterior mean  = 0.6429\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-18 fw=all -->\n",
    "\n",
    "The posterior mean sits between the prior's $0.5$ and the data's $0.7$, exactly as\n",
    "the pseudo-count sum predicts. the figure shows the\n",
    "same mechanism at work over a longer run of data: the flat $\\mathrm{Beta}(1,1)$\n",
    "prior sharpens into a posterior that piles up on the true bias, its width\n",
    "shrinking like $1/\\sqrt n$ as the real counts swamp the phantom ones.\n",
    "\n",
    "![Bayesian updating as sharpening. Starting from the flat $\\mathrm{Beta}(1,1)$ prior (left), observing $9$ heads in $13$ flips of a coin with true bias $\\theta^\\ast=0.7$ (dashed line) gives the $\\mathrm{Beta}(10,5)$ posterior (middle); after $130$ flips the $\\mathrm{Beta}(91,41)$ posterior has piled up tightly around $\\theta^\\ast$ (right). Each update just adds the observed heads and tails to the pseudo-counts, and the posterior standard deviation shrinks like $1/\\sqrt{n}$.](../img/mdl-prob-beta-posterior.svg)\n",
    "\n",
    "### The Rest of the Tier, and the General Fact\n",
    "\n",
    "The other discrete laws have their own conjugate partners, built the same way.\n",
    "\n",
    "* **Gamma--Poisson.** The **Gamma** distribution\n",
    "  $\\mathrm{Gamma}(\\lambda\\mid\\alpha,\\beta)=\\frac{\\beta^\\alpha}{\\Gamma(\\alpha)}\\,\\lambda^{\\alpha-1}e^{-\\beta\\lambda}$\n",
    "  is a density on the rate $\\lambda>0$ with mean $\\alpha/\\beta$ and variance\n",
    "  $\\alpha/\\beta^2$. It is conjugate to the Poisson by the same one-line move as\n",
    "  the Beta: $n$ observed counts $x_1,\\dots,x_n$ contribute the likelihood\n",
    "  $\\propto\\lambda^{\\sum_i x_i}e^{-n\\lambda}$, and multiplying by the prior\n",
    "  collects powers into $\\lambda^{\\alpha+\\sum_i x_i-1}e^{-(\\beta+n)\\lambda}$, the\n",
    "  posterior $\\mathrm{Gamma}\\bigl(\\alpha+\\sum_i x_i,\\ \\beta+n\\bigr)$: pseudo-events\n",
    "  over a pseudo-window. Averaging the Poisson over this Gamma uncertainty in its\n",
    "  rate yields the *negative binomial*, the over-dispersed fix promised in the\n",
    "  Poisson section.\n",
    "* **Gamma as a waiting time.** The Gamma is more than a prior: with integer shape\n",
    "  $\\alpha=k$, $\\mathrm{Gamma}(k,\\beta)$ is the waiting time to the *$k$-th* event\n",
    "  of the Poisson process we named at the bus stop, the sum of $k$ independent\n",
    "  $\\mathrm{Exp}(\\beta)$ waits, extending the exponential's arrow in the family\n",
    "  tree just as the binomial extends the Bernoulli's.\n",
    "* **Dirichlet--Multinomial.** The **Dirichlet** distribution\n",
    "  $\\mathrm{Dir}(\\mathbf p\\mid\\boldsymbol\\alpha)=\\frac{\\Gamma(\\sum_k\\alpha_k)}{\\prod_k\\Gamma(\\alpha_k)}\\prod_k p_k^{\\alpha_k-1}$\n",
    "  is the multivariate Beta, a density over probability vectors on the simplex,\n",
    "  with mean $\\mathbb E[p_k]=\\alpha_k/\\sum_j\\alpha_j$. It is conjugate to the\n",
    "  categorical/multinomial by the same multiplication: the likelihood contributes\n",
    "  $\\prod_k p_k^{x_k}$, so observing class counts $\\mathbf x$ updates the prior to\n",
    "  $\\mathrm{Dir}(\\boldsymbol\\alpha+\\mathbf x)$, one pseudo-count per class.\n",
    "\n",
    "These are three instances of one theorem. **Every exponential-family likelihood\n",
    "has a conjugate prior**, provided the hyperparameters are chosen so that the\n",
    "prior below is normalizable, and it is itself an exponential family in the\n",
    "natural parameters $\\boldsymbol\\eta$: writing the prior as\n",
    "$p(\\boldsymbol\\eta)\\propto\\exp\\!\\bigl(\\boldsymbol\\nu^\\top\\boldsymbol\\eta - \\kappa\\,A(\\boldsymbol\\eta)\\bigr)$,\n",
    "multiplying by the likelihood the equation just adds the data's\n",
    "sufficient statistics into $\\boldsymbol\\nu$ and increments $\\kappa$ by the sample\n",
    "size. The pseudo-count update of the equation is this general\n",
    "mechanism specialized to the Bernoulli, and the hyperparameters\n",
    "$(\\boldsymbol\\nu,\\kappa)$ are *pseudo-data*: a prior sufficient statistic and a prior\n",
    "sample size [@Bishop.2006; @Murphy.2022]. Conjugacy makes the exponential\n",
    "family the natural home of tractable Bayesian inference as well as of the\n",
    "standard losses.\n",
    "\n",
    "## Summary\n",
    "\n",
    "The eleven data distributions of this section in one view (their three conjugate\n",
    "priors, the Beta, Gamma, and Dirichlet, close the family). For the categorical\n",
    "and multinomial rows we code a draw as a one-hot vector, respectively as the sum\n",
    "of $n$ one-hot vectors: the coding the cross-entropy loss already uses, and the\n",
    "object whose mean and covariance the table states.\n",
    "\n",
    "| Law | pmf or pdf | Mean | Variance | Where it shows up in DL |\n",
    "|:--|:--|:--|:--|:--|\n",
    "| $\\mathrm{Bernoulli}(p)$ | $p^x(1-p)^{1-x}$ | $p$ | $p(1-p)$ | binary classifier output, BCE loss |\n",
    "| $\\mathrm{Categorical}(\\mathbf p)$ | $p_k$ | $\\mathbf p$ | $\\operatorname{diag}(\\mathbf p)-\\mathbf p\\mathbf p^\\top$ | softmax output, cross-entropy loss |\n",
    "| $\\mathrm{Multinomial}(n,\\mathbf p)$ | the equation | $n\\mathbf p$ | $n\\bigl(\\operatorname{diag}(\\mathbf p)-\\mathbf p\\mathbf p^\\top\\bigr)$ | class counts over $n$ draws |\n",
    "| Uniform $U(n)$ | $1/n$ | $(n+1)/2$ | $(n^2-1)/12$ | index sampling, shuffling |\n",
    "| $\\mathrm{Binomial}(n,p)$ | $\\binom nk p^k(1-p)^{n-k}$ | $np$ | $np(1-p)$ | success counts over $n$ trials |\n",
    "| $\\mathrm{Poisson}(\\lambda)$ | $\\lambda^k e^{-\\lambda}/k!$ | $\\lambda$ | $\\lambda$ | rare-event counts |\n",
    "| Uniform $U(a,b)$ | $1/(b-a)$ on $[a,b]$ | $(a+b)/2$ | $(b-a)^2/12$ | raw randomness, initialization |\n",
    "| $\\mathrm{Exp}(\\lambda)$ | $\\lambda e^{-\\lambda x}$, $x\\ge0$ | $1/\\lambda$ | $1/\\lambda^2$ | waiting times, survival |\n",
    "| $\\mathcal N(\\mu,\\sigma^2)$ | $\\frac{1}{\\sqrt{2\\pi\\sigma^2}}e^{-(x-\\mu)^2/2\\sigma^2}$ | $\\mu$ | $\\sigma^2$ | MSE noise model, CLT, priors |\n",
    "| $\\mathrm{Laplace}(\\mu,b)$ | $\\frac{1}{2b}e^{-\\lvert x-\\mu\\rvert/b}$ | $\\mu$ | $2b^2$ | MAE loss, $L_1$ sparsity prior |\n",
    "| $\\mathcal N(\\boldsymbol\\mu,\\boldsymbol\\Sigma)$ | the equation | $\\boldsymbol\\mu$ | $\\boldsymbol\\Sigma$ | latent priors, Gaussian processes |\n",
    "\n",
    "* Distributions form a *family*: Bernoulli is the seed; summing $n$\n",
    "  gives the **Binomial** ($\\mu=np$, $\\sigma^2=np(1-p)$, derived as a sum of\n",
    "  Bernoullis); the many-rare limit gives the **Poisson** ($\\mu=\\sigma^2=\\lambda$,\n",
    "  derived from the binomial); the many-ordinary limit (CLT) gives the **Gaussian**.\n",
    "* The **Categorical** generalizes Bernoulli to $K$ outcomes; its softmax form is\n",
    "  every classifier's output layer and its NLL is the cross-entropy loss. The\n",
    "  **Multinomial** counts $n$ categorical draws.\n",
    "* The **continuous** laws: the **uniform** (raw randomness, $U(0,1)$ powers\n",
    "  sampling), the **exponential** (memoryless waiting times, partner of the\n",
    "  Poisson), the **Gaussian** (CLT limit, MSE noise model, max-entropy default), the\n",
    "  **Laplace** ($L_1$/MAE loss, sparsity prior), and the **multivariate Gaussian**,\n",
    "  whose elliptical contours are the eigendecomposition of its covariance, which is\n",
    "  closed under both linear maps and conditioning (the Schur-complement formulas,\n",
    "  the equation, that drive Gaussian-process regression), and\n",
    "  which is sampled by the Cholesky recipe\n",
    "  $\\mathbf x=\\boldsymbol\\mu+\\mathbf L\\mathbf z$.\n",
    "* Means and variances follow from structure (linearity of expectation,\n",
    "  limits, and standard integrals) rather than memorization.\n",
    "* The **exponential family** $p(\\mathbf x)=h(\\mathbf x)\\exp(\\boldsymbol\\eta^\\top\n",
    "  T(\\mathbf x)-A(\\boldsymbol\\eta))$ unifies almost all of them (the uniforms, whose\n",
    "  support moves with their parameters, stay outside); it is exactly the\n",
    "  maximum-entropy family (entropy taken relative to the base measure $h$) for a\n",
    "  fixed set of expected sufficient statistics. Its\n",
    "  log-partition $A$ generates moments: $\\nabla A(\\boldsymbol\\eta)=\\mathbb E[T(\\mathbf x)]$,\n",
    "  and $A$ is convex. Thus its negative log-likelihood is convex in the natural\n",
    "  parameters; with an affine parameter map this yields a convex generalized\n",
    "  linear model, whereas a neural-network parameterization remains nonconvex.\n",
    "* Every exponential-family likelihood has a **conjugate prior**: the **Beta** for the\n",
    "  Bernoulli/binomial, the **Gamma** for the Poisson, the **Dirichlet** for the\n",
    "  categorical/multinomial. Bayesian updating then just adds the data's sufficient\n",
    "  statistics to the prior's *pseudo-counts*: e.g. a $\\mathrm{Beta}(\\alpha,\\beta)$\n",
    "  prior and $x$ heads in $n$ flips give a $\\mathrm{Beta}(\\alpha+x,\\beta+n-x)$\n",
    "  posterior.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Let $X,Y\\sim\\mathrm{Binomial}(16,1/2)$ be independent. Find the standard\n",
    "   deviation of $X-Y$. (*Hint:* variances add for independent variables, even under\n",
    "   subtraction.)\n",
    "2. Show that the categorical negative log-likelihood $-\\sum_k y_k\\log p_k$ with\n",
    "   $p_k$ from the softmax the equation has gradient\n",
    "   $\\hat{\\mathbf p}-\\mathbf y$ with respect to the logits $\\mathbf z$. Why does this\n",
    "   form make softmax classification well behaved?\n",
    "3. Verify that the multinomial pmf the equation for $n$ trials,\n",
    "   $K=2$, reduces exactly to the binomial pmf the equation.\n",
    "4. Prove the converse of memorylessness: if a positive continuous random variable\n",
    "   has survival function $G(t)=P(X>t)$ satisfying $G(s+t)=G(s)\\,G(t)$ for all\n",
    "   $s,t\\ge0$, show that $G(t)=e^{-\\lambda t}$ for some $\\lambda>0$ (use\n",
    "   monotonicity of $G$). Then show that the minimum of independent\n",
    "   $\\mathrm{Exp}(\\lambda_i)$ is again exponential with rate $\\sum_i\\lambda_i$.\n",
    "5. For $X\\sim\\mathrm{Poisson}(\\lambda)$, argue that $(X-\\lambda)/\\sqrt\\lambda$\n",
    "   becomes approximately Gaussian as $\\lambda\\to\\infty$. (*Hint:* for integer\n",
    "   $\\lambda=m$, a $\\mathrm{Poisson}(m)$ variable is the sum of $m$ independent\n",
    "   $\\mathrm{Poisson}(1)$ variables, so the iid CLT applies as $m\\to\\infty$;\n",
    "   general $\\lambda$ follows by splitting off the fractional part.)\n",
    "6. Show that the maximum-likelihood location $\\mu$ under a $\\mathrm{Laplace}(\\mu,b)$\n",
    "   model minimizes $\\sum_i|x_i-\\mu|$ and hence is the sample *median*, while under a\n",
    "   Gaussian it is the mean.\n",
    "7. Write the **categorical** distribution over $K$ classes in exponential-family\n",
    "   form the equation. Identify $T(\\mathbf x)$ (one-hot) and the natural\n",
    "   parameters, and verify that $\\nabla A(\\boldsymbol\\eta)=\\mathbb E[T]$ recovers the\n",
    "   softmax the equation.\n",
    "8. Given $\\boldsymbol\\Sigma=\\left(\\begin{smallmatrix}2&1\\\\1&2\\end{smallmatrix}\\right)$,\n",
    "   find the eigenvalues and eigenvectors and state the axis directions and\n",
    "   half-lengths of the $\\mathcal N(\\mathbf 0,\\boldsymbol\\Sigma)$ density contours.\n",
    "   Show that a linear map $\\mathbf A\\mathbf x+\\mathbf b$ of a multivariate Gaussian\n",
    "   is again Gaussian with covariance $\\mathbf A\\boldsymbol\\Sigma\\mathbf A^\\top$.\n",
    "9. Start from a uniform $\\mathrm{Beta}(1,1)$ prior on a coin's bias and observe the\n",
    "   sequence \"HHHTHTTHHHHHT\" ($9$ heads, $4$ tails) of\n",
    "   that section. Give the posterior the equation\n",
    "   and its mean, and show that as the data grows the posterior mean converges to the\n",
    "   maximum-likelihood frequency $n_H/(n_H+n_T)$.\n",
    "10. Using the conditional formula the equation, take the\n",
    "    bivariate Gaussian with $\\boldsymbol\\mu=\\mathbf 0$ and\n",
    "    $\\boldsymbol\\Sigma=\\left(\\begin{smallmatrix}1&\\rho\\\\\\rho&1\\end{smallmatrix}\\right)$\n",
    "    and find the distribution of $x_1$ given $x_2$. Show its mean is $\\rho x_2$ and\n",
    "    its variance is $1-\\rho^2$, and explain why observing $x_2$ never *increases* the\n",
    "    uncertainty about $x_1$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-distributions-md-19 fw=mxnet -->\n",
    "\n",
    "[Discussions](https://d2l.discourse.group/t/417)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-mxnet",
   "display_name": "d2l (MXNet)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
