{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-mutual-information-md-1 fw=all -->\n",
    "\n",
    "# Mutual Information and Representation Learning\n",
    "\n",
    "The entropies and divergences of that section score\n",
    "one distribution at a time. This section asks the question that powers modern\n",
    "representation learning: how much information do *two* random variables share?\n",
    "The answer, *mutual information*, is the engine behind contrastive and\n",
    "self-supervised training: SimCLR, CPC, and CLIP-style dual encoders\n",
    "[@Chen.Kornblith.Norouzi.ea.2020; @Oord.Li.Vinyals.2018; @radford2021learning]\n",
    "are, formally, mutual-information maximizers, and the InfoNCE loss they\n",
    "minimize is a *lower bound* on mutual information. The same section also\n",
    "carries a warning: certifying a large mutual information from samples is\n",
    "provably hard (no estimator can certify more than about $\\log N$ nats from\n",
    "$N$ samples), so these objectives must be read as training signals, not as\n",
    "measurements. We build the quantity and its calculus first (including the\n",
    "*data-processing inequality*, the theorem that says representations can only\n",
    "lose information), then confront the estimation barrier, then derive the\n",
    "variational bounds and InfoNCE, and close with the information bottleneck and\n",
    "an account of what mutual information can and cannot tell you.\n",
    "\n",
    "As in that section, **we work in nats** (natural\n",
    "logarithm) throughout, flagging the occasional conversion to bits explicitly;\n",
    "recall $1 \\textrm{ bit} = \\ln 2 \\approx 0.693$ nats. We also lean on two\n",
    "facts from that section without re-proving them: Gibbs' inequality\n",
    "($D_{\\textrm{KL}} \\geq 0$ with equality iff the distributions agree) and the\n",
    "decomposition $\\textrm{CE} = H + D_{\\textrm{KL}}$.\n",
    "\n",
    "Most of this section is plain-NumPy probability; the deep-learning library\n",
    "appears when we *train* a critic at the end of the variational-bounds\n",
    "section."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mutual-information-imports"
   },
   "source": [
    "%matplotlib inline\n",
    "from d2l import mxnet as d2l\n",
    "from mxnet import autograd, gluon, init, np, npx\n",
    "npx.set_np()\n",
    "import numpy as onp  # plain NumPy for the probability demos\n",
    "from scipy.special import digamma"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-mutual-information-md-2 fw=all -->\n",
    "\n",
    "## Mutual Information: Definitions and Properties\n",
    "\n",
    "Throughout, $(X,Y)$ is a pair of random variables with joint distribution\n",
    "$P_{X,Y}$ and marginals $P_X,P_Y$. We begin the entropy bookkeeping in the\n",
    "discrete case. For continuous variables the same symbols denote differential\n",
    "entropies only when the relevant integrals are finite; the KL definition of\n",
    "mutual information below remains primary and avoids undefined differences such\n",
    "as $\\infty-\\infty$. Everything we build answers one question:\n",
    "what does observing $Y$ tell us about $X$?\n",
    "\n",
    "### From One Variable to Two: Joint and Conditional Entropy\n",
    "\n",
    "Entropy extends from one variable to a pair the obvious way. The *joint\n",
    "entropy* of $(X, Y)$ is the average surprise of observing the pair,\n",
    "\n",
    "$$H(X, Y) = -E_{(x, y) \\sim P_{X,Y}} [\\log p_{X, Y}(x, y)],$$\n",
    "\n",
    "a sum over outcomes. The two extremes orient the discrete definition: if\n",
    "$Y=X$, the pair carries no more information than either variable alone,\n",
    "$H(X,Y)=H(X)$; if $X$ and $Y$ are independent, surprises add,\n",
    "$H(X,Y)=H(X)+H(Y)$. Every discrete dependent pair sits between these bounds.\n",
    "For a continuous $Y=X$, by contrast, the joint law lies on a lower-dimensional\n",
    "diagonal, has no ordinary joint density with respect to area, and has infinite\n",
    "mutual information; the discrete identity must not be transferred verbatim.\n",
    "\n",
    "Often we care about a directed version of the question. Take $X$ to be the\n",
    "pixels of an image and $Y$ its class label. The image is a complicated,\n",
    "information-rich object, but once we have *seen* it, the label holds few\n",
    "surprises: a legible digit already tells you which digit it is. The\n",
    "*conditional entropy*\n",
    "\n",
    "$$H(Y \\mid X) = - E_{(x, y) \\sim P_{X,Y}} [\\log p(y \\mid x)],$$\n",
    "\n",
    "with $p(y \\mid x) = p_{X,Y}(x, y)/p_X(x)$, measures exactly this residual:\n",
    "the average surprise in $Y$ *after* $X$ is known. The three quantities are\n",
    "linked by a bookkeeping identity.\n",
    "\n",
    "**Proposition (chain rule for entropy).** *For discrete variables, or for\n",
    "continuous variables when the displayed entropies are finite,\n",
    "$H(X,Y)=H(X)+H(Y\\mid X)$.*\n",
    "\n",
    "**Proof.** Inside the expectation defining $H(X,Y)$, factor the joint density:\n",
    "$\\log p_{X,Y}(x,y) = \\log p_X(x) + \\log p(y \\mid x)$. Taking\n",
    "$-E_{(x,y) \\sim P_{X,Y}}$ of both sides turns the left side into $H(X,Y)$ and\n",
    "the two right-hand terms into $H(X)$ and $H(Y \\mid X)$.\n",
    "$\\blacksquare$\n",
    "\n",
    "Read it as an accounting statement: the total information in the pair is the\n",
    "information in $X$ plus whatever $Y$ adds once $X$ is known.\n",
    "\n",
    "Let's make the definitions concrete on a $2 \\times 2$ joint distribution.\n",
    "Take\n",
    "\n",
    "$$\n",
    "p_{X,Y} = \\begin{pmatrix} 0.1 & 0.4 \\\\ 0.2 & 0.3 \\end{pmatrix},\n",
    "$$\n",
    "\n",
    "with rows indexing $x \\in \\{0, 1\\}$ and columns $y \\in \\{0, 1\\}$; the entries\n",
    "sum to $1$ as a joint p.m.f. must. The marginals come *from the joint* by\n",
    "summing rows and columns: $p_X = (0.5, 0.5)$ and $p_Y = (0.3, 0.7)$. From the\n",
    "definitions, $H(X,Y) \\approx 1.2799$ nats, $H(X) = \\ln 2 \\approx 0.6931$\n",
    "nats, $H(Y) \\approx 0.6109$ nats, and the chain rule predicts\n",
    "$H(Y \\mid X) = H(X,Y) - H(X) \\approx 0.5867$ nats, which the direct\n",
    "definition the equation confirms."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mutual-information-joint-conditional"
   },
   "source": [
    "p_xy = onp.array([[0.1, 0.4],\n",
    "                  [0.2, 0.3]])           # a valid joint: entries sum to 1\n",
    "p_x, p_y = p_xy.sum(axis=1), p_xy.sum(axis=0)  # marginals FROM the joint\n",
    "\n",
    "entropy = lambda p: float(-(p * onp.log(p)).sum())\n",
    "H_xy, H_x, H_y = entropy(p_xy), entropy(p_x), entropy(p_y)\n",
    "# Conditional entropy directly: p(y|x) = p(x,y) / p(x), weighted by p(x,y)\n",
    "H_y_given_x = float(-(p_xy * onp.log(p_xy / p_x[:, None])).sum())\n",
    "\n",
    "print(f'marginals: p_x = {p_x}, p_y = {p_y}')\n",
    "print(f'H(X,Y) = {H_xy:.4f}, H(X) = {H_x:.4f}, H(Y) = {H_y:.4f} nats')\n",
    "print(f'H(Y|X) = {H_y_given_x:.4f} = H(X,Y) - H(X) = {H_xy - H_x:.4f} nats')"
   ],
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "marginals: p_x = [0.5 0.5], p_y = [0.3 0.7]\nH(X,Y) = 1.2799, H(X) = 0.6931, H(Y) = 0.6109 nats\nH(Y|X) = 0.5867 = H(X,Y) - H(X) = 0.5867 nats\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-mutual-information-md-3 fw=all -->\n",
    "\n",
    "### Mutual Information as a Divergence from Independence\n",
    "\n",
    "The conditional entropy measured what $Y$ keeps *to itself*; we want what\n",
    "$X$ and $Y$ *share*. The definition needs no entropy at all. If $X$ and $Y$\n",
    "were independent, their joint\n",
    "distribution would be the product of marginals $P_X \\otimes P_Y$, with density\n",
    "$p_X(x)\\,p_Y(y)$. The *mutual information* is the KL divergence from that\n",
    "hypothetical independent world to the actual joint:\n",
    "\n",
    "$$I(X; Y) = D_{\\textrm{KL}}\\!\\left(P_{X,Y} \\,\\|\\, P_X \\otimes P_Y\\right) = E_{(x, y) \\sim P_{X,Y}} \\left[ \\log\\frac{p_{X, Y}(x, y)}{p_X(x)\\, p_Y(y)} \\right].$$\n",
    "\n",
    "In words: mutual information is *how far the pair is from being independent*,\n",
    "measured in the same units as every other quantity in this chapter. (The\n",
    "semicolon in $I(X; Y)$ is the standard notation, distinguishing the two\n",
    "arguments from a single joint object.) The definition presumes the joint has\n",
    "a density with respect to the product of the marginals; when it does not,\n",
    "for instance when $Y = f(X)$ is a deterministic function of a continuous\n",
    "$X$, the divergence, and hence $I(X; Y)$, is infinite. Symmetry and\n",
    "nonnegativity are immediate from the divergence form, and the entropy\n",
    "identities follow by expanding the logarithm.\n",
    "\n",
    "**Proposition (properties of mutual information).** *For any pair $(X,Y)$,\n",
    "$I(X;Y)$ is symmetric and non-negative, with equality if and only if $X$ and\n",
    "$Y$ are independent. When the relevant discrete or differential entropies are\n",
    "finite, it also satisfies*\n",
    "\n",
    "$$\n",
    "I(X;Y)=H(X)+H(Y)-H(X,Y)=H(X)-H(X\\mid Y)=H(Y)-H(Y\\mid X).\n",
    "$$\n",
    "\n",
    "**Proof.** When the entropy terms are finite, expand the logarithm in\n",
    "the equation:\n",
    "\n",
    "$$\n",
    "I(X; Y)\n",
    "= E\\left[\\log p_{X,Y}(x,y)\\right] - E\\left[\\log p_X(x)\\right] - E\\left[\\log p_Y(y)\\right]\n",
    "= -H(X,Y) + H(X) + H(Y).\n",
    "$$\n",
    "\n",
    "Substituting the chain rule $H(X,Y) = H(Y) + H(X \\mid Y)$ gives\n",
    "$I(X;Y) = H(X) - H(X \\mid Y)$, and the symmetric substitution gives\n",
    "$H(Y) - H(Y \\mid X)$. Symmetry is immediate from\n",
    "the equation, whose density ratio is unchanged when $X$ and $Y$\n",
    "are swapped. Nonnegativity is Gibbs' inequality\n",
    "(that section) applied to the two distributions\n",
    "$P_{X,Y}$ and $P_X \\otimes P_Y$: the divergence is non-negative, and zero\n",
    "exactly when they coincide, i.e., when\n",
    "$p_{X,Y}(x,y) = p_X(x)\\,p_Y(y)$ everywhere, which is the definition of\n",
    "independence. $\\blacksquare$\n",
    "\n",
    "The identities have a picture, the figure: draw $H(X)$ and\n",
    "$H(Y)$ as overlapping disks. The overlap is $I(X;Y)$, the crescents are the\n",
    "conditional entropies, the union is the joint entropy, and each identity in\n",
    "claim 1 is one way of reading off the overlap's area. The picture also\n",
    "explains the directed reading: $I(X;Y) = H(X) - H(X \\mid Y)$ is the\n",
    "*reduction in surprise* about $X$ that observing $Y$ buys.\n",
    "\n",
    "![Mutual information $I(X;Y)$ as the overlap of the entropies $H(X)$ and $H(Y)$: the crescents are the conditional entropies $H(X\\mid Y)$ and $H(Y\\mid X)$, the union is the joint entropy $H(X,Y)$, and the three equivalent expressions for $I(X;Y)$ follow by reading off the areas.](../img/mdl-it-mi-overlap.svg)\n",
    "\n",
    "Back to the worked example. With marginals computed from the joint, the\n",
    "divergence form the equation and the entropy identity must\n",
    "return the same number, and they do: $I(X;Y) \\approx 0.0242$ nats\n",
    "($\\approx 0.035$ bits after dividing by $\\ln 2$). The pair is *barely*\n",
    "dependent: the joint $\\begin{smallmatrix}0.1 & 0.4\\\\0.2 & 0.3\\end{smallmatrix}$\n",
    "is close to the product of its marginals\n",
    "$\\begin{smallmatrix}0.15 & 0.35\\\\0.15 & 0.35\\end{smallmatrix}$, and mutual\n",
    "information quantifies the gap."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mutual-information-discrete-joint"
   },
   "source": [
    "def mutual_information(p_xy):\n",
    "    \"\"\"I(X;Y) of a discrete joint p.m.f., in nats.\"\"\"\n",
    "    p_x = p_xy.sum(axis=1, keepdims=True)   # marginals from the joint\n",
    "    p_y = p_xy.sum(axis=0, keepdims=True)\n",
    "    return float((p_xy * onp.log(p_xy / (p_x * p_y))).sum())\n",
    "\n",
    "mi_kl = mutual_information(p_xy)            # divergence form\n",
    "mi_ent = H_x + H_y - H_xy                    # entropy identity\n",
    "print(f'I(X;Y) = {mi_kl:.4f} nats (KL form) '\n",
    "      f'= {mi_ent:.4f} nats (entropy identity)')\n",
    "print(f'       = {mi_kl / onp.log(2):.4f} bits')"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "I(X;Y) = 0.0242 nats (KL form) = 0.0242 nats (entropy identity)\n       = 0.0349 bits\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-mutual-information-md-4 fw=all -->\n",
    "\n",
    "### The Gaussian Anchor\n",
    "\n",
    "For continuous variables the canonical worked example is the bivariate\n",
    "Gaussian, where mutual information has a closed form that we will use as\n",
    "*ground truth* for every estimator in the rest of this section.\n",
    "\n",
    "**Proposition (Gaussian mutual information).** *If $(X, Y)$ is bivariate\n",
    "Gaussian with correlation coefficient $\\rho$, then*\n",
    "\n",
    "$$I(X; Y) = -\\tfrac{1}{2} \\log\\left(1 - \\rho^2\\right).$$\n",
    "\n",
    "**Proof.** Mutual information is invariant under separate invertible maps of\n",
    "$X$ and of $Y$ (proved in the next subsection), so we may standardize both\n",
    "variables and take\n",
    "\n",
    "$$\n",
    "\\Sigma = \\begin{pmatrix} 1 & \\rho \\\\ \\rho & 1 \\end{pmatrix},\n",
    "\\qquad \\det \\Sigma = 1 - \\rho^2.\n",
    "$$\n",
    "\n",
    "For a standard Gaussian the differential entropy is\n",
    "$h(X) = h(Y) = \\tfrac{1}{2}\\log(2\\pi e)$, a value computed explicitly inside\n",
    "the maximum-entropy proof of that section; the\n",
    "same integration against the bivariate density gives\n",
    "$h(X, Y) = \\tfrac{1}{2}\\log\\left((2\\pi e)^2 \\det\\Sigma\\right)$. The entropy\n",
    "identity of claim 1 carries over: joint and conditional differential\n",
    "entropies are the same formulas with integrals in place of sums, and the\n",
    "chain-rule algebra never used discreteness. It gives\n",
    "\n",
    "$$\n",
    "I(X; Y) = h(X) + h(Y) - h(X, Y)\n",
    "= \\log(2\\pi e) - \\tfrac{1}{2}\\log\\left((2\\pi e)^2 (1 - \\rho^2)\\right)\n",
    "= -\\tfrac{1}{2}\\log(1 - \\rho^2).\n",
    "$$\n",
    "\n",
    "Each differential entropy individually depends on the coordinates (recall the\n",
    "caveat of that section: differential entropy can even\n",
    "be negative), but the offending terms cancel in the combination: mutual\n",
    "information is a *relative* quantity. $\\blacksquare$\n",
    "\n",
    "Read the limits: at $\\rho = 0$ the variables are independent (for Gaussians,\n",
    "uncorrelated implies independent) and $I = 0$; as $\\rho \\to \\pm 1$ the joint\n",
    "density collapses onto a line, $Y$ determines $X$, and $I \\to \\infty$. In\n",
    "between, the formula gives concrete targets: $\\rho = 0.5$ carries\n",
    "$\\approx 0.144$ nats, $\\rho = 0.9$ carries $\\approx 0.830$ nats, and\n",
    "$\\rho = 0.99$ carries $\\approx 1.96$ nats.\n",
    "\n",
    "Let's verify the equation empirically with the simplest\n",
    "estimator there is: bin the samples into a 2-D histogram and apply the\n",
    "discrete formula to the bin frequencies."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mutual-information-gaussian-binned"
   },
   "source": [
    "def gauss_pair(rng, rho, n):\n",
    "    \"\"\"n samples of a standard bivariate Gaussian with correlation rho.\"\"\"\n",
    "    x = rng.standard_normal(n)\n",
    "    y = rho * x + onp.sqrt(1 - rho**2) * rng.standard_normal(n)\n",
    "    return x, y\n",
    "\n",
    "def binned_mi(x, y, bins=40):\n",
    "    \"\"\"Plug-in MI estimate from a 2-D histogram, in nats.\"\"\"\n",
    "    counts, _, _ = onp.histogram2d(x, y, bins=bins)\n",
    "    p = counts / counts.sum()\n",
    "    p_x, p_y = p.sum(axis=1, keepdims=True), p.sum(axis=0, keepdims=True)\n",
    "    mask = p > 0\n",
    "    return float((p[mask] * onp.log(p[mask] / (p_x * p_y)[mask])).sum())\n",
    "\n",
    "rng = onp.random.default_rng(42)\n",
    "rhos = onp.array([0.0, 0.3, 0.6, 0.9, 0.99])\n",
    "est = [binned_mi(*gauss_pair(rng, r, 200_000)) for r in rhos]\n",
    "true = [-0.5 * onp.log(1 - r**2) for r in rhos]\n",
    "for r, e, t in zip(rhos, est, true):\n",
    "    print(f'rho = {r:4.2f}: binned = {e:.4f}, closed form = {t:.4f} nats')\n",
    "d2l.plot(rhos, [onp.array(est), onp.array(true)], 'correlation rho',\n",
    "         'I(X;Y) (nats)', legend=['binned estimate', 'closed form'],\n",
    "         figsize=(5, 3))"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "rho = 0.00: binned = 0.0027, closed form = -0.0000 nats\nrho = 0.30: binned = 0.0504, closed form = 0.0472 nats\nrho = 0.60: binned = 0.2235, closed form = 0.2231 nats\nrho = 0.90: binned = 0.8111, closed form = 0.8304 nats\nrho = 0.99: binned = 1.7829, closed form = 1.9585 nats\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"329.98125pt\" height=\"211.07625pt\" viewBox=\"0 0 329.98125 211.07625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-19T14:00:59.258149</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 211.07625 \nL 329.98125 211.07625 \nL 329.98125 0 \nL 0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 43.78125 173.52 \nL 322.78125 173.52 \nL 322.78125 7.2 \nL 43.78125 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path d=\"M 56.463068 173.52 \nL 56.463068 7.2 \n\" clip-path=\"url(#p1c1c9ae910)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path id=\"m4a0ce2b1a0\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m4a0ce2b1a0\" x=\"56.463068\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0.0 -->\n      <g transform=\"translate(48.511506 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \nL 1344 794 \nL 1344 0 \nL 684 0 \nL 684 794 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path d=\"M 107.702738 173.52 \nL 107.702738 7.2 \n\" clip-path=\"url(#p1c1c9ae910)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#m4a0ce2b1a0\" x=\"107.702738\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 0.2 -->\n      <g transform=\"translate(99.751175 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path d=\"M 158.942407 173.52 \nL 158.942407 7.2 \n\" clip-path=\"url(#p1c1c9ae910)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#m4a0ce2b1a0\" x=\"158.942407\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 0.4 -->\n      <g transform=\"translate(150.990845 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_7\">\n      <path d=\"M 210.182076 173.52 \nL 210.182076 7.2 \n\" clip-path=\"url(#p1c1c9ae910)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#m4a0ce2b1a0\" x=\"210.182076\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 0.6 -->\n      <g transform=\"translate(202.230514 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-36\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_9\">\n      <path d=\"M 261.421746 173.52 \nL 261.421746 7.2 \n\" clip-path=\"url(#p1c1c9ae910)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#m4a0ce2b1a0\" x=\"261.421746\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 0.8 -->\n      <g transform=\"translate(253.470183 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-38\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_11\">\n      <path d=\"M 312.661415 173.52 \nL 312.661415 7.2 \n\" clip-path=\"url(#p1c1c9ae910)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_12\">\n      <g>\n       <use xlink:href=\"#m4a0ce2b1a0\" x=\"312.661415\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 1.0 -->\n      <g transform=\"translate(304.709853 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- correlation rho -->\n     <g transform=\"translate(146.669531 201.796563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \nL 3122 2828 \nQ 2878 2963 2633 3030 \nQ 2388 3097 2138 3097 \nQ 1578 3097 1268 2742 \nQ 959 2388 959 1747 \nQ 959 1106 1268 751 \nQ 1578 397 2138 397 \nQ 2388 397 2633 464 \nQ 2878 531 3122 666 \nL 3122 134 \nQ 2881 22 2623 -34 \nQ 2366 -91 2075 -91 \nQ 1284 -91 818 406 \nQ 353 903 353 1747 \nQ 353 2603 823 3093 \nQ 1294 3584 2113 3584 \nQ 2378 3584 2631 3529 \nQ 2884 3475 3122 3366 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-63\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(54.980469 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(116.162109 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(155.525391 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(194.388672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(255.912109 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(283.695312 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(344.974609 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(384.183594 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(411.966797 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(473.148438 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(536.527344 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(568.314453 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(607.677734 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(671.056641 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_13\">\n      <path d=\"M 43.78125 165.96 \nL 322.78125 165.96 \n\" clip-path=\"url(#p1c1c9ae910)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_14\">\n      <defs>\n       <path id=\"m94e93802e1\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m94e93802e1\" x=\"43.78125\" y=\"165.96\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.0 -->\n      <g transform=\"translate(20.878125 169.759219) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_15\">\n      <path d=\"M 43.78125 127.35938 \nL 322.78125 127.35938 \n\" clip-path=\"url(#p1c1c9ae910)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_16\">\n      <g>\n       <use xlink:href=\"#m94e93802e1\" x=\"43.78125\" y=\"127.35938\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.5 -->\n      <g transform=\"translate(20.878125 131.158599) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \nL 3169 4666 \nL 3169 4134 \nL 1269 4134 \nL 1269 2991 \nQ 1406 3038 1543 3061 \nQ 1681 3084 1819 3084 \nQ 2600 3084 3056 2656 \nQ 3513 2228 3513 1497 \nQ 3513 744 3044 326 \nQ 2575 -91 1722 -91 \nQ 1428 -91 1123 -41 \nQ 819 9 494 109 \nL 494 744 \nQ 775 591 1075 516 \nQ 1375 441 1709 441 \nQ 2250 441 2565 725 \nQ 2881 1009 2881 1497 \nQ 2881 1984 2565 2268 \nQ 2250 2553 1709 2553 \nQ 1456 2553 1204 2497 \nQ 953 2441 691 2322 \nL 691 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_17\">\n      <path d=\"M 43.78125 88.75876 \nL 322.78125 88.75876 \n\" clip-path=\"url(#p1c1c9ae910)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_18\">\n      <g>\n       <use xlink:href=\"#m94e93802e1\" x=\"43.78125\" y=\"88.75876\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 1.0 -->\n      <g transform=\"translate(20.878125 92.557979) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_19\">\n      <path d=\"M 43.78125 50.158141 \nL 322.78125 50.158141 \n\" clip-path=\"url(#p1c1c9ae910)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_20\">\n      <g>\n       <use xlink:href=\"#m94e93802e1\" x=\"43.78125\" y=\"50.158141\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 1.5 -->\n      <g transform=\"translate(20.878125 53.957359) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_21\">\n      <path d=\"M 43.78125 11.557521 \nL 322.78125 11.557521 \n\" clip-path=\"url(#p1c1c9ae910)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_22\">\n      <g>\n       <use xlink:href=\"#m94e93802e1\" x=\"43.78125\" y=\"11.557521\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 2.0 -->\n      <g transform=\"translate(20.878125 15.356739) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-32\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_13\">\n     <!-- I(X;Y) (nats) -->\n     <g transform=\"translate(14.798438 120.188125) rotate(-90) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-49\" d=\"M 628 4666 \nL 1259 4666 \nL 1259 0 \nL 628 0 \nL 628 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \nQ 1566 4138 1362 3434 \nQ 1159 2731 1159 2009 \nQ 1159 1288 1364 580 \nQ 1569 -128 1984 -844 \nL 1484 -844 \nQ 1016 -109 783 600 \nQ 550 1309 550 2009 \nQ 550 2706 781 3412 \nQ 1013 4119 1484 4856 \nL 1984 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-58\" d=\"M 403 4666 \nL 1081 4666 \nL 2241 2931 \nL 3406 4666 \nL 4084 4666 \nL 2584 2425 \nL 4184 0 \nL 3506 0 \nL 2194 1984 \nL 872 0 \nL 191 0 \nL 1856 2491 \nL 403 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-3b\" d=\"M 750 3309 \nL 1409 3309 \nL 1409 2516 \nL 750 2516 \nL 750 3309 \nz\nM 750 794 \nL 1409 794 \nL 1409 256 \nL 897 -744 \nL 494 -744 \nL 750 256 \nL 750 794 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-59\" d=\"M -13 4666 \nL 666 4666 \nL 1959 2747 \nL 3244 4666 \nL 3922 4666 \nL 2272 2222 \nL 2272 0 \nL 1638 0 \nL 1638 2222 \nL -13 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \nL 1013 4856 \nQ 1481 4119 1714 3412 \nQ 1947 2706 1947 2009 \nQ 1947 1309 1714 600 \nQ 1481 -109 1013 -844 \nL 513 -844 \nQ 928 -128 1133 580 \nQ 1338 1288 1338 2009 \nQ 1338 2731 1133 3434 \nQ 928 4138 513 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-49\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(29.492188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-58\" transform=\"translate(68.505859 0)\"/>\n      <use xlink:href=\"#DejaVuSans-3b\" transform=\"translate(137.011719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-59\" transform=\"translate(170.703125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(231.787109 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(270.800781 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(302.587891 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(341.601562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(404.980469 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(466.259766 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(505.46875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(557.568359 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_23\">\n    <path d=\"M 56.463068 165.749039 \nL 133.322572 162.070757 \nL 210.182076 148.705821 \nL 287.041581 103.341158 \nL 310.099432 28.317992 \n\" clip-path=\"url(#p1c1c9ae910)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_24\">\n    <path d=\"M 56.463068 165.96 \nL 133.322572 162.319549 \nL 210.182076 148.733041 \nL 287.041581 101.854746 \nL 310.099432 14.76 \n\" clip-path=\"url(#p1c1c9ae910)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 43.78125 173.52 \nL 43.78125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 322.78125 173.52 \nL 322.78125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 43.78125 173.52 \nL 322.78125 173.52 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 43.78125 7.2 \nL 322.78125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 50.78125 44.55625 \nL 164.265625 44.55625 \nQ 166.265625 44.55625 166.265625 42.55625 \nL 166.265625 14.2 \nQ 166.265625 12.2 164.265625 12.2 \nL 50.78125 12.2 \nQ 48.78125 12.2 48.78125 14.2 \nL 48.78125 42.55625 \nQ 48.78125 44.55625 50.78125 44.55625 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_25\">\n     <path d=\"M 52.78125 20.298438 \nL 62.78125 20.298438 \nL 72.78125 20.298438 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_14\">\n     <!-- binned estimate -->\n     <g transform=\"translate(80.78125 23.798438) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-62\" d=\"M 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\nM 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2969 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \nL 2906 4863 \nL 3481 4863 \nL 3481 0 \nL 2906 0 \nL 2906 525 \nQ 2725 213 2448 61 \nQ 2172 -91 1784 -91 \nQ 1150 -91 751 415 \nQ 353 922 353 1747 \nQ 353 2572 751 3078 \nQ 1150 3584 1784 3584 \nQ 2172 3584 2448 3432 \nQ 2725 3281 2906 2969 \nz\nM 947 1747 \nQ 947 1113 1208 752 \nQ 1469 391 1925 391 \nQ 2381 391 2643 752 \nQ 2906 1113 2906 1747 \nQ 2906 2381 2643 2742 \nQ 2381 3103 1925 3103 \nQ 1469 3103 1208 2742 \nQ 947 2381 947 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \nQ 3544 3216 3844 3400 \nQ 4144 3584 4550 3584 \nQ 5097 3584 5394 3201 \nQ 5691 2819 5691 2113 \nL 5691 0 \nL 5113 0 \nL 5113 2094 \nQ 5113 2597 4934 2840 \nQ 4756 3084 4391 3084 \nQ 3944 3084 3684 2787 \nQ 3425 2491 3425 1978 \nL 3425 0 \nL 2847 0 \nL 2847 2094 \nQ 2847 2600 2669 2842 \nQ 2491 3084 2119 3084 \nQ 1678 3084 1418 2786 \nQ 1159 2488 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1356 3278 1631 3431 \nQ 1906 3584 2284 3584 \nQ 2666 3584 2933 3390 \nQ 3200 3197 3328 2828 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-62\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(63.476562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(91.259766 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(154.638672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(218.017578 0)\"/>\n      <use xlink:href=\"#DejaVuSans-64\" transform=\"translate(279.541016 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(343.017578 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(374.804688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(436.328125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(488.427734 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(527.636719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6d\" transform=\"translate(555.419922 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(652.832031 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(714.111328 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(753.320312 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_26\">\n     <path d=\"M 52.78125 34.976563 \nL 62.78125 34.976563 \nL 72.78125 34.976563 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_15\">\n     <!-- closed form -->\n     <g transform=\"translate(80.78125 38.476563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \nL 2375 4384 \nL 1825 4384 \nQ 1516 4384 1395 4259 \nQ 1275 4134 1275 3809 \nL 1275 3500 \nL 2222 3500 \nL 2222 3053 \nL 1275 3053 \nL 1275 0 \nL 697 0 \nL 697 3053 \nL 147 3053 \nL 147 3500 \nL 697 3500 \nL 697 3744 \nQ 697 4328 969 4595 \nQ 1241 4863 1831 4863 \nL 2375 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-63\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(54.980469 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(82.763672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(143.945312 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(196.044922 0)\"/>\n      <use xlink:href=\"#DejaVuSans-64\" transform=\"translate(257.568359 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(321.044922 0)\"/>\n      <use xlink:href=\"#DejaVuSans-66\" transform=\"translate(352.832031 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(388.037109 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(449.21875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6d\" transform=\"translate(488.582031 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p1c1c9ae910\">\n   <rect x=\"43.78125\" y=\"7.2\" width=\"279\" height=\"166.32\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-mutual-information-md-5 fw=all -->\n",
    "\n",
    "With $200{,}000$ samples and a $40 \\times 40$ grid, the histogram tracks the\n",
    "closed form well at moderate correlation, but look closely at\n",
    "$\\rho = 0.99$: the estimate comes in at roughly $1.78$ nats against a truth\n",
    "of $1.96$, missing about $0.18$ nats because the probability mass hides in a\n",
    "ridge thinner than the bins. Even in *two* dimensions with abundant data,\n",
    "the obvious estimator is already biased once the dependence is strong. The\n",
    "second part of this section is about how much worse this gets.\n",
    "\n",
    "### Pointwise Mutual Information\n",
    "\n",
    "Mutual information is an *average*. The quantity being averaged in\n",
    "the equation is interesting in its own right: the\n",
    "*pointwise mutual information*\n",
    "\n",
    "$$\\textrm{pmi}(x, y) = \\log\\frac{p_{X, Y}(x, y)}{p_X(x)\\, p_Y(y)}$$\n",
    "\n",
    "compares how often the specific pair $(x, y)$ occurs to how often it *would*\n",
    "occur if $X$ and $Y$ were independent. Positive pmi means the outcomes attract\n",
    "(they co-occur more than chance predicts); negative pmi means they repel; and\n",
    "$I(X;Y) = E[\\textrm{pmi}(x,y)]$ is the average attraction.\n",
    "\n",
    "Pointwise mutual information is a classic workhorse of natural language\n",
    "processing [@Church.Hanks.1990], where it separates genuine\n",
    "*collocations* from words that merely co-occur because both are common. Here\n",
    "is the effect in miniature. Suppose we count consecutive word pairs (bigrams)\n",
    "in a small corpus and tally how often each of the first words *new*,\n",
    "*machine*, *the* is followed by each of *york*, *learning*, *day*:\n",
    "\n",
    "| count | york | learning | day |\n",
    "| :-- | --: | --: | --: |\n",
    "| **new** | 38 | 3 | 9 |\n",
    "| **machine** | 1 | 46 | 3 |\n",
    "| **the** | 11 | 21 | 168 |\n",
    "\n",
    "The raw counts mislead: \"the day\" is by far the most frequent pair ($168$\n",
    "occurrences against $38$ for \"new york\"), simply because *the* and *day* are\n",
    "both frequent words. Pointwise mutual information corrects for the marginals."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mutual-information-pmi-corpus"
   },
   "source": [
    "counts = onp.array([[38., 3., 9.],     # new york / new learning / new day\n",
    "                    [1., 46., 3.],     # machine ...\n",
    "                    [11., 21., 168.]]) # the ...\n",
    "first, second = ['new', 'machine', 'the'], ['york', 'learning', 'day']\n",
    "\n",
    "p = counts / counts.sum()\n",
    "p_w, p_c = p.sum(axis=1, keepdims=True), p.sum(axis=0, keepdims=True)\n",
    "pmi = onp.log(p / (p_w * p_c))\n",
    "for (i, j) in [(0, 0), (1, 1), (2, 2), (2, 0)]:\n",
    "    print(f'pmi({first[i]}, {second[j]}) = {pmi[i, j]:6.3f} nats '\n",
    "          f'(count {counts[i, j]:.0f})')\n",
    "print(f'I(first; second) = {float((p * pmi).sum()):.4f} nats')"
   ],
   "execution_count": 5,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "pmi(new, york) =  1.517 nats (count 38)\npmi(machine, learning) =  1.372 nats (count 46)\npmi(the, day) =  0.336 nats (count 168)\npmi(the, york) = -1.109 nats (count 11)\nI(first; second) = 0.4146 nats\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-mutual-information-md-6 fw=all -->\n",
    "\n",
    "The ranking flips, exactly as a fluent speaker would rank the collocations:\n",
    "\"new york\" scores $\\textrm{pmi} \\approx 1.52$ nats and \"machine learning\"\n",
    "$\\approx 1.37$, while the frequent-but-unremarkable \"the day\" manages only\n",
    "$\\approx 0.34$; the pair \"the york\" comes out *negative*\n",
    "($\\approx -1.11$ nats), occurring three times less often than independence\n",
    "would predict. The table's overall mutual information, $\\approx 0.41$ nats,\n",
    "summarizes how strongly first words constrain second words on average. This\n",
    "count-divide-and-log pattern (and its smoothed variants) underpins\n",
    "classical word-association mining and survives inside modern embedding\n",
    "methods, where factorizing a pmi matrix recovers word vectors\n",
    "[@Levy.Goldberg.2014].\n",
    "\n",
    "### Mutual Information as Nonlinear Correlation\n",
    "\n",
    "How does mutual information relate to the correlation coefficient of\n",
    "that section? Correlation detects *linear* dependence:\n",
    "$\\rho(X, Y) = 0$ does **not** imply independence, only the absence of a\n",
    "linear trend. Mutual information detects *any* dependence: by claim 3 of the\n",
    "properties proposition, $I(X; Y) = 0$ holds *if and only if* $X \\perp Y$.\n",
    "Two further properties sharpen the contrast.\n",
    "\n",
    "**Proposition (reparameterization invariance).** *If $g$ and $h$ are\n",
    "invertible maps (in the continuous case, smooth with smooth inverses), then\n",
    "$I(g(X); h(Y)) = I(X; Y)$.*\n",
    "\n",
    "**Proof.** For discrete variables, invertibility means $g$ and $h$ merely\n",
    "relabel outcomes: the joint p.m.f. of $(g(X), h(Y))$ takes the same values as\n",
    "that of $(X, Y)$, just at different labels, and the sum in\n",
    "the equation is unchanged. For continuous variables, write\n",
    "mutual information as the KL divergence\n",
    "$D_{\\textrm{KL}}(P_{X,Y} \\| P_X \\otimes P_Y)$ and apply the smooth change of\n",
    "variables $(x, y) \\mapsto (g(x), h(y))$: *both* densities in the ratio pick\n",
    "up the same Jacobian factor $|g'(x)|^{-1} |h'(y)|^{-1}$, which cancels.\n",
    "(Contrast differential entropy, where the uncancelled Jacobian is exactly why\n",
    "it is coordinate-dependent; see that section.)\n",
    "$\\blacksquare$\n",
    "\n",
    "Correlation enjoys no such invariance: it is preserved only by *affine* maps\n",
    "(up to sign). So mutual information is the more principled notion of\n",
    "dependence: it asks whether two variables are informative about each other\n",
    "in *any* coordinates, not whether they happen to line up. A standard\n",
    "counterexample makes both points at once: let $X \\sim \\mathcal{N}(0, 1)$ and\n",
    "$Y = X^2 + 0.3\\,\\epsilon$ with independent noise\n",
    "$\\epsilon \\sim \\mathcal{N}(0,1)$. Then $\\textrm{Cov}(X, Y) = E[X^3] = 0$ by\n",
    "symmetry, so the correlation vanishes, yet $Y$ is nearly a function of $X$,\n",
    "and the dependence is strong."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mutual-information-nonlinear-correlation"
   },
   "source": [
    "rng = onp.random.default_rng(0)\n",
    "n = 200_000\n",
    "x = rng.standard_normal(n)\n",
    "y = x**2 + 0.3 * rng.standard_normal(n)    # zero correlation by symmetry\n",
    "\n",
    "print(f'sample correlation rho = {onp.corrcoef(x, y)[0, 1]:+.4f}')\n",
    "print(f'binned MI estimate     = {binned_mi(x, y):.4f} nats')\n",
    "print(f'after x -> 2x + 3      = {binned_mi(2 * x + 3, y):.4f} nats')"
   ],
   "execution_count": 6,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "sample correlation rho = -0.0028\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "binned MI estimate     = 0.9814 nats\nafter x -> 2x + 3      = 0.9814 nats\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-mutual-information-md-7 fw=all -->\n",
    "\n",
    "The correlation is numerically zero while the mutual information estimate is\n",
    "about one nat, *more* shared information than the $\\rho = 0.9$ Gaussian pair\n",
    "above ($0.830$ nats). The third line checks invariance: an affine\n",
    "reparameterization of $X$ leaves the estimate exactly unchanged (the\n",
    "histogram bins stretch with the data, so even the bin counts are identical).\n",
    "One caution for later: this invariance is also why mutual information is hard\n",
    "to *estimate*: an estimator must implicitly cope with every\n",
    "reparameterization of the data at once, and we will see in\n",
    "that section that this costs dearly.\n",
    "\n",
    "### Conditional Mutual Information and the Chain Rule\n",
    "\n",
    "One more piece of calculus and the toolbox is complete. Just as entropy has a\n",
    "conditional version, so does mutual information: the *conditional mutual\n",
    "information*\n",
    "\n",
    "$$I(X; Y \\mid Z) = H(X \\mid Z) - H(X \\mid Y, Z)$$\n",
    "\n",
    "is the information $X$ and $Y$ share *once $Z$ is already known*.\n",
    "Equivalently, $I(X; Y \\mid Z) = E_{z}\\left[D_{\\textrm{KL}}(P_{X,Y \\mid z}\n",
    "\\,\\|\\, P_{X \\mid z} \\otimes P_{Y \\mid z})\\right]$: expanding each\n",
    "conditional entropy as an expectation and collecting the logarithms into the\n",
    "ratio $p_{X,Y \\mid z}/(p_{X \\mid z}\\, p_{Y \\mid z})$ repeats the algebra of\n",
    "claim 1 under each conditional law. As an average of per-$z$ divergences it\n",
    "inherits nonnegativity from Gibbs, with $I(X; Y \\mid Z) = 0$ if and only if\n",
    "$X$ and $Y$ are conditionally independent given $Z$. The chain rule then\n",
    "splits joint information into sequential contributions.\n",
    "\n",
    "**Proposition (chain rule for mutual information).** *For any $X, Y, Z$,*\n",
    "\n",
    "$$\n",
    "I(X; Y, Z) = I(X; Z) + I(X; Y \\mid Z).\n",
    "$$\n",
    "\n",
    "**Proof.** Telescope with the entropy identities:\n",
    "\n",
    "$$\n",
    "I(X; Y, Z) = H(X) - H(X \\mid Y, Z)\n",
    "= \\underbrace{H(X) - H(X \\mid Z)}_{I(X;\\, Z)}\n",
    "+ \\underbrace{H(X \\mid Z) - H(X \\mid Y, Z)}_{I(X;\\, Y \\mid Z)}.\n",
    "\\quad\\blacksquare\n",
    "$$\n",
    "\n",
    "**Corollary (independent side information is free).** *If $W$ is independent\n",
    "of the pair $(X, Y)$, then $I(X; (Y, W)) = I(X; Y)$.*\n",
    "\n",
    "**Proof.** By the chain rule, $I(X; Y, W) = I(X; W) + I(X; Y \\mid W)$. The\n",
    "first term is zero since $W \\perp X$. For the second, independence of $W$\n",
    "from the pair means conditioning on $W$ changes neither $H(X)$ nor\n",
    "$H(X \\mid Y)$, so $I(X; Y \\mid W) = I(X; Y)$. $\\blacksquare$\n",
    "\n",
    "The corollary is exactly the accounting we will need for InfoNCE in\n",
    "that section, where a batch of independent \"negative\" samples\n",
    "is appended to $Y$ and must add no information about $X$. The chain rule's\n",
    "main consequence deserves its own subsection.\n",
    "\n",
    "### The Data-Processing Inequality\n",
    "\n",
    "Random variables $X, Y, Z$ form a *Markov chain*, written\n",
    "$X \\to Y \\to Z$, when $Z$ depends on $(X, Y)$ only through $Y$:\n",
    "$p(z \\mid x, y) = p(z \\mid y)$. This is the structure of *processing*: $Y$ is\n",
    "computed from $X$ (a measurement, a feature map, a hidden layer), then $Z$ is\n",
    "computed from $Y$ alone. Equivalently, $X$ and $Z$ are conditionally\n",
    "independent given $Y$, i.e., $I(X; Z \\mid Y) = 0$. The following one-line\n",
    "theorem [@Cover.Thomas.1999] is the conservation law that governs it.\n",
    "\n",
    "**Proposition (data-processing inequality).** *If $X \\to Y \\to Z$ is a\n",
    "Markov chain, then*\n",
    "\n",
    "$$\n",
    "I(X; Z) \\leq I(X; Y),\n",
    "$$\n",
    "\n",
    "*with equality if and only if $I(X; Y \\mid Z) = 0$ (i.e., $X \\to Z \\to Y$ is\n",
    "also a Markov chain).*\n",
    "\n",
    "**Proof.** Expand $I(X; Y, Z)$ by the chain rule in both orders:\n",
    "\n",
    "$$\n",
    "I(X; Z) + I(X; Y \\mid Z) = I(X; Y, Z) = I(X; Y) + I(X; Z \\mid Y).\n",
    "$$\n",
    "\n",
    "The Markov property kills the last term, $I(X; Z \\mid Y) = 0$, leaving\n",
    "$I(X; Z) = I(X; Y) - I(X; Y \\mid Z) \\leq I(X; Y)$, since conditional mutual\n",
    "information is non-negative. Equality holds exactly when\n",
    "$I(X; Y \\mid Z) = 0$. $\\blacksquare$\n",
    "\n",
    "No processing of $Y$, deterministic or random, clever or dumb, can\n",
    "*increase* the information it carries about $X$. Three readings:\n",
    "\n",
    "* **Representations only lose.** A network computes its representation\n",
    "  layer-by-layer, $X \\to Z_1 \\to Z_2 \\to \\cdots$, so\n",
    "  $I(X; Z_1) \\geq I(X; Z_2) \\geq \\cdots$ and likewise\n",
    "  $I(Y; Z_1) \\geq I(Y; Z_2) \\geq \\cdots$ for a label $Y$ at the *input* end\n",
    "  of the chain (since $Y \\to X \\to Z_\\ell$ is Markov). Whatever label\n",
    "  information the input lacks, no architecture can conjure; whatever a layer\n",
    "  discards, no later layer recovers. This is the theorem that makes the\n",
    "  information bottleneck of that section a\n",
    "  sensible objective rather than wishful thinking.\n",
    "* **Equality is achievable.** If $Z = g(Y)$ for an *invertible* $g$, then the\n",
    "  chain runs both ways and $I(X; Z) = I(X; Y)$, consistent with the\n",
    "  reparameterization invariance proved earlier. A many-to-one or noisy map may lose information, but it need not: a\n",
    "  sufficient statistic can be noninvertible or noisy while preserving all the\n",
    "  information about $X$. Equality in data processing is characterized by such\n",
    "  sufficiency, not by invertibility alone.\n",
    "* **A sufficient statistic is a lossless compression.** When equality holds\n",
    "  with $Z$ much \"smaller\" than $Y$, the map has discarded only what was\n",
    "  irrelevant to $X$: this is the information-theoretic reading of a\n",
    "  sufficient statistic (that section).\n",
    "\n",
    "One caution about scope. The data-processing inequality is a statement about\n",
    "Markov chains, not about conditioning in general: conditioning can *create*\n",
    "dependence, and $I(X; Y \\mid Z) > I(X; Y)$ is perfectly possible: in the\n",
    "extreme case of two independent fair bits with $Z$ their XOR, conditioning\n",
    "turns zero dependence into a full $\\ln 2$ nats (Exercise 6 works this out).\n",
    "\"Processing destroys information\" is a theorem about the chain\n",
    "$X \\to Y \\to Z$ alone; outside that structure, conditioning can raise or\n",
    "lower dependence, as the XOR example shows.\n",
    "\n",
    "Let's verify the bookkeeping numerically on the simplest nontrivial chain:\n",
    "a fair bit $X$, flipped with probability $0.1$ to give $Y$, flipped again\n",
    "with probability $0.2$ to give $Z$."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mutual-information-dpi-check"
   },
   "source": [
    "p_x = onp.array([0.5, 0.5])\n",
    "K1 = onp.array([[0.9, 0.1], [0.1, 0.9]])   # p(y|x): flip w.p. 0.1\n",
    "K2 = onp.array([[0.8, 0.2], [0.2, 0.8]])   # p(z|y): flip w.p. 0.2\n",
    "p_xyz = p_x[:, None, None] * K1[:, :, None] * K2[None, :, :]\n",
    "\n",
    "def cond_mi(p_abc):\n",
    "    \"\"\"I(A;B|C) for a joint array indexed [a, b, c], in nats.\"\"\"\n",
    "    p_ac, p_bc, p_c = p_abc.sum(1), p_abc.sum(0), p_abc.sum((0, 1))\n",
    "    ratio = p_abc * p_c[None, None, :] / (p_ac[:, None, :] * p_bc[None, :, :])\n",
    "    return float((p_abc * onp.log(ratio)).sum())\n",
    "\n",
    "I_xy = mutual_information(p_xyz.sum(axis=2))           # I(X;Y)\n",
    "I_xz = mutual_information(p_xyz.sum(axis=1))           # I(X;Z)\n",
    "I_xz_y = cond_mi(onp.transpose(p_xyz, (0, 2, 1)))      # I(X;Z|Y)\n",
    "I_xy_z = cond_mi(p_xyz)                                # I(X;Y|Z)\n",
    "print(f'I(X;Y) = {I_xy:.4f},  I(X;Z) = {I_xz:.4f} nats  (DPI: smaller)')\n",
    "print(f'I(X;Z|Y) = {I_xz_y:.4f}  (Markov: zero)')\n",
    "print(f'chain rule, both orders: {I_xz + I_xy_z:.4f} = {I_xy + I_xz_y:.4f}')"
   ],
   "execution_count": 7,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "I(X;Y) = 0.3681,  I(X;Z) = 0.1201 nats  (DPI: smaller)\nI(X;Z|Y) = 0.0000  (Markov: zero)\nchain rule, both orders: 0.3681 = 0.3681\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-mutual-information-md-8 fw=all -->\n",
    "\n",
    "Every claim checks out: $I(X; Z \\mid Y) = 0$ certifies the Markov structure,\n",
    "both expansions of $I(X; Y, Z)$ agree at $0.3681$ nats, and processing\n",
    "through the second noisy channel costs two thirds of the information:\n",
    "$I(X;Y) \\approx 0.368$ nats falls to $I(X;Z) \\approx 0.120$ nats. (For\n",
    "calibration, a noiseless bit would carry $\\ln 2 \\approx 0.693$ nats.)\n",
    "\n",
    "## Why Measuring Mutual Information Is Hard\n",
    "\n",
    "We now have a quantity that is invariant, zero exactly at independence,\n",
    "closed-form for Gaussians, and governed by a conservation law. The natural\n",
    "next step is to *measure* it for real data: images and their augmentations,\n",
    "sentences and their continuations. That measurement problem is fundamentally\n",
    "hard, and the hardness shapes everything that practitioners do with mutual\n",
    "information.\n",
    "\n",
    "### The Curse of Estimation\n",
    "\n",
    "The plug-in recipe that worked above (bin the data, apply the discrete\n",
    "formula) dies in high dimensions. With $b$ bins per axis, a $d$-dimensional\n",
    "histogram has $b^d$ cells; at $d = 10$ and a modest $b = 10$ that is ten\n",
    "billion cells, almost all empty at any realistic sample size, and the\n",
    "plug-in estimate is dominated by sampling noise in the occupied few. Already\n",
    "at $d = 2$ and $\\rho = 0.99$ we watched the histogram miss $0.18$ nats.\n",
    "\n",
    "Smarter nonparametric estimators improve the constants, not the story. The\n",
    "classic of the genre, due to @Kraskov.Stogbauer.Grassberger.2004\n",
    "(KSG), replaces bins with $k$-nearest-neighbor distances: around each sample\n",
    "find the Chebyshev (max-coordinate) distance $\\epsilon_i$ to its $k$-th\n",
    "nearest neighbor in\n",
    "the *joint* space, count the marginal neighbors $n_x^{(i)}$ and $n_y^{(i)}$\n",
    "lying within $\\epsilon_i$, and average\n",
    "\n",
    "$$\\hat{I}_{\\textrm{KSG}} = \\psi(k) + \\psi(n) - \\frac{1}{n} \\sum_{i=1}^{n} \\left(\\psi\\left(n_x^{(i)} + 1\\right) + \\psi\\left(n_y^{(i)} + 1\\right)\\right),$$\n",
    "\n",
    "where $\\psi = (\\log \\Gamma)'$ is the digamma function. The local radii\n",
    "$\\epsilon_i$ adapt the resolution to wherever the data actually sit: no bins\n",
    "to choose, no bandwidth to tune."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mdl-mutual-information-the-curse-of-estimation"
   },
   "source": [
    "def ksg_mi(x, y, k=5):\n",
    "    \"\"\"Kraskov et al. (2004) k-NN estimate of I(X;Y), in nats.\"\"\"\n",
    "    n = len(x)\n",
    "    dx = onp.abs(x[:, None, :] - x[None, :, :]).max(axis=-1)\n",
    "    dy = onp.abs(y[:, None, :] - y[None, :, :]).max(axis=-1)\n",
    "    d = onp.maximum(dx, dy)               # Chebyshev distance in joint space\n",
    "    onp.fill_diagonal(d, onp.inf)\n",
    "    eps = onp.sort(d, axis=1)[:, k - 1]   # distance to the k-th neighbor\n",
    "    n_x = (dx < eps[:, None]).sum(axis=1) - 1   # marginal counts, minus self\n",
    "    n_y = (dy < eps[:, None]).sum(axis=1) - 1\n",
    "    return float(digamma(k) + digamma(n)\n",
    "                 - (digamma(n_x + 1) + digamma(n_y + 1)).mean())\n",
    "\n",
    "rng = onp.random.default_rng(1)\n",
    "n = 2000\n",
    "for dim in (1, 2):\n",
    "    for rho in (0.5, 0.7, 0.9):\n",
    "        pairs = [gauss_pair(rng, rho, n) for _ in range(dim)]\n",
    "        x = onp.stack([u for u, _ in pairs], axis=1)\n",
    "        y = onp.stack([v for _, v in pairs], axis=1)\n",
    "        print(f'd = {dim}, rho = {rho}: KSG = {ksg_mi(x, y):.3f}, '\n",
    "              f'closed form = {-dim / 2 * onp.log(1 - rho**2):.3f} nats')"
   ],
   "execution_count": 8,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "d = 1, rho = 0.5: KSG = 0.121, closed form = 0.144 nats\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "d = 1, rho = 0.7: KSG = 0.335, closed form = 0.337 nats\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "d = 1, rho = 0.9: KSG = 0.863, closed form = 0.830 nats\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "d = 2, rho = 0.5: KSG = 0.267, closed form = 0.288 nats\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "d = 2, rho = 0.7: KSG = 0.664, closed form = 0.673 nats\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "d = 2, rho = 0.9: KSG = 1.627, closed form = 1.661 nats\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-mutual-information-md-9 fw=all -->\n",
    "\n",
    "With $2{,}000$ samples, a hundredth of what the histogram consumed, KSG\n",
    "lands within a few hundredths of a nat of the closed form across\n",
    "correlations, in one and two dimensions alike, with nothing to tune. That is\n",
    "what \"improving the constants\" buys. What it does not buy is an escape:\n",
    "KSG's guarantees rest on smoothness assumptions that become more restrictive\n",
    "as $d$ rises, nearest-neighbor distances concentrate in high dimension, and\n",
    "by the time $X$ and $Y$ are images the estimator is as lost as the\n",
    "histogram.\n",
    "\n",
    "The reparameterization invariance we celebrated is part of the problem.\n",
    "Mutual information cares only about the copula of the joint distribution,\n",
    "the joint law of the marginal ranks: any invertible warping of either\n",
    "marginal leaves it unchanged, so an estimator cannot rely on the data looking Gaussian, or\n",
    "bounded, or smooth in any particular coordinates. Distribution-free\n",
    "estimation of an invariant functional is a statistician's nightmare, and the\n",
    "nightmare has a theorem.\n",
    "\n",
    "### A Ceiling at log N\n",
    "\n",
    "Suppose we are conservative and ask only for a *lower bound*: an estimator\n",
    "$\\hat{I}_N$, computed from $N$ samples of the joint, such that\n",
    "$\\hat{I}_N \\leq I(X; Y)$ with high probability *for every distribution*.\n",
    "(Lower bounds are the useful direction for representation learning, where we\n",
    "want to certify that two views share a lot of information.)\n",
    "@McAllester.Stratos.2020 proved that any such estimator is\n",
    "essentially capped:\n",
    "\n",
    "> **Theorem (McAllester--Stratos, informal).** Any distribution-free,\n",
    "> high-confidence lower bound on $I(X;Y)$ computed from $N$ samples cannot\n",
    "> exceed $O(\\log N)$.\n",
    "\n",
    "The intuition is a counting argument about rare events. Mutual information\n",
    "lives in the likelihood ratio $p_{X,Y}/(p_X\\, p_Y)$, and when $I$ is large,\n",
    "the average of this ratio under the *product* of marginals (which equals\n",
    "$1$) is carried by a vanishingly rare set of \"matched\" pairs of probability\n",
    "roughly $e^{-I}$. A sample of size $N$ simply never sees those events unless\n",
    "$N \\gtrsim e^{I}$; and a *certifiable* bound cannot presume what it has never\n",
    "observed. Flip the inequality and you get the ceiling: $N$ samples certify\n",
    "at most about $\\log N$ nats.\n",
    "\n",
    "The practical consequence is immediate, because every estimator we are about\n",
    "to meet is computed on minibatches. A batch of $N = 256$ pairs can certify at\n",
    "most $\\ln 256 \\approx 5.5$ nats (exactly $8$ bits, since $256 = 2^8$), no\n",
    "matter how expressive the critic network is. Two views of the same image share far more information\n",
    "than that. So when a contrastive method reports its \"estimated MI,\" the\n",
    "number is best read as $\\min(I, \\log N)$ plus noise.\n",
    "\n",
    "### Watching the Ceiling: a Perfect-Critic Simulation\n",
    "\n",
    "The theorem bounds what *any* estimator can certify; let's watch the ceiling\n",
    "materialize for the estimator family we will actually use. Here is the\n",
    "estimation game in its purest form. Nature draws one *positive* pair\n",
    "$(x, y_1) \\sim P_{X,Y}$ and $N - 1$ *negatives* $y_2, \\ldots, y_N \\sim P_Y$,\n",
    "and asks us to score how well we can tell the positive apart. The\n",
    "Bayes-optimal player scores candidates by the exact likelihood ratio, the\n",
    "exponentiated pointwise mutual information the equation, and the\n",
    "average log-probability it assigns to the truth (plus $\\log N$) is an\n",
    "estimate of $I(X;Y)$. We will *derive* this estimator properly in\n",
    "that section; for now the point is that we can simulate it with\n",
    "**zero training error**: on a Gaussian pair we know the density ratio in\n",
    "closed form, so the simulation shows the best this kind of estimator could\n",
    "ever do."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mutual-information-log-n-ceiling"
   },
   "source": [
    "def pmi_gauss(x, y, rho):\n",
    "    \"\"\"Exact pmi(x, y) of the standard bivariate Gaussian, in nats.\"\"\"\n",
    "    c = 1 - rho**2\n",
    "    return (-0.5 * onp.log(c) - (x**2 - 2 * rho * x * y + y**2) / (2 * c)\n",
    "            + (x**2 + y**2) / 2)\n",
    "\n",
    "def infonce_oracle(rng, rho, N, batches):\n",
    "    \"\"\"InfoNCE estimate with the *true* density-ratio critic.\"\"\"\n",
    "    vals = []\n",
    "    for _ in range(batches):\n",
    "        x, y = gauss_pair(rng, rho, N)\n",
    "        f = pmi_gauss(x[:, None], y[None, :], rho)     # all N x N scores\n",
    "        f -= f.max(axis=1, keepdims=True)              # stable log-softmax\n",
    "        ll = onp.diag(f) - onp.log(onp.exp(f).sum(axis=1))\n",
    "        vals.append(ll.mean() + onp.log(N))\n",
    "    return onp.array(vals)\n",
    "\n",
    "rng = onp.random.default_rng(7)\n",
    "true_mi = onp.array([0.25, 0.5, 1, 1.5, 2, 2.5, 3, 4, 5, 6, 7])\n",
    "rhos = onp.sqrt(1 - onp.exp(-2 * true_mi))             # invert the closed form\n",
    "ests = [onp.array([infonce_oracle(rng, r, N, 200 if N <= 128 else 50).mean()\n",
    "                   for r in rhos]) for N in (16, 128, 1024)]\n",
    "d2l.plot(true_mi, ests + [true_mi], 'true I(X;Y) (nats)', 'estimate (nats)',\n",
    "         legend=['N=16', 'N=128', 'N=1024', 'truth'], figsize=(5, 3))"
   ],
   "execution_count": 9,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"320.440625pt\" height=\"211.07625pt\" viewBox=\"0 0 320.440625 211.07625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-19T14:01:16.991202</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 211.07625 \nL 320.440625 211.07625 \nL 320.440625 0 \nL 0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 34.240625 173.52 \nL 313.240625 173.52 \nL 313.240625 7.2 \nL 34.240625 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path d=\"M 37.528504 173.52 \nL 37.528504 7.2 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path id=\"m8bef7165d4\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m8bef7165d4\" x=\"37.528504\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <g transform=\"translate(34.347254 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path d=\"M 75.104261 173.52 \nL 75.104261 7.2 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#m8bef7165d4\" x=\"75.104261\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 1 -->\n      <g transform=\"translate(71.923011 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path d=\"M 112.680019 173.52 \nL 112.680019 7.2 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#m8bef7165d4\" x=\"112.680019\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 2 -->\n      <g transform=\"translate(109.498769 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-32\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_7\">\n      <path d=\"M 150.255777 173.52 \nL 150.255777 7.2 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#m8bef7165d4\" x=\"150.255777\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 3 -->\n      <g transform=\"translate(147.074527 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \nQ 3050 2419 3304 2112 \nQ 3559 1806 3559 1356 \nQ 3559 666 3084 287 \nQ 2609 -91 1734 -91 \nQ 1441 -91 1130 -33 \nQ 819 25 488 141 \nL 488 750 \nQ 750 597 1062 519 \nQ 1375 441 1716 441 \nQ 2309 441 2620 675 \nQ 2931 909 2931 1356 \nQ 2931 1769 2642 2001 \nQ 2353 2234 1838 2234 \nL 1294 2234 \nL 1294 2753 \nL 1863 2753 \nQ 2328 2753 2575 2939 \nQ 2822 3125 2822 3475 \nQ 2822 3834 2567 4026 \nQ 2313 4219 1838 4219 \nQ 1578 4219 1281 4162 \nQ 984 4106 628 3988 \nL 628 4550 \nQ 988 4650 1302 4700 \nQ 1616 4750 1894 4750 \nQ 2613 4750 3031 4423 \nQ 3450 4097 3450 3541 \nQ 3450 3153 3228 2886 \nQ 3006 2619 2597 2516 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-33\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_9\">\n      <path d=\"M 187.831534 173.52 \nL 187.831534 7.2 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#m8bef7165d4\" x=\"187.831534\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 4 -->\n      <g transform=\"translate(184.650284 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-34\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_11\">\n      <path d=\"M 225.407292 173.52 \nL 225.407292 7.2 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_12\">\n      <g>\n       <use xlink:href=\"#m8bef7165d4\" x=\"225.407292\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 5 -->\n      <g transform=\"translate(222.226042 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \nL 3169 4666 \nL 3169 4134 \nL 1269 4134 \nL 1269 2991 \nQ 1406 3038 1543 3061 \nQ 1681 3084 1819 3084 \nQ 2600 3084 3056 2656 \nQ 3513 2228 3513 1497 \nQ 3513 744 3044 326 \nQ 2575 -91 1722 -91 \nQ 1428 -91 1123 -41 \nQ 819 9 494 109 \nL 494 744 \nQ 775 591 1075 516 \nQ 1375 441 1709 441 \nQ 2250 441 2565 725 \nQ 2881 1009 2881 1497 \nQ 2881 1984 2565 2268 \nQ 2250 2553 1709 2553 \nQ 1456 2553 1204 2497 \nQ 953 2441 691 2322 \nL 691 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-35\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_7\">\n     <g id=\"line2d_13\">\n      <path d=\"M 262.983049 173.52 \nL 262.983049 7.2 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_14\">\n      <g>\n       <use xlink:href=\"#m8bef7165d4\" x=\"262.983049\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_7\">\n      <!-- 6 -->\n      <g transform=\"translate(259.801799 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-36\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_8\">\n     <g id=\"line2d_15\">\n      <path d=\"M 300.558807 173.52 \nL 300.558807 7.2 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_16\">\n      <g>\n       <use xlink:href=\"#m8bef7165d4\" x=\"300.558807\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 7 -->\n      <g transform=\"translate(297.377557 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-37\" d=\"M 525 4666 \nL 3525 4666 \nL 3525 4397 \nL 1831 0 \nL 1172 0 \nL 2766 4134 \nL 525 4134 \nL 525 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-37\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_9\">\n     <!-- true I(X;Y) (nats) -->\n     <g transform=\"translate(132.0625 201.796563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-75\" d=\"M 544 1381 \nL 544 3500 \nL 1119 3500 \nL 1119 1403 \nQ 1119 906 1312 657 \nQ 1506 409 1894 409 \nQ 2359 409 2629 706 \nQ 2900 1003 2900 1516 \nL 2900 3500 \nL 3475 3500 \nL 3475 0 \nL 2900 0 \nL 2900 538 \nQ 2691 219 2414 64 \nQ 2138 -91 1772 -91 \nQ 1169 -91 856 284 \nQ 544 659 544 1381 \nz\nM 1991 3584 \nL 1991 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-49\" d=\"M 628 4666 \nL 1259 4666 \nL 1259 0 \nL 628 0 \nL 628 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \nQ 1566 4138 1362 3434 \nQ 1159 2731 1159 2009 \nQ 1159 1288 1364 580 \nQ 1569 -128 1984 -844 \nL 1484 -844 \nQ 1016 -109 783 600 \nQ 550 1309 550 2009 \nQ 550 2706 781 3412 \nQ 1013 4119 1484 4856 \nL 1984 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-58\" d=\"M 403 4666 \nL 1081 4666 \nL 2241 2931 \nL 3406 4666 \nL 4084 4666 \nL 2584 2425 \nL 4184 0 \nL 3506 0 \nL 2194 1984 \nL 872 0 \nL 191 0 \nL 1856 2491 \nL 403 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-3b\" d=\"M 750 3309 \nL 1409 3309 \nL 1409 2516 \nL 750 2516 \nL 750 3309 \nz\nM 750 794 \nL 1409 794 \nL 1409 256 \nL 897 -744 \nL 494 -744 \nL 750 256 \nL 750 794 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-59\" d=\"M -13 4666 \nL 666 4666 \nL 1959 2747 \nL 3244 4666 \nL 3922 4666 \nL 2272 2222 \nL 2272 0 \nL 1638 0 \nL 1638 2222 \nL -13 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \nL 1013 4856 \nQ 1481 4119 1714 3412 \nQ 1947 2706 1947 2009 \nQ 1947 1309 1714 600 \nQ 1481 -109 1013 -844 \nL 513 -844 \nQ 928 -128 1133 580 \nQ 1338 1288 1338 2009 \nQ 1338 2731 1133 3434 \nQ 928 4138 513 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(39.208984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-75\" transform=\"translate(80.322266 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(143.701172 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(205.224609 0)\"/>\n      <use xlink:href=\"#DejaVuSans-49\" transform=\"translate(237.011719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(266.503906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-58\" transform=\"translate(305.517578 0)\"/>\n      <use xlink:href=\"#DejaVuSans-3b\" transform=\"translate(374.023438 0)\"/>\n      <use xlink:href=\"#DejaVuSans-59\" transform=\"translate(407.714844 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(468.798828 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(507.8125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(539.599609 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(578.613281 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(641.992188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(703.271484 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(742.480469 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(794.580078 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_17\">\n      <path d=\"M 34.240625 171.178639 \nL 313.240625 171.178639 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_18\">\n      <defs>\n       <path id=\"mcd31b24757\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#mcd31b24757\" x=\"34.240625\" y=\"171.178639\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 0 -->\n      <g transform=\"translate(20.878125 174.977858) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_19\">\n      <path d=\"M 34.240625 148.833119 \nL 313.240625 148.833119 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_20\">\n      <g>\n       <use xlink:href=\"#mcd31b24757\" x=\"34.240625\" y=\"148.833119\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 1 -->\n      <g transform=\"translate(20.878125 152.632338) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_21\">\n      <path d=\"M 34.240625 126.487599 \nL 313.240625 126.487599 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_22\">\n      <g>\n       <use xlink:href=\"#mcd31b24757\" x=\"34.240625\" y=\"126.487599\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 2 -->\n      <g transform=\"translate(20.878125 130.286818) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-32\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_23\">\n      <path d=\"M 34.240625 104.142079 \nL 313.240625 104.142079 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_24\">\n      <g>\n       <use xlink:href=\"#mcd31b24757\" x=\"34.240625\" y=\"104.142079\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_13\">\n      <!-- 3 -->\n      <g transform=\"translate(20.878125 107.941298) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-33\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_25\">\n      <path d=\"M 34.240625 81.79656 \nL 313.240625 81.79656 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_26\">\n      <g>\n       <use xlink:href=\"#mcd31b24757\" x=\"34.240625\" y=\"81.79656\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_14\">\n      <!-- 4 -->\n      <g transform=\"translate(20.878125 85.595778) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-34\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_27\">\n      <path d=\"M 34.240625 59.45104 \nL 313.240625 59.45104 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_28\">\n      <g>\n       <use xlink:href=\"#mcd31b24757\" x=\"34.240625\" y=\"59.45104\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_15\">\n      <!-- 5 -->\n      <g transform=\"translate(20.878125 63.250258) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-35\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_29\">\n      <path d=\"M 34.240625 37.10552 \nL 313.240625 37.10552 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_30\">\n      <g>\n       <use xlink:href=\"#mcd31b24757\" x=\"34.240625\" y=\"37.10552\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_16\">\n      <!-- 6 -->\n      <g transform=\"translate(20.878125 40.904739) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-36\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_8\">\n     <g id=\"line2d_31\">\n      <path d=\"M 34.240625 14.76 \nL 313.240625 14.76 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_32\">\n      <g>\n       <use xlink:href=\"#mcd31b24757\" x=\"34.240625\" y=\"14.76\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_17\">\n      <!-- 7 -->\n      <g transform=\"translate(20.878125 18.559219) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-37\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_18\">\n     <!-- estimate (nats) -->\n     <g transform=\"translate(14.798438 128.649844) rotate(-90) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \nQ 3544 3216 3844 3400 \nQ 4144 3584 4550 3584 \nQ 5097 3584 5394 3201 \nQ 5691 2819 5691 2113 \nL 5691 0 \nL 5113 0 \nL 5113 2094 \nQ 5113 2597 4934 2840 \nQ 4756 3084 4391 3084 \nQ 3944 3084 3684 2787 \nQ 3425 2491 3425 1978 \nL 3425 0 \nL 2847 0 \nL 2847 2094 \nQ 2847 2600 2669 2842 \nQ 2491 3084 2119 3084 \nQ 1678 3084 1418 2786 \nQ 1159 2488 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1356 3278 1631 3431 \nQ 1906 3584 2284 3584 \nQ 2666 3584 2933 3390 \nQ 3200 3197 3328 2828 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-65\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(61.523438 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(113.623047 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(152.832031 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6d\" transform=\"translate(180.615234 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(278.027344 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(339.306641 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(378.515625 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(440.039062 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(471.826172 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(510.839844 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(574.21875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(635.498047 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(674.707031 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(726.806641 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_33\">\n    <path d=\"M 46.922443 165.96 \nL 56.316383 161.098744 \nL 75.104261 150.772194 \nL 93.89214 142.039234 \nL 112.680019 133.48613 \nL 131.467898 126.95108 \nL 150.255777 120.990038 \nL 187.831534 114.128461 \nL 225.407292 111.524895 \nL 262.983049 110.004809 \nL 300.558807 109.597255 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_34\">\n    <path d=\"M 46.922443 165.706124 \nL 56.316383 160.031779 \nL 75.104261 148.994626 \nL 93.89214 138.265931 \nL 112.680019 127.501092 \nL 131.467898 117.2625 \nL 150.255777 107.159651 \nL 187.831534 89.084433 \nL 225.407292 75.866292 \nL 262.983049 68.302165 \nL 300.558807 64.941105 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"line2d_35\">\n    <path d=\"M 46.922443 165.648218 \nL 56.316383 159.81341 \nL 75.104261 148.892187 \nL 93.89214 137.535117 \nL 112.680019 126.728935 \nL 131.467898 115.617077 \nL 150.255777 104.60666 \nL 187.831534 83.033275 \nL 225.407292 62.303753 \nL 262.983049 43.831898 \nL 300.558807 30.120779 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"line2d_36\">\n    <path d=\"M 46.922443 165.592259 \nL 56.316383 160.005879 \nL 75.104261 148.833119 \nL 93.89214 137.660359 \nL 112.680019 126.487599 \nL 131.467898 115.314839 \nL 150.255777 104.142079 \nL 187.831534 81.79656 \nL 225.407292 59.45104 \nL 262.983049 37.10552 \nL 300.558807 14.76 \n\" clip-path=\"url(#p4b1a96ee8b)\" style=\"fill: none; stroke-dasharray: 1.5,2.475; stroke-dashoffset: 0; stroke: #ff0000; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 34.240625 173.52 \nL 34.240625 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 313.240625 173.52 \nL 313.240625 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 34.240625 173.52 \nL 313.240625 173.52 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 34.240625 7.2 \nL 313.240625 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 41.240625 73.9125 \nL 114.551563 73.9125 \nQ 116.551563 73.9125 116.551563 71.9125 \nL 116.551563 14.2 \nQ 116.551563 12.2 114.551563 12.2 \nL 41.240625 12.2 \nQ 39.240625 12.2 39.240625 14.2 \nL 39.240625 71.9125 \nQ 39.240625 73.9125 41.240625 73.9125 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_37\">\n     <path d=\"M 43.240625 20.298438 \nL 53.240625 20.298438 \nL 63.240625 20.298438 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_19\">\n     <!-- N=16 -->\n     <g transform=\"translate(71.240625 23.798438) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-4e\" d=\"M 628 4666 \nL 1478 4666 \nL 3547 763 \nL 3547 4666 \nL 4159 4666 \nL 4159 0 \nL 3309 0 \nL 1241 3903 \nL 1241 0 \nL 628 0 \nL 628 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-3d\" d=\"M 678 2906 \nL 4684 2906 \nL 4684 2381 \nL 678 2381 \nL 678 2906 \nz\nM 678 1631 \nL 4684 1631 \nL 4684 1100 \nL 678 1100 \nL 678 1631 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-4e\"/>\n      <use xlink:href=\"#DejaVuSans-3d\" transform=\"translate(74.804688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(158.59375 0)\"/>\n      <use xlink:href=\"#DejaVuSans-36\" transform=\"translate(222.216797 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_38\">\n     <path d=\"M 43.240625 34.976563 \nL 53.240625 34.976563 \nL 63.240625 34.976563 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_20\">\n     <!-- N=128 -->\n     <g transform=\"translate(71.240625 38.476563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-4e\"/>\n      <use xlink:href=\"#DejaVuSans-3d\" transform=\"translate(74.804688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(158.59375 0)\"/>\n      <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(222.216797 0)\"/>\n      <use xlink:href=\"#DejaVuSans-38\" transform=\"translate(285.839844 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_39\">\n     <path d=\"M 43.240625 49.654688 \nL 53.240625 49.654688 \nL 63.240625 49.654688 \n\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_21\">\n     <!-- N=1024 -->\n     <g transform=\"translate(71.240625 53.154688) scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-4e\"/>\n      <use xlink:href=\"#DejaVuSans-3d\" transform=\"translate(74.804688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(158.59375 0)\"/>\n      <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(222.216797 0)\"/>\n      <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(285.839844 0)\"/>\n      <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(349.462891 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_40\">\n     <path d=\"M 43.240625 64.332813 \nL 53.240625 64.332813 \nL 63.240625 64.332813 \n\" style=\"fill: none; stroke-dasharray: 1.5,2.475; stroke-dashoffset: 0; stroke: #ff0000; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_22\">\n     <!-- truth -->\n     <g transform=\"translate(71.240625 67.832813) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(39.208984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-75\" transform=\"translate(80.322266 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(143.701172 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(182.910156 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p4b1a96ee8b\">\n   <rect x=\"34.240625\" y=\"7.2\" width=\"279\" height=\"166.32\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-mutual-information-md-10 fw=all -->\n",
    "\n",
    "Each curve hugs the diagonal while the true mutual information is small, then\n",
    "bends flat as it approaches its ceiling $\\ln N$ ($\\approx 2.77$, $4.85$, and\n",
    "$6.93$ nats for the three batch sizes), *even though the critic is exact*.\n",
    "No amount of architecture search or training fixes this; only a bigger $N$\n",
    "raises the ceiling, and only logarithmically. This is the McAllester--Stratos\n",
    "phenomenon in one picture.\n",
    "\n",
    "Two readings of this hardness, before we proceed. Pessimistic: reported MI\n",
    "numbers in the literature are not measurements; treat them as such.\n",
    "Optimistic: representation learning never needed the *number*. Maximizing a\n",
    "lower bound can shape features well even when the bound is slack: a hiker\n",
    "can climb by always stepping uphill without ever knowing the summit's\n",
    "altitude.\n",
    "\n",
    "## Variational Bounds and InfoNCE\n",
    "\n",
    "Mutual information involves the unknown densities of\n",
    "the equation, so we cannot compute it, but we can *bound it\n",
    "from below* by an expression involving only samples and an auxiliary model\n",
    "(a \"decoder\" or a \"critic\"), then make the bound as tight as we can by\n",
    "optimizing the auxiliary model. This turns estimation into optimization,\n",
    "exactly the trick deep learning is good at, and tightening a lower bound on\n",
    "shared information is precisely the training signal that contrastive methods\n",
    "exploit. We derive the three classical bounds and then InfoNCE, the bound\n",
    "modern practice rests on; @Poole.Oord.Alemi.ea.2019 give the unified\n",
    "treatment.\n",
    "\n",
    "### The Barber--Agakov Bound\n",
    "\n",
    "The first bound replaces the unknown posterior with a model of it.\n",
    "\n",
    "**Proposition (Barber--Agakov bound).** *For any conditional density\n",
    "$q(x \\mid y)$,*\n",
    "\n",
    "$$I(X; Y) \\geq H(X) + E_{(x,y) \\sim P_{X,Y}}\\left[\\log q(x \\mid y)\\right],$$\n",
    "\n",
    "*with equality if and only if $q(x \\mid y) = p(x \\mid y)$.*\n",
    "\n",
    "**Proof.** Write $I(X; Y) = H(X) - H(X \\mid Y)$ and insert the model:\n",
    "\n",
    "$$\n",
    "I(X; Y) - H(X) - E\\left[\\log q(x \\mid y)\\right]\n",
    "= E\\left[\\log \\frac{p(x \\mid y)}{q(x \\mid y)}\\right]\n",
    "= E_{y}\\left[D_{\\textrm{KL}}\\!\\left(p(\\cdot \\mid y) \\,\\|\\, q(\\cdot \\mid y)\\right)\\right]\n",
    "\\geq 0\n",
    "$$\n",
    "\n",
    "by Gibbs' inequality, applied for each value of $y$ and averaged. Equality\n",
    "holds iff the per-$y$ divergences all vanish, i.e., $q$ *is* the posterior.\n",
    "$\\blacksquare$\n",
    "\n",
    "The bound [@Barber.Agakov.2003] is the information-theoretic reading of\n",
    "an encoder--decoder: if a decoder can reconstruct $X$ from $Y$ with low\n",
    "cross-entropy, the pair must share a lot of information. Its weakness is the\n",
    "$H(X)$ term, which is unknown in exactly the situations where we need the\n",
    "bound (though it is a constant: *maximizing* the bound never needs it). The\n",
    "same proof with $q(x \\mid y)/p(x)$ in place of $q(x \\mid y)$ gives the\n",
    "equivalent ratio form $I \\geq E[\\log\\left(q(x \\mid y)/p(x)\\right)]$, which we\n",
    "will reuse for InfoNCE.\n",
    "\n",
    "### Donsker--Varadhan and MINE\n",
    "\n",
    "The second bound dispenses with normalized models altogether: it\n",
    "characterizes KL divergence, hence mutual information, as a supremum over\n",
    "arbitrary *critic* functions $T(x, y)$, with no requirement that anything\n",
    "integrate to one. This is the same convex-duality pattern that drives the\n",
    "f-GAN construction in that section.\n",
    "\n",
    "**Proposition (Donsker--Varadhan representation).** *For any distributions\n",
    "$P, Q$ and any function $T$ with $E_Q[e^T] < \\infty$,*\n",
    "\n",
    "$$D_{\\textrm{KL}}(P \\| Q) \\geq E_P[T] - \\log E_Q\\left[e^{T}\\right],$$\n",
    "\n",
    "*with equality if and only if $T = \\log \\frac{dP}{dQ} + c$ for a constant\n",
    "$c$; taking the supremum over $T$ attains $D_{\\textrm{KL}}(P \\| Q)$ (both\n",
    "statements for $P \\ll Q$ with finite divergence)\n",
    "[@Donsker.Varadhan.1983].*\n",
    "\n",
    "**Proof.** Given $T$, define the *Gibbs distribution*\n",
    "$g(x) = q(x)\\, e^{T(x)} / Z$ with normalizer $Z = E_Q[e^T]$, the\n",
    "distribution $Q$ \"tilted\" by the critic. Then\n",
    "\n",
    "$$\n",
    "D_{\\textrm{KL}}(P \\| Q) - \\left(E_P[T] - \\log Z\\right)\n",
    "= E_P\\left[\\log \\frac{p}{q} - T + \\log Z\\right]\n",
    "= E_P\\left[\\log \\frac{p}{g}\\right]\n",
    "= D_{\\textrm{KL}}(P \\| G) \\geq 0,\n",
    "$$\n",
    "\n",
    "by Gibbs' inequality (the eponymous coincidence is no accident). Equality\n",
    "holds iff $P = G$, i.e., $p/q = e^{T}/Z$, i.e., $T = \\log(p/q) + \\log Z$.\n",
    "$\\blacksquare$\n",
    "\n",
    "Specializing $P = P_{X,Y}$ and $Q = P_X \\otimes P_Y$ bounds mutual\n",
    "information: for any critic $T(x,y)$,\n",
    "\n",
    "$$\n",
    "I(X; Y) \\geq E_{P_{X,Y}}[T] - \\log E_{P_X \\otimes P_Y}\\left[e^{T}\\right],\n",
    "$$\n",
    "\n",
    "where the second expectation is estimated by *shuffling*: pairing each $x$\n",
    "with a $y$ drawn from a different example destroys the dependence and\n",
    "samples the product of marginals. **MINE** (mutual information neural\n",
    "estimation, @Belghazi.Baratin.Rajeshwar.ea.2018) is exactly this\n",
    "bound with $T$ a neural network trained by gradient ascent on\n",
    "the equation, plus a fix for the subtlety that the\n",
    "$\\log E[e^T]$ term makes naive minibatch gradients biased.\n",
    "\n",
    "### The NWJ Bound and the Bias--Variance Spectrum\n",
    "\n",
    "The logarithm in front of $E_Q[e^T]$ is the troublemaker: it couples all\n",
    "samples in the batch and gives the DV bound its bias and, at high MI, its\n",
    "exploding variance (the expectation of $e^T$ under $Q$ is carried by the\n",
    "same rare events that the McAllester--Stratos argument warned about). The\n",
    "tangent-line inequality $\\log z \\leq z/e$ (equality at $z = e$) trades\n",
    "tightness for tractability: applying it to $Z = E_Q[e^T]$ in\n",
    "the equation gives, for every critic $T$,\n",
    "\n",
    "$$I(X; Y) \\geq E_{P_{X,Y}}[T] - e^{-1}\\, E_{P_X \\otimes P_Y}\\left[e^{T}\\right],$$\n",
    "\n",
    "the **NWJ bound** [@Nguyen.Wainwright.Jordan.2010], tight at\n",
    "$T^* = 1 + \\log\\frac{dP}{dQ}$. Since $e^{-1} E_Q[e^T] = E_Q[e^{T-1}]$, the\n",
    "right-hand side is exactly the f-GAN bound for the KL divergence,\n",
    "$E_P[T] - E_Q[e^{T-1}]$, that Fenchel duality produced in\n",
    "that section: this is where we meet that bound\n",
    "again. Because no log-of-average appears, plain\n",
    "minibatch estimates of the equation are unbiased for the bound\n",
    "itself, but the $e^{T}$ term still has heavy tails, so the variance problem\n",
    "survives. the figure sketches the resulting\n",
    "family: estimators stacked under the true value, trading bias\n",
    "for variance, with the contrastive bound we meet next sitting lowest but\n",
    "steadiest.\n",
    "\n",
    "![Variational lower bounds on mutual information, after Poole et al. (2019). Every estimator sits at or below the true-MI diagonal, and each fails in its own way as $I$ grows: InfoNCE is low-variance but capped at $\\log N$; the decoder-based Barber--Agakov bound is low-variance but biased downward by its decoder's gap (it flattens); and the critic-based bounds lose control of their batch-to-batch spread (shaded band): NWJ's spread grows first while its mean falls below the truth, and the log-of-average in Donsker--Varadhan / MINE adds upward bias and the largest blow-up.](../img/mdl-it-mi-variational-bounds.svg)\n",
    "\n",
    "### InfoNCE: Estimation as Classification\n",
    "\n",
    "InfoNCE takes a different route to taming the partition\n",
    "function: it embeds the problem in a *classification* task, the guessing\n",
    "game we simulated in that section. Draw one positive pair and\n",
    "$N-1$ negatives, score every candidate with a critic $f(x, y)$, and ask a\n",
    "softmax to point at the positive. The **InfoNCE loss**\n",
    "[@Oord.Li.Vinyals.2018] is the resulting cross-entropy,\n",
    "\n",
    "$$\\mathcal{L}_{\\textrm{NCE}} = -E\\left[\\log \\frac{e^{f(x, y_1)}}{\\sum_{j=1}^{N} e^{f(x, y_j)}}\\right],$$\n",
    "\n",
    "where the expectation is over $(x, y_1) \\sim P_{X,Y}$ and\n",
    "$y_2, \\ldots, y_N \\sim P_Y$ i.i.d., and the associated mutual-information\n",
    "*estimator* is the batch ceiling minus the loss,\n",
    "\n",
    "$$\\hat{I}_{\\textrm{NCE}} = \\log N - \\mathcal{L}_{\\textrm{NCE}}.$$\n",
    "\n",
    "(Many papers, following @Oord.Li.Vinyals.2018, write the loss with a\n",
    "$\\frac{1}{N}$ inside the logarithm and a compensating $+\\log N$ outside; the\n",
    "two cancel, so that convention and the equation are the same\n",
    "number.) Note what the loss *is*: a $k$-class categorical cross-entropy\n",
    "with $k = N$, the same loss dissected in\n",
    "that section and\n",
    "that section, with classes that are *other examples\n",
    "in the batch* rather than fixed labels. That is the entire engineering\n",
    "appeal (this loss is exactly what deep-learning libraries optimize best),\n",
    "and it is the loss\n",
    "of CPC, SimCLR, and CLIP-style dual encoders, where $x$ and $y$ are two\n",
    "views or two modalities of the same datum and the batch supplies the\n",
    "negatives [@Oord.Li.Vinyals.2018; @radford2021learning].\n",
    "\n",
    "![The InfoNCE game. Left: an anchor $x$ is scored by the critic $f$ against the positive $y^+$ drawn jointly with it and $N-1$ negatives drawn independently from the marginal. Right: a softmax over the $N$ scores must point at the positive; the resulting cross-entropy certifies $I(X;Y) \\geq \\log N - \\mathcal{L}_{\\textrm{NCE}}$, a bound that can never exceed $\\log N$.](../img/mdl-it-infonce-pos-neg.svg)\n",
    "\n",
    "the figure draws the game. The payoff is a theorem\n",
    "[@Poole.Oord.Alemi.ea.2019]: small InfoNCE loss *certifies* mutual\n",
    "information.\n",
    "\n",
    "**Proposition (InfoNCE bound).** *For any critic $f$ and any $N \\geq 1$,*\n",
    "\n",
    "$$I(X; Y) \\geq \\hat{I}_{\\textrm{NCE}} = \\log N - \\mathcal{L}_{\\textrm{NCE}}, \\qquad \\textrm{and } \\hat{I}_{\\textrm{NCE}} \\textrm{ never exceeds } \\log N.$$\n",
    "\n",
    "**Proof.** Write $V = (Y_1, \\ldots, Y_N)$ for the whole bag of candidates,\n",
    "where $(X, Y_1) \\sim P_{X,Y}$ and the negatives $Y_{2:N} \\sim P_Y$ are\n",
    "independent of $(X, Y_1)$.\n",
    "\n",
    "*Step 1: the negatives are free.* By the corollary to the chain rule,\n",
    "independent side information adds nothing: $I(X; V) = I(X; Y_1)$.\n",
    "\n",
    "*Step 2: a normalized variational model in disguise.* Define\n",
    "\n",
    "$$\n",
    "g(x, v) = \\frac{e^{f(x, y_1)}}{\\frac{1}{N}\\sum_{j=1}^N e^{f(x, y_j)}}.\n",
    "$$\n",
    "\n",
    "Under the product distribution $p(x)\\,p(v)$, where all $N$ candidates,\n",
    "including $y_1$, are i.i.d. draws from $P_Y$, exchangeability of the $y_j$\n",
    "makes each of the $N$ terms $E\\big[e^{f(x,y_i)} / \\frac{1}{N}\\sum_j\n",
    "e^{f(x,y_j)}\\big]$ equal, so for every fixed $x$,\n",
    "\n",
    "$$\n",
    "E_{p(v)}\\left[g(x, V)\\right]\n",
    "= \\frac{1}{N}\\sum_{i=1}^{N} E\\left[\\frac{e^{f(x, y_i)}}{\\frac{1}{N}\\sum_{j} e^{f(x, y_j)}}\\right]\n",
    "= E\\left[\\frac{\\frac{1}{N}\\sum_{i} e^{f(x, y_i)}}{\\frac{1}{N}\\sum_{j} e^{f(x, y_j)}}\\right] = 1.\n",
    "$$\n",
    "\n",
    "Hence $q(v \\mid x) = p(v)\\, g(x, v)$ is a *bona fide* normalized conditional\n",
    "distribution over bags.\n",
    "\n",
    "*Step 3: Barber--Agakov on the bag.* The ratio form of\n",
    "the equation applied to $(X, V)$ with the model $q(v \\mid x)$\n",
    "gives\n",
    "\n",
    "$$\n",
    "I(X; V) \\geq E_{p(x, v)}\\left[\\log \\frac{q(v \\mid x)}{p(v)}\\right]\n",
    "= E\\left[\\log g(X, V)\\right]\n",
    "= \\log N - \\mathcal{L}_{\\textrm{NCE}}.\n",
    "$$\n",
    "\n",
    "Combining with Step 1 proves the bound. For the ceiling, the softmax assigns\n",
    "the positive a probability at most $1$, so\n",
    "$\\mathcal{L}_{\\textrm{NCE}} \\geq 0$ and the estimator\n",
    "the equation is at most $\\log N$. $\\blacksquare$\n",
    "\n",
    "The two halves of the proposition are the two faces of contrastive learning.\n",
    "The bound explains *why it works*: driving the classification loss down\n",
    "provably drives shared information up, and the optimal critic recovers the\n",
    "density ratio $f^*(x,y) = \\textrm{pmi}(x,y) + c(x)$ (Exercise 8;\n",
    "[@Poole.Oord.Alemi.ea.2019]). The ceiling explains\n",
    "*why batch size matters so much* in practice: with $N$ candidates the bound\n",
    "simply cannot certify more than $\\log N$ nats, which is why CPC, SimCLR, and\n",
    "CLIP push toward enormous batches, and why even those raise the certifiable\n",
    "information only logarithmically. One practical dial deserves naming: in\n",
    "SimCLR-style systems the critic is a scaled cosine similarity between the\n",
    "two embeddings, $f(x, y) = \\textrm{sim}(z_x, z_y)/\\tau$, and the\n",
    "*temperature* $\\tau$\n",
    "[@Chen.Kornblith.Norouzi.ea.2020] is the same knob as the distillation\n",
    "temperature of that section: lowering $\\tau$\n",
    "sharpens the softmax over candidates and concentrates the loss on the\n",
    "hardest negatives.\n",
    "\n",
    "Before training anything, let's compare all three bounds in their ideal\n",
    "state, critics set to their optima exactly using the closed-form Gaussian\n",
    "ratio, so that everything we see is a property of the *bounds*, not of\n",
    "optimization. We use a batch of $N = 128$ and report the mean and standard\n",
    "deviation across $200$ batches."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mutual-information-perfect-critic-bounds"
   },
   "source": [
    "rng = onp.random.default_rng(3)\n",
    "N, B = 128, 200\n",
    "print('true I    DV (MINE)         NWJ              InfoNCE')\n",
    "for tm in [0.5, 1.0, 2.0, 4.0, 6.0]:\n",
    "    rho = onp.sqrt(1 - onp.exp(-2 * tm))\n",
    "    dv, nwj, nce = [], [], []\n",
    "    for _ in range(B):\n",
    "        x, y = gauss_pair(rng, rho, N)\n",
    "        joint = pmi_gauss(x, y, rho)               # T* on positive pairs\n",
    "        # A random permutation leaves ~1 fixed point per batch of N=128, so\n",
    "        # one \"negative\" is secretly a true pair; the O(1/N) contamination\n",
    "        # is negligible next to the heavy tails probed here.\n",
    "        prod = pmi_gauss(x, rng.permutation(y), rho)   # T* on shuffled pairs\n",
    "        dv.append(joint.mean() - onp.log(onp.mean(onp.exp(prod))))\n",
    "        nwj.append(joint.mean() + 1 - onp.mean(onp.exp(prod)))\n",
    "        f = pmi_gauss(x[:, None], y[None, :], rho)\n",
    "        f -= f.max(axis=1, keepdims=True)\n",
    "        ll = onp.diag(f) - onp.log(onp.exp(f).sum(axis=1))\n",
    "        nce.append(ll.mean() + onp.log(N))\n",
    "    print(f'{tm:5.1f}   {onp.mean(dv):6.2f} +-{onp.std(dv):6.2f} '\n",
    "          f'  {onp.mean(nwj):6.2f} +-{onp.std(nwj):6.2f} '\n",
    "          f'  {onp.mean(nce):6.2f} +-{onp.std(nce):6.2f}')"
   ],
   "execution_count": 10,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "true I    DV (MINE)         NWJ              InfoNCE\n  0.5     0.50 +-  0.12     0.50 +-  0.12     0.50 +-  0.07\n  1.0     1.01 +-  0.19     0.99 +-  0.22     1.00 +-  0.07\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "  2.0     1.93 +-  0.35     1.83 +-  0.97     1.94 +-  0.09\n  4.0     3.79 +-  0.92     3.36 +-  1.18     3.67 +-  0.08\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "  6.0    23.22 +- 56.65     1.23 +- 11.74     4.61 +-  0.05\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-mutual-information-md-11 fw=all -->\n",
    "\n",
    "The table is the figure in numbers. At low\n",
    "mutual information all three estimators are accurate and well-behaved. As\n",
    "the truth climbs past $\\ln N \\approx 4.85$ nats, they fail in their\n",
    "characteristic ways: InfoNCE stays steady (standard deviation a few\n",
    "hundredths of a nat) but saturates at its ceiling; NWJ's batch-to-batch\n",
    "spread grows first, exploding to many nats while its mean falls far below\n",
    "the truth; and DV's log-of-average adds upward bias and the largest blow-up,\n",
    "until single batches return values that are pure noise: at a true MI of $6$\n",
    "nats its batch estimates average far *above* the truth with a spread of tens\n",
    "of nats. With a perfect\n",
    "critic. Estimator choice is a bias--variance decision, not a ranking.\n",
    "\n",
    "### Experiment: Learning the Critic\n",
    "\n",
    "Finally, let the critic be learned, as in practice. We train a small MLP\n",
    "critic $f(x, y)$ with the InfoNCE loss on a correlated Gaussian pair with\n",
    "$\\rho = 0.99$, whose true mutual information\n",
    "$-\\tfrac{1}{2}\\log(1 - 0.99^2) \\approx 1.958$ nats we know exactly. The\n",
    "data are generated once in NumPy with a fixed\n",
    "seed; training uses batches of $N = 128$, and we then evaluate the trained\n",
    "critic's InfoNCE bound at several evaluation batch sizes to watch the\n",
    "$\\log N$ ceiling and the approach to the truth in one table."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mutual-information-infonce-train"
   },
   "source": [
    "rho, n_pool = 0.99, 4096\n",
    "data_rng = onp.random.default_rng(0)\n",
    "x_tr, y_tr = (a.astype('float32') for a in gauss_pair(data_rng, rho, n_pool))\n",
    "x_ev, y_ev = (a.astype('float32') for a in gauss_pair(data_rng, rho, n_pool))\n",
    "\n",
    "np.random.seed(0)\n",
    "critic = gluon.nn.Sequential()\n",
    "critic.add(gluon.nn.Dense(64, activation='relu'),\n",
    "           gluon.nn.Dense(64, activation='relu'), gluon.nn.Dense(1))\n",
    "critic.initialize(init.Xavier())\n",
    "trainer = gluon.Trainer(critic.collect_params(), 'adam',\n",
    "                        {'learning_rate': 1e-3})\n",
    "\n",
    "def scores(x, y):\n",
    "    \"\"\"Matrix of critic scores f(x_i, y_j) for all pairs in a batch.\"\"\"\n",
    "    N = x.shape[0]\n",
    "    xy = np.stack([np.tile(x.reshape(-1, 1), (1, N)),\n",
    "                   np.tile(y.reshape(1, -1), (N, 1))], axis=-1)\n",
    "    return critic(xy.reshape(-1, 2)).reshape(N, N)\n",
    "\n",
    "def diag_minus_logsumexp(f):\n",
    "    \"\"\"Per-row log-softmax of the diagonal (positive-pair) scores.\"\"\"\n",
    "    m = f.max(axis=1, keepdims=True)\n",
    "    lse = (m + np.log(np.exp(f - m).sum(axis=1, keepdims=True)))[:, 0]\n",
    "    return (f * np.eye(f.shape[0])).sum(axis=1) - lse\n",
    "\n",
    "N = 128\n",
    "for step in range(400):\n",
    "    idx = data_rng.choice(n_pool, N, replace=False)\n",
    "    x, y = np.array(x_tr[idx]), np.array(y_tr[idx])\n",
    "    with autograd.record():\n",
    "        loss = -diag_minus_logsumexp(scores(x, y)).mean()\n",
    "    loss.backward()\n",
    "    trainer.step(1)\n",
    "\n",
    "for N_ev in [2, 8, 32, 128]:\n",
    "    vals = [float(diag_minus_logsumexp(\n",
    "                scores(np.array(x_ev[i:i + N_ev]),\n",
    "                       np.array(y_ev[i:i + N_ev]))).mean())\n",
    "            for i in range(0, n_pool, N_ev)]\n",
    "    print(f'N = {N_ev:4d}: ln N = {onp.log(N_ev):.3f}, '\n",
    "          f'InfoNCE bound = {onp.log(N_ev) + onp.mean(vals):.3f} nats')\n",
    "print(f'true I(X;Y) = {-0.5 * onp.log(1 - rho**2):.3f} nats')"
   ],
   "execution_count": 11,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "[14:01:17] /home/smola/mxnet/src/base.cc:48: GPU context requested, but no GPUs found.\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "N =    2: ln N = 0.693, InfoNCE bound = 0.570 nats\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "N =    8: ln N = 2.079, InfoNCE bound = 1.408 nats\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "N =   32: ln N = 3.466, InfoNCE bound = 1.764 nats\nN =  128: ln N = 4.852, InfoNCE bound = 1.891 nats\ntrue I(X;Y) = 1.959 nats\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-mutual-information-md-12 fw=all -->\n",
    "\n",
    "A two-layer critic trained for $400$ steps recovers the structure almost\n",
    "perfectly. Reading the table against the truth of $1.958$ nats: at\n",
    "$N = 2$ the bound is pinned below its ceiling $\\ln 2 \\approx 0.693$; at\n",
    "$N = 8$ the ceiling ($\\approx 2.08$) finally clears the truth and the bound\n",
    "jumps to about $1.4$; and by $N = 128$ the learned critic certifies about\n",
    "$1.89$ nats, within $0.07$ of the truth, the residual being the cost of an\n",
    "imperfect critic and finite evaluation data. The same critic, the same data:\n",
    "only the number of negatives changed. Batch size sets the resolution of the\n",
    "instrument.\n",
    "\n",
    "## The Information Bottleneck and the Limits of Mutual Information\n",
    "\n",
    "We close with the principle that joins this section's two themes, what\n",
    "representations preserve and what we can measure, into a single objective,\n",
    "and with an assessment of where its empirical support stands.\n",
    "\n",
    "### Compression with a Purpose: the IB Lagrangian\n",
    "\n",
    "What makes a representation $Z$ of an input $X$ *good* for predicting a\n",
    "label $Y$? The *information bottleneck* (IB) principle\n",
    "[@Tishby.Pereira.Bialek.1999] answers with two mutual informations: a\n",
    "good $Z$ keeps little of $X$ (it *compresses*) while keeping much of $Y$ (it\n",
    "*predicts*). Over stochastic encoders $p(z \\mid x)$, minimize the Lagrangian\n",
    "\n",
    "$$\\min_{p(z \\mid x)}\\; I(X; Z) - \\beta\\, I(Y; Z).$$\n",
    "\n",
    "The data-processing inequality is what makes this well-posed: since\n",
    "$Y \\to X \\to Z$ is a Markov chain (the encoder sees only $X$), we always\n",
    "have $I(Y; Z) \\leq \\min\\{I(X; Y),\\, I(X; Z)\\}$: the representation can\n",
    "never know more about the label than the input does, and predicting well\n",
    "forces $Z$ to retain input information. The multiplier $\\beta$ sets the\n",
    "tradeoff: as $\\beta \\to 0$ the optimum collapses to an uninformative $Z$\n",
    "(maximal compression), as $\\beta \\to \\infty$ it retains everything relevant\n",
    "(and, en route, recovers sufficient statistics in the sense of\n",
    "that section); sweeping $\\beta$\n",
    "traces a frontier in the *information plane* with coordinates\n",
    "$(I(X;Z),\\, I(Y;Z))$. Rate--distortion theory [@Shannon.1959] is the\n",
    "special case where\n",
    "\"relevance\" is a hand-chosen distortion measure; IB lets the label define\n",
    "relevance instead. The *deep variational information bottleneck* (VIB)\n",
    "[@Alemi.Fischer.Dillon.ea.2017] makes the equation\n",
    "trainable by bounding both terms with the machinery we just built: a\n",
    "Barber--Agakov-style decoder bound for $I(Y;Z)$ from below and a variational\n",
    "upper bound for $I(X;Z)$, the same replace-the-intractable-posterior pattern\n",
    "as the ELBO of that section: another reminder that in\n",
    "deep learning, mutual informations are optimized through bounds, not\n",
    "computed. The frontier this\n",
    "tradeoff traces is shown in the figure.\n",
    "\n",
    "![The information plane of the Gaussian bottleneck with $\\rho = 0.9$: the concave frontier traces the best attainable $I(Y;Z)$ at each compression level $I(X;Z)$, the shaded region above it is infeasible, and the dashed ceiling is the data-processing limit $I(X;Y) = -\\tfrac{1}{2}\\log(1-\\rho^2) \\approx 0.830$ nats. Dots mark the operating points the multiplier $\\beta$ selects; for $\\beta \\leq 1/\\rho^2$ the optimum collapses to the origin.](../img/mdl-it-ib-tradeoff.svg)\n",
    "\n",
    "### The Information Plane: a Gaussian Bottleneck in Closed Form\n",
    "\n",
    "Rather than train a VIB (two stacked variational bounds would put us two\n",
    "bounds away from anything checkable), we exhibit the IB tradeoff in a model\n",
    "where every quantity is exact [@Chechik.Globerson.Tishby.ea.2005]. Let\n",
    "$(X, Y)$ be our standard Gaussian pair with $\\rho = 0.9$, so\n",
    "$I(X; Y) \\approx 0.830$ nats, and let the encoder be a noisy channel\n",
    "\n",
    "$$\n",
    "Z = X + \\sigma\\, \\epsilon, \\qquad \\epsilon \\sim \\mathcal{N}(0, 1),\n",
    "$$\n",
    "\n",
    "with the noise scale $\\sigma$ as the compression knob. Both informations\n",
    "follow from the Gaussian anchor the equation: the correlations\n",
    "are $\\rho_{XZ}^2 = 1/(1 + \\sigma^2)$ and\n",
    "$\\rho_{YZ}^2 = \\rho^2/(1 + \\sigma^2)$, so\n",
    "\n",
    "$$\n",
    "I(X; Z) = \\tfrac{1}{2}\\log\\left(1 + \\sigma^{-2}\\right), \\qquad\n",
    "I(Y; Z) = -\\tfrac{1}{2}\\log\\left(1 - \\frac{\\rho^2}{1 + \\sigma^2}\\right).\n",
    "$$\n",
    "\n",
    "In this scalar Gaussian family the noise level is the *only* degree of\n",
    "freedom, so sweeping $\\sigma$ traces the family's entire curve; that this\n",
    "curve is the *optimal* frontier over all encoders $p(z \\mid x)$ is the\n",
    "Gaussian information bottleneck theorem of\n",
    "[@Chechik.Globerson.Tishby.ea.2005]. Minimizing the Lagrangian\n",
    "the equation over $\\sigma$ for\n",
    "each $\\beta$ picks out one operating point on the frontier."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mutual-information-ib-plane"
   },
   "source": [
    "rho = 0.9\n",
    "sigma = onp.logspace(-2, 2, 401)            # noise scale: the beta knob\n",
    "I_xz = 0.5 * onp.log(1 + sigma**-2)         # compression cost\n",
    "I_yz = -0.5 * onp.log(1 - rho**2 / (1 + sigma**2))  # predictive value\n",
    "I_xy = -0.5 * onp.log(1 - rho**2)\n",
    "\n",
    "d2l.plot(I_xz, [I_yz, onp.full_like(I_xz, I_xy)], 'I(X;Z) (nats)',\n",
    "         'I(Y;Z) (nats)', legend=['IB frontier', 'I(X;Y) ceiling'],\n",
    "         xlim=[0, 3], figsize=(5, 3))\n",
    "for beta in [1.0, 1.5, 2.0, 4.0, 16.0]:\n",
    "    i = onp.argmin(I_xz - beta * I_yz)      # IB Lagrangian on the grid\n",
    "    d2l.plt.plot(I_xz[i], I_yz[i], 'ro')\n",
    "    d2l.plt.annotate(f'{beta:g}', (I_xz[i], I_yz[i]),\n",
    "                     textcoords='offset points', xytext=(6, -2))\n",
    "    print(f'beta = {beta:4.1f}: I(X;Z) = {I_xz[i]:.3f}, '\n",
    "          f'I(Y;Z) = {I_yz[i]:.3f} nats')"
   ],
   "execution_count": 12,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "beta =  1.0: I(X;Z) = 0.000, I(Y;Z) = 0.000 nats\nbeta =  1.5: I(X;Z) = 0.382, I(Y;Z) = 0.284 nats\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "beta =  2.0: I(X;Z) = 0.731, I(Y;Z) = 0.487 nats\nbeta =  4.0: I(X;Z) = 1.283, I(Y;Z) = 0.689 nats\nbeta = 16.0: I(X;Z) = 2.080, I(Y;Z) = 0.798 nats\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"337.932812pt\" height=\"211.07625pt\" viewBox=\"0 0 337.932812 211.07625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-19T14:01:28.699738</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 211.07625 \nL 337.932812 211.07625 \nL 337.932812 0 \nL 0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 43.78125 173.52 \nL 322.78125 173.52 \nL 322.78125 7.2 \nL 43.78125 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path d=\"M 43.78125 173.52 \nL 43.78125 7.2 \n\" clip-path=\"url(#p39173d56ab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path id=\"m38c9f90918\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m38c9f90918\" x=\"43.78125\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0.0 -->\n      <g transform=\"translate(35.829688 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \nL 1344 794 \nL 1344 0 \nL 684 0 \nL 684 794 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path d=\"M 90.28125 173.52 \nL 90.28125 7.2 \n\" clip-path=\"url(#p39173d56ab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#m38c9f90918\" x=\"90.28125\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 0.5 -->\n      <g transform=\"translate(82.329688 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \nL 3169 4666 \nL 3169 4134 \nL 1269 4134 \nL 1269 2991 \nQ 1406 3038 1543 3061 \nQ 1681 3084 1819 3084 \nQ 2600 3084 3056 2656 \nQ 3513 2228 3513 1497 \nQ 3513 744 3044 326 \nQ 2575 -91 1722 -91 \nQ 1428 -91 1123 -41 \nQ 819 9 494 109 \nL 494 744 \nQ 775 591 1075 516 \nQ 1375 441 1709 441 \nQ 2250 441 2565 725 \nQ 2881 1009 2881 1497 \nQ 2881 1984 2565 2268 \nQ 2250 2553 1709 2553 \nQ 1456 2553 1204 2497 \nQ 953 2441 691 2322 \nL 691 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path d=\"M 136.78125 173.52 \nL 136.78125 7.2 \n\" clip-path=\"url(#p39173d56ab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#m38c9f90918\" x=\"136.78125\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 1.0 -->\n      <g transform=\"translate(128.829688 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_7\">\n      <path d=\"M 183.28125 173.52 \nL 183.28125 7.2 \n\" clip-path=\"url(#p39173d56ab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#m38c9f90918\" x=\"183.28125\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 1.5 -->\n      <g transform=\"translate(175.329688 188.118438) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_9\">\n      <path d=\"M 229.78125 173.52 \nL 229.78125 7.2 \n\" clip-path=\"url(#p39173d56ab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#m38c9f90918\" x=\"229.78125\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 2.0 -->\n      <g transform=\"translate(221.829688 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-32\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_11\">\n      <path d=\"M 276.28125 173.52 \nL 276.28125 7.2 \n\" clip-path=\"url(#p39173d56ab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_12\">\n      <g>\n       <use xlink:href=\"#m38c9f90918\" x=\"276.28125\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 2.5 -->\n      <g transform=\"translate(268.329687 188.118438) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-32\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_7\">\n     <g id=\"line2d_13\">\n      <path d=\"M 322.78125 173.52 \nL 322.78125 7.2 \n\" clip-path=\"url(#p39173d56ab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_14\">\n      <g>\n       <use xlink:href=\"#m38c9f90918\" x=\"322.78125\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_7\">\n      <!-- 3.0 -->\n      <g transform=\"translate(314.829687 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \nQ 3050 2419 3304 2112 \nQ 3559 1806 3559 1356 \nQ 3559 666 3084 287 \nQ 2609 -91 1734 -91 \nQ 1441 -91 1130 -33 \nQ 819 25 488 141 \nL 488 750 \nQ 750 597 1062 519 \nQ 1375 441 1716 441 \nQ 2309 441 2620 675 \nQ 2931 909 2931 1356 \nQ 2931 1769 2642 2001 \nQ 2353 2234 1838 2234 \nL 1294 2234 \nL 1294 2753 \nL 1863 2753 \nQ 2328 2753 2575 2939 \nQ 2822 3125 2822 3475 \nQ 2822 3834 2567 4026 \nQ 2313 4219 1838 4219 \nQ 1578 4219 1281 4162 \nQ 984 4106 628 3988 \nL 628 4550 \nQ 988 4650 1302 4700 \nQ 1616 4750 1894 4750 \nQ 2613 4750 3031 4423 \nQ 3450 4097 3450 3541 \nQ 3450 3153 3228 2886 \nQ 3006 2619 2597 2516 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-33\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_8\">\n     <!-- I(X;Z) (nats) -->\n     <g transform=\"translate(153.082031 201.796563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-49\" d=\"M 628 4666 \nL 1259 4666 \nL 1259 0 \nL 628 0 \nL 628 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \nQ 1566 4138 1362 3434 \nQ 1159 2731 1159 2009 \nQ 1159 1288 1364 580 \nQ 1569 -128 1984 -844 \nL 1484 -844 \nQ 1016 -109 783 600 \nQ 550 1309 550 2009 \nQ 550 2706 781 3412 \nQ 1013 4119 1484 4856 \nL 1984 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-58\" d=\"M 403 4666 \nL 1081 4666 \nL 2241 2931 \nL 3406 4666 \nL 4084 4666 \nL 2584 2425 \nL 4184 0 \nL 3506 0 \nL 2194 1984 \nL 872 0 \nL 191 0 \nL 1856 2491 \nL 403 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-3b\" d=\"M 750 3309 \nL 1409 3309 \nL 1409 2516 \nL 750 2516 \nL 750 3309 \nz\nM 750 794 \nL 1409 794 \nL 1409 256 \nL 897 -744 \nL 494 -744 \nL 750 256 \nL 750 794 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-5a\" d=\"M 359 4666 \nL 4025 4666 \nL 4025 4184 \nL 1075 531 \nL 4097 531 \nL 4097 0 \nL 288 0 \nL 288 481 \nL 3238 4134 \nL 359 4134 \nL 359 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \nL 1013 4856 \nQ 1481 4119 1714 3412 \nQ 1947 2706 1947 2009 \nQ 1947 1309 1714 600 \nQ 1481 -109 1013 -844 \nL 513 -844 \nQ 928 -128 1133 580 \nQ 1338 1288 1338 2009 \nQ 1338 2731 1133 3434 \nQ 928 4138 513 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-49\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(29.492188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-58\" transform=\"translate(68.505859 0)\"/>\n      <use xlink:href=\"#DejaVuSans-3b\" transform=\"translate(137.011719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5a\" transform=\"translate(170.703125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(239.208984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(278.222656 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(310.009766 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(349.023438 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(412.402344 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(473.681641 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(512.890625 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(564.990234 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_15\">\n      <path d=\"M 43.78125 165.967375 \nL 322.78125 165.967375 \n\" clip-path=\"url(#p39173d56ab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_16\">\n      <defs>\n       <path id=\"m36793608f7\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m36793608f7\" x=\"43.78125\" y=\"165.967375\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.0 -->\n      <g transform=\"translate(20.878125 169.766593) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_17\">\n      <path d=\"M 43.78125 129.547905 \nL 322.78125 129.547905 \n\" clip-path=\"url(#p39173d56ab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_18\">\n      <g>\n       <use xlink:href=\"#m36793608f7\" x=\"43.78125\" y=\"129.547905\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 0.2 -->\n      <g transform=\"translate(20.878125 133.347124) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_19\">\n      <path d=\"M 43.78125 93.128435 \nL 322.78125 93.128435 \n\" clip-path=\"url(#p39173d56ab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_20\">\n      <g>\n       <use xlink:href=\"#m36793608f7\" x=\"43.78125\" y=\"93.128435\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 0.4 -->\n      <g transform=\"translate(20.878125 96.927654) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_21\">\n      <path d=\"M 43.78125 56.708966 \nL 322.78125 56.708966 \n\" clip-path=\"url(#p39173d56ab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_22\">\n      <g>\n       <use xlink:href=\"#m36793608f7\" x=\"43.78125\" y=\"56.708966\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 0.6 -->\n      <g transform=\"translate(20.878125 60.508184) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-36\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_23\">\n      <path d=\"M 43.78125 20.289496 \nL 322.78125 20.289496 \n\" clip-path=\"url(#p39173d56ab)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_24\">\n      <g>\n       <use xlink:href=\"#m36793608f7\" x=\"43.78125\" y=\"20.289496\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_13\">\n      <!-- 0.8 -->\n      <g transform=\"translate(20.878125 24.088715) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-38\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_14\">\n     <!-- I(Y;Z) (nats) -->\n     <g transform=\"translate(14.798438 120.188125) rotate(-90) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-59\" d=\"M -13 4666 \nL 666 4666 \nL 1959 2747 \nL 3244 4666 \nL 3922 4666 \nL 2272 2222 \nL 2272 0 \nL 1638 0 \nL 1638 2222 \nL -13 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-49\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(29.492188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-59\" transform=\"translate(68.505859 0)\"/>\n      <use xlink:href=\"#DejaVuSans-3b\" transform=\"translate(129.589844 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5a\" transform=\"translate(163.28125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(231.787109 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(270.800781 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(302.587891 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(341.601562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(404.980469 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(466.259766 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(505.46875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(557.568359 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_25\">\n    <path d=\"M 338.932812 15.437403 \nL 315.87349 15.869445 \nL 296.669476 16.431555 \nL 279.628178 17.161777 \nL 266.872451 17.906885 \nL 254.145943 18.875446 \nL 243.569324 19.897373 \nL 233.02595 21.159852 \nL 224.62071 22.376125 \nL 216.247179 23.806422 \nL 207.911455 25.481662 \nL 199.620734 27.434682 \nL 193.437263 29.102126 \nL 187.287931 30.959196 \nL 181.177333 33.019764 \nL 175.110608 35.29681 \nL 169.093486 37.801901 \nL 163.132322 40.544599 \nL 157.234135 43.531851 \nL 151.406633 46.76735 \nL 145.658224 50.250949 \nL 139.998016 53.978122 \nL 134.435791 57.939557 \nL 128.981961 62.120888 \nL 123.647482 66.502638 \nL 118.443746 71.060354 \nL 113.382428 75.764991 \nL 108.4753 80.583508 \nL 102.19238 87.122374 \nL 96.230612 93.707473 \nL 90.613957 100.244883 \nL 84.10893 108.210843 \nL 78.203288 115.797831 \nL 71.926738 124.216585 \nL 64.917107 134.029767 \nL 57.62358 144.672568 \nL 50.059042 156.142013 \nL 43.7859 165.96 \nL 43.7859 165.96 \n\" clip-path=\"url(#p39173d56ab)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_26\">\n    <path d=\"M 338.932812 14.76 \nL 43.7859 14.76 \nL 43.7859 14.76 \n\" clip-path=\"url(#p39173d56ab)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"line2d_27\">\n    <defs>\n     <path id=\"m500b0ded29\" d=\"M 0 3 \nC 0.795609 3 1.55874 2.683901 2.12132 2.12132 \nC 2.683901 1.55874 3 0.795609 3 0 \nC 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \nC 1.55874 -2.683901 0.795609 -3 0 -3 \nC -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \nC -2.683901 -1.55874 -3 -0.795609 -3 0 \nC -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \nC -1.55874 2.683901 -0.795609 3 0 3 \nz\n\" style=\"stroke: #ff0000\"/>\n    </defs>\n    <g clip-path=\"url(#p39173d56ab)\">\n     <use xlink:href=\"#m500b0ded29\" x=\"43.7859\" y=\"165.96\" style=\"fill: #ff0000; stroke: #ff0000\"/>\n    </g>\n   </g>\n   <g id=\"line2d_28\">\n    <g clip-path=\"url(#p39173d56ab)\">\n     <use xlink:href=\"#m500b0ded29\" x=\"79.335554\" y=\"114.317643\" style=\"fill: #ff0000; stroke: #ff0000\"/>\n    </g>\n   </g>\n   <g id=\"line2d_29\">\n    <g clip-path=\"url(#p39173d56ab)\">\n     <use xlink:href=\"#m500b0ded29\" x=\"111.729007\" y=\"77.360242\" style=\"fill: #ff0000; stroke: #ff0000\"/>\n    </g>\n   </g>\n   <g id=\"line2d_30\">\n    <g clip-path=\"url(#p39173d56ab)\">\n     <use xlink:href=\"#m500b0ded29\" x=\"163.132322\" y=\"40.544599\" style=\"fill: #ff0000; stroke: #ff0000\"/>\n    </g>\n   </g>\n   <g id=\"line2d_31\">\n    <g clip-path=\"url(#p39173d56ab)\">\n     <use xlink:href=\"#m500b0ded29\" x=\"237.238818\" y=\"20.622974\" style=\"fill: #ff0000; stroke: #ff0000\"/>\n    </g>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 43.78125 173.52 \nL 43.78125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 322.78125 173.52 \nL 322.78125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 43.78125 173.52 \nL 322.78125 173.52 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 43.78125 7.2 \nL 322.78125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"text_15\">\n    <!-- 1 -->\n    <g transform=\"translate(49.7859 167.96) scale(0.1 -0.1)\">\n     <use xlink:href=\"#DejaVuSans-31\"/>\n    </g>\n   </g>\n   <g id=\"text_16\">\n    <!-- 1.5 -->\n    <g transform=\"translate(85.335554 116.317643) scale(0.1 -0.1)\">\n     <use xlink:href=\"#DejaVuSans-31\"/>\n     <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n     <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(95.410156 0)\"/>\n    </g>\n   </g>\n   <g id=\"text_17\">\n    <!-- 2 -->\n    <g transform=\"translate(117.729007 79.360242) scale(0.1 -0.1)\">\n     <use xlink:href=\"#DejaVuSans-32\"/>\n    </g>\n   </g>\n   <g id=\"text_18\">\n    <!-- 4 -->\n    <g transform=\"translate(169.132322 42.544599) scale(0.1 -0.1)\">\n     <use xlink:href=\"#DejaVuSans-34\"/>\n    </g>\n   </g>\n   <g id=\"text_19\">\n    <!-- 16 -->\n    <g transform=\"translate(243.238818 22.622974) scale(0.1 -0.1)\">\n     <use xlink:href=\"#DejaVuSans-31\"/>\n     <use xlink:href=\"#DejaVuSans-36\" transform=\"translate(63.623047 0)\"/>\n    </g>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 220.851562 168.52 \nL 315.78125 168.52 \nQ 317.78125 168.52 317.78125 166.52 \nL 317.78125 138.16375 \nQ 317.78125 136.16375 315.78125 136.16375 \nL 220.851562 136.16375 \nQ 218.851562 136.16375 218.851562 138.16375 \nL 218.851562 166.52 \nQ 218.851562 168.52 220.851562 168.52 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_32\">\n     <path d=\"M 222.851562 144.262188 \nL 232.851562 144.262188 \nL 242.851562 144.262188 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_20\">\n     <!-- IB frontier -->\n     <g transform=\"translate(250.851562 147.762188) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \nL 1259 519 \nL 2272 519 \nQ 2781 519 3026 730 \nQ 3272 941 3272 1375 \nQ 3272 1813 3026 2020 \nQ 2781 2228 2272 2228 \nL 1259 2228 \nz\nM 1259 4147 \nL 1259 2741 \nL 2194 2741 \nQ 2656 2741 2882 2914 \nQ 3109 3088 3109 3444 \nQ 3109 3797 2882 3972 \nQ 2656 4147 2194 4147 \nL 1259 4147 \nz\nM 628 4666 \nL 2241 4666 \nQ 2963 4666 3353 4366 \nQ 3744 4066 3744 3513 \nQ 3744 3084 3544 2831 \nQ 3344 2578 2956 2516 \nQ 3422 2416 3680 2098 \nQ 3938 1781 3938 1306 \nQ 3938 681 3513 340 \nQ 3088 0 2303 0 \nL 628 0 \nL 628 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \nL 2375 4384 \nL 1825 4384 \nQ 1516 4384 1395 4259 \nQ 1275 4134 1275 3809 \nL 1275 3500 \nL 2222 3500 \nL 2222 3053 \nL 1275 3053 \nL 1275 0 \nL 697 0 \nL 697 3053 \nL 147 3053 \nL 147 3500 \nL 697 3500 \nL 697 3744 \nQ 697 4328 969 4595 \nQ 1241 4863 1831 4863 \nL 2375 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-49\"/>\n      <use xlink:href=\"#DejaVuSans-42\" transform=\"translate(29.492188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(98.095703 0)\"/>\n      <use xlink:href=\"#DejaVuSans-66\" transform=\"translate(129.882812 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(165.087891 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(203.951172 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(265.132812 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(328.511719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(367.720703 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(395.503906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(457.027344 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_33\">\n     <path d=\"M 222.851562 158.940313 \nL 232.851562 158.940313 \nL 242.851562 158.940313 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_21\">\n     <!-- I(X;Y) ceiling -->\n     <g transform=\"translate(250.851562 162.440313) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \nL 3122 2828 \nQ 2878 2963 2633 3030 \nQ 2388 3097 2138 3097 \nQ 1578 3097 1268 2742 \nQ 959 2388 959 1747 \nQ 959 1106 1268 751 \nQ 1578 397 2138 397 \nQ 2388 397 2633 464 \nQ 2878 531 3122 666 \nL 3122 134 \nQ 2881 22 2623 -34 \nQ 2366 -91 2075 -91 \nQ 1284 -91 818 406 \nQ 353 903 353 1747 \nQ 353 2603 823 3093 \nQ 1294 3584 2113 3584 \nQ 2378 3584 2631 3529 \nQ 2884 3475 3122 3366 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \nQ 2906 2416 2648 2759 \nQ 2391 3103 1925 3103 \nQ 1463 3103 1205 2759 \nQ 947 2416 947 1791 \nQ 947 1169 1205 825 \nQ 1463 481 1925 481 \nQ 2391 481 2648 825 \nQ 2906 1169 2906 1791 \nz\nM 3481 434 \nQ 3481 -459 3084 -895 \nQ 2688 -1331 1869 -1331 \nQ 1566 -1331 1297 -1286 \nQ 1028 -1241 775 -1147 \nL 775 -588 \nQ 1028 -725 1275 -790 \nQ 1522 -856 1778 -856 \nQ 2344 -856 2625 -561 \nQ 2906 -266 2906 331 \nL 2906 616 \nQ 2728 306 2450 153 \nQ 2172 0 1784 0 \nQ 1141 0 747 490 \nQ 353 981 353 1791 \nQ 353 2603 747 3093 \nQ 1141 3584 1784 3584 \nQ 2172 3584 2450 3431 \nQ 2728 3278 2906 2969 \nL 2906 3500 \nL 3481 3500 \nL 3481 434 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-49\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(29.492188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-58\" transform=\"translate(68.505859 0)\"/>\n      <use xlink:href=\"#DejaVuSans-3b\" transform=\"translate(137.011719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-59\" transform=\"translate(170.703125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(231.787109 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(270.800781 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(302.587891 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(357.568359 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(419.091797 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(446.875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(474.658203 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(502.441406 0)\"/>\n      <use xlink:href=\"#DejaVuSans-67\" transform=\"translate(565.820312 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p39173d56ab\">\n   <rect x=\"43.78125\" y=\"7.2\" width=\"279\" height=\"166.32\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-mutual-information-md-13 fw=all -->\n",
    "\n",
    "The frontier rises steeply from the origin (the first fraction of a nat of\n",
    "retained input information buys label information at the best exchange rate,\n",
    "$\\rho^2$ nats per nat), then flattens as $I(Y;Z)$ approaches its DPI ceiling\n",
    "$I(X;Y)$: keeping ever more of $X$ buys less and less about $Y$. The marked\n",
    "points show the $\\beta$ knob at work, including the degenerate regime: for\n",
    "$\\beta \\leq 1/\\rho^2 \\approx 1.23$ the exchange rate never beats the price\n",
    "and the optimum is total collapse, $I(X;Z) = I(Y;Z) = 0$; beyond it, rising\n",
    "$\\beta$ slides the operating point up the frontier. A deep VIB does exactly\n",
    "this on real data (encoder noise instead of $\\sigma$, variational bounds\n",
    "instead of closed forms), and its $\\beta$-sweep traces a noisy version of\n",
    "the same curve.\n",
    "\n",
    "### The Compression-Phase Debate\n",
    "\n",
    "The information plane also hosts one of deep learning's livelier disputes.\n",
    "@Shwartz-Ziv.Tishby.2017 plotted estimated $(I(X;Z), I(Y;Z))$\n",
    "trajectories of ordinary classifiers *during training* and reported a\n",
    "two-phase dynamic, a fast *fitting* phase (both informations rise) followed\n",
    "by a long *compression* phase ($I(X;Z)$ falls while $I(Y;Z)$ holds),\n",
    "proposed as the mechanism by which deep networks generalize.\n",
    "@Saxe.Bansal.Dapello.ea.2018 re-examined the claim and found the\n",
    "compression phase to be largely an artifact: it appears with saturating\n",
    "activations (tanh squashes activations into the bins' edges, which the\n",
    "binned MI estimator reads as compression) and disappears with ReLU networks,\n",
    "which generalize just as well without any measured compression. Worse, for\n",
    "a *deterministic* network with continuous inputs, $I(X; Z)$ is in fact\n",
    "infinite or constant (the singular case flagged at the definition\n",
    "the equation): whatever the plotted trajectories track, it is\n",
    "a property of the binning, not a mutual information. Where the debate\n",
    "stands: the\n",
    "IB *objective* the equation is well-posed for explicitly\n",
    "stochastic encoders and trains well (VIB improves robustness and\n",
    "calibration in practice), but the claim that standard training *implicitly*\n",
    "performs IB compression remains unsettled, and the dispute itself is a case\n",
    "study in how treacherous MI estimation is, which is precisely this\n",
    "section's second theme.\n",
    "\n",
    "### What Mutual Information Guarantees: Fano's Inequality\n",
    "\n",
    "One guarantee explains why anyone wants high mutual information in the\n",
    "first place: enough shared information is a *prerequisite* for accurate\n",
    "prediction.\n",
    "\n",
    "**Proposition (Fano's inequality).** *Let $X$ take values in a set of size\n",
    "$k$, and let $\\hat{X} = g(Y)$ be any estimate of $X$ computed from $Y$, with\n",
    "error probability $P_e = P(\\hat{X} \\neq X)$. Then*\n",
    "\n",
    "$$H_b(P_e) + P_e \\log(k - 1) \\geq H(X \\mid Y) = H(X) - I(X; Y),$$\n",
    "\n",
    "*where $H_b$ is the binary entropy in nats* [@Cover.Thomas.1999].\n",
    "\n",
    "**Proof.** Let $E = \\mathbf{1}_{\\hat{X} \\neq X}$ be the error indicator and\n",
    "expand $H(E, X \\mid \\hat{X})$ by the chain rule in both orders. One order:\n",
    "$H(E, X \\mid \\hat{X}) = H(X \\mid \\hat{X})$, since $E$ is a function of\n",
    "$(X, \\hat{X})$. The other:\n",
    "$H(E, X \\mid \\hat{X}) = H(E \\mid \\hat{X}) + H(X \\mid E, \\hat{X})\n",
    "\\leq H_b(P_e) + P_e \\log(k-1)$, because conditioning cannot raise entropy\n",
    "($H(E \\mid \\hat{X}) \\leq H(E) = H_b(P_e)$, since $I(E; \\hat{X}) \\geq 0$),\n",
    "and given $E$: when $E = 0$, $X = \\hat{X}$ is known (zero entropy), and\n",
    "when $E = 1$ (probability $P_e$), $X$ ranges over at most $k - 1$ values.\n",
    "Finally $X \\to Y \\to \\hat{X}$ is a Markov chain, so the data-processing\n",
    "inequality gives $I(X; \\hat{X}) \\leq I(X; Y)$, i.e.,\n",
    "$H(X \\mid \\hat{X}) \\geq H(X \\mid Y)$. Chaining the three displays proves the\n",
    "claim. $\\blacksquare$\n",
    "\n",
    "Fano converts information into a floor on achievable error: no\n",
    "representation, classifier, or amount of compute can beat it. It quantifies\n",
    "the \"prerequisite\" reading of contrastive learning: features that support\n",
    "accurate downstream classification *must* carry high mutual information with\n",
    "the label. The numbers are sobering when combined with this section's\n",
    "estimation ceiling:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "mutual-information-fano"
   },
   "source": [
    "k, p_e = 1000, 0.05\n",
    "H_b = lambda p: -p * onp.log(p) - (1 - p) * onp.log(1 - p)\n",
    "need = onp.log(k) - H_b(p_e) - p_e * onp.log(k - 1)\n",
    "print(f'{p_e:.0%} error on {k} balanced classes needs '\n",
    "      f'I(X;Y) >= {need:.3f} nats')\n",
    "print(f'certifying that much MI needs N > e^I = {onp.exp(need):.0f} samples')"
   ],
   "execution_count": 13,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "5% error on 1000 balanced classes needs I(X;Y) >= 6.364 nats\ncertifying that much MI needs N > e^I = 581 samples\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=mdl-mutual-information-md-14 fw=all -->\n",
    "\n",
    "Five percent error on a thousand balanced classes requires at least\n",
    "$6.36$ nats of label information; taking the $\\log N$ ceiling at face\n",
    "value, *certifying* that much mutual information needs\n",
    "$N \\gtrsim e^{I} \\approx 581$ samples per batch. The\n",
    "two halves of this section are one subject.\n",
    "\n",
    "### What Mutual Information Estimates Can and Cannot Tell You\n",
    "\n",
    "Let's consolidate the section's findings into usage guidance, because mutual\n",
    "information is at once indispensable and routinely over-read.\n",
    "\n",
    "* **A neural bound is not a direct measurement.** The high-dimensional neural\n",
    "  estimators discussed above (InfoNCE, NWJ, DV/MINE, and decoder bounds) are\n",
    "  variational bounds estimated from batches and often optimized with imperfect\n",
    "  critics: noisy estimates of bounds. Histogram, kernel, and nearest-neighbor\n",
    "  estimators are not variational lower bounds, although they have their own\n",
    "  dimensionality, bias, and tuning problems. The $\\log N$\n",
    "  ceiling caps what any of them can certify, and past the ceiling an\n",
    "  estimator returns either a saturated number (InfoNCE) or an exploding one\n",
    "  (DV/NWJ).\n",
    "* **The objective can be good even when the number is bad.**\n",
    "  @Tschannen.Djolonga.Rubenstein.ea.2020 show that downstream\n",
    "  representation quality correlates poorly with the tightness of the MI\n",
    "  bound being maximized: looser bounds sometimes yield *better*\n",
    "  representations, and InfoNCE's success owes as much to the inductive\n",
    "  biases of the critic architecture and the choice of views as to its\n",
    "  information content. Maximizing an MI bound is a useful *training signal*\n",
    "  wrapped in a partly-aesthetic theory, and the wrapping should not be\n",
    "  mistaken for a guarantee.\n",
    "* **Invariances cut both ways.** MI's reparameterization invariance makes it\n",
    "  the right notion of dependence and simultaneously the wrong target for\n",
    "  easy estimation; any pipeline that needs a trustworthy dependence number\n",
    "  on high-dimensional data should confront that tension directly (e.g., by\n",
    "  validating its estimator on a known-MI synthetic pair, as we did here).\n",
    "* **What survives the caveats** is exactly what we proved: the definitions\n",
    "  and their calculus, the data-processing inequality, Fano's floor, the\n",
    "  bounds' validity as *bounds*, and the closed-form anchors used to test\n",
    "  estimators. Build on those, and treat everything else as engineering.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Mutual information $I(X;Y) = D_{\\textrm{KL}}(P_{X,Y} \\| P_X \\otimes P_Y)$\n",
    "  measures how far a pair of variables is from independence. It is\n",
    "  symmetric, non-negative, zero exactly at independence, equals\n",
    "  $H(X) - H(X \\mid Y) = H(X) + H(Y) - H(X,Y)$, and is invariant under\n",
    "  invertible reparameterization of either variable; unlike correlation, it\n",
    "  detects *any* dependence.\n",
    "* The Gaussian pair gives the closed-form anchor\n",
    "  $I = -\\tfrac{1}{2}\\log(1 - \\rho^2)$; pointwise mutual information scores\n",
    "  individual co-occurrences and powers collocation mining in NLP.\n",
    "* The chain rule $I(X; Y, Z) = I(X;Z) + I(X; Y \\mid Z)$ yields the\n",
    "  data-processing inequality: along a Markov chain $X \\to Y \\to Z$,\n",
    "  $I(X;Z) \\leq I(X;Y)$. Processing never creates information;\n",
    "  representations only lose it.\n",
    "* Estimating MI is statistically hard: any distribution-free,\n",
    "  high-confidence lower bound from $N$ samples is capped near $\\log N$\n",
    "  nats (McAllester--Stratos), a ceiling that even a perfect critic hits.\n",
    "* Practical estimators are variational lower bounds: Barber--Agakov\n",
    "  (decoder), Donsker--Varadhan/MINE and NWJ (critics), and InfoNCE\n",
    "  (contrastive classification, a categorical cross-entropy). InfoNCE\n",
    "  certifies $I \\geq \\log N - \\mathcal{L}$, is low-variance but saturates at\n",
    "  $\\log N$; DV/NWJ are tighter in expectation but their variance explodes\n",
    "  at high MI.\n",
    "* The information bottleneck $\\min I(X;Z) - \\beta\\, I(Y;Z)$ frames good\n",
    "  representation as a compression--prediction tradeoff, made trainable by\n",
    "  the same variational bounds (VIB); the claim that ordinary training\n",
    "  implicitly compresses is contested (Saxe et al.). Fano's inequality is\n",
    "  the guarantee that survives: low prediction error *requires* high mutual\n",
    "  information.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Prove the discrete bounds $\\max\\{H(X), H(Y)\\} \\leq H(X, Y) \\leq\n",
    "   H(X) + H(Y)$ from the chain rule and the properties of (conditional)\n",
    "   entropy. Which inequality fails for differential entropy, and why?\n",
    "1. Compute $I(X;Y)$ by hand for the joint p.m.f.\n",
    "   $p_{X,Y} = \\left(\\begin{smallmatrix} 0.3 & 0.2 \\\\ 0.1 & 0.4\n",
    "   \\end{smallmatrix}\\right)$: form the marginals, evaluate the pointwise\n",
    "   mutual information at each cell, and average. Check your answer against\n",
    "   the `mutual_information` function.\n",
    "1. Derive the Gaussian formula the equation directly by\n",
    "   integrating the equation (without the entropy\n",
    "   identities). How large must $\\rho$ be for $I(X;Y)$ to exceed the\n",
    "   certification ceiling of a batch of $N = 256$?\n",
    "1. Construct a pair of *discrete* random variables with zero correlation\n",
    "   but positive mutual information, and verify both numerically. Then prove\n",
    "   that for jointly Gaussian variables, zero correlation does imply\n",
    "   $I = 0$.\n",
    "1. (Data-processing) Let $X \\to Y \\to Z$ with a *deterministic* second step\n",
    "   $Z = g(Y)$. Show $I(X; g(Y)) \\leq I(X; Y)$ with equality when $g$ is\n",
    "   invertible, and give an example where $g$ is not invertible yet equality\n",
    "   still holds. Can post-processing by any randomized $g$ ever increase\n",
    "   mutual information?\n",
    "1. (Synergy) Conditioning can *create* dependence: let $X, Y$ be\n",
    "   independent fair bits and $Z = X \\oplus Y$ (exclusive or). Show\n",
    "   $I(X; Y) = 0$ but $I(X; Y \\mid Z) = \\ln 2$. Why does this not contradict\n",
    "   the data-processing inequality?\n",
    "1. Derive the Barber--Agakov bound the equation from\n",
    "   $I = H(X) - H(X \\mid Y)$ and Gibbs' inequality, and show that the gap of\n",
    "   the bound equals $E_y[D_{\\textrm{KL}}(p(\\cdot \\mid y) \\| q(\\cdot \\mid\n",
    "   y))]$.\n",
    "1. Show from the equation that\n",
    "   $\\mathcal{L}_{\\textrm{NCE}} \\geq 0$, conclude that the InfoNCE estimator\n",
    "   the equation can never exceed $\\log N$, and explain in one paragraph\n",
    "   why increasing the batch size tightens the bound. What does the optimal\n",
    "   critic look like?\n",
    "1. For the scalar Gaussian bottleneck, derive the critical value\n",
    "   $\\beta^* = 1/\\rho^2$ below which the IB optimum collapses to an\n",
    "   uninformative $Z$, by examining the slope of the frontier at\n",
    "   $I(X;Z) = 0$. Characterize the optimal $Z$ as $\\beta \\to \\infty$.\n",
    "1. Using Fano's inequality the equation and the $\\log N$ ceiling\n",
    "   taken at face value, estimate the smallest contrastive batch size that\n",
    "   could in principle certify enough mutual information for $1\\%$ error on\n",
    "   a balanced $10{,}000$-class problem. Then design (and run) a sanity check for any\n",
    "   MI estimator using a known-MI Gaussian pair."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-mxnet",
   "display_name": "d2l (MXNet)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
