{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-1 fw=all -->\n",
    "\n",
    "# Linear Recurrence and State Space Models\n",
    "\n",
    "The gated cells of that section solved the memory problem and promptly\n",
    "ran into a compute problem. A recurrent network must consume its input one\n",
    "step at a time: $\\mathbf{H}_t$ cannot be computed before\n",
    "$\\mathbf{H}_{t-1}$ exists, so a sequence of length $T$ costs $T$\n",
    "*sequential* rounds of work no matter how many processors we own. Modern\n",
    "accelerators are throughput machines, built to run tens of thousands of\n",
    "threads at once, and a recurrence at inference batch sizes leaves nearly all\n",
    "of them idle. Compare the convolutional networks of that section: a\n",
    "convolution applies the same weights at every position, just as a recurrence\n",
    "applies the same cell at every step, but all positions are computed *at\n",
    "once*. That difference in training throughput, far more than any difference\n",
    "in modeling power, is what pushed recurrent networks out of large-scale use.\n",
    "\n",
    "This section is about getting the parallelism back without giving up the\n",
    "thing that makes recurrence attractive: a state of fixed size, updated in\n",
    "constant time per token at inference (that section).\n",
    "The obstacle turns out to be the nonlinearity wrapped around the state\n",
    "update, and the section removes it in two passes. First we strip the GRU\n",
    "down to a minimal gated cell whose update is *linear* in the state, which\n",
    "makes the whole sequence computable by a parallel *scan* in logarithmic\n",
    "depth. Then we rebuild the linear recurrence properly, starting from a\n",
    "continuous-time *state space model* (SSM): discretization will hand us\n",
    "principled gates, stability by construction rather than by hope, an exact\n",
    "equivalence between recurrence and convolution, and, through the HiPPO\n",
    "theory, a principled answer to what dynamics make a fixed-size state a\n",
    "good memory of the past. The result, the S4 family of models\n",
    "[@Gu.Goel.Re.2022], is the backbone of the selective state space\n",
    "models we meet in the next section and of recurrent layers inside several\n",
    "production language models.\n",
    "\n",
    "*Prerequisites: the gated cells and language-modeling recipe of\n",
    "that section, the gradient analysis of\n",
    "that section, and, for the classifier\n",
    "experiment, Fashion-MNIST from that section.*"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-linear-recurrence-and-state-space-models"
   },
   "source": [
    "%matplotlib inline\n",
    "from d2l import torch as d2l\n",
    "import math\n",
    "import numpy as np\n",
    "import time\n",
    "import torch\n",
    "from torch import nn"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-2 fw=all -->\n",
    "\n",
    "## Linearizing the Recurrence\n",
    "\n",
    "Why exactly can a recurrence not be parallelized? Take the vanilla update\n",
    "$\\mathbf{H}_t = \\phi(\\mathbf{X}_t \\mathbf{W}_{\\textrm{xh}} +\n",
    "\\mathbf{H}_{t-1} \\mathbf{W}_{\\textrm{hh}} + \\mathbf{b})$ and try to jump\n",
    "ahead two steps: substituting the update into itself buries\n",
    "$\\mathbf{H}_{t-2}$ under two nested applications of $\\phi$, and nothing\n",
    "simplifies. A composition of nonlinear maps has no compact closed form, so\n",
    "the only way to evaluate step $t$ is to have evaluated step $t-1$. The\n",
    "nonlinearity in the *state path* is the exact culprit. Everything else\n",
    "about the cell, gates computed from the input, nonlinear read-outs of the\n",
    "state, deep stacks of layers, creates no such obstruction.\n",
    "\n",
    "That observation suggests a surgical experiment: keep the gating idea of\n",
    "that section, but remove every appearance of $\\mathbf{H}_{t-1}$\n",
    "inside a nonlinearity. Recall the GRU update, the equation\n",
    "through the equation: two gates and a candidate, each a function of\n",
    "$\\mathbf{X}_t$ *and* $\\mathbf{H}_{t-1}$, followed by a convex blend.\n",
    "@Feng.Tung.Ahmed.ea.2024 asked how much of that machinery is\n",
    "load-bearing and arrived at the *minGRU* by making two cuts. First, the\n",
    "update gate is computed from the input alone. Second, the candidate state\n",
    "is computed from the input alone, which leaves the reset gate with nothing\n",
    "to reset, so it is deleted outright. What survives is\n",
    "\n",
    "$$\n",
    "\\mathbf{Z}_t = \\sigma(\\mathbf{X}_t \\mathbf{W}_{\\textrm{xz}} + \\mathbf{b}_\\textrm{z}),\n",
    "\\qquad\n",
    "\\tilde{\\mathbf{H}}_t = \\tanh(\\mathbf{X}_t \\mathbf{W}_{\\textrm{xh}} + \\mathbf{b}_\\textrm{h}),\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\mathbf{H}_t = (1 - \\mathbf{Z}_t) \\odot \\mathbf{H}_{t-1} + \\mathbf{Z}_t \\odot \\tilde{\\mathbf{H}}_t.\n",
    "$$\n",
    "\n",
    "Here $\\mathbf{Z}_t$ gates the write, so $\\mathbf{Z}_t \\approx 0$ means\n",
    "\"hold the state\"; substituting $\\mathbf{Z}_t \\leftarrow 1 - \\mathbf{Z}_t$\n",
    "recovers the orientation of the equation. (Feng et al. also drop the\n",
    "$\\tanh$ on the candidate; we keep it because a bounded candidate trains\n",
    "more stably under the plain SGD recipe we reuse below, and an exercise asks\n",
    "you to remove it.) The cell looks almost unchanged, but its character is\n",
    "transformed: for fixed inputs, the equation is an *affine* function\n",
    "of $\\mathbf{H}_{t-1}$,\n",
    "\n",
    "$$\n",
    "\\mathbf{H}_t = \\mathbf{a}_t \\odot \\mathbf{H}_{t-1} + \\mathbf{b}_t,\n",
    "\\qquad\n",
    "\\mathbf{a}_t = 1 - \\mathbf{Z}_t,\n",
    "\\quad\n",
    "\\mathbf{b}_t = \\mathbf{Z}_t \\odot \\tilde{\\mathbf{H}}_t,\n",
    "$$\n",
    "\n",
    "where every coefficient $\\mathbf{a}_t, \\mathbf{b}_t$ depends only on\n",
    "$\\mathbf{X}_t$ and hence can be computed for *all* $t$ simultaneously, in\n",
    "one batched matrix product. Two consequences follow, and they organize the\n",
    "rest of this section. Parallelism: affine maps compose into affine maps,\n",
    "so the $T$-step recurrence can be evaluated by a divide-and-conquer scheme\n",
    "rather than a left-to-right walk. Analyzable memory: unrolling\n",
    "the equation shows that the input from $k$ steps ago\n",
    "reaches $\\mathbf{H}_t$ scaled by exactly\n",
    "$\\prod_{j=t-k+1}^{t} \\mathbf{a}_j$, a product of gate values you can print,\n",
    "plot, and reason about. Because $\\mathbf{a}_t = 1 - \\mathbf{Z}_t \\in (0, 1)$\n",
    "elementwise, that product can never explode, and how fast it decays is a\n",
    "learned, per-unit property: the eigenvalue analysis of\n",
    "that section applies with the eigenvalues in\n",
    "plain sight. Linear recurrent units (LRUs) push the same idea further with\n",
    "complex-valued decays and careful initialization, and match well-tuned\n",
    "nonlinear RNNs on long-range benchmarks [@Orvieto.Smith.Gu.ea.2023].\n",
    "\n",
    "Does linearity in the state cost us expressivity? Within a single layer,\n",
    "yes: a linear state can only accumulate and forget, not compute. The\n",
    "answer, as everywhere else in deep learning, is depth: nonlinearities\n",
    "remain between layers, where they do not obstruct parallelism, and the\n",
    "stack recovers the lost capacity. We will see the pattern concretely in\n",
    "the S4D block near the end of this section.\n",
    "\n",
    "## Parallel Scans\n",
    "\n",
    "### An Associative Combine\n",
    "\n",
    "The classic *prefix sum* problem asks for all running totals\n",
    "$s_t = x_1 + \\cdots + x_t$ of a sequence. It looks as sequential as our\n",
    "recurrence, yet it parallelizes beautifully, because addition is\n",
    "*associative*: sums may be grouped arbitrarily, so partial sums of blocks\n",
    "can be computed independently and merged. The general result, due to\n",
    "@Blelloch.1990, is that a running \"total\" under *any* associative\n",
    "binary operator can be computed in $O(\\log T)$ parallel rounds; the\n",
    "operation is called a *parallel scan*, and it is exactly as general as it\n",
    "sounds.\n",
    "\n",
    "Our recurrence fits once we find the right operator. Represent the affine\n",
    "update at step $t$ by its coefficient pair $(\\mathbf{a}_t, \\mathbf{b}_t)$,\n",
    "standing for the map $\\mathbf{h} \\mapsto \\mathbf{a}_t \\odot \\mathbf{h} +\n",
    "\\mathbf{b}_t$. Applying $(\\mathbf{a}_1, \\mathbf{b}_1)$ first and\n",
    "$(\\mathbf{a}_2, \\mathbf{b}_2)$ second gives\n",
    "$\\mathbf{a}_2 \\odot (\\mathbf{a}_1 \\odot \\mathbf{h} + \\mathbf{b}_1) +\n",
    "\\mathbf{b}_2$, which is again affine:\n",
    "\n",
    "$$\n",
    "(\\mathbf{a}_2, \\mathbf{b}_2) \\circ (\\mathbf{a}_1, \\mathbf{b}_1)\n",
    "= (\\mathbf{a}_1 \\odot \\mathbf{a}_2,\\ \\mathbf{a}_2 \\odot \\mathbf{b}_1 + \\mathbf{b}_2).\n",
    "$$\n",
    "\n",
    "The operator $\\circ$ is associative because function composition is\n",
    "associative, and composing the first $t$ pairs yields the map that carries\n",
    "$\\mathbf{h}_0$ all the way to $\\mathbf{h}_t$; with\n",
    "$\\mathbf{h}_0 = \\mathbf{0}$ the state is simply the $\\mathbf{b}$-component\n",
    "of that composite. So the entire trajectory\n",
    "$\\mathbf{h}_1, \\ldots, \\mathbf{h}_T$ is the prefix scan of\n",
    "$(\\mathbf{a}_1, \\mathbf{b}_1), \\ldots, (\\mathbf{a}_T, \\mathbf{b}_T)$\n",
    "under the equation.\n",
    "\n",
    "the figure shows the parallel schedule we will implement, on\n",
    "eight elements. In round one, every position combines with its immediate\n",
    "predecessor; in round two, with the position two back; then four back, and\n",
    "so on, doubling the stride. After round $r$ position $t$ holds the\n",
    "composite of elements $\\max(1, t - 2^r + 1)$ through $t$, so after\n",
    "$\\lceil \\log_2 T \\rceil$ rounds every position holds its full prefix. Each\n",
    "round is one batched elementwise multiply-add over the whole sequence,\n",
    "perfect work for an accelerator.\n",
    "\n",
    "![A parallel prefix scan on eight elements. Each round combines every position with the one a fixed stride back (arrows), doubling the stride each round; dashed arrows copy unchanged values. Labels show which input elements each node has absorbed; after $\\log_2 8 = 3$ rounds, every position holds its full prefix (green).](../img/mdl-modernrnn-scan-tree.svg)\n",
    "\n",
    "This doubling schedule, the scheme of Hillis and Steele, performs\n",
    "$O(T \\log T)$ total work, a $\\log T$ factor more than the sequential loop,\n",
    "in exchange for $O(\\log T)$ depth.\n",
    "Blelloch's classic two-sweep scan gets the work back down to $O(T)$ at\n",
    "slightly more than twice the depth; production kernels use it, and we\n",
    "happily pay the log factor for code that fits on a slide.\n",
    "\n",
    "### Implementation\n",
    "\n",
    "We implement the doubling scan for the elementwise-affine recurrence\n",
    "$\\mathbf{h}_t = \\mathbf{a}_t \\odot \\mathbf{h}_{t-1} + \\mathbf{b}_t$ with\n",
    "$\\mathbf{h}_0 = \\mathbf{0}$. Time runs along the first axis; any batch or\n",
    "feature axes ride along elementwise, and the coefficient tensors follow\n",
    "ordinary broadcasting rules, a convenience we will lean on later when\n",
    "$\\mathbf{a}$ is shared across the batch. Each iteration of the loop is one\n",
    "round of the figure: positions $t < s$ keep their values (the\n",
    "`[:s]` prefix), and every other position combines with the value at stride\n",
    "$s$ by the equation. The function is built from differentiable\n",
    "tensor ops, so gradients flow through training without further ado.\n",
    "\n",
    "We save the helper in the `d2l` library: it is the computational core of\n",
    "this section and the next."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-implementation-1"
   },
   "source": [
    "def associative_scan(a, b, dim=0):\n",
    "    \"\"\"Parallel prefix scan for h_t = a_t * h_{t-1} + b_t with h_0 = 0.\"\"\"\n",
    "    a, b = a.movedim(dim, 0), b.movedim(dim, 0)\n",
    "    step = 1\n",
    "    while step < b.shape[0]:  # ceil(log2 T) rounds of combines\n",
    "        a_prev, b_prev = a[:-step], b[:-step]\n",
    "        a, b = (torch.cat([a[:step], a_prev * a[step:]]),\n",
    "                torch.cat([b[:step], a[step:] * b_prev + b[step:]]))\n",
    "        step *= 2\n",
    "    return b.movedim(0, dim)"
   ],
   "execution_count": 2,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-3 fw=all -->\n",
    "\n",
    "Correctness first: on random coefficients (decays $\\mathbf{a}_t$ drawn in\n",
    "$(0, 1)$, a deliberately awkward length that is not a power of two), the\n",
    "scan must agree with the sequential loop it replaces."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-implementation-2"
   },
   "source": [
    "def sequential_scan(a, b):\n",
    "    h, outputs = d2l.zeros_like(b[0]), []\n",
    "    for t in range(b.shape[0]):\n",
    "        h = a[t] * h + b[t]\n",
    "        outputs.append(h)\n",
    "    return d2l.stack(outputs, 0)\n",
    "\n",
    "a, b = torch.rand(100, 4, 8), torch.randn(100, 4, 8)\n",
    "err = (associative_scan(a, b) - sequential_scan(a, b)).abs().max()\n",
    "print(f'maximum deviation: {float(err):.2e}')"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "maximum deviation: 4.77e-07\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-4 fw=all -->\n",
    "\n",
    "Now the payoff. We time the parallel scan against the sequential\n",
    "evaluation as the sequence grows from 256 to 16,384 steps, at a fixed\n",
    "batch of 32 sequences with 128 units."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-implementation-3"
   },
   "source": [
    "device = d2l.try_gpu()\n",
    "\n",
    "def wall_clock(f, *args, reps=3):\n",
    "    f(*args)  # Warm up\n",
    "    if device.type == 'cuda':\n",
    "        torch.cuda.synchronize()\n",
    "    start = time.time()\n",
    "    for _ in range(reps):\n",
    "        f(*args)\n",
    "    if device.type == 'cuda':\n",
    "        torch.cuda.synchronize()\n",
    "    return (time.time() - start) / reps\n",
    "\n",
    "lengths, times = [256, 1024, 4096, 16384], [[], []]\n",
    "for T in lengths:\n",
    "    a = torch.rand(T, 32, 128, device=device)\n",
    "    b = torch.randn(T, 32, 128, device=device)\n",
    "    times[0].append(wall_clock(associative_scan, a, b) * 1e3)\n",
    "    times[1].append(wall_clock(sequential_scan, a, b) * 1e3)\n",
    "d2l.plot(lengths, times, 'sequence length', 'time per call (ms)',\n",
    "         legend=['parallel scan', 'sequential loop'],\n",
    "         xscale='log', yscale='log', figsize=(5, 3))"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"331.678125pt\" height=\"211.07625pt\" viewBox=\"0 0 331.678125 211.07625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-20T00:58:22.290668</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M -0 211.07625 \nL 331.678125 211.07625 \nL 331.678125 0 \nL -0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 45.478125 173.52 \nL 324.478125 173.52 \nL 324.478125 7.2 \nL 45.478125 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path d=\"M 141.259006 173.52 \nL 141.259006 7.2 \n\" clip-path=\"url(#pb5a266a4f3)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path id=\"me555aa1801\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#me555aa1801\" x=\"141.259006\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- $\\mathdefault{10^{3}}$ -->\n      <g transform=\"translate(132.459006 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \nQ 3050 2419 3304 2112 \nQ 3559 1806 3559 1356 \nQ 3559 666 3084 287 \nQ 2609 -91 1734 -91 \nQ 1441 -91 1130 -33 \nQ 819 25 488 141 \nL 488 750 \nQ 750 597 1062 519 \nQ 1375 441 1716 441 \nQ 2309 441 2620 675 \nQ 2931 909 2931 1356 \nQ 2931 1769 2642 2001 \nQ 2353 2234 1838 2234 \nL 1294 2234 \nL 1294 2753 \nL 1863 2753 \nQ 2328 2753 2575 2939 \nQ 2822 3125 2822 3475 \nQ 2822 3834 2567 4026 \nQ 2313 4219 1838 4219 \nQ 1578 4219 1281 4162 \nQ 984 4106 628 3988 \nL 628 4550 \nQ 988 4650 1302 4700 \nQ 1616 4750 1894 4750 \nQ 2613 4750 3031 4423 \nQ 3450 4097 3450 3541 \nQ 3450 3153 3228 2886 \nQ 3006 2619 2597 2516 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(128.203125 39.046875) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path d=\"M 281.685966 173.52 \nL 281.685966 7.2 \n\" clip-path=\"url(#pb5a266a4f3)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#me555aa1801\" x=\"281.685966\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- $\\mathdefault{10^{4}}$ -->\n      <g transform=\"translate(272.885966 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n       <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(128.203125 38.965625) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <defs>\n       <path id=\"m050af7d1ea\" d=\"M 0 0 \nL 0 2 \n\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"67.832733\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"85.3775\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_7\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"98.986279\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"110.105461\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_7\">\n     <g id=\"line2d_9\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"119.506595\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_8\">\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"127.650228\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_9\">\n     <g id=\"line2d_11\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"134.833421\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_10\">\n     <g id=\"line2d_12\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"183.531733\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_11\">\n     <g id=\"line2d_13\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"208.259694\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_12\">\n     <g id=\"line2d_14\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"225.804461\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_13\">\n     <g id=\"line2d_15\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"239.413239\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_14\">\n     <g id=\"line2d_16\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"250.532421\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_15\">\n     <g id=\"line2d_17\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"259.933555\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_16\">\n     <g id=\"line2d_18\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"268.077188\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_17\">\n     <g id=\"line2d_19\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"275.260381\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_18\">\n     <g id=\"line2d_20\">\n      <g>\n       <use xlink:href=\"#m050af7d1ea\" x=\"323.958694\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_3\">\n     <!-- sequence length -->\n     <g transform=\"translate(143.35625 201.796563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-71\" d=\"M 947 1747 \nQ 947 1113 1208 752 \nQ 1469 391 1925 391 \nQ 2381 391 2643 752 \nQ 2906 1113 2906 1747 \nQ 2906 2381 2643 2742 \nQ 2381 3103 1925 3103 \nQ 1469 3103 1208 2742 \nQ 947 2381 947 1747 \nz\nM 2906 525 \nQ 2725 213 2448 61 \nQ 2172 -91 1784 -91 \nQ 1150 -91 751 415 \nQ 353 922 353 1747 \nQ 353 2572 751 3078 \nQ 1150 3584 1784 3584 \nQ 2172 3584 2448 3432 \nQ 2725 3281 2906 2969 \nL 2906 3500 \nL 3481 3500 \nL 3481 -1331 \nL 2906 -1331 \nL 2906 525 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-75\" d=\"M 544 1381 \nL 544 3500 \nL 1119 3500 \nL 1119 1403 \nQ 1119 906 1312 657 \nQ 1506 409 1894 409 \nQ 2359 409 2629 706 \nQ 2900 1003 2900 1516 \nL 2900 3500 \nL 3475 3500 \nL 3475 0 \nL 2900 0 \nL 2900 538 \nQ 2691 219 2414 64 \nQ 2138 -91 1772 -91 \nQ 1169 -91 856 284 \nQ 544 659 544 1381 \nz\nM 1991 3584 \nL 1991 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \nL 3122 2828 \nQ 2878 2963 2633 3030 \nQ 2388 3097 2138 3097 \nQ 1578 3097 1268 2742 \nQ 959 2388 959 1747 \nQ 959 1106 1268 751 \nQ 1578 397 2138 397 \nQ 2388 397 2633 464 \nQ 2878 531 3122 666 \nL 3122 134 \nQ 2881 22 2623 -34 \nQ 2366 -91 2075 -91 \nQ 1284 -91 818 406 \nQ 353 903 353 1747 \nQ 353 2603 823 3093 \nQ 1294 3584 2113 3584 \nQ 2378 3584 2631 3529 \nQ 2884 3475 3122 3366 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \nQ 2906 2416 2648 2759 \nQ 2391 3103 1925 3103 \nQ 1463 3103 1205 2759 \nQ 947 2416 947 1791 \nQ 947 1169 1205 825 \nQ 1463 481 1925 481 \nQ 2391 481 2648 825 \nQ 2906 1169 2906 1791 \nz\nM 3481 434 \nQ 3481 -459 3084 -895 \nQ 2688 -1331 1869 -1331 \nQ 1566 -1331 1297 -1286 \nQ 1028 -1241 775 -1147 \nL 775 -588 \nQ 1028 -725 1275 -790 \nQ 1522 -856 1778 -856 \nQ 2344 -856 2625 -561 \nQ 2906 -266 2906 331 \nL 2906 616 \nQ 2728 306 2450 153 \nQ 2172 0 1784 0 \nQ 1141 0 747 490 \nQ 353 981 353 1791 \nQ 353 2603 747 3093 \nQ 1141 3584 1784 3584 \nQ 2172 3584 2450 3431 \nQ 2728 3278 2906 2969 \nL 2906 3500 \nL 3481 3500 \nL 3481 434 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(52.099609 0)\"/>\n      <use xlink:href=\"#DejaVuSans-71\" transform=\"translate(113.623047 0)\"/>\n      <use xlink:href=\"#DejaVuSans-75\" transform=\"translate(177.099609 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(240.478516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(302.001953 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(365.380859 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(420.361328 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(481.884766 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(513.671875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(541.455078 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(602.978516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-67\" transform=\"translate(666.357422 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(729.833984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(769.042969 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_21\">\n      <path d=\"M 45.478125 153.66475 \nL 324.478125 153.66475 \n\" clip-path=\"url(#pb5a266a4f3)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_22\">\n      <defs>\n       <path id=\"mcc28c12a8a\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#mcc28c12a8a\" x=\"45.478125\" y=\"153.66475\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <g transform=\"translate(20.878125 157.463969) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_23\">\n      <path d=\"M 45.478125 99.607822 \nL 324.478125 99.607822 \n\" clip-path=\"url(#pb5a266a4f3)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_24\">\n      <g>\n       <use xlink:href=\"#mcc28c12a8a\" x=\"45.478125\" y=\"99.607822\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- $\\mathdefault{10^{1}}$ -->\n      <g transform=\"translate(20.878125 103.407041) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_25\">\n      <path d=\"M 45.478125 45.550895 \nL 324.478125 45.550895 \n\" clip-path=\"url(#pb5a266a4f3)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_26\">\n      <g>\n       <use xlink:href=\"#mcc28c12a8a\" x=\"45.478125\" y=\"45.550895\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- $\\mathdefault{10^{2}}$ -->\n      <g transform=\"translate(20.878125 49.350114) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_27\">\n      <defs>\n       <path id=\"m495154fee8\" d=\"M 0 0 \nL -2 0 \n\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"169.937506\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_28\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"165.657212\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_29\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"162.038274\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_30\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"158.903407\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_8\">\n     <g id=\"line2d_31\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"156.138259\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_9\">\n     <g id=\"line2d_32\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"137.391993\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_10\">\n     <g id=\"line2d_33\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"127.873041\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_11\">\n     <g id=\"line2d_34\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"121.119237\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_12\">\n     <g id=\"line2d_35\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"115.880579\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_13\">\n     <g id=\"line2d_36\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"111.600284\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_14\">\n     <g id=\"line2d_37\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"107.981346\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_15\">\n     <g id=\"line2d_38\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"104.84648\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_16\">\n     <g id=\"line2d_39\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"102.081332\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_17\">\n     <g id=\"line2d_40\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"83.335066\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_18\">\n     <g id=\"line2d_41\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"73.816113\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_19\">\n     <g id=\"line2d_42\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"67.062309\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_20\">\n     <g id=\"line2d_43\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"61.823652\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_21\">\n     <g id=\"line2d_44\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"57.543357\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_22\">\n     <g id=\"line2d_45\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"53.924419\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_23\">\n     <g id=\"line2d_46\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"50.789553\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_24\">\n     <g id=\"line2d_47\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"48.024404\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_25\">\n     <g id=\"line2d_48\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"29.278138\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_26\">\n     <g id=\"line2d_49\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"19.759186\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_27\">\n     <g id=\"line2d_50\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"13.005382\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_28\">\n     <g id=\"line2d_51\">\n      <g>\n       <use xlink:href=\"#m495154fee8\" x=\"45.478125\" y=\"7.766724\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- time per call (ms) -->\n     <g transform=\"translate(14.798438 134.6975) rotate(-90) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \nQ 3544 3216 3844 3400 \nQ 4144 3584 4550 3584 \nQ 5097 3584 5394 3201 \nQ 5691 2819 5691 2113 \nL 5691 0 \nL 5113 0 \nL 5113 2094 \nQ 5113 2597 4934 2840 \nQ 4756 3084 4391 3084 \nQ 3944 3084 3684 2787 \nQ 3425 2491 3425 1978 \nL 3425 0 \nL 2847 0 \nL 2847 2094 \nQ 2847 2600 2669 2842 \nQ 2491 3084 2119 3084 \nQ 1678 3084 1418 2786 \nQ 1159 2488 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1356 3278 1631 3431 \nQ 1906 3584 2284 3584 \nQ 2666 3584 2933 3390 \nQ 3200 3197 3328 2828 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \nL 1159 -1331 \nL 581 -1331 \nL 581 3500 \nL 1159 3500 \nL 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nz\nM 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \nQ 1566 4138 1362 3434 \nQ 1159 2731 1159 2009 \nQ 1159 1288 1364 580 \nQ 1569 -128 1984 -844 \nL 1484 -844 \nQ 1016 -109 783 600 \nQ 550 1309 550 2009 \nQ 550 2706 781 3412 \nQ 1013 4119 1484 4856 \nL 1984 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \nL 1013 4856 \nQ 1481 4119 1714 3412 \nQ 1947 2706 1947 2009 \nQ 1947 1309 1714 600 \nQ 1481 -109 1013 -844 \nL 513 -844 \nQ 928 -128 1133 580 \nQ 1338 1288 1338 2009 \nQ 1338 2731 1133 3434 \nQ 928 4138 513 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(39.208984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6d\" transform=\"translate(66.992188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(164.404297 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(225.927734 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(257.714844 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(321.191406 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(382.714844 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(423.828125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(455.615234 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(510.595703 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(571.875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(599.658203 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(627.441406 0)\"/>\n      <use xlink:href=\"#DejaVuSans-28\" transform=\"translate(659.228516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6d\" transform=\"translate(698.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(795.654297 0)\"/>\n      <use xlink:href=\"#DejaVuSans-29\" transform=\"translate(847.753906 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_52\">\n    <path d=\"M 58.159943 165.96 \nL 142.705398 160.349459 \nL 227.250852 100.248437 \nL 311.796307 62.946025 \n\" clip-path=\"url(#pb5a266a4f3)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_53\">\n    <path d=\"M 58.159943 112.586382 \nL 142.705398 80.087695 \nL 227.250852 47.595981 \nL 311.796307 14.76 \n\" clip-path=\"url(#pb5a266a4f3)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 45.478125 173.52 \nL 45.478125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 324.478125 173.52 \nL 324.478125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 45.478125 173.52 \nL 324.478125 173.52 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 45.478125 7.2 \nL 324.478125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 52.478125 44.55625 \nL 161.164062 44.55625 \nQ 163.164062 44.55625 163.164062 42.55625 \nL 163.164062 14.2 \nQ 163.164062 12.2 161.164062 12.2 \nL 52.478125 12.2 \nQ 50.478125 12.2 50.478125 14.2 \nL 50.478125 42.55625 \nQ 50.478125 44.55625 52.478125 44.55625 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_54\">\n     <path d=\"M 54.478125 20.298438 \nL 64.478125 20.298438 \nL 74.478125 20.298438 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_8\">\n     <!-- parallel scan -->\n     <g transform=\"translate(82.478125 23.798438) scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-70\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(63.476562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(124.755859 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(165.869141 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(227.148438 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(254.931641 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(282.714844 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(344.238281 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(372.021484 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(403.808594 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(455.908203 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(510.888672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(572.167969 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_55\">\n     <path d=\"M 54.478125 34.976562 \nL 64.478125 34.976562 \nL 74.478125 34.976562 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_9\">\n     <!-- sequential loop -->\n     <g transform=\"translate(82.478125 38.476562) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-73\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(52.099609 0)\"/>\n      <use xlink:href=\"#DejaVuSans-71\" transform=\"translate(113.623047 0)\"/>\n      <use xlink:href=\"#DejaVuSans-75\" transform=\"translate(177.099609 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(240.478516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(302.001953 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(365.380859 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(404.589844 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(432.373047 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(493.652344 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(521.435547 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(553.222656 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(581.005859 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(642.1875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(703.369141 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pb5a266a4f3\">\n   <rect x=\"45.478125\" y=\"7.2\" width=\"279\" height=\"166.32\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-5 fw=all -->\n",
    "\n",
    "The sequential curve climbs linearly with $T$, dominated by the cost of\n",
    "launching $T$ tiny dependent steps. The scan's curve stays well below it\n",
    "at every length, by roughly an order of magnitude in our runs: its\n",
    "handful of rounds are large batched operations that actually saturate the\n",
    "hardware. This plot is the\n",
    "\"trains like a CNN\" moment for recurrence, and everything else in this\n",
    "section and the next is built on top of it. Note what the scan did *not*\n",
    "change: at inference we still update\n",
    "$\\mathbf{h}_t = \\mathbf{a}_t \\odot \\mathbf{h}_{t-1} + \\mathbf{b}_t$ one\n",
    "token at a time, in constant memory, exactly like any RNN. We cash that\n",
    "claim, with a stopwatch, in that section.\n",
    "\n",
    "### A minGRU Language Model\n",
    "\n",
    "Assembling the equation through\n",
    "the equation around the scan yields a drop-in\n",
    "replacement for the recurrent cells of that section: same\n",
    "`(inputs, state)` interface, no step-by-step loop. A carried-in state\n",
    "folds into the first element, since\n",
    "$\\mathbf{h}_1 = \\mathbf{a}_1 \\odot \\mathbf{h}_0 + \\mathbf{b}_1$."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-a-mingru-language-model-1"
   },
   "source": [
    "class MinGRU(d2l.Module):\n",
    "    \"\"\"The minimal GRU: input-only gates and a linear state path.\"\"\"\n",
    "    def __init__(self, num_inputs, num_hiddens):\n",
    "        super().__init__()\n",
    "        self.save_hyperparameters()\n",
    "        self.W_xz = nn.Linear(num_inputs, num_hiddens)\n",
    "        self.W_xh = nn.Linear(num_inputs, num_hiddens)\n",
    "\n",
    "    def forward(self, inputs, H=None):\n",
    "        Z = torch.sigmoid(self.W_xz(inputs))     # (num_steps, batch, hiddens)\n",
    "        H_tilde = torch.tanh(self.W_xh(inputs))\n",
    "        a, b = 1 - Z, Z * H_tilde\n",
    "        if H is not None:  # Fold the carried-in state into the first step\n",
    "            b = torch.cat([b[:1] + a[:1] * H, b[1:]])\n",
    "        outputs = associative_scan(a, b)\n",
    "        return outputs, outputs[-1]"
   ],
   "execution_count": 5,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-6 fw=all -->\n",
    "\n",
    "Is stripping the GRU this aggressively survivable? We rerun the exact\n",
    "recipe of that section: *The Time Machine* under the 1,024-token\n",
    "BPE tokenizer, 50,000 windows of 32 tokens, embedding dimension 64, 128\n",
    "hidden units, ten epochs, gradients clipped to norm 1, so the numbers are\n",
    "directly comparable with that section's scoreboard."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-a-mingru-language-model-2"
   },
   "source": [
    "data = d2l.TimeMachine(batch_size=1024, num_steps=32,\n",
    "                       num_train=50000, num_val=5000)"
   ],
   "execution_count": 6,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-a-mingru-language-model-3"
   },
   "source": [
    "mingru = MinGRU(num_inputs=64, num_hiddens=128)\n",
    "model = d2l.RNNLM(mingru, vocab_size=len(data.vocab), lr=4)"
   ],
   "execution_count": 7,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-a-mingru-language-model-4"
   },
   "source": [
    "trainer = d2l.Trainer(max_epochs=10, gradient_clip_val=1, num_gpus=1)\n",
    "model.board.yscale = 'log'\n",
    "start = time.time()\n",
    "trainer.fit(model, data)\n",
    "fit_time = time.time() - start"
   ],
   "execution_count": 8,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"240.6625pt\" height=\"183.35625pt\" viewBox=\"0 0 240.6625 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-20T00:58:49.900295</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 183.35625 \nL 240.6625 183.35625 \nL 240.6625 0 \nL 0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 31.8 145.8 \nL 227.1 145.8 \nL 227.1 7.2 \nL 31.8 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path id=\"mffa8a7544d\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#mffa8a7544d\" x=\"31.8\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <g transform=\"translate(28.61875 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use xlink:href=\"#mffa8a7544d\" x=\"70.86\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 2 -->\n      <g transform=\"translate(67.67875 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-32\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use xlink:href=\"#mffa8a7544d\" x=\"109.92\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 4 -->\n      <g transform=\"translate(106.73875 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-34\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#mffa8a7544d\" x=\"148.98\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 6 -->\n      <g transform=\"translate(145.79875 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-36\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use xlink:href=\"#mffa8a7544d\" x=\"188.04\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 8 -->\n      <g transform=\"translate(184.85875 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-38\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#mffa8a7544d\" x=\"227.1\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 10 -->\n      <g transform=\"translate(220.7375 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- epoch -->\n     <g transform=\"translate(114.221875 174.076563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \nL 1159 -1331 \nL 581 -1331 \nL 581 3500 \nL 1159 3500 \nL 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nz\nM 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \nL 3122 2828 \nQ 2878 2963 2633 3030 \nQ 2388 3097 2138 3097 \nQ 1578 3097 1268 2742 \nQ 959 2388 959 1747 \nQ 959 1106 1268 751 \nQ 1578 397 2138 397 \nQ 2388 397 2633 464 \nQ 2878 531 3122 666 \nL 3122 134 \nQ 2881 22 2623 -34 \nQ 2366 -91 2075 -91 \nQ 1284 -91 818 406 \nQ 353 903 353 1747 \nQ 353 2603 823 3093 \nQ 1294 3584 2113 3584 \nQ 2378 3584 2631 3529 \nQ 2884 3475 3122 3366 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-65\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(61.523438 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(186.181641 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(241.162109 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_7\">\n      <defs>\n       <path id=\"m6fa93cf867\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m6fa93cf867\" x=\"31.8\" y=\"97.764749\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- $\\mathdefault{10^{2}}$ -->\n      <g transform=\"translate(7.2 101.563968) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_8\">\n      <defs>\n       <path id=\"mebeb36525b\" d=\"M 0 0 \nL -2 0 \n\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#mebeb36525b\" x=\"31.8\" y=\"144.675524\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_9\">\n      <g>\n       <use xlink:href=\"#mebeb36525b\" x=\"31.8\" y=\"133.251381\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#mebeb36525b\" x=\"31.8\" y=\"123.917176\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_11\">\n      <g>\n       <use xlink:href=\"#mebeb36525b\" x=\"31.8\" y=\"116.025218\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_12\">\n      <g>\n       <use xlink:href=\"#mebeb36525b\" x=\"31.8\" y=\"109.188893\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_13\">\n      <g>\n       <use xlink:href=\"#mebeb36525b\" x=\"31.8\" y=\"103.158827\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_8\">\n     <g id=\"line2d_14\">\n      <g>\n       <use xlink:href=\"#mebeb36525b\" x=\"31.8\" y=\"62.278117\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_9\">\n     <g id=\"line2d_15\">\n      <g>\n       <use xlink:href=\"#mebeb36525b\" x=\"31.8\" y=\"41.519768\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_10\">\n     <g id=\"line2d_16\">\n      <g>\n       <use xlink:href=\"#mebeb36525b\" x=\"31.8\" y=\"26.791485\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_11\">\n     <g id=\"line2d_17\">\n      <g>\n       <use xlink:href=\"#mebeb36525b\" x=\"31.8\" y=\"15.367342\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_18\">\n    <path d=\"M 36.383571 13.5 \nL 45.949286 45.189089 \nL 55.515 55.777636 \nL 65.080714 65.029731 \nL 74.646429 73.848941 \nL 84.212143 81.857552 \nL 93.777857 89.473724 \nL 103.343571 95.739927 \nL 112.909286 102.137757 \nL 122.475 107.328384 \nL 132.040714 112.36843 \nL 141.606429 116.454584 \nL 151.172143 120.011715 \nL 160.737857 123.959295 \nL 170.303571 127.087051 \nL 179.869286 129.783485 \nL 189.435 132.546615 \nL 199.000714 134.877467 \nL 208.566429 137.480311 \nL 218.132143 139.5 \n\" clip-path=\"url(#p33bd6a1d70)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_19\">\n    <path d=\"M 51.33 53.104926 \nL 70.86 67.150147 \nL 90.39 78.559623 \nL 109.92 86.981041 \nL 129.45 93.775593 \nL 148.98 98.106083 \nL 168.51 101.255048 \nL 188.04 103.947944 \nL 207.57 105.401351 \nL 227.1 106.951256 \n\" clip-path=\"url(#p33bd6a1d70)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #ff7f0e; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 31.8 145.8 \nL 31.8 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 227.1 145.8 \nL 227.1 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 31.8 145.8 \nL 227.1 145.8 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 31.8 7.2 \nL 227.1 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 144.35 45.1125 \nL 220.1 45.1125 \nQ 222.1 45.1125 222.1 43.1125 \nL 222.1 14.2 \nQ 222.1 12.2 220.1 12.2 \nL 144.35 12.2 \nQ 142.35 12.2 142.35 14.2 \nL 142.35 43.1125 \nQ 142.35 45.1125 144.35 45.1125 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_20\">\n     <path d=\"M 146.35 20.298438 \nL 156.35 20.298438 \nL 166.35 20.298438 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_9\">\n     <!-- train_ppl -->\n     <g transform=\"translate(174.35 23.798438) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \nL 3263 -1509 \nL -63 -1509 \nL -63 -1063 \nL 3263 -1063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(39.208984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(80.322266 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(141.601562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(169.384766 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(232.763672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(282.763672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(346.240234 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(409.716797 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_21\">\n     <path d=\"M 146.35 35.254688 \nL 156.35 35.254688 \nL 166.35 35.254688 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #ff7f0e; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_10\">\n     <!-- val_ppl -->\n     <g transform=\"translate(174.35 38.754688) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-76\" d=\"M 191 3500 \nL 800 3500 \nL 1894 563 \nL 2988 3500 \nL 3597 3500 \nL 2284 0 \nL 1503 0 \nL 191 3500 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-76\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(59.179688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(120.458984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(148.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(198.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(261.71875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(325.195312 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p33bd6a1d70\">\n   <rect x=\"31.8\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-a-mingru-language-model-5"
   },
   "source": [
    "ppl = float(model.board.data['val_ppl'][-1].y)\n",
    "pred = model.predict('the time traveller', 30, data.tokenizer, d2l.try_gpu())\n",
    "print(f'validation perplexity {ppl:.1f}, {fit_time:.0f}s wall clock')\n",
    "print(pred)"
   ],
   "execution_count": 9,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "validation perplexity 83.6, 9s wall clock\nthe time traveller. I had a\ndised. I was in the\nside, and the\nside, and the\nside, and\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-7 fw=all -->\n",
    "\n",
    "In our runs the minGRU's validation perplexity comes in a few points\n",
    "behind the GRU of that section (low-to-mid eighties versus high\n",
    "seventies to low eighties under the identical recipe), despite having\n",
    "input-only gates and roughly a quarter of the GRU's recurrent parameters,\n",
    "at $2h(d+1)$\n",
    "versus $3h(d+h+1)$ weights. The claim here is deliberately modest: not\n",
    "that the minGRU is better, but that very little of the GRU's quality is\n",
    "lost at this scale while the training computation becomes a parallel\n",
    "scan. At 32-step windows the\n",
    "sequential loop was never the bottleneck; the wall-clock argument is the\n",
    "benchmark above, whose gap keeps widening as sequences grow. What\n",
    "the stripped cell *does* give up is any dependence of the gates on the\n",
    "state: the decay $\\mathbf{a}_t$ is decided by the current input alone.\n",
    "Hold that thought; it returns as the central limitation at the end of this\n",
    "section.\n",
    "\n",
    "## State Space Models\n",
    "\n",
    "The minGRU linearized an RNN by deletion. State space models arrive at the\n",
    "same destination from the opposite direction, starting from a classical\n",
    "object with sixty years of control-theory pedigree: the continuous-time\n",
    "linear dynamical system\n",
    "\n",
    "$$\n",
    "\\dot{\\mathbf{x}}(t) = \\mathbf{A}\\, \\mathbf{x}(t) + \\mathbf{B}\\, u(t),\n",
    "\\qquad\n",
    "y(t) = \\mathbf{C}\\, \\mathbf{x}(t) + D\\, u(t),\n",
    "$$\n",
    "\n",
    "in which a scalar input signal $u(t)$ drives a hidden state\n",
    "$\\mathbf{x}(t) \\in \\mathbb{R}^N$ through the *state matrix*\n",
    "$\\mathbf{A} \\in \\mathbb{R}^{N \\times N}$ and input vector\n",
    "$\\mathbf{B} \\in \\mathbb{R}^{N}$, and a scalar output $y(t)$ reads the\n",
    "state through $\\mathbf{C} \\in \\mathbb{R}^{1 \\times N}$ plus a direct\n",
    "skip term $D u(t)$. (For vector-valued sequences we will simply run one\n",
    "such system per feature channel.)\n",
    "\n",
    "Why go continuous when our data are discrete tokens or pixels? Because\n",
    "the equation parameterizes *dynamics*, a law of motion, rather\n",
    "than a table of per-step transition weights. The law is independent of\n",
    "how finely we sample it: one extra scalar, a step size $\\Delta$, converts\n",
    "the same $(\\mathbf{A}, \\mathbf{B})$ into a recurrence at any rate, so\n",
    "resolution becomes a knob rather than something baked into the weights.\n",
    "Better still, the conversion has a closed form that will hand us, for\n",
    "free, the gate that that section had to engineer by hand.\n",
    "\n",
    "### Discretization: the Step Size is a Gate\n",
    "\n",
    "To turn the equation into a recurrence, hold the input constant\n",
    "over each sampling interval of length $\\Delta$ (a *zero-order hold*, ZOH)\n",
    "and integrate the ODE exactly across the interval. For the linear system\n",
    "this integral is explicit, giving the discrete recurrence\n",
    "\n",
    "$$\n",
    "\\mathbf{x}_t = \\bar{\\mathbf{A}}\\, \\mathbf{x}_{t-1} + \\bar{\\mathbf{B}}\\, u_t,\n",
    "\\qquad\n",
    "y_t = \\mathbf{C}\\, \\mathbf{x}_t + D\\, u_t,\n",
    "$$\n",
    "\n",
    "with\n",
    "\n",
    "$$\n",
    "\\bar{\\mathbf{A}} = \\exp(\\Delta \\mathbf{A}),\n",
    "\\qquad\n",
    "\\bar{\\mathbf{B}} = \\left( \\int_0^{\\Delta} \\exp(\\tau \\mathbf{A})\\, d\\tau \\right) \\mathbf{B},\n",
    "$$\n",
    "\n",
    "where $\\exp$ is the matrix exponential: the state decays under the flow\n",
    "for a full interval, while the held input is carried in by the flow from\n",
    "every instant $\\tau$ at which it entered. The integral is the definition;\n",
    "when $\\mathbf{A}$ is invertible it has the familiar closed form\n",
    "$\\bar{\\mathbf{B}} = \\mathbf{A}^{-1}(\\exp(\\Delta \\mathbf{A}) -\n",
    "\\mathbf{I})\\, \\mathbf{B}$, and for singular $\\mathbf{A}$ the integral\n",
    "still exists (at $\\mathbf{A} = \\mathbf{0}$ it is simply $\\Delta\n",
    "\\mathbf{B}$). (A cruder alternative, the Euler\n",
    "step $\\bar{\\mathbf{A}} \\approx \\mathbf{I} + \\Delta \\mathbf{A}$, is the\n",
    "subject of an exercise.) In practice $\\mathbf{A}$ is taken *diagonal*,\n",
    "$\\mathbf{A} = \\mathrm{diag}(a_1, \\ldots, a_N)$, which costs surprisingly\n",
    "little modeling power and makes the equation elementwise:\n",
    "\n",
    "$$\n",
    "\\bar{a}_n = e^{\\Delta a_n},\n",
    "\\qquad\n",
    "\\bar{b}_n = \\frac{e^{\\Delta a_n} - 1}{a_n}\\, b_n\n",
    "\\quad (a_n \\neq 0; \\textrm{ the limit at } a_n = 0 \\textrm{ is } \\Delta b_n).\n",
    "$$\n",
    "\n",
    "One numerical note travels with the equation: for small\n",
    "$|\\Delta a_n|$ the numerator $e^{\\Delta a_n} - 1$ cancels catastrophically\n",
    "in floating point, so code should call `expm1`, which computes\n",
    "$e^x - 1$ accurately near zero. Our implementations below do.\n",
    "\n",
    "Now make $\\Delta$ a *learnable parameter* and watch what it does.\n",
    "\n",
    "> **The step size is a gate.** As $\\Delta \\to 0$, we get\n",
    "> $\\bar{\\mathbf{A}} \\to \\mathbf{I}$ and $\\bar{\\mathbf{B}} \\to \\mathbf{0}$:\n",
    "> the state copies itself forward and ignores the input, a frozen memory.\n",
    "> As $\\Delta$ grows (with the real parts of the eigenvalues of\n",
    "> $\\mathbf{A}$ negative), $\\bar{\\mathbf{A}} \\to \\mathbf{0}$ and the state\n",
    "> is rewritten from the current input alone. A learned $\\Delta$ is\n",
    "> therefore an update gate in the sense of that section,\n",
    "> interpolating between \"remember everything\" and \"overwrite everything\",\n",
    "> except that it was not bolted on: it fell out of asking how fast a\n",
    "> continuous memory should run. This one correspondence is the bridge\n",
    "> between the gated RNNs behind us and the selective models ahead, where\n",
    "> $\\Delta$ becomes input-dependent and gating is rediscovered a third\n",
    "> time.\n",
    "\n",
    "The correspondence becomes an identity, not an analogy, in one worked\n",
    "scalar case. Discretize the\n",
    "same ODE with the *backward* Euler rule instead of the ZOH and, for\n",
    "the scalar mode $a = -1$, $b = 1$, the update becomes\n",
    "$x_t = (1 - z)\\, x_{t-1} + z\\, u_t$ with\n",
    "$z = \\Delta / (1 + \\Delta)$; store the step size in log space, as we are\n",
    "about to, and $z = \\sigma(\\log \\Delta)$ is *exactly* a sigmoid gate. A\n",
    "GRU-style gated update is the backward-Euler discretization of this\n",
    "scalar mode, with the gate's pre-activation\n",
    "playing the role of the log step size [@Gu.2023]. Keep the scope of\n",
    "the identity in view: for general matrix $\\mathbf{A}$ the backward-Euler\n",
    "transition $(\\mathbf{I} - \\Delta \\mathbf{A})^{-1}$ is matrix-valued and\n",
    "each mode carries its own decay and tied input response, so not every\n",
    "discretized SSM is a GRU-style update. The gates that\n",
    "that section engineered and the step size that calculus hands us\n",
    "here are, in this scalar case, one object seen through two\n",
    "discretization rules.\n",
    "\n",
    "ZOH and backward Euler are two entries in a standard menu. The rules\n",
    "differ in the transition they produce, in whether they preserve\n",
    "stability, and in what happens when the sampling rate changes:\n",
    "\n",
    "| Rule | $\\bar{\\mathbf{A}}$ | $\\mathrm{Re}(a) < 0$ maps to | Accuracy | Changing $\\Delta$ |\n",
    "| :-- | :-- | :-- | :-- | :-- |\n",
    "| Zero-order hold | $e^{\\Delta \\mathbf{A}}$ | $\\lvert\\bar{a}\\rvert < 1$, always | exact for piecewise-constant input | rediscretize; stays stable |\n",
    "| Bilinear (Tustin) | $(\\mathbf{I} - \\tfrac{\\Delta}{2}\\mathbf{A})^{-1}(\\mathbf{I} + \\tfrac{\\Delta}{2}\\mathbf{A})$ | $\\lvert\\bar{a}\\rvert < 1$, always | second order in $\\Delta$ | rediscretize; stays stable |\n",
    "| Forward Euler | $\\mathbf{I} + \\Delta \\mathbf{A}$ | $\\lvert 1 + \\Delta a\\rvert < 1$ only for small $\\Delta$ | first order in $\\Delta$ | a coarser $\\Delta$ can destabilize |\n",
    "| Backward Euler | $(\\mathbf{I} - \\Delta \\mathbf{A})^{-1}$ | $\\lvert\\bar{a}\\rvert < 1$, always | first order in $\\Delta$ | rediscretize; stays stable |\n",
    "\n",
    "The choice is less about accuracy than about guarantees: every rule but\n",
    "forward Euler maps the stable half-plane into the stable disk for any\n",
    "$\\Delta$. S4 used the bilinear transform [@Gu.Goel.Re.2022]; we use\n",
    "ZOH throughout this chapter, as Mamba does (with a further first-order\n",
    "simplification we meet in that section); forward Euler earns its\n",
    "place in the exercise that shows how it fails.\n",
    "\n",
    "Stability, which that section taught us to\n",
    "fear, is now a matter of *parameterization* rather than luck. Store\n",
    "$a_n$ as $-e^{\\theta_n}$ (or any form pinned to the left half-plane), and\n",
    "the discrete eigenvalues satisfy\n",
    "\n",
    "$$\n",
    "|\\bar{a}_n| = e^{\\Delta\\, \\mathrm{Re}(a_n)} < 1\n",
    "\\quad \\textrm{whenever } \\mathrm{Re}(a_n) < 0,\n",
    "$$\n",
    "\n",
    "for *every* value of the learned parameters: the spectral radius lives\n",
    "inside the unit circle *by construction*, and no gradient update can push\n",
    "the recurrence into the exploding regime. One scoping remark keeps this\n",
    "claim exact. For a *diagonal* — hence normal — $\\mathbf{A}$, the\n",
    "eigenvalue bound controls every transient too: each coordinate shrinks at\n",
    "every single step. A general non-normal matrix is slipperier, since its\n",
    "powers can grow substantially for many steps before eigenvalue decay\n",
    "wins (that section); every $\\mathbf{A}$ this\n",
    "chapter trains is diagonal, so the stronger, per-step reading applies.\n",
    "Contrast the vanilla RNN, whose\n",
    "$\\mathbf{W}_{\\textrm{hh}}$ had to be coaxed toward the knife-edge\n",
    "$\\rho \\approx 1$ by clipping and prayer. Allowing complex $a_n$\n",
    "(conjugate pairs) adds rotation to the decay: each state dimension\n",
    "becomes a damped oscillator with a learned frequency, enriching the\n",
    "dynamics at no cost to stability. For everything we train below, plain\n",
    "negative-real $a_n$ suffices.\n",
    "\n",
    "> **Four meanings of \"stable\".** The word is doing several jobs in this\n",
    "> chapter, and they are not interchangeable. *Asymptotic (internal)\n",
    "> stability*: the autonomous state decays, $\\mathbf{x}_t \\to \\mathbf{0}$;\n",
    "> for our diagonal systems this is exactly $|\\bar{a}_n| < 1$, and it is\n",
    "> what \"by construction\" above means. *BIBO stability*: bounded inputs\n",
    "> yield bounded outputs; internal stability implies it here, though the\n",
    "> two can part ways when an unstable mode hides from the output (a story\n",
    "> told below). *Nonexpansive transition*: a single step never increases\n",
    "> the state norm — a stronger per-step property that returns for the\n",
    "> delta-rule transitions of that section. *Numerical\n",
    "> stability*: the floating-point computation neither loses accuracy nor\n",
    "> overflows; a mathematically stable scan can still hit it (notes at the\n",
    "> end of this subsection).\n",
    "\n",
    "A last word on precision, since discretized recurrences live or die by\n",
    "it. Three habits cover this chapter. First, compute $e^x - 1$ with\n",
    "`expm1`, as above. Second, respect what the scan does to rounding: it\n",
    "*reassociates* a length-$T$ product of decays, so its output matches the\n",
    "sequential loop only to float tolerance, never bitwise — every check in\n",
    "this chapter compares against a tolerance scaled to the dtype. Third,\n",
    "watch long products: with all $|\\bar{a}| < 1$ a product of $T$ decays\n",
    "underflows gracefully toward zero, but *ratios* of separately\n",
    "exponentiated prefix products, which the chunked forms of\n",
    "that section are built from, can overflow; log-domain and\n",
    "chunk-rescaled implementations are the production answer.\n",
    "\n",
    "### Recurrence is Convolution\n",
    "\n",
    "The system the equation is linear and *time-invariant* (LTI): the\n",
    "same $\\bar{\\mathbf{A}}, \\bar{\\mathbf{B}}, \\mathbf{C}$ act at every step.\n",
    "Unroll the recurrence from $\\mathbf{x}_0 = \\mathbf{0}$ and every output\n",
    "becomes a weighted sum of all past inputs:\n",
    "\n",
    "$$\n",
    "y_t = \\sum_{k=0}^{t-1} \\mathbf{C} \\bar{\\mathbf{A}}^{k} \\bar{\\mathbf{B}}\\, u_{t-k} + D\\, u_t\n",
    "= (\\bar{\\mathbf{K}} * \\mathbf{u})_t + D\\, u_t,\n",
    "\\qquad\n",
    "\\bar{\\mathbf{K}} = (\\mathbf{C}\\bar{\\mathbf{B}},\\\n",
    "\\mathbf{C}\\bar{\\mathbf{A}}\\bar{\\mathbf{B}},\\\n",
    "\\mathbf{C}\\bar{\\mathbf{A}}^2\\bar{\\mathbf{B}}, \\ldots).\n",
    "$$\n",
    "\n",
    "That is a plain causal convolution, with a kernel $\\bar{\\mathbf{K}}$ as\n",
    "long as the sequence itself, materialized from three small matrices. One\n",
    "model is thus three models (the figure): a continuous ODE\n",
    "(elegant for analysis and parameterization), a recurrence (constant-state\n",
    "inference, one token at a time), and a convolution (parallel training).\n",
    "The S4 line of work trains through this convolutional view, computing\n",
    "$\\bar{\\mathbf{K}}$ and applying it with FFTs in $O(T \\log T)$ time\n",
    "[@Gu.Goel.Re.2022]; it is a beautiful trick, but it leans on\n",
    "time-invariance, and the next section will make the dynamics\n",
    "input-dependent precisely to gain selectivity. The parallel scan covers\n",
    "both cases, so the scan is the path we implement.\n",
    "\n",
    "![Three views of one linear state space model. Left: a continuous-time ODE responds to an input pulse. Middle: the discretized system is a linear recurrence, one step per token. Right: unrolled, the same system is a causal convolution whose kernel collects the impulse response.](../img/mdl-modernrnn-ssm-views.svg)\n",
    "\n",
    "Seeing is believing: for a random diagonal SSM we now compute the same\n",
    "outputs three ways, with the sequential recurrence, with the parallel\n",
    "scan, and by convolving with the materialized kernel from\n",
    "the equation (we drop the $D$ skip here, since all three views\n",
    "share it verbatim)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-recurrence-is-convolution"
   },
   "source": [
    "num_states, num_steps, delta = 8, 64, 0.1\n",
    "a = -torch.arange(1., num_states + 1)               # Re(a) < 0\n",
    "b, c = torch.ones(num_states), torch.randn(num_states)\n",
    "a_bar = torch.exp(delta * a)                        # Zero-order hold\n",
    "b_bar = torch.expm1(delta * a) / a * b              # expm1: accurate e^x - 1\n",
    "u = torch.randn(num_steps)\n",
    "\n",
    "x, ys = torch.zeros(num_states), []                 # (i) recurrence\n",
    "for t in range(num_steps):\n",
    "    x = a_bar * x + b_bar * u[t]\n",
    "    ys.append(torch.dot(c, x))\n",
    "y_loop = torch.stack(ys)\n",
    "\n",
    "xs = associative_scan(a_bar.tile(num_steps, 1),     # (ii) parallel scan\n",
    "                      b_bar * u[:, None])\n",
    "y_scan = xs @ c\n",
    "\n",
    "k = torch.arange(num_steps)[:, None]                # (iii) convolution\n",
    "kernel = (c * b_bar * a_bar ** k).sum(-1)\n",
    "y_conv = torch.tensor(np.convolve(u, kernel)[:num_steps])\n",
    "\n",
    "print(f'scan vs loop: {float((y_scan - y_loop).abs().max()):.2e}, '\n",
    "      f'conv vs loop: {float((y_conv - y_loop).abs().max()):.2e}')"
   ],
   "execution_count": 10,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "scan vs loop: 2.38e-07, conv vs loop: 2.38e-07\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-8 fw=all -->\n",
    "\n",
    "The three views agree to floating-point precision. Each is the right tool\n",
    "for a different job, and the freedom to switch among them at will is the\n",
    "practical superpower of keeping the recurrence linear.\n",
    "\n",
    "### What Control Theory Already Knew\n",
    "\n",
    "State space models were not invented for deep learning: they are the\n",
    "core formalism of 1960s control theory, and four of its ideas earn their\n",
    "keep in this chapter. All four show up in the smallest interesting\n",
    "example, a two-state diagonal system\n",
    "\n",
    "$$\n",
    "\\dot{\\mathbf{x}}(t) = \\begin{pmatrix} -1 & 0 \\\\ 0 & -2 \\end{pmatrix} \\mathbf{x}(t) + \\mathbf{B}\\, u(t),\n",
    "\\qquad\n",
    "y(t) = \\mathbf{C}\\, \\mathbf{x}(t).\n",
    "$$\n",
    "\n",
    "*Controllability and observability.* With $\\mathbf{B} = (1, 0)^\\top$ the\n",
    "input never touches the second state: that mode is *uncontrollable*, and\n",
    "no input signal can move it off its decay. With $\\mathbf{C} = (1, 0)$\n",
    "the output never sees it: the mode is *unobservable*. Either way, the\n",
    "two-state system behaves at its terminals exactly like a one-state\n",
    "system. State size is an upper bound on dynamical richness, not a\n",
    "measurement of it — worth remembering whenever a later section prices a\n",
    "model by its state, since $N$ counts coordinates, not used memory.\n",
    "\n",
    "*Transfer function and poles.* Solved in the frequency domain, the LTI\n",
    "system is multiplication by the *transfer function*\n",
    "$H(s) = \\mathbf{C}(s\\mathbf{I} - \\mathbf{A})^{-1}\\mathbf{B}$, a rational\n",
    "function whose poles are a subset of the eigenvalues of $\\mathbf{A}$.\n",
    "For the example with $\\mathbf{B} = (1, 0)^\\top$ and\n",
    "$\\mathbf{C} = (1, 1)$, $H(s) = 1/(s+1)$: one pole, not two, because the\n",
    "unreachable mode cancels. The impulse response — our convolution kernel\n",
    "the equation — is the same object in the time domain; the\n",
    "kernel view above is control theory's oldest view, rediscovered.\n",
    "\n",
    "*Internal versus BIBO stability.* The cancellation is also where the two\n",
    "stability notions of the box above part ways: a system can be BIBO\n",
    "stable, with bounded inputs producing bounded outputs, while an unstable\n",
    "mode grows unseen because it is hidden from the output. Internal\n",
    "stability, every mode decaying, is the stronger certificate, and it is\n",
    "the one our left-half-plane parameterization provides.\n",
    "\n",
    "*Similarity and non-identifiability.* Change state coordinates by any\n",
    "invertible $\\mathbf{T}$, sending $(\\mathbf{A}, \\mathbf{B}, \\mathbf{C})$\n",
    "to $(\\mathbf{T}\\mathbf{A}\\mathbf{T}^{-1}, \\mathbf{T}\\mathbf{B},\n",
    "\\mathbf{C}\\mathbf{T}^{-1})$, and the input–output map is unchanged. The\n",
    "state is a coordinate system, not a canonical object. That freedom is\n",
    "why restricting $\\mathbf{A}$ to diagonal form (as S4D will) is less of a\n",
    "sacrifice than it looks, and why two different-looking recurrences can\n",
    "compute one function — a theme that returns as the state-space duality\n",
    "of that section. For the classical theory at full\n",
    "strength, see @Astrom.Murray.2021 .\n",
    "\n",
    "A scope note, finally, because the term is older than our usage of it.\n",
    "In statistics and signal processing a \"state space model\" is usually\n",
    "*stochastic*: process noise drives the state, observation noise corrupts\n",
    "the output, and the computational problem is inference — recovering a\n",
    "posterior over the latent trajectory, by the Kalman filter\n",
    "[@Kalman.1960] in the linear-Gaussian case and by particle methods\n",
    "beyond it, with system identification as the matching learning problem.\n",
    "The SSMs of this chapter share the state equation but not the problem:\n",
    "they are deterministic feature extractors trained by backpropagation,\n",
    "with no noise model and no posterior. The shared name records lineage,\n",
    "not equivalence.\n",
    "\n",
    "## Remembering the Past: HiPPO\n",
    "\n",
    "One question remains before we can call this a principled design: what\n",
    "should $\\mathbf{A}$ *be*? Stability constrains its eigenvalues to the left\n",
    "half-plane but otherwise leaves $N^2$ (or $N$ diagonal) numbers to choose,\n",
    "and early attempts to train SSMs from random initializations performed\n",
    "poorly. The question has a crisp mathematical form. The state\n",
    "$\\mathbf{x}(t)$ is a fixed budget of $N$ numbers summarizing the input\n",
    "seen so far; which $N$ numbers lose the least?\n",
    "\n",
    "HiPPO (high-order polynomial projection operators) is the answer\n",
    "[@Gu.Dao.Ermon.ea.2020]. Fix a measure over the past, say uniform\n",
    "weight on everything seen so far, and ask that $\\mathbf{x}(t)$ hold the\n",
    "$N$ coefficients of the best *orthogonal polynomial approximation* (of\n",
    "degree less than $N$) of the input's history under that measure, the same\n",
    "least-squares projection that powers Fourier and Legendre series. The remarkable\n",
    "theorem is that this optimal compression can be maintained *online*: as\n",
    "new input arrives, the optimal coefficients evolve according to a\n",
    "linear, *time-varying* ODE,\n",
    "\n",
    "$$\n",
    "\\dot{\\mathbf{x}}(t) = \\frac{1}{t} \\left( \\mathbf{A}\\, \\mathbf{x}(t) + \\mathbf{B}\\, u(t) \\right),\n",
    "$$\n",
    "\n",
    "with a specific, input-independent matrix. For the uniform (\"scaled\n",
    "Legendre\", LegS) measure it is\n",
    "\n",
    "$$\n",
    "A_{nk} = -\\begin{cases}\n",
    "\\sqrt{(2n+1)(2k+1)} & \\textrm{if } n > k, \\\\\n",
    "n + 1 & \\textrm{if } n = k, \\\\\n",
    "0 & \\textrm{if } n < k,\n",
    "\\end{cases}\n",
    "\\qquad\n",
    "B_n = \\sqrt{2n + 1},\n",
    "$$\n",
    "\n",
    "for $n, k = 0, \\ldots, N-1$. We state the matrix and cite the\n",
    "derivation; the $1/t$, though, deserves a sentence, because it is what\n",
    "makes the equation *not* an instance of the LTI system\n",
    "the equation. The measure spreads uniform weight over all of\n",
    "$[0, t]$, so by time $t$ each new instant is only a $1/t$ fraction of\n",
    "the history it joins, and the dynamics must slow down accordingly:\n",
    "discretized at unit steps the update is\n",
    "$\\mathbf{x}_k = (\\mathbf{I} + \\mathbf{A}/k)\\, \\mathbf{x}_{k-1} +\n",
    "(\\mathbf{B}/k)\\, u_k$, a recurrence whose effective step size decays\n",
    "like $1/k$ rather than holding constant. (A pleasant consequence: the\n",
    "system has no timescale $\\Delta$ to tune, because rescaling time maps\n",
    "its trajectories onto themselves.) What the theorem buys is shown in\n",
    "the figure, whose generator integrates exactly\n",
    "this time-varying system: a signal is compressed online into a LegS\n",
    "state of size $N$ and then reconstructed from the state alone at the\n",
    "final time. With $N = 64$\n",
    "numbers, a thousand-step history is recovered almost perfectly, smoothing\n",
    "only the sharpest transients. A fixed-size state provably *can* carry a\n",
    "long past, provided its dynamics are chosen for the job.\n",
    "\n",
    "![Compressing a function into $N$ numbers, online. A HiPPO-LegS state of size $N$ is updated as the signal (black) streams in; afterwards, the full history is reconstructed (color) from the final state alone. Larger states recover the past in increasing detail.](../img/mdl-modernrnn-hippo-reconstruction.svg)\n",
    "\n",
    "### From HiPPO to S4 to S4D\n",
    "\n",
    "The path from this theorem to the layer we implement passes through\n",
    "three different objects, and it pays to keep them distinct.\n",
    "\n",
    "*The online LegS projection* is the time-varying system\n",
    "the equation. It, and only it, carries the optimality theorem:\n",
    "run those dynamics and the state is, at every moment, the best\n",
    "degree-$N$ polynomial summary of the entire past under the uniform\n",
    "measure.\n",
    "\n",
    "*S4 is an LTI model initialized at the HiPPO matrix.* S4\n",
    "[@Gu.Goel.Re.2022] freezes the equation into the\n",
    "time-invariant system the equation, adds a learnable step size\n",
    "per channel, and trains everything by gradient descent. The optimality\n",
    "theorem does not transfer — a time-invariant system maintains no\n",
    "uniform-measure projection; tracking one is precisely what the $1/t$ was\n",
    "for — but the HiPPO matrix proved to be a superb *initialization*,\n",
    "giving the state a spread of decay timescales and long-memory structure\n",
    "that random matrices lack. Speed came from structure: the HiPPO matrix\n",
    "is *normal plus low-rank* (NPLR), so it is unitarily similar to a\n",
    "diagonal-plus-low-rank (DPLR) matrix with complex-conjugate modes, and\n",
    "with that form the convolution kernel the equation reduces to\n",
    "Cauchy-kernel evaluations computable in near-linear time — machinery we\n",
    "cite rather than rebuild. The payoff landed on the Long Range Arena\n",
    "benchmark [@Tay.Dehghani.Abnar.ea.2021]: tasks with dependencies\n",
    "over 16,000 steps, solved ahead of every transformer of S4's day.\n",
    "\n",
    "*S4D is the diagonal approximation.* Drop the low-rank correction and\n",
    "keep a diagonal state matrix, collapsing the machinery to the\n",
    "elementwise formulas the equation. @Gu.Gupta.Goel.Re.2022\n",
    "supply the evidence that this step is safe: suitably initialized\n",
    "diagonal systems approximate S4's kernel as the state grows and match\n",
    "its accuracy across the Long Range Arena in practice. The simplest of\n",
    "their initializations is the one we implement below, real\n",
    "$a_n = -(n + 1)$, the diagonal of the equation. Be clear about what\n",
    "that inherits and what it does not: it is a *multiscale initialization*,\n",
    "a bank of decay rates from slow to fast, not the optimal Legendre\n",
    "projection — the theorem stays behind with the time-varying system, and\n",
    "the case for the diagonal toy is the S4D paper's empirical one. Finally,\n",
    "S5 [@Smith.Warrington.Linderman.2023] replaced the per-channel\n",
    "FFTs with one multi-input state space layer evaluated by the\n",
    "parallel scan, the same computational pattern we built above. Our\n",
    "implementation sits deliberately at the intersection: S4D's diagonal\n",
    "parameterization, S5's scan.\n",
    "\n",
    "## S4D in Practice\n",
    "\n",
    "Time to assemble a working network. The `S4D` layer below runs one\n",
    "independent scalar SSM per feature channel, each with $N$ internal states:\n",
    "its parameters are the diagonal $\\mathbf{a}$ (stored as\n",
    "$-e^{\\log \\textrm{magnitude}}$ to pin eigenvalues to the left half-plane,\n",
    "initialized to $-(n+1)$), a per-channel learnable step size $\\Delta$\n",
    "(stored as $\\log \\Delta$, initialized log-uniformly between $10^{-3}$ and\n",
    "$10^{-1}$, so different channels start as gates of very different speeds),\n",
    "the read-out $\\mathbf{C}$, and the skip $D$; following S4D we fix\n",
    "$\\mathbf{B} = \\mathbf{1}$ and let $\\mathbf{C}$ absorb its scale. The\n",
    "forward pass is three lines of ZOH followed by the scan from\n",
    "that section; since the dynamics are time-invariant,\n",
    "the decay coefficients are the same at every position, and broadcasting\n",
    "(a singleton batch axis for `a`) spares us materializing them per element.\n",
    "We save the layer, and the block that follows, in the `d2l` library: the\n",
    "next section builds directly on both."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-s4d-in-practice-1"
   },
   "source": [
    "class S4D(nn.Module):\n",
    "    \"\"\"A diagonal state space layer: one SSM per feature channel.\"\"\"\n",
    "    def __init__(self, num_hiddens, num_states=4, dt_min=0.001, dt_max=0.1):\n",
    "        super().__init__()\n",
    "        H, N = num_hiddens, num_states\n",
    "        self.log_a = nn.Parameter(\n",
    "            torch.log(torch.arange(1., N + 1)).repeat(H, 1))\n",
    "        self.log_dt = nn.Parameter(\n",
    "            torch.rand(H, 1) * math.log(dt_max / dt_min) + math.log(dt_min))\n",
    "        self.C = nn.Parameter(torch.randn(H, N) / math.sqrt(N))\n",
    "        self.D = nn.Parameter(torch.ones(H))\n",
    "\n",
    "    def forward(self, u):                    # (num_steps, batch, num_hiddens)\n",
    "        a = -torch.exp(self.log_a)                    # (H, N), Re(a) < 0\n",
    "        da = torch.exp(self.log_dt) * a\n",
    "        a_bar = torch.exp(da)\n",
    "        b_bar = torch.expm1(da) / a                   # ZOH with B = 1\n",
    "        a_elems = a_bar.expand(u.shape[0], 1, -1, -1) # Same at every step\n",
    "        b_elems = b_bar * u.unsqueeze(-1)             # (T, batch, H, N)\n",
    "        x = associative_scan(a_elems, b_elems)\n",
    "        return (x * self.C).sum(-1) + self.D * u"
   ],
   "execution_count": 11,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-9 fw=all -->\n",
    "\n",
    "A single `S4D` layer mixes information across *time* but never across\n",
    "*channels* (each channel owns a private SSM), and it is linear. Both gaps\n",
    "are closed the modern way, by wrapping the layer in a residual block with\n",
    "normalization and a gated MLP (the figure): the MLP mixes\n",
    "channels and supplies the nonlinearity that the state path gave up, and\n",
    "the gate is one more echo of that section. Stacking $L$ such blocks\n",
    "yields the standard deep SSM of the S4/S4D/S5 papers.\n",
    "\n",
    "![The S4D residual block: sequence mixing by a per-channel diagonal SSM, channel mixing by a gated MLP, each behind LayerNorm with a residual connection, stacked $L$ times.](../img/mdl-modernrnn-ssm-block.svg)"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-s4d-in-practice-2"
   },
   "source": [
    "class S4DBlock(nn.Module):\n",
    "    def __init__(self, num_hiddens, num_states):\n",
    "        super().__init__()\n",
    "        self.ln1 = nn.LayerNorm(num_hiddens)\n",
    "        self.ssm = S4D(num_hiddens, num_states)\n",
    "        self.ln2 = nn.LayerNorm(num_hiddens)\n",
    "        self.W_v = nn.Linear(num_hiddens, 2 * num_hiddens)\n",
    "        self.W_g = nn.Linear(num_hiddens, 2 * num_hiddens)\n",
    "        self.W_o = nn.Linear(2 * num_hiddens, num_hiddens)\n",
    "\n",
    "    def forward(self, X):\n",
    "        X = X + self.ssm(self.ln1(X))\n",
    "        Y = self.ln2(X)\n",
    "        return X + self.W_o(self.W_v(Y) * torch.sigmoid(self.W_g(Y)))"
   ],
   "execution_count": 12,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-10 fw=all -->\n",
    "\n",
    "### Sequential Image Classification\n",
    "\n",
    "Where should a long-memory architecture prove itself? A classic stress\n",
    "test is *sequential image classification*: feed a model the\n",
    "Fashion-MNIST images of that section one pixel at a time,\n",
    "as a plain sequence of 784 scalars, and predict the class. No\n",
    "spatial structure is given: any relation between pixel 3 and pixel 787\n",
    "must travel through the sequence model. How it is allowed to travel is\n",
    "decided by the *readout*, and the choice changes what the experiment\n",
    "measures. Mean-pooling all 784 per-step outputs is the standard\n",
    "benchmark form of the task: every pixel contributes through its own\n",
    "output, so the score measures how well the encoder *mixes* information\n",
    "across the sequence, not whether anything survives to the end. Reading\n",
    "only the *final* step's features is the sterner retention diagnostic:\n",
    "whatever the model uses must still be present in the state after\n",
    "hundreds of updates. The classifier below embeds\n",
    "each pixel, runs an encoder over the sequence, and supports both\n",
    "readouts; it accepts any encoder with our\n",
    "`(num_steps, batch, features)` calling convention, which lets us swap an\n",
    "S4D stack against an LSTM without touching anything else. We train with\n",
    "Adam (that section), whose per-parameter step sizes suit\n",
    "a model that mixes log-decays with ordinary linear weights; plain SGD\n",
    "handles the SSM's exponentials awkwardly."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-sequential-image-classification-1"
   },
   "source": [
    "class SeqClassifier(d2l.Classifier):\n",
    "    \"\"\"Classify a sequence: encoder, then mean or final-step readout.\"\"\"\n",
    "    def __init__(self, encoder, num_hiddens, pool='mean', lr=3e-3):\n",
    "        super().__init__()\n",
    "        self.save_hyperparameters()\n",
    "        self.emb = nn.Linear(1, num_hiddens)\n",
    "        self.head = nn.LazyLinear(10)\n",
    "\n",
    "    def forward(self, X):\n",
    "        X = X.reshape(X.shape[0], -1, 1).movedim(1, 0)  # Pixel sequence\n",
    "        Y = self.encoder(self.emb(X))\n",
    "        Y = Y[0] if isinstance(Y, tuple) else Y         # RNNs return a state\n",
    "        return self.head(Y.mean(dim=0) if self.pool == 'mean' else Y[-1])\n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        return torch.optim.Adam(self.parameters(), lr=self.lr)"
   ],
   "execution_count": 13,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-11 fw=all -->\n",
    "\n",
    "Our S4D model stacks two blocks of width 48 with $N = 4$ states per\n",
    "channel; the baseline is the LSTM of that section sized to the same\n",
    "parameter count (48-dimensional pixel embeddings, 64 hidden units,\n",
    "roughly 30,000 parameters each), trained under the benchmark mean-pool\n",
    "readout. Ten epochs each, identical data\n",
    "pipeline, identical head, and gradients clipped to norm 1 for both\n",
    "models: at 784 steps the LSTM otherwise destabilizes mid-training, its\n",
    "loss spiking exactly as that section would\n",
    "predict, while the S4D, whose recurrence cannot leave the unit circle,\n",
    "reaches the same accuracy with or without clipping."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-sequential-image-classification-2"
   },
   "source": [
    "data = d2l.FashionMNIST(batch_size=128)\n",
    "results = {}\n",
    "\n",
    "def train_and_report(name, model, epochs=10):\n",
    "    trainer = d2l.Trainer(max_epochs=epochs, gradient_clip_val=1, num_gpus=1)\n",
    "    start = time.time()\n",
    "    trainer.fit(model, data)\n",
    "    params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "    results[name] = (params, float(model.board.data['val_acc'][-1].y),\n",
    "                     (time.time() - start) / epochs)\n",
    "\n",
    "s4d = SeqClassifier(nn.Sequential(*[S4DBlock(48, 4) for _ in range(2)]),\n",
    "                    num_hiddens=48)\n",
    "train_and_report('S4D', s4d)"
   ],
   "execution_count": 14,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"245.328125pt\" height=\"183.35625pt\" viewBox=\"0 0 245.328125 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-20T01:06:46.269967</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 183.35625 \nL 245.328125 183.35625 \nL 245.328125 0 \nL 0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 36.465625 145.8 \nL 231.765625 145.8 \nL 231.765625 7.2 \nL 36.465625 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path id=\"meeca73870a\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#meeca73870a\" x=\"36.465625\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <g transform=\"translate(33.284375 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use xlink:href=\"#meeca73870a\" x=\"75.525625\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 2 -->\n      <g transform=\"translate(72.344375 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-32\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use xlink:href=\"#meeca73870a\" x=\"114.585625\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 4 -->\n      <g transform=\"translate(111.404375 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-34\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#meeca73870a\" x=\"153.645625\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 6 -->\n      <g transform=\"translate(150.464375 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-36\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use xlink:href=\"#meeca73870a\" x=\"192.705625\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 8 -->\n      <g transform=\"translate(189.524375 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-38\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#meeca73870a\" x=\"231.765625\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 10 -->\n      <g transform=\"translate(225.403125 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- epoch -->\n     <g transform=\"translate(118.8875 174.076563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \nL 1159 -1331 \nL 581 -1331 \nL 581 3500 \nL 1159 3500 \nL 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nz\nM 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \nL 3122 2828 \nQ 2878 2963 2633 3030 \nQ 2388 3097 2138 3097 \nQ 1578 3097 1268 2742 \nQ 959 2388 959 1747 \nQ 959 1106 1268 751 \nQ 1578 397 2138 397 \nQ 2388 397 2633 464 \nQ 2878 531 3122 666 \nL 3122 134 \nQ 2881 22 2623 -34 \nQ 2366 -91 2075 -91 \nQ 1284 -91 818 406 \nQ 353 903 353 1747 \nQ 353 2603 823 3093 \nQ 1294 3584 2113 3584 \nQ 2378 3584 2631 3529 \nQ 2884 3475 3122 3366 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-65\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(61.523438 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(186.181641 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(241.162109 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_7\">\n      <defs>\n       <path id=\"ma927e672d5\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#ma927e672d5\" x=\"36.465625\" y=\"137.012391\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.50 -->\n      <g transform=\"translate(7.2 140.811609) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \nL 1344 794 \nL 1344 0 \nL 684 0 \nL 684 794 \nz\n\" transform=\"scale(0.015625)\"/>\n        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \nL 3169 4666 \nL 3169 4134 \nL 1269 4134 \nL 1269 2991 \nQ 1406 3038 1543 3061 \nQ 1681 3084 1819 3084 \nQ 2600 3084 3056 2656 \nQ 3513 2228 3513 1497 \nQ 3513 744 3044 326 \nQ 2575 -91 1722 -91 \nQ 1428 -91 1123 -41 \nQ 819 9 494 109 \nL 494 744 \nQ 775 591 1075 516 \nQ 1375 441 1709 441 \nQ 2250 441 2565 725 \nQ 2881 1009 2881 1497 \nQ 2881 1984 2565 2268 \nQ 2250 2553 1709 2553 \nQ 1456 2553 1204 2497 \nQ 953 2441 691 2322 \nL 691 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#ma927e672d5\" x=\"36.465625\" y=\"108.178848\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.75 -->\n      <g transform=\"translate(7.2 111.978067) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-37\" d=\"M 525 4666 \nL 3525 4666 \nL 3525 4397 \nL 1831 0 \nL 1172 0 \nL 2766 4134 \nL 525 4134 \nL 525 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-37\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_9\">\n      <g>\n       <use xlink:href=\"#ma927e672d5\" x=\"36.465625\" y=\"79.345306\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 1.00 -->\n      <g transform=\"translate(7.2 83.144525) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#ma927e672d5\" x=\"36.465625\" y=\"50.511763\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 1.25 -->\n      <g transform=\"translate(7.2 54.310982) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_11\">\n      <g>\n       <use xlink:href=\"#ma927e672d5\" x=\"36.465625\" y=\"21.678221\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 1.50 -->\n      <g transform=\"translate(7.2 25.47744) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_12\">\n    <path d=\"M 41.316894 13.5 \nL 51.061073 68.416444 \nL 60.805252 78.284213 \nL 70.549431 96.555947 \nL 80.29361 107.616025 \nL 90.037789 112.635966 \nL 99.781968 117.37394 \nL 109.526147 118.259815 \nL 119.270326 122.986046 \nL 129.014506 123.850775 \nL 138.758685 126.257964 \nL 148.502864 129.17791 \nL 158.247043 130.906589 \nL 167.991222 132.414555 \nL 177.735401 133.649668 \nL 187.47958 136.447817 \nL 197.223759 136.883798 \nL 206.967938 138.444805 \nL 216.712118 139.394649 \nL 226.456297 139.5 \n\" clip-path=\"url(#pa36530779a)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_13\">\n    <path d=\"M 55.995625 62.52889 \nL 75.525625 105.940155 \nL 95.055625 113.235871 \nL 114.585625 117.790281 \nL 134.115625 124.369431 \nL 153.645625 131.319695 \nL 173.175625 130.259932 \nL 192.705625 133.995701 \nL 212.235625 135.380267 \nL 231.765625 133.290719 \n\" clip-path=\"url(#pa36530779a)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #ff7f0e; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"line2d_14\">\n    <path d=\"M 55.995625 127.796607 \nL 75.525625 112.056777 \nL 95.055625 108.281499 \nL 114.585625 108.555236 \nL 134.115625 104.791363 \nL 153.645625 102.065407 \nL 173.175625 102.145247 \nL 192.705625 100.981868 \nL 212.235625 100.970463 \nL 231.765625 101.438095 \n\" clip-path=\"url(#pa36530779a)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #2ca02c; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 36.465625 145.8 \nL 36.465625 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 231.765625 145.8 \nL 231.765625 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 36.465625 145.8 \nL 231.765625 145.8 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 36.465625 7.2 \nL 231.765625 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 145.175 60.06875 \nL 224.765625 60.06875 \nQ 226.765625 60.06875 226.765625 58.06875 \nL 226.765625 14.2 \nQ 226.765625 12.2 224.765625 12.2 \nL 145.175 12.2 \nQ 143.175 12.2 143.175 14.2 \nL 143.175 58.06875 \nQ 143.175 60.06875 145.175 60.06875 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_15\">\n     <path d=\"M 147.175 20.298438 \nL 157.175 20.298438 \nL 167.175 20.298438 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_13\">\n     <!-- train_loss -->\n     <g transform=\"translate(175.175 23.798438) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \nL 3263 -1509 \nL -63 -1509 \nL -63 -1063 \nL 3263 -1063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(39.208984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(80.322266 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(141.601562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(169.384766 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(232.763672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(282.763672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(310.546875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(371.728516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(423.828125 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_16\">\n     <path d=\"M 147.175 35.254688 \nL 157.175 35.254688 \nL 167.175 35.254688 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #ff7f0e; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_14\">\n     <!-- val_loss -->\n     <g transform=\"translate(175.175 38.754688) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-76\" d=\"M 191 3500 \nL 800 3500 \nL 1894 563 \nL 2988 3500 \nL 3597 3500 \nL 2284 0 \nL 1503 0 \nL 191 3500 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-76\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(59.179688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(120.458984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(148.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(198.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(226.025391 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(287.207031 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(339.306641 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_17\">\n     <path d=\"M 147.175 50.210938 \nL 157.175 50.210938 \nL 167.175 50.210938 \n\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #2ca02c; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_15\">\n     <!-- val_acc -->\n     <g transform=\"translate(175.175 53.710938) scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-76\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(59.179688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(120.458984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(148.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(198.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(259.521484 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(314.501953 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pa36530779a\">\n   <rect x=\"36.465625\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-12 fw=all -->\n",
    "\n",
    "The retention diagnostic is the same mixer read only at the final step,\n",
    "one more training run."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-sequential-image-classification-5"
   },
   "source": [
    "s4d_final = SeqClassifier(\n",
    "    nn.Sequential(*[S4DBlock(48, 4) for _ in range(2)]),\n",
    "    num_hiddens=48, pool='last')\n",
    "train_and_report('S4D (final)', s4d_final)"
   ],
   "execution_count": 15,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"245.328125pt\" height=\"185.317272pt\" viewBox=\"0 0 245.328125 185.317272\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-20T01:12:55.726208</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 185.317272 \nL 245.328125 185.317272 \nL 245.328125 0 \nL 0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 36.465625 147.761022 \nL 231.765625 147.761022 \nL 231.765625 9.161022 \nL 36.465625 9.161022 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path id=\"m59fee9bb5a\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m59fee9bb5a\" x=\"36.465625\" y=\"147.761022\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <g transform=\"translate(33.284375 162.359459) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use xlink:href=\"#m59fee9bb5a\" x=\"75.525625\" y=\"147.761022\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 2 -->\n      <g transform=\"translate(72.344375 162.359459) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-32\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use xlink:href=\"#m59fee9bb5a\" x=\"114.585625\" y=\"147.761022\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 4 -->\n      <g transform=\"translate(111.404375 162.359459) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-34\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#m59fee9bb5a\" x=\"153.645625\" y=\"147.761022\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 6 -->\n      <g transform=\"translate(150.464375 162.359459) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-36\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use xlink:href=\"#m59fee9bb5a\" x=\"192.705625\" y=\"147.761022\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 8 -->\n      <g transform=\"translate(189.524375 162.359459) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-38\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#m59fee9bb5a\" x=\"231.765625\" y=\"147.761022\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 10 -->\n      <g transform=\"translate(225.403125 162.359459) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- epoch -->\n     <g transform=\"translate(118.8875 176.037584) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \nL 1159 -1331 \nL 581 -1331 \nL 581 3500 \nL 1159 3500 \nL 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nz\nM 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \nL 3122 2828 \nQ 2878 2963 2633 3030 \nQ 2388 3097 2138 3097 \nQ 1578 3097 1268 2742 \nQ 959 2388 959 1747 \nQ 959 1106 1268 751 \nQ 1578 397 2138 397 \nQ 2388 397 2633 464 \nQ 2878 531 3122 666 \nL 3122 134 \nQ 2881 22 2623 -34 \nQ 2366 -91 2075 -91 \nQ 1284 -91 818 406 \nQ 353 903 353 1747 \nQ 353 2603 823 3093 \nQ 1294 3584 2113 3584 \nQ 2378 3584 2631 3529 \nQ 2884 3475 3122 3366 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-65\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(61.523438 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(186.181641 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(241.162109 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_7\">\n      <defs>\n       <path id=\"m0b7baa4d1d\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m0b7baa4d1d\" x=\"36.465625\" y=\"142.176241\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.50 -->\n      <g transform=\"translate(7.2 145.975459) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \nL 1344 794 \nL 1344 0 \nL 684 0 \nL 684 794 \nz\n\" transform=\"scale(0.015625)\"/>\n        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \nL 3169 4666 \nL 3169 4134 \nL 1269 4134 \nL 1269 2991 \nQ 1406 3038 1543 3061 \nQ 1681 3084 1819 3084 \nQ 2600 3084 3056 2656 \nQ 3513 2228 3513 1497 \nQ 3513 744 3044 326 \nQ 2575 -91 1722 -91 \nQ 1428 -91 1123 -41 \nQ 819 9 494 109 \nL 494 744 \nQ 775 591 1075 516 \nQ 1375 441 1709 441 \nQ 2250 441 2565 725 \nQ 2881 1009 2881 1497 \nQ 2881 1984 2565 2268 \nQ 2250 2553 1709 2553 \nQ 1456 2553 1204 2497 \nQ 953 2441 691 2322 \nL 691 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#m0b7baa4d1d\" x=\"36.465625\" y=\"115.940836\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.75 -->\n      <g transform=\"translate(7.2 119.740055) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-37\" d=\"M 525 4666 \nL 3525 4666 \nL 3525 4397 \nL 1831 0 \nL 1172 0 \nL 2766 4134 \nL 525 4134 \nL 525 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-37\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_9\">\n      <g>\n       <use xlink:href=\"#m0b7baa4d1d\" x=\"36.465625\" y=\"89.705432\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 1.00 -->\n      <g transform=\"translate(7.2 93.504651) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#m0b7baa4d1d\" x=\"36.465625\" y=\"63.470028\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 1.25 -->\n      <g transform=\"translate(7.2 67.269246) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_11\">\n      <g>\n       <use xlink:href=\"#m0b7baa4d1d\" x=\"36.465625\" y=\"37.234623\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 1.50 -->\n      <g transform=\"translate(7.2 41.033842) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_12\">\n      <g>\n       <use xlink:href=\"#m0b7baa4d1d\" x=\"36.465625\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_13\">\n      <!-- 1.75 -->\n      <g transform=\"translate(7.2 14.798438) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-37\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_13\">\n    <path d=\"M 41.316894 15.461022 \nL 51.061073 84.725802 \nL 60.805252 102.553672 \nL 70.549431 112.058362 \nL 80.29361 116.954701 \nL 90.037789 122.559433 \nL 99.781968 124.891011 \nL 109.526147 127.805731 \nL 119.270326 130.277538 \nL 129.014506 130.440103 \nL 138.758685 131.833517 \nL 148.502864 133.157023 \nL 158.247043 134.796028 \nL 167.991222 134.099825 \nL 177.735401 135.95953 \nL 187.47958 137.273984 \nL 197.223759 138.853852 \nL 206.967938 139.094887 \nL 216.712118 140.944369 \nL 226.456297 141.461022 \n\" clip-path=\"url(#p92dbc5eb86)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_14\">\n    <path d=\"M 55.995625 92.400772 \nL 75.525625 113.875063 \nL 95.055625 123.205409 \nL 114.585625 126.555269 \nL 134.115625 131.094692 \nL 153.645625 129.52874 \nL 173.175625 133.886627 \nL 192.705625 134.909922 \nL 212.235625 136.077389 \nL 231.765625 139.252683 \n\" clip-path=\"url(#p92dbc5eb86)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #ff7f0e; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"line2d_15\">\n    <path d=\"M 55.995625 127.802809 \nL 75.525625 119.521222 \nL 95.055625 116.200284 \nL 114.585625 115.255893 \nL 134.115625 113.543535 \nL 153.645625 114.103943 \nL 173.175625 112.41234 \nL 192.705625 112.121758 \nL 212.235625 111.073588 \nL 231.765625 110.015039 \n\" clip-path=\"url(#p92dbc5eb86)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #2ca02c; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 36.465625 147.761022 \nL 36.465625 9.161022 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 231.765625 147.761022 \nL 231.765625 9.161022 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 36.465625 147.761022 \nL 231.765625 147.761022 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 36.465625 9.161022 \nL 231.765625 9.161022 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 145.175 62.029772 \nL 224.765625 62.029772 \nQ 226.765625 62.029772 226.765625 60.029772 \nL 226.765625 16.161022 \nQ 226.765625 14.161022 224.765625 14.161022 \nL 145.175 14.161022 \nQ 143.175 14.161022 143.175 16.161022 \nL 143.175 60.029772 \nQ 143.175 62.029772 145.175 62.029772 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_16\">\n     <path d=\"M 147.175 22.259459 \nL 157.175 22.259459 \nL 167.175 22.259459 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_14\">\n     <!-- train_loss -->\n     <g transform=\"translate(175.175 25.759459) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \nL 3263 -1509 \nL -63 -1509 \nL -63 -1063 \nL 3263 -1063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(39.208984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(80.322266 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(141.601562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(169.384766 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(232.763672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(282.763672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(310.546875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(371.728516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(423.828125 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_17\">\n     <path d=\"M 147.175 37.215709 \nL 157.175 37.215709 \nL 167.175 37.215709 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #ff7f0e; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_15\">\n     <!-- val_loss -->\n     <g transform=\"translate(175.175 40.715709) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-76\" d=\"M 191 3500 \nL 800 3500 \nL 1894 563 \nL 2988 3500 \nL 3597 3500 \nL 2284 0 \nL 1503 0 \nL 191 3500 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-76\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(59.179688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(120.458984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(148.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(198.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(226.025391 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(287.207031 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(339.306641 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_18\">\n     <path d=\"M 147.175 52.171959 \nL 157.175 52.171959 \nL 167.175 52.171959 \n\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #2ca02c; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_16\">\n     <!-- val_acc -->\n     <g transform=\"translate(175.175 55.671959) scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-76\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(59.179688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(120.458984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(148.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(198.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(259.521484 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(314.501953 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p92dbc5eb86\">\n   <rect x=\"36.465625\" y=\"9.161022\" width=\"195.3\" height=\"138.6\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-sequential-image-classification-3"
   },
   "source": [
    "lstm = SeqClassifier(d2l.LSTM(num_inputs=48, num_hiddens=64),\n",
    "                     num_hiddens=48)\n",
    "train_and_report('LSTM', lstm)"
   ],
   "execution_count": 16,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"245.328125pt\" height=\"183.35625pt\" viewBox=\"0 0 245.328125 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-20T01:13:35.933704</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 183.35625 \nL 245.328125 183.35625 \nL 245.328125 0 \nL 0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 36.465625 145.8 \nL 231.765625 145.8 \nL 231.765625 7.2 \nL 36.465625 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path id=\"m241f35090c\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m241f35090c\" x=\"36.465625\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <g transform=\"translate(33.284375 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use xlink:href=\"#m241f35090c\" x=\"75.525625\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 2 -->\n      <g transform=\"translate(72.344375 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-32\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use xlink:href=\"#m241f35090c\" x=\"114.585625\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 4 -->\n      <g transform=\"translate(111.404375 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-34\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#m241f35090c\" x=\"153.645625\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 6 -->\n      <g transform=\"translate(150.464375 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-36\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use xlink:href=\"#m241f35090c\" x=\"192.705625\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 8 -->\n      <g transform=\"translate(189.524375 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-38\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#m241f35090c\" x=\"231.765625\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 10 -->\n      <g transform=\"translate(225.403125 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- epoch -->\n     <g transform=\"translate(118.8875 174.076563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \nL 1159 -1331 \nL 581 -1331 \nL 581 3500 \nL 1159 3500 \nL 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nz\nM 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \nL 3122 2828 \nQ 2878 2963 2633 3030 \nQ 2388 3097 2138 3097 \nQ 1578 3097 1268 2742 \nQ 959 2388 959 1747 \nQ 959 1106 1268 751 \nQ 1578 397 2138 397 \nQ 2388 397 2633 464 \nQ 2878 531 3122 666 \nL 3122 134 \nQ 2881 22 2623 -34 \nQ 2366 -91 2075 -91 \nQ 1284 -91 818 406 \nQ 353 903 353 1747 \nQ 353 2603 823 3093 \nQ 1294 3584 2113 3584 \nQ 2378 3584 2631 3529 \nQ 2884 3475 3122 3366 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-65\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(61.523438 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(186.181641 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(241.162109 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_7\">\n      <defs>\n       <path id=\"m0519e9487f\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m0519e9487f\" x=\"36.465625\" y=\"139.264454\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.50 -->\n      <g transform=\"translate(7.2 143.063673) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \nL 1344 794 \nL 1344 0 \nL 684 0 \nL 684 794 \nz\n\" transform=\"scale(0.015625)\"/>\n        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \nL 3169 4666 \nL 3169 4134 \nL 1269 4134 \nL 1269 2991 \nQ 1406 3038 1543 3061 \nQ 1681 3084 1819 3084 \nQ 2600 3084 3056 2656 \nQ 3513 2228 3513 1497 \nQ 3513 744 3044 326 \nQ 2575 -91 1722 -91 \nQ 1428 -91 1123 -41 \nQ 819 9 494 109 \nL 494 744 \nQ 775 591 1075 516 \nQ 1375 441 1709 441 \nQ 2250 441 2565 725 \nQ 2881 1009 2881 1497 \nQ 2881 1984 2565 2268 \nQ 2250 2553 1709 2553 \nQ 1456 2553 1204 2497 \nQ 953 2441 691 2322 \nL 691 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#m0519e9487f\" x=\"36.465625\" y=\"110.394867\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.75 -->\n      <g transform=\"translate(7.2 114.194085) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-37\" d=\"M 525 4666 \nL 3525 4666 \nL 3525 4397 \nL 1831 0 \nL 1172 0 \nL 2766 4134 \nL 525 4134 \nL 525 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-37\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_9\">\n      <g>\n       <use xlink:href=\"#m0519e9487f\" x=\"36.465625\" y=\"81.525279\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 1.00 -->\n      <g transform=\"translate(7.2 85.324498) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#m0519e9487f\" x=\"36.465625\" y=\"52.655691\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 1.25 -->\n      <g transform=\"translate(7.2 56.45491) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_11\">\n      <g>\n       <use xlink:href=\"#m0519e9487f\" x=\"36.465625\" y=\"23.786103\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 1.50 -->\n      <g transform=\"translate(7.2 27.585322) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(95.410156 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(159.033203 0)\"/>\n      </g>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_12\">\n    <path d=\"M 41.316894 13.5 \nL 51.061073 79.054342 \nL 60.805252 97.403256 \nL 70.549431 101.959805 \nL 80.29361 113.132394 \nL 90.037789 121.865034 \nL 99.781968 120.356737 \nL 109.526147 120.046518 \nL 119.270326 13.88878 \nL 129.014506 115.07205 \nL 138.758685 123.656015 \nL 148.502864 109.897502 \nL 158.247043 123.776223 \nL 167.991222 133.136666 \nL 177.735401 134.545917 \nL 187.47958 136.013893 \nL 197.223759 135.69488 \nL 206.967938 137.595757 \nL 216.712118 137.62614 \nL 226.456297 139.5 \n\" clip-path=\"url(#pb55ea71009)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_13\">\n    <path d=\"M 55.995625 98.709766 \nL 75.525625 33.276201 \nL 95.055625 123.638779 \nL 114.585625 127.593893 \nL 134.115625 126.418286 \nL 153.645625 39.558978 \nL 173.175625 133.644451 \nL 192.705625 134.888449 \nL 212.235625 136.855695 \nL 231.765625 138.929539 \n\" clip-path=\"url(#pb55ea71009)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #ff7f0e; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"line2d_14\">\n    <path d=\"M 55.995625 114.87148 \nL 75.525625 138.864757 \nL 95.055625 106.614869 \nL 114.585625 105.769794 \nL 134.115625 106.215172 \nL 153.645625 131.898598 \nL 173.175625 103.954025 \nL 192.705625 103.816986 \nL 212.235625 102.960491 \nL 231.765625 102.480854 \n\" clip-path=\"url(#pb55ea71009)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #2ca02c; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 36.465625 145.8 \nL 36.465625 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 231.765625 145.8 \nL 231.765625 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 36.465625 145.8 \nL 231.765625 145.8 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 36.465625 7.2 \nL 231.765625 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 145.175 100.434375 \nL 224.765625 100.434375 \nQ 226.765625 100.434375 226.765625 98.434375 \nL 226.765625 54.565625 \nQ 226.765625 52.565625 224.765625 52.565625 \nL 145.175 52.565625 \nQ 143.175 52.565625 143.175 54.565625 \nL 143.175 98.434375 \nQ 143.175 100.434375 145.175 100.434375 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_15\">\n     <path d=\"M 147.175 60.664063 \nL 157.175 60.664063 \nL 167.175 60.664063 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_13\">\n     <!-- train_loss -->\n     <g transform=\"translate(175.175 64.164063) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \nL 3263 -1509 \nL -63 -1509 \nL -63 -1063 \nL 3263 -1063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(39.208984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(80.322266 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(141.601562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(169.384766 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(232.763672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(282.763672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(310.546875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(371.728516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(423.828125 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_16\">\n     <path d=\"M 147.175 75.620313 \nL 157.175 75.620313 \nL 167.175 75.620313 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #ff7f0e; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_14\">\n     <!-- val_loss -->\n     <g transform=\"translate(175.175 79.120313) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-76\" d=\"M 191 3500 \nL 800 3500 \nL 1894 563 \nL 2988 3500 \nL 3597 3500 \nL 2284 0 \nL 1503 0 \nL 191 3500 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-76\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(59.179688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(120.458984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(148.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(198.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(226.025391 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(287.207031 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(339.306641 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_17\">\n     <path d=\"M 147.175 90.576563 \nL 157.175 90.576563 \nL 167.175 90.576563 \n\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #2ca02c; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_15\">\n     <!-- val_acc -->\n     <g transform=\"translate(175.175 94.076563) scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-76\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(59.179688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(120.458984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(148.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(198.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(259.521484 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(314.501953 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pb55ea71009\">\n   <rect x=\"36.465625\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-sequential-image-classification-4"
   },
   "source": [
    "print(f'{\"model\":>12} {\"params\":>8} {\"val acc\":>8} {\"s/epoch\":>8}')\n",
    "for name, (params, acc, secs) in results.items():\n",
    "    print(f'{name:>12} {params:>8,} {acc:>8.3f} {secs:>8.1f}')"
   ],
   "execution_count": 17,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "       model   params  val acc  s/epoch\n         S4D   30,058    0.808     47.6\n S4D (final)   30,058    0.806     36.9\n        LSTM   29,930    0.819      4.0\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-13 fw=all -->\n",
    "\n",
    "Read the table one comparison at a time. On the mean-pool benchmark the\n",
    "S4D lands in the low-to-mid eighties in both frameworks\n",
    "and every rerun, clipped or unclipped. The final-state run answers the\n",
    "retention question the mean pool cannot: reading nothing but the state\n",
    "after 784 updates costs the S4D little to nothing in our runs — the two\n",
    "readouts finish within run-to-run noise of each other, at most a few\n",
    "points apart and sometimes nearly tied — so what the classifier uses\n",
    "genuinely survives to the end of the\n",
    "sequence rather than being rescued by early outputs. The LSTM baseline\n",
    "is another\n",
    "story: across repeated runs (with and without clipping, at nearby\n",
    "learning rates) its final accuracy has ranged from roughly 60 to 84\n",
    "percent, and where a given run lands is a matter of initialization and\n",
    "luck. Plain uniform recurrent weights (PyTorch's default) leave\n",
    "long-range memory to chance; orthogonal recurrent weights (Flax's\n",
    "default) improve the odds without securing them; reliably closing the\n",
    "gap takes the rest of the folklore of that section, starting with\n",
    "a forget-gate bias of 1. The contrast is the section's thesis in\n",
    "miniature. At initialization the S4D's channels already implement a\n",
    "bank of exponential memories with time constants spanning more than two\n",
    "orders of magnitude (what the log-uniform $\\Delta$ init provides), so\n",
    "pixel-to-pixel structure hundreds of steps apart is visible to the model\n",
    "from the first gradient step, in any framework; an LSTM holds long-range\n",
    "state only when a decade's worth of initialization folklore is applied\n",
    "on its behalf, and hopes for a lucky draw otherwise. What the SSM\n",
    "parameterization provides — stability by construction and a multiscale\n",
    "initialization — is that memory *by design*, robustly, rather than\n",
    "by folklore. Wall-clock per epoch, also in the table,\n",
    "depends on what each framework fuses: against PyTorch's cuDNN-fused LSTM\n",
    "our teaching-grade scan pays its log-factor overhead, while in JAX,\n",
    "where both models run compiled scans, the parallel scan is several times\n",
    "faster than the sequential LSTM; production SSM kernels close the\n",
    "remaining gap the same way cuDNN once did for the LSTM. On the Long\n",
    "Range Arena [@Tay.Dehghani.Abnar.ea.2021], with sequences up to\n",
    "16,000 steps, this architecture family was the first to solve tasks on\n",
    "which both RNNs and transformers had failed, which is what first made\n",
    "the field take state space models seriously.\n",
    "\n",
    "## Inference, One Token at a Time\n",
    "\n",
    "This section opened with a bargain: train in parallel, then run the model\n",
    "the old recurrent way at inference, a fixed-size state updated in constant\n",
    "time per token. We have delivered the first half with the scan and, so\n",
    "far, only asserted the second. Time to collect. There is nothing to\n",
    "derive: the equation *is* the inference algorithm. Given the last\n",
    "state $\\mathbf{x}_{t-1}$ and one new input $u_t$,\n",
    "\n",
    "$$\n",
    "\\mathbf{x}_t = \\bar{\\mathbf{A}}\\, \\mathbf{x}_{t-1} + \\bar{\\mathbf{B}}\\, u_t,\n",
    "\\qquad\n",
    "y_t = \\mathbf{C}\\, \\mathbf{x}_t + D\\, u_t,\n",
    "$$\n",
    "\n",
    "and the state is all the model ever needs to remember: for our `S4D`\n",
    "layer, one $N$-vector per channel, an $(H, N)$ block of numbers per\n",
    "layer. The `step` method below advances that block by one token; the\n",
    "matching method on `S4DBlock` threads the state through the layer while\n",
    "the gated MLP, which acts position-wise, needs no state at all. Both go\n",
    "into the `d2l` library next to their classes."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-inference-one-token-at-a-time-1"
   },
   "source": [
    "@d2l.add_to_class(S4D)\n",
    "def step(self, u, x=None):\n",
    "    \"\"\"Advance one token: u is (batch, H); x is the (batch, H, N) state.\"\"\"\n",
    "    a = -torch.exp(self.log_a)\n",
    "    da = torch.exp(self.log_dt) * a\n",
    "    a_bar = torch.exp(da)                             # Same ZOH as forward\n",
    "    b_bar = torch.expm1(da) / a\n",
    "    if x is None:\n",
    "        x = u.new_zeros(*u.shape, a_bar.shape[-1])\n",
    "    x = a_bar * x + b_bar * u.unsqueeze(-1)           # One recurrence step\n",
    "    return (x * self.C).sum(-1) + self.D * u, x\n",
    "\n",
    "@d2l.add_to_class(S4DBlock)\n",
    "def step(self, X, x=None):\n",
    "    \"\"\"Advance the block one token; only the SSM carries state.\"\"\"\n",
    "    y, x = self.ssm.step(self.ln1(X), x)\n",
    "    X = X + y\n",
    "    Y = self.ln2(X)\n",
    "    return X + self.W_o(self.W_v(Y) * torch.sigmoid(self.W_g(Y))), x"
   ],
   "execution_count": 18,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-14 fw=all -->\n",
    "\n",
    "A claim of equivalence should be checked on a model whose answers we care\n",
    "about, so we replay validation images through the *trained* classifier's\n",
    "encoder both ways: all 784 pixels at once through the scan, then one\n",
    "pixel at a time through `step`, carrying one state per block. The two\n",
    "schedules perform the same arithmetic in different association orders,\n",
    "so they can differ only by float32 rounding, and over 784 steps of\n",
    "trained, near-unit decays that rounding accumulates; the comparison\n",
    "must therefore be *relative* to the activation scale. (Checked\n",
    "against exact float64 stepping, each path sits about ten times farther\n",
    "from the true answer than the two sit from each other.) The assertion\n",
    "is a regression guard as much as a demonstration: a genuine mismatch\n",
    "between `forward` and `step` shows up at relative errors near one, five\n",
    "orders of magnitude above this gate."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-inference-one-token-at-a-time-2"
   },
   "source": [
    "X_val = next(iter(data.val_dataloader()))[0][:64].to(d2l.try_gpu())\n",
    "with torch.no_grad():\n",
    "    seq = s4d.emb(X_val.reshape(X_val.shape[0], -1, 1).movedim(1, 0))\n",
    "    y_scan = s4d.encoder(seq)                    # All 784 steps at once\n",
    "    states, ys = [None] * len(s4d.encoder), []\n",
    "    for t in range(seq.shape[0]):                # One pixel at a time\n",
    "        X, new_states = seq[t], []\n",
    "        for blk, x in zip(s4d.encoder, states):\n",
    "            X, x = blk.step(X, x)\n",
    "            new_states.append(x)\n",
    "        states, ys = new_states, ys + [X]\n",
    "err = float((torch.stack(ys) - y_scan).abs().max())\n",
    "scale = float(y_scan.abs().max())\n",
    "print(f'stepped vs scanned: deviation {err:.2e} '\n",
    "      f'on activations of scale {scale:.0f}: relative {err / scale:.2e}')\n",
    "assert err < 1e-3 * scale"
   ],
   "execution_count": 19,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "stepped vs scanned: deviation 4.41e-04 on activations of scale 71: relative 6.21e-06\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-15 fw=all -->\n",
    "\n",
    "Now the stopwatch. Picture the model serving a stream, a batch of 32\n",
    "sequences arriving one token at a time, and consider the cost of\n",
    "absorbing the next token once $P$ tokens have already passed. Without a\n",
    "carried state we would do what our training path does: re-run the scan\n",
    "over all $P + 1$ positions, paying for the entire history again. With\n",
    "the state, the past is already summarized and one `step` suffices. We\n",
    "time both as the prefix grows."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ssm-inference-one-token-at-a-time-3"
   },
   "source": [
    "def one_step(X, states):\n",
    "    with torch.no_grad():\n",
    "        for i, blk in enumerate(s4d.encoder):\n",
    "            X, states[i] = blk.step(X, states[i])\n",
    "    return X\n",
    "\n",
    "def rerun(seq):\n",
    "    with torch.no_grad():\n",
    "        return s4d.encoder(seq)\n",
    "\n",
    "lengths, times = [256, 1024, 4096, 16384], [[], []]\n",
    "states = [None] * len(s4d.encoder)\n",
    "for P in lengths:\n",
    "    seq = torch.randn(P, 32, 48, device=device)\n",
    "    one_step(seq[-1], states)                    # Materialize the state\n",
    "    times[0].append(wall_clock(one_step, seq[-1], states) * 1e3)\n",
    "    times[1].append(wall_clock(rerun, seq) * 1e3)\n",
    "d2l.plot(lengths, times, 'prefix length', 'ms per new token',\n",
    "         legend=['recurrent step', 're-run the prefix'],\n",
    "         xscale='log', yscale='log', figsize=(5, 3))\n",
    "print(f'carried state: {sum(4 * x.numel() for x in states) // 32} '\n",
    "      f'bytes per sequence, at every prefix length')"
   ],
   "execution_count": 20,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "carried state: 1536 bytes per sequence, at every prefix length\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"331.678125pt\" height=\"211.07625pt\" viewBox=\"0 0 331.678125 211.07625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-20T01:13:37.558445</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M -0 211.07625 \nL 331.678125 211.07625 \nL 331.678125 0 \nL -0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 45.478125 173.52 \nL 324.478125 173.52 \nL 324.478125 7.2 \nL 45.478125 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path d=\"M 141.259006 173.52 \nL 141.259006 7.2 \n\" clip-path=\"url(#p93525ea2fb)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path id=\"m58346d8714\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m58346d8714\" x=\"141.259006\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- $\\mathdefault{10^{3}}$ -->\n      <g transform=\"translate(132.459006 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \nQ 3050 2419 3304 2112 \nQ 3559 1806 3559 1356 \nQ 3559 666 3084 287 \nQ 2609 -91 1734 -91 \nQ 1441 -91 1130 -33 \nQ 819 25 488 141 \nL 488 750 \nQ 750 597 1062 519 \nQ 1375 441 1716 441 \nQ 2309 441 2620 675 \nQ 2931 909 2931 1356 \nQ 2931 1769 2642 2001 \nQ 2353 2234 1838 2234 \nL 1294 2234 \nL 1294 2753 \nL 1863 2753 \nQ 2328 2753 2575 2939 \nQ 2822 3125 2822 3475 \nQ 2822 3834 2567 4026 \nQ 2313 4219 1838 4219 \nQ 1578 4219 1281 4162 \nQ 984 4106 628 3988 \nL 628 4550 \nQ 988 4650 1302 4700 \nQ 1616 4750 1894 4750 \nQ 2613 4750 3031 4423 \nQ 3450 4097 3450 3541 \nQ 3450 3153 3228 2886 \nQ 3006 2619 2597 2516 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(128.203125 39.046875) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path d=\"M 281.685966 173.52 \nL 281.685966 7.2 \n\" clip-path=\"url(#p93525ea2fb)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#m58346d8714\" x=\"281.685966\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- $\\mathdefault{10^{4}}$ -->\n      <g transform=\"translate(272.885966 188.118438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n       <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(128.203125 38.965625) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <defs>\n       <path id=\"m5101668a0e\" d=\"M 0 0 \nL 0 2 \n\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"67.832733\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"85.3775\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_7\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"98.986279\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"110.105461\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_7\">\n     <g id=\"line2d_9\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"119.506595\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_8\">\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"127.650228\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_9\">\n     <g id=\"line2d_11\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"134.833421\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_10\">\n     <g id=\"line2d_12\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"183.531733\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_11\">\n     <g id=\"line2d_13\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"208.259694\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_12\">\n     <g id=\"line2d_14\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"225.804461\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_13\">\n     <g id=\"line2d_15\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"239.413239\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_14\">\n     <g id=\"line2d_16\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"250.532421\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_15\">\n     <g id=\"line2d_17\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"259.933555\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_16\">\n     <g id=\"line2d_18\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"268.077188\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_17\">\n     <g id=\"line2d_19\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"275.260381\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_18\">\n     <g id=\"line2d_20\">\n      <g>\n       <use xlink:href=\"#m5101668a0e\" x=\"323.958694\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_3\">\n     <!-- prefix length -->\n     <g transform=\"translate(153.149219 201.796563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \nL 1159 -1331 \nL 581 -1331 \nL 581 3500 \nL 1159 3500 \nL 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nz\nM 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \nL 2375 4384 \nL 1825 4384 \nQ 1516 4384 1395 4259 \nQ 1275 4134 1275 3809 \nL 1275 3500 \nL 2222 3500 \nL 2222 3053 \nL 1275 3053 \nL 1275 0 \nL 697 0 \nL 697 3053 \nL 147 3053 \nL 147 3500 \nL 697 3500 \nL 697 3744 \nQ 697 4328 969 4595 \nQ 1241 4863 1831 4863 \nL 2375 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \nL 2247 1797 \nL 3578 0 \nL 2900 0 \nL 1881 1375 \nL 863 0 \nL 184 0 \nL 1544 1831 \nL 300 3500 \nL 978 3500 \nL 1906 2253 \nL 2834 3500 \nL 3513 3500 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \nQ 2906 2416 2648 2759 \nQ 2391 3103 1925 3103 \nQ 1463 3103 1205 2759 \nQ 947 2416 947 1791 \nQ 947 1169 1205 825 \nQ 1463 481 1925 481 \nQ 2391 481 2648 825 \nQ 2906 1169 2906 1791 \nz\nM 3481 434 \nQ 3481 -459 3084 -895 \nQ 2688 -1331 1869 -1331 \nQ 1566 -1331 1297 -1286 \nQ 1028 -1241 775 -1147 \nL 775 -588 \nQ 1028 -725 1275 -790 \nQ 1522 -856 1778 -856 \nQ 2344 -856 2625 -561 \nQ 2906 -266 2906 331 \nL 2906 616 \nQ 2728 306 2450 153 \nQ 2172 0 1784 0 \nQ 1141 0 747 490 \nQ 353 981 353 1791 \nQ 353 2603 747 3093 \nQ 1141 3584 1784 3584 \nQ 2172 3584 2450 3431 \nQ 2728 3278 2906 2969 \nL 2906 3500 \nL 3481 3500 \nL 3481 434 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-70\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(63.476562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(102.339844 0)\"/>\n      <use xlink:href=\"#DejaVuSans-66\" transform=\"translate(163.863281 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(199.068359 0)\"/>\n      <use xlink:href=\"#DejaVuSans-78\" transform=\"translate(226.851562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(286.03125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(317.818359 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(345.601562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(407.125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-67\" transform=\"translate(470.503906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(533.980469 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(573.189453 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_21\">\n      <path d=\"M 45.478125 153.0939 \nL 324.478125 153.0939 \n\" clip-path=\"url(#p93525ea2fb)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_22\">\n      <defs>\n       <path id=\"m39d7658d23\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m39d7658d23\" x=\"45.478125\" y=\"153.0939\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <g transform=\"translate(20.878125 156.893119) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_23\">\n      <path d=\"M 45.478125 83.850011 \nL 324.478125 83.850011 \n\" clip-path=\"url(#p93525ea2fb)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_24\">\n      <g>\n       <use xlink:href=\"#m39d7658d23\" x=\"45.478125\" y=\"83.850011\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- $\\mathdefault{10^{1}}$ -->\n      <g transform=\"translate(20.878125 87.649229) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_25\">\n      <path d=\"M 45.478125 14.606121 \nL 324.478125 14.606121 \n\" clip-path=\"url(#p93525ea2fb)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_26\">\n      <g>\n       <use xlink:href=\"#m39d7658d23\" x=\"45.478125\" y=\"14.606121\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- $\\mathdefault{10^{2}}$ -->\n      <g transform=\"translate(20.878125 18.40534) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_27\">\n      <defs>\n       <path id=\"m295843d6c2\" d=\"M 0 0 \nL -2 0 \n\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"168.455571\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_28\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"163.819915\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_29\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"159.804327\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_30\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"156.262327\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_8\">\n     <g id=\"line2d_31\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"132.249413\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_9\">\n     <g id=\"line2d_32\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"120.056169\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_10\">\n     <g id=\"line2d_33\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"111.404925\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_11\">\n     <g id=\"line2d_34\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"104.694499\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_12\">\n     <g id=\"line2d_35\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"99.211681\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_13\">\n     <g id=\"line2d_36\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"94.576025\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_14\">\n     <g id=\"line2d_37\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"90.560437\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_15\">\n     <g id=\"line2d_38\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"87.018437\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_16\">\n     <g id=\"line2d_39\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"63.005523\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_17\">\n     <g id=\"line2d_40\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"50.812279\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_18\">\n     <g id=\"line2d_41\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"42.161035\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_19\">\n     <g id=\"line2d_42\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"35.450609\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_20\">\n     <g id=\"line2d_43\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"29.967791\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_21\">\n     <g id=\"line2d_44\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"25.332135\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_22\">\n     <g id=\"line2d_45\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"21.316547\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_23\">\n     <g id=\"line2d_46\">\n      <g>\n       <use xlink:href=\"#m295843d6c2\" x=\"45.478125\" y=\"17.774548\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- ms per new token -->\n     <g transform=\"translate(14.798438 135.221719) rotate(-90) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \nQ 3544 3216 3844 3400 \nQ 4144 3584 4550 3584 \nQ 5097 3584 5394 3201 \nQ 5691 2819 5691 2113 \nL 5691 0 \nL 5113 0 \nL 5113 2094 \nQ 5113 2597 4934 2840 \nQ 4756 3084 4391 3084 \nQ 3944 3084 3684 2787 \nQ 3425 2491 3425 1978 \nL 3425 0 \nL 2847 0 \nL 2847 2094 \nQ 2847 2600 2669 2842 \nQ 2491 3084 2119 3084 \nQ 1678 3084 1418 2786 \nQ 1159 2488 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1356 3278 1631 3431 \nQ 1906 3584 2284 3584 \nQ 2666 3584 2933 3390 \nQ 3200 3197 3328 2828 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-77\" d=\"M 269 3500 \nL 844 3500 \nL 1563 769 \nL 2278 3500 \nL 2956 3500 \nL 3675 769 \nL 4391 3500 \nL 4966 3500 \nL 4050 0 \nL 3372 0 \nL 2619 2869 \nL 1863 0 \nL 1184 0 \nL 269 3500 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6b\" d=\"M 581 4863 \nL 1159 4863 \nL 1159 1991 \nL 2875 3500 \nL 3609 3500 \nL 1753 1863 \nL 3688 0 \nL 2938 0 \nL 1159 1709 \nL 1159 0 \nL 581 0 \nL 581 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-6d\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(97.412109 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(149.511719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(181.298828 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(244.775391 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(306.298828 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(347.412109 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(379.199219 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(442.578125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-77\" transform=\"translate(504.101562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(585.888672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(617.675781 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(656.884766 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6b\" transform=\"translate(718.066406 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(772.351562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(833.875 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_47\">\n    <path d=\"M 58.159943 165.307255 \nL 142.705398 165.635483 \nL 227.250852 165.96 \nL 311.796307 165.382682 \n\" clip-path=\"url(#p93525ea2fb)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_48\">\n    <path d=\"M 58.159943 133.512133 \nL 142.705398 127.969349 \nL 227.250852 61.025348 \nL 311.796307 14.76 \n\" clip-path=\"url(#p93525ea2fb)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 45.478125 173.52 \nL 45.478125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 324.478125 173.52 \nL 324.478125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 45.478125 173.52 \nL 324.478125 173.52 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 45.478125 7.2 \nL 324.478125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 52.478125 44.55625 \nL 166.282813 44.55625 \nQ 168.282813 44.55625 168.282813 42.55625 \nL 168.282813 14.2 \nQ 168.282813 12.2 166.282813 12.2 \nL 52.478125 12.2 \nQ 50.478125 12.2 50.478125 14.2 \nL 50.478125 42.55625 \nQ 50.478125 44.55625 52.478125 44.55625 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_49\">\n     <path d=\"M 54.478125 20.298438 \nL 64.478125 20.298438 \nL 74.478125 20.298438 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_8\">\n     <!-- recurrent step -->\n     <g transform=\"translate(82.478125 23.798438) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \nL 3122 2828 \nQ 2878 2963 2633 3030 \nQ 2388 3097 2138 3097 \nQ 1578 3097 1268 2742 \nQ 959 2388 959 1747 \nQ 959 1106 1268 751 \nQ 1578 397 2138 397 \nQ 2388 397 2633 464 \nQ 2878 531 3122 666 \nL 3122 134 \nQ 2881 22 2623 -34 \nQ 2366 -91 2075 -91 \nQ 1284 -91 818 406 \nQ 353 903 353 1747 \nQ 353 2603 823 3093 \nQ 1294 3584 2113 3584 \nQ 2378 3584 2631 3529 \nQ 2884 3475 3122 3366 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-75\" d=\"M 544 1381 \nL 544 3500 \nL 1119 3500 \nL 1119 1403 \nQ 1119 906 1312 657 \nQ 1506 409 1894 409 \nQ 2359 409 2629 706 \nQ 2900 1003 2900 1516 \nL 2900 3500 \nL 3475 3500 \nL 3475 0 \nL 2900 0 \nL 2900 538 \nQ 2691 219 2414 64 \nQ 2138 -91 1772 -91 \nQ 1169 -91 856 284 \nQ 544 659 544 1381 \nz\nM 1991 3584 \nL 1991 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-72\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(38.863281 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(100.386719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-75\" transform=\"translate(155.367188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(218.746094 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(258.109375 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(296.972656 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(358.496094 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(421.875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(461.083984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(492.871094 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(544.970703 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(584.179688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(645.703125 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_50\">\n     <path d=\"M 54.478125 34.976562 \nL 64.478125 34.976562 \nL 74.478125 34.976562 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_9\">\n     <!-- re-run the prefix -->\n     <g transform=\"translate(82.478125 38.476562) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-2d\" d=\"M 313 2009 \nL 1997 2009 \nL 1997 1497 \nL 313 1497 \nL 313 2009 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-72\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(38.863281 0)\"/>\n      <use xlink:href=\"#DejaVuSans-2d\" transform=\"translate(100.386719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(136.470703 0)\"/>\n      <use xlink:href=\"#DejaVuSans-75\" transform=\"translate(177.583984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(240.962891 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(304.341797 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(336.128906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(375.337891 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(438.716797 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(500.240234 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(532.027344 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(595.503906 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(634.367188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-66\" transform=\"translate(695.890625 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(731.095703 0)\"/>\n      <use xlink:href=\"#DejaVuSans-78\" transform=\"translate(758.878906 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p93525ea2fb\">\n   <rect x=\"45.478125\" y=\"7.2\" width=\"279\" height=\"166.32\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=ssm-md-16 fw=all -->\n",
    "\n",
    "The two curves have the shapes the theory promises. Re-running the\n",
    "prefix costs more the longer the history, linearly once the sequence is\n",
    "long enough for real work to dominate per-call overhead; the recurrent\n",
    "step costs the same at every prefix length. How wide the gap has grown\n",
    "by our longest prefix depends on the framework's per-call overhead\n",
    "(severalfold to two orders of magnitude in our runs), and it keeps\n",
    "widening for as long as the prefix, and the memory holding it, keep\n",
    "growing. At this toy width the absolute\n",
    "times are launch-overhead-bound, so read the shapes, not the\n",
    "milliseconds.\n",
    "\n",
    "The printed number is the punchline's other half. A transformer\n",
    "generating with a KV cache also avoids re-running its prefix, but it\n",
    "does so by *storing* the past: its cache grows linearly with context, at\n",
    "a rate we measured in that section (tens of kibibytes per\n",
    "token for even a small model), and serving budgets are set by that\n",
    "growth. The recurrent model's entire memory of an arbitrarily long\n",
    "history is the $(H, N)$ state per layer printed above, about a\n",
    "kibibyte here, the same at token 100 as at token 100,000. This\n",
    "constant-memory inference is why recurrent layers are back inside\n",
    "production language models, and it is the prize for which linear\n",
    "attention gave up its softmax in that section. What\n",
    "the fixed-size state *costs*, and when it is worth paying, is a question\n",
    "this chapter returns to once the selective models are on the table.\n",
    "\n",
    "## Summary\n",
    "\n",
    "Recurrence buys constant-memory inference but serializes training; this\n",
    "section removed the serialization by making the state update linear. The\n",
    "minGRU shows the deletion route: gates computed from the input alone turn\n",
    "the GRU into an elementwise affine recurrence\n",
    "$\\mathbf{h}_t = \\mathbf{a}_t \\odot \\mathbf{h}_{t-1} + \\mathbf{b}_t$, which\n",
    "an associative combine evaluates in $O(\\log T)$ parallel rounds, at\n",
    "language-modeling quality on par with the GRU. State space models reach\n",
    "the same recurrence from first principles: discretizing continuous linear\n",
    "dynamics yields the ZOH formulas, in which the learnable step size\n",
    "$\\Delta$ *is* an update gate and stability holds by construction, with\n",
    "eigenvalues pinned inside the unit circle by parameterization rather than\n",
    "by clipping (that section). Time-invariance\n",
    "makes the same model a recurrence, a convolution, and an ODE at once.\n",
    "HiPPO supplies dynamics under which a fixed-size state is a\n",
    "provably good compression of the whole past — a time-varying system\n",
    "whose matrix, frozen, initializes the LTI S4 family, with the diagonal\n",
    "S4D as the empirically vindicated simplification we implement.\n",
    "Assembled into residual\n",
    "blocks, the S4D trains reliably on 784-step sequential image\n",
    "classification where a plainly initialized,\n",
    "parameter-matched LSTM succeeds only on a lucky draw, and it holds its\n",
    "accuracy to within a few points when read from the final state alone,\n",
    "the retention diagnostic. Finally we ran the trained model\n",
    "the other way, one token at a time: stepping agrees with the scan to\n",
    "floating-point accuracy, costs the same at any prefix length, and\n",
    "carries a state a few hundred bytes per layer where a transformer's KV\n",
    "cache grows with the length of the context (that section).\n",
    "\n",
    "Everything here, though, is *linear and time-invariant*: the kernel\n",
    "$\\bar{\\mathbf{K}}$, and with it the decision of what to remember, is fixed\n",
    "before the model ever sees the input. An S4D *layer* cannot read a token\n",
    "and decide that *this one* matters while its neighbor is noise: its\n",
    "gates, unlike the LSTM's, do not look at the data as it flows past, so\n",
    "the state-update operator is in this precise sense content-blind. The\n",
    "qualifier matters: the blocks *around* each SSM are input-dependent, and\n",
    "a deep stack can modulate between layers what it cannot modulate within\n",
    "the state update — which is why such models earn partial credit, though\n",
    "not success, on the selectivity task of the next section. Curing the\n",
    "limitation at the operator level, and keeping the\n",
    "scan, is the subject of that section.\n",
    "\n",
    "**What the experiments show, and what they do not.** The scan-vs-loop\n",
    "and step-vs-rerun benchmarks establish *shapes* (logarithmic against\n",
    "linear, flat against growing), measured with teaching-grade kernels on\n",
    "one GPU; they are not production throughput numbers. The equivalence\n",
    "checks (scan against loop, stepping against scanning) are identities up\n",
    "to float rounding, the strongest claims here. The classifier is one\n",
    "seeded run per framework per readout: it shows that the multiscale\n",
    "initialization trains reliably where the plainly initialized LSTM is at\n",
    "the mercy of its draw, not a general architecture ranking.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. [short-code] *Effective memory.* For a scalar recurrence\n",
    "   $h_t = \\bar{a} h_{t-1} + u_t$, define the effective memory horizon as\n",
    "   the lag $k$ at which the influence $\\bar{a}^k$ of an input drops below\n",
    "   $0.01$. Plot the horizon as a function of $\\bar{a} \\in (0.9, 0.999)$\n",
    "   and as a function of $\\Delta$ under $\\bar{a} = e^{-\\Delta}$. How must\n",
    "   $\\Delta$ scale for the model to remember 10 times longer, and how does\n",
    "   this connect the step-size-as-gate box to the eigenvalue analysis of\n",
    "   that section?\n",
    "1. [short-code] *ZOH versus Euler.* The Euler discretization uses\n",
    "   $\\bar{\\mathbf{A}} = \\mathbf{I} + \\Delta \\mathbf{A}$ and\n",
    "   $\\bar{\\mathbf{B}} = \\Delta \\mathbf{B}$. For the scalar system\n",
    "   $a = -1, b = 1$ driven by $u_t = \\sin(0.1 t)$, compare Euler and ZOH\n",
    "   outputs against a fine-grained reference solution for\n",
    "   $\\Delta \\in \\{0.1, 0.5, 1.0, 2.5\\}$. Which discretization degrades\n",
    "   gracefully, and what goes visibly wrong with Euler at $\\Delta = 2.5$?\n",
    "   (Check the eigenvalue $|1 + \\Delta a|$ first.)\n",
    "1. [short-code] *Kernels as fingerprints.* Materialize and plot the\n",
    "   convolution kernel\n",
    "   the equation for (a) our diagonal initialization\n",
    "   $a_n = -(n+1)$ with $\\mathbf{B} = \\mathbf{1}$ and random $\\mathbf{C}$,\n",
    "   and (b) the LTI system that S4 initializes, the full (non-diagonal)\n",
    "   HiPPO matrix the equation frozen into the equation, using\n",
    "   matrix powers. How do the kernels differ in shape and in how much mass\n",
    "   they place on the distant past?\n",
    "1. [extended] *minLSTM.* Apply the minGRU recipe to the LSTM of that section:\n",
    "   make all gates functions of the input only, drop the output gate's\n",
    "   dependence on the state, and write the cell update\n",
    "   $\\mathbf{C}_t = \\mathbf{F}_t \\odot \\mathbf{C}_{t-1} + \\mathbf{I}_t\n",
    "   \\odot \\tilde{\\mathbf{C}}_t$ as an affine recurrence\n",
    "   the equation. Implement it with `associative_scan`\n",
    "   and train it with this section's language-modeling recipe. Does the\n",
    "   extra gate help? (Compare @Feng.Tung.Ahmed.ea.2024, who also\n",
    "   normalize the two gates to sum to one.)\n",
    "1. [short-code] *The candidate nonlinearity.* Our minGRU keeps a $\\tanh$ on the\n",
    "   candidate $\\tilde{\\mathbf{H}}_t$, while @Feng.Tung.Ahmed.ea.2024\n",
    "   use a purely linear candidate. Remove the $\\tanh$ and retrain. What\n",
    "   happens to training stability and final perplexity, and why does an\n",
    "   unbounded candidate interact badly with a convex-combination update?\n",
    "1. [short-code] *Streaming classification.* The stepped path of that section\n",
    "   makes anytime prediction almost free: the classifier head only needs the\n",
    "   running mean of the encoder features seen so far, which you can carry as\n",
    "   one more piece of constant-size state. Stream validation images pixel by\n",
    "   pixel, classify after every step, and plot accuracy as a function of\n",
    "   pixels seen. After how many of the 784 pixels does the model commit to\n",
    "   its final answer, and does the shape of that curve match what mean\n",
    "   pooling would lead you to expect?\n",
    "1. [short-code] *State passing.* The scan assumes $\\mathbf{h}_0 = \\mathbf{0}$, and\n",
    "   `MinGRU` folds a carried state into $\\mathbf{b}_1$. Use this to process\n",
    "   a length-$T$ sequence in $T/\\tau$ chunks of length $\\tau$, passing the\n",
    "   final state of each chunk into the next, and verify that the outputs\n",
    "   match the unchunked scan to floating-point tolerance (chunking\n",
    "   reassociates the arithmetic, so compare against a tolerance scaled to\n",
    "   the dtype, not bitwise). Why might chunking be useful at\n",
    "   training time? (Think memory, and recall truncated BPTT from\n",
    "   that section.)\n",
    "\n",
    "[Discussions](https://d2l.discourse.group/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-pytorch",
   "display_name": "d2l (PyTorch)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
