{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=deltanet-md-1 fw=all -->\n",
    "\n",
    "# DeltaNet: Memory That Edits\n",
    "\n",
    "Read down the write column of the table one more time. Seven\n",
    "models, three research lineages, and every write is an addition: whatever\n",
    "$\\mathbf{k}_t \\mathbf{v}_t^\\top$ says, the memory takes it on top of what is\n",
    "already there. The previous section measured what crowding does to such a\n",
    "memory and built the decay ladder to manage it, but decay is indiscriminate:\n",
    "to weaken one stale entry, the model must weaken everything. This section\n",
    "changes the write rule instead. The fix, proposed for exactly this failure\n",
    "by @Schlag.Irie.Schmidhuber.2021, is to *read before writing*: look\n",
    "up what the memory currently returns for $\\mathbf{k}_t$, subtract it, and\n",
    "write only the correction. We first isolate the failure the additive write\n",
    "cannot escape, with a memory whose entries are *overwritten* rather than\n",
    "merely accumulated. We then derive the corrective write, the *delta rule*,\n",
    "and find that it is one step of gradient descent on a recall loss, taken\n",
    "inside the forward pass. A memory whose transition is no longer diagonal\n",
    "resists the chunked training schedule of that section, so we\n",
    "rebuild that schedule around a triangular solve; with a decay gate added\n",
    "back we arrive at Gated DeltaNet, the cell inside several current\n",
    "production models, and train it on our language-modeling scoreboard.\n",
    "Finally we ask what the new transition can *compute*, and meet a capability\n",
    "that no model of that section has: state tracking.\n",
    "\n",
    "*Prerequisites: the matrix-state recurrence and its capacity law\n",
    "(that section, the equation), the chunked\n",
    "training schedule of that section, and the optimizer\n",
    "vocabulary of that section, in particular what \"decoupled\"\n",
    "weight decay means (that section). The training cells use a GPU\n",
    "when one is available; the identity checks run anywhere.*"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "deltanet-deltanet-memory-that-edits"
   },
   "source": [
    "%matplotlib inline\n",
    "from d2l import torch as d2l\n",
    "import numpy as np\n",
    "import time\n",
    "import torch\n",
    "from torch import nn\n",
    "import torch.nn.functional as F"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=deltanet-md-2 fw=all -->\n",
    "\n",
    "## The Trouble with Adding\n",
    "\n",
    "The capacity law of that section blamed *crowding*: store\n",
    "more unit-norm pairs than the key dimension and reads drown in the\n",
    "interference term of the equation. There is a second\n",
    "failure, and it survives even when crowding is engineered away. Suppose the\n",
    "keys are few and mutually orthogonal, so that interference between\n",
    "*different* keys is exactly zero, but the stream keeps *re-binding* old\n",
    "keys to new values: today's stock price under the same ticker, a variable\n",
    "reassigned in code, a character's location as a story moves on. An additive\n",
    "memory handles re-binding badly, because addition cannot replace. After $R$\n",
    "writes to the same key $\\mathbf{k}$, the Hebbian state holds\n",
    "\n",
    "$$\n",
    "\\mathbf{S}^\\top \\mathbf{k}\n",
    "= \\mathbf{v}^{(1)} + \\mathbf{v}^{(2)} + \\cdots + \\mathbf{v}^{(R)}:\n",
    "$$\n",
    "\n",
    "every value ever filed under $\\mathbf{k}$, superposed, with the current one\n",
    "enjoying no privilege over the stale ones. A reader who wants the *latest*\n",
    "binding is reduced to guessing among $R$ candidates.\n",
    "\n",
    "The corrective write repairs exactly this. Before writing, retrieve what\n",
    "the memory currently returns for the key; subtract it; then write. For\n",
    "now we state the repair operationally, with everything it does visible in\n",
    "the code below, and derive it properly in the next section:\n",
    "\n",
    "$$\n",
    "\\mathbf{S}_t\n",
    "= \\mathbf{S}_{t-1}\n",
    "+ \\mathbf{k}_t \\big(\\mathbf{v}_t - \\mathbf{S}_{t-1}^\\top \\mathbf{k}_t\\big)^\\top .\n",
    "$$\n",
    "\n",
    "For a unit-norm key this *replaces* the stored value: reading back\n",
    "immediately after the write returns $\\mathbf{v}_t$ exactly, however many\n",
    "bindings preceded it. The experiment below makes the contrast quantitative.\n",
    "We hand-build both memories, with no training anywhere: $8$ orthonormal\n",
    "keys in $64$ dimensions (orthonormal so that cross-key interference is zero\n",
    "by construction and staleness is the *only* effect in view; unit-norm keys\n",
    "are also what the equation assumes), values drawn from a\n",
    "codebook of $32$ unit vectors, $R$ writes per key in shuffled order. After\n",
    "the stream, we read every key and decode by the nearest codebook entry,\n",
    "scoring a hit only if the memory returns the key's *latest* value."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "deltanet-the-trouble-with-adding"
   },
   "source": [
    "rng = np.random.default_rng(0)\n",
    "\n",
    "def overwrite_recall(rule, reps, num_keys=8, num_values=32, d=64, trials=200):\n",
    "    \"\"\"Recall of the latest value bound to each key, after R writes per key.\"\"\"\n",
    "    accs = []\n",
    "    for R in reps:\n",
    "        correct = total = 0\n",
    "        for _ in range(trials):\n",
    "            K = np.linalg.qr(rng.standard_normal((d, num_keys)))[0].T\n",
    "            codebook = rng.standard_normal((num_values, d))\n",
    "            codebook /= np.linalg.norm(codebook, axis=1, keepdims=True)\n",
    "            writes = [(k, rng.integers(num_values)) for k in range(num_keys)\n",
    "                      for _ in range(R)]\n",
    "            rng.shuffle(writes)\n",
    "            latest, S = {}, np.zeros((d, d))\n",
    "            for kk, vv in writes:\n",
    "                k, v = K[kk], codebook[vv]\n",
    "                latest[kk] = vv\n",
    "                if rule == 'delta':\n",
    "                    S = S + np.outer(k, v - S.T @ k)   # Erase, then write\n",
    "                else:\n",
    "                    S = S + np.outer(k, v)             # Superimpose\n",
    "            for kk in range(num_keys):\n",
    "                read = S.T @ K[kk]\n",
    "                correct += (codebook @ read).argmax() == latest[kk]\n",
    "                total += 1\n",
    "        accs.append(correct / total)\n",
    "    return accs\n",
    "\n",
    "reps = [1, 2, 4, 8]\n",
    "hebbian, delta = (overwrite_recall(r, reps) for r in ('hebbian', 'delta'))\n",
    "print(f\"{'writes/key':>10} {'Hebbian':>9} {'delta':>7}\")\n",
    "for row in zip(reps, hebbian, delta):\n",
    "    print('{:>10} {:>9.3f} {:>7.3f}'.format(*row))\n",
    "assert min(delta) > 0.99, 'delta write must recall the latest value exactly'\n",
    "assert hebbian[-1] < 0.4, 'Hebbian recall must collapse under overwrites'\n",
    "d2l.plot(reps, [hebbian, delta], 'writes per key', 'recall of latest value',\n",
    "         legend=['Hebbian write', 'delta write'], fmts=('-o', '-s'))"
   ],
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "writes/key   Hebbian   delta\n         1     1.000   1.000\n         2     0.510   1.000\n         4     0.299   1.000\n         8     0.217   1.000\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"246.28125pt\" height=\"183.35625pt\" viewBox=\"0 0 246.28125 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-20T01:04:52.969038</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 183.35625 \nL 246.28125 183.35625 \nL 246.28125 0 \nL 0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 43.78125 145.8 \nL 239.08125 145.8 \nL 239.08125 7.2 \nL 43.78125 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path d=\"M 78.022159 145.8 \nL 78.022159 7.2 \n\" clip-path=\"url(#p6ff747f7d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path id=\"m576e2da8ff\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m576e2da8ff\" x=\"78.022159\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 2 -->\n      <g transform=\"translate(74.840909 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-32\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path d=\"M 128.749432 145.8 \nL 128.749432 7.2 \n\" clip-path=\"url(#p6ff747f7d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#m576e2da8ff\" x=\"128.749432\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 4 -->\n      <g transform=\"translate(125.568182 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-34\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path d=\"M 179.476705 145.8 \nL 179.476705 7.2 \n\" clip-path=\"url(#p6ff747f7d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#m576e2da8ff\" x=\"179.476705\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 6 -->\n      <g transform=\"translate(176.295455 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-36\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_7\">\n      <path d=\"M 230.203977 145.8 \nL 230.203977 7.2 \n\" clip-path=\"url(#p6ff747f7d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#m576e2da8ff\" x=\"230.203977\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 8 -->\n      <g transform=\"translate(227.022727 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-38\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_5\">\n     <!-- writes per key -->\n     <g transform=\"translate(106.021875 174.076563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-77\" d=\"M 269 3500 \nL 844 3500 \nL 1563 769 \nL 2278 3500 \nL 2956 3500 \nL 3675 769 \nL 4391 3500 \nL 4966 3500 \nL 4050 0 \nL 3372 0 \nL 2619 2869 \nL 1863 0 \nL 1184 0 \nL 269 3500 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \nL 1159 -1331 \nL 581 -1331 \nL 581 3500 \nL 1159 3500 \nL 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nz\nM 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6b\" d=\"M 581 4863 \nL 1159 4863 \nL 1159 1991 \nL 2875 3500 \nL 3609 3500 \nL 1753 1863 \nL 3688 0 \nL 2938 0 \nL 1159 1709 \nL 1159 0 \nL 581 0 \nL 581 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \nQ 1816 -950 1584 -1140 \nQ 1353 -1331 966 -1331 \nL 506 -1331 \nL 506 -850 \nL 844 -850 \nQ 1081 -850 1212 -737 \nQ 1344 -625 1503 -206 \nL 1606 56 \nL 191 3500 \nL 800 3500 \nL 1894 763 \nL 2988 3500 \nL 3597 3500 \nL 2059 -325 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-77\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(81.787109 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(122.900391 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(150.683594 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(189.892578 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(251.416016 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(303.515625 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(335.302734 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(398.779297 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(460.302734 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(501.416016 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6b\" transform=\"translate(533.203125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(587.488281 0)\"/>\n      <use xlink:href=\"#DejaVuSans-79\" transform=\"translate(649.011719 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_9\">\n      <path d=\"M 43.78125 142.317891 \nL 239.08125 142.317891 \n\" clip-path=\"url(#p6ff747f7d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_10\">\n      <defs>\n       <path id=\"mc40ec4352f\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#mc40ec4352f\" x=\"43.78125\" y=\"142.317891\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 0.2 -->\n      <g transform=\"translate(20.878125 146.11711) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \nL 1344 794 \nL 1344 0 \nL 684 0 \nL 684 794 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_11\">\n      <path d=\"M 43.78125 110.113419 \nL 239.08125 110.113419 \n\" clip-path=\"url(#p6ff747f7d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_12\">\n      <g>\n       <use xlink:href=\"#mc40ec4352f\" x=\"43.78125\" y=\"110.113419\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_7\">\n      <!-- 0.4 -->\n      <g transform=\"translate(20.878125 113.912637) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_13\">\n      <path d=\"M 43.78125 77.908946 \nL 239.08125 77.908946 \n\" clip-path=\"url(#p6ff747f7d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_14\">\n      <g>\n       <use xlink:href=\"#mc40ec4352f\" x=\"43.78125\" y=\"77.908946\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.6 -->\n      <g transform=\"translate(20.878125 81.708164) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-36\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_15\">\n      <path d=\"M 43.78125 45.704473 \nL 239.08125 45.704473 \n\" clip-path=\"url(#p6ff747f7d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_16\">\n      <g>\n       <use xlink:href=\"#mc40ec4352f\" x=\"43.78125\" y=\"45.704473\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.8 -->\n      <g transform=\"translate(20.878125 49.503692) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-38\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_17\">\n      <path d=\"M 43.78125 13.5 \nL 239.08125 13.5 \n\" clip-path=\"url(#p6ff747f7d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_18\">\n      <g>\n       <use xlink:href=\"#mc40ec4352f\" x=\"43.78125\" y=\"13.5\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 1.0 -->\n      <g transform=\"translate(20.878125 17.299219) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-2e\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(95.410156 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_11\">\n     <!-- recall of latest value -->\n     <g transform=\"translate(14.798438 127.410156) rotate(-90) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \nL 3122 2828 \nQ 2878 2963 2633 3030 \nQ 2388 3097 2138 3097 \nQ 1578 3097 1268 2742 \nQ 959 2388 959 1747 \nQ 959 1106 1268 751 \nQ 1578 397 2138 397 \nQ 2388 397 2633 464 \nQ 2878 531 3122 666 \nL 3122 134 \nQ 2881 22 2623 -34 \nQ 2366 -91 2075 -91 \nQ 1284 -91 818 406 \nQ 353 903 353 1747 \nQ 353 2603 823 3093 \nQ 1294 3584 2113 3584 \nQ 2378 3584 2631 3529 \nQ 2884 3475 3122 3366 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \nL 2375 4384 \nL 1825 4384 \nQ 1516 4384 1395 4259 \nQ 1275 4134 1275 3809 \nL 1275 3500 \nL 2222 3500 \nL 2222 3053 \nL 1275 3053 \nL 1275 0 \nL 697 0 \nL 697 3053 \nL 147 3053 \nL 147 3500 \nL 697 3500 \nL 697 3744 \nQ 697 4328 969 4595 \nQ 1241 4863 1831 4863 \nL 2375 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-76\" d=\"M 191 3500 \nL 800 3500 \nL 1894 563 \nL 2988 3500 \nL 3597 3500 \nL 2284 0 \nL 1503 0 \nL 191 3500 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-75\" d=\"M 544 1381 \nL 544 3500 \nL 1119 3500 \nL 1119 1403 \nQ 1119 906 1312 657 \nQ 1506 409 1894 409 \nQ 2359 409 2629 706 \nQ 2900 1003 2900 1516 \nL 2900 3500 \nL 3475 3500 \nL 3475 0 \nL 2900 0 \nL 2900 538 \nQ 2691 219 2414 64 \nQ 2138 -91 1772 -91 \nQ 1169 -91 856 284 \nQ 544 659 544 1381 \nz\nM 1991 3584 \nL 1991 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-72\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(38.863281 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(100.386719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(155.367188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(216.646484 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(244.429688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(272.212891 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(304 0)\"/>\n      <use xlink:href=\"#DejaVuSans-66\" transform=\"translate(365.181641 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(400.386719 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(432.173828 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(459.957031 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(521.236328 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(560.445312 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(621.96875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(674.068359 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(713.277344 0)\"/>\n      <use xlink:href=\"#DejaVuSans-76\" transform=\"translate(745.064453 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(804.244141 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(865.523438 0)\"/>\n      <use xlink:href=\"#DejaVuSans-75\" transform=\"translate(893.306641 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(956.685547 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_19\">\n    <path d=\"M 52.658523 13.5 \nL 78.022159 92.400958 \nL 128.749432 126.316294 \nL 230.203977 139.5 \n\" clip-path=\"url(#p6ff747f7d9)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    <defs>\n     <path id=\"ma6c82a6d26\" d=\"M 0 3 \nC 0.795609 3 1.55874 2.683901 2.12132 2.12132 \nC 2.683901 1.55874 3 0.795609 3 0 \nC 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \nC 1.55874 -2.683901 0.795609 -3 0 -3 \nC -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \nC -2.683901 -1.55874 -3 -0.795609 -3 0 \nC -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \nC -1.55874 2.683901 -0.795609 3 0 3 \nz\n\" style=\"stroke: #1f77b4\"/>\n    </defs>\n    <g clip-path=\"url(#p6ff747f7d9)\">\n     <use xlink:href=\"#ma6c82a6d26\" x=\"52.658523\" y=\"13.5\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n     <use xlink:href=\"#ma6c82a6d26\" x=\"78.022159\" y=\"92.400958\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n     <use xlink:href=\"#ma6c82a6d26\" x=\"128.749432\" y=\"126.316294\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n     <use xlink:href=\"#ma6c82a6d26\" x=\"230.203977\" y=\"139.5\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n    </g>\n   </g>\n   <g id=\"line2d_20\">\n    <path d=\"M 52.658523 13.5 \nL 78.022159 13.5 \nL 128.749432 13.5 \nL 230.203977 13.5 \n\" clip-path=\"url(#p6ff747f7d9)\" style=\"fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square\"/>\n    <defs>\n     <path id=\"m4f2a5366c9\" d=\"M -3 3 \nL 3 3 \nL 3 -3 \nL -3 -3 \nz\n\" style=\"stroke: #ff7f0e; stroke-linejoin: miter\"/>\n    </defs>\n    <g clip-path=\"url(#p6ff747f7d9)\">\n     <use xlink:href=\"#m4f2a5366c9\" x=\"52.658523\" y=\"13.5\" style=\"fill: #ff7f0e; stroke: #ff7f0e; stroke-linejoin: miter\"/>\n     <use xlink:href=\"#m4f2a5366c9\" x=\"78.022159\" y=\"13.5\" style=\"fill: #ff7f0e; stroke: #ff7f0e; stroke-linejoin: miter\"/>\n     <use xlink:href=\"#m4f2a5366c9\" x=\"128.749432\" y=\"13.5\" style=\"fill: #ff7f0e; stroke: #ff7f0e; stroke-linejoin: miter\"/>\n     <use xlink:href=\"#m4f2a5366c9\" x=\"230.203977\" y=\"13.5\" style=\"fill: #ff7f0e; stroke: #ff7f0e; stroke-linejoin: miter\"/>\n    </g>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 43.78125 145.8 \nL 43.78125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 239.08125 145.8 \nL 239.08125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 43.78125 145.8 \nL 239.08125 145.8 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 43.78125 7.2 \nL 239.08125 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 130.148438 92.678125 \nL 232.08125 92.678125 \nQ 234.08125 92.678125 234.08125 90.678125 \nL 234.08125 62.321875 \nQ 234.08125 60.321875 232.08125 60.321875 \nL 130.148438 60.321875 \nQ 128.148438 60.321875 128.148438 62.321875 \nL 128.148438 90.678125 \nQ 128.148438 92.678125 130.148438 92.678125 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_21\">\n     <path d=\"M 132.148438 68.420313 \nL 142.148438 68.420313 \nL 152.148438 68.420313 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n     <g>\n      <use xlink:href=\"#ma6c82a6d26\" x=\"142.148438\" y=\"68.420313\" style=\"fill: #1f77b4; stroke: #1f77b4\"/>\n     </g>\n    </g>\n    <g id=\"text_12\">\n     <!-- Hebbian write -->\n     <g transform=\"translate(160.148438 71.920313) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-48\" d=\"M 628 4666 \nL 1259 4666 \nL 1259 2753 \nL 3553 2753 \nL 3553 4666 \nL 4184 4666 \nL 4184 0 \nL 3553 0 \nL 3553 2222 \nL 1259 2222 \nL 1259 0 \nL 628 0 \nL 628 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-62\" d=\"M 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\nM 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2969 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-48\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(75.195312 0)\"/>\n      <use xlink:href=\"#DejaVuSans-62\" transform=\"translate(136.71875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-62\" transform=\"translate(200.195312 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(263.671875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(291.455078 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(352.734375 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(416.113281 0)\"/>\n      <use xlink:href=\"#DejaVuSans-77\" transform=\"translate(447.900391 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(529.6875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(570.800781 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(598.583984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(637.792969 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_22\">\n     <path d=\"M 132.148438 83.098438 \nL 142.148438 83.098438 \nL 152.148438 83.098438 \n\" style=\"fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square\"/>\n     <g>\n      <use xlink:href=\"#m4f2a5366c9\" x=\"142.148438\" y=\"83.098438\" style=\"fill: #ff7f0e; stroke: #ff7f0e; stroke-linejoin: miter\"/>\n     </g>\n    </g>\n    <g id=\"text_13\">\n     <!-- delta write -->\n     <g transform=\"translate(160.148438 86.598438) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \nL 2906 4863 \nL 3481 4863 \nL 3481 0 \nL 2906 0 \nL 2906 525 \nQ 2725 213 2448 61 \nQ 2172 -91 1784 -91 \nQ 1150 -91 751 415 \nQ 353 922 353 1747 \nQ 353 2572 751 3078 \nQ 1150 3584 1784 3584 \nQ 2172 3584 2448 3432 \nQ 2725 3281 2906 2969 \nz\nM 947 1747 \nQ 947 1113 1208 752 \nQ 1469 391 1925 391 \nQ 2381 391 2643 752 \nQ 2906 1113 2906 1747 \nQ 2906 2381 2643 2742 \nQ 2381 3103 1925 3103 \nQ 1469 3103 1208 2742 \nQ 947 2381 947 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-64\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(63.476562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(152.783203 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(191.992188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-20\" transform=\"translate(253.271484 0)\"/>\n      <use xlink:href=\"#DejaVuSans-77\" transform=\"translate(285.058594 0)\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(366.845703 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(407.958984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-74\" transform=\"translate(435.742188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-65\" transform=\"translate(474.951172 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p6ff747f7d9\">\n   <rect x=\"43.78125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=deltanet-md-3 fw=all -->\n",
    "\n",
    "The additive memory collapses just as the superposition argument predicts:\n",
    "recall roughly halves by two writes per key and is down to about $0.2$ by\n",
    "eight, close to picking at random among the values superposed on the key.\n",
    "The corrective write holds recall at $1.000$ at every overwrite count, and\n",
    "not approximately: with $\\beta = 1$ and orthonormal keys, each write\n",
    "replaces the row it addresses and touches nothing else, so the last write\n",
    "always wins. Note what this experiment does *not* show: with only $8$\n",
    "orthogonal keys there is no crowding, so the decay ladder of\n",
    "that section would not help here. Decay discounts by\n",
    "*age*; the failure is indexed by *key*.\n",
    "\n",
    "### Trained Models Hit the Same Ceiling\n",
    "\n",
    "A fair objection: no trained model is forced to use its memory the way we\n",
    "just hand-operated it. Perhaps gradient descent, given an additive memory\n",
    "and a task full of re-bindings, finds some encoding that sidesteps\n",
    "superposition. So we let it try. The task generator below emits sequences\n",
    "with a write phase, $R$ re-bindings of each of $8$ keys in shuffled order,\n",
    "followed by a query phase asking for each key's latest value; the model is\n",
    "trained end-to-end on cross-entropy over its answers."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "deltanet-trained-models-hit-the-same-ceiling-1"
   },
   "source": [
    "def make_task(num_seqs, num_keys=8, num_values=16, R=4, seed=0):\n",
    "    \"\"\"R shuffled writes per key, then one query per key for its latest value.\"\"\"\n",
    "    task_rng = np.random.default_rng(seed)\n",
    "    W = num_keys * R\n",
    "    QUERY = num_values                     # A reserved \"query\" content token\n",
    "    keys = np.zeros((num_seqs, W + num_keys), np.int64)\n",
    "    vals = np.full((num_seqs, W + num_keys), QUERY, np.int64)\n",
    "    tgt = np.full((num_seqs, W + num_keys), -1, np.int64)\n",
    "    for b in range(num_seqs):\n",
    "        ks = np.tile(np.arange(num_keys), R)\n",
    "        vs = task_rng.integers(0, num_values, W)\n",
    "        perm = task_rng.permutation(W)\n",
    "        ks, vs = ks[perm], vs[perm]\n",
    "        keys[b, :W], vals[b, :W] = ks, vs\n",
    "        latest = {int(k): int(v) for k, v in zip(ks, vs)}\n",
    "        qorder = task_rng.permutation(num_keys)\n",
    "        keys[b, W:] = qorder\n",
    "        tgt[b, W:] = [latest[int(k)] for k in qorder]\n",
    "    return keys, vals, tgt                 # tgt = -1 outside the query phase"
   ],
   "execution_count": 3,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=deltanet-md-4 fw=all -->\n",
    "\n",
    "The model is deliberately small: one matrix-memory layer, one MLP block, a\n",
    "classification head. One design decision is load-bearing, and we learned it\n",
    "the hard way. The write *address*, the projections producing\n",
    "$\\mathbf{q}_t$, $\\mathbf{k}_t$, and $\\beta_t$, must be computed from the\n",
    "key token alone, and the *content* $\\mathbf{v}_t$ from the value token\n",
    "alone. Give the address projection access to the value token and a trained\n",
    "Hebbian model escapes superposition entirely, by assigning every\n",
    "(key, value) *pair* its own private address so that no two writes ever\n",
    "collide; recall stops distinguishing the two rules. That escape is itself\n",
    "instructive, superposition only binds when addresses are reused, but it\n",
    "dodges the question we are asking. Real linear-attention layers offer no\n",
    "such dodge at this task's scale: $\\mathbf{q}$, $\\mathbf{k}$, $\\mathbf{v}$\n",
    "are all projections of the same token embedding, so a model that answers\n",
    "\"what is ticker X now\" must address by the ticker. Our split enforces the\n",
    "shared-address regime we set out to measure."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "deltanet-trained-models-hit-the-same-ceiling-2"
   },
   "source": [
    "class MemoryModel(nn.Module):\n",
    "    \"\"\"One matrix-memory layer plus an MLP block; address from the key only.\"\"\"\n",
    "    def __init__(self, num_keys, num_values, rule, d=64, d_head=48):\n",
    "        super().__init__()\n",
    "        self.rule = rule\n",
    "        self.key_emb = nn.Embedding(num_keys, d)\n",
    "        self.val_emb = nn.Embedding(num_values + 1, d)\n",
    "        self.W_q = nn.Linear(d, d_head, bias=False)   # Address: from key only\n",
    "        self.W_k = nn.Linear(d, d_head, bias=False)\n",
    "        self.W_beta = nn.Linear(d, 1)\n",
    "        self.W_v = nn.Linear(d, d_head, bias=False)   # Content: from value only\n",
    "        self.W_o = nn.Linear(d_head, d, bias=False)\n",
    "        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)\n",
    "        self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(),\n",
    "                                 nn.Linear(4 * d, d))\n",
    "        self.head = nn.Linear(d, num_values)\n",
    "\n",
    "    def forward(self, keys, vals):\n",
    "        a, c = self.key_emb(keys), self.val_emb(vals)\n",
    "        q = F.normalize(self.W_q(a), dim=-1)   # x / max(|x|, 1e-12)\n",
    "        k = F.normalize(self.W_k(a), dim=-1)\n",
    "        v, beta = self.W_v(c), torch.sigmoid(self.W_beta(a))\n",
    "        B, T, d_head = v.shape\n",
    "        S = v.new_zeros(B, d_head, d_head)\n",
    "        outputs = []\n",
    "        for t in range(T):\n",
    "            k_t, v_t = k[:, t], v[:, t]\n",
    "            if self.rule == 'delta':                   # Erase, then write\n",
    "                u = beta[:, t] * (v_t - torch.einsum('bkv,bk->bv', S, k_t))\n",
    "            else:                                      # Hebbian: superimpose\n",
    "                u = v_t\n",
    "            S = S + k_t[:, :, None] * u[:, None, :]\n",
    "            outputs.append(torch.einsum('bkv,bk->bv', S, q[:, t]))\n",
    "        h = a + c + self.W_o(torch.stack(outputs, 1))\n",
    "        h = h + self.mlp(self.ln1(h))\n",
    "        return self.head(self.ln2(h))"
   ],
   "execution_count": 4,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=deltanet-md-5 fw=all -->\n",
    "\n",
    "Training is a plain Adam loop, a couple of minutes in total for the whole\n",
    "sweep; the only asymmetry between the two runs is the single line that\n",
    "writes the state."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "deltanet-trained-models-hit-the-same-ceiling-3"
   },
   "source": [
    "def train_recall(rule, R, num_keys=8, num_values=16, epochs=12,\n",
    "                 num_train=4096, num_val=1024, batch_size=256):\n",
    "    device = d2l.try_gpu()\n",
    "    torch.manual_seed(0)\n",
    "    data = [torch.tensor(x, device=device)\n",
    "            for x in make_task(num_train, num_keys, num_values, R, seed=1)\n",
    "            + make_task(num_val, num_keys, num_values, R, seed=2)]\n",
    "    Xk, Xv, y, Vk, Vv, Vy = data\n",
    "    model = MemoryModel(num_keys, num_values, rule).to(device)\n",
    "    opt = torch.optim.Adam(model.parameters(), lr=3e-3)\n",
    "    for epoch in range(epochs):\n",
    "        for i in torch.randperm(num_train, device=device).split(batch_size):\n",
    "            loss = F.cross_entropy(model(Xk[i], Xv[i]).reshape(-1, num_values),\n",
    "                                   y[i].reshape(-1), ignore_index=-1)\n",
    "            opt.zero_grad(); loss.backward(); opt.step()\n",
    "    with torch.no_grad():\n",
    "        pred, mask = model(Vk, Vv).argmax(-1), Vy != -1\n",
    "        return float((pred[mask] == Vy[mask]).float().mean())\n",
    "\n",
    "print(f\"{'writes/key':>10} {'Hebbian':>9} {'delta':>7}\")\n",
    "final = {}\n",
    "for R in [1, 2, 4, 6]:\n",
    "    accs = [train_recall(rule, R) for rule in ('hebbian', 'delta')]\n",
    "    final[R] = accs\n",
    "    print(f'{R:>10} {accs[0]:>9.3f} {accs[1]:>7.3f}')\n",
    "assert final[6][1] > 0.95, 'trained delta must keep near-perfect recall'\n",
    "assert final[6][0] < 0.5, 'trained Hebbian must collapse under overwrites'"
   ],
   "execution_count": 5,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "writes/key   Hebbian   delta\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "         1     1.000   1.000\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "         2     0.537   1.000\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "         4     0.327   1.000\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "         6     0.268   1.000\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=deltanet-md-6 fw=all -->\n",
    "\n",
    "Training does not rescue the additive write. The trained Hebbian model\n",
    "tracks the mechanistic collapse, near $0.5$ at two writes per key and\n",
    "roughly $0.25$--$0.35$ by four to six, while the delta model learns a\n",
    "perfect overwriting memory and stays at $1.000$ throughout. Both models\n",
    "have identical parameter counts and see identical data; the entire gap is\n",
    "the write rule. This synthetic failure has a production-scale echo:\n",
    "*multi-query associative recall*, essentially this task embedded in\n",
    "natural text, is the single synthetic that best predicts which efficient\n",
    "architectures close the quality gap to attention, a diagnosis\n",
    "that section takes up in earnest.\n",
    "\n",
    "## The Delta Rule\n",
    "\n",
    "Time to earn the equation rather than assert it. The memory's job,\n",
    "when token $t$ arrives, is to associate $\\mathbf{k}_t$ with\n",
    "$\\mathbf{v}_t$. Phrase that as an objective: the state\n",
    "$\\mathbf{S}$ should make the momentary *recall loss*\n",
    "\n",
    "$$\n",
    "\\ell_t(\\mathbf{S})\n",
    "= \\tfrac{1}{2}\\,\n",
    "\\big\\| \\mathbf{S}^\\top \\mathbf{k}_t - \\mathbf{v}_t \\big\\|^2\n",
    "$$\n",
    "\n",
    "small: reading the memory at $\\mathbf{k}_t$ should return\n",
    "$\\mathbf{v}_t$. The gradient is\n",
    "\n",
    "$$\n",
    "\\nabla_{\\mathbf{S}}\\, \\ell_t(\\mathbf{S})\n",
    "= \\mathbf{k}_t \\big(\\mathbf{S}^\\top \\mathbf{k}_t - \\mathbf{v}_t\\big)^\\top,\n",
    "$$\n",
    "\n",
    "an outer product of the key with the *prediction error*, what the memory\n",
    "answered minus what it should have answered. One step of gradient descent\n",
    "from $\\mathbf{S}_{t-1}$ with step size $\\beta_t$ is\n",
    "\n",
    "$$\n",
    "\\mathbf{S}_t\n",
    "= \\mathbf{S}_{t-1}\n",
    "- \\beta_t\\, \\mathbf{k}_t\n",
    "  \\big(\\mathbf{k}_t^\\top \\mathbf{S}_{t-1} - \\mathbf{v}_t^\\top\\big),\n",
    "$$\n",
    "\n",
    "which at $\\beta_t = 1$ is exactly the repair we used above. This is the\n",
    "*delta rule*, and it is old: it is the Widrow--Hoff least-mean-squares\n",
    "update of adaptive filtering [@Widrow.Hoff.1960], rediscovered for\n",
    "fast-weight memories by @Schlag.Irie.Schmidhuber.2021 and made\n",
    "parallel, and thereby practical at modern scale, by\n",
    "@Yang.Wang.Zhang.ea.2024 . Multiplying out\n",
    "the equation puts it in the transition-plus-write shape of\n",
    "the equation:\n",
    "\n",
    "$$\n",
    "\\mathbf{S}_t\n",
    "= \\big(\\mathbf{I} - \\beta_t\\, \\mathbf{k}_t \\mathbf{k}_t^\\top\\big)\\,\n",
    "  \\mathbf{S}_{t-1}\n",
    "+ \\beta_t\\, \\mathbf{k}_t \\mathbf{v}_t^\\top,\n",
    "\\qquad\n",
    "\\mathbf{o}_t = \\mathbf{S}_t^\\top \\mathbf{q}_t .\n",
    "$$\n",
    "\n",
    "The two readings teach different things. The error form\n",
    "the equation is the algorithm, four lines below, and the\n",
    "cheap one to run: two matrix--vector products per token, never a\n",
    "$d_k \\times d_k$ matrix in sight. The matrix form\n",
    "the equation slots DeltaNet into the family template: the\n",
    "transition $\\mathbf{I} - \\beta_t \\mathbf{k}_t \\mathbf{k}_t^\\top$ is no\n",
    "longer diagonal, and this section's closing pages are about what that\n",
    "buys. For a unit-norm key, the update interpolates: the value stored at\n",
    "$\\mathbf{k}_t$ moves to $(1 - \\beta_t)\\,\\mathbf{v}_{\\textrm{old}} +\n",
    "\\beta_t\\, \\mathbf{v}_t$, so $\\beta_t$, produced by a sigmoid from the\n",
    "token, is a learned *write strength*, $\\beta_t \\approx 0$ preserving the\n",
    "old binding and $\\beta_t \\approx 1$ replacing it, the input gate of\n",
    "that section reborn as a per-token learning rate.\n",
    "\n",
    "That last phrase is worth dwelling on, because it reframes the whole\n",
    "family. Compare the Hebbian write under the same lens: adding\n",
    "$\\beta_t \\mathbf{k}_t \\mathbf{v}_t^\\top$ is one gradient step on the\n",
    "*alignment* objective $-\\langle \\mathbf{S}^\\top \\mathbf{k}_t,\n",
    "\\mathbf{v}_t\\rangle$, a loss that only ever asks the read-out to grow\n",
    "along $\\mathbf{v}_t$, never to be *correct*, so the updates pile up\n",
    "without feedback. The delta rule optimizes recall error and gets\n",
    "error-correction; linear attention optimizes alignment and gets\n",
    "superposition. Both are online learners running *inside* the forward pass,\n",
    "at test time, with no autograd involved; they differ only in their loss.\n",
    "Once seen this way, the natural question is what other optimizers one\n",
    "could run in that inner loop, and that section\n",
    "develops exactly that view, with the delta rule as one point, one\n",
    "plain-SGD step on a least-squares loss, in a family of solvers.\n",
    "\n",
    "One practical note before the code: all delta-rule implementations in this\n",
    "chapter $L2$-normalize keys (and queries), as shipped DeltaNet variants\n",
    "do. The reason is visible in the equation: along\n",
    "$\\mathbf{k}_t$ the transition scales the state by\n",
    "$1 - \\beta_t \\|\\mathbf{k}_t\\|^2$, which for unnormalized keys and\n",
    "$\\beta_t$ near $1$ can exceed $1$ in magnitude and let repeated writes\n",
    "amplify the state geometrically. Unit keys pin that factor to\n",
    "$1 - \\beta_t \\in (0, 1)$: stable, and cleanly interpretable as partial\n",
    "replacement."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "deltanet-the-delta-rule"
   },
   "source": [
    "def delta_recurrent(Q, K, V, beta):\n",
    "    \"\"\"The delta rule in the error form: read, subtract, write, read out.\"\"\"\n",
    "    S = Q.new_zeros(Q.shape[-1], V.shape[-1])\n",
    "    outputs = []\n",
    "    for t in range(Q.shape[0]):\n",
    "        error = V[t] - S.T @ K[t]                # What the memory got wrong\n",
    "        S = S + beta[t] * K[t][:, None] * error[None, :]\n",
    "        outputs.append(S.T @ Q[t])\n",
    "    return torch.stack(outputs)\n",
    "\n",
    "def delta_recurrent_matrix(Q, K, V, beta):\n",
    "    \"\"\"The same update as transition-then-write, for the family template.\"\"\"\n",
    "    S = Q.new_zeros(Q.shape[-1], V.shape[-1])\n",
    "    I = torch.eye(Q.shape[-1], device=Q.device)\n",
    "    outputs = []\n",
    "    for t in range(Q.shape[0]):\n",
    "        S = (I - beta[t] * torch.outer(K[t], K[t])) @ S \\\n",
    "            + beta[t] * torch.outer(K[t], V[t])\n",
    "        outputs.append(S.T @ Q[t])\n",
    "    return torch.stack(outputs)\n",
    "\n",
    "torch.manual_seed(0)\n",
    "d_k, d_v, T = 64, 64, 512\n",
    "Q, K, V = (torch.randn(T, d, device=d2l.try_gpu()) for d in (d_k, d_k, d_v))\n",
    "K = F.normalize(K, dim=-1)                       # Unit-norm keys, throughout\n",
    "beta = torch.sigmoid(torch.randn(T, device=d2l.try_gpu()))\n",
    "y_delta = delta_recurrent(Q, K, V, beta)\n",
    "err = (y_delta - delta_recurrent_matrix(Q, K, V, beta)).abs().max() \\\n",
    "    / y_delta.abs().max()\n",
    "print(f'error form vs matrix form: relative deviation {float(err):.2e}')\n",
    "assert err < 1e-5                  # One rule, two readings"
   ],
   "execution_count": 6,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "error form vs matrix form: relative deviation 1.88e-06\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=deltanet-md-7 fw=all -->\n",
    "\n",
    "## Training It: the WY Trick\n",
    "\n",
    "The recurrence above is fine for generation, one token, constant memory,\n",
    "but training wants the whole sequence at once, and here DeltaNet is\n",
    "genuinely harder than everything in that section. The\n",
    "chunked schedule of that section leaned on the transition\n",
    "being a scalar: products of decays collapse into `segsum`, an elementwise\n",
    "log-space cumulative sum. DeltaNet's transitions are the matrices\n",
    "$\\mathbf{I} - \\beta_t \\mathbf{k}_t \\mathbf{k}_t^\\top$, and a product of\n",
    "those is not elementwise anything; materializing the cumulative products\n",
    "would cost a $d_k \\times d_k$ matrix multiply per token, exactly the\n",
    "parallelism-killing serial chain we are trying to avoid.\n",
    "\n",
    "The escape is to look at *what the updates share*. Write the state at the\n",
    "end of a chunk of $C$ tokens, entered with state $\\mathbf{S}$, in the\n",
    "error form: each token contributes a rank-one write\n",
    "$\\mathbf{k}_t \\mathbf{u}_t^\\top$ with update vector\n",
    "$\\mathbf{u}_t = \\beta_t (\\mathbf{v}_t - \\mathbf{S}_{t-1}^\\top\n",
    "\\mathbf{k}_t)$. The trouble is that $\\mathbf{u}_t$ depends on\n",
    "$\\mathbf{S}_{t-1}$, which contains every *earlier* update in the chunk.\n",
    "Substituting $\\mathbf{S}_{t-1} = \\mathbf{S} + \\sum_{s<t} \\mathbf{k}_s\n",
    "\\mathbf{u}_s^\\top$ makes the dependency explicit:\n",
    "\n",
    "$$\n",
    "\\mathbf{u}_t\n",
    "= \\beta_t \\big(\\mathbf{v}_t - \\mathbf{S}^\\top \\mathbf{k}_t\\big)\n",
    "- \\beta_t \\sum_{s<t} (\\mathbf{k}_s^\\top \\mathbf{k}_t)\\, \\mathbf{u}_s .\n",
    "$$\n",
    "\n",
    "The updates couple only through the scalars\n",
    "$\\mathbf{k}_s^\\top \\mathbf{k}_t$, key overlaps we can compute for the\n",
    "whole chunk in one matmul. Stack the $\\mathbf{u}_t^\\top$ as the rows of\n",
    "$\\mathbf{U} \\in \\mathbb{R}^{C \\times d_v}$ and\n",
    "the equation becomes a *unit lower triangular* linear\n",
    "system,\n",
    "\n",
    "$$\n",
    "(\\mathbf{I} + \\mathbf{A})\\, \\mathbf{U}\n",
    "= \\mathrm{diag}(\\boldsymbol{\\beta})\n",
    "  \\big(\\mathbf{V} - \\mathbf{K}\\mathbf{S}\\big),\n",
    "\\qquad\n",
    "A_{ts} = \\beta_t\\, (\\mathbf{k}_t^\\top \\mathbf{k}_s)\\;\n",
    "\\textrm{for } s < t, \\textrm{ else } 0,\n",
    "$$\n",
    "\n",
    "solved by forward substitution, one triangular solve per chunk. The\n",
    "widths matter in the accounting: forming the coupling matrix $\\mathbf{A}$\n",
    "is a $\\mathcal{O}(C^2 d_k)$ matmul, and the forward substitution, whose\n",
    "right-hand side has $d_v$ columns, costs $\\mathcal{O}(C^2 d_v)$ — the\n",
    "same order as the attention-shaped matmuls around it, not free.\n",
    "(Numerical linear algebra veterans will recognize the\n",
    "maneuver: expressing a product of rank-one modifications,\n",
    "$\\prod_t (\\mathbf{I} - \\beta_t \\mathbf{k}_t \\mathbf{k}_t^\\top)$, through\n",
    "a compact coupling matrix is the classical *WY representation* of\n",
    "products of Householder-like transformations, which is why the fast form\n",
    "of DeltaNet goes by that name [@Yang.Wang.Zhang.ea.2024].) With\n",
    "$\\mathbf{U}$ in hand, everything else is the familiar two-piece schedule\n",
    "of that section: outputs are the read of the inherited\n",
    "state plus a causal attention-shaped matmul against the chunk's own\n",
    "updates,\n",
    "$\\mathbf{Y} = \\mathbf{Q}\\mathbf{S} +\n",
    "\\mathrm{tril}(\\mathbf{Q}\\mathbf{K}^\\top)\\,\\mathbf{U}$,\n",
    "and the state advances by $\\mathbf{S} \\leftarrow \\mathbf{S} +\n",
    "\\mathbf{K}^\\top \\mathbf{U}$. Attention inside, recurrence across, plus\n",
    "one small solve:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "deltanet-training-it-the-wy-trick"
   },
   "source": [
    "def delta_chunked(Q, K, V, beta, chunk_size=64):\n",
    "    \"\"\"Chunked WY form: one triangular solve per chunk, a state across chunks.\"\"\"\n",
    "    C = chunk_size\n",
    "    Qc, Kc = Q.view(-1, C, Q.shape[-1]), K.view(-1, C, K.shape[-1])\n",
    "    Vc, bc = V.view(-1, C, V.shape[-1]), beta.view(-1, C)\n",
    "    I = torch.eye(C, device=Q.device)\n",
    "    A = torch.tril(bc[:, :, None] * (Kc @ Kc.transpose(-1, -2)), -1)\n",
    "    S = Q.new_zeros(Q.shape[-1], V.shape[-1])\n",
    "    Y = []\n",
    "    for c in range(Q.shape[0] // C):       # T/C boundary steps\n",
    "        U = torch.linalg.solve_triangular(  # Solve against the actual RHS:\n",
    "            I + A[c], bc[c][:, None] * (Vc[c] - Kc[c] @ S),\n",
    "            upper=False, unitriangular=True)  # never form the inverse\n",
    "        Y.append(Qc[c] @ S + torch.tril(Qc[c] @ Kc[c].T) @ U)\n",
    "        S = S + Kc[c].T @ U\n",
    "    return torch.cat(Y)\n",
    "\n",
    "for chunk_size in [16, 64, 256]:\n",
    "    err = (delta_chunked(Q, K, V, beta, chunk_size) - y_delta).abs().max() \\\n",
    "        / y_delta.abs().max()\n",
    "    print(f'chunk size {chunk_size:4d}: relative deviation {float(err):.2e}')\n",
    "    assert err < 1e-3                      # Chunked == recurrence"
   ],
   "execution_count": 7,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "chunk size   16: relative deviation 3.00e-07\nchunk size   64: relative deviation 4.03e-07\nchunk size  256: relative deviation 3.63e-07\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=deltanet-md-8 fw=all -->\n",
    "\n",
    "Agreement to floating-point rounding at every chunk size: the triangular\n",
    "solve *is* the sequential recurrence, reorganized. The cost accounting\n",
    "matches the scalar-decay case of that section almost line\n",
    "for line, diagonal blocks of attention-shaped matmuls, a state carried\n",
    "across $T/C$ boundaries, with one addition: the per-chunk unit-triangular\n",
    "solve for $\\mathbf{U}$, taken directly against the chunk's right-hand\n",
    "side $\\mathrm{diag}(\\boldsymbol{\\beta})(\\mathbf{V} - \\mathbf{K}\n",
    "\\mathbf{S})$ at $\\mathcal{O}(C^2 d_v)$. Materializing\n",
    "$(\\mathbf{I} + \\mathbf{A})^{-1}$ first and multiplying would give the\n",
    "same answer at $\\mathcal{O}(C^3)$ plus a stored matrix nobody needs;\n",
    "solving against the right-hand side is both the cheaper and the more\n",
    "numerically well-behaved habit. This is teaching\n",
    "altitude, deliberately: the production kernels add head batching, gating,\n",
    "tensor-core-aware tiling of the solve, and numerical care in low\n",
    "precision, and live in the open-source `flash-linear-attention` library\n",
    "that also serves as the reference implementation for most of the model\n",
    "family of this chapter. The algorithm is the one you just verified.\n",
    "\n",
    "## Gating and the Modern Cell\n",
    "\n",
    "DeltaNet edits the row it writes; every other row it preserves *exactly*,\n",
    "since the transition acts as the identity off $\\mathbf{k}_t$. That is the\n",
    "opposite failure from the decay ladder: a pure-decay model forgets\n",
    "everything indiscriminately, a pure-delta model forgets nothing unless a\n",
    "new write happens to land on the same key direction. A season of\n",
    "tickers never queried again would sit in a DeltaNet state forever,\n",
    "crowding the geometry for no benefit. The modern cell therefore uses both\n",
    "knobs, a scalar decay *and* the delta write\n",
    "[@Yang.Kautz.Hatamizadeh.2024]:\n",
    "\n",
    "$$\n",
    "\\mathbf{S}_t\n",
    "= \\alpha_t \\big(\\mathbf{I} - \\beta_t\\, \\mathbf{k}_t \\mathbf{k}_t^\\top\\big)\\,\n",
    "  \\mathbf{S}_{t-1}\n",
    "+ \\beta_t\\, \\mathbf{k}_t \\mathbf{v}_t^\\top,\n",
    "\\qquad \\alpha_t \\in (0, 1),\\; \\beta_t \\in (0, 1).\n",
    "$$\n",
    "\n",
    "This is *Gated DeltaNet*, and the design has a familiar shape. In the\n",
    "online-learner reading, $\\beta_t$ is a learning rate and $\\alpha_t$ is\n",
    "weight decay, and the equation applies them *decoupled*: the decay\n",
    "shrinks the whole state first, and the gradient step is taken at its own,\n",
    "independently controlled strength, the same fix that distinguishes\n",
    "AdamW from Adam-with-$L2$ in that section. The analogy is worth\n",
    "its keep on exactly that one axis — forget at one rate, learn at\n",
    "another, let the model set each per token — and it ends there: there are\n",
    "no gradient moments or bias corrections in the equation, only\n",
    "AdamW's decoupling of decay from step. On the ladder of\n",
    "the figure, this is the dashed rung, one step past the\n",
    "diagonal: decay everywhere, plus a rank-one edit.\n",
    "\n",
    "Strike the erase term $-\\alpha_t \\beta_t \\mathbf{k}_t \\mathbf{k}_t^\\top\n",
    "\\mathbf{S}_{t-1}$, keeping decay and write, and you are back at Mamba-2's\n",
    "row of the table; set $\\alpha_t = 1$ and you have pure\n",
    "DeltaNet, so this one cell holds both parents of the family in two\n",
    "sigmoids. It is also, at the time of writing, the family's production\n",
    "representative: Qwen3-Next interleaves Gated DeltaNet with a few\n",
    "full-attention layers, and Kimi Linear's KDA layer is a Gated DeltaNet\n",
    "whose forgetting is refined from a scalar to a per-channel diagonal\n",
    "[@Kimi.Team.2025b], with the ratio of recurrent to attention layers\n",
    "a design axis that section will price.\n",
    "\n",
    "### A Language Model\n",
    "\n",
    "Every architecture this chapter takes seriously must face the scoreboard\n",
    "of that section. The block below wraps the gated delta\n",
    "recurrence in the now-routine production dressing, and nothing in it is\n",
    "new to you: pre-norm residuals, two heads with per-head states\n",
    "($\\mathbf{q}, \\mathbf{k}$ normalized per head, matching the unit-key\n",
    "convention above), an output RMSNorm and a SiLU gate on the read-out\n",
    "(this family normalizes the *output* rather than carrying a normalizer\n",
    "state, as discussed in that section), and a small MLP for\n",
    "channel mixing, since the memory read is linear across channels. We bias\n",
    "$\\alpha_t$ toward $1$ at initialization so an untrained model retains by\n",
    "default, the same open-the-forget-gate trick used since the LSTM. For\n",
    "training we keep the transparent token loop, fast enough at $32$ steps\n",
    "per window; the WY form above is how the same model would train at real\n",
    "sequence lengths."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "deltanet-a-language-model-1"
   },
   "source": [
    "class GatedDeltaNetBlock(nn.Module):\n",
    "    \"\"\"Pre-norm residual: gated delta-rule memory, then a position-wise MLP.\"\"\"\n",
    "    def __init__(self, num_hiddens, num_heads=2, dropout=0):\n",
    "        super().__init__()\n",
    "        d, self.num_heads = num_hiddens, num_heads\n",
    "        self.d_head = d // num_heads\n",
    "        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)\n",
    "        self.W_qkv = nn.Linear(d, 3 * d, bias=False)\n",
    "        self.W_ab = nn.Linear(d, 2 * num_heads)   # alpha, beta per head\n",
    "        self.norm = nn.RMSNorm(self.d_head)\n",
    "        self.W_gate = nn.Linear(d, d)\n",
    "        self.W_o = nn.Linear(d, d, bias=False)\n",
    "        self.mlp = nn.Sequential(nn.Linear(d, 2 * d), nn.GELU(),\n",
    "                                 nn.Linear(2 * d, d))\n",
    "        self.drop = nn.Dropout(dropout)\n",
    "        nn.init.constant_(self.W_ab.bias[:num_heads], 2.0)  # Retain by default\n",
    "\n",
    "    def forward(self, X):                    # (num_steps, batch, num_hiddens)\n",
    "        T, B, d = X.shape\n",
    "        h = self.ln1(X)\n",
    "        q, k, v = self.W_qkv(h).reshape(T, B, self.num_heads, 3,\n",
    "                                        self.d_head).unbind(3)\n",
    "        q, k = F.normalize(q, dim=-1), F.normalize(k, dim=-1)\n",
    "        alpha, beta = torch.sigmoid(self.W_ab(h)).chunk(2, -1)\n",
    "        S = X.new_zeros(B, self.num_heads, self.d_head, self.d_head)\n",
    "        outputs = []\n",
    "        for t in range(T):                   # Decay, then erase-and-write\n",
    "            S = alpha[t].unsqueeze(-1).unsqueeze(-1) * S\n",
    "            error = v[t] - torch.einsum('bhkv,bhk->bhv', S, k[t])\n",
    "            S = S + (beta[t].unsqueeze(-1) * k[t]).unsqueeze(-1) \\\n",
    "                * error.unsqueeze(-2)\n",
    "            outputs.append(torch.einsum('bhkv,bhk->bhv', S, q[t]))\n",
    "        o = self.norm(torch.stack(outputs))  # (T, B, heads, d_head)\n",
    "        o = (o * F.silu(self.W_gate(h)).reshape(T, B, self.num_heads,\n",
    "                                                self.d_head)).reshape(T, B, d)\n",
    "        X = X + self.drop(self.W_o(o))\n",
    "        return X + self.drop(self.mlp(self.ln2(X)))\n",
    "\n",
    "class GatedDeltaNet(d2l.Module):\n",
    "    \"\"\"A stack of blocks with the recurrent-cell interface.\"\"\"\n",
    "    def __init__(self, num_inputs, num_blocks=2, num_heads=2, dropout=0):\n",
    "        super().__init__()\n",
    "        self.save_hyperparameters()\n",
    "        self.num_hiddens = num_inputs\n",
    "        self.blocks = nn.Sequential(*[\n",
    "            GatedDeltaNetBlock(num_inputs, num_heads, dropout)\n",
    "            for _ in range(num_blocks)])\n",
    "        self.ln = nn.LayerNorm(num_inputs)\n",
    "\n",
    "    def forward(self, X, state=None):\n",
    "        return self.ln(self.blocks(X)), None"
   ],
   "execution_count": 8,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=deltanet-md-9 fw=all -->\n",
    "\n",
    "The training recipe is copied from the Mamba row of\n",
    "that section, *The Time Machine* under the BPE tokenizer,\n",
    "50,000 windows of 32 tokens, ten epochs, Adam at a gentle learning rate\n",
    "with dropout on the residual branches, because this model shares Mamba's\n",
    "appetite for memorizing a small corpus. `d2l.RNNLM` supplies embedding\n",
    "and head unchanged."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "deltanet-a-language-model-2"
   },
   "source": [
    "class GatedDeltaNetLM(d2l.RNNLM):\n",
    "    def configure_optimizers(self):\n",
    "        return torch.optim.Adam(self.parameters(), lr=self.lr)\n",
    "\n",
    "data = d2l.TimeMachine(batch_size=1024, num_steps=32,\n",
    "                       num_train=50000, num_val=5000)\n",
    "torch.manual_seed(0)\n",
    "model = GatedDeltaNetLM(GatedDeltaNet(num_inputs=128, dropout=0.3),\n",
    "                        vocab_size=len(data.vocab), lr=3e-4)\n",
    "trainer = d2l.Trainer(max_epochs=10, gradient_clip_val=1, num_gpus=1)\n",
    "model.board.yscale = 'log'\n",
    "start = time.time()\n",
    "trainer.fit(model, data)\n",
    "secs = (time.time() - start) / 10\n",
    "ppl = float(model.board.data['val_ppl'][-1].y)\n",
    "params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "print(f'{\"model\":>15} {\"val ppl\":>8} {\"params\":>9} {\"s/epoch\":>8}')\n",
    "print(f'{\"Gated DeltaNet\":>15} {ppl:>8.1f} {params:>9,} {secs:>8.1f}')"
   ],
   "execution_count": 9,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "          model  val ppl    params  s/epoch\n Gated DeltaNet     83.7   562,315      5.4\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"240.6625pt\" height=\"184.296751pt\" viewBox=\"0 0 240.6625 184.296751\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-20T01:06:58.699922</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 184.296751 \nL 240.6625 184.296751 \nL 240.6625 -0 \nL 0 -0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 31.8 146.740501 \nL 227.1 146.740501 \nL 227.1 8.140501 \nL 31.8 8.140501 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path id=\"m8e954602e2\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m8e954602e2\" x=\"31.8\" y=\"146.740501\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <g transform=\"translate(28.61875 161.338939) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use xlink:href=\"#m8e954602e2\" x=\"70.86\" y=\"146.740501\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 2 -->\n      <g transform=\"translate(67.67875 161.338939) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-32\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use xlink:href=\"#m8e954602e2\" x=\"109.92\" y=\"146.740501\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 4 -->\n      <g transform=\"translate(106.73875 161.338939) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-34\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#m8e954602e2\" x=\"148.98\" y=\"146.740501\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 6 -->\n      <g transform=\"translate(145.79875 161.338939) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-36\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use xlink:href=\"#m8e954602e2\" x=\"188.04\" y=\"146.740501\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 8 -->\n      <g transform=\"translate(184.85875 161.338939) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-38\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#m8e954602e2\" x=\"227.1\" y=\"146.740501\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 10 -->\n      <g transform=\"translate(220.7375 161.338939) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- epoch -->\n     <g transform=\"translate(114.221875 175.017064) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \nL 1159 -1331 \nL 581 -1331 \nL 581 3500 \nL 1159 3500 \nL 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nz\nM 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \nL 3122 2828 \nQ 2878 2963 2633 3030 \nQ 2388 3097 2138 3097 \nQ 1578 3097 1268 2742 \nQ 959 2388 959 1747 \nQ 959 1106 1268 751 \nQ 1578 397 2138 397 \nQ 2388 397 2633 464 \nQ 2878 531 3122 666 \nL 3122 134 \nQ 2881 22 2623 -34 \nQ 2366 -91 2075 -91 \nQ 1284 -91 818 406 \nQ 353 903 353 1747 \nQ 353 2603 823 3093 \nQ 1294 3584 2113 3584 \nQ 2378 3584 2631 3529 \nQ 2884 3475 3122 3366 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-65\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(61.523438 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(186.181641 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(241.162109 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_7\">\n      <defs>\n       <path id=\"m413143c229\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m413143c229\" x=\"31.8\" y=\"90.844415\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- $\\mathdefault{10^{2}}$ -->\n      <g transform=\"translate(7.2 94.643633) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#m413143c229\" x=\"31.8\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- $\\mathdefault{10^{3}}$ -->\n      <g transform=\"translate(7.2 14.798437) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \nQ 3050 2419 3304 2112 \nQ 3559 1806 3559 1356 \nQ 3559 666 3084 287 \nQ 2609 -91 1734 -91 \nQ 1441 -91 1130 -33 \nQ 819 25 488 141 \nL 488 750 \nQ 750 597 1062 519 \nQ 1375 441 1716 441 \nQ 2309 441 2620 675 \nQ 2931 909 2931 1356 \nQ 2931 1769 2642 2001 \nQ 2353 2234 1838 2234 \nL 1294 2234 \nL 1294 2753 \nL 1863 2753 \nQ 2328 2753 2575 2939 \nQ 2822 3125 2822 3475 \nQ 2822 3834 2567 4026 \nQ 2313 4219 1838 4219 \nQ 1578 4219 1281 4162 \nQ 984 4106 628 3988 \nL 628 4550 \nQ 988 4650 1302 4700 \nQ 1616 4750 1894 4750 \nQ 2613 4750 3031 4423 \nQ 3450 4097 3450 3541 \nQ 3450 3153 3228 2886 \nQ 3006 2619 2597 2516 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(128.203125 39.046875) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_9\">\n      <defs>\n       <path id=\"m435575d87d\" d=\"M 0 0 \nL -2 0 \n\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"146.653812\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"132.593771\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_11\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"122.618013\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_12\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"114.880214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_13\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"108.557972\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_8\">\n     <g id=\"line2d_14\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"103.212592\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_9\">\n     <g id=\"line2d_15\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"98.582214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_10\">\n     <g id=\"line2d_16\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"94.497931\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_11\">\n     <g id=\"line2d_17\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"66.808616\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_12\">\n     <g id=\"line2d_18\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"52.748575\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_13\">\n     <g id=\"line2d_19\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"42.772817\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_14\">\n     <g id=\"line2d_20\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"35.035018\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_15\">\n     <g id=\"line2d_21\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"28.712776\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_16\">\n     <g id=\"line2d_22\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"23.367396\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_17\">\n     <g id=\"line2d_23\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"18.737018\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_18\">\n     <g id=\"line2d_24\">\n      <g>\n       <use xlink:href=\"#m435575d87d\" x=\"31.8\" y=\"14.652735\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_25\">\n    <path d=\"M 36.383571 14.440501 \nL 45.949286 39.083451 \nL 55.515 54.450642 \nL 65.080714 62.994891 \nL 74.646429 72.239599 \nL 84.212143 81.072181 \nL 93.777857 89.047758 \nL 103.343571 95.693984 \nL 112.909286 101.647069 \nL 122.475 107.084228 \nL 132.040714 111.924324 \nL 141.606429 116.052708 \nL 151.172143 120.154459 \nL 160.737857 123.419057 \nL 170.303571 126.872872 \nL 179.869286 129.832166 \nL 189.435 132.914957 \nL 199.000714 135.409655 \nL 208.566429 138.084485 \nL 218.132143 140.440501 \n\" clip-path=\"url(#p892755a76b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_26\">\n    <path d=\"M 51.33 53.861241 \nL 70.86 69.417379 \nL 90.39 83.500267 \nL 109.92 91.970295 \nL 129.45 96.735881 \nL 148.98 99.038676 \nL 168.51 99.915682 \nL 188.04 99.739551 \nL 207.57 98.7212 \nL 227.1 97.024871 \n\" clip-path=\"url(#p892755a76b)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #ff7f0e; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 31.8 146.740501 \nL 31.8 8.140501 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 227.1 146.740501 \nL 227.1 8.140501 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 31.8 146.740501 \nL 227.1 146.740501 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 31.8 8.140501 \nL 227.1 8.140501 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 144.35 46.053001 \nL 220.1 46.053001 \nQ 222.1 46.053001 222.1 44.053001 \nL 222.1 15.140501 \nQ 222.1 13.140501 220.1 13.140501 \nL 144.35 13.140501 \nQ 142.35 13.140501 142.35 15.140501 \nL 142.35 44.053001 \nQ 142.35 46.053001 144.35 46.053001 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_27\">\n     <path d=\"M 146.35 21.238939 \nL 156.35 21.238939 \nL 166.35 21.238939 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_10\">\n     <!-- train_ppl -->\n     <g transform=\"translate(174.35 24.738939) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \nL 3263 -1509 \nL -63 -1509 \nL -63 -1063 \nL 3263 -1063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(39.208984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(80.322266 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(141.601562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(169.384766 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(232.763672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(282.763672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(346.240234 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(409.716797 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_28\">\n     <path d=\"M 146.35 36.195189 \nL 156.35 36.195189 \nL 166.35 36.195189 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #ff7f0e; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_11\">\n     <!-- val_ppl -->\n     <g transform=\"translate(174.35 39.695189) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-76\" d=\"M 191 3500 \nL 800 3500 \nL 1894 563 \nL 2988 3500 \nL 3597 3500 \nL 2284 0 \nL 1503 0 \nL 191 3500 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-76\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(59.179688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(120.458984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(148.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(198.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(261.71875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(325.195312 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p892755a76b\">\n   <rect x=\"31.8\" y=\"8.140501\" width=\"195.3\" height=\"138.6\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=deltanet-md-10 fw=all -->\n",
    "\n",
    "In our runs the row lands in the same band as the strongest models of\n",
    "that section, mid-seventies to mid-eighties of validation\n",
    "perplexity depending on framework and seed, at a comparable parameter\n",
    "count, without touching that scoreboard's recipe beyond what its Mamba\n",
    "row already used. At this corpus size that is all the claim we make: the\n",
    "gated delta cell is *competitive as a language model* while carrying the\n",
    "overwriting memory the additive family lacks. The published version of\n",
    "the same comparison, at 1.3B parameters and real token budgets, has\n",
    "Gated DeltaNet beating Mamba-2 and DeltaNet individually on perplexity\n",
    "and recall-heavy evaluations [@Yang.Kautz.Hatamizadeh.2024], with\n",
    "the two mechanisms' contributions cleanly ablatable: decay without delta\n",
    "(Mamba-2) trails on recall, delta without decay (DeltaNet) trails on\n",
    "modeling; the cell that does both leads on both.\n",
    "\n",
    "### The Generalized Delta Rule\n",
    "\n",
    "Once the transition is allowed to be \"diagonal plus rank-one\", Gated\n",
    "DeltaNet's $\\alpha_t (\\mathbf{I} - \\beta_t \\mathbf{k}_t\n",
    "\\mathbf{k}_t^\\top)$ is one point in a small design space, and the other\n",
    "prominent occupant arrived from the RNN lineage. RWKV-7\n",
    "[@Peng.Zhang.Goldstein.ea.2025] decouples every role the delta rule\n",
    "ties together:\n",
    "\n",
    "$$\n",
    "\\mathbf{S}_t\n",
    "= \\big(\\mathrm{diag}(\\mathbf{w}_t)\n",
    "- (\\mathbf{a}_t \\odot \\hat{\\boldsymbol{\\kappa}}_t)\\,\n",
    "  \\hat{\\boldsymbol{\\kappa}}_t^\\top\\big)\\, \\mathbf{S}_{t-1}\n",
    "+ \\tilde{\\mathbf{k}}_t \\mathbf{v}_t^\\top .\n",
    "$$\n",
    "\n",
    "Here $\\mathbf{w}_t$ is a per-channel decay (the GLA rung of the ladder,\n",
    "where Gated DeltaNet keeps a scalar), the unit vector\n",
    "$\\hat{\\boldsymbol{\\kappa}}_t$ is a *removal key* that need not equal the\n",
    "write key $\\tilde{\\mathbf{k}}_t$, and $\\mathbf{a}_t$ is a per-channel\n",
    "in-context learning rate. The model can erase one address while writing\n",
    "another, decay each coordinate at its own rate, and modulate the erase\n",
    "per channel. Set all three to the delta-rule tying, no decay, removal\n",
    "key equal to write key, uniform rate $\\beta$, and the update collapses\n",
    "back to DeltaNet; the cell below asserts as much, which is also a\n",
    "useful check that the equation (stated here in this chapter's\n",
    "convention; see the margin note of that section) is the\n",
    "update we claim it is. We reimplement the step from the paper's\n",
    "equations; the reference code, released under Apache 2.0 alongside a\n",
    "line-by-line explainer, matches it in about ten lines."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "deltanet-the-generalized-delta-rule"
   },
   "source": [
    "def rwkv7_step(S, w, kappa, a, k_tilde, v):\n",
    "    \"\"\"One RWKV-7 state update: per-channel decay, rank-1 erase, write.\"\"\"\n",
    "    erase = (a * kappa)[:, None] * (kappa @ S)[None, :]\n",
    "    return w[:, None] * S - erase + np.outer(k_tilde, v)\n",
    "\n",
    "d = 8\n",
    "S = rng.standard_normal((d, d))\n",
    "k, v = rng.standard_normal(d), rng.standard_normal(d)\n",
    "k /= np.linalg.norm(k)\n",
    "b = 0.7                                    # One write strength for every role\n",
    "S_rwkv = rwkv7_step(S, np.ones(d), k, b * np.ones(d), b * k, v)\n",
    "S_delta = (np.eye(d) - b * np.outer(k, k)) @ S + b * np.outer(k, v)\n",
    "print(f'RWKV-7, delta-rule setting: max deviation '\n",
    "      f'{np.abs(S_rwkv - S_delta).max():.2e}')\n",
    "assert np.abs(S_rwkv - S_delta).max() < 1e-12"
   ],
   "execution_count": 10,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "RWKV-7, delta-rule setting: max deviation 4.44e-16\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=deltanet-md-11 fw=all -->\n",
    "\n",
    "Two shipped families, Gated DeltaNet in Qwen3-Next and KDA in Kimi\n",
    "Linear [@Kimi.Team.2025b], plus RWKV-7's own model line, all sit\n",
    "within a couple of parameter-tying choices of the equation. What\n",
    "in that section was a table of independent-looking models\n",
    "has become, one write rule later, a family with two parents: choose the\n",
    "decay structure (scalar, diagonal), choose the write rule (additive,\n",
    "delta, generalized delta), and the model largely follows.\n",
    "\n",
    "## What the Transition Can Compute\n",
    "\n",
    "The memory story is done; a computational story has been hiding inside\n",
    "it. Look at the equation once more, not as a write to a\n",
    "memory but as a *linear dynamical system*: DeltaNet's transition\n",
    "$\\mathbf{I} - \\beta_t \\mathbf{k}_t \\mathbf{k}_t^\\top$ is the first\n",
    "transition in this chapter that is not diagonal. Its spectrum is easy to\n",
    "read off: eigenvalue $1 - \\beta_t$ along $\\mathbf{k}_t$, eigenvalue $1$\n",
    "on the orthogonal complement. Every transition in\n",
    "the table, by contrast, has *non-negative* eigenvalues:\n",
    "decays in $(0, 1)$ can shrink a state coordinate, but can never flip its\n",
    "sign. Does that limit what these models can compute, as opposed to\n",
    "remember?\n",
    "\n",
    "It does, and the cleanest witness is the humblest function in this book:\n",
    "*parity*. Read a bit stream and answer whether the number of ones so far\n",
    "is even. A one-dimensional recurrence solves it exactly, multiply the\n",
    "state by $-1$ on every one-bit, and the sign of the state *is* the\n",
    "answer, but that solution needs a transition value of $-1$. A model whose\n",
    "transitions are confined to $[0, 1]$ can represent parity of the count\n",
    "only through precision tricks, and finite-precision analysis says\n",
    "essentially not at all [@Grazzi.Siems.Franke.ea.2025]. The\n",
    "experiment below stages exactly this boundary, and two design choices,\n",
    "both discovered by breaking the experiment first, matter. First, the\n",
    "recurrence is *purely multiplicative*, $\\mathbf{h}_t = \\mathbf{a}_t\n",
    "\\odot \\mathbf{h}_{t-1}$ from a nonzero learned $\\mathbf{h}_0$, with no\n",
    "input drive: parity lives in sign flips, and an additive drive hands the\n",
    "model a way to bury the flip mechanism in magnitudes, which in our\n",
    "trials left *both* variants at chance. Second, the sequences are short,\n",
    "training at $T \\le 24$: the point is the eigenvalue range, not\n",
    "long-horizon optimization, which fails for its own reasons soon enough.\n",
    "The two models below differ in one activation: the per-channel\n",
    "transition $\\mathbf{a}_t$ comes from a sigmoid, range $(0, 1)$, or from\n",
    "a tanh, range $(-1, 1)$. Because success at the edge of trainability is\n",
    "seed-dependent, we train three seeds per configuration and report each."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "deltanet-what-the-transition-can-compute"
   },
   "source": [
    "class MultiplicativeRNN(nn.Module):\n",
    "    \"\"\"h_t = a_t * h_{t-1}, elementwise; nonzero h_0; no input drive.\"\"\"\n",
    "    def __init__(self, act, num_hiddens=8):\n",
    "        super().__init__()\n",
    "        self.emb = nn.Embedding(2, num_hiddens)\n",
    "        self.W_a = nn.Linear(num_hiddens, num_hiddens)\n",
    "        self.h0 = nn.Parameter(torch.ones(num_hiddens)\n",
    "                               + 0.1 * torch.randn(num_hiddens))\n",
    "        self.head = nn.Linear(num_hiddens, 2)\n",
    "        self.act = act\n",
    "\n",
    "    def forward(self, X):\n",
    "        a = self.act(self.W_a(self.emb(X)))      # (batch, T, num_hiddens)\n",
    "        h = self.h0.expand(X.shape[0], -1)\n",
    "        for t in range(X.shape[1]):\n",
    "            h = a[:, t] * h\n",
    "        return self.head(h)\n",
    "\n",
    "def train_parity(act, T, seed, num_train=8192, num_val=2048, epochs=60,\n",
    "                 batch_size=256):\n",
    "    torch.manual_seed(seed)\n",
    "    data_rng = np.random.default_rng(1)\n",
    "    X = data_rng.integers(0, 2, (num_train + num_val, T))\n",
    "    X, y = torch.tensor(X), torch.tensor(X.sum(1) % 2)\n",
    "    model = MultiplicativeRNN(act)\n",
    "    opt = torch.optim.Adam(model.parameters(), lr=5e-3)\n",
    "    for epoch in range(epochs):\n",
    "        for i in torch.randperm(num_train).split(batch_size):\n",
    "            loss = F.cross_entropy(model(X[i]), y[i])\n",
    "            opt.zero_grad(); loss.backward()\n",
    "            nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()\n",
    "    with torch.no_grad():\n",
    "        return float((model(X[num_train:]).argmax(-1)\n",
    "                      == y[num_train:]).float().mean())\n",
    "\n",
    "for T in [8, 16, 24]:\n",
    "    for act, name in [(torch.sigmoid, 'sigmoid, a in (0,1)'),\n",
    "                      (torch.tanh, 'tanh,    a in (-1,1)')]:\n",
    "        accs = [train_parity(act, T, seed) for seed in range(3)]\n",
    "        print(f'T={T:2d}  {name}: ' + '  '.join(f'{a:.3f}' for a in accs))"
   ],
   "execution_count": 11,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "T= 8  sigmoid, a in (0,1): 0.372  0.375  0.650\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "T= 8  tanh,    a in (-1,1): 1.000  1.000  1.000\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "T=16  sigmoid, a in (0,1): 0.486  0.497  0.497\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "T=16  tanh,    a in (-1,1): 1.000  1.000  1.000\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "T=24  sigmoid, a in (0,1): 0.517  0.517  0.483\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "T=24  tanh,    a in (-1,1): 0.517  1.000  1.000\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=deltanet-md-12 fw=all -->\n",
    "\n",
    "The separation is stark. The non-negative model solves parity in none of\n",
    "our runs: its best seeds reach only the $0.6$--$0.7$ range, at $T = 8$,\n",
    "where magnitudes can weakly encode the count, and from $T = 16$ on every\n",
    "run sits at chance; more data or more epochs cannot supply what the\n",
    "dynamics lacks, a way to flip a sign. The $(-1, 1)$ model solves parity\n",
    "on every seed at the shortest length and then runs into a different\n",
    "wall: optimization grows less reliable with length. In our runs, failing\n",
    "seeds appear from $T = 16$ in one framework and by $T = 24$ in both,\n",
    "each failed seed at chance while the surviving seeds still find the\n",
    "sign-flip solution; which seeds fail moves between frameworks and\n",
    "re-runs, and only the pattern, reliability decaying with $T$, is stable.\n",
    "Keep the two walls distinct, because they have different remedies. The\n",
    "sigmoid model fails for *representational* reasons: by the\n",
    "finite-precision result cited below, no parameter setting of a\n",
    "fixed-size recurrence with non-negative transitions computes parity at\n",
    "arbitrary length, and our short-$T$ runs show the failure already in\n",
    "force at toy scale. The tanh model at larger $T$ fails, when it fails,\n",
    "for *optimization* reasons: the solution exists (set every one-bit's\n",
    "transition to $-1$; done), but gradient descent through a product of\n",
    "many near-sign-flips has a hard time finding it. The next cell shows the\n",
    "solution existing regardless of length, by setting the weights by hand.\n",
    "\n",
    "### The Reflection in the Delta Rule\n",
    "\n",
    "What has this to do with DeltaNet? The transition\n",
    "$\\mathbf{I} - \\beta \\mathbf{k} \\mathbf{k}^\\top$ carries the same\n",
    "capability, hidden behind the range of $\\beta$. Its eigenvalue along\n",
    "$\\mathbf{k}$ is $1 - \\beta$: for $\\beta \\in (0, 1)$, always positive, a\n",
    "partial erase; at $\\beta = 1$, exactly zero, a *projection* that deletes\n",
    "the $\\mathbf{k}$-component outright; and if $\\beta$ were allowed up to\n",
    "$2$, the eigenvalue reaches $-1$ and the transition becomes a\n",
    "*reflection*, a sign flip along $\\mathbf{k}$ and identity elsewhere, a\n",
    "Householder transformation. We verify the mechanism with no training at\n",
    "all, a delta cell whose weights we set by hand: state initialized to\n",
    "$\\mathbf{k}$, transition applied on every one-bit, parity read from the\n",
    "sign of $\\mathbf{k}^\\top \\mathbf{h}_T$. One caution from our own\n",
    "attempts: this *must* be a hand-set demonstration. The trained analogue\n",
    "with a single learnable reflection axis is an optimization trap, it can\n",
    "represent parity but in our runs never learned it at any $T$, so we\n",
    "teach the mechanism mechanistically and let the diagonal experiment\n",
    "above carry the trained evidence."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "deltanet-the-reflection-in-the-delta-rule"
   },
   "source": [
    "d = 6\n",
    "k = rng.standard_normal(d)\n",
    "k /= np.linalg.norm(k)\n",
    "for b in [0.5, 1.0, 1.5, 2.0]:\n",
    "    eigs = np.sort(np.linalg.eigvalsh(np.eye(d) - b * np.outer(k, k)))\n",
    "    print(f'beta = {b:.1f}: eigenvalues {eigs[0]:+.1f} (along k), '\n",
    "          f'{eigs[-1]:+.1f} (elsewhere)')\n",
    "\n",
    "for T in [8, 24, 64]:\n",
    "    bits = rng.integers(0, 2, (2000, T))\n",
    "    parity = bits.sum(1) % 2\n",
    "    for b, tag in [(1.0, 'projection'), (2.0, 'reflection')]:\n",
    "        h = np.tile(k, (len(bits), 1))            # h_0 = k\n",
    "        for t in range(T):                        # (I - beta k k^T) on one-bits\n",
    "            h -= (b * bits[:, t])[:, None] * (h @ k)[:, None] * k\n",
    "        acc = ((h @ k < 0).astype(int) == parity).mean()\n",
    "        print(f'T = {T:2d}, beta = {b:.0f} ({tag}): parity accuracy {acc:.3f}')"
   ],
   "execution_count": 12,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "beta = 0.5: eigenvalues +0.5 (along k), +1.0 (elsewhere)\nbeta = 1.0: eigenvalues -0.0 (along k), +1.0 (elsewhere)\nbeta = 1.5: eigenvalues -0.5 (along k), +1.0 (elsewhere)\nbeta = 2.0: eigenvalues -1.0 (along k), +1.0 (elsewhere)\nT =  8, beta = 1 (projection): parity accuracy 0.529\nT =  8, beta = 2 (reflection): parity accuracy 1.000\nT = 24, beta = 1 (projection): parity accuracy 0.518\nT = 24, beta = 2 (reflection): parity accuracy 1.000\nT = 64, beta = 1 (projection): parity accuracy 0.501\nT = 64, beta = 2 (reflection): parity accuracy 1.000\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=deltanet-md-13 fw=all -->\n",
    "\n",
    "The projection erases the very component that counts, accuracy at\n",
    "chance at every length, while the reflection computes parity *exactly*\n",
    "at $T = 64$ as easily as at $T = 8$: no optimization horizon, because\n",
    "nothing was optimized. This one eigenvalue is the entire content of a\n",
    "design flag you will meet in production DeltaNet code, `allow_neg_eigval`,\n",
    "which simply doubles the gate to $\\beta_t = 2\\,\\sigma(\\cdot) \\in (0, 2)$.\n",
    "Note that the interval is open: for finite logits $2\\sigma(\\cdot)$\n",
    "approaches but never reaches $2$, so the exact reflection we just ran by\n",
    "hand is a *boundary construction*, the limit of the trainable family\n",
    "rather than a member of it — close enough for the capability, since an\n",
    "eigenvalue near $-1$ still flips signs, at the price of a slow decay of\n",
    "the tracked component. @Grazzi.Siems.Franke.ea.2025 supply the\n",
    "theory and the scaled-up measurements: for finite-precision linear\n",
    "recurrences with fixed state size, transitions confined to non-negative\n",
    "spectra provably cannot solve parity at arbitrary input length, and\n",
    "trained DeltaNet variants jump from near-zero length generalization on\n",
    "parity to essentially perfect once $\\beta$ may exceed $1$, with modular\n",
    "arithmetic improving in step.\n",
    "\n",
    "### The Ladder, and Its Ceiling\n",
    "\n",
    "Climbing one more rung: a reflection along one axis per token buys\n",
    "sign-flip state, but composing *rotations*, tracking a permutation as it\n",
    "composes, say the word problem of the symmetric group $S_3$, needs\n",
    "transitions that mix several directions at once. DeltaProduct\n",
    "[@Siems.Carstensen.Zela.ea.2025] gets there by taking $n_h$ delta\n",
    "micro-steps per token, making the transition a product of $n_h$\n",
    "Householder-like factors, rank-$n_h$ beyond diagonal, which\n",
    "interpolates between DeltaNet ($n_h = 1$) and a dense (but\n",
    "norm-bounded, hence stable) transition; on group word problems the\n",
    "reachable groups and the length extrapolation grow with $n_h$, at\n",
    "$n_h$ times the write cost. And there, as of the 2024--2025 results\n",
    "cited here, the ladder stops.\n",
    "@Merrill.Petty.Sabharwal.2024 place fixed-depth, log-precision\n",
    "transformers, S4, and Mamba in the complexity class $\\mathsf{TC}^0$:\n",
    "under standard conjectures, no such model, and no diagonal-transition\n",
    "recurrence under the same assumptions, solves the word problem of $S_5$\n",
    "at arbitrary length. That group is the algebra those authors put forward\n",
    "as the shared kernel of state tracking, tasks like following chess\n",
    "positions, evaluating code, or tracking entities — a motivation drawn\n",
    "from the cited work, not a proved reduction of any of those full tasks.\n",
    "A plain nonlinear RNN, paying the sequential price this\n",
    "chapter has been trying to avoid, solves the $S_5$ word problem with one\n",
    "layer. Their phrase for the diagonal families' predicament: an\n",
    "*illusion of state*,\n",
    "a recurrence that carries a state but cannot compose with it. The\n",
    "delta-rule rungs recover a real, if bounded, slice of that power\n",
    "without giving up parallel training; at the time of writing (2025), no\n",
    "published rung climbs higher without giving up the parallel schedule.\n",
    "the table summarizes the ladder with\n",
    "the evidence this section put on the table.\n",
    "\n",
    "<!-- tbl-caption:What a transition can track. Each rung adds structure to $\\mathbf{D}_t$ in the equation; the right column names the cheapest capability the rung unlocks, per this section's experiments and the cited theory. -->\n",
    "\n",
    "| transition $\\mathbf{D}_t$ | eigenvalues | models | unlocks |\n",
    "|:--|:--|:--|:--|\n",
    "| $\\mathbf{I}$ | all $1$ | linear attention | accumulation only |\n",
    "| scalar or diagonal, $(0, 1)$ | non-negative | RetNet, Mamba-2, GLA | forgetting; parity out of reach (fixed state, finite precision, arbitrary length) |\n",
    "| diagonal, $(-1, 1)$ | signed | negative-eigenvalue variants | parity, via sign flips |\n",
    "| $\\alpha_t(\\mathbf{I} - \\beta_t \\mathbf{k}_t \\mathbf{k}_t^\\top)$, $\\beta_t \\in (0, 1)$ | non-negative | DeltaNet, Gated DeltaNet | targeted erasure (this section's memory story) |\n",
    "| same, $\\beta_t \\in (0, 2)$ | signed | `allow_neg_eigval` variants, RWKV-7 | sign flips: parity (exact reflection at the $\\beta = 2$ boundary) |\n",
    "| product of $n_h$ Householders | signed, rank-$n_h$ | DeltaProduct | composing reflections: group word problems ($S_3$, $S_4$) |\n",
    "| dense nonlinear | unrestricted | plain RNN (that section) | $S_5$ and beyond, at sequential cost |\n",
    "\n",
    "Where does this leave the family? With a memory that can edit, a decay\n",
    "that can forget, a training schedule that runs on matmuls, and a\n",
    "transition that computes more than any diagonal model, yet still a\n",
    "fixed-size state with the capacity ceiling measured in\n",
    "that section, and still, under $\\mathsf{TC}^0$, shy of\n",
    "what a true recurrent state can track. Both residual gaps point to the\n",
    "same production answer, pairing these layers with a few attention\n",
    "layers, and that is that section. Before the hybrids, though,\n",
    "one more unification is owed: the delta rule arrived here as *one* SGD\n",
    "step on *one* loss. that section asks the general\n",
    "question, what if a sequence layer is an optimizer, full stop, and\n",
    "re-derives this whole chapter from it.\n",
    "\n",
    "**What this section's experiments do and do not show.** The overwrite\n",
    "cells are diagnostics under a stated restriction, write addresses\n",
    "computed from the key token alone, and show that within it the additive\n",
    "write superposes and the delta write does not, both mechanistically and\n",
    "after end-to-end training; they do not bound what an architecture with\n",
    "pair-dependent addressing can learn. The WY check is an identity,\n",
    "verified to floating-point tolerance. The scoreboard row is one seeded\n",
    "run per framework on one small corpus; the conclusions at 1.3B scale\n",
    "belong to the cited paper. The parity sweep shows a representational\n",
    "wall for non-negative transitions at toy scale — the arbitrary-length\n",
    "statement is the cited theorem's, with its fixed-state and\n",
    "finite-precision assumptions — and a seed-dependent optimization horizon\n",
    "for signed ones; the hand-set reflection is a mechanism demonstration,\n",
    "not a trained result.\n",
    "\n",
    "## Summary\n",
    "\n",
    "An additive matrix memory cannot re-bind a key: after $R$ writes to the\n",
    "same key it returns the superposition of all $R$ values, and recall of\n",
    "the latest collapses like guesswork, a failure that survives end-to-end\n",
    "training whenever write addresses must be reused (and one that decay\n",
    "cannot fix, since decay discounts by age, not by key). The delta rule\n",
    "reads before writing and stores only the correction,\n",
    "$\\mathbf{S}_t = \\mathbf{S}_{t-1} - \\beta_t \\mathbf{k}_t\n",
    "(\\mathbf{k}_t^\\top \\mathbf{S}_{t-1} - \\mathbf{v}_t^\\top)$, which is one\n",
    "SGD step on the recall loss $\\frac{1}{2}\\|\\mathbf{S}^\\top \\mathbf{k}_t -\n",
    "\\mathbf{v}_t\\|^2$ taken inside the forward pass, Widrow--Hoff's 1960\n",
    "update running as a neural memory, holding recall at $1.000$ under\n",
    "overwrites in both our mechanistic and trained experiments. Its\n",
    "transition $\\mathbf{I} - \\beta_t \\mathbf{k}_t \\mathbf{k}_t^\\top$ is not\n",
    "diagonal, so chunked training replaces `segsum` with a unit-triangular\n",
    "solve, the WY trick, verified here equal to the recurrence at every\n",
    "chunk size. Gated DeltaNet adds decoupled forgetting,\n",
    "$\\alpha_t(\\mathbf{I} - \\beta_t \\mathbf{k}_t \\mathbf{k}_t^\\top)$, its\n",
    "gate playing the role of AdamW's decoupled weight decay, and lands in\n",
    "the strongest band of our language-modeling scoreboard; RWKV-7\n",
    "generalizes the same cell with per-channel decay and a removal key\n",
    "decoupled from the write key. The\n",
    "non-diagonal transition also computes: eigenvalue $1 - \\beta_t$ along\n",
    "the key means $\\beta > 1$ buys sign flips, up to the exact reflection at\n",
    "the boundary $\\beta = 2$, and with them parity\n",
    "(the `allow_neg_eigval` flag); products of reflections (DeltaProduct)\n",
    "buy group word problems, and the $\\mathsf{TC}^0$ ceiling marks where\n",
    "parallel-trainable recurrences still end.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. **[short-code]** In the trained overwrite experiment of\n",
    "   that section,\n",
    "   change the write-strength gate from $\\sigma(\\cdot)$ to\n",
    "   $2\\sigma(\\cdot)$, so that $\\beta_t \\in (0, 2)$, and rerun the delta\n",
    "   sweep. Does recall change? Explain why the overwrite task gives the\n",
    "   model no reason to use $\\beta > 1$, and predict, before running\n",
    "   anything, a task from this section that would.\n",
    "1. **[short-code]** The mechanistic experiment of\n",
    "   that section uses\n",
    "   orthonormal keys. Re-run it with random unit-norm keys (skip the QR\n",
    "   step) and measure, separately, recall of keys that were just\n",
    "   overwritten and recall of the *other* keys. Use\n",
    "   the equation to explain why the delta write now\n",
    "   damages neighboring bindings, and why the damage grows with key\n",
    "   overlap.\n",
    "1. **[short-code]** Repeat the capacity sweep of\n",
    "   that section with the\n",
    "   delta write, storing $n$ pairs under $n$ *distinct* random keys. Show\n",
    "   that the delta rule obeys essentially the same $(n-1)/d_k$ error law.\n",
    "   Reconcile this with this section: which failure does the delta rule\n",
    "   fix, crowding or staleness?\n",
    "1. **[short-code]** Replace `solve_triangular` in `delta_chunked` with\n",
    "   an explicit forward-substitution loop over the chunk and verify\n",
    "   agreement. Count\n",
    "   the FLOPs of the solve against the two matmuls that surround it, at\n",
    "   chunk size $C$ and dimensions $d_k = d_v = d$: for which $C/d$ ratio\n",
    "   does the solve stop being negligible?\n",
    "1. **[extended]** Extend `delta_chunked` to the gated update\n",
    "   the equation.\n",
    "   Hint: fold the running product of decays $\\alpha$ into the keys and\n",
    "   values entering each chunk, in log space as in\n",
    "   that section, and rescale the carried state at the\n",
    "   boundary; verify against a token loop.\n",
    "1. **[extended]** Two reflections compose into a rotation: taking\n",
    "   $n_h = 2$ delta\n",
    "   micro-steps per token yields transitions that can rotate a plane by\n",
    "   an input-dependent angle, which is enough to track the word problem\n",
    "   of $S_3$ (three-element permutation composition). Following\n",
    "   that section, hand-set a two-step cell whose two\n",
    "   reflection axes realize each generator of $S_3$ and verify perfect\n",
    "   tracking on random words of length up to $64$. What does your\n",
    "   construction need that a single micro-step ($n_h = 1$) cannot\n",
    "   provide?\n",
    "\n",
    "[Discussions](https://d2l.discourse.group/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-pytorch",
   "display_name": "d2l (PyTorch)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
