{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-1 fw=all -->\n",
    "\n",
    "# Custom Layers and Functions\n",
    "\n",
    "The library ships over a hundred layers, yet every one of them started life as\n",
    "code somebody wrote because the layer they needed did not exist. Sooner or\n",
    "later you will be in the same position: a new normalization, an unusual\n",
    "residual block, an operation with no gradient. This section shows what to do.\n",
    "A custom layer is a subclass of the module class from\n",
    "that section with a forward method, and if you\n",
    "register its state properly you inherit everything a built-in layer gets:\n",
    "parameter tracking, serialization, and device movement, with no extra code. We\n",
    "build up from a stateless five-liner to RMSNorm, a normalization used by many\n",
    "current language models, then to layers with precomputed non-trainable state,\n",
    "and finally to the case where the forward computation alone is not enough\n",
    "because the gradient itself must be redefined."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "custom-layers-custom-layers-and-functions"
   },
   "source": [
    "import os\n",
    "import tempfile\n",
    "from mxnet import autograd, gluon, init, np, npx\n",
    "from mxnet.gluon import nn\n",
    "npx.set_np()"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-2 fw=all -->\n",
    "\n",
    "## Layers without Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-3 fw=mxnet -->\n",
    "\n",
    "The smallest custom layer has no state at all. `CenteredLayer` subtracts the\n",
    "mean from its input. To build it, we inherit from the base block class and\n",
    "implement `forward`; there is nothing to set up, so `__init__` only calls the\n",
    "parent constructor."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "custom-layers-layers-without-parameters-1"
   },
   "source": [
    "class CenteredLayer(nn.Block):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    def forward(self, X):\n",
    "        return X - X.mean()"
   ],
   "execution_count": 2,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-4 fw=all -->\n",
    "\n",
    "Feeding data through confirms that it does what it says."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "custom-layers-layers-without-parameters-2"
   },
   "source": [
    "layer = CenteredLayer()\n",
    "layer(np.array([1.0, 2, 3, 4, 5]))"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "array([-2., -1.,  0.,  1.,  2.])"
     },
     "metadata": {},
     "execution_count": 3
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-5 fw=all -->\n",
    "\n",
    "Nothing distinguishes this class from a built-in layer. We can place it inside\n",
    "a `Sequential`, and the container neither knows nor cares that one of its\n",
    "children is user code. The output mean should be zero; because we are adding\n",
    "up floating-point numbers, we may see a very small nonzero value instead,\n",
    "which is roundoff, not a bug."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-6 fw=mxnet -->\n",
    "\n",
    "One gluon-specific step: parameter arrays are allocated lazily, so a network\n",
    "must run `initialize()` before its first call. Our layer contributes nothing\n",
    "to initialize, but the `Dense` beside it does."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "custom-layers-layers-without-parameters-3"
   },
   "source": [
    "net = nn.Sequential()\n",
    "net.add(nn.Dense(128), CenteredLayer())\n",
    "net.initialize()\n",
    "Y = net(np.random.uniform(size=(4, 8)))\n",
    "Y.mean()"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "array(6.073151e-10)"
     },
     "metadata": {},
     "execution_count": 4
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-7 fw=all -->\n",
    "\n",
    "## Layers with Parameters: RMSNorm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-8 fw=mxnet -->\n",
    "\n",
    "A layer with something to learn must create its own parameters, and assigning\n",
    "a `gluon.Parameter` to an attribute is what registers it\n",
    "(that section). The parameter is declared with a name, a\n",
    "shape, and an initializer; its array is allocated only when `initialize()`\n",
    "runs, and the forward pass fetches the copy on the input's device with\n",
    "`.data(X.device)`. We could show the mechanics by re-implementing `nn.Dense`,\n",
    "but that teaches nothing the built-in does not already do. Instead we\n",
    "implement *RMSNorm* [@Zhang.Sennrich.2019], a normalization used by many\n",
    "current language models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-9 fw=all -->\n",
    "\n",
    "Layer normalization standardizes each input vector: subtract the mean, divide\n",
    "by the standard deviation, then apply a learned scale and shift. Zhang and\n",
    "Sennrich observed that the re-centering contributes little and dropped it,\n",
    "along with the shift. What remains is: divide by the root mean square,\n",
    "multiply by a learned gain $\\mathbf{g}$:\n",
    "\n",
    "$$\n",
    "\\textrm{RMSNorm}(\\mathbf{x}) = \\frac{\\mathbf{x}}{\\sqrt{\\frac{1}{d} \\sum_{i=1}^{d} x_i^2 + \\epsilon}} \\odot \\mathbf{g},\n",
    "$$\n",
    "\n",
    "where $d$ is the width of $\\mathbf{x}$ and $\\epsilon$ guards against division\n",
    "by zero. One parameter vector, one reduction, no shift. Why dropping the mean\n",
    "costs so little in practice is a question for the normalization discussion in\n",
    "later chapters; here it is our stateful worked example."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "custom-layers-layers-with-parameters-rmsnorm-1"
   },
   "source": [
    "class RMSNorm(nn.Block):\n",
    "    def __init__(self, d, eps=1e-6):\n",
    "        super().__init__()\n",
    "        self.gain = gluon.Parameter('gain', shape=(d,), init=init.One())\n",
    "        self.eps = eps\n",
    "\n",
    "    def forward(self, X):\n",
    "        rms = np.sqrt((X ** 2).mean(-1, keepdims=True) + self.eps)\n",
    "        return self.gain.data(X.device) * X / rms"
   ],
   "execution_count": 5,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-10 fw=all -->\n",
    "\n",
    "Let $m_2$ denote an input row's mean square. With gain one, the output mean\n",
    "square is $m_2/(m_2 + \\epsilon)$. It is therefore close to one when\n",
    "$m_2 \\gg \\epsilon$, as the badly scaled inputs below verify."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "custom-layers-layers-with-parameters-rmsnorm-2"
   },
   "source": [
    "norm = RMSNorm(8)\n",
    "norm.initialize()\n",
    "X = 100 * np.random.randn(4, 8)\n",
    "(norm(X) ** 2).mean(-1)"
   ],
   "execution_count": 6,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "array([1.       , 1.0000002, 0.9999999, 1.0000001])"
     },
     "metadata": {},
     "execution_count": 6
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-11 fw=all -->\n",
    "\n",
    "### The Composability Guarantee\n",
    "\n",
    "The reason to write RMSNorm as a module, rather than as a function with a\n",
    "gain tensor floating around beside it, is what registration buys. A correctly\n",
    "written custom layer is indistinguishable from a built-in one along four\n",
    "axes: its parameters are tracked, it composes inside containers, its state\n",
    "serializes, and it moves across devices. We check each once."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-12 fw=mxnet -->\n",
    "\n",
    "First, the gain registered itself the moment we assigned it to an attribute:\n",
    "the block files every `gluon.Parameter` it is handed. A `Trainer` given\n",
    "`norm.collect_params()` will find and update it."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "custom-layers-the-composability-guarantee-1"
   },
   "source": [
    "norm.collect_params()"
   ],
   "execution_count": 7,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "{'gain': Parameter (shape=(8,), dtype=<class 'numpy.float32'>)}"
     },
     "metadata": {},
     "execution_count": 7
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-13 fw=all -->\n",
    "\n",
    "Second, the layer drops into a `Sequential` next to built-ins."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "custom-layers-the-composability-guarantee-2"
   },
   "source": [
    "net = nn.Sequential()\n",
    "net.add(nn.Dense(8), RMSNorm(8), nn.Dense(2))\n",
    "net.initialize()\n",
    "net(np.random.randn(4, 20)).shape"
   ],
   "execution_count": 8,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "(4, 2)"
     },
     "metadata": {},
     "execution_count": 8
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-14 fw=mxnet -->\n",
    "\n",
    "Third, its state serializes with everything else. `save_parameters` writes\n",
    "every parameter, the gain included, to a file keyed by position in the block\n",
    "tree, and `load_parameters` reads that file into a fresh clone, filling in\n",
    "the deferred shapes as it goes; after the load the two models agree exactly\n",
    "(that section covers the file formats)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "custom-layers-the-composability-guarantee-3"
   },
   "source": [
    "with tempfile.NamedTemporaryFile(suffix='.params', delete=False) as f:\n",
    "    path = f.name\n",
    "net.save_parameters(path)\n",
    "clone = nn.Sequential()\n",
    "clone.add(nn.Dense(8), RMSNorm(8), nn.Dense(2))\n",
    "clone.load_parameters(path)\n",
    "os.remove(path)\n",
    "X = np.random.randn(4, 20)\n",
    "bool((net(X) == clone(X)).all())"
   ],
   "execution_count": 9,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "True"
     },
     "metadata": {},
     "execution_count": 9
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-15 fw=mxnet -->\n",
    "\n",
    "Fourth, it moves. Gluon settles device placement at initialization:\n",
    "`initialize(device=...)` puts every registered parameter, the gain included,\n",
    "on the device you name, and `reset_device` moves an initialized block later.\n",
    "The forward pass then fetches the copy living where the input does, which is\n",
    "what `self.gain.data(X.device)` was for. that section exercises\n",
    "these moves on real hardware, so we leave this axis as prose here. The\n",
    "guarantee is the same: because the gain went through the proper channel,\n",
    "every placement and move includes it, custom layer or built-in alike."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-16 fw=all -->\n",
    "\n",
    "None of this took any code beyond the class definition. The guarantee comes\n",
    "from the base class: subclass it, register state through the proper channels,\n",
    "and containers, optimizers, checkpoints, and devices all treat your layer as\n",
    "native.\n",
    "\n",
    "### Checking against the Built-in"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-17 fw=mxnet -->\n",
    "\n",
    "RMSNorm proved useful enough that most libraries now ship their own\n",
    "implementation. This Gluon version is the exception: it has `nn.LayerNorm`\n",
    "but no RMSNorm, so there is no referee\n",
    "to check our five-liner against. The general rule for custom layers, build\n",
    "one to understand it, then use the native implementation in production,\n",
    "therefore resolves differently on this tab: when the library ships nothing\n",
    "to prefer, keep the tested custom implementation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-18 fw=all -->\n",
    "\n",
    "## Precomputed State: Buffers\n",
    "\n",
    "Some layer state is neither a parameter nor a passing activation: a causal\n",
    "mask, a fixed positional table, running statistics. Such values must\n",
    "persist, serialize, and move to the device together with the model — yet no\n",
    "optimizer may ever touch them. Frameworks give this third kind of state its\n",
    "own channel, and custom layers are where you create it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-19 fw=mxnet -->\n",
    "\n",
    "that section introduced state that persists and travels with\n",
    "the model but receives no gradient. Gluon's channel is `gluon.Constant`, a\n",
    "`Parameter` subclass whose `grad_req` is fixed to `'null'`: autograd ignores\n",
    "it and no `Trainer` will update it, yet it initializes, saves, and moves with\n",
    "the rest of the block's parameters. A common case is a layer built around a\n",
    "precomputed table, for instance the causal mask that keeps attention scores\n",
    "from looking at future positions. The mask is fixed, so an ordinary parameter\n",
    "is wrong (the optimizer would update it) and a plain array attribute is wrong\n",
    "too (`save_parameters` would omit it and `reset_device` would skip it)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-20 fw=all -->\n",
    "\n",
    "One caveat before the code: the compact mask below accepts square\n",
    "self-attention score matrices. Cached decoding, where query and key lengths\n",
    "differ, needs an offset mask rather than this $T \\times T$ slice."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "custom-layers-precomputed-state-buffers-1"
   },
   "source": [
    "class CausalMask(nn.Block):\n",
    "    def __init__(self, max_len):\n",
    "        super().__init__()\n",
    "        # Precompute once for the longest sequence; slice per call\n",
    "        self.mask = gluon.Constant(np.triu(np.ones((max_len, max_len)), 1))\n",
    "\n",
    "    def forward(self, scores):\n",
    "        T = scores.shape[-1]\n",
    "        return np.where(self.mask.data(scores.device)[:T, :T].astype(bool),\n",
    "                        -np.inf, scores)"
   ],
   "execution_count": 10,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-21 fw=mxnet -->\n",
    "\n",
    "The layer masks the strict upper triangle, and its one entry in\n",
    "`collect_params` is a `Constant` with `grad_req` `'null'`, so a `Trainer`\n",
    "skips it while serialization and device moves still carry it."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "custom-layers-precomputed-state-buffers-2"
   },
   "source": [
    "mask = CausalMask(max_len=8)\n",
    "mask.initialize()\n",
    "print(mask(np.zeros((3, 3))))\n",
    "print(mask.collect_params())"
   ],
   "execution_count": 11,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[[  0. -inf -inf]\n [  0.   0. -inf]\n [  0.   0.   0.]]\n{'mask': Constant (shape=(8, 8), dtype=float32)}\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-22 fw=all -->\n",
    "\n",
    "## Custom Gradients\n",
    "\n",
    "So far we customized the forward computation and let automatic\n",
    "differentiation derive the backward pass. That derivation is literal: it\n",
    "differentiates exactly the operations you ran. Occasionally literalness is\n",
    "the problem. Quantization rounds values to a grid, and rounding is flat\n",
    "almost everywhere, so its true derivative is zero. Any parameter sitting\n",
    "behind a rounding operation stops learning:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "custom-layers-custom-gradients-1"
   },
   "source": [
    "w = np.array([0.9, 1.4, 2.6])\n",
    "w.attach_grad()\n",
    "with autograd.record():\n",
    "    y = np.round(w).sum()\n",
    "y.backward()\n",
    "w.grad"
   ],
   "execution_count": 12,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "array([0., 0., 0.])"
     },
     "metadata": {},
     "execution_count": 12
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-23 fw=all -->\n",
    "\n",
    "Zero gradient, exactly as calculus demands, and useless for training. The\n",
    "*straight-through estimator* [@Bengio.Leonard.Courville.2013] resolves\n",
    "this with a controlled lie: keep the rounding in the forward pass, but\n",
    "pretend in the backward pass that it was the identity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-24 fw=mxnet -->\n",
    "\n",
    "No automatic system can derive a lie for us, so we override the chain rule\n",
    "with `autograd.Function`, supplying both directions ourselves as methods of a\n",
    "class, the split the figure draws explicitly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-25 fw=all -->\n",
    "\n",
    "![The straight-through estimator, forward versus backward. Forward keeps the true staircase round(x), close to but not the same as the identity it approximates; backward substitutes a constant surrogate gradient of 1, the identity's own derivative, for the true gradient, which is zero almost everywhere and would stop all learning.](../img/bg-ste.svg)"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "custom-layers-custom-gradients-2"
   },
   "source": [
    "class RoundSTE(autograd.Function):\n",
    "    def forward(self, X):\n",
    "        return np.round(X)\n",
    "\n",
    "    def backward(self, grad_output):\n",
    "        # Pretend forward was the identity: pass the gradient straight through\n",
    "        return grad_output"
   ],
   "execution_count": 13,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-26 fw=mxnet -->\n",
    "\n",
    "Two rules govern the class. First, a `Function` instance is single-use: the\n",
    "library asserts that each instance is called exactly once, so create a fresh\n",
    "`RoundSTE()` at every application rather than reusing one, or the second call\n",
    "fails with an assertion naming this rule. Second, `backward` receives the\n",
    "loss gradient with respect to each output of `forward` and must return the\n",
    "gradient with respect to each input. Ours ignores the input values entirely;\n",
    "a backward that needs them must stash them in `forward` with\n",
    "`self.save_for_backward` and read them back from `self.saved_tensors`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-27 fw=all -->\n",
    "\n",
    "A small example verifies that the gradient now flows. We multiply the rounded\n",
    "values by known weights, so we know what gradient to expect at `w`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "custom-layers-custom-gradients-3"
   },
   "source": [
    "with autograd.record():\n",
    "    y = (RoundSTE()(w) * np.array([1.0, 2.0, 3.0])).sum()\n",
    "y.backward()\n",
    "w.grad"
   ],
   "execution_count": 14,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "array([1., 2., 3.])"
     },
     "metadata": {},
     "execution_count": 14
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-28 fw=all -->\n",
    "\n",
    "The downstream gradient arrives at `w` untouched, as if the rounding were the\n",
    "identity, while the forward pass still computed with rounded values. Some\n",
    "estimators instead zero the gradient where a saved input had saturated, while\n",
    "ordinary gradient clipping bounds a large upstream gradient independently of\n",
    "the quantizer. These are different approximations. Neither is a universal\n",
    "default; the surrogate backward rule must match the training objective it is\n",
    "meant to approximate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-29 fw=all -->\n",
    "\n",
    "A scope note to close. A custom gradient redefines the gradient of a\n",
    "computation assembled from existing tensor operations; it does not create new\n",
    "ones. Writing new *kernels*, code that runs on the accelerator itself, is a\n",
    "separate craft that we do not cover in this book; that section\n",
    "discusses how to get performance out of the operations you already have.\n",
    "\n",
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-30 fw=mxnet -->\n",
    "\n",
    "A custom layer is a block subclass: `forward` defines the computation,\n",
    "assigning a `gluon.Parameter` registers learnable state, and `gluon.Constant`\n",
    "registers persistent state that autograd and the `Trainer` ignore.\n",
    "Registration is what buys composability; a correctly written layer gets\n",
    "parameter tracking, container compatibility, serialization, and device\n",
    "movement for free, as we verified on RMSNorm axis by axis. When the chain\n",
    "rule itself must be overridden, as in the straight-through estimator,\n",
    "`autograd.Function` lets you supply `forward` and `backward` as a pair, one\n",
    "fresh instance per call. Build custom implementations to understand them;\n",
    "prefer the native ones in production, and where gluon ships none, as with\n",
    "RMSNorm, keep your own."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-31 fw=all -->\n",
    "\n",
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=custom-layers-md-32 fw=mxnet -->\n",
    "\n",
    "1. Add an optional learned bias to `RMSNorm` (a shift applied after the\n",
    "   scale, restoring part of what LayerNorm had). Verify that the file written\n",
    "   by `save_parameters` for a model containing it grows by the expected\n",
    "   entry, and observe what `load_parameters` does when handed a file saved\n",
    "   without the bias. Which of its `allow_missing` and `ignore_extra` flags\n",
    "   changes the outcome?\n",
    "1. Implement `Dropout` from scratch as a custom layer that zeroes each entry\n",
    "   with probability $p$ and rescales the survivors during training, but is\n",
    "   the identity during evaluation. Gluon has no `.train()`/`.eval()` switch;\n",
    "   the flag your `forward` consults is `autograd.is_training()`. What sets\n",
    "   that flag, and what does it imply for calling the layer outside\n",
    "   `autograd.record()`?\n",
    "1. Implement a clamp with a custom gradient: an `autograd.Function` whose\n",
    "   forward is `np.clip(X, lo, hi)` and whose backward passes the gradient\n",
    "   only where the input lay strictly inside the clamp range; `forward` must\n",
    "   stash `X` with `save_for_backward`. Compare its gradients with those of\n",
    "   the native `np.clip` for inputs inside, outside, and exactly on the\n",
    "   boundary. Which convention does the native operation use at the boundary?\n",
    "1. Write a layer that returns the leading half of the Fourier coefficients of\n",
    "   its input. It has no parameters, so nothing registers. What do you still\n",
    "   gain by wrapping the computation in a module instead of calling the\n",
    "   transform inline wherever you need it?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-mxnet",
   "display_name": "d2l (MXNet)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
