{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-1 fw=all -->\n",
    "\n",
    "# Reproducibility and Inspection\n",
    "\n",
    "Run yesterday's experiment again and the loss curve comes out different.\n",
    "Is this morning's change an improvement, or a lucky seed? Answering that\n",
    "question requires knowing where every random number in a training run comes\n",
    "from. And once two runs *do* disagree, or a loss turns into NaN, the next\n",
    "question is what the model computed layer by layer, ideally without editing\n",
    "its source. This section covers both skills: controlling randomness (seeds,\n",
    "generators, determinism) and observing a running model from the outside\n",
    "(hooks)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-reproducibility-and-inspection"
   },
   "source": [
    "from mxnet import autograd, gluon, init, np, npx\n",
    "from mxnet.gluon import nn\n",
    "npx.set_np()"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-2 fw=all -->\n",
    "\n",
    "## Seeds and Randomness\n",
    "\n",
    "Randomness enters a training run in more places than you might list on a\n",
    "first try: initialization draws every weight (that section),\n",
    "dropout samples a fresh mask at each step\n",
    "[@Srivastava.Hinton.Krizhevsky.ea.2014], the data loader shuffles\n",
    "examples differently in every epoch, augmentations sample crops and flips,\n",
    "and the loader's worker processes each do all of this in parallel. Every one\n",
    "of these consults a pseudorandom number generator, a deterministic algorithm\n",
    "whose entire output sequence is fixed by its seed. Seed every generator and\n",
    "the run is repeatable; miss one and it is not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-3 fw=mxnet -->\n",
    "\n",
    "MXNet keeps one global generator per device. `np.random.seed(k)` (MXNet's\n",
    "`np`, an alias of `mx.random.seed`) seeds all of them in one call, each\n",
    "from the seed and its device id, so initialization, sampling, and dropout\n",
    "are pinned on every device at once (pass `device=` instead to bring a\n",
    "single generator to a device-independent state). It does not touch\n",
    "Python's `random` module or classic NumPy, which keep their own global\n",
    "generators, a gap that will matter when the data pipeline enters. The\n",
    "function below draws everything, weights and data alike, from MXNet's\n",
    "seeded generators, so one call pins the whole run:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-seeds-and-randomness"
   },
   "source": [
    "def train_once(seed):\n",
    "    np.random.seed(seed)  # seeds the generator on every device\n",
    "    net = nn.Sequential()\n",
    "    net.add(nn.Dense(32, activation='relu'), nn.Dense(1))\n",
    "    net.initialize()\n",
    "    trainer = gluon.Trainer(net.collect_params(),\n",
    "                            'sgd', {'learning_rate': 0.1})\n",
    "    X = np.random.normal(0, 1, (128, 20))\n",
    "    y = np.random.normal(0, 1, (128, 1))\n",
    "    for _ in range(5):\n",
    "        with autograd.record():\n",
    "            loss = ((net(X) - y) ** 2).mean()\n",
    "        loss.backward()\n",
    "        trainer.step(1)  # loss is a mean already: no batch-size rescale\n",
    "    return loss.item(), net[0].weight.data().copy()\n",
    "\n",
    "loss_a, w_a = train_once(seed=0)\n",
    "loss_b, w_b = train_once(seed=0)\n",
    "loss_c, _ = train_once(seed=1)\n",
    "print('same seed, identical loss and weights:',\n",
    "      loss_a == loss_b, bool((w_a == w_b).all()))\n",
    "print('different seed:', loss_a, 'vs', loss_c)"
   ],
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "same seed, identical loss and weights: True True\ndifferent seed: 1.2009189128875732 vs 0.9397940039634705\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-4 fw=all -->\n",
    "\n",
    "The two seeded runs agree *bitwise*, down to the last floating-point bit:\n",
    "same initial weights, same data, same gradients, same operations in the\n",
    "same order. One detail deserves a flag: we pinned the TensorFlow run to\n",
    "the CPU. On a GPU the same seeded code produces equal losses but weights\n",
    "that differ in their last bits — the reason emerges in the next section.\n",
    "\n",
    "### Generator Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-5 fw=mxnet -->\n",
    "\n",
    "A single global stream is fragile: every consumer shares it, so inserting\n",
    "one extra random call shifts everything drawn after it. MXNet offers no\n",
    "private stream to retreat to: there is no generator object, and no\n",
    "sampling function takes one. The substitute is seeding discipline: re-seed\n",
    "immediately before the draws that must be pinned. It works, at a price the\n",
    "private stream does not charge, since re-seeding also resets the stream\n",
    "for everything drawn after it:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-generator-objects"
   },
   "source": [
    "np.random.seed(42)\n",
    "split = np.arange(10)\n",
    "np.random.shuffle(split)  # mx.np has no permutation; shuffle in place\n",
    "_ = np.random.normal(0, 1, (1000,))  # unrelated consumption of the stream\n",
    "np.random.seed(42)  # re-seed at the boundary that must be pinned\n",
    "split_again = np.arange(10)\n",
    "np.random.shuffle(split_again)\n",
    "print(bool((split == split_again).all()), split)"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "True [9. 1. 4. 5. 8. 7. 6. 2. 3. 0.]\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-6 fw=mxnet -->\n",
    "\n",
    "There is no `generator=` argument anywhere in Gluon: a consumer that must\n",
    "own its randomness owns a seed instead and calls `np.random.seed` itself,\n",
    "accepting the reset of everything drawn after it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-7 fw=all -->\n",
    "\n",
    "### DataLoader Workers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-8 fw=mxnet -->\n",
    "\n",
    "Gluon's `DataLoader` runs `num_workers > 0` on a pool of worker\n",
    "*processes*, so the fork-inheritance lesson applies verbatim: on Linux\n",
    "each worker starts as a copy of the parent, NumPy state included, and the\n",
    "pool's initializer does no seeding at all; it only ships the dataset and\n",
    "the `set_np` flags to the child (see `_worker_initializer` in\n",
    "`gluon/data/dataloader.py`). There is no `worker_init_fn` equivalent to\n",
    "bridge the gap, and the pool outlives any one epoch. Two consequences.\n",
    "First, deliberate per-worker seeding has to live in the dataset itself:\n",
    "make augmentation a deterministic function of the sample by seeding from\n",
    "the sample index inside `__getitem__`, and it stops mattering which\n",
    "worker runs it. Second, a surprise from the source: the shuffle order of\n",
    "`DataLoader(shuffle=True)` is drawn from *classic* NumPy's global\n",
    "generator (`RandomSampler` calls `numpy.random.shuffle`), so pinning it\n",
    "takes a NumPy seed; MXNet's own seed does not reach it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-9 fw=all -->\n",
    "\n",
    "### Randomness as a Value\n",
    "\n",
    "Every failure above is hidden global state: a generator that lives\n",
    "somewhere your code does not mention, silently copied or shared. Some\n",
    "library designs remove the hiding place altogether. In an explicit-PRNG\n",
    "design (JAX is the prominent example), randomness is threaded through the\n",
    "program as a value: every random operation takes a key argument, using the\n",
    "same key twice yields the same numbers by definition, and independent\n",
    "streams are obtained by explicitly splitting a key in two. Accidentally\n",
    "duplicating a stream would require writing the duplication into the code,\n",
    "so the worker bug cannot be expressed. The price is bookkeeping, since\n",
    "every function that needs randomness must accept and split keys. PyTorch's\n",
    "global seed is the convenient version of the same contract, one implicit\n",
    "key that everything shares; generator objects recover the explicit version\n",
    "where it matters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-10 fw=mxnet -->\n",
    "\n",
    "MXNet sits at the fully implicit end of this spectrum: per-device global\n",
    "streams, no generator objects, no stateless variants. Seeding discipline,\n",
    "one call up front and a re-seed at any boundary that must be pinned, is\n",
    "the entire toolkit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-11 fw=all -->\n",
    "\n",
    "## Determinism and Its Price\n",
    "\n",
    "Seeding fixes which numbers the program draws. It does not fix how the\n",
    "arithmetic evaluates. Floating-point addition is not associative\n",
    "(that section), so summing the same numbers in a different\n",
    "order or grouping may give a different answer:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-determinism-and-its-price-1"
   },
   "source": [
    "a, b, c = [np.array([v], dtype='float32') for v in (1e8, -1e8, 1.0)]\n",
    "s_left, s_right = (a + b) + c, a + (b + c)  # same numbers, regrouped\n",
    "print(bool(s_left == s_right), float(s_left - s_right))"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "False 1.0\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-12 fw=mxnet -->\n",
    "\n",
    "We regroup three numbers rather than reversing a million, because MXNet's\n",
    "`sum` accumulates in double precision and its result is impressively\n",
    "insensitive to element order — the associativity failure is a property of\n",
    "the *format*, and the three-line version exhibits it in any framework.\n",
    "On a CPU the summation order inside one operation is at least fixed, so\n",
    "seeded runs repeat; the bitwise agreement of `train_once` above is exactly\n",
    "that. On a GPU it often is not: kernels built on atomic additions commit\n",
    "their partial sums in whatever order threads happen to finish, so two\n",
    "seeded runs on the *same machine* can differ in the last bits, and after\n",
    "enough training steps, in the loss curve. MXNet ships no switch that\n",
    "forbids this and no error-raising mode. The one lever it does have is the\n",
    "analogue of cuDNN benchmark mode: by default cuDNN times several\n",
    "convolution algorithms on the first batch and keeps the winner, and since\n",
    "the winner can change from run to run, reproducibility work sets the\n",
    "environment variable `MXNET_CUDNN_AUTOTUNE_DEFAULT=0` (before any\n",
    "operator runs) to pin the algorithm choice. Beyond that, staying away\n",
    "from operations whose kernels are nondeterministic is up to you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-13 fw=all -->\n",
    "\n",
    "This guarantee has a narrow scope. No framework promises bitwise agreement\n",
    "across releases, platforms, or CPU versus GPU: it holds only for a\n",
    "pinned machine, library version, and flag configuration. That makes bitwise\n",
    "reproducibility a *debugging* tool, the setting that lets you bisect\n",
    "exactly where two runs diverge. The *scientific* goal is statistical\n",
    "reproducibility: the same conclusions across seeds, reported as a mean and\n",
    "spread over several runs rather than one fortunate curve. The distinction\n",
    "mirrors that section: changing dtype changes results in the\n",
    "last bits by design, and an experimental claim that survives neither a new\n",
    "seed nor bfloat16 was never a result.\n",
    "\n",
    "## Hooks: Looking Inside"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-14 fw=mxnet -->\n",
    "\n",
    "In that section we noted that `net(X)` does not\n",
    "call `forward` directly: it calls `__call__`, which wraps `forward` with\n",
    "extra machinery. Hooks are that machinery, exposed. Calling\n",
    "`block.register_forward_hook(f)` arranges for `f(block, inputs, output)`\n",
    "to run after every forward pass of that block, and\n",
    "`register_forward_pre_hook` attaches before it, with no change to the\n",
    "model's source, which matters precisely when the model came from a\n",
    "library or a checkpoint you do not want to edit. Gluon itself relies on\n",
    "the mechanism: `Block.summary()` builds its per-layer table by\n",
    "registering a forward hook on every block. the figure draws\n",
    "the wrapper as a pipeline: hooks slot into the gap the `__call__` wrapper\n",
    "already leaves around `forward`, so an observer can capture or check\n",
    "without a single line of the model changing (Gluon's contract is\n",
    "observe-only: a hook's return value is ignored, so unlike PyTorch's it\n",
    "cannot modify the output)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-15 fw=all -->\n",
    "\n",
    "![The `__call__` wrapper as a pipeline: input flows through pre-hooks, then forward, then hooks, to the output, with the two hook stages dashed and orange against forward's solid blue, and a side arrow from the hooks stage to an observer that can capture, check, or modify.](../img/bg-hooks.svg)\n",
    "\n",
    "We reuse the residual stack of\n",
    "that section, rebuilt compactly:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-hooks-looking-inside"
   },
   "source": [
    "class ResidualBlock(nn.Block):\n",
    "    def __init__(self, num_hiddens):\n",
    "        super().__init__()\n",
    "        self.body = nn.Sequential()\n",
    "        self.body.add(nn.Dense(num_hiddens, activation='relu'),\n",
    "                      nn.Dense(num_hiddens))\n",
    "\n",
    "    def forward(self, X):\n",
    "        return X + self.body(X)\n",
    "\n",
    "np.random.seed(0)\n",
    "net = nn.Sequential()\n",
    "net.add(nn.Dense(64), *[ResidualBlock(64) for _ in range(8)], nn.Dense(10))\n",
    "net.initialize(init.Xavier())  # variance-preserving, as in the init chapter"
   ],
   "execution_count": 5,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-16 fw=all -->\n",
    "\n",
    "### Capturing Activation Statistics\n",
    "\n",
    "The initialization experiments of that section measured the\n",
    "standard deviation of activations at every depth. The same measurement\n",
    "takes a few lines on an unmodified model:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-capturing-activation-statistics"
   },
   "source": [
    "stats, handles = [], []\n",
    "\n",
    "def record(block, inputs, output):\n",
    "    stats.append((type(block).__name__, output.std().item()))\n",
    "\n",
    "for m in net:\n",
    "    handles.append(m.register_forward_hook(record))\n",
    "net(np.random.normal(0, 1, (256, 20)))\n",
    "for h in handles:\n",
    "    h.detach()\n",
    "for name, std in stats:\n",
    "    print(f'{name:15s} std {std:.2f}')"
   ],
   "execution_count": 6,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Dense           std 0.68\nResidualBlock   std 0.83\nResidualBlock   std 1.03\nResidualBlock   std 1.21\nResidualBlock   std 1.46\nResidualBlock   std 1.83\nResidualBlock   std 2.28\nResidualBlock   std 2.75\nResidualBlock   std 3.43\nDense           std 4.41\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-17 fw=all -->\n",
    "\n",
    "The residual stream's spread grows block by block, since each block adds\n",
    "its body's output on top of the stream, exactly the depth effect that\n",
    "motivated the scaled initializations of that section, measured\n",
    "here without touching the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-18 fw=mxnet -->\n",
    "\n",
    "Two rules keep hooks safe. First, *detach before you stash*: the hook\n",
    "above reduces the output to a Python float on the spot; a hook that\n",
    "stores arrays should store `output.detach()` (add `.copy()` if the array\n",
    "may later be updated in place), because under `autograd.record` a stashed\n",
    "output keeps the computation graph of every forward pass alive until the\n",
    "list is cleared. Second, *keep the handle and call* `handle.detach()`:\n",
    "`register_forward_hook` returns a `HookHandle`, and the hook stays\n",
    "registered for the block's lifetime otherwise (the handle also works as a\n",
    "context manager that detaches on exit). One caveat specific to Gluon:\n",
    "hooks live in the Python `__call__` wrapper, so `hybridize()` warns that\n",
    "hooks on the children of a hybridized block will not fire; keep a block\n",
    "unhybridized while observing it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-19 fw=all -->\n",
    "\n",
    "### A NaN Finder\n",
    "\n",
    "When a loss becomes NaN at step 40,000, the useful question is which layer\n",
    "produced the first non-finite value. Check every leaf module's output for\n",
    "finiteness and the answer arrives in one forward pass. We sabotage a\n",
    "weight deep in the stack to watch it fire:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-a-nan-finder"
   },
   "source": [
    "def make_finite_check(name):\n",
    "    def check(block, inputs, output):\n",
    "        if not np.isfinite(output).all():\n",
    "            raise RuntimeError(f'first non-finite output in {name}')\n",
    "    return check\n",
    "\n",
    "leaves = ([('0', net[0])]\n",
    "          + [(f'{k}.body.{j}', net[k].body[j])\n",
    "             for k in range(1, 9) for j in (0, 1)]\n",
    "          + [('9', net[9])])\n",
    "handles = [blk.register_forward_hook(make_finite_check(name))\n",
    "           for name, blk in leaves]\n",
    "net[3].body[1].weight.data()[0, 0] = float('nan')  # sabotage one layer\n",
    "try:\n",
    "    net(np.random.normal(0, 1, (2, 20)))\n",
    "except RuntimeError as e:\n",
    "    print(e)\n",
    "for h in handles:\n",
    "    h.detach()"
   ],
   "execution_count": 7,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "first non-finite output in 3.body.1\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-20 fw=mxnet -->\n",
    "\n",
    "The report names block `3.body.1`, the layer we poisoned, rather than\n",
    "leaving you to bisect with print statements while NaNs propagate through\n",
    "everything downstream. Two Gluon-specific details show in the setup.\n",
    "First, blocks carry no path names and there is no `named_modules`\n",
    "equivalent, so we spell out the list of leaves ourselves, easy here\n",
    "because the structure is ours. Second, we deliberately poison the block's\n",
    "*second* dense layer: Gluon's fused `Dense(..., activation='relu')`\n",
    "kernel maps NaN to $0$, so a NaN planted before the fused activation is\n",
    "silently erased before any hook can see it — a small object lesson, in a\n",
    "section about diagnosis, that the instrument must be placed where the\n",
    "symptom can actually appear. Registration order does not matter: hooks\n",
    "fire in execution order, so the first completed forward with a\n",
    "non-finite output is the culprit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-21 fw=all -->\n",
    "\n",
    "### Backward Hooks and Beyond"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-22 fw=mxnet -->\n",
    "\n",
    "Gluon has no backward hooks: nothing can be attached to observe gradients\n",
    "*as* the backward pass computes them. What it has is post-hoc inspection,\n",
    "and for most debugging that is enough: after `loss.backward()` every\n",
    "parameter's gradient sits in `param.grad()`, so to log per-layer gradient\n",
    "norms or catch an explosion you iterate `net.collect_params()` and read\n",
    "them like any other array. What post-hoc inspection cannot do is act\n",
    "*during* the pass, so per-layer gradient clipping in the style of\n",
    "exercise 3 is written into the training step instead; Gluon's `Trainer`\n",
    "splits `step` into `allreduce_grads` and `update` precisely so that such\n",
    "code can stand between them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-23 fw=all -->\n",
    "\n",
    "## Summary\n",
    "\n",
    "Reproducibility is an inventory problem: initialization, dropout,\n",
    "shuffling, augmentation, and loader workers each draw from some generator,\n",
    "and a run repeats only if every one of them is seeded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-24 fw=mxnet -->\n",
    "\n",
    "`np.random.seed` covers MXNet's generator on every device in one call;\n",
    "classic NumPy and Python's `random` need their own seeds, and Gluon's\n",
    "loader consults NumPy for its shuffle order while offering no per-worker\n",
    "seeding hook, so worker randomness must be pinned inside the dataset.\n",
    "Seeding makes the program repeatable, not the arithmetic: MXNet has no\n",
    "determinism switch, only the autotuning lever\n",
    "`MXNET_CUDNN_AUTOTUNE_DEFAULT=0`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-25 fw=all -->\n",
    "\n",
    "Bitwise agreement is a debugging tool;\n",
    "conclusions that hold across seeds are the scientific goal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-26 fw=mxnet -->\n",
    "\n",
    "For looking inside a model, forward hooks and pre-hooks observe any block\n",
    "without editing it, subject to two rules: detach what you stash, and\n",
    "detach the `HookHandle` when done. Backward hooks do not exist; gradients\n",
    "are read after the fact from `param.grad()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-27 fw=all -->\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Extend `train_once` to load data through your framework's parallel data\n",
    "   loader (e.g., PyTorch's `DataLoader`, with `num_workers=4`). Give the\n",
    "   dataset its own `np.random.default_rng()` object and use it to add noise\n",
    "   as each example loads. On a process-based loader, check whether workers\n",
    "   copied the same generator state. Using the per-worker seeding hook this\n",
    "   section introduced for your framework, replace the dataset's stashed\n",
    "   generator with one derived from the worker's own seed, then verify\n",
    "   agreement across two runs.\n",
    "2. Write a forward hook (or your framework's equivalent introduced in this\n",
    "   section) that counts multiply-accumulate operations for every linear\n",
    "   layer from the shapes of its input and weight, and use it to report\n",
    "   per-layer and total FLOPs for the residual stack above. Check the total\n",
    "   against a hand count.\n",
    "3. Using a backward hook (or your framework's equivalent introduced in this\n",
    "   section), record the norm of each block's output gradient. After the\n",
    "   backward pass, compare these activation-gradient norms with the\n",
    "   parameter-gradient norms in the same block. Which one first reveals an\n",
    "   exploding backward signal?\n",
    "4. A forward hook that returns a value *replaces* the module's output. Use\n",
    "   one to zero out the output of a single residual block's body (turning\n",
    "   the block into the identity) and measure how the network's output\n",
    "   changes. Which block matters most, and how would you find out with one\n",
    "   loop?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-28 fw=mxnet -->\n",
    "\n",
    "5. Rebuild the activation-statistics table two more ways: with\n",
    "   `register_forward_pre_hook`, recording the standard deviation of each\n",
    "   block's *input* (the pre-hook receives the argument tuple, so unpack\n",
    "   it), and by reading the source of `Block.summary()`, which builds its\n",
    "   whole table from one forward hook, then calling it on the residual\n",
    "   stack. Then revisit exercise 4: Gluon ignores a hook's return value,\n",
    "   so an output-replacing hook cannot be written; perform the same\n",
    "   ablation by editing the block's `forward` instead, and compare the two\n",
    "   contracts, observe-only hooks against PyTorch's mutable ones, for a\n",
    "   model you own and for one you do not."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-mxnet",
   "display_name": "d2l (MXNet)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
