{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-1 fw=all -->\n",
    "\n",
    "# Reproducibility and Inspection\n",
    "\n",
    "Run yesterday's experiment again and the loss curve comes out different.\n",
    "Is this morning's change an improvement, or a lucky seed? Answering that\n",
    "question requires knowing where every random number in a training run comes\n",
    "from. And once two runs *do* disagree, or a loss turns into NaN, the next\n",
    "question is what the model computed layer by layer, ideally without editing\n",
    "its source. This section covers both skills: controlling randomness (seeds,\n",
    "generators, determinism) and observing a running model from the outside\n",
    "(hooks)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-reproducibility-and-inspection"
   },
   "source": [
    "import random\n",
    "import numpy as np\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.utils.data import DataLoader, TensorDataset"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-2 fw=all -->\n",
    "\n",
    "## Seeds and Randomness\n",
    "\n",
    "Randomness enters a training run in more places than you might list on a\n",
    "first try: initialization draws every weight (that section),\n",
    "dropout samples a fresh mask at each step\n",
    "[@Srivastava.Hinton.Krizhevsky.ea.2014], the data loader shuffles\n",
    "examples differently in every epoch, augmentations sample crops and flips,\n",
    "and the loader's worker processes each do all of this in parallel. Every one\n",
    "of these consults a pseudorandom number generator, a deterministic algorithm\n",
    "whose entire output sequence is fixed by its seed. Seed every generator and\n",
    "the run is repeatable; miss one and it is not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-3 fw=pytorch -->\n",
    "\n",
    "In PyTorch, `torch.manual_seed(k)` seeds the global generator on the CPU\n",
    "*and* on every CUDA device. It does not touch Python's `random` module or\n",
    "NumPy, which keep their own global generators (`random.seed`,\n",
    "`np.random.seed`), nor NumPy `Generator` objects created by\n",
    "`np.random.default_rng`, which carry their own seeds. The function below\n",
    "draws everything, weights, data, and shuffling, from torch's global\n",
    "generator, so one call pins the whole run:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-seeds-and-randomness"
   },
   "source": [
    "def train_once(seed):\n",
    "    torch.manual_seed(seed)  # seeds the CPU and every CUDA device\n",
    "    net = nn.Sequential(nn.Linear(20, 32), nn.ReLU(), nn.Linear(32, 1))\n",
    "    opt = torch.optim.SGD(net.parameters(), lr=0.1)\n",
    "    X, y = torch.randn(128, 20), torch.randn(128, 1)\n",
    "    for _ in range(5):\n",
    "        opt.zero_grad()\n",
    "        loss = ((net(X) - y) ** 2).mean()\n",
    "        loss.backward()\n",
    "        opt.step()\n",
    "    return loss.item(), net[0].weight.detach().clone()\n",
    "\n",
    "loss_a, w_a = train_once(seed=0)\n",
    "loss_b, w_b = train_once(seed=0)\n",
    "loss_c, _ = train_once(seed=1)\n",
    "print('same seed, identical loss and weights:',\n",
    "      loss_a == loss_b, torch.equal(w_a, w_b))\n",
    "print('different seed:', loss_a, 'vs', loss_c)"
   ],
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "same seed, identical loss and weights: True True\ndifferent seed: 0.8643209338188171 vs 0.9312677383422852\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-4 fw=all -->\n",
    "\n",
    "The two seeded runs agree *bitwise*, down to the last floating-point bit:\n",
    "same initial weights, same data, same gradients, same operations in the\n",
    "same order. One detail deserves a flag: we pinned the TensorFlow run to\n",
    "the CPU. On a GPU the same seeded code produces equal losses but weights\n",
    "that differ in their last bits — the reason emerges in the next section.\n",
    "\n",
    "### Generator Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-5 fw=pytorch -->\n",
    "\n",
    "A single global stream is fragile: every consumer shares it, so inserting\n",
    "one extra random call (a new layer's initialization, a stray `torch.randn`\n",
    "while debugging) shifts everything drawn after it. A `torch.Generator` is a\n",
    "private stream with its own seed. Here a data split stays fixed no matter\n",
    "what else consumes randomness in between:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-generator-objects"
   },
   "source": [
    "g = torch.Generator().manual_seed(42)\n",
    "split = torch.randperm(10, generator=g)\n",
    "_ = torch.randn(1000)  # unrelated consumption of the global stream\n",
    "split_again = torch.randperm(10, generator=g.manual_seed(42))\n",
    "print(torch.equal(split, split_again), split)"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "True tensor([2, 6, 1, 8, 4, 5, 0, 9, 3, 7])\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-6 fw=pytorch -->\n",
    "\n",
    "Most sampling functions accept `generator=`, and so does `DataLoader`,\n",
    "where it controls the shuffle order. We use that below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-7 fw=all -->\n",
    "\n",
    "### DataLoader Workers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-8 fw=pytorch -->\n",
    "\n",
    "`DataLoader` creates a distinct torch seed for every worker from a base seed\n",
    "and the worker id. Current PyTorch also initializes Python's `random` module\n",
    "and NumPy's legacy global generator in each worker. This covers common dataset\n",
    "code, but it cannot discover generator objects owned by a dataset or a\n",
    "third-party library. Those sources still need an explicit worker initializer.\n",
    "The loader's `generator` controls both its base seed and the shuffled index\n",
    "order, so one generator can pin the whole input pipeline."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-dataloader-workers-2"
   },
   "source": [
    "def seed_worker(worker_id):\n",
    "    worker_seed = torch.initial_seed() % 2**32  # this worker's torch seed\n",
    "    np.random.seed(worker_seed)\n",
    "    random.seed(worker_seed)\n",
    "\n",
    "g = torch.Generator().manual_seed(0)\n",
    "data = TensorDataset(torch.arange(8))\n",
    "# num_workers=0 keeps this notebook cell portable. In a training script,\n",
    "# num_workers>0 invokes seed_worker once in each worker process.\n",
    "loader = DataLoader(data, batch_size=4, shuffle=True, num_workers=0,\n",
    "                    worker_init_fn=seed_worker, generator=g)\n",
    "print([batch[0].tolist() for batch in loader])\n",
    "g.manual_seed(0)\n",
    "print([batch[0].tolist() for batch in loader])"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[[5, 3, 1, 7], [2, 6, 4, 0]]\n[[5, 3, 1, 7], [2, 6, 4, 0]]\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-9 fw=pytorch -->\n",
    "\n",
    "Resetting the generator reproduces the exact shuffle order. With workers\n",
    "enabled, `seed_worker` derives the legacy NumPy and Python streams from the\n",
    "same base seed. A dataset-owned `Generator` needs to be replaced there as\n",
    "well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-10 fw=all -->\n",
    "\n",
    "### Randomness as a Value\n",
    "\n",
    "Every failure above is hidden global state: a generator that lives\n",
    "somewhere your code does not mention, silently copied or shared. Some\n",
    "library designs remove the hiding place altogether. In an explicit-PRNG\n",
    "design (JAX is the prominent example), randomness is threaded through the\n",
    "program as a value: every random operation takes a key argument, using the\n",
    "same key twice yields the same numbers by definition, and independent\n",
    "streams are obtained by explicitly splitting a key in two. Accidentally\n",
    "duplicating a stream would require writing the duplication into the code,\n",
    "so the worker bug cannot be expressed. The price is bookkeeping, since\n",
    "every function that needs randomness must accept and split keys. PyTorch's\n",
    "global seed is the convenient version of the same contract, one implicit\n",
    "key that everything shares; generator objects recover the explicit version\n",
    "where it matters.\n",
    "\n",
    "\n",
    "## Determinism and Its Price\n",
    "\n",
    "Seeding fixes which numbers the program draws. It does not fix how the\n",
    "arithmetic evaluates. Floating-point addition is not associative\n",
    "(that section), so summing the same numbers in a different\n",
    "order or grouping may give a different answer:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-determinism-and-its-price-1"
   },
   "source": [
    "torch.manual_seed(0)\n",
    "x = torch.randn(1_000_000)\n",
    "s_fwd, s_rev = x.sum(), x.flip(0).sum()  # same numbers, different order\n",
    "print(s_fwd == s_rev, (s_fwd - s_rev).item())"
   ],
   "execution_count": 5,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor(False) -0.000244140625\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-11 fw=pytorch -->\n",
    "\n",
    "On a CPU the summation order is at least fixed, so seeded runs repeat. On a\n",
    "GPU it often is not: kernels built on atomic additions (`scatter_add`, some\n",
    "convolution and embedding backward passes) commit their partial sums in\n",
    "whatever order threads happen to finish, so two seeded runs on the *same\n",
    "machine* can differ in the last bits, and after enough training steps, in\n",
    "the loss curve. `torch.use_deterministic_algorithms(True)` is the switch\n",
    "that forbids this: operations with a deterministic implementation use it\n",
    "(often at some speed cost), and operations without one raise a\n",
    "`RuntimeError` rather than silently varying. On CUDA you must additionally\n",
    "set the environment variable `CUBLAS_WORKSPACE_CONFIG=:4096:8`, or cuBLAS\n",
    "matrix multiplications themselves raise. A related but narrower setting is\n",
    "`torch.backends.cudnn.benchmark`: when true, cuDNN times several\n",
    "convolution algorithms on the first batch and keeps the winner, and since\n",
    "the winner can change from run to run, reproducibility work sets it to\n",
    "`False` (its sibling `cudnn.deterministic` covers convolutions only and is\n",
    "subsumed by the global switch)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-determinism-and-its-price-2"
   },
   "source": [
    "torch.use_deterministic_algorithms(True)\n",
    "if torch.cuda.is_available():\n",
    "    try:\n",
    "        x = torch.randn(1, 1, 2, 2, 2, requires_grad=True, device='cuda')\n",
    "        nn.AvgPool3d(1)(x).sum().backward()\n",
    "    except RuntimeError as e:\n",
    "        print(str(e).split('.')[0])\n",
    "else:\n",
    "    print('CPU run: every kernel used above is already deterministic;',\n",
    "          'on CUDA, ops lacking a deterministic kernel raise RuntimeError')\n",
    "torch.use_deterministic_algorithms(False)"
   ],
   "execution_count": 6,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "avg_pool3d_backward_cuda does not have a deterministic implementation, but you set 'torch\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-12 fw=all -->\n",
    "\n",
    "This guarantee has a narrow scope. No framework promises bitwise agreement\n",
    "across releases, platforms, or CPU versus GPU: it holds only for a\n",
    "pinned machine, library version, and flag configuration. That makes bitwise\n",
    "reproducibility a *debugging* tool, the setting that lets you bisect\n",
    "exactly where two runs diverge. The *scientific* goal is statistical\n",
    "reproducibility: the same conclusions across seeds, reported as a mean and\n",
    "spread over several runs rather than one fortunate curve. The distinction\n",
    "mirrors that section: changing dtype changes results in the\n",
    "last bits by design, and an experimental claim that survives neither a new\n",
    "seed nor bfloat16 was never a result.\n",
    "\n",
    "## Hooks: Looking Inside"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-13 fw=pytorch -->\n",
    "\n",
    "In that section we noted that `net(X)` does not call\n",
    "`forward` directly: it calls `__call__`, which wraps `forward` with extra\n",
    "machinery. Hooks are that machinery, exposed. Calling\n",
    "`module.register_forward_hook(f)` arranges for `f(module, inputs, output)`\n",
    "to run after every forward pass of that module, with no change to the\n",
    "model's source, which matters precisely when the model came from a library\n",
    "or a checkpoint you do not want to edit.\n",
    "the figure draws the wrapper as a pipeline: hooks slot into the\n",
    "gap the `__call__` wrapper already leaves around `forward`, so an observer\n",
    "can capture, check, or modify without a single line of the model changing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-14 fw=all -->\n",
    "\n",
    "![The `__call__` wrapper as a pipeline: input flows through pre-hooks, then forward, then hooks, to the output, with the two hook stages dashed and orange against forward's solid blue, and a side arrow from the hooks stage to an observer that can capture, check, or modify.](../img/bg-hooks.svg)\n",
    "\n",
    "We reuse the residual stack of\n",
    "that section, rebuilt compactly:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-hooks-looking-inside"
   },
   "source": [
    "class ResidualBlock(nn.Module):\n",
    "    def __init__(self, num_hiddens):\n",
    "        super().__init__()\n",
    "        self.body = nn.Sequential(nn.Linear(num_hiddens, num_hiddens),\n",
    "                                  nn.ReLU(),\n",
    "                                  nn.Linear(num_hiddens, num_hiddens))\n",
    "\n",
    "    def forward(self, X):\n",
    "        return X + self.body(X)\n",
    "\n",
    "torch.manual_seed(0)\n",
    "net = nn.Sequential(nn.Linear(20, 64),\n",
    "                    *[ResidualBlock(64) for _ in range(8)],\n",
    "                    nn.Linear(64, 10))"
   ],
   "execution_count": 7,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-15 fw=all -->\n",
    "\n",
    "### Capturing Activation Statistics\n",
    "\n",
    "The initialization experiments of that section measured the\n",
    "standard deviation of activations at every depth. The same measurement\n",
    "takes a few lines on an unmodified model:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-capturing-activation-statistics"
   },
   "source": [
    "stats, handles = [], []\n",
    "\n",
    "def record(module, inputs, output):\n",
    "    stats.append((type(module).__name__, output.detach().std().item()))\n",
    "\n",
    "for m in net:\n",
    "    handles.append(m.register_forward_hook(record))\n",
    "net(torch.randn(256, 20))\n",
    "for h in handles:\n",
    "    h.remove()\n",
    "for name, std in stats:\n",
    "    print(f'{name:15s} std {std:.2f}')"
   ],
   "execution_count": 8,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Linear          std 0.60\nResidualBlock   std 0.62\nResidualBlock   std 0.66\nResidualBlock   std 0.68\nResidualBlock   std 0.69\nResidualBlock   std 0.72\nResidualBlock   std 0.74\nResidualBlock   std 0.77\nResidualBlock   std 0.78\nLinear          std 0.48\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-16 fw=all -->\n",
    "\n",
    "The residual stream's spread grows block by block, since each block adds\n",
    "its body's output on top of the stream, exactly the depth effect that\n",
    "motivated the scaled initializations of that section, measured\n",
    "here without touching the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-17 fw=pytorch -->\n",
    "\n",
    "Two rules keep hooks safe. First, *detach before you stash*: the hook above\n",
    "stores `output.detach().std()`; storing `output` itself would keep the\n",
    "autograd graph of every forward pass alive until `stats` is cleared, a\n",
    "memory leak that grows with each batch. Second, *keep the handle and call*\n",
    "`handle.remove()`: a hook stays registered for the module's lifetime\n",
    "otherwise, taxing every later forward pass and accumulating stashed\n",
    "tensors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-18 fw=all -->\n",
    "\n",
    "### A NaN Finder\n",
    "\n",
    "When a loss becomes NaN at step 40,000, the useful question is which layer\n",
    "produced the first non-finite value. Check every leaf module's output for\n",
    "finiteness and the answer arrives in one forward pass. We sabotage a\n",
    "weight deep in the stack to watch it fire:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "reproducibility-inspection-a-nan-finder"
   },
   "source": [
    "def make_finite_check(name):\n",
    "    def check(module, inputs, output):\n",
    "        if not torch.isfinite(output).all():\n",
    "            raise RuntimeError(f'first non-finite output in {name}')\n",
    "    return check\n",
    "\n",
    "handles = [m.register_forward_hook(make_finite_check(name))\n",
    "           for name, m in net.named_modules()\n",
    "           if len(list(m.children())) == 0]\n",
    "with torch.no_grad():\n",
    "    net[3].body[0].weight[0, 0] = float('nan')  # sabotage one layer\n",
    "try:\n",
    "    net(torch.randn(2, 20))\n",
    "except RuntimeError as e:\n",
    "    print(e)\n",
    "for h in handles:\n",
    "    h.remove()"
   ],
   "execution_count": 9,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "first non-finite output in 3.body.0\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-19 fw=pytorch -->\n",
    "\n",
    "The report names module `3.body.0`, the layer we poisoned, rather than\n",
    "leaving you to bisect with print statements while NaNs propagate through\n",
    "everything downstream."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-20 fw=all -->\n",
    "\n",
    "### Backward Hooks and Beyond"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-21 fw=pytorch -->\n",
    "\n",
    "Gradients get the same treatment.\n",
    "`module.register_full_backward_hook(f)` runs\n",
    "`f(module, grad_input, grad_output)` after the module's gradients are\n",
    "computed, which is how you log activation-gradient norms or catch an explosion\n",
    "at its source. A returned replacement changes `grad_input`; it does not edit\n",
    "the module's parameter gradients. Use this API rather than the older\n",
    "`register_backward_hook`, which is deprecated and unreliable for modules with\n",
    "multiple inputs. Like forward hooks, it returns a handle to remove and should\n",
    "detach anything it stores. For the specific job of\n",
    "extracting intermediate features from a standard vision backbone,\n",
    "torchvision's `create_feature_extractor` is the production upgrade: it\n",
    "traces the model with `torch.fx` and returns a module that outputs the\n",
    "requested internal nodes directly, provided the model is traceable, whereas\n",
    "hooks work on anything."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-22 fw=all -->\n",
    "\n",
    "## Summary\n",
    "\n",
    "Reproducibility is an inventory problem: initialization, dropout,\n",
    "shuffling, augmentation, and loader workers each draw from some generator,\n",
    "and a run repeats only if every one of them is seeded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-23 fw=pytorch -->\n",
    "\n",
    "`torch.manual_seed` covers torch's CPU and CUDA generators; NumPy and\n",
    "`random` need their own seeds in the main process. `DataLoader` derives\n",
    "distinct worker seeds and a supplied `generator` pins the shuffle order;\n",
    "`worker_init_fn` also configures generator objects that the loader cannot\n",
    "discover. Seeding\n",
    "makes the program repeatable, not the arithmetic:\n",
    "`torch.use_deterministic_algorithms(True)` pins kernel choice too, raising\n",
    "on operations that cannot comply."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-24 fw=all -->\n",
    "\n",
    "Bitwise agreement is a debugging tool;\n",
    "conclusions that hold across seeds are the scientific goal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-25 fw=pytorch -->\n",
    "\n",
    "For looking\n",
    "inside a model, forward and backward hooks observe any module without\n",
    "editing it, subject to two rules: detach what you stash, and remove the\n",
    "handle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=reproducibility-inspection-md-26 fw=all -->\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Extend `train_once` to load data through your framework's parallel data\n",
    "   loader (e.g., PyTorch's `DataLoader`, with `num_workers=4`). Give the\n",
    "   dataset its own `np.random.default_rng()` object and use it to add noise\n",
    "   as each example loads. On a process-based loader, check whether workers\n",
    "   copied the same generator state. Using the per-worker seeding hook this\n",
    "   section introduced for your framework, replace the dataset's stashed\n",
    "   generator with one derived from the worker's own seed, then verify\n",
    "   agreement across two runs.\n",
    "2. Write a forward hook (or your framework's equivalent introduced in this\n",
    "   section) that counts multiply-accumulate operations for every linear\n",
    "   layer from the shapes of its input and weight, and use it to report\n",
    "   per-layer and total FLOPs for the residual stack above. Check the total\n",
    "   against a hand count.\n",
    "3. Using a backward hook (or your framework's equivalent introduced in this\n",
    "   section), record the norm of each block's output gradient. After the\n",
    "   backward pass, compare these activation-gradient norms with the\n",
    "   parameter-gradient norms in the same block. Which one first reveals an\n",
    "   exploding backward signal?\n",
    "4. A forward hook that returns a value *replaces* the module's output. Use\n",
    "   one to zero out the output of a single residual block's body (turning\n",
    "   the block into the identity) and measure how the network's output\n",
    "   changes. Which block matters most, and how would you find out with one\n",
    "   loop?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-pytorch",
   "display_name": "d2l (PyTorch)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
