{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-1 fw=all -->\n",
    "\n",
    "# Initialization\n",
    "\n",
    "Every parameter in a network has a value before training starts: a fully\n",
    "connected layer mapping 512 inputs to 256 outputs brings a $256 \\times 512$\n",
    "weight matrix filled with random numbers. that section explains why the scale of those\n",
    "numbers decides whether a deep network trains at all, and derives the Xavier\n",
    "[@Glorot.Bengio.2010] and He [@He.Zhang.Ren.ea.2015] schemes that\n",
    "keep signal variance stable across layers. This section is the practical\n",
    "companion: what the library does when you say nothing, how to impose a scheme\n",
    "of your own on a whole model or a single block, what transformer-era code adds\n",
    "on top of Xavier and He, and how to write an initializer that no menu provides."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "init-initialization"
   },
   "source": [
    "import math\n",
    "import torch\n",
    "from torch import nn"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-2 fw=all -->\n",
    "\n",
    "## Defaults and When to Override Them\n",
    "\n",
    "You can usually ignore initialization because the default is sensible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-3 fw=pytorch -->\n",
    "\n",
    "PyTorch initializes an `nn.Linear` the moment it is constructed: weight and\n",
    "bias are both drawn from the uniform distribution $U(-b, b)$ with\n",
    "$b = 1/\\sqrt{n_\\textrm{in}}$, where $n_\\textrm{in}$ is the number of input\n",
    "features. (The documentation describes the weight rule as a variant of He\n",
    "initialization, `kaiming_uniform_` with `a=math.sqrt(5)`; the algebra collapses\n",
    "to the bound above.) A uniform distribution on $[-b, b]$ has standard deviation\n",
    "$b/\\sqrt{3}$, so both claims are cheap to check against a fresh layer:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "init-defaults-and-when-to-override-them"
   },
   "source": [
    "torch.manual_seed(0)\n",
    "layer = nn.Linear(512, 256)\n",
    "bound = 1 / math.sqrt(layer.in_features)\n",
    "w = layer.weight\n",
    "print(f'range [{w.min():.4f}, {w.max():.4f}], predicted bound {bound:.4f}')\n",
    "print(f'std {w.std():.4f}, predicted {bound / math.sqrt(3):.4f}')"
   ],
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "range [-0.0442, 0.0442], predicted bound 0.0442\nstd 0.0255, predicted 0.0255\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-4 fw=all -->\n",
    "\n",
    "Other libraries make different but equally fan-aware choices, with one legacy\n",
    "exception:\n",
    "\n",
    "| Library | Default for a dense layer's weight |\n",
    "|:--|:--|\n",
    "| PyTorch | $U(-1/\\sqrt{n_\\textrm{in}},\\, 1/\\sqrt{n_\\textrm{in}})$, bias likewise |\n",
    "| Flax (JAX) | LeCun normal: truncated normal, std $1/\\sqrt{n_\\textrm{in}}$; zero bias |\n",
    "| Keras (TensorFlow) | Glorot uniform: $U(\\pm\\sqrt{6/(n_\\textrm{in} + n_\\textrm{out})})$; zero bias |\n",
    "| MXNet Gluon | $U(-0.07, 0.07)$, not fan-aware, so override it; zero bias |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-5 fw=pytorch -->\n",
    "\n",
    "When should you override? Four situations recur: a deep network without\n",
    "normalization layers, where variance compounds across depth (we demonstrate\n",
    "this below); parameters you created by hand, since\n",
    "`nn.Parameter(torch.empty(...))` contains whatever bytes the allocator\n",
    "happened to return; reproducing a paper whose results depend on its\n",
    "initialization recipe; and architecture-specific corrections such as the\n",
    "residual scaling later in this section. Note that the defaults are computed\n",
    "from fan-in at construction time; a lazy layer (that section) has\n",
    "no fan-in until its first forward pass, so it runs the same rule at\n",
    "materialization instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-6 fw=all -->\n",
    "\n",
    "## Applying Initializers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-7 fw=pytorch -->\n",
    "\n",
    "To impose a scheme we do not touch layers one at a time. `net.apply(fn)` walks\n",
    "the module tree of that section and calls `fn` on every\n",
    "submodule, children first. The function inspects each module's type and\n",
    "decides what, if anything, to do with it. The routines in `torch.nn.init`\n",
    "follow PyTorch's convention that a trailing underscore means *in place*:\n",
    "`nn.init.normal_(w)` overwrites `w` rather than returning a fresh tensor. In\n",
    "place is what we want here, since the parameter must remain the same tensor\n",
    "object the module registered; replacing it would leave `net.parameters()` and\n",
    "any existing optimizer pointing at the old one."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "init-applying-initializers-1"
   },
   "source": [
    "net = nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 1))\n",
    "\n",
    "def init_normal(module):\n",
    "    if isinstance(module, nn.Linear):\n",
    "        nn.init.normal_(module.weight, mean=0, std=0.01)\n",
    "        nn.init.zeros_(module.bias)\n",
    "\n",
    "net.apply(init_normal)\n",
    "net[0].weight.data[0], net[0].bias.data[0]"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "(tensor([ 1.3090e-02,  7.2679e-03, -8.8525e-05,  9.0255e-03]), tensor(0.))"
     },
     "metadata": {},
     "execution_count": 3
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-8 fw=pytorch -->\n",
    "\n",
    "The `isinstance` dispatch is what makes the pattern compose: one function can\n",
    "treat `nn.Linear`, `nn.Conv2d`, and `nn.Embedding` differently, and it\n",
    "silently skips containers and activations. Because `apply` runs on any\n",
    "subtree, we can also mix schemes per block. Below we give the first layer\n",
    "Xavier initialization (that section) and set the last to a\n",
    "constant, a poor idea for training by the symmetry argument of\n",
    "that section, but it makes the mechanics visible:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "init-applying-initializers-2"
   },
   "source": [
    "def init_xavier(module):\n",
    "    if isinstance(module, nn.Linear):\n",
    "        nn.init.xavier_uniform_(module.weight)\n",
    "\n",
    "def init_42(module):\n",
    "    if isinstance(module, nn.Linear):\n",
    "        nn.init.constant_(module.weight, 42)\n",
    "\n",
    "net[0].apply(init_xavier)\n",
    "net[2].apply(init_42)\n",
    "net[0].weight.data[0], net[2].weight.data"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "(tensor([ 0.6055, -0.2622, -0.1840,  0.2908]),\n tensor([[42., 42., 42., 42., 42., 42., 42., 42.]]))"
     },
     "metadata": {},
     "execution_count": 4
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-9 fw=pytorch -->\n",
    "\n",
    "This ten-line pattern is the entire API. Later chapters wrap it into\n",
    "one-liners, `init_cnn` for convolutional networks and `init_seq2seq` for\n",
    "encoder-decoder models, but each is just a function like `init_normal` handed\n",
    "to `apply`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-10 fw=all -->\n",
    "\n",
    "## Modern Schemes: Truncation, Depth, and Zeros\n",
    "\n",
    "Xavier and He set the variance of a single layer. The schemes below, standard\n",
    "in transformer codebases, adjust what happens in the distribution's tails,\n",
    "across depth, and at a block's start.\n",
    "\n",
    "### Truncated Normals\n",
    "\n",
    "A Gaussian gets the variance right, but its tails are unbounded. That is\n",
    "harmless for one draw and a near-certainty at scale: among the $10^8$ weights\n",
    "of a BERT-sized model, dozens land beyond five standard deviations. Large\n",
    "draws also consume disproportionate dynamic range once a model is cast to low\n",
    "precision (that section). BERT and implementations in the ViT\n",
    "lineage use truncated-normal initialization\n",
    "[@Devlin.Chang.Lee.ea.2018; @Dosovitskiy.Beyer.Kolesnikov.ea.2021]: the\n",
    "tails are cut off at a fixed multiple of the nominal standard deviation. Raw\n",
    "truncated-normal initializers do not necessarily restore the variance removed\n",
    "with the tails; fan-aware variance-scaling initializers often do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-11 fw=pytorch -->\n",
    "\n",
    "`nn.init.trunc_normal_` takes absolute cutoffs `a` and `b` (defaulting to\n",
    "$\\pm 2$, which means $\\pm 2\\sigma$ only when `std=1`), so a clip at two\n",
    "standard deviations must state them explicitly:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "init-truncated-normals"
   },
   "source": [
    "torch.manual_seed(0)\n",
    "w = torch.empty(1000, 1000)\n",
    "nn.init.normal_(w, std=0.02)\n",
    "print(f'normal:    std {w.std():.4f}, max weight {w.abs().max():.4f}')\n",
    "nn.init.trunc_normal_(w, std=0.02, a=-0.04, b=0.04)\n",
    "print(f'truncated: std {w.std():.4f}, max weight {w.abs().max():.4f}')"
   ],
   "execution_count": 5,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "normal:    std 0.0200, max weight 0.0952\ntruncated: std 0.0176, max weight 0.0400\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-12 fw=all -->\n",
    "\n",
    "A million plain-normal draws produce entries near $5\\sigma = 0.1$; the\n",
    "truncated version guarantees $|w| \\leq 0.04$. Its printed standard deviation\n",
    "dips slightly below the nominal 0.02 because truncation removes tail mass;\n",
    "practice ignores the difference.\n",
    "\n",
    "### Scaling Down Residual Branches\n",
    "\n",
    "A residual network computes $\\mathbf{x}_{k+1} = \\mathbf{x}_k +\n",
    "f_k(\\mathbf{x}_k)$, so its output is the input plus the sum of $N$ block\n",
    "contributions. If every block is initialized identically, each contribution\n",
    "has variance proportional to that of the already-inflated stream it reads, and\n",
    "the stream's variance compounds geometrically with depth. This is the\n",
    "depth-wise cousin of the layer-wise explosion in\n",
    "that section. GPT-2's fix is to shrink only the *last*\n",
    "linear layer of each residual branch, the output projection that writes into\n",
    "the stream, by a factor of $1/\\sqrt{N}$: with $N$ roughly independent\n",
    "contributions each scaled down that way, the sum's variance stays $O(1)$\n",
    "regardless of depth [@Radford.Wu.Child.ea.2019]. For an $L$-layer\n",
    "transformer the published recipe is a normal with std $0.02/\\sqrt{2L}$ on the\n",
    "residual projections — the number of contributions is $N = 2L$, because each\n",
    "layer writes to the stream twice, once from attention and once from the MLP.\n",
    "Unlike Xavier or He, the base 0.02 is not derived from fan-in; it is an\n",
    "empirical constant that worked at GPT-2's widths and stuck.\n",
    "\n",
    "the figure draws the two regimes side by side.\n",
    "\n",
    "![A residual stream with additive block contributions, unscaled (left) versus scaled by 1/sqrt(N) (right). Under the independent-contribution approximation, unscaled variance grows with N and scaling keeps the sum O(1). In an unnormalized stack, each branch reads an already inflated stream, so growth can be faster, as the experiment below shows.](../img/bg-residual-stream.svg)\n",
    "\n",
    "### Starting a Block at Zero\n",
    "\n",
    "Zero-initializing *every* weight is fatal: all units in a layer compute the\n",
    "same output, receive the same gradient, and stay identical forever\n",
    "(that section). Zero-initializing just the *last* layer\n",
    "of a residual block is a different and useful move. The branch then\n",
    "contributes exactly nothing, each block is the identity map, and the network\n",
    "starts as a shallow function whose depth switches on gradually during\n",
    "training. Symmetry is not a problem because the branch's earlier layers keep\n",
    "their random weights. On the first backward pass, the zeroed projection\n",
    "receives a nonzero gradient because its input is nonzero, while the earlier\n",
    "layers receive zero gradient through that projection. After the projection\n",
    "moves off zero, gradient reaches the whole branch. Keep this scheme distinct\n",
    "from the previous one: GPT-2 makes every residual projection *small but\n",
    "nonzero*, whereas zero-init makes one layer *exactly zero* so the block starts\n",
    "as an exact identity.\n",
    "\n",
    "### Watching the Variance Compound\n",
    "\n",
    "Claims about variance at depth are cheap to test. We reuse a compact residual\n",
    "block (it repeats the one from that section) and stack\n",
    "it $N$ deep:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "init-watching-the-variance-compound-1"
   },
   "source": [
    "class ResidualBlock(nn.Module):\n",
    "    def __init__(self, d):\n",
    "        super().__init__()\n",
    "        self.body = nn.Sequential(nn.Linear(d, 4 * d), nn.ReLU(),\n",
    "                                  nn.Linear(4 * d, d))\n",
    "\n",
    "    def forward(self, X):\n",
    "        return X + self.body(X)"
   ],
   "execution_count": 6,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-13 fw=pytorch -->\n",
    "\n",
    "Every linear layer gets He initialization, appropriate for the ReLU inside\n",
    "the branch. Then a `named_parameters` loop, the same shape as nanoGPT's\n",
    "initialization code, singles out each block's output projection and applies\n",
    "one of three treatments: leave it alone, scale it by $1/\\sqrt{N}$, or zero it."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "init-watching-the-variance-compound-2"
   },
   "source": [
    "def init_he(module):\n",
    "    if isinstance(module, nn.Linear):\n",
    "        nn.init.kaiming_normal_(module.weight, nonlinearity='relu')\n",
    "        nn.init.zeros_(module.bias)\n",
    "\n",
    "def output_std(num_blocks, tweak):\n",
    "    torch.manual_seed(0)\n",
    "    net = nn.Sequential(*[ResidualBlock(64) for _ in range(num_blocks)])\n",
    "    net.apply(init_he)\n",
    "    for name, param in net.named_parameters():\n",
    "        if name.endswith('body.2.weight'):  # each block's output projection\n",
    "            tweak(param, num_blocks)\n",
    "    torch.manual_seed(1)\n",
    "    with torch.no_grad():\n",
    "        return net(torch.randn(256, 64)).std().item()"
   ],
   "execution_count": 7,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-14 fw=all -->\n",
    "\n",
    "One forward pass on unit-variance inputs per depth and treatment:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "init-watching-the-variance-compound-3"
   },
   "source": [
    "tweaks = {'default': lambda p, n: None,\n",
    "          'scaled': lambda p, n: p.data.mul_(n ** -0.5),\n",
    "          'zero': lambda p, n: nn.init.zeros_(p)}\n",
    "print(f'{\"N\":>3}' + ''.join(f'{name:>10}' for name in tweaks))\n",
    "for n in (2, 8, 32):\n",
    "    stds = (output_std(n, tweak) for tweak in tweaks.values())\n",
    "    print(f'{n:>3}' + ''.join(f'{s:>10.3g}' for s in stds))"
   ],
   "execution_count": 8,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "  N   default    scaled      zero\n  2      2.81      1.91         1\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "  8      83.5      2.38         1\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      " 32  6.37e+07      2.63         1\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-15 fw=all -->\n",
    "\n",
    "The default column multiplies by a roughly constant factor per block in this\n",
    "unnormalized stack, reaching tens of millions by $N=32$. Such initial\n",
    "activations make optimization impractical. The scaled column stays near a\n",
    "small constant over the depths tested: scaling cuts each branch's variance by\n",
    "a factor of $N$, following the independent-contribution approximation above.\n",
    "The zero column reproduces the input's standard deviation, since every block\n",
    "is the identity. This forward-pass test does not prove how an optimizer will\n",
    "behave, but it catches an unusable initialization before training begins.\n",
    "\n",
    "## Custom Initializers\n",
    "\n",
    "Occasionally the menu has nothing you need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-16 fw=pytorch -->\n",
    "\n",
    "An initializer is just a function that mutates a parameter, so writing one is\n",
    "no harder than using one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-17 fw=all -->\n",
    "\n",
    "Suppose, to make the point vividly, we want weights distributed as\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "    w \\sim \\begin{cases}\n",
    "        U(5, 10) & \\textrm{ with probability } \\frac{1}{4} \\\\\n",
    "            0    & \\textrm{ with probability } \\frac{1}{2} \\\\\n",
    "        U(-10, -5) & \\textrm{ with probability } \\frac{1}{4}\n",
    "    \\end{cases}\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-18 fw=pytorch -->\n",
    "\n",
    "Draw uniformly from $U(-10, 10)$, then zero every entry of magnitude below 5:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "init-custom-initializers-1"
   },
   "source": [
    "def my_init(module):\n",
    "    if isinstance(module, nn.Linear):\n",
    "        nn.init.uniform_(module.weight, -10, 10)\n",
    "        with torch.no_grad():\n",
    "            module.weight *= module.weight.abs() >= 5\n",
    "\n",
    "net.apply(my_init)\n",
    "net[0].weight[:2]"
   ],
   "execution_count": 9,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "tensor([[-0.0000, -0.0000, -8.7540,  0.0000],\n        [ 0.0000, -6.8736,  0.0000, -0.0000]], grad_fn=<SliceBackward0>)"
     },
     "metadata": {},
     "execution_count": 9
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-19 fw=pytorch -->\n",
    "\n",
    "The `torch.no_grad()` block is required: parameters are leaf tensors that\n",
    "track gradients, and PyTorch rejects in-place arithmetic on them unless we\n",
    "declare that the mutation is not part of any computation to differentiate.\n",
    "(The `nn.init` routines run under their own `no_grad` internally, which is why\n",
    "the first line needs none.) The same escape hatch handles one-off surgery,\n",
    "such as offsetting a whole matrix or pinning a single entry:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "init-custom-initializers-2"
   },
   "source": [
    "with torch.no_grad():\n",
    "    net[0].weight[:] += 1\n",
    "    net[0].weight[0, 0] = 42\n",
    "net[0].weight[0]"
   ],
   "execution_count": 10,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "tensor([42.0000,  1.0000, -7.7540,  1.0000], grad_fn=<SelectBackward0>)"
     },
     "metadata": {},
     "execution_count": 10
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-20 fw=pytorch -->\n",
    "\n",
    "One caveat when building with lazy layers (that section): before\n",
    "the first forward pass their parameters are placeholders with no shape, so\n",
    "`apply`-based initializers and direct surgery alike must come *after* the dry\n",
    "run that materializes them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-21 fw=all -->\n",
    "\n",
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-22 fw=pytorch -->\n",
    "\n",
    "PyTorch initializes parameters at construction with fan-aware defaults;\n",
    "override them when depth, hand-made parameters, or a paper's recipe demands\n",
    "it. The mechanism is one pattern: a function that dispatches on module type\n",
    "and mutates parameters in place with the trailing-underscore `nn.init`\n",
    "routines, walked over the model by `net.apply`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-23 fw=all -->\n",
    "\n",
    "On top of Xavier and He,\n",
    "transformer-era code truncates normal tails to bound outliers, shrinks each\n",
    "residual output projection by $1/\\sqrt{N}$ so the stream's variance stays\n",
    "$O(1)$ at any depth, and zero-initializes a block's last layer to start it as\n",
    "the identity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-24 fw=pytorch -->\n",
    "\n",
    "Anything the menu lacks is a few lines of tensor code under\n",
    "`torch.no_grad()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=init-md-25 fw=all -->\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Instrument the residual stack: record the standard deviation of the\n",
    "   activation after every block (run the stack one block at a time, or\n",
    "   capture per-block activations with the tools of that section)\n",
    "   for the default and scaled treatments at $N=32$, and plot it against\n",
    "   depth. Which curve matches the geometric-growth prediction?\n",
    "1. Zero-initialize *all* layers of every block instead of just the output\n",
    "   projection. The forward pass still returns the input, but what can the\n",
    "   network learn? Work out which parameters receive a nonzero gradient, and\n",
    "   relate the answer to the symmetry-breaking argument of\n",
    "   that section.\n",
    "1. Write an initializer that fills each parameter from a dictionary keyed by\n",
    "   parameter name (walk `net.named_parameters()` in PyTorch, `net.weights`\n",
    "   in TensorFlow, `nnx.iter_modules(net)` in JAX, `net.collect_params()`\n",
    "   in MXNet). You have re-invented part of checkpoint loading,\n",
    "   which that section covers.\n",
    "1. For a normal distribution truncated at $\\pm 2\\sigma$: what fraction of\n",
    "   draws does the clip discard, and by how much does it shrink the standard\n",
    "   deviation? Verify both numbers against the printed output of the truncation\n",
    "   demo above."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-pytorch",
   "display_name": "d2l (PyTorch)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
