{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=convnext-md-1 fw=all -->\n",
    "\n",
    "# ConvNeXt: A ConvNet for the 2020s\n",
    "\n",
    "By 2021, vision transformers [@Dosovitskiy.Beyer.Kolesnikov.ea.2021] had taken over the top of the ImageNet leaderboard, and hierarchical variants such as the Swin transformer [@liu2021swin] were displacing convnets as the default backbone for detection and segmentation. It was tempting to conclude that attention had made convolution obsolete. But a transformer differs from a 2015 ResNet in many ways besides attention: its training recipe, its stem, its activation function, its normalization layers, the shape and depth of its stages. Which of these differences actually carried the improvement?\n",
    "\n",
    "@liu2022convnet answered by experiment. Starting from a ResNet-50, they applied one transformer-inspired design change at a time, measured ImageNet accuracy after each, kept what helped, and never added attention. The end point, named ConvNeXt, reaches 82.0% top-1 accuracy where the Swin-T transformer of the same computational cost reaches 81.3%. The exercise is the best guided tour of modern architecture design we know of, because every step uses a concept this book has already taught: training recipes (that section), depthwise convolutions (that section), the inverted bottleneck (that section), receptive fields (the equation), and normalization layers (that section). In this section we walk the roadmap, implement the result, and train a scaled-down ConvNeXt with the modern recipe of that section."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "convnext-imports"
   },
   "source": [
    "from d2l import mxnet as d2l\n",
    "import math\n",
    "import mxnet as mx\n",
    "from mxnet import gluon, init, np, npx\n",
    "from mxnet.gluon import nn\n",
    "npx.set_np()"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=convnext-md-2 fw=all -->\n",
    "\n",
    "## The Modernization Roadmap\n",
    "\n",
    "the table shows the full sequence. Each row keeps every change above it, so the accuracies are cumulative; the computational cost is held near that of Swin-T (about 4.5 GFLOPs) throughout.\n",
    "\n",
    "<!-- tbl-caption:The ConvNeXt modernization roadmap [@liu2022convnet]. ImageNet-1K top-1 accuracy of a ResNet-50 as design changes accumulate; each row includes all rows above it, at roughly constant cost. Swin-T, the transformer of the same cost, reaches 81.3%. -->\n",
    "\n",
    "| Change | What it means | Top-1 |\n",
    "|:--|:--|:--|\n",
    "| (starting point) | ResNet-50 with its 2015 recipe | 76.1% |\n",
    "| Modern training recipe | 300 epochs, AdamW, Mixup, label smoothing, stochastic depth (that section) | 78.8% |\n",
    "| Stage ratio | blocks per stage $(3, 4, 6, 3) \\to (3, 3, 9, 3)$ | 79.4% |\n",
    "| Patchify stem | $7 \\times 7$ stride-2 convolution plus pooling $\\to$ one $4 \\times 4$ stride-4 convolution | 79.5% |\n",
    "| Depthwise convolutions | the $3 \\times 3$ convolution becomes depthwise; width grows from 64 to 96 channels | 80.5% |\n",
    "| Inverted bottleneck | expand $4\\times$ inside the block instead of compressing | 80.6% |\n",
    "| Large kernels | depthwise convolution moved to the front and enlarged to $7 \\times 7$ | 80.6% |\n",
    "| ReLU $\\to$ GELU | a smooth activation, as in transformers | 80.6% |\n",
    "| Fewer activations | one activation per block instead of three | 81.3% |\n",
    "| Fewer normalizations | one normalization per block instead of three | 81.4% |\n",
    "| BN $\\to$ LN | layer normalization over channels at each position | 81.5% |\n",
    "| Separate downsampling | an LN plus $2 \\times 2$ stride-2 convolution between stages | 82.0% |\n",
    "\n",
    "The first row is the largest single jump, and it is not an architecture change at all. The 2.7 points it contributes are the recipe effect of that section (the \"ResNet strikes back\" study pushed the same network to 80.4% with a still longer 600-epoch procedure [@wightman2021resnet]). Keep this split in mind for everything below: of the 5.9 points separating the 2015 ResNet-50 from ConvNeXt, 2.7 are training and 3.2 are architecture. Papers that compared a well-tuned transformer against a 2015-recipe ResNet were, in large part, comparing recipes.\n",
    "\n",
    "### Macro design\n",
    "\n",
    "The next two rows change the network's silhouette. Swin-T distributes its blocks across the four stages in the ratio $1{:}1{:}3{:}1$, spending most of its depth where the feature maps are small and each block is cheap; adopting the same ratio, $(3, 3, 9, 3)$ blocks instead of ResNet-50's $(3, 4, 6, 3)$, adds 0.6 points. The stem changes more. ResNet begins with a $7 \\times 7$ stride-2 convolution followed by max-pooling, a fourfold reduction computed with overlapping windows. Vision transformers instead slice the image into non-overlapping patches, which is just a strided convolution whose kernel size equals its stride (that section): a $4 \\times 4$ convolution with stride 4. This *patchify* stem replaces the ResNet stem with no loss (79.5%), and its output, one feature vector per $4 \\times 4$ patch, is the same object a transformer calls a patch embedding.\n",
    "\n",
    "### Depthwise convolutions and the inverted bottleneck\n",
    "\n",
    "The attention core of a transformer mixes information *across positions* using\n",
    "one weight pattern per head. Its query, key, value, and output projections also\n",
    "mix channels. The MLP that follows mixes channels independently at each\n",
    "position. MobileNet factorizes a convolution in a related way: a depthwise\n",
    "convolution mixes positions within each channel, then $1 \\times 1$ convolutions\n",
    "mix channels (that section). Making the ResNet\n",
    "bottleneck's $3 \\times 3$ convolution depthwise, and spending the savings on\n",
    "width (64 to 96 channels, matching Swin-T), brings 80.5%. Inverting the\n",
    "bottleneck so that the block expands its thin residual stream by $4\\times$ and\n",
    "projects back adds a little more (80.6%). We saw in\n",
    "that section why this shape is economical once the spatial\n",
    "convolution is depthwise. The transformer's MLP uses the same factor-four\n",
    "pointwise expand--activate--project substructure; ConvNeXt adds the depthwise\n",
    "spatial mixer before it.\n",
    "\n",
    "With the expensive dense convolution gone, kernel size stops being a luxury. Moving the depthwise convolution to the front of the block (so the cheap operation sees the unexpanded stream; this alone temporarily costs 0.7 points) and enlarging it from $3 \\times 3$ to $7 \\times 7$ recovers the loss at lower cost. By the equation, a $7 \\times 7$ kernel grows the receptive field as fast as three stacked $3 \\times 3$ layers, the trade VGG once resolved in favor of depth (that section) when kernels were dense and their cost quadratic. Depthwise, a $7 \\times 7$ kernel costs only $49/9 \\approx 5\\times$ a $3 \\times 3$ one on the depthwise term alone, a small fraction of the block. The authors report that accuracy saturates at $7 \\times 7$: kernels of $9 \\times 9$ and $11 \\times 11$ buy nothing further at this scale.\n",
    "\n",
    "### Micro design\n",
    "\n",
    "The remaining rows are small and, cumulatively, decisive. Replacing ReLU by GELU, the smooth activation used in transformers and in essentially every large language model, changes nothing by itself (80.6%). What matters is *how many* activations and normalizations the block contains. A ResNet bottleneck interleaves a normalization and an activation after every convolution: three of each. A transformer block applies one activation, inside its MLP, and one or two normalizations. Deleting all but one GELU (between the two $1 \\times 1$ convolutions) adds 0.7 points, more than any other micro-design change; deleting all but one normalization adds a little more, and swapping that survivor from batch normalization to layer normalization, applied over the channels at each spatial position, another tenth (81.5%). The lesson of that section reappears: normalization is scaffolding for optimization, and less of it, placed well, can be better. the figure shows the before and after.\n",
    "\n",
    "![The ResNet-50 bottleneck block and the ConvNeXt block that the roadmap turns it into. Three normalizations and three activations become one of each; the compress-process-expand bottleneck becomes an inverted bottleneck led by a $7 \\times 7$ depthwise convolution.](../img/arch-resnet-vs-convnext-block.svg)\n",
    "\n",
    "Finally, ResNet downsamples inside the first block of each stage, by giving its $3 \\times 3$ convolution a stride of 2. Swin instead uses a separate patch-merging layer between stages. Adopting the same separation, a layer normalization followed by a $2 \\times 2$ stride-2 convolution between stages (plus a normalization after the stem and one before the classifier head, which the authors found necessary for stable training), completes the roadmap at 82.0%. the figure shows the assembled ConvNeXt-T: a patchify stem, four stages of $(3, 3, 9, 3)$ blocks at widths $(96, 192, 384, 768)$, downsampling layers between them, and a global-average-pooling head, the stem-body-head layout this chapter has used since that section.\n",
    "\n",
    "![ConvNeXt-T. A patchify stem replaces convolution-plus-pooling; stages of 3, 3, 9, and 3 blocks are joined by separate LN plus $2 \\times 2$ stride-2 downsampling layers; there is no pooling anywhere in the body.](../img/arch-convnext.svg)\n",
    "\n",
    "Where does this leave the convnet-versus-transformer question? At this scale, even: a pure convnet, given the transformer's recipe and design sensibilities, matches or slightly beats the transformer, and the paper shows the result persists for larger variants and transfers to COCO detection and ADE20K segmentation, the benchmarks that that section argued still discriminate. What the roadmap does not show is a convnet *advantage*: the two families, tuned by the same hands, land in the same place. We return to what that means in that section.\n",
    "\n",
    "## Implementation\n",
    "\n",
    "ConvNeXt is easy to build precisely because the roadmap removed things. A block is six operations; the network is the familiar arch-tuple loop.\n",
    "\n",
    "### The ConvNeXt block\n",
    "\n",
    "The block applies, in order: a $7 \\times 7$ depthwise convolution, layer normalization, a $1 \\times 1$ convolution expanding the channels $4\\times$, a GELU, and a $1 \\times 1$ convolution projecting back, with the result added to the input. Two refinements from the paper's training setup come along. *Layer scale* multiplies the branch by a learnable per-channel vector $\\gamma$ initialized to $10^{-6}$, so every block starts as a near-identity and the network begins training as a shallow function that gradually deepens; the technique was introduced for very deep vision transformers [@touvron2021cait] and helps stability here too. *Stochastic depth*, the equation of that section, randomly drops the whole branch per sample during training.\n",
    "\n",
    "One implementation detail deserves attention, because it is the\n",
    "layer-normalization placement that the roadmap's LN row depends on. ConvNeXt\n",
    "normalizes over the *channels at each spatial position*, the same normalization\n",
    "a transformer applies to each token. In a channels-last layout (samples,\n",
    "height, width, channels) this is the library default, and a $1 \\times 1$\n",
    "convolution is a linear layer applied to the last axis. Our PyTorch\n",
    "implementation therefore permutes to channels-last inside the block, while\n",
    "TensorFlow and JAX are channels-last natively. MXNet stays channels-first and\n",
    "normalizes `axis=1`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=convnext-md-3 fw=mxnet -->\n",
    "\n",
    "The MXNet block implements the deterministic ConvNeXt transformation and layer\n",
    "scale. This reduced variant omits stochastic depth; PyTorch, JAX, and\n",
    "TensorFlow include the per-sample mask used for training."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "convnext-block"
   },
   "source": [
    "class ConvNeXtBlock(nn.Block):\n",
    "    \"\"\"Depthwise 7x7, LN, 1x1 expand, GELU, 1x1 project, scaled residual.\"\"\"\n",
    "    def __init__(self, dim, layer_scale=1e-6):\n",
    "        super().__init__()\n",
    "        self.dwconv = nn.Conv2D(dim, kernel_size=7, padding=3, groups=dim)\n",
    "        self.norm = nn.LayerNorm(axis=1, epsilon=1e-6)\n",
    "        self.pwconv1 = nn.Conv2D(4 * dim, kernel_size=1)\n",
    "        self.act = nn.GELU()\n",
    "        self.pwconv2 = nn.Conv2D(dim, kernel_size=1)\n",
    "        self.gamma = gluon.Parameter('gamma', shape=(1, dim, 1, 1),\n",
    "                                     init=init.Constant(layer_scale))\n",
    "\n",
    "    def forward(self, X):\n",
    "        Y = self.norm(self.dwconv(X))\n",
    "        Y = self.pwconv2(self.act(self.pwconv1(Y)))\n",
    "        return X + self.gamma.data() * Y"
   ],
   "execution_count": 2,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=convnext-md-4 fw=all -->\n",
    "\n",
    "### The full network\n",
    "\n",
    "The network is the arch-tuple pattern of that section one more time: a tuple of (depth, channels) pairs, a stem, a loop, a head. The stem is the patchify convolution plus a normalization. Between stages sits the separate downsampling layer, a normalization followed by a $2 \\times 2$ stride-2 convolution; there is no pooling anywhere in the body. Following the paper, the per-block stochastic-depth probability ramps linearly from zero at the first block to its maximum at the last. Our dimensions are a scaled-down ConvNeXt sized for Fashion-MNIST, near the \"atto\" end of the family: widths $(40, 80, 160, 320)$ and depths $(2, 2, 6, 2)$, with the third stage deepest as in the full model."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "convnext-model"
   },
   "source": [
    "class ConvNeXt(d2l.Classifier):\n",
    "    def __init__(self, arch=((2, 40), (2, 80), (6, 160), (2, 320)),\n",
    "                 lr=2e-3, num_classes=10, drop_path_max=0.1):\n",
    "        super().__init__()\n",
    "        self.save_hyperparameters()\n",
    "        self.net = nn.Sequential()\n",
    "        self.net.add(nn.Conv2D(arch[0][1], kernel_size=4, strides=4),\n",
    "                     nn.LayerNorm(axis=1, epsilon=1e-6))\n",
    "        for i, (depth, c) in enumerate(arch):\n",
    "            if i > 0:  # separate downsampling layer between stages\n",
    "                self.net.add(nn.LayerNorm(axis=1, epsilon=1e-6),\n",
    "                             nn.Conv2D(c, kernel_size=2, strides=2))\n",
    "            for _ in range(depth):\n",
    "                self.net.add(ConvNeXtBlock(c))\n",
    "        self.net.add(nn.GlobalAvgPool2D(), nn.Flatten(),\n",
    "                     nn.LayerNorm(axis=-1, epsilon=1e-6),\n",
    "                     nn.Dense(num_classes))\n",
    "        self.net.initialize(init.Xavier())"
   ],
   "execution_count": 3,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=convnext-md-5 fw=all -->\n",
    "\n",
    "A $96 \\times 96$ input leaves the stem as a $24 \\times 24$ map, and the three downsampling layers reduce it to $12 \\times 12$, $6 \\times 6$, and finally $3 \\times 3$ before the head pools it away. We check the output shape and count parameters: 3,376,450, about a third of the 11.2 million in the ResNet-18 we trained in that section. The exact count is a stringent correctness check for any reimplementation, ours included, since a single wrongly sized layer changes it."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "convnext-params"
   },
   "source": [
    "model = ConvNeXt()\n",
    "X = np.random.normal(0, 1, (1, 1, 96, 96))\n",
    "assert model.net(X).shape == (1, 10)\n",
    "sum(p.data().size for p in model.collect_params().values()\n",
    "    if p.grad_req != 'null')"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "3376450"
     },
     "metadata": {},
     "execution_count": 4
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=convnext-md-6 fw=all -->\n",
    "\n",
    "### Training with the modern recipe\n",
    "\n",
    "We train ConvNeXt with the compact modern recipe used in\n",
    "that section: AdamW, cosine decay with warmup, label\n",
    "smoothing, and Mixup. For a controlled local comparison, we train a ResNet-18\n",
    "whose base width is reduced from 64 to 35 channels, giving it nearly the same\n",
    "parameter count as our ConvNeXt. The two models share the data, recipe, epoch\n",
    "budget, and the same seed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=convnext-md-7 fw=mxnet -->\n",
    "\n",
    "The MXNet path above implements the deterministic ConvNeXt transformation but\n",
    "omits stochastic depth and this repeated training comparison. Its available\n",
    "AdamW, cosine-schedule, label-smoothing, and Mixup components are demonstrated\n",
    "in that section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=convnext-md-8 fw=all -->\n",
    "\n",
    "We train for 30 epochs on the full Fashion-MNIST training set at $96 \\times\n",
    "96$, matching the modern-recipe ResNet-18 run of\n",
    "that section epoch for epoch. We set ConvNeXt's stochastic\n",
    "depth rate to zero in this comparison because the compact ResNet control does\n",
    "not use it; the block implementation above still exercises the complete\n",
    "stochastic-depth path."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=convnext-md-9 fw=all -->\n",
    "\n",
    "| Model | Parameters | PyTorch accuracy | JAX accuracy |\n",
    "|---|---:|---:|---:|\n",
    "| ConvNeXt | 3,376,450 | ≈92% | ≈92% |\n",
    "| Compact ResNet-18 | 3,348,320 | ≈94% | ≈94.5% |\n",
    "\n",
    "This comparison asks a narrower question than the earlier run against the\n",
    "11.2-million-parameter ResNet-18: at roughly 3.4 million parameters, does the\n",
    "ConvNeXt block and stem help on upsampled Fashion-MNIST? Each entry is one\n",
    "seeded run, rounded to the half point; the independent JAX run checks that\n",
    "the result is not tied to one implementation. Here the compact\n",
    "ResNet is ahead by about two points in both implementations. Even this control\n",
    "is task-specific: it matches parameters, not latency or multiply-adds, and\n",
    "every step in the published roadmap was selected on ImageNet at\n",
    "$224\\times224$. The local result therefore says that ConvNeXt's ImageNet\n",
    "design choices do not automatically transfer to this small grayscale task; it\n",
    "is not a general ranking of the architecture families.\n",
    "\n",
    "## Beyond ConvNeXt\n",
    "\n",
    "### ConvNeXt V2: pretraining and GRN\n",
    "\n",
    "ConvNeXt closed the supervised gap, but by 2023 the frontier had moved to self-supervised pretraining, where transformers held an advantage: masked autoencoders [@he2022masked] drop most input patches and train the network to reconstruct them, which is natural for a patch-sequence model and awkward for a convolution that slides across the holes. ConvNeXt V2 [@woo2023convnextv2] adapted the idea with sparse convolutions that skip masked regions during pretraining. Doing so exposed a failure the authors traced to feature *collapse*: with the V1 block, many channels of the pretrained network go dead or redundant. Their fix, *Global Response Normalization* (GRN), is a three-line layer inserted after the block's GELU that computes each channel's global response norm, divides by the mean over channels, and uses the ratio to recalibrate the features, sharpening the contrast between channels and preventing collapse; it also replaces layer scale. The combination of the masked-autoencoder pretraining and GRN lifted the largest model to 88.9% ImageNet top-1 accuracy, at the time the best result trained on public data. Implementing GRN is one of the exercises, and it fits in about five lines.\n",
    "\n",
    "### Large Kernels and Other Spatial Mixers\n",
    "\n",
    "The roadmap's $7 \\times 7$ kernel invited an obvious question: why stop there? RepLKNet [@ding2022replknet] pushed depthwise kernels to $31 \\times 31$, using the re-parameterization trick of that section to train a parallel small kernel that folds away at inference, and showed the effective receptive field widening dramatically, with the gains showing up mostly in detection and segmentation rather than classification. SLaK [@liu2022slak] reached $51 \\times 51$ by factorizing the kernel into two thin rectangular stripes plus dynamic sparsity. InternImage [@wang2023internimage] took the opposite route to the same goal, a large *adaptive* receptive field, by building its blocks from deformable convolutions (DCNv3) whose sampling locations are input-dependent, and scaled to 3 billion parameters and state-of-the-art COCO detection. UniRepLKNet [@ding2024unireplknet] carried large-kernel design across modalities, to audio, point clouds, and time series.\n",
    "\n",
    "The durable result is narrower than a verdict on one winning operator. Modern\n",
    "backbones need access to wide context, and ConvNeXt showed that a moderately\n",
    "large depthwise kernel is one practical way to obtain it. Fixed kernels of\n",
    "$31 \\times 31$ and beyond have remained specialized: their reported gains are\n",
    "strongest in dense prediction, and efficient training or inference often needs\n",
    "re-parameterization, sparsity, or specialized kernels. Deformable operators\n",
    "and attention provide alternative, input-dependent routes to wide context.\n",
    "\n",
    "### ConvNeXt in 2026\n",
    "\n",
    "ConvNeXt has aged into infrastructure. It is a standard strong-CNN backbone\n",
    "in detection and segmentation toolkits, a common encoder choice where a\n",
    "transformer's quadratic attention cost is unwelcome at high resolution, and\n",
    "the convolutional tower in several widely used OpenCLIP image-text models\n",
    "[@radford2021learning], where ConvNeXt encoders trained on\n",
    "billion-scale image-text corpora remain among the strongest public\n",
    "convolutional models. When a practitioner in 2026 says \"just use a CNN\",\n",
    "the CNN they reach for is more often than not a ConvNeXt or something\n",
    "shaped like one.\n",
    "\n",
    "## Summary and Discussion\n",
    "\n",
    "ConvNeXt is a controlled experiment wearing an architecture's name. One change\n",
    "at a time, it turned a 2015 ResNet-50 into a network that matches Swin at equal\n",
    "cost: a modern recipe, a Swin-like stage ratio, a patchify stem, depthwise\n",
    "convolutions in an inverted bottleneck led by a $7 \\times 7$ kernel, GELU, and\n",
    "fewer normalization and activation layers. At the scales tested in the paper,\n",
    "well-tuned convnets and transformers are competitive; the experiment does not\n",
    "establish that either family dominates in every data and compute regime. Our\n",
    "parameter-matched Fashion-MNIST comparison applies the same control discipline\n",
    "to the smaller setting used in this book.\n",
    "\n",
    "The block itself is a piece of convergent evolution. Depthwise convolution mixing space but not channels, a pointwise MLP expanding by four mixing channels but not space, one normalization, one activation, a residual connection: strip the names and the ConvNeXt block and the transformer block are the same design with different spatial mixers. That reading, which that section develops, is more durable than any single leaderboard number, including the ones in this section.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Implement Global Response Normalization. For a channels-last feature map $X$, compute per-channel global norms $g_c = \\lVert X_{:,:,c} \\rVert_2$, normalize them as $n_c = g_c / \\bar{g}$ where $\\bar{g}$ is the mean over channels, and return $\\gamma \\odot (X \\cdot n) + \\beta + X$ with learnable per-channel $\\gamma, \\beta$ initialized to zero. Insert it after the GELU in `ConvNeXtBlock` (ConvNeXt V2 also removes layer scale when doing so), retrain, and compare.\n",
    "1. Swap the depthwise kernel size: train the model with $3 \\times 3$ and with $11 \\times 11$ depthwise convolutions (adjust the padding) and compare accuracy, parameter count, and time per epoch against the $7 \\times 7$ baseline. Do you see the saturation that @liu2022convnet report?\n",
    "1. Count where the parameters live: for our ConvNeXt, compute the fraction of parameters in depthwise convolutions, in the $1 \\times 1$ expansions and projections, and in the downsampling layers, and compare with the ResNet-18 of that section. Which design decision explains why ConvNeXt is three times smaller at the same depth-times-width feel?\n",
    "1. Ablate layer scale: train with $\\gamma$ initialized to $10^{-6}$ (the default), to $1$, and with the parameter removed entirely. Relate what you observe to stochastic depth, which also shrinks the effective contribution of each residual branch early in training.\n",
    "1. Our run gives the modern recipe to a modern architecture. Complete the other half of the ablation square: train this ConvNeXt with the 2015 recipe of that section (SGD with momentum, step decay, no Mixup or smoothing). How much of the network's quality survives the recipe downgrade?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-mxnet",
   "display_name": "d2l (MXNet)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
