ConvNeXt: A ConvNet for the 2020s

The question in 2021

Vision transformers (ViT, then Swin) took over ImageNet and detection/segmentation backbones.

But a transformer differs from a 2015 ResNet in many ways besides attention: recipe, stem, activations, normalization, stage shape.

Liu et al., 2022: change a ResNet-50 one step at a time toward Swin’s design, measure after each step, never add attention.

End point: ConvNeXt, 82.0% vs. Swin-T’s 81.3% at equal FLOPs.

The modernization roadmap

Change Top-1
ResNet-50, 2015 recipe 76.1%
modern training recipe 78.8%
stage ratio (3,4,6,3) → (3,3,9,3) 79.4%
patchify stem (4×4, stride 4) 79.5%
depthwise conv + width 64 → 96 80.5%
inverted bottleneck 80.6%
7×7 depthwise, moved first 80.6%
GELU; fewer activations 81.3%
fewer norms; BN → LN 81.5%
separate downsampling 82.0%

Read the first row first

The largest single jump, +2.7 points, is the recipe, not the architecture (and RSB pushed the same ResNet-50 to 80.4%).

Of the 5.9 points from 2015 ResNet-50 to ConvNeXt:

  • 2.7 training, 3.2 architecture.

Papers comparing tuned transformers against 2015-recipe ResNets were largely comparing recipes.

Macro design

  • Stage ratio: spend depth where maps are small, (3,3,9,3) like Swin-T (+0.6).
  • Patchify stem: a ViT patch embedding is a convolution whose kernel equals its stride: 4×4, stride 4 (79.5%).

No pooling, no 7×7 stem: one strided conv slices the image into patches.

The block, before and after

Three norms and three activations become one of each; the bottleneck inverts and leads with a 7×7 depthwise conv.

Why this shape

The transformer block factorizes: attention mixes positions, the MLP mixes channels.

Convnets have owned that factorization since MobileNet: depthwise conv + 1×1 convs.

  • depthwise 3×3, width to 96: 80.5%
  • inverted bottleneck (expand 4×, like a transformer MLP): 80.6%
  • depthwise kernel to 7×7 (cheap once depthwise): 80.6%, saturates beyond 7×7
  • one GELU, one LayerNorm per block: 81.5%

ConvNeXt-T assembled

Patchify stem, stages 3:3:9:3, LN + 2×2 s2 downsampling between stages.

The block in code

LN over channels at each position (a transformer’s LN): go channels-last, then 1×1 convs are Linear layers:

def drop_path(Y, p, training):  # Stochastic depth, as in the previous section
    if not training or p == 0:
        return Y
    keep = (torch.rand(Y.shape[0], 1, 1, 1, device=Y.device) > p).float()
    return Y * keep / (1 - p)

class ConvNeXtBlock(nn.Module):
    """Depthwise 7x7, LN, 1x1 expand, GELU, 1x1 project, scaled residual."""
    def __init__(self, dim, drop_prob=0.0, layer_scale=1e-6):
        super().__init__()
        self.dwconv = nn.Conv2d(dim, dim, kernel_size=7, padding=3,
                                groups=dim)
        self.norm = nn.LayerNorm(dim, eps=1e-6)
        self.pwconv1 = nn.Linear(dim, 4 * dim)  # 1x1 conv in channels-last
        self.pwconv2 = nn.Linear(4 * dim, dim)
        self.gamma = nn.Parameter(layer_scale * torch.ones(dim))
        self.drop_prob = drop_prob

    def forward(self, X):
        Y = self.dwconv(X).permute(0, 2, 3, 1)  # to channels-last
        Y = self.pwconv2(F.gelu(self.pwconv1(self.norm(Y))))
        Y = (self.gamma * Y).permute(0, 3, 1, 2)  # back to channels-first
        return X + drop_path(Y, self.drop_prob, self.training)

Layer scale (\gamma \approx 10^{-6}): every block starts near-identity. Stochastic depth: the same drop_path as in the previous section.

The network: the arch tuple again

Widths (40, 80, 160, 320), depths (2, 2, 6, 2), a scaled-down “atto” ConvNeXt:

class LayerNorm2d(nn.LayerNorm):
    """LayerNorm over the channel axis of an NCHW tensor."""
    def forward(self, X):
        return super().forward(X.permute(0, 2, 3, 1)).permute(0, 3, 1, 2)

class ConvNeXt(d2l.Classifier):
    arch = ((2, 40), (2, 80), (6, 160), (2, 320))

    def __init__(self, lr=2e-3, num_classes=10, drop_path_max=0.1):
        super().__init__()
        self.save_hyperparameters()
        depths = [d for d, c in self.arch]
        rates = torch.linspace(0, drop_path_max, sum(depths)).tolist()
        layers = [nn.Conv2d(1, self.arch[0][1], kernel_size=4, stride=4),
                  LayerNorm2d(self.arch[0][1], eps=1e-6)]
        c_prev, b = self.arch[0][1], 0
        for i, (depth, c) in enumerate(self.arch):
            if i > 0:  # separate downsampling layer between stages
                layers += [LayerNorm2d(c_prev, eps=1e-6),
                           nn.Conv2d(c_prev, c, kernel_size=2, stride=2)]
            for _ in range(depth):
                layers.append(ConvNeXtBlock(c, drop_prob=rates[b]))
                b += 1
            c_prev = c
        layers += [nn.AdaptiveAvgPool2d((1, 1)), nn.Flatten(),
                   nn.LayerNorm(c_prev, eps=1e-6),
                   nn.Linear(c_prev, num_classes)]
        self.net = nn.Sequential(*layers)

Parameter count as a checksum

model = ConvNeXt()
X = np.random.normal(0, 1, (1, 1, 96, 96))
assert model.net(X).shape == (1, 10)
sum(p.data().size for p in model.collect_params().values()
    if p.grad_req != 'null')
3376450

3,376,450 parameters: one wrongly sized layer changes this number, so matching it validates a reimplementation. A third of ResNet-18’s 11.2M.

Train it with the modern recipe

ConvNeXt was never trained any other way: AdamW + cosine warmup + label smoothing + Mixup, from the previous section:

def cosine_warmup(epoch, max_epochs, base_lr, warmup=3):
    if epoch < warmup:
        return base_lr * (epoch + 1) / warmup
    t = (epoch - warmup) / (max_epochs - warmup)
    return base_lr * 0.5 * (1 + math.cos(math.pi * t))

def mixup(X, y, alpha):
    lam = float(torch.distributions.Beta(alpha, alpha).sample())
    perm = torch.randperm(X.shape[0], device=X.device)
    return lam * X + (1 - lam) * X[perm], y, y[perm], lam

class RecipeTrainer(d2l.Trainer):
    """A Trainer that sets the learning rate from the model's schedule."""
    def fit_epoch(self):
        for group in self.optim.param_groups:
            group['lr'] = cosine_warmup(self.epoch, self.max_epochs,
                                        self.model.lr)
        super().fit_epoch()

class ModernConvNeXt(ConvNeXt):
    """ConvNeXt under the modern recipe of the previous section."""
    def __init__(self, lr=2e-3, num_classes=10, drop_path_max=0.0):
        super().__init__(lr, num_classes, drop_path_max)

    def configure_optimizers(self):
        return torch.optim.AdamW(self.parameters(), lr=self.lr,
                                 weight_decay=0.05)

    def loss(self, y_hat, y):
        return F.cross_entropy(y_hat, y, label_smoothing=0.1)

    def training_step(self, batch):
        X, y_a, y_b, lam = mixup(*batch, alpha=0.2)
        y_hat = self(X)
        l = lam * self.loss(y_hat, y_a) + (1 - lam) * self.loss(y_hat, y_b)
        self.plot('loss', l, train=True)
        return l

class CompactResNet18(d2l.Classifier):
    """A parameter-matched ResNet-18 with base width 35."""
    def __init__(self, lr=2e-3, num_classes=10, base=35):
        super().__init__()
        self.save_hyperparameters()
        channels = (base, 2 * base, 4 * base, 8 * base)
        layers = [nn.Conv2d(1, base, 7, stride=2, padding=3),
                  nn.BatchNorm2d(base), nn.ReLU(),
                  nn.MaxPool2d(3, stride=2, padding=1)]
        for i, c in enumerate(channels):
            for j in range(2):
                down = i > 0 and j == 0
                layers.append(d2l.Residual(c, use_1x1conv=down,
                                           strides=2 if down else 1))
        layers += [nn.AdaptiveAvgPool2d((1, 1)), nn.Flatten(),
                   nn.Linear(channels[-1], num_classes)]
        self.net = nn.Sequential(*layers)

    configure_optimizers = ModernConvNeXt.configure_optimizers
    loss = ModernConvNeXt.loss
    training_step = ModernConvNeXt.training_step

Results

data = d2l.FashionMNIST(batch_size=128, resize=(96, 96))

def train_model(model_cls, seed):
    torch.manual_seed(seed)
    model = model_cls(lr=2e-3)
    trainer = RecipeTrainer(max_epochs=30, num_gpus=1)
    trainer.fit(model, data)
    model.eval()
    correct, n = 0.0, 0
    with torch.no_grad():
        for X, y in map(trainer.prepare_batch, data.val_dataloader()):
            correct += float(model.accuracy(
                model(X), y, averaged=False).sum())
            n += len(y)
    return model, correct / n

scores, models = {}, {}
for name, cls in (('ConvNeXt', ModernConvNeXt),
                  ('compact ResNet-18', CompactResNet18)):
    model, acc = train_model(cls, seed=1)
    models[name] = model
    scores[name] = acc
for name, acc in scores.items():
    count = sum(p.numel() for p in models[name].parameters())
    print(f'{name}: {count:,} parameters, val acc {acc:.3f}')

Read the result plainly

  • ConvNeXt, 3.4M params: about 92%.
  • Parameter-matched Compact ResNet-18, 3.35M, same recipe and budget: about 94%.

The compact ResNet is ahead by about two points in both PyTorch and JAX. The control matches parameters, not latency or multiply-adds.

Every roadmap step was selected on ImageNet at 224 px. On upsampled 28 px data, ResNet’s overlapping stem + BN + size win. Architectures are good for a regime, not in the abstract.

ConvNeXt V2, briefly

Masked-autoencoder pretraining for convnets (FCMAE): sparse convolutions skip the masked holes.

Pretraining exposed feature collapse; the fix is Global Response Normalization: normalize each channel’s global response by the mean over channels (~5 lines; exercise 1).

Result: 88.9% top-1, the best public-data model at the time.

Large kernels, honestly

  • RepLKNet: 31×31 depthwise (re-param trick to train).
  • SLaK: 51×51 via stripes + sparsity.
  • InternImage: deformable conv (DCNv3), adaptive field, 3B params.
  • UniRepLKNet: large kernels across audio/points/time series.

Verdict: the principle (large effective receptive field, bought depthwise) stuck. Giant dense kernels did not become defaults; deformable/adaptive operators and attention carried it further.

Recap

  • ConvNeXt = a controlled ablation: ResNet-50 → 82.0%, matching Swin-T, with no attention.
  • Largest step: the recipe. Then: patchify, depthwise + inverted bottleneck, 7×7, one norm + one activation, LN.
  • Strip the names: ConvNeXt block ≡ transformer block with a depthwise conv as the spatial mixer.
  • 2026: a default strong-CNN backbone (OpenCLIP towers, detection/segmentation encoders).