Training Recipes Matter

Recipes matter more than architectures

Take ResNet-50, unchanged since 2015, and retrain it with 2022 methods (Wightman et al., 2021):

• 2015 recipe: 76.1% ImageNet top-1
• modern recipe, 100 epochs: 78.1%
• modern recipe, 600 epochs + heavy augmentation: 80.4%

Four points from the training procedure alone: more than most architecture generations. Consequence: pre- and post-2021 paper numbers are not comparable.

What changed, 2015 → 2022

Ingredient	2015	2022
Optimizer	SGD + momentum	AdamW / LAMB
Schedule	step decay	cosine + warmup
Augmentation	flips, crops	RandAugment, Mixup, CutMix
Regularization	weight decay	+ smoothing, stochastic depth
Duration	90 epochs	300 - 600 epochs
Evaluated weights	last checkpoint	EMA of weights

Label smoothing

One-hot targets reward unbounded confidence: the loss keeps falling as the correct logit grows. Smoothing puts weight 1-\epsilon on the label, \epsilon/K everywhere, so the optimum is a finite logit gap:

logits, y = d2l.tensor([[10.0, 0.0, 0.0]]), d2l.tensor([0])
for epsilon in (0.0, 0.1):
    loss = F.cross_entropy(logits, y, label_smoothing=epsilon)
    print(f'epsilon={epsilon}: loss={float(loss):.4f}')

epsilon=0.0: loss=0.0001
epsilon=0.1: loss=0.6668

A confident correct prediction no longer has near-zero loss.

Cosine schedule with warmup

Two phases: linear warmup (a fresh network produces large, badly scaled gradients), then a half-cosine decay to zero. No drop times to tune.

def step_decay(epoch, max_epochs, base_lr):
    """Recipe A: multiply the rate by 0.1 at 60% and 85% of the budget."""
    return base_lr * 0.1 ** sum(epoch >= int(f * max_epochs)
                                for f in (0.6, 0.85))

def cosine_warmup(epoch, max_epochs, base_lr, warmup=3):
    """Recipe B: linear warmup, then a half-cosine decay to zero."""
    if epoch < warmup:
        return base_lr * (epoch + 1) / warmup
    t = (epoch - warmup) / (max_epochs - warmup)
    return base_lr * 0.5 * (1 + math.cos(math.pi * t))

epochs = list(range(45))
d2l.plot(epochs, [[step_decay(e, 45, 0.1) for e in epochs],
                  [cosine_warmup(e, 45, 0.1) for e in epochs]],
         xlabel='epoch', ylabel='learning rate',
         legend=['step decay', 'cosine with warmup'])

Mixup: train between the data points

Blend two images and their labels with \lambda \sim \mathrm{Beta}(\alpha, \alpha):

\tilde{\mathbf{x}} = \lambda \mathbf{x}_i + (1-\lambda) \mathbf{x}_j, \qquad \tilde{y} = \lambda y_i + (1-\lambda) y_j.

Cross-entropy is linear in the target, so this is just a \lambda-weighted pair of ordinary losses:

def mixup(X, y, alpha):
    """Return a mixed batch, both label sets, and the mixing weight."""
    lam = float(torch.distributions.Beta(alpha, alpha).sample())
    perm = torch.randperm(X.shape[0], device=X.device)
    return lam * X + (1 - lam) * X[perm], y, y[perm], lam

Mixed images

data = d2l.FashionMNIST(batch_size=8)
X, y = next(iter(data.train_dataloader()))
X_mix, y_a, y_b, lam = mixup(X, y, alpha=2.0)
d2l.show_images(torch.cat([X, X_mix]).squeeze(1), 2, 8)
print(f'lambda = {float(lam):.2f}')

lambda = 0.17

CutMix is the spatial sibling: paste a rectangle instead of blending, mix labels by area. Modern recipes alternate both.

Stochastic depth

Drop a residual block’s entire branch per sample with probability p; the identity path carries the signal. Rescale by 1/(1-p) so evaluation needs no change:

def drop_path(Y, p, training):
    if not training or p == 0:
        return Y
    keep = (torch.rand(Y.shape[0], 1, 1, 1, device=Y.device) > p).float()
    return Y * keep / (1 - p)

class StochasticResidual(d2l.Residual):
    """A residual block whose branch is dropped with probability p."""
    def __init__(self, num_channels, p):
        super().__init__(num_channels)
        self.p = p

    def forward(self, X):
        Y = self.bn2(self.conv2(F.relu(self.bn1(self.conv1(X)))))
        return F.relu(drop_path(Y, self.p, self.training) + X)

blk, X = StochasticResidual(3, p=0.5), torch.randn(1000, 3, 8, 8)
blk(X)  # Initialize the lazy layers
blk.train()
dropped = (blk(X) == F.relu(X)).flatten(1).all(1).float().mean()
print(f'fraction of samples with a dropped branch: {float(dropped):.3f}')

fraction of samples with a dropped branch: 0.495

EMA: evaluate the average, not the last step

SGD iterates jitter around a good region. A shadow copy \bar{\theta} \leftarrow \beta \bar{\theta} + (1-\beta) \theta cancels the jitter for the price of one parameter copy:

class EMA:
    """A shadow copy of a model's parameters, updated after each step."""
    def __init__(self, model, decay=0.99):
        self.decay = decay
        self.shadow = {k: v.detach().clone()
                       for k, v in model.state_dict().items()}

    def update(self, model):
        for k, v in model.state_dict().items():
            if v.dtype.is_floating_point:
                self.shadow[k].mul_(self.decay).add_(v, alpha=1 - self.decay)
            else:
                self.shadow[k].copy_(v)  # e.g., batch-norm step counters

net = nn.Linear(1, 1, bias=False)
ema, raw, avg = EMA(net, decay=0.9), [], []
for t in range(100):
    with torch.no_grad():  # Simulate noisy SGD iterates around 1.0
        net.weight.copy_(1 + 0.3 * torch.randn(1, 1))
    ema.update(net)
    raw.append(float(net.weight))
    avg.append(float(ema.shadow['weight']))
d2l.plot(list(range(100)), [raw, avg], xlabel='step', ylabel='weight',
         legend=['raw iterates', 'EMA'])

The experiment: one network, two recipes

Same ResNet-18, Fashion-MNIST at 96×96, 10k training images (with all 60k images the gap largely closes; scarcity is where regularization earns its keep).

class RecipeTrainer(d2l.Trainer):
    """A Trainer that sets the learning rate from the model's schedule."""
    def fit_epoch(self):
        for group in self.optim.param_groups:
            group['lr'] = self.model.lr_at(self.epoch, self.max_epochs)
        super().fit_epoch()

class ClassicRecipe(ResNet18):
    """Recipe A, ca. 2015: SGD with momentum, step decay, plain loss."""
    def configure_optimizers(self):
        return torch.optim.SGD(self.parameters(), lr=self.lr,
                               momentum=0.9, weight_decay=5e-4)

    def lr_at(self, epoch, max_epochs):
        return step_decay(epoch, max_epochs, self.lr)

class ModernRecipe(ResNet18):
    """Recipe B, ca. 2022: AdamW, cosine warmup, smoothing, Mixup."""
    def configure_optimizers(self):
        return torch.optim.AdamW(self.parameters(), lr=self.lr,
                                 weight_decay=0.05)

    def lr_at(self, epoch, max_epochs):
        return cosine_warmup(epoch, max_epochs, self.lr)

    def loss(self, y_hat, y):
        return F.cross_entropy(y_hat, y, label_smoothing=0.1)

    def training_step(self, batch):
        X, y_a, y_b, lam = mixup(*batch, alpha=0.2)
        y_hat = self(X)
        l = lam * self.loss(y_hat, y_a) + (1 - lam) * self.loss(y_hat, y_b)
        self.plot('loss', l, train=True)
        return l

Recipe A: 2015

SGD + momentum, step decay, plain cross-entropy, 30 epochs:

data = FashionMNIST10k()
torch.manual_seed(1)
model = ClassicRecipe(lr=0.05)
trainer = RecipeTrainer(max_epochs=30, num_gpus=1)
model.apply_init([next(iter(data.get_dataloader(True)))[0]], d2l.init_cnn)
trainer.fit(model, data)
classic_score = val_accuracy(model, data, trainer)
classic_score

0.9042

Recipe B: modern

AdamW + cosine warmup + label smoothing + Mixup, 45 epochs:

torch.manual_seed(1)
model = ModernRecipe(lr=2e-3)
trainer = RecipeTrainer(max_epochs=45, num_gpus=1)
model.apply_init([next(iter(data.get_dataloader(True)))[0]], d2l.init_cnn)
trainer.fit(model, data)
modern_score = val_accuracy(model, data, trainer)
modern_score

0.9138

Results

10k training images; test accuracy.

Implementation	Recipe A	Recipe B
PyTorch	≈90.5%	≈91%
JAX	≈90.5%	≈91%

A consistent (if modest) gain; it grows as data gets scarcer. Caveats: retune when ablating (A’s learning rate diverges under AdamW), and never compare training losses across recipes.

Reading the scoreboard

• ImageNet top-1 is saturated and recipe-confounded: attach mental error bars to cross-paper gaps.
• The ImageNet-V2 “drop” is largely a label-noise artifact: cleaner ReaL labels close most of it.
• What still separates backbones: transfer to COCO detection and ADE20K segmentation.

Recap

• Recipe gains on a fixed ResNet-50: 76.1% → 80.4%, bigger than most architecture jumps.
• Ingredients are each a few lines: smoothing, cosine + warmup, Mixup/CutMix, stochastic depth, weight EMA.
• Regularization and duration move together; heavily regularized short runs underperform.
• No architecture comparison is meaningful without matched recipes: the strong baseline is the control experiment.