Generalization in Deep Learning

Dive into Deep Learning · §4.5

Why over-parametrized networks generalize
double descent · the optimizer as regularizer · grokking.

Optimization is a means; generalization is the goal

Motivation

Fitting the training data is only the intermediate step. The real quest is to find patterns that predict well on unseen data.

For deep networks, fitting is easy: they can interpolate almost any training set, even random labels.
So nearly every gain comes from closing the generalization gap.
Yet why remains a major open question, with two surprises ahead: a second descent, and generalization long after memorization.

Bigger past the interpolation threshold is *better*, not worse: the deep-learning surprise this section explains.

Overfitting, revisited

inductive bias and the interpolation regime

No free lunch: every model needs a bias

Overfitting

No algorithm generalizes better on all distributions. Given finite data, a model must lean on inductive biases that prefer some solutions over others.

A deep MLP is biased toward functions built by composing simpler ones.
Good biases match how the world is structured.

The generalization gap is the difference between test and training error. A large gap means we have overfit.

Classical recipe to close the gap: make the model less complex.

fewer features
fewer nonzero parameters
smaller parameters

Deep networks strain the simplest classical picture

Overfitting

Modern networks can be expressive enough to fit large training sets and even random labels.

So the classical levers behave strangely:

Many candidate architectures reach zero training error, leaving their test behavior to distinguish them.
Increasing capacity can lower test error in some interpolating regimes.

Worst-case bounds based on the complete hypothesis class are often vacuous at modern scales. Data- and algorithm-dependent explanations remain active work.

Double descent

the U-curve, and what lies past it

The classical bias-variance tradeoff

Double descent

Classical theory predicts a U-shaped test-error curve as capacity grows:

\underbrace{\text{error}}_{\text{test}} \;=\; \underbrace{\text{bias}^2}_{\downarrow\ \text{with capacity}} \;+\; \underbrace{\text{variance}}_{\uparrow\ \text{with capacity}}.

Too little capacity underfits; too much overfits; a sweet spot sits in between.

This is the left half of the picture on the next slide, and it is all the classical story predicts.

Double descent: a second curve observed in some regimes

One curve, two regimes: the classical U ends at the interpolation threshold, where training error reaches zero and test error peaks — then descends a second time. That second descent is contingent on the model, data, optimizer, and training budget.

One mechanism in tractable models

Double descent

In linear least-squares and random-feature models, the minimum-norm interpolant can have high variance near a rank transition.

Past that transition, additional features create more interpolating solutions, and the minimum-norm solution can have lower variance.

The appendix (the concentration-and-generalization section) derives this mechanism in a specified model. It is not a theorem that every deep network follows the same curve or selects the same kind of interpolant.

Three knobs trace the curve

Double descent

Model size is only one of three ways to cross the threshold (Nakkiran et al., 2021):

model-wise: grow the network (the flavor above);
epoch-wise: fix the network, train longer: error falls, rises while the model interpolates noise, then falls again;
sample-wise: adding training examples can hurt, because more data moves the threshold and can push a fixed model back into the spike.

Nakkiran et al. propose effective model complexity, how many examples the full training procedure (model, optimizer, and budget) can fit. It organizes their observed model-, epoch-, and sample-wise curves, but it is an empirical account rather than a universal law. Exercise 6 has you produce the epoch-wise flavor yourself.

What optimization selects

implicit bias, grokking, and its limits

The optimizer imposes an implicit bias

Implicit bias

Among settings that interpolate the data, the initialization and optimizer make some solutions more likely than others. This bias is model-dependent.

A theorem underlies it:

For a linear predictor on separable data, gradient descent on logistic loss converges in direction to the \ell_2 maximum-margin separator, with no explicit penalty (Soudry et al., 2018).

The linear theorem does not transfer automatically to deep networks. Raw parameter-space flatness is also changed by function-preserving reparameterizations. Weight decay, early stopping, and methods such as SAM interact with the optimizer’s bias, and their effect is empirical.

Grokking: generalization arrives orders of magnitude later

Implicit bias · the second surprise

On small algorithmic tasks, a network first memorizes the training set, with test accuracy flat near chance.

Then, after orders of magnitude more steps at zero training loss, it suddenly generalizes, long after any early-stopping rule would have given up.

Same architecture, same data, same loss: only time spent training changes the outcome. Dynamics, not architecture alone, govern generalization.

Training accuracy (gray) saturates early; validation accuracy (blue) lingers near chance for orders of magnitude more steps, then rises sharply (after Power et al., 2022).

A nonparametric lens: the neural tangent kernel

Implicit bias

Networks have parameters, yet often behave like nonparametric models, whose complexity grows with the data.

As a randomly initialized MLP grows infinitely wide, it becomes equivalent to a kernel method for a fixed kernel:

the neural tangent kernel, a distance function the architecture induces.

Memorization and generalization can coexist: 1-nearest-neighbor memorizes everything, achieves zero training error, and still lands asymptotically within a factor of two of the optimal Bayes error (Cover & Hart, 1967).

Over-parametrized nets likewise interpolate, so the nonparametric intuition is often the more reliable one.

Regularization in practice

early stopping and classical penalties, reinterpreted

Early stopping: fit clean data first

Practice

With label noise, networks fit the cleanly labeled examples first and only later interpolate the mislabeled ones.

Stop while the clean data is fit but the noise is not, and you can certify generalization (with high probability).

Patience criterion: monitor validation error each epoch; stop when it fails to improve by \epsilon for a few epochs. It also saves training time and money.

Matters most under label noise or intrinsic label variability. On clean, separable data it changes little.

Classical penalties, reinterpreted

Practice

Weight decay adds a norm penalty to the loss:

\ell_2\ (\text{ridge}):\ \lambda\lVert\mathbf{w}\rVert_2^2 \qquad \ell_1\ (\text{lasso}):\ \lambda\lVert\mathbf{w}\rVert_1.

But at typical strengths, \ell_2 is too weak to stop a network from interpolating the data.

So its benefit may come not from constraining capacity but from encoding a better inductive bias, much like the choice of distance in k-NN. Often it helps only together with early stopping. (Dropout is the next such tool.)

Recap

Wrap-up

Many deep nets are over-parametrized and can interpolate their training sets; whether they do so depends on the training procedure.
Double descent occurs in some regimes near an interpolation threshold; other regimes retain a U-shaped or monotone curve.
Worst-case class-level bounds are often vacuous at modern scales; tighter explanations remain an open problem.

Implicit bias: optimizers favor some interpolants. Maximum-margin convergence is proved for separable linear logistic regression, not for arbitrary deep networks.
Nonparametric view: infinitely wide nets become kernel methods.
In practice: validate early stopping and weight decay rather than assigning them one universal mechanism.

Why certain choices improve generalization remains a massive open question. Try the surprises yourself: exercise 6 reproduces epoch-wise double descent; exercise 7, grokking on modular addition. Next (the dropout section): dropout, regularization by structured noise.