Generalization

Dive into Deep Learning · §2.6

Fitting the training data is not the goal
telling memorization apart from learning · the U-curve · model selection.

Two students, one exam

The parable

Two students prepare from the same stack of past exams.

Extraordinary Ellie memorizes every answer: 100% on any question she has seen, and frozen by one she has not.
Inductive Irene can barely memorize, but picks up patterns: a steady 90%, seen or unseen.

If the exam recycles old questions, Ellie wins. If it is fresh, Irene does. Every trained model is one of these two students, and the training error alone cannot tell you which.

This section builds the instruments that can: held-out data, the generalization gap, and the bias-variance trade-off.

Two Errors

the statistic we see vs. the expectation we want

Training error vs. generalization error

Two Errors

Training error, an average over the data we have: R_\textrm{emp} = \tfrac1n \sum_{i=1}^n l\bigl(\mathbf{x}^{(i)}, y^{(i)}, f\bigr)

Generalization error, an expectation over data we will never fully see: R = E_{(\mathbf{x},y)\sim P}\bigl[\,l(\mathbf{x}, y, f)\,\bigr]

We can never compute R. We estimate it on held-out data: a fixed model on fresh samples is just mean estimation.

The IID assumption

Two Errors

Train and test are drawn independently from the same distribution P(X,Y).

The training error is a biased gauge of R: the model was chosen using that very data, so it is optimistically biased.

Drop IID, let the distribution shift from P to Q, and without an assumption relating them, source performance does not determine target performance.

Model Complexity

the bias-variance trade-off and the U-curve

The bias-variance trade-off

Model Complexity

Too simple → misses the signal: high bias (underfitting).
Too flexible → fits sample-specific noise: high variance (overfitting).

Their sum, plus an irreducible noise floor, is the test error, which is minimized at an intermediate capacity in this classical picture.

Bias falls and variance rises with complexity; their sum, plus an irreducible noise floor, is the U-shaped test error.

Reading the gap

Model Complexity

Both errors high, small gap → too simple. Underfitting; consider a more expressive model.
Train error far below test → severe overfitting.

The generalization gap is R - R_\textrm{emp}.

Overfitting is not always bad: the best deep models often fit training data far better than holdout. The objective is low R; the gap is diagnostic rather than an objective by itself.

What makes a model complex?

Model Complexity

A model class that can fit any labeling receives no support from training fit alone (Popper’s falsifiability).

Complexity is more than parameter count: it is also the range of values parameters may take. Kernel methods have infinitely many parameters yet stay controlled.

Low training error alone neither certifies nor rules out low generalization error.

The Demo

fit polynomials of growing degree to a noisy cubic

Polynomial fitting is linear regression in disguise

The Demo

Predict \hat y = \sum_{i=0}^d x^i w_i: take the powers of x as features and it is plain least squares, with the degree d as a capacity control. The experiment uses a degree-3 target and only 20 training points, so high-degree fits can overfit:

np.random.seed(0)
max_degree = 20                  # highest polynomial degree we will fit
n_train, n_test = 20, 100        # few training points, so high degrees overfit
true_w = np.zeros(max_degree)
true_w[:4] = np.array([5, 1.2, -3.4, 5.6])

x = np.random.uniform(-1, 1, size=n_train + n_test)
poly = np.power(x.reshape(-1, 1), np.arange(max_degree))   # column i holds x**i
labels = poly @ true_w + np.random.normal(scale=0.1, size=n_train + n_test)

One fit per degree, scored on both splits

The Demo

Fit the first d{+}1 power columns by least squares; record the loss on train and on 100 held-out test points.

def fit_degree(d):
    cols = slice(0, d + 1)
    w, *_ = np.linalg.lstsq(poly[:n_train, cols], labels[:n_train], rcond=None)
    err = poly[:, cols] @ w - labels
    return (err[:n_train] ** 2).mean(), (err[n_train:] ** 2).mean()

The three cases illustrate degree 1 (too rigid), degree 3 (the generating degree), degree 19 (one parameter per data point).

What the sweep produces

The Demo · result

Sweep the degree from 1 to 19: training loss falls monotonically, while test loss reaches a minimum near the generating degree 3, then rises as surplus capacity fits noise.

Redrawing the training noise 200 times and refitting decomposes that test error into its two estimated parts: bias² becomes small once the model class contains the cubic target, while variance rises at high degrees. Their sum is smallest near degree 3 for this setup.

Measured test error first falls, then rises: the U-curve, traced by fitting polynomials of growing degree.

Dataset size and model capacity

The Demo

Fix the model: fewer samples means more, and more severe, overfitting.

So complexity should grow with data, not ahead of it. Larger datasets can support more expressive models, although the required sample size depends on the task and representation.

Model Selection

keep the test set honest with a validation split

Never select on the test set

Model Selection

Using test data to choose a model makes the reported test score adaptively biased and removes its role as an independent final evaluation.

So split three ways: train, validation (for model selection), test (touched once). Most “test” accuracy in practice is really validation accuracy.

K-fold cross-validation

Model Selection

When data is too scarce to spare a validation set: split into K folds, train on K{-}1, validate on the held-out one, rotate, and average the K scores.

Choosing K trades bias, variance, and compute: each fold trains on (K{-}1)/K of the data. If performance improves monotonically with more data, the estimate is pessimistic relative to the final full-data fit. Larger K narrows that gap but costs more fits and uses nearly identical, correlated training sets. K = 5 or 10 is a common compromise.

Each of the K folds serves once as the validation set; average the K validation scores.

Beyond the Classical U-Curve

when more capacity helps again

Double descent

Beyond the Classical U-Curve

The classical U-curve does not describe every overparameterized regime. Once capacity is large enough to interpolate the data, pushing it further often makes test error fall again.

The generalization-in-deep-learning section takes up the modern story; the concentration-and-generalization section reproduces this curve from scratch and explains the peak.

Past the interpolation threshold, test error descends a second time, the over-parametrized regime of deep learning.

Rules of thumb

Wrap-up

Generalization, not training fit, is the goal: mind the gap R - R_\textrm{emp}.
Zero training error alone does not certify generalization: degree 19 fit 20 points exactly and tested at 5\times10^{13}.
Bias-variance: bias² falls, variance rises; their sum (plus a noise floor) is the test-error U-curve, and we computed both.

Select models with a validation set or K-fold CV (K=5–10), never the test set.
Additional representative data often helps; let complexity grow with it, not ahead of it.
All of this rests on IID, and huge models can defy the classical U via double descent.