Environment and Distribution Shift
Dive into Deep Learning · §3.7
When the world stops matching the training set
Distribution shift: how it breaks models, and what we can do about it.
The question we skipped
Why this matters
We fit models to data and measure test accuracy. But we rarely ask where the data came from or what the prediction will be used for.
A loan model finds that Oxfords repay, sneakers default. Approve everyone in Oxfords, and soon everyone wears Oxfords, with no change in who actually repays. The decision broke the signal.
This is Goodhart’s law: when a measure becomes a target, it ceases to be a good measure. Deploying a model can perturb the very distribution it was trained on.
Train here, deploy there
The setup
Training data is drawn from a source distribution p_S(\mathbf{x}, y); at test time we meet a target p_T(\mathbf{x}, y) that may differ.
With no link between p_S and p_T, learning cannot transfer. Suppose the inputs are unchanged, p_S(\mathbf{x})=p_T(\mathbf{x}), but every label flips, p_S(y\mid\mathbf{x})=1-p_T(y\mid\mathbf{x}): “cats” become “dogs” overnight. No algorithm can tell this apart from no shift at all.
The way out is structure: assume how the world may change, and that assumption buys us detection, sometimes correction.
01
Three Kinds of Shift
what stays fixed tells you what to do
Label shift: the mix moves
Three kinds of shift
The label frequencies P(y) change, but each class still looks the same: P(\mathbf{x}\mid y) is fixed.
The natural assumption when y causes \mathbf{x}: diseases cause symptoms, so as an outbreak shifts how common a diagnosis is, the symptom pattern per disease is unchanged.
Why prefer it when both could apply? Its corrections live in label space (low-dimensional categories), not in high-dimensional input space, exactly the cheap side in deep learning.
Concept shift: the labels themselves move
Three kinds of shift
Now the definition of a label drifts: P(y\mid\mathbf{x}) changes because what counts as the answer changed.
What people call a soft drink depends on where you ask (“soda”, “pop”, “coke”).
Diagnostic criteria, fashion, and job titles can drift this way across time or geography. Gradual drift can sometimes be tracked with fresh labeled data; abrupt changes require faster detection and may require a new model.
02
When It Bites
spurious features and slow drift
The model learned the wrong thing
When it bites
A blood-test startup drew healthy controls from students, sick patients from the clinic. The classifier hit near-perfect accuracy, on age, hormones, and diet, not the disease.
The tank fable. A net “detects tanks” perfectly on held-out images, then fails in the field. The tank photos were shot at noon, the empty ones at dawn: it learned the lighting, never the tank.
A spurious feature, present in your sample but gone at deployment, fools a model that never saw the distinction you care about.
Slow drift, stale model
When it bites
The subtler failure: the distribution moves gradually (a nonstationary world) and the model is never refreshed.
A spam filter stops working once spammers craft messages unlike any seen before.
A recommender keeps pushing Santa hats long after Christmas.
The signal did not break all at once, it eroded while nobody was watching.
03
Correcting Shift
reweighting the risk we cannot see
Risk vs. empirical risk
The frame
What we want to minimize is the risk: expected loss under the true distribution p(\mathbf{x}, y).
R(f) = \mathbb{E}_{(\mathbf{x}, y)\sim p}\,[\,l(f(\mathbf{x}), y)\,].
We cannot evaluate it, so we minimize the empirical risk, the average loss on the training sample, and hope the two agree.
\hat{R}(f) = \frac{1}{n}\sum_{i=1}^{n} l(f(\mathbf{x}_i), y_i).
Under shift, the training sample comes from the wrong distribution, so this hope fails, unless we correct the average.
Reweight to the right distribution
Covariate shift correction
Labeled data comes from source q(\mathbf{x}), but we care about target p(\mathbf{x}). Because p(y\mid\mathbf{x})=q(y\mid\mathbf{x}), one identity fixes the risk:
\mathbb{E}_{p}[\,l\,] = \mathbb{E}_{q}\!\left[\,\frac{p(\mathbf{x})}{q(\mathbf{x})}\, l\,\right].
So reweight each example by how much more likely it is under the target than the source, and minimize a weighted empirical risk:
\beta_i = \frac{p(\mathbf{x}_i)}{q(\mathbf{x}_i)}, \qquad \min_f\ \frac{1}{n}\sum_{i=1}^{n}\beta_i\, l(f(\mathbf{x}_i), y_i).
A classifier estimates the weights
Covariate shift correction
We do not know p/q. But train a classifier to tell source from target (z=+1 for target, -1 for source), and the odds are the ratio:
\frac{P(z{=}1\mid\mathbf{x})}{P(z{=}{-}1\mid\mathbf{x})} = \frac{p(\mathbf{x})}{q(\mathbf{x})}.
With a logistic model P(z{=}1\mid\mathbf{x})=\sigma(h(\mathbf{x})) this collapses to \beta_i = \exp(h(\mathbf{x}_i)). We need only unlabeled target features \mathbf{x}\sim p.
Where the weights explode, and why we clip
Covariate shift correction · geometry
![]()
Training data comes from the source q (left curve); the risk we care about weights points by the target p (right curve). The weight \beta = p/q is near zero where only the source has mass, crosses 1 where the densities agree, and explodes out in the tail where the source has almost nothing; the dashed line clips it at a ceiling c.
Clip \beta_i \leftarrow \min(\exp(h(\mathbf{x}_i)), c): where the domains barely overlap, a few examples grab enormous weights and dominate the objective, so a little bias buys much less variance. If p > 0 where q = 0, the true weight is infinite: no reweighting can conjure data that was never sampled.
The discriminator recovers the truth
Covariate shift correction · watch it work
A 2-D rig: source Gaussian at the origin, target the same Gaussian shifted to (2, 0), one shared curved labeling rule (covariate shift by construction), with a known answer: the true log-ratio is 2x_1 - 2. Pool the inputs, train the domain classifier h:
learned h(x) = 2.06 x1 +0.09 x2 -2.03 (true log-ratio: 2 x1 - 2)
beta on source data: mean 0.86, max 56.2
Learned: 2.06\,x_1 + 0.09\,x_2 - 2.03. The discriminator is the density ratio, and note the \beta tail: one source point already carries weight 56.
Reweighting turns a coin flip into 0.93
Covariate shift correction · the verdict
Train the actual classifier three ways on the same labeled source data; evaluate on the target, the domain we care about:
unweighted target accuracy: 0.502 (source accuracy: 0.860)
weighted target accuracy: 0.933 (source accuracy: 0.795)
clipped, c=5 target accuracy: 0.945 (source accuracy: 0.817)
Unweighted fits where the source lives: 0.502 on the target, a coin flip. Reweighting: 0.933, bought by a worse fit on the discounted source region, exactly the trade the identity prescribes. Clipping at c=5 tames the \beta > 50 outliers and even helps: 0.945.
Label shift: invert a confusion matrix
Label shift correction
Here P(y) shifts while P(\mathbf{x}\mid y) is fixed, so the weights are label ratios \beta_i=p(y_i)/q(y_i), and we never touch the high-dimensional inputs.
Take an off-the-shelf classifier, measure its k\times k confusion matrix \mathbf{C} on a source validation set (the very matrix we computed for Fashion-MNIST in the softmax-from-scratch section, column-normalized), and the average prediction \mu(\hat{\mathbf{y}}) on the (unlabeled) target. They are linked by total probability:
\mathbf{C}\, p(\mathbf{y}) = \mu(\hat{\mathbf{y}}) \quad\Longrightarrow\quad p(\mathbf{y}) = \mathbf{C}^{-1}\mu(\hat{\mathbf{y}}).
The system requires a nonsingular \mathbf{C}. Strict diagonal dominance (each diagonal entry exceeds the sum of the other entries in its row) is one sufficient condition; then form \beta_i and reweight.
Concept shift: just keep up
Concept shift correction
When the labels are redefined, there is no clever reweighting, the old answers are simply wrong.
When concept shift is gradual, as in changing ads or news, fresh labeled data can reveal the moving target. One practical response is:
Keep the current weights and take a few update steps on fresh data, rather than retraining from scratch. Let the model track the moving target.
04
Beyond Passive Prediction
when the environment reacts to you
A taxonomy of learning problems
The bigger picture
Everything above assumed we passively predict. The environment can also react:
- Batch: train once, deploy, never update (the smart catdoor).
- Online: data arrives one point at a time; predict, then learn from the outcome.
- Bandits: online, but a finite set of actions, so stronger guarantees.
- Control & RL: the environment remembers and responds, possibly adversarially (a thermostat, a chess opponent, other cars).
A strategy that is safe in a stationary world can fail once the world adapts to it, an arbitrage trade vanishes the moment it is exploited.
Predictions become decisions
Fairness & feedback
Deploying a model is rarely just prediction, it automates decisions about people, where accuracy is seldom the right measure (the costs of different errors differ).
Predictive policing runaway loop. More patrols → more crime recorded in that area → the model predicts even more crime there → still more patrols. The data feeds back into the model, and the loop runs away.
Watch for feedback loops, cost-sensitive errors, and whether you are solving the right problem at all.
Shift in the foundation-model era
The modern picture
Benchmarks like WILDS collect real shifts (hospitals, cameras, countries, time) along an axis orthogonal to our mechanism taxonomy:
Domain generalization: test domains never seen in training. Camelyon17: a tumor classifier trained on a few hospitals’ slides must survive a new hospital’s staining quirks.
Subpopulation shift: same domains, new proportions, so what matters is worst-group accuracy. CivilComments: average toxicity accuracy conceals much larger errors on some demographic groups.
OOD detection ≠ shift correction. Detection rejects inputs the model cannot handle; correction reweights for a target that is here to stay. A deployed system needs both, and fixes that shine on one shift routinely fail on another, so measure on the shift you actually face.
Summary
Wrap-up
- Shift = train and test distributions differ; failing to notice is a top cause of deployment disasters.
- Three kinds: covariate (P(\mathbf{x}) moves, \mathbf{x}\!\to\!y), label (P(y) moves, y\!\to\!\mathbf{x}), concept (the labels themselves move).
- Correct covariate shift by reweighting with \beta_i=p(\mathbf{x}_i)/q(\mathbf{x}_i), estimated by a source-vs-target classifier (demo: target accuracy 0.502 \to 0.933, clipped 0.945); label shift by inverting the confusion matrix.
- Beware the environment: it may remember your actions and feed them back. Measure on the shift you actually face (WILDS), and keep monitoring live systems.
