from d2l import torch as d2l
import torch3.3 The Base Classification Model
Classification models require a validation step that reports both loss and accuracy, together with an optimizer. We implement these shared operations in a Classifier base class extending the d2l.Module introduced in Section 2.2. A subclass then supplies the model-specific forward method and, when needed, a loss other than cross-entropy.
from d2l import tensorflow as d2l
import tensorflow as tffrom d2l import jax as d2l
from jax import numpy as jnp
import optaxfrom d2l import mxnet as d2l
from mxnet import autograd, np, npx, gluon
npx.set_np()3.3.1 The Classifier Class
The Classifier class reports loss and accuracy for each validation batch. These curves are monitoring diagnostics: an unweighted average of batch means differs from the exact dataset metric when the final batch is smaller. Final evaluation should instead sum per-example losses and correct indicators and divide by the total number of examples.
The _report_val method mirrors validation_step but accepts a precomputed y_hat, so the compiled validation graph (see Section 2.4) only needs to run the forward pass once.
NNX modules own their parameters and other state. The compiled trainer can therefore call the model directly, while NNX tracks updates such as BatchNorm running statistics automatically. The validation step returns its two metrics to the trainer, which records them outside the compiled computation.
class Classifier(d2l.Module):
"""The base class of classification models."""
def validation_step(self, batch):
Y_hat = self(*batch[:-1])
self.plot('loss', self.loss(Y_hat, batch[-1]), train=False)
self.plot('acc', self.accuracy(Y_hat, batch[-1]), train=False)
def _report_val(self, y_hat, batch):
self.plot('loss', self.loss(y_hat, batch[-1]), train=False)
self.plot('acc', self.accuracy(y_hat, batch[-1]), train=False)class Classifier(d2l.Module):
"""The base class of classification models."""
def validation_step(self, batch):
Y_hat = self(*batch[:-1])
self.plot('loss', self.loss(Y_hat, batch[-1]), train=False)
self.plot('acc', self.accuracy(Y_hat, batch[-1]), train=False)
def _report_val(self, y_hat, batch):
self.plot('loss', self.loss(y_hat, batch[-1]), train=False)
self.plot('acc', self.accuracy(y_hat, batch[-1]), train=False)class Classifier(d2l.Module):
"""The base class of classification models."""
def validation_step(self, batch):
Y_hat = self(*batch[:-1])
return self.loss(Y_hat, batch[-1]), self.accuracy(Y_hat, batch[-1])class Classifier(d2l.Module):
"""The base class of classification models."""
def validation_step(self, batch):
Y_hat = self(*batch[:-1])
self.plot('loss', self.loss(Y_hat, batch[-1]), train=False)
self.plot('acc', self.accuracy(Y_hat, batch[-1]), train=False)
def _report_val(self, y_hat, batch):
self.plot('loss', self.loss(y_hat, batch[-1]), train=False)
self.plot('acc', self.accuracy(y_hat, batch[-1]), train=False)By default we use a stochastic gradient descent optimizer operating on minibatches, just as we did in the context of linear regression. configure_optimizers is a hook: Trainer calls it once at the start of training (see Section 2.2), and it returns the optimizer object that Trainer then uses to update the parameters after each backward pass. We install it on d2l.Module rather than Classifier because regression models use the same default; a subclass can still override it (later chapters do exactly that) to switch optimizers.
@d2l.add_to_class(d2l.Module)
def configure_optimizers(self):
return torch.optim.SGD(self.parameters(), lr=self.lr)@d2l.add_to_class(d2l.Module)
def configure_optimizers(self):
return tf.keras.optimizers.SGD(float(self.lr))@d2l.add_to_class(d2l.Module)
def configure_optimizers(self):
return optax.sgd(self.lr)@d2l.add_to_class(d2l.Module)
def configure_optimizers(self):
params = self.parameters()
if isinstance(params, list):
return d2l.SGD(params, self.lr)
return gluon.Trainer(params, 'sgd', {'learning_rate': self.lr})3.3.2 Accuracy
A classifier uses the same score vector \(\mathbf{o}\in\mathbb{R}^q\) for training and evaluation in different ways (Figure 3.3.1). During training, softmax converts the scores to probabilities and cross-entropy provides a smooth loss for gradient descent. The loss distinguishes, for example, correct-class probabilities of \(0.51\) and \(0.99\) even though both produce the same hard decision. During evaluation, \(\arg\max\) converts the scores to a class \(\hat{y}\), and accuracy records whether it matches the label. Accuracy is discrete and has zero gradient almost everywhere, so it is unsuitable as the direct training objective. Whether it is an appropriate deployment metric depends on the costs of different errors and on how predictions are used.
We therefore report both quantities. The loss measures probability quality and supplies the optimization objective; accuracy measures hard decisions. A decreasing loss with unchanged accuracy can indicate increasing margins or changing calibration rather than an implementation error.
Many applications require a hard decision. Given the predicted distribution y_hat, we select its highest-probability class. An email system, for example, may estimate probabilities internally but ultimately assign each message to one folder. Accuracy is the fraction of selected classes that match the labels y.
Accuracy is computed as follows. First, if y_hat is a matrix, we assume that the second dimension stores prediction scores for each class. We use argmax to obtain the predicted class by the index for the largest entry in each row. Then we compare the predicted class with the ground truth y elementwise. Since the equality operator == is sensitive to data types, we cast the predicted classes (preds) to y’s dtype. The result is a tensor containing entries of 0 (false) and 1 (true). Averaging the 0/1 entries yields the fraction correct (or, with averaged=False, the raw 0/1 vector).
@d2l.add_to_class(Classifier)
def accuracy(self, Y_hat, Y, averaged=True):
"""Compute the fraction of correct predictions."""
Y_hat = d2l.reshape(Y_hat, (-1, Y_hat.shape[-1]))
preds = d2l.astype(d2l.argmax(Y_hat, axis=1), Y.dtype)
compare = d2l.astype(preds == d2l.reshape(Y, (-1,)), d2l.float32)
return d2l.reduce_mean(compare) if averaged else compare@d2l.add_to_class(Classifier)
def accuracy(self, Y_hat, Y, averaged=True):
"""Compute the fraction of correct predictions."""
Y_hat = d2l.reshape(Y_hat, (-1, Y_hat.shape[-1]))
preds = d2l.astype(d2l.argmax(Y_hat, axis=1), Y.dtype)
compare = d2l.astype(preds == d2l.reshape(Y, (-1,)), d2l.float32)
return d2l.reduce_mean(compare) if averaged else compare@d2l.add_to_class(Classifier)
def accuracy(self, Y_hat, Y, averaged=True):
"""Compute the fraction of correct predictions."""
Y_hat = d2l.reshape(Y_hat, (-1, Y_hat.shape[-1]))
preds = d2l.astype(d2l.argmax(Y_hat, axis=1), Y.dtype)
compare = d2l.astype(preds == d2l.reshape(Y, (-1,)), d2l.float32)
return d2l.reduce_mean(compare) if averaged else compareThe NNX version receives the precomputed scores just like the other frameworks. The surrounding validation step is compiled, so this small metric does not need its own jit decorator.
@d2l.add_to_class(Classifier)
def accuracy(self, Y_hat, Y, averaged=True):
"""Compute the fraction of correct predictions."""
Y_hat = d2l.reshape(Y_hat, (-1, Y_hat.shape[-1]))
preds = d2l.astype(d2l.argmax(Y_hat, axis=1), Y.dtype)
compare = d2l.astype(preds == d2l.reshape(Y, (-1,)), d2l.float32)
return d2l.reduce_mean(compare) if averaged else compareMXNet’s gluon.Block.collect_params only finds parameters declared through Gluon’s Parameter machinery, so it misses the bare np.ndarray attributes that the from-scratch implementations in this book use. We extend d2l.Module with a fallback get_scratch_params that walks attributes recursively, and a parameters method that returns Gluon’s params when present and the scratch params otherwise. This fallback is specific to Gluon’s parameter API.
@d2l.add_to_class(d2l.Module)
def get_scratch_params(self):
# collect_params() only finds Parameters declared via Gluon's Parameter
# API. For from-scratch models that store weights as bare np.ndarrays, we
# walk the object's attributes recursively and gather those instead.
params = []
for attr in dir(self):
a = getattr(self, attr)
if isinstance(a, np.ndarray):
params.append(a)
if isinstance(a, d2l.Module):
params.extend(a.get_scratch_params())
return params
@d2l.add_to_class(d2l.Module)
def parameters(self):
# Return the Gluon ParameterDict when the model uses Gluon layers; fall
# back to the bare-array scan for from-scratch implementations.
params = self.collect_params()
return params if isinstance(params, dict) and len(
params.keys()) else self.get_scratch_params()3.3.3 Beyond Accuracy
Accuracy treats every example, and every kind of mistake, as equally important. Once classes are imbalanced, that assumption fails in a way that can make accuracy actively misleading. Consider screening for a disease that affects 1% of the population. A classifier that ignores its input and always predicts healthy is right 99% of the time, so its accuracy is 0.99, but it finds not a single sick patient. The counts make this concrete:
n, sick = 100_000, 1_000 # 1% of the population carries the disease
tp, fp, fn = 0, 0, sick # "always predict healthy" never fires
accuracy = 1 - (fp + fn) / n
recall = tp / sick # fraction of sick patients found
accuracy, recall(0.99, 0.0)The numbers that expose the failure come from breaking the counts down by what was predicted and what was true. For a binary problem there are four cases, which we arrange in a \(2\times2\) table: true positives (TP, predicted positive, was positive), false positives (FP, predicted positive, was negative), false negatives (FN, predicted negative, was positive), and true negatives (TN). Two ratios summarize the two failure modes:
\[\textrm{precision} = \frac{\textrm{TP}}{\textrm{TP} + \textrm{FP}}, \qquad \textrm{recall} = \frac{\textrm{TP}}{\textrm{TP} + \textrm{FN}}. \tag{3.3.1}\]
Precision asks: of the examples we flagged, how many were real? Recall asks: of the real positives, how many did we find? The always-healthy classifier above has recall \(0\) (and its precision is undefined, since it never flags anyone), which tells the true story that the 99% accuracy conceals. When a single summary number is needed, the conventional compromise is the F1 score, the harmonic mean \(2\,\textrm{PR}/(\textrm{P}+\textrm{R})\) of precision and recall, which is high only when both are.
For \(q\) classes the same bookkeeping becomes a \(q \times q\) confusion matrix: entry \((i, j)\) counts the examples of true class \(j\) that the model predicted as class \(i\), so the diagonal holds the correct decisions and every off-diagonal cell isolates one specific kind of error. A confusion matrix is one useful classification diagnostic; a single accuracy number is its normalized trace (the fraction on the diagonal). We will meet it twice more in this chapter: in Section 3.4 we compute one for our trained Fashion-MNIST classifier to see which classes it confuses, and in Section 3.7 the same matrix becomes the key computational object for correcting label shift.
3.3.4 Summary
The Classifier class adds two things to d2l.Module: an overridden validation_step that logs both the loss and the accuracy, and a default configure_optimizers that returns a minibatch SGD optimizer. Because of this, every classification model in the rest of the book can subclass Classifier and supply only its forward pass (and a custom loss, where the default cross-entropy will not do), inheriting the whole training and evaluation loop. Accuracy itself is the fraction of examples whose predicted class matches the true label, where the predicted class is the \(\arg\max\) of the score vector. It is a discrete metric and so cannot serve as a training objective, but it is commonly reported in benchmarks and should be watched alongside the loss when all mistakes have comparable costs. On imbalanced or cost-sensitive problems, accuracy alone can badly mislead; precision, recall, and the confusion matrix break the errors down by kind, and the confusion matrix in particular will return twice more in this chapter.
3.3.5 Exercises
- Denote by \(L_\textrm{v}\) the validation loss, and let \(L_\textrm{v}^\textrm{q}\) be its unweighted batch-mean estimate computed by the loss averaging in this section. Lastly, denote by \(l_\textrm{v}^\textrm{b}\) the loss on the last minibatch. Express \(L_\textrm{v}\) in terms of \(L_\textrm{v}^\textrm{q}\), \(l_\textrm{v}^\textrm{b}\), and the sample and minibatch sizes.
- Show that the unweighted batch-mean estimate \(L_\textrm{v}^\textrm{q}\) is unbiased. That is, show that \(E[L_\textrm{v}] = E[L_\textrm{v}^\textrm{q}]\). Why would you still want to use \(L_\textrm{v}\) instead?
- Given a multiclass classification loss, denoting by \(l(y,y')\) the penalty of estimating \(y'\) when we see \(y\) and given a probability \(p(y \mid x)\), formulate the rule for an optimal selection of \(y'\). Hint: express the expected loss, using \(l\) and \(p(y \mid x)\).
- Suppose two classifiers \(A\) and \(B\) both achieve 90% accuracy on a ten-class test set, but on the examples they get right, \(A\) assigns probability \(0.91\) on average to the correct class while \(B\) assigns only \(0.51\). (i) Compute the average cross-entropy loss each incurs on those examples. (ii) Explain why these averages are insufficient to decide which classifier is safer: what must we know about their probabilities on incorrect predictions, calibration by subgroup, and the costs of their errors? (iii) Construct a simple monotone rescaling of the scores (a temperature) that sharpens \(B\)’s probabilities without changing any of its \(\arg\max\) decisions, and argue why its accuracy is therefore unchanged.
- Generalize
accuracyto top-\(k\) accuracy, which counts a prediction as correct when the true class is among the \(k\) highest-scoring classes. (i) Modify the four-line implementation to take akargument (hint: replace the singleargmaxwith the indices of the \(k\) largest scores). (ii) On a \(q\)-class problem, what is top-\(q\) accuracy always equal to, and why? (iii) Why is top-5 accuracy a standard companion to top-1 on benchmarks with many fine-grained classes? - In the disease-screening example, suppose we care much more about missing a sick patient than about a false alarm. (i) Modify the cross-entropy loss so that each class \(j\) carries a weight \(w_j\), multiplying the loss of every example of true class \(j\); this is a one-line change to
loss. (ii) Show that this weighted loss is exactly the weighted empirical risk minimization objective Equation 3.7.2 of Section 3.7 with weights \(\beta_i = w_{y_i}\), and explain why upweighting the rare class shifts the learned decision boundary toward higher recall. (iii) Give a second, data-side intervention with the same effect (hint: sampling). - A binary classifier’s score can be thresholded at any \(\tau \in (0, 1)\), not just \(\frac{1}{2}\): predict positive whenever \(\hat{y}_1 > \tau\). Sweep \(\tau\) from \(1\) down to \(0\) for a classifier of your choice (for instance a two-class subset of Fashion-MNIST, such as sneaker versus sandal, once you have trained the model of Section 3.4). (i) Compute the true-positive rate (recall) and the false-positive rate at each \(\tau\) and plot one against the other; this curve is called the receiver operating characteristic (ROC). (ii) What do the endpoints \(\tau=1\) and \(\tau=0\) correspond to? (iii) Argue that a classifier whose scores are a random permutation of the data traces the diagonal in expectation, and that the area under the curve is therefore a threshold-free summary of ranking quality.