{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=classification-md-1 fw=all -->\n",
    "\n",
    "# The Base Classification Model\n",
    "\n",
    "Every classification model in this book, from the linear softmax regressor we build next to the deep convolutional networks of later chapters, shares two common needs: a *validation step* that reports both loss and accuracy, and a default optimizer. Rather than re-implementing these in every subclass, we collect them once in a `Classifier` base class that extends the `d2l.Module` scaffold introduced in that section. The payoff is the same one that motivated `Module` itself: a new classifier supplies only what is genuinely model-specific (its `forward` pass, and a `loss` if it is not plain cross-entropy), and inherits the training and evaluation machinery for free."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "classification-the-base-classification-model"
   },
   "source": [
    "from d2l import mxnet as d2l\n",
    "from mxnet import autograd, np, npx, gluon\n",
    "npx.set_np()"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=classification-md-2 fw=all -->\n",
    "\n",
    "## The `Classifier` Class\n",
    "<!-- d2l:prose id=classification-md-3 fw=pytorch, mxnet, tensorflow -->\n",
    "\n",
    "We define the `Classifier` class below. In the `validation_step` we report both the loss value and the classification accuracy on a validation batch. The plotting machinery records one point per batch, and since all validation batches (except possibly the last) contain the same number of examples, the average of the plotted per-batch values equals the loss and accuracy over the whole validation set. That average is off only when the final batch is smaller, a minor discrepancy we ignore to keep the code simple."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "classification-the-classifier-class-1"
   },
   "source": [
    "class Classifier(d2l.Module):\n",
    "    \"\"\"The base class of classification models.\"\"\"\n",
    "    def validation_step(self, batch):\n",
    "        Y_hat = self(*batch[:-1])\n",
    "        self.plot('loss', self.loss(Y_hat, batch[-1]), train=False)\n",
    "        self.plot('acc', self.accuracy(Y_hat, batch[-1]), train=False)\n",
    "\n",
    "    def _report_val(self, y_hat, batch):\n",
    "        self.plot('loss', self.loss(y_hat, batch[-1]), train=False)\n",
    "        self.plot('acc', self.accuracy(y_hat, batch[-1]), train=False)"
   ],
   "execution_count": 2,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=classification-md-4 fw=all -->\n",
    "\n",
    "By default we use a stochastic gradient descent optimizer operating on minibatches, just as we did in the context of linear regression. `configure_optimizers` is a hook: `Trainer` calls it once at the start of training (see that section), and it returns the optimizer object that `Trainer` then uses to update the parameters after each backward pass. We install it on `d2l.Module` rather than `Classifier` because regression models use the same default; a subclass can still override it (later chapters do exactly that) to switch optimizers."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "classification-the-classifier-class-2"
   },
   "source": [
    "@d2l.add_to_class(d2l.Module)\n",
    "def configure_optimizers(self):\n",
    "    params = self.parameters()\n",
    "    if isinstance(params, list):\n",
    "        return d2l.SGD(params, self.lr)\n",
    "    return gluon.Trainer(params, 'sgd', {'learning_rate': self.lr})"
   ],
   "execution_count": 3,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=classification-md-5 fw=all -->\n",
    "\n",
    "## Accuracy\n",
    "\n",
    "Before we implement the accuracy metric, consider why a classifier needs *two* numbers at all. A single forward pass produces a vector of scores $\\mathbf{o}\\in\\mathbb{R}^q$, one per class, and from there the picture forks into two branches that read the *same* scores for very different purposes (the figure). On the training branch we turn the scores into probabilities with the softmax and read off the cross-entropy loss. This loss is a smooth function of the parameters, so gradient descent can minimize it; and it keeps rewarding the model for putting more probability on the correct class even after the decision is already right, nudging a confidence of $0.51$ toward $0.99$. On the evaluation branch we take the $\\arg\\max$ of the scores to a single hard decision $\\hat{y}$, compare it with the label, and count the hit. This is the accuracy: the fraction of correct decisions. It is a common benchmark metric but a *discrete* quantity whose gradient is zero almost everywhere, since a tiny change to the scores almost never flips which entry is largest. Whether accuracy is the right deployment metric depends on the costs of different errors and on how the scores will be used.\n",
    "\n",
    "So we report both, and for complementary reasons. Two models can reach identical accuracy while one is confidently right and the other barely so, and only the loss can tell them apart, which is why it, not accuracy, is what we optimize. Accuracy in turn measures the hard-decision quality that the loss only stands in for. When the two disagree (accuracy flat while the loss still drops, say) that is diagnostic information about optimization and calibration (how well the predicted probabilities match empirical frequencies), not a bug.\n",
    "\n",
    "![From model scores to a training loss and an evaluation accuracy. One forward pass produces the logits $\\mathbf{o}$; the top branch softmaxes them into probabilities $\\hat{\\mathbf{y}}$ and reads off the differentiable cross-entropy loss that drives gradient descent, while the bottom branch takes the $\\arg\\max$ to a hard decision $\\hat{y}$, compares it with the label $y$, and counts it for accuracy. The numbers shown are the exact softmax and cross-entropy of the logits $(1.0, 2.2, 0.3)$ for true class $y=1$.](../img/mdl-clf-loss-accuracy.svg)\n",
    "\n",
    "Taking the hard decision is what many applications require. Given the predicted probability distribution `y_hat`, we choose the class with the highest predicted probability whenever we must commit to one. Gmail, for instance, must file an email under \"Primary\", \"Social\", \"Updates\", \"Forums\", or \"Spam\": it might estimate probabilities internally, but at the end of the day it has to pick a single folder. A prediction that matches the label class `y` is correct, and accuracy is simply the fraction of predictions that are.\n",
    "\n",
    "Accuracy is computed as follows.\n",
    "First, if `y_hat` is a matrix,\n",
    "we assume that the second dimension stores prediction scores for each class.\n",
    "We use `argmax` to obtain the predicted class by the index for the largest entry in each row.\n",
    "Then we compare the predicted class with the ground truth `y` elementwise.\n",
    "Since the equality operator `==` is sensitive to data types,\n",
    "we cast the predicted classes (`preds`) to `y`'s dtype.\n",
    "The result is a tensor containing entries of 0 (false) and 1 (true).\n",
    "Averaging the 0/1 entries yields the fraction correct (or, with `averaged=False`, the raw 0/1 vector)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "classification-accuracy-1"
   },
   "source": [
    "@d2l.add_to_class(Classifier)\n",
    "def accuracy(self, Y_hat, Y, averaged=True):\n",
    "    \"\"\"Compute the fraction of correct predictions.\"\"\"\n",
    "    Y_hat = d2l.reshape(Y_hat, (-1, Y_hat.shape[-1]))\n",
    "    preds = d2l.astype(d2l.argmax(Y_hat, axis=1), Y.dtype)\n",
    "    compare = d2l.astype(preds == d2l.reshape(Y, (-1,)), d2l.float32)\n",
    "    return d2l.reduce_mean(compare) if averaged else compare"
   ],
   "execution_count": 4,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=classification-md-6 fw=mxnet -->\n",
    "\n",
    "MXNet's `gluon.Block.collect_params` only finds parameters declared\n",
    "through Gluon's `Parameter` machinery, so it misses the bare `np.ndarray`\n",
    "attributes that the from-scratch implementations in this book use.\n",
    "We extend `d2l.Module` with a fallback `get_scratch_params` that\n",
    "walks attributes recursively, and a `parameters` method that returns\n",
    "Gluon's params when present and the scratch params otherwise. This\n",
    "fallback is specific to Gluon's parameter API."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "classification-accuracy-2"
   },
   "source": [
    "\n",
    "@d2l.add_to_class(d2l.Module)\n",
    "def get_scratch_params(self):\n",
    "    # collect_params() only finds Parameters declared via Gluon's Parameter\n",
    "    # API. For from-scratch models that store weights as bare np.ndarrays, we\n",
    "    # walk the object's attributes recursively and gather those instead.\n",
    "    params = []\n",
    "    for attr in dir(self):\n",
    "        a = getattr(self, attr)\n",
    "        if isinstance(a, np.ndarray):\n",
    "            params.append(a)\n",
    "        if isinstance(a, d2l.Module):\n",
    "            params.extend(a.get_scratch_params())\n",
    "    return params\n",
    "\n",
    "@d2l.add_to_class(d2l.Module)\n",
    "def parameters(self):\n",
    "    # Return the Gluon ParameterDict when the model uses Gluon layers; fall\n",
    "    # back to the bare-array scan for from-scratch implementations.\n",
    "    params = self.collect_params()\n",
    "    return params if isinstance(params, dict) and len(\n",
    "        params.keys()) else self.get_scratch_params()"
   ],
   "execution_count": 5,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=classification-md-7 fw=all -->\n",
    "\n",
    "## Beyond Accuracy\n",
    "\n",
    "Accuracy treats every example, and every kind of mistake, as equally important. Once classes are *imbalanced*, that assumption fails in a way that can make accuracy actively misleading. Consider screening for a disease that affects 1% of the population. A \"classifier\" that ignores its input and always predicts *healthy* is right 99% of the time, so its accuracy is 0.99, and it is also perfectly useless: it finds not a single sick patient. The counts make this concrete:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "classification-beyond-accuracy"
   },
   "source": [
    "n, sick = 100_000, 1_000            # 1% of the population carries the disease\n",
    "tp, fp, fn = 0, 0, sick             # \"always predict healthy\" never fires\n",
    "accuracy = 1 - (fp + fn) / n\n",
    "recall = tp / sick                  # fraction of sick patients found\n",
    "accuracy, recall"
   ],
   "execution_count": 6,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "(0.99, 0.0)"
     },
     "metadata": {},
     "execution_count": 6
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=classification-md-8 fw=all -->\n",
    "\n",
    "The numbers that expose the failure come from breaking the counts down by what was predicted and what was true. For a binary problem there are four cases, which we arrange in a $2\\times2$ table: *true positives* (TP, predicted positive, was positive), *false positives* (FP, predicted positive, was negative), *false negatives* (FN, predicted negative, was positive), and *true negatives* (TN). Two ratios summarize the two failure modes:\n",
    "\n",
    "$$\\textrm{precision} = \\frac{\\textrm{TP}}{\\textrm{TP} + \\textrm{FP}}, \\qquad \\textrm{recall} = \\frac{\\textrm{TP}}{\\textrm{TP} + \\textrm{FN}}.$$\n",
    "\n",
    "Precision asks: of the examples we *flagged*, how many were real? Recall asks: of the real positives, how many did we *find*? The always-healthy classifier above has recall $0$ (and its precision is undefined, since it never flags anyone), which tells the true story that the 99% accuracy conceals. When a single summary number is needed, the *F1 score*, the harmonic mean $2\\,\\textrm{PR}/(\\textrm{P}+\\textrm{R})$ of precision and recall, is the conventional compromise, high only when both are.\n",
    "\n",
    "For $q$ classes the same bookkeeping becomes a $q \\times q$ *confusion matrix*: entry $(i, j)$ counts the examples of true class $j$ that the model predicted as class $i$, so the diagonal holds the correct decisions and every off-diagonal cell isolates one specific kind of error. The confusion matrix is the classification diagnostic; a single accuracy number is just the normalized trace (the fraction on the diagonal). We will meet it twice more in this chapter: in that section we compute one for our trained Fashion-MNIST classifier to see *which* classes it confuses, and in that section the very same matrix becomes the key computational object for correcting label shift.\n",
    "\n",
    "## Summary\n",
    "\n",
    "The `Classifier` class adds two things to `d2l.Module`: an overridden `validation_step` that logs *both* the loss and the accuracy, and a default `configure_optimizers` that returns a minibatch SGD optimizer. Because of this, every classification model in the rest of the book can subclass `Classifier` and supply only its `forward` pass (and a custom `loss`, where the default cross-entropy will not do), inheriting the whole training and evaluation loop. Accuracy itself is the fraction of examples whose predicted class, the $\\arg\\max$ of the score vector, matches the true label. It is a discrete metric and so cannot serve as a training objective, but it is commonly reported in benchmarks and should be watched alongside the loss when all mistakes have comparable costs. On imbalanced or cost-sensitive problems, accuracy alone can badly mislead; precision, recall, and the confusion matrix break the errors down by kind, and the confusion matrix in particular will return twice more in this chapter.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Denote by $L_\\textrm{v}$ the validation loss, and let $L_\\textrm{v}^\\textrm{q}$ be its quick and dirty estimate computed by the loss function averaging in this section. Lastly, denote by $l_\\textrm{v}^\\textrm{b}$ the loss on the last minibatch. Express $L_\\textrm{v}$ in terms of $L_\\textrm{v}^\\textrm{q}$, $l_\\textrm{v}^\\textrm{b}$, and the sample and minibatch sizes.\n",
    "1. Show that the quick and dirty estimate $L_\\textrm{v}^\\textrm{q}$ is unbiased. That is, show that $E[L_\\textrm{v}] = E[L_\\textrm{v}^\\textrm{q}]$. Why would you still want to use $L_\\textrm{v}$ instead?\n",
    "1. Given a multiclass classification loss, denoting by $l(y,y')$ the penalty of estimating $y'$ when we see $y$ and given a probability $p(y \\mid x)$, formulate the rule for an optimal selection of $y'$. Hint: express the expected loss, using $l$ and $p(y \\mid x)$.\n",
    "1. Suppose two classifiers $A$ and $B$ both achieve 90% accuracy on a ten-class test set, but on the examples they get right, $A$ assigns probability $0.91$ on average to the correct class while $B$ assigns only $0.51$. (i) Compute the average cross-entropy loss each incurs on those examples. (ii) Explain why these averages are insufficient to decide which classifier is safer: what must we know about their probabilities on incorrect predictions, calibration by subgroup, and the costs of their errors? (iii) Construct a simple monotone rescaling of the scores (a temperature) that sharpens $B$'s probabilities without changing any of its $\\arg\\max$ decisions, and argue why its accuracy is therefore unchanged.\n",
    "1. Generalize `accuracy` to *top-$k$ accuracy*, which counts a prediction as correct when the true class is among the $k$ highest-scoring classes. (i) Modify the four-line implementation to take a `k` argument (hint: replace the single `argmax` with the indices of the $k$ largest scores). (ii) On a $q$-class problem, what is top-$q$ accuracy always equal to, and why? (iii) Why is top-5 accuracy a standard companion to top-1 on benchmarks with many fine-grained classes?\n",
    "1. In the disease-screening example, suppose we care much more about missing a sick patient than about a false alarm. (i) Modify the cross-entropy loss so that each class $j$ carries a weight $w_j$, multiplying the loss of every example of true class $j$; this is a one-line change to `loss`. (ii) Show that this weighted loss is exactly the weighted empirical risk minimization objective the equation of that section with weights $\\beta_i = w_{y_i}$, and explain why upweighting the rare class shifts the learned decision boundary toward higher recall. (iii) Give a second, data-side intervention with the same effect (hint: sampling).\n",
    "1. A binary classifier's score can be thresholded at any $\\tau \\in (0, 1)$, not just $\\frac{1}{2}$: predict positive whenever $\\hat{y}_1 > \\tau$. Sweep $\\tau$ from $1$ down to $0$ for a classifier of your choice (for instance a two-class subset of Fashion-MNIST, such as sneaker versus sandal, once you have trained the model of that section). (i) Compute the true-positive rate (recall) and the false-positive rate at each $\\tau$ and plot one against the other; this curve is called the *receiver operating characteristic* (ROC). (ii) What do the endpoints $\\tau=1$ and $\\tau=0$ correspond to? (iii) Argue that a classifier whose scores are a random permutation of the data traces the diagonal in expectation, and that the area under the curve is therefore a threshold-free summary of ranking quality."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=classification-md-9 fw=mxnet -->\n",
    "\n",
    "[Discussions](https://d2l.discourse.group/t/6808)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-mxnet",
   "display_name": "d2l (MXNet)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
