3  Linear Classification in Neural Networks

Classification predicts a category rather than a numerical value. The training procedure remains the same as for regression: load data, compute model outputs and a loss, differentiate the loss, and update the parameters. The target representation, output layer, and loss must now describe a distribution over discrete classes.

This chapter derives softmax regression and cross-entropy, implements the model on Fashion-MNIST, and introduces common classification metrics. It then examines generalization and the consequences of distribution shift.

Resources and Further Reading

The classical machine-learning texts listed in the previous chapter Linear Regression in Neural Networks also cover classification, so we do not repeat them here. The references below focus on softmax regression, cross-entropy, calibration, and distribution shift. All are freely accessible online except where noted.

Tutorials and notes

• CS229 Machine Learning lecture notes — Andrew Ng & Tengyu Ma (Stanford) — free PDF; derives logistic and softmax (multinomial) regression and the cross-entropy loss from the exponential-family / GLM viewpoint.
• Information Theory — this book’s Mathematics for Deep Learning appendix — entropy, cross-entropy, and KL divergence developed in full, the information-theoretic background behind the softmax cross-entropy loss.

Foundational papers

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms — Xiao, Rasul & Vollgraf (2017) — the dataset used throughout this chapter’s image-classification examples.
• On Calibration of Modern Neural Networks — Guo, Pleiss, Sun & Weinberger (2017) — shows that a classifier’s softmax confidences are often miscalibrated and that temperature scaling can improve calibration; useful context for interpreting softmax outputs as probabilities.

Books

• Dataset Shift in Machine Learning — Quiñonero-Candela, Sugiyama, Schwaighofer & Lawrence (eds.), MIT Press — the standard reference on the distribution-shift problems (covariate, label, and concept shift) discussed in the final section (an open-access edition is available from MIT Press).
• Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation — Sugiyama & Kawanabe, MIT Press — a focused, in-depth treatment of covariate shift and importance-weighted adaptation (print edition; not open access).