2  Linear Regression in Neural Networks

Linear models provide the simplest setting in which to study the complete learning procedure: represent a model, define a loss, organize data, optimize parameters, and evaluate predictions on unseen examples. They are also useful statistical methods in their own right and provide standard baselines for more complex models.

This chapter develops linear regression as a one-layer neural network. We derive the model and its loss, implement its components from first principles, and then replace those components with framework abstractions. The final sections introduce generalization and weight decay. The next chapter applies the same framework to classification.

Resources and Further Reading

The references below cover the statistical foundations shared with classification, followed by material specific to regression, least squares, generalization, and weight decay as \(L_2\) regularization. The next chapter builds on the shared references rather than listing them again. All are freely accessible online except where noted.

Books

• An Introduction to Statistical Learning (ISL) — James, Witten, Hastie & Tibshirani — free PDF (R and Python editions); the gentlest rigorous treatment of regression, classification, and regularization, with worked labs.
• The Elements of Statistical Learning (ESL) — Hastie, Tibshirani & Friedman — free PDF; the graduate companion to ISL, with deep coverage of shrinkage, the bias–variance tradeoff, and model selection.
• Pattern Recognition and Machine Learning — Christopher Bishop — free PDF from Microsoft Research; a probabilistic, Bayesian-flavoured account of linear models and regularization.
• Probabilistic Machine Learning: An Introduction — Kevin Murphy — free draft PDF; a modern, unifying probabilistic perspective spanning regression through deep learning.
• Understanding Deep Learning (UDL) — Simon Prince — free PDF; a figure-rich modern treatment whose loss-function chapter generalizes the “match the loss to the noise model” recipe of this chapter into a full menu.
• Deep Learning: Foundations and Concepts — Bishop & Bishop (2024) — free to read online; the successor to PRML, connecting the probabilistic view of linear regression and regularization directly to deep networks.
• Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares (VMLS) — Boyd & Vandenberghe — free PDF, slides, and code; an applications-first text built around least squares, the engine of linear regression.

Courses and video lectures

• Supervised Machine Learning: Regression and Classification — Andrew Ng (Stanford / DeepLearning.AI) — free to audit; the foundational course covering both settings of this part of the book, starting from linear and logistic regression.
• Statistical Learning — Hastie & Tibshirani (Stanford Online, edX) — free to audit; lecture series following ISL, including regression, ridge, and the LASSO.

Foundational papers

• Ridge Regression: Biased Estimation for Nonorthogonal Problems — Hoerl & Kennard (1970), Technometrics — the origin of \(L_2\) regularization, exactly the weight decay introduced in this chapter (paywalled, noted).
• Regression Shrinkage and Selection via the Lasso — Tibshirani (1996), JRSS-B — introduces the \(L_1\)-penalized counterpart, contrasting sparse selection with ridge’s shrinkage (paywalled, noted).