2  Linear Regression in Neural Networks

Before we worry about making our neural networks deep, it will be helpful to implement some shallow ones, for which the inputs connect directly to the outputs. This will prove important for a few reasons. First, rather than getting distracted by complicated architectures, we can focus on the basics of neural network training, including parametrizing the output layer, handling data, specifying a loss function, and training the model. Second, this class of shallow networks happens to comprise the set of linear models, which subsumes many classical methods of statistical prediction, including linear and softmax regression. Understanding these classical tools matters because they are widely used in many contexts and we will often need to use them as baselines when justifying the use of fancier architectures. This chapter will focus narrowly on linear regression and the next one will extend our modeling repertoire by developing linear neural networks for classification.

Resources and Further Reading

The references below develop the linear-model family this chapter introduces: the statistical foundations shared with classification (these classical texts are listed here once, and the next chapter builds on them), then material specific to regression, least squares, generalization, and the weight decay we use as \(L_2\) regularization. All are freely accessible online except where noted.

Books

Courses and video lectures

Foundational papers