x = np.arange(1.0, 21.0)
y = 2 * x
y[5] = 10000 # corrupt a single label
w_sq = (x * y).sum() / (x * x).sum() # closed-form squared-loss fit
w_mae = 0.0 # subgradient descent on MAE
for _ in range(2000):
w_mae -= 0.002 * float((np.sign(w_mae * x - y) * x).mean())
print(f'true w: 2.00, squared loss: {float(w_sq):.2f}, MAE: {w_mae:.2f}')Predicting a number
Motivation
- Collect house sales: each has an area, an age, a price.
- Bigger houses cost more, not exactly, but on average.
- Regression draws the line and turns it into a prediction for a house nobody has seen.
Features \mathbf{x}, label y; we model E[Y \mid \mathbf{x}]. Two things are missing: a measure of how wrong we are, and a way to improve. This section supplies both.
Squared loss charges each miss by its square
The Model
Average the per-example penalties \tfrac12\bigl(\hat{y}^{(i)} - y^{(i)}\bigr)^2:
L(\mathbf{w}, b) = \frac{1}{n} \sum_{i=1}^{n} \tfrac{1}{2}\bigl(\hat{y}^{(i)} - y^{(i)}\bigr)^2.
L is convex in (\mathbf{w}, b), so every local minimum is the global one.
Large errors hurt quadratically: strong incentive to avoid big misses, and outsized sensitivity to anomalous points.
The closed form is a projection
The Model · geometry
As \mathbf{w} varies, \mathbf{X}\mathbf{w} sweeps the column space of \mathbf{X}. The best fit is the point of that subspace closest to \mathbf{y}: the orthogonal projection.
The residual \mathbf{y}-\mathbf{X}\mathbf{w}^* is what is left over, and it must be perpendicular to every feature column, exactly the normal equation \mathbf{X}^\top(\mathbf{X}\mathbf{w}^*-\mathbf{y})=\mathbf{0}.
Assume the errors are Gaussian
Where losses come from
Model each label as the linear prediction plus bell-curve noise:
y = \mathbf{w}^\top \mathbf{x} + b + \epsilon, \qquad \epsilon \sim \mathcal{N}(0, \sigma^2).
Shifting the mean slides the bell; growing the variance flattens it. The Gaussian density assigns very low probability to errors many standard deviations from the mean.
Match the loss to the noise model
Where losses come from · comparison
Choose a noise model, minimize its negative log-likelihood.
- Gaussian → squared error
- Laplace → absolute error
- Gaussian on \log y → log-price regression
- Poisson → \lambda - k\log\lambda (counts)
The Laplace model assigns more probability to large residuals and penalizes them linearly, producing the robust MAE fit in the outlier example.
The weight-decay section adds a prior to this likelihood → weight decay; the next chapter applies the same likelihood principle to categorical data → softmax.
Linear regression is a one-neuron network
A Neural Network
Connect every input x_1,\ldots,x_d directly to a single output o_1.
The output is the same weighted sum \sum_i w_i x_i + b, so linear regression is a single-layer, fully connected network: d inputs, one computed neuron, the atom that deep networks stack.
Inspiration, not blueprint
A Neural Network
The cartoon that inspired the name: dendrites collect inputs x_i, weighted by synaptic strengths w_i; the nucleus sums them; the axon carries the result on.
The biological analogy motivated the terminology, but modern deep learning is principally described with concepts from mathematics, statistics, and computer science.
