x = jnp.arange(1.0, 21.0)
y = 2 * x
y = y.at[5].set(10000) # corrupt a single label
w_sq = (x * y).sum() / (x * x).sum() # closed-form squared-loss fit
w_mae = 0.0 # subgradient descent on MAE
for _ in range(2000):
w_mae -= 0.002 * (jnp.sign(w_mae * x - y) * x).mean()
print(f'true w: 2.00, squared loss: {float(w_sq):.2f}, MAE: {float(w_mae):.2f}')Predicting a number
Motivation
- Collect house sales: each has an area, an age, a price.
- Bigger houses cost more, not exactly, but on average.
- Regression draws the line and turns it into a prediction for a house nobody has seen.
Features \mathbf{x}, label y; we model E[Y \mid \mathbf{x}]. Two things are missing: a measure of how wrong we are, and a way to improve. This section supplies both.
Squared loss charges each miss by its square
The Model
Average the per-example penalties \tfrac12\bigl(\hat{y}^{(i)} - y^{(i)}\bigr)^2:
L(\mathbf{w}, b) = \frac{1}{n} \sum_{i=1}^{n} \tfrac{1}{2}\bigl(\hat{y}^{(i)} - y^{(i)}\bigr)^2.
L is convex in (\mathbf{w}, b), so every local minimum is the global one.
Large errors hurt quadratically: strong incentive to avoid big misses, and outsized sensitivity to anomalous points.
The closed form is a projection
The Model · geometry
As \mathbf{w} varies, \mathbf{X}\mathbf{w} sweeps the column space of \mathbf{X}. The best fit is the point of that subspace closest to \mathbf{y}: the orthogonal projection.
The residual \mathbf{y}-\mathbf{X}\mathbf{w}^* is what is left over, and it must be perpendicular to every feature column, exactly the normal equation \mathbf{X}^\top(\mathbf{X}\mathbf{w}^*-\mathbf{y})=\mathbf{0}.
Assume the errors are Gaussian
Where losses come from
Model each label as the linear prediction plus bell-curve noise:
y = \mathbf{w}^\top \mathbf{x} + b + \epsilon, \qquad \epsilon \sim \mathcal{N}(0, \sigma^2).
Shifting the mean slides the bell; growing the variance flattens it. Note how fast the tails die: under a Gaussian, a huge error is essentially impossible.
The recipe: match the loss to the noise model
Where losses come from · payoff
Choose a noise model, minimize its negative log-likelihood.
- Gaussian → squared error
- Laplace → absolute error
- Gaussian on \log y → log-price regression
- Poisson → \lambda - k\log\lambda (counts)
Laplace’s heavy tails expect the occasional wild point and penalize it only linearly, the robust MAE fit from the outlier demo.
The weight-decay section adds a prior to this likelihood → weight decay; the next chapter runs the recipe on categorical noise → softmax.
Linear regression is a one-neuron network
A Neural Network
Wire every input x_1,\ldots,x_d directly to a single output o_1.
The output is the same weighted sum \sum_i w_i x_i + b, so linear regression is a single-layer, fully connected network: d inputs, one computed neuron, the atom that deep networks stack.
Inspiration, not blueprint
A Neural Network
The cartoon that inspired the name: dendrites collect inputs x_i, weighted by synaptic strengths w_i; the nucleus sums them; the axon carries the result on.
Planes were inspired by birds, but aeronautics is not ornithology: today’s deep learning draws at least as much on mathematics, statistics, and computer science as on the brain.
