X = onp.array([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
W1 = onp.array([[1., 1.], [1., 1.]])
b1 = onp.array([0., -1.])
w2 = onp.array([[1.], [-2.]])
H = onp.maximum(X @ W1 + b1, 0)
O = (H @ w2).squeeze()
onp.column_stack([X, (O > 0.5).astype(float)])A linear model draws one straight boundary
Motivation
Softmax regression is a single affine map: monotonic, line-shaped decisions.
- Body temperature → risk rises on both sides of 37°C.
- Cat vs dog: pixel (13,17) means nothing without its neighbours.
- XOR: a line provably cannot separate it.
The fix: learn the features, keep the linear predictor on top. A two-unit net computes XOR exactly, and depth multiplies what width merely adds.
XOR: impossible for a line, easy after a fold
Why nonlinearity matters
Label each corner of the unit square by whether its coordinates differ. The two classes sit on opposite diagonals (left), so no straight line works.
One hidden layer \mathbf{h} = \operatorname{ReLU}(\mathbf{x}\mathbf{W}^{(1)} + \mathbf{b}^{(1)}) then folds the two label-1 corners onto the same point (right), and now a single line separates them.
Why it is plausible: one hinge at a time
Expressive power
Each ReLU unit contributes a hinge a_k\operatorname{ReLU}(x - t_k): with D units the output is piecewise linear with at most D+1 pieces. Approximating a curve is then just fitting a polyline: more joints, less error.
Three hinges (left) sum to a 4-piece polyline that tracks the smooth target (right); the shaded band is the error.
ReLU: the modern default
Activations
\operatorname{ReLU}(x) = \max(0, x).
x = torch.arange(-8.0, 8.0, 0.1, requires_grad=True)
y = torch.relu(x)
d2l.plot(x.detach(), y.detach(), 'x', 'relu(x)', figsize=(5, 2.5))
Keep the positive part, zero the rest. Why it won:
- No right-side saturation: gradient is exactly 1 for x>0.
- Cheap: a single comparison, no exponential.
- Sparse: about half the units output 0.
ReLU’s gradient: an on/off step
Activations
The derivative is a step: 0 on the left, 1 on the right:
\operatorname{ReLU}'(x) = \mathbb{1}[x > 0].
y.backward(torch.ones_like(x), retain_graph=True)
d2l.plot(x.detach(), x.grad, 'x', 'grad of relu', figsize=(5, 2.5))
Dead ReLU: a unit pushed negative for every example gets zero gradient forever. LeakyReLU / PReLU, \max(0,x)+\alpha\min(0,x), leak a little signal to keep it alive.
Sigmoid: squashing into (0, 1)
Activations
\operatorname{sigmoid}(x) = \frac{1}{1 + e^{-x}}.
y = torch.sigmoid(x)
d2l.plot(x.detach(), y.detach(), 'x', 'sigmoid(x)', figsize=(5, 2.5))
A smooth, differentiable threshold, and the original neuron activation. Today it lives mostly at the edges of a net:
- Binary output, read as a probability.
- Gates in LSTM/GRU and attention.
Why sigmoid stalls deep networks
Activations · the catch
\operatorname{sigmoid}'(x) = \operatorname{sigmoid}(x)\,(1 - \operatorname{sigmoid}(x)).
# Clear out previous gradients
x.grad.zero_()
y.backward(torch.ones_like(x),retain_graph=True)
d2l.plot(x.detach(), x.grad, 'x', 'grad of sigmoid', figsize=(5, 2.5))
The gradient peaks at just 0.25 and vanishes past |x|\gtrsim 5. Even at its best, ten stacked layers attenuate the backward signal by 0.25^{10} \approx 10^{-6}: the vanishing-gradient problem ReLU fixed (the full story in the numerical-stability section).
Tanh: sigmoid’s zero-centered cousin
Activations
\tanh(x) = \frac{1 - e^{-2x}}{1 + e^{-2x}} = 2\,\operatorname{sigmoid}(2x) - 1.
y = torch.tanh(x)
d2l.plot(x.detach(), y.detach(), 'x', 'tanh(x)', figsize=(5, 2.5))
Same S-shape, but range (-1,1) and zero-centered, which mildly eases optimization. The default inside RNN cells, where bounded activations help.
Still saturates at both tails, so its gradient vanishes just like sigmoid’s.