X = onp.array([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
W1 = onp.array([[1., 1.], [1., 1.]])
b1 = onp.array([0., -1.])
w2 = onp.array([[1.], [-2.]])
H = onp.maximum(X @ W1 + b1, 0)
O = (H @ w2).squeeze()
onp.column_stack([X, (O > 0.5).astype(float)])A linear model draws one straight boundary
Motivation
Softmax regression is a single affine map: monotonic, line-shaped decisions.
- Body temperature → risk rises on both sides of 37°C.
- Cat vs dog: pixel (13,17) means nothing without its neighbours.
- XOR: a line provably cannot separate it.
Learn the features and retain a linear predictor at the output. A two-unit net computes XOR exactly, and depth multiplies what width merely adds.
A one-dimensional hinge construction
Expressive power
For a one-dimensional construction, each ReLU unit contributes a hinge a_k\operatorname{ReLU}(x - t_k): with D units the output is piecewise linear with at most D+1 pieces. Additional joints can refine this polyline approximation.
Three hinges (left) sum to a 4-piece polyline that tracks the smooth target (right); the shaded band is the error.
ReLU: the modern default
Activations
\operatorname{ReLU}(x) = \max(0, x).
x = torch.arange(-8.0, 8.0, 0.1, requires_grad=True)
y = torch.relu(x)
d2l.plot(x.detach(), y.detach(), 'x', 'relu(x)', figsize=(5, 2.5))
The activation retains positive inputs and maps negative inputs to zero:
- No right-side saturation: gradient is exactly 1 for x>0.
- Cheap: a single comparison, no exponential.
- Sparse: about half the units output 0.
ReLU’s gradient: an on/off step
Activations
The derivative is a step: 0 on the left, 1 on the right:
\operatorname{ReLU}'(x) = \mathbb{1}[x > 0].
y.backward(torch.ones_like(x), retain_graph=True)
d2l.plot(x.detach(), x.grad, 'x', 'grad of relu', figsize=(5, 2.5))
Dead ReLU: a unit pushed negative for every example receives zero gradient. LeakyReLU / PReLU, \max(0,x)+\alpha\min(0,x), retain a nonzero negative-side slope.
Sigmoid: squashing into (0, 1)
Activations
\operatorname{sigmoid}(x) = \frac{1}{1 + e^{-x}}.
y = torch.sigmoid(x)
d2l.plot(x.detach(), y.detach(), 'x', 'sigmoid(x)', figsize=(5, 2.5))
This smooth approximation to a threshold is now used mainly at network outputs and in gating mechanisms:
- Binary output, read as a probability.
- Gates in LSTM/GRU and attention.
Sigmoid saturation attenuates gradients
Activation saturation
\operatorname{sigmoid}'(x) = \operatorname{sigmoid}(x)\,(1 - \operatorname{sigmoid}(x)).
# Clear out previous gradients
x.grad.zero_()
y.backward(torch.ones_like(x),retain_graph=True)
d2l.plot(x.detach(), x.grad, 'x', 'grad of sigmoid', figsize=(5, 2.5))
The gradient peaks at just 0.25 and vanishes past |x|\gtrsim 5. Even at its best, ten stacked layers attenuate the backward signal by 0.25^{10} \approx 10^{-6}: the vanishing-gradient problem discussed in the numerical-stability section.
Tanh: sigmoid’s zero-centered cousin
Activations
\tanh(x) = \frac{1 - e^{-2x}}{1 + e^{-2x}} = 2\,\operatorname{sigmoid}(2x) - 1.
y = torch.tanh(x)
d2l.plot(x.detach(), y.detach(), 'x', 'tanh(x)', figsize=(5, 2.5))
Same S-shape, but range (-1,1) and zero-centered, which mildly eases optimization. The default inside RNN cells, where bounded activations help.
Still saturates at both tails, so its gradient vanishes just like sigmoid’s.