6.2  Convolutions for Images

The preceding section derived convolutional layers from locality and translation equivariance. We now define the two-dimensional cross-correlation operation, implement it directly, and show how its kernels can be learned from image data.

PyTorch

from d2l import torch as d2l
import torch
from torch import nn
from torch.nn import functional as F

TensorFlow

from d2l import tensorflow as d2l
import tensorflow as tf

JAX

from d2l import jax as d2l
from flax import nnx
import jax
from jax import numpy as jnp

MXNet

from d2l import mxnet as d2l
from mxnet import autograd, gluon, np, npx
from mxnet.gluon import nn
npx.set_np()

6.2.1 The Cross-Correlation Operation

Strictly speaking, the operation used by a convolutional layer is cross-correlation rather than mathematical convolution. As described in Section 6.1, such a layer combines an input tensor and a kernel tensor into an output tensor through a cross-correlation operation.

We first omit channels and consider two-dimensional inputs and hidden representations. In Figure 6.2.1, the input is a two-dimensional tensor with a height of 3 and width of 3. We mark the shape of the tensor as \(3 \times 3\) or (\(3\), \(3\)). The height and width of the kernel are both 2. The shape of the kernel window (or convolution window) is given by the height and width of the kernel (here it is \(2 \times 2\)).

Figure 6.2.1: Two-dimensional cross-correlation operation. The shaded portions are the first output element as well as the input and kernel tensor elements used for the output computation: \(0\times0+1\times1+3\times2+4\times3=19\).

In the two-dimensional cross-correlation operation, we begin with the convolution window positioned at the upper-left corner of the input tensor and slide it across the input tensor, both from left to right and top to bottom. When the convolution window slides to a certain position, the input subtensor contained in that window and the kernel tensor are multiplied elementwise and the resulting tensor is summed up yielding a single scalar value. This result gives the value of the output tensor at the corresponding location. Here, the output tensor has a height of 2 and width of 2 and the four elements are derived from the two-dimensional cross-correlation operation:

\[ \begin{aligned} 0\times0+1\times1+3\times2+4\times3 &= 19,\\ 1\times0+2\times1+4\times2+5\times3 &= 25,\\ 3\times0+4\times1+6\times2+7\times3 &= 37,\\ 4\times0+5\times1+7\times2+8\times3 &= 43. \end{aligned} \]

Along each axis, the output is smaller than the input. Because the kernel has width and height greater than \(1\), we can only properly compute the cross-correlation for locations where the kernel fits wholly within the image. The output size is therefore the input size \(n_\textrm{h} \times n_\textrm{w}\) minus the size of the convolution kernel \(k_\textrm{h} \times k_\textrm{w}\), that is,

\[(n_\textrm{h}-k_\textrm{h}+1) \times (n_\textrm{w}-k_\textrm{w}+1). \tag{6.2.1}\]

The kernel must fit within the image at every evaluated position. The next section shows how to keep the size unchanged by padding the image with zeros around its boundary so that there is enough space to shift the kernel. Next, we implement this process in the corr2d function, which accepts an input tensor X and a kernel tensor K and returns an output tensor Y.

PyTorch

def corr2d(X, K):
    """Compute 2D cross-correlation."""
    h, w = K.shape
    Y = d2l.zeros((X.shape[0] - h + 1, X.shape[1] - w + 1))
    for i in range(Y.shape[0]):
        for j in range(Y.shape[1]):
            Y[i, j] = d2l.reduce_sum((X[i: i + h, j: j + w] * K))
    return Y

TensorFlow

def corr2d(X, K):
    """Compute 2D cross-correlation."""
    h, w = K.shape
    Y = tf.Variable(tf.zeros((X.shape[0] - h + 1, X.shape[1] - w + 1)))
    for i in range(Y.shape[0]):
        for j in range(Y.shape[1]):
            Y[i, j].assign(tf.reduce_sum(
                X[i: i + h, j: j + w] * K))
    return Y

JAX

def corr2d(X, K):
    """Compute 2D cross-correlation."""
    h, w = K.shape
    Y = jnp.zeros((X.shape[0] - h + 1, X.shape[1] - w + 1))
    for i in range(Y.shape[0]):
        for j in range(Y.shape[1]):
            Y = Y.at[i, j].set((X[i:i + h, j:j + w] * K).sum())
    return Y

MXNet

def corr2d(X, K):
    """Compute 2D cross-correlation."""
    h, w = K.shape
    Y = d2l.zeros((X.shape[0] - h + 1, X.shape[1] - w + 1))
    for i in range(Y.shape[0]):
        for j in range(Y.shape[1]):
            Y[i, j] = d2l.reduce_sum((X[i: i + h, j: j + w] * K))
    return Y

We can construct the input tensor X and the kernel tensor K from Figure 6.2.1 to validate the output of the above implementation of the two-dimensional cross-correlation operation.

X = d2l.tensor([[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]])
K = d2l.tensor([[0.0, 1.0], [2.0, 3.0]])
corr2d(X, K)

PyTorch

tensor([[19., 25.],
        [37., 43.]])

TensorFlow

<tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=
array([[19., 25.],
       [37., 43.]], dtype=float32)>

JAX

Array([[19., 25.],
       [37., 43.]], dtype=float32)

MXNet

array([[19., 25.],
       [37., 43.]])

6.2.2 Convolutional Layers

A convolutional layer cross-correlates the input and kernel and adds a scalar bias to produce an output. The two parameters of a convolutional layer are the kernel and the scalar bias. When training models based on convolutional layers, we typically initialize the kernels randomly, as with a fully connected layer.

We are now ready to implement a two-dimensional convolutional layer based on the corr2d function defined above. In the __init__ constructor method, we declare weight and bias as the two model parameters. The forward propagation method calls the corr2d function and adds the bias.

PyTorch

class Conv2D(nn.Module):
    def __init__(self, kernel_size):
        super().__init__()
        self.weight = nn.Parameter(torch.rand(kernel_size))
        self.bias = nn.Parameter(torch.zeros(1))

    def forward(self, x):
        return corr2d(x, self.weight) + self.bias

TensorFlow

class Conv2D(tf.keras.layers.Layer):
    def __init__(self):
        super().__init__()

    def build(self, kernel_size):
        initializer = tf.random_normal_initializer()
        self.weight = self.add_weight(name='w', shape=kernel_size,
                                      initializer=initializer)
        self.bias = self.add_weight(name='b', shape=(1, ),
                                    initializer=initializer)

    def call(self, inputs):
        return corr2d(inputs, self.weight) + self.bias

JAX

class Conv2D(nnx.Module):
    def __init__(self, kernel_size, rngs=None):
        rngs = nnx.Rngs(d2l.get_key()) if rngs is None else rngs
        self.weight = nnx.Param(
            nnx.initializers.uniform()(rngs.params(), kernel_size))
        self.bias = nnx.Param(jnp.zeros(1))

    def __call__(self, x):
        return corr2d(x, self.weight) + self.bias

MXNet

class Conv2D(nn.Block):
    def __init__(self, kernel_size):
        super().__init__()
        self.weight = gluon.Parameter('weight', shape=kernel_size)
        self.bias = gluon.Parameter('bias', shape=(1,))

    def forward(self, x):
        return corr2d(x, self.weight.data()) + self.bias.data()

In an \(h \times w\) convolution, or an \(h \times w\) convolution kernel, the height and width of the kernel are \(h\) and \(w\), respectively. We call a convolutional layer with this kernel an \(h \times w\) convolutional layer.

6.2.3 Object Edge Detection in Images

As a first application, we use a convolutional layer to detect an object’s edge by locating changes in pixel values. First, we construct an “image” of \(6\times 8\) pixels. The middle four columns are black (\(0\)) and the rest are white (\(1\)).

PyTorch

X = d2l.ones((6, 8))
X[:, 2:6] = 0
X

tensor([[1., 1., 0., 0., 0., 0., 1., 1.],
        [1., 1., 0., 0., 0., 0., 1., 1.],
        [1., 1., 0., 0., 0., 0., 1., 1.],
        [1., 1., 0., 0., 0., 0., 1., 1.],
        [1., 1., 0., 0., 0., 0., 1., 1.],
        [1., 1., 0., 0., 0., 0., 1., 1.]])

TensorFlow

X = tf.Variable(tf.ones((6, 8)))
X[:, 2:6].assign(tf.zeros(X[:, 2:6].shape))
X

<tf.Variable 'Variable:0' shape=(6, 8) dtype=float32, numpy=
array([[1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.]], dtype=float32)>

JAX

X = jnp.ones((6, 8))
X = X.at[:, 2:6].set(0)
X

Array([[1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.]], dtype=float32)

MXNet

X = d2l.ones((6, 8))
X[:, 2:6] = 0
X

array([[1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.],
       [1., 1., 0., 0., 0., 0., 1., 1.]])

Next, we construct a kernel K with a height of 1 and a width of 2,

\[\mathbf{K} = \begin{bmatrix} 1 & -1 \end{bmatrix}. \tag{6.2.2}\]

When we perform the cross-correlation operation with the input, if the horizontally adjacent elements are the same, the output is 0. Otherwise, the output is nonzero. This kernel is a special case of a finite-difference operator. At location \((i,j)\) it computes \(x_{i,j} - x_{i,j+1}\), the difference between horizontally adjacent pixels. Up to a sign, this is a discrete approximation to the first derivative in the horizontal direction. For a function \(f(i,j)\), the derivative is

\[\partial_j f(i,j) = \lim_{\epsilon \to 0} \frac{f(i,j+\epsilon) - f(i,j)}{\epsilon},\]

so the kernel output \(x_{i,j} - x_{i,j+1}\) approximates \(-\partial_j f(i,j)\).

K = d2l.tensor([[1.0, -1.0]])

We are ready to perform the cross-correlation operation with arguments X (our input) and K (our kernel). The output is \(1\) at the edge from white to black and \(-1\) for the edge from black to white. All other outputs take value \(0\).

Y = corr2d(X, K)
Y

PyTorch

tensor([[ 0.,  1.,  0.,  0.,  0., -1.,  0.],
        [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
        [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
        [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
        [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
        [ 0.,  1.,  0.,  0.,  0., -1.,  0.]])

TensorFlow

<tf.Variable 'Variable:0' shape=(6, 7) dtype=float32, numpy=
array([[ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.]], dtype=float32)>

JAX

Array([[ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.]], dtype=float32)

MXNet

array([[ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
       [ 0.,  1.,  0.,  0.,  0., -1.,  0.]])

We can now apply the kernel to the transposed image. As expected, it vanishes. The kernel K only detects vertical edges.

corr2d(d2l.transpose(X), K)

PyTorch

tensor([[0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.]])

TensorFlow

<tf.Variable 'Variable:0' shape=(8, 5) dtype=float32, numpy=
array([[0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.]], dtype=float32)>

JAX

Array([[0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.]], dtype=float32)

MXNet

array([[0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.]])

6.2.4 Learning a Kernel

The finite difference [1, -1] works when the desired feature is known in advance. For larger kernels and successive convolutional layers, manually specifying every filter is generally impractical.

We therefore learn the kernel that maps X to Y from input–output pairs. We first construct a convolutional layer and initialize its kernel as a random tensor. Next, in each iteration, we will use the squared error to compare Y with the output of the convolutional layer. We can then calculate the gradient to update the kernel. The following example uses the built-in two-dimensional convolutional layer and omits the bias.

PyTorch

# Construct a two-dimensional convolutional layer with 1 output channel and a
# kernel of shape (1, 2). For the sake of simplicity, we ignore the bias here
conv2d = nn.LazyConv2d(1, kernel_size=(1, 2), bias=False)

# The two-dimensional convolutional layer uses four-dimensional input and
# output in the format of (example, channel, height, width), where the batch
# size (number of examples in the batch) and the number of channels are both 1
X = X.reshape((1, 1, 6, 8))
Y = Y.reshape((1, 1, 6, 7))
lr = 3e-2  # Learning rate

for i in range(10):
    Y_hat = conv2d(X)
    l = (Y_hat - Y) ** 2
    conv2d.zero_grad()
    l.sum().backward()
    # Update the kernel
    with torch.no_grad():
        conv2d.weight[:] -= lr * conv2d.weight.grad
    if (i + 1) % 2 == 0:
        print(f'epoch {i + 1}, loss {l.sum():.3f}')

epoch 2, loss 10.235
epoch 4, loss 3.103
epoch 6, loss 1.088
epoch 8, loss 0.415
epoch 10, loss 0.165

TensorFlow

# Construct a two-dimensional convolutional layer with 1 output channel and a
# kernel of shape (1, 2). For the sake of simplicity, we ignore the bias here
conv2d = tf.keras.layers.Conv2D(1, (1, 2), use_bias=False)

# The two-dimensional convolutional layer uses four-dimensional input and
# output in the format of (example, height, width, channel), where the batch
# size (number of examples in the batch) and the number of channels are both 1
X = tf.reshape(X, (1, 6, 8, 1))
Y = tf.reshape(Y, (1, 6, 7, 1))
lr = 3e-2  # Learning rate

for i in range(10):
    with tf.GradientTape() as g:
        Y_hat = conv2d(X)
        l = (Y_hat - Y) ** 2
    # Update the kernel
    update = tf.multiply(lr, g.gradient(l, conv2d.trainable_weights)[0])
    conv2d.kernel.assign(conv2d.kernel - update)
    if (i + 1) % 2 == 0:
        print(f'epoch {i + 1}, loss {tf.reduce_sum(l):.3f}')

epoch 2, loss 0.035
epoch 4, loss 0.010
epoch 6, loss 0.003
epoch 8, loss 0.001
epoch 10, loss 0.000

JAX

# Construct a two-dimensional convolutional layer with 1 output channel and a
# kernel of shape (1, 2). For the sake of simplicity, we ignore the bias here.
# Use a small-stddev normal init so the toy 10-step SGD has time to converge
# (Flax's default lecun_normal yields a much larger initial loss).
conv2d = nnx.Conv(1, 1, kernel_size=(1, 2), use_bias=False, padding='VALID',
                  kernel_init=nnx.initializers.normal(stddev=0.01),
                  rngs=nnx.Rngs(d2l.get_key()))

# The two-dimensional convolutional layer uses four-dimensional input and
# output in the format of (example, height, width, channel), where the batch
# size (number of examples in the batch) and the number of channels are both 1
X = X.reshape((1, 6, 8, 1))
Y = Y.reshape((1, 6, 7, 1))
lr = 3e-2  # Learning rate

def loss(model, X, Y):
    Y_hat = model(X)
    return ((Y_hat - Y) ** 2).sum()

for i in range(10):
    l, grads = nnx.value_and_grad(loss)(conv2d, X, Y)
    conv2d.kernel[...] -= lr * grads.kernel[...]
    if (i + 1) % 2 == 0:
        print(f'epoch {i + 1}, loss {l:.3f}')

epoch 2, loss 4.868
epoch 4, loss 0.817
epoch 6, loss 0.137
epoch 8, loss 0.023
epoch 10, loss 0.004

MXNet

# Construct a two-dimensional convolutional layer with 1 output channel and a
# kernel of shape (1, 2). For the sake of simplicity, we ignore the bias here
conv2d = nn.Conv2D(1, kernel_size=(1, 2), use_bias=False)
conv2d.initialize()

# The two-dimensional convolutional layer uses four-dimensional input and
# output in the format of (example, channel, height, width), where the batch
# size (number of examples in the batch) and the number of channels are both 1
X = X.reshape(1, 1, 6, 8)
Y = Y.reshape(1, 1, 6, 7)
lr = 3e-2  # Learning rate

for i in range(10):
    with autograd.record():
        Y_hat = conv2d(X)
        l = (Y_hat - Y) ** 2
    l.backward()
    # Update the kernel
    conv2d.weight.data()[:] -= lr * conv2d.weight.grad()
    if (i + 1) % 2 == 0:
        print(f'epoch {i + 1}, loss {float(l.sum()):.3f}')

epoch 2, loss 5.413
epoch 4, loss 0.909
epoch 6, loss 0.153
epoch 8, loss 0.026
epoch 10, loss 0.004

After 10 iterations, the error is small. We can inspect the learned kernel.

PyTorch

d2l.reshape(conv2d.weight.data, (1, 2))

tensor([[ 0.9475, -1.0304]])

TensorFlow

d2l.reshape(conv2d.get_weights()[0], (1, 2))

<tf.Tensor: shape=(1, 2), dtype=float32, numpy=array([[ 1.0029018, -0.9985178]], dtype=float32)>

JAX

conv2d.kernel[...].reshape((1, 2))

Array([[ 0.98822   , -0.98883337]], dtype=float32)

MXNet

d2l.reshape(conv2d.weight.data(), (1, 2))

array([[ 0.98865455, -0.9871513 ]])

Indeed, the learned kernel tensor is close to the kernel tensor K we defined earlier.

6.2.5 Cross-Correlation and Convolution

Recall our observation from Section 6.1 of the correspondence between the cross-correlation and convolution operations. For two-dimensional layers, strict convolution as defined in Equation 6.1.5 differs from cross-correlation only by flipping the kernel horizontally and vertically before applying cross-correlation.

Since kernels are learned from data in deep learning, the outputs of convolutional layers remain unaffected whether the layers perform strict convolution or cross-correlation.

To illustrate this, suppose that a convolutional layer performs cross-correlation and learns the kernel in Figure 6.2.1, which is here denoted as the matrix \(\mathbf{K}\). Assuming that other conditions remain unchanged, when this layer instead performs strict convolution, the learned kernel \(\mathbf{K}'\) will be the same as \(\mathbf{K}\) after \(\mathbf{K}'\) is flipped both horizontally and vertically. Thus, when the convolutional layer performs strict convolution on the input in Figure 6.2.1 with \(\mathbf{K}'\), it produces the same output as in Figure 6.2.1 (the cross-correlation of the input and \(\mathbf{K}\)).

In keeping with standard terminology in deep learning literature, we will continue to refer to the cross-correlation operation as a convolution even though the operations differ mathematically. We use the term element to refer to an entry (or component) of any tensor representing a layer representation or a convolution kernel.

6.2.6 Convolution as Matrix Multiplication

Every output element in Figure 6.2.1 is a dot product: the kernel, flattened into a vector of length \(k_\textrm{h} k_\textrm{w}\), multiplied with the input patch under the window, flattened the same way. For an input of spatial size \(h \times w\), the number of such patches depends on the kernel, padding, and stride. If we extract each patch the sliding window visits, flatten it into a row, and stack the rows, we obtain a matrix with one row per output position. The entire cross-correlation then collapses into a single product of this patch matrix with the flattened kernel. The rearrangement is called im2col (it turns image patches into the columns, here rows, of a matrix).

This rewriting motivates one important family of convolution implementations. GPUs and other accelerators are built around fast dense matrix multiplication, and libraries often lower convolutions to explicit or implicit matrix products. Depending on the shapes and hardware, they may instead select direct, Winograd, FFT-based, or other specialized kernels. The explicit form costs memory: each input element is duplicated in up to \(k_\textrm{h} k_\textrm{w}\) rows. Let’s build the patch matrix for the input and kernel of Figure 6.2.1 and check that the matrix product reproduces corr2d.

X = d2l.tensor([[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]])
K = d2l.tensor([[0.0, 1.0], [2.0, 3.0]])
h, w = K.shape
p_h, p_w = X.shape[0] - h + 1, X.shape[1] - w + 1
patches = d2l.stack([d2l.reshape(X[i:i + h, j:j + w], (-1,))
                     for i in range(p_h) for j in range(p_w)])
Y_mat = d2l.reshape(d2l.matmul(patches, d2l.reshape(K, (-1, 1))), (p_h, p_w))
Y_mat, d2l.reduce_sum(d2l.abs(Y_mat - corr2d(X, K)))

PyTorch

(tensor([[19., 25.],
         [37., 43.]]),
 tensor(0.))

TensorFlow

(<tf.Tensor: shape=(2, 2), dtype=float32, numpy=
 array([[19., 25.],
        [37., 43.]], dtype=float32)>,
 <tf.Tensor: shape=(), dtype=float32, numpy=0.0>)

JAX

(Array([[19., 25.],
        [37., 43.]], dtype=float32),
 Array(0., dtype=float32))

MXNet

(array([[19., 25.],
        [37., 43.]]),
 array(0.))

The sum of absolute differences is zero: the one matrix product and the sliding-window loop compute the same output.

PyTorch

PyTorch exposes this rearrangement directly: F.unfold extracts the patch matrix (transposed, and with batch and channel dimensions), so any convolution can be written as unfold, matrix multiplication, and reshape.

PyTorch

X_batch = d2l.reshape(X, (1, 1, 3, 3))
patches_unfold = F.unfold(X_batch, kernel_size=(2, 2))[0].T
torch.allclose(patches_unfold, patches)

True

The same idea also works in the opposite direction: instead of unfolding the input, we can unroll the kernel into a sparse, banded matrix that multiplies the flattened image. The exercises explore this view.

6.2.7 Feature Map and Receptive Field

As described in Section 6.1.4, the convolutional layer output in Figure 6.2.1 is sometimes called a feature map, as it can be regarded as the learned representations (features) in the spatial dimensions (e.g., width and height) that it passes on to the subsequent layer. In CNNs, for any element \(x\) of some layer, its receptive field refers to all the elements (from all the previous layers) that may affect the calculation of \(x\) during the forward propagation. The receptive field can extend beyond the valid input region when padding is used.

We use Figure 6.2.1 to illustrate the receptive field. Given the \(2 \times 2\) convolution kernel, the receptive field of the shaded output element (of value \(19\)) is the four elements in the shaded portion of the input. Now let’s denote the \(2 \times 2\) output as \(\mathbf{Y}\) and consider a deeper CNN with an additional \(2 \times 2\) convolutional layer that takes \(\mathbf{Y}\) as its input, outputting a single element \(z\). In this case, the receptive field of \(z\) on \(\mathbf{Y}\) includes all the four elements of \(\mathbf{Y}\), while the receptive field on the input includes all the nine input elements. Stacking layers therefore lets an element respond to input features over a broader area.

This layer-by-layer counting has a closed form. Consider a stack of \(L\) convolutional layers in which layer \(i\) has a \(k_i \times k_i\) kernel and stride \(s_i\) (padding affects only which outputs exist, not how far each one sees). One step at the input of layer \(i\) corresponds to \(\prod_{j=1}^{i-1} s_j\) steps at the original input, since every earlier layer with stride \(s_j\) multiplies the step size by \(s_j\). Layer \(i\)’s kernel spans \(k_i - 1\) steps of its own input, so it widens the receptive field by \((k_i - 1) \prod_{j=1}^{i-1} s_j\) input pixels. Starting from a single pixel and summing over layers, an element at the top of the stack has a receptive field of side length

\[ r = 1 + \sum_{i=1}^{L} \left( k_i - 1 \right) \prod_{j=1}^{i-1} s_j, \tag{6.2.3}\]

where the empty product for \(i = 1\) equals \(1\).

The most common case is \(L\) stacked \(3 \times 3\) layers with stride \(1\): each layer adds \(2\), so the stack sees \((2L + 1) \times (2L + 1)\) input pixels. Two such layers cover \(5 \times 5\), three cover \(7 \times 7\). This is why deep stacks of small kernels can replace single large ones: three \(3 \times 3\) layers match the receptive field of one \(7 \times 7\) layer with fewer parameters (\(27\) weights instead of \(49\) per input–output channel pair) and three nonlinearities instead of one. Strides enter through the product: after a stride-\(2\) layer (or a pooling step, Section 6.5), every later kernel counts double, so downsampling makes the receptive field grow geometrically with depth. We will use Equation 6.2.3 repeatedly when we analyze modern architectures in Chapter 7.

Equation Equation 6.2.3 gives the theoretical receptive field: the set of inputs that can affect an activation at all. Their influence is not uniform. Luo et al. (2016) measured the effective receptive field through gradients and found that it concentrates near the center of the theoretical region, with a roughly Gaussian profile in common randomly initialized and trained networks. Depth, dilation, and larger kernels enlarge the set of possible inputs; they do not guarantee that optimization will use all of it equally.

Receptive fields derive their name from neurophysiology. Experiments recording from the visual cortex of several animal species (Hubel and Wiesel 1959, 1962, 1968) found that its lower levels respond to edges and related shapes. Later, Field (1987) modeled these responses on natural images with what are, in effect, convolutional kernels. We reprint a key figure in Figure 6.2.2.

Figure 6.2.2: Six receptive-field filters and their responses to a natural image, adapted from Field (1987) . The left panel shows the filters, the middle panel the input image, and the right panel the filtered responses sampled at intervals proportional to each filter’s size. Different filters respond to different local edge patterns.

The correspondence extends to features computed by deeper layers of networks trained on image classification (Kuzovkin et al. 2018).

6.2.8 Summary

The core computation in a convolutional layer is cross-correlation. A nested loop states the operation directly; practical libraries select among implicit matrix multiplication, direct, Winograd, FFT-based, and specialized kernels. Locality creates extensive reuse of weights and overlapping input patches, but real performance depends on both arithmetic throughput and memory movement.

Convolutions can detect edges and lines, blur images, or sharpen them. In a neural network, the filters are learned from data rather than specified by hand. The language of receptive fields has a historical connection to visual neurophysiology, but that analogy is motivation rather than evidence that a particular network mirrors the brain.

6.2.9 Exercises

1. [code] Diagonal edges. Construct a \(6 \times 6\) image X whose entries are \(1\) on and below the main diagonal and \(0\) above it. Let \(\mathbf{K}\) be the kernel of Equation 6.2.2.

   1. Apply \(\mathbf{K}\) and \(\mathbf{K}^\top\) to X with corr2d. Give the positions and signs of the nonzero entries of each output, and explain them from the pixel differences that each kernel computes.
   2. Let \(\mathbf{A}\) be any input and \(\mathbf{B}\) any kernel. Show that cross-correlating \(\mathbf{A}^\top\) with \(\mathbf{B}\) gives the transpose of cross-correlating \(\mathbf{A}\) with \(\mathbf{B}^\top\). What does this identity predict for corr2d applied to the transpose of X with \(\mathbf{K}\)?

2. [code] Hand-designed kernels. Design some kernels manually.

   1. Given a directional vector \(\mathbf{v} = (v_1, v_2)\), derive an edge-detection kernel that detects edges orthogonal to \(\mathbf{v}\), i.e., edges in the direction \((v_2, -v_1)\).
   2. Derive a finite difference operator for the second derivative. Which structures in images respond most strongly to it?
   3. What is the minimum size of a kernel to obtain a derivative of order \(d\)?
   4. How would you design a blur kernel? Why might you want to use such a kernel?
   5. Load a single image, convert it to one channel, and downsample it to about \(100 \times 100\) pixels, since corr2d loops over every output pixel in Python. Apply three kernels with corr2d: the edge detector, the blur kernel, and a sharpening kernel defined as twice the identity kernel minus the blur kernel, where the identity kernel has a \(1\) at its center and \(0\) elsewhere. Display each result next to the original.

   Adapted from Stanford CS231n, Assignment 2, ConvolutionalNetworks notebook.

3. [code] Convolution as a banded matrix. The patch matrix of Section 6.2.6 unfolds the input. The opposite construction unrolls the kernel. Let \(\mathbf{X}\) be a \(4 \times 4\) input, \(\mathbf{K}\) a \(3 \times 3\) kernel, and \(\textrm{vec}(\cdot)\) the row-by-row flattening.

   1. Construct the \(4 \times 16\) matrix \(\mathbf{W}\) with \(\mathbf{W}\,\textrm{vec}(\mathbf{X}) = \textrm{vec}(\mathbf{Y})\), where \(\mathbf{Y}\) is the cross-correlation of \(\mathbf{X}\) with \(\mathbf{K}\). How many nonzero entries does each row have, and how many distinct values appear in \(\mathbf{W}\)?
   2. Build \(\mathbf{W}\) in code from K and confirm that the product reproduces corr2d(X, K).
   3. For an \(n \times n\) input and a \(k \times k\) kernel, count the entries of \(\mathbf{W}\) stored as a dense array, the nonzero entries of \(\mathbf{W}\), and the entries of the patch matrix. How does each count grow with \(n\)?
   4. Keep the sparsity pattern of \(\mathbf{W}\) but give every row its own \(k^2\) values. The result is a locally connected layer: the fourth-order tensor \(\mathsf{V}\) of Equation 6.1.2 restricted to a \(k \times k\) window of offsets \((a, b)\), before translation equivariance ties the weights. For an \(n \times n\) input, count the parameters of this layer, of a fully connected layer with the same input and output shapes, and of the convolution.

   Adapted from CMU 11-785, Homework 2 Part 1.

4. [code] Backward pass of a convolution. ● Let \(\mathbf{Y}\) be the single-channel cross-correlation of an input \(\mathbf{X}\) with a kernel \(\mathbf{K}\), as computed by corr2d, and let \(\ell\) be a scalar loss that depends on \(\mathbf{X}\) and \(\mathbf{K}\) only through \(\mathbf{Y}\).

   1. Derive \(\partial \ell/\partial \mathbf{K}\) and \(\partial \ell/\partial \mathbf{X}\) in terms of \(\partial \ell/\partial \mathbf{Y}\), in the style of the backpropagation equations of Section 4.3.
   2. Implement your expression for \(\partial \ell/\partial \mathbf{K}\) without automatic differentiation for the loss \(\ell = \sum_{i,j} G_{ij} Y_{ij}\), where \(\mathbf{G}\) is a fixed random matrix of the shape of \(\mathbf{Y}\). Compare it with a centered finite-difference gradient on a small random X and K in single precision. Report the largest relative error for step sizes \(10^{-2}\), \(10^{-3}\), and \(10^{-4}\), and explain how the error changes with the step size.
   3. Check your expression for \(\partial \ell/\partial \mathbf{K}\) against the kernel gradient that automatic differentiation computes in the experiment of Section 6.2.4.

   Adapted from Michael Nielsen, Neural Networks and Deep Learning, Chapter 6, and Stanford CS231n, Assignment 2, ConvolutionalNetworks notebook.

5. [code] Effective receptive field. The theoretical receptive field of Equation 6.2.3 bounds which inputs can influence an output; the effective receptive field measures how much they actually do (Luo et al. 2016). Build a stack of \(L\) single-channel \(3 \times 3\) convolutional layers with stride 1, no padding, no bias, and no nonlinearity on a \(32 \times 32\) input, using the built-in layer of Section 6.2.4. Draw every kernel entry independently from a standard normal distribution.

   For \(L = 3\) and a single draw of the kernels, backpropagate a unit gradient from an output unit near the center of the last layer to the input and plot the magnitude of the input gradient. Compare its support with the \(7 \times 7\) theoretical receptive field, and say where the magnitude is largest.

6. [code] Effective width versus depth. ● Use the stack of \(L\) random linear layers of the previous problem.

   1. Average the squared input gradient over 500 draws of the kernels, for \(L = 3\) and for \(L = 10\). Sum each averaged map over its rows and normalize the resulting column profile to sum to one. How does the ratio of the profile’s standard deviation to the theoretical side length \(2L + 1\) change with depth?
   2. Show that the expected squared input gradient, summed over rows, is proportional to the convolution of \(L\) copies of the vector \((1, 1, 1)\). Derive how the standard deviation of the normalized profile grows with \(L\).

PyTorch

7. [code] Autodiff on the custom layer. corr2d fills its output with the in-place assignment Y[i, j] = ... into a tensor that does not itself require gradients.
   1. Create K with requires_grad=True, run corr2d(X, K).sum().backward(), and inspect K.grad and the grad_fn of the output. Does the assignment block differentiation? Explain what autograd records for it.
   2. Rebuild, as two-dimensional tensors, the \(6 \times 8\) image X of Section 6.2.3 and its edge map Y = corr2d(X, K) for the kernel K of Equation 6.2.2. Train Conv2D((1, 2)) on them by plain gradient descent and compare the learned weight with K.

TensorFlow

7. [code] Autodiff on the custom layer. corr2d writes its output into a tf.Variable with Y[i, j].assign(...). Watch K on a tf.GradientTape, evaluate corr2d(X, K), and request the gradient of its sum with respect to K. What does the tape return, and why does a variable assignment break the chain from K to the output? Rewrite corr2d so that it collects the window sums in a Python list and assembles the output with tf.stack, and verify that the gradient is now defined.

JAX

7. [code] Autodiff on the custom layer. corr2d builds its output with the functional update Y.at[i, j].set(...). Compute jax.grad(lambda K: corr2d(X, K).sum())(K) and explain why this update poses no obstacle to differentiation. Then wrap corr2d in jax.jit and compare the compile time for a \(6 \times 8\) and a \(28 \times 28\) input. Why does the Python loop make the compiled program grow with the input size?

MXNet

7. [code] Autodiff on the custom layer. Call K.attach_grad(), evaluate corr2d(X, K) inside autograd.record(), and call backward(). What error message do you see, and which line of corr2d triggers it? Rewrite corr2d without in-place assignment so that the gradient with respect to K can be computed.

Field, David J. 1987. “Relations Between the Statistics of Natural Images and the Response Properties of Cortical Cells.” JOSA A 4 (12): 2379–94. https://doi.org/10.1364/josaa.4.002379.

Hubel, David H, and Torsten N Wiesel. 1959. “Receptive Fields of Single Neurones in the Cat’s Striate Cortex.” Journal of Physiology 148 (3): 574–91. https://doi.org/10.1113/jphysiol.1959.sp006308.

Hubel, David H, and Torsten N Wiesel. 1962. “Receptive Fields, Binocular Interaction and Functional Architecture in the Cat’s Visual Cortex.” Journal of Physiology 160 (1): 106–54. https://doi.org/10.1113/jphysiol.1962.sp006837.

Hubel, David H, and Torsten N Wiesel. 1968. “Receptive Fields and Functional Architecture of Monkey Striate Cortex.” Journal of Physiology 195 (1): 215–43. https://doi.org/10.1113/jphysiol.1968.sp008455.

Kuzovkin, Ilya, Raul Vicente, Mathilde Petton, et al. 2018. “Activations of Deep Convolutional Neural Networks Are Aligned with Gamma Band Activity of Human Visual Cortex.” Communications Biology 1 (1): 1–12. https://doi.org/10.1038/s42003-018-0110-y.

Luo, Wenjie, Yujia Li, Raquel Urtasun, and Richard Zemel. 2016. “Understanding the Effective Receptive Field in Deep Convolutional Neural Networks.” Advances in Neural Information Processing Systems. https://arxiv.org/abs/1701.04128.