6  Convolutional Neural Networks

Image data is represented as a two-dimensional grid of pixels, be the image monochromatic or in color. Accordingly each pixel corresponds to one or multiple numerical values respectively. So far we have ignored this rich structure and treated images as vectors of numbers by flattening them, irrespective of the spatial relation between pixels. This deeply unsatisfying approach was necessary in order to feed the resulting one-dimensional vectors through a fully connected MLP.

An MLP assigns a separate weight to every input coordinate but has no built-in notion that two coordinates are neighbors. If we permute the pixels in every image consistently and retrain the MLP, we have merely renamed its input coordinates; the model class has not changed, even though the spatial layout has disappeared. Ideally, we would use our prior knowledge that nearby pixels are typically related to each other, to build efficient models for learning from image data.

This chapter introduces convolutional neural networks (CNNs) (LeCun, Jackel, et al. 1995), a powerful family of neural networks that are designed for precisely this purpose. On the ImageNet collection (Deng et al. 2009) it was the use of convolutional neural networks, in short CNNs, that provided significant performance improvements (Krizhevsky et al. 2012), and CNN-based architectures dominated computer vision from roughly 2012 to 2021. Today they share the field with vision transformers (Chapter 11) and remain the default where latency, small datasets, or dense prediction dominate.

Modern CNNs, as they are called colloquially, owe their design to inspirations from biology, group theory, and a healthy dose of experimental tinkering. In addition to their sample efficiency in achieving accurate models, CNNs tend to be computationally efficient, both because they require fewer parameters than fully connected architectures and because convolutions are easy to parallelize across GPU cores (Chetlur et al. 2014). Consequently, practitioners often apply CNNs whenever possible, and increasingly they have emerged as credible competitors even on tasks with a one-dimensional sequence structure, such as audio (Abdel-Hamid et al. 2014), text (Kalchbrenner et al. 2014), and time series analysis (LeCun, Bengio, et al. 1995), where recurrent neural networks are conventionally used. Some clever adaptations of CNNs have also brought them to bear on graph-structured data (Kipf and Welling 2017) and in recommender systems.

First, we will examine the motivation for convolutional neural networks. This is followed by a walk through the basic operations that comprise the backbone of all convolutional networks. These include the convolutional layers themselves, nitty-gritty details including padding, stride, and dilation, the pooling layers used to aggregate information across adjacent spatial regions, the use of multiple channels at each layer, including grouped and depthwise-separable convolutions, and a careful discussion of the structure of modern architectures. We will conclude the chapter with a full working example of LeNet, one of the earliest convolutional networks deployed at scale, long before the rise of modern deep learning. In the next chapter, we will develop full implementations of some popular and comparatively recent CNN architectures whose designs represent most of the techniques commonly used by modern practitioners.

Resources and Further Reading

The references below develop the fundamentals of this chapter: why convolutions, the mechanics of the operation (padding, stride, dilation, channels), receptive fields, pooling, and the first convolutional networks. All are freely accessible online except where noted.

Books

Courses and video lectures

Tutorials, notes, and interactive

Foundational papers