6 Convolutional Neural Networks

An image is a two-dimensional grid of pixels, with one or more values at each location. The models introduced so far flatten this grid into a vector before applying a fully connected layer. Flattening discards the explicit spatial relationship between neighboring pixels.

An MLP assigns a separate weight to every input coordinate but has no built-in notion that two coordinates are neighbors. If we permute the pixels in every image consistently and retrain the MLP, we have only renamed its input coordinates; the model class has not changed, even though the spatial layout has disappeared. Ideally, we would use our prior knowledge that nearby pixels are typically related to each other, to build efficient models for learning from image data.

This chapter introduces convolutional neural networks (CNNs) (LeCun et al. 1995). CNNs preserve spatial organization by connecting each output to a local neighborhood and sharing the same weights across locations. On the ImageNet collection (Deng et al. 2009), convolutional neural networks delivered substantial performance improvements (Krizhevsky et al. 2012), and CNN-based architectures dominated computer vision from roughly 2012 to 2021. Today they share the field with vision transformers (Chapter 11) and remain the default where latency, small datasets, or dense prediction dominate.

These restrictions reduce the parameter count relative to a fully connected layer, and convolution kernels can be evaluated in parallel on GPUs (Chetlur et al. 2014). The same operation also appears in models for audio, text, and time series, but this chapter concentrates on images, where locality and translation equivariance have a direct spatial interpretation.

We first derive convolution from locality and translation equivariance in Section 6.1. We then define the operation and show how padding, stride, dilation, channels, and pooling control its shape and information flow. LeNet combines these components in a complete image classifier. The next chapter studies the architectural and training changes that made substantially deeper CNNs practical.

Resources and Further Reading

The references below develop the fundamentals of this chapter: why convolutions, the mechanics of the operation (padding, stride, dilation, channels), receptive fields, pooling, and the first convolutional networks. All are freely accessible online except where noted.

Books

Deep Learning — Goodfellow, Bengio & Courville — free HTML; Chapter 9 develops sparse interactions, parameter sharing, and equivariance in detail.
Understanding Deep Learning — Simon J. D. Prince — free PDF; Chapter 10 covers convolutions, stride, dilation, and receptive fields with unusually good figures.
Neural Networks and Deep Learning — Michael Nielsen — free online; Chapter 6 introduces convolutional layers, shared weights, and pooling from first principles.

Courses and video lectures

Stanford CS231n: Deep Learning for Computer Vision — free notes; the “Convolutional Networks” module (layer arithmetic, parameter counting, layer patterns) is the most widely used online companion to this chapter.
Michigan EECS 498-007: Deep Learning for Computer Vision — Justin Johnson — free lecture videos; the convolution and pooling lectures work through kernel/stride/padding arithmetic on the board at exactly this chapter’s pace.

Tutorials, notes, and interactive

A guide to convolution arithmetic for deep learning — Dumoulin & Visin (2016) — free, with the famous animations; the definitive visual reference for padding, stride, dilation, and transposed convolutions.
CNN Explainer — Wang et al. (Polo Club) — free, zero-install; an interactive convnet running in the browser where you can inspect every activation, kernel, and receptive field.
Computing Receptive Fields of Convolutional Neural Networks — Araujo, Norris & Sim (2019), Distill — free; derives the closed-form receptive-field arithmetic that Section 6.2 introduces, including strided and multi-path cases.
Conv Nets: A Modular Perspective — Chris Olah — free; a short classic building the “convolution as structured weight sharing” intuition.
Image Kernels Explained Visually — Victor Powell — free, interactive; slide hand-designed kernels over an image, a perfect warm-up for the edge-detection example in Section 6.2.
Feature Visualization — Olah, Mordvintsev & Schubert (2017), Distill — free; what convnet units actually respond to, a companion to the activation-visualization exercise in Section 6.6.

Foundational papers

Gradient-Based Learning Applied to Document Recognition — LeCun, Bottou, Bengio & Haffner (1998), Proc. IEEE — free PDF; the LeNet-5 paper behind Section 6.6, still worth reading for its systems-level completeness.
Backpropagation Applied to Handwritten Zip Code Recognition — LeCun et al. (1989), Neural Computation — the first convnet trained end-to-end with backprop (paywalled, noted; widely reproduced online).
Neocognitron — Fukushima (1980), Biological Cybernetics — the pre-backprop ancestor of alternating convolution/pooling stages (paywalled, noted).
Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex — Hubel & Wiesel (1962), J. Physiology — free (PMC); the biological root of local receptive fields and hierarchical feature detection.