7 Modern Convnets
Now that we understand how to wire up convolutional networks, let’s tour the architectures that made them the dominant tool of computer vision, and the practices that keep them competitive today. First, the honest picture. Since roughly 2021, vision transformers have led large-scale image classification benchmarks, beginning with Dosovitskiy et al. (2021) and the Swin Transformer (Liu et al. 2021); we cover them in Chapter 11. Convnets remain the workhorse wherever latency budgets are tight, training data is scarce, or the prediction is dense, as in segmentation (Long et al. 2015) and object detection (Redmon and Farhadi 2018). When trained with modern recipes at matched compute, they still match transformers on image classification (Smith et al. 2023). This chapter brings you to that state of the art as of 2026.
The chapter tells the story in two eras. The first is the architecture race of 2012–2015, set off when AlexNet (Krizhevsky et al. 2012) won the 2012 ImageNet competition by a wide margin (Section 7.1). The years that followed were about organizing convolutions: repeated blocks in VGG (Simonyan and Zisserman 2015), \(1 \times 1\) channel mixing and global pooling in NiN (Lin et al. 2013), and multi-branch design in GoogLeNet (Szegedy et al. 2015), all covered in Section 7.2. Batch normalization (Ioffe and Szegedy 2015) made deep networks train reliably (Section 7.3), and residual connections (He et al. 2016) removed the optimization barrier to depth (Section 7.4), with ResNeXt (Xie et al. 2017) and DenseNet (Huang et al. 2017) as the surviving variations. The race produced more than vision models: normalization layers and residual connections escaped computer vision entirely and now sit inside nearly every deep network, transformers included.
The second era, from 2016 to today, is one of maturation. Network topology stopped changing quickly and progress moved elsewhere. Deployment under latency and memory constraints produced a line of designs from depthwise-separable MobileNets (Howard et al. 2017) to structural re-parameterization (Section 7.5). Training recipes improved so much that an unmodified ResNet-50 gains over four points of ImageNet accuracy from the recipe alone (Wightman et al. 2021); Section 7.6 teaches the modern recipe, without which comparisons between architectures mislead. Building on that recipe, ConvNeXt (Liu et al. 2022) modernized a ResNet step by step, using only ideas from this book, into a network that matches a vision transformer of equal cost (Section 7.7). Finally, network design itself became an empirical science: rather than crafting a single network, the RegNet methodology (Radosavovic et al. 2020) explores whole design spaces, and Section 7.8 uses it to close the chapter with the big picture of where convnets and transformers each stand.
We present the architectures in roughly chronological order, partly to convey a sense of the history so that you can form your own intuitions about where the field is heading, and partly because each design responds to a concrete failure of its predecessors. The networks in this chapter are the product of intuition, a few mathematical insights, and a lot of trial and error; knowing why each idea was introduced is the best guide to when to reach for it.
Resources and Further Reading
The references below trace the architecture race this chapter narrates and the modern practice it lands on: landmark networks from AlexNet to ConvNeXt, the training recipes that confound naive comparisons between them, and efficient networks for deployment. All are freely accessible online except where noted.
Books
- Deep Learning for Coders with fastai and PyTorch — Howard & Gugger — free notebooks; builds ResNets from scratch and applies the modern training tricks (augmentation, schedules, mixed precision) of Section 7.6 in working code.
- Understanding Deep Learning — Simon J. D. Prince — free PDF; Chapter 11 (Residual networks) analyzes why residual connections ease optimization — loss-surface and gradient-propagation arguments complementing Section 7.4.
Courses and video lectures
- Michigan EECS 498-007, Lecture: CNN Architectures — Justin Johnson — free videos; walks AlexNet → VGG → GoogLeNet → ResNet with parameter/FLOP accounting, the same tour as this chapter’s first half.
- MIT 6.5940: TinyML and Efficient Deep Learning — Song Han — free lectures; depthwise separability, quantization, pruning, and edge deployment — the systems side of Section 7.5.
- Hugging Face Computer Vision Course — free; modern practice with pretrained backbones (including ConvNeXt) and transfer learning, the “what you actually do in 2026” companion to this chapter.
Tutorials, notes, and surveys
- timm (pytorch-image-models) — Ross Wightman et al. — free; the reference implementation zoo where every architecture in this chapter lives, with trained weights and results tables comparing them under consistent evaluation.
- A Recipe for Training Neural Networks — Andrej Karpathy — free; the debugging-first training discipline that Section 7.6 systematizes.
- Zoom In: An Introduction to Circuits — Olah et al. (2020), Distill — free; opens trained vision models to inspect the features and circuits they learn, useful perspective once you can train the architectures in this chapter.
Foundational papers
All free on arXiv or the proceedings site; these are the primary sources this chapter retells, worth reading in the original:
- ImageNet Classification with Deep Convolutional Neural Networks — Krizhevsky, Sutskever & Hinton (2012), NeurIPS — AlexNet (Section 7.1).
- Batch Normalization — Ioffe & Szegedy (2015) — read together with its critique in Section 7.3.
- Deep Residual Learning for Image Recognition — He, Zhang, Ren & Sun (2015) — ResNet (Section 7.4), the most-cited paper in deep learning.
- Bag of Tricks for Image Classification with CNNs — He et al. (2019) — the first systematic demonstration that recipe details rival architecture changes.
- ResNet Strikes Back — Wightman, Touvron & Jégou (2021) — the definitive recipe-vs-architecture accounting behind Section 7.6.
- A ConvNet for the 2020s — Liu et al. (2022) — ConvNeXt (Section 7.7), a controlled ablation worth studying as method, not just result.
- ConvNets Match Vision Transformers at Scale — Smith et al. (2023) — the scaling-law resolution of the convnet-vs-transformer debate closing this chapter.