Blocks, Bottlenecks, and Branches: VGG, NiN, GoogLeNet

After AlexNet: organize the convolutions

AlexNet proved deep CNNs work, but gave no template: every layer was designed individually.

The next generation contributed one organizing idea each:

• VGG repeats identical blocks; a network becomes a tuple.
• NiN mixes channels with 1×1 convs and replaces the FC head with global average pooling.
• GoogLeNet runs branches of several filter sizes in parallel and concatenates.

All three ideas are still in every modern network.

VGG: regular blocks at scale

VGG (Simonyan & Zisserman, 2014) is AlexNet taken seriously: stack more layers, but make them regular blocks of 3×3 conv + ReLU, ending in a 2×2 max-pool.

From AlexNet’s hand-tuned layers to VGG’s repeated 3×3 blocks.

Receptive field arithmetic

Why 3×3 only? Stacking small kernels grows the visible patch without paying for a large kernel in one step. For stride 1:

r = 1 + \sum_{i=1}^{L} (k_i - 1).

Two 3×3 convolutions see 1 + 2 + 2 = 5 pixels across: the same receptive field as one 5×5 conv, with 18c^2 instead of 25c^2 weights and one extra ReLU. Deep-and-narrow beats shallow-and-wide.

The VGG block

A reusable subunit: num_convs consecutive Conv-ReLU pairs, then a 2×2 MaxPool:

# Memory-footprint knobs (set before importing mxnet). VGG and NiN train
# back-to-back at 224x224; MXNet's default "Naive" GPU pool keeps each
# training's freed blocks in size-exact free lists, so NiN's differently
# shaped activations cannot reuse VGG's and the pool high-water grows. The
# "Round" pool buckets allocations by rounded size, letting the second model
# reuse the first's memory; disabling cuDNN autotune drops its scratch
# workspace. Together: ~7.7 GiB -> ~6.4 GiB true peak, with identical results.
import os
os.environ['MXNET_GPU_MEM_POOL_TYPE'] = 'Round'
os.environ['MXNET_CUDNN_AUTOTUNE_DEFAULT'] = '0'
from d2l import mxnet as d2l
from mxnet import np, npx, init
from mxnet.gluon import nn
npx.set_np()

def vgg_block(num_convs, num_channels):
    blk = nn.Sequential()
    for _ in range(num_convs):
        blk.add(nn.Conv2D(num_channels, kernel_size=3,
                          padding=1, activation='relu'))
    blk.add(nn.MaxPool2D(pool_size=2, strides=2))
    return blk

The VGG network

Five blocks at growing channel counts plus a 3-layer dense head. The “named architecture” is just a tuple of (n_convs, channels) pairs; a different tuple gives VGG-13/16/19:

class VGG(d2l.Classifier):
    def __init__(self, arch, lr=0.1, num_classes=10):
        super().__init__()
        self.save_hyperparameters()
        self.net = nn.Sequential()
        for (num_convs, num_channels) in arch:
            self.net.add(vgg_block(num_convs, num_channels))
        self.net.add(nn.Dense(4096, activation='relu'), nn.Dropout(0.5),
                     nn.Dense(4096, activation='relu'), nn.Dropout(0.5),
                     nn.Dense(num_classes))
        self.net.initialize(init.Xavier())

VGG-11 shape check

Each block halves the resolution; channels double until 512:

VGG(arch=((1, 64), (1, 128), (2, 256), (2, 512), (2, 512))).layer_summary(
    (1, 1, 224, 224))

Sequential output shape:     (1, 64, 112, 112)
Sequential output shape:     (1, 128, 56, 56)
Sequential output shape:     (1, 256, 28, 28)
Sequential output shape:     (1, 512, 14, 14)
Sequential output shape:     (1, 512, 7, 7)
Dense output shape:  (1, 4096)
Dropout output shape:    (1, 4096)
Dense output shape:  (1, 4096)
Dropout output shape:    (1, 4096)
Dense output shape:  (1, 10)

Training a thin VGG

Full VGG-11 is heavy for a notebook, so we thin the channels (16/32/64/128/128) and train on Fashion-MNIST:

model = VGG(arch=((1, 16), (1, 32), (2, 64), (2, 128), (2, 128)), lr=0.01)
trainer = d2l.Trainer(max_epochs=10, num_gpus=1)
data = d2l.FashionMNIST(batch_size=128, resize=(224, 224))
trainer.fit(model, data)

Same pipeline as AlexNet; the block design is what changed.

NiN: 1×1 convolutions and GAP

Network in network (Lin et al., 2013) attacks the FC head: VGG-11’s first dense layer alone needs ~400 MB in FP32.

• 1×1 convolutions: an MLP applied at every pixel; channel mixing and extra nonlinearity at zero spatial cost.
• Global average pooling: one number per channel replaces the giant dense classifier.

NiN vs. VGG: same body idea, radically different head.

The NiN block

A regular convolution followed by two 1×1 convolutions with ReLUs in between:

def nin_block(num_channels, kernel_size, strides, padding):
    blk = nn.Sequential()
    blk.add(nn.Conv2D(num_channels, kernel_size, strides, padding,
                      activation='relu'),
            nn.Conv2D(num_channels, kernel_size=1, activation='relu'),
            nn.Conv2D(num_channels, kernel_size=1, activation='relu'))
    return blk

The NiN model

Four NiN blocks (AlexNet’s kernel sizes), max-pool between them, and the last block emits num_classes channels. Then global average pooling. No fully connected layers at all.

class NiN(d2l.Classifier):
    def __init__(self, lr=0.1, num_classes=10):
        super().__init__()
        self.save_hyperparameters()
        self.net = nn.Sequential()
        self.net.add(
            nin_block(96, kernel_size=11, strides=4, padding=0),
            nn.MaxPool2D(pool_size=3, strides=2),
            nin_block(256, kernel_size=5, strides=1, padding=2),
            nn.MaxPool2D(pool_size=3, strides=2),
            nin_block(384, kernel_size=3, strides=1, padding=1),
            nn.MaxPool2D(pool_size=3, strides=2),
            nn.Dropout(0.5),
            nin_block(num_classes, kernel_size=3, strides=1, padding=1),
            nn.GlobalAvgPool2D(),
            nn.Flatten())
        self.net.initialize(init.Xavier())

NiN shape inspection

Spatial dims shrink, channels grow, and the final block already has one channel per class:

NiN().layer_summary((1, 1, 224, 224))

Sequential output shape:     (1, 96, 54, 54)
MaxPool2D output shape:  (1, 96, 26, 26)
Sequential output shape:     (1, 256, 26, 26)
MaxPool2D output shape:  (1, 256, 12, 12)
Sequential output shape:     (1, 384, 12, 12)
MaxPool2D output shape:  (1, 384, 5, 5)
Dropout output shape:    (1, 384, 5, 5)
Sequential output shape:     (1, 10, 5, 5)
GlobalAvgPool2D output shape:    (1, 10, 1, 1)
Flatten output shape:    (1, 10)

Training NiN

model = NiN(lr=0.05)
trainer = d2l.Trainer(max_epochs=10, num_gpus=1)
data = d2l.FashionMNIST(batch_size=128, resize=(224, 224))
trainer.fit(model, data)

The averaging head costs nothing and does not hurt accuracy; that surprise made GAP the default head ever since.

GoogLeNet: go wide

GoogLeNet (Szegedy et al., 2015) won ImageNet 2014 with two lasting contributions:

• The stem / body / head decomposition of a network, still the universal vocabulary.
• The Inception block: don’t pick a filter size, run 1×1, 3×3, 5×5, and pooling in parallel and concatenate the channels.

Four branches, four scales, one shared input.

The Inception block in code

Four branches, channel-concatenated; 1×1 convs shrink channels before the costly 3×3 and 5×5:

class Inception(nn.Block):
    # c1--c4 are the number of output channels for each branch
    def __init__(self, c1, c2, c3, c4):
        super().__init__()
        # Branch 1
        self.b1_1 = nn.Conv2D(c1, kernel_size=1, activation='relu')
        # Branch 2
        self.b2_1 = nn.Conv2D(c2[0], kernel_size=1, activation='relu')
        self.b2_2 = nn.Conv2D(c2[1], kernel_size=3, padding=1,
                              activation='relu')
        # Branch 3
        self.b3_1 = nn.Conv2D(c3[0], kernel_size=1, activation='relu')
        self.b3_2 = nn.Conv2D(c3[1], kernel_size=5, padding=2,
                              activation='relu')
        # Branch 4
        self.b4_1 = nn.MaxPool2D(pool_size=3, strides=1, padding=1)
        self.b4_2 = nn.Conv2D(c4, kernel_size=1, activation='relu')

    def forward(self, x):
        b1 = self.b1_1(x)
        b2 = self.b2_2(self.b2_1(x))
        b3 = self.b3_2(self.b3_1(x))
        b4 = self.b4_2(self.b4_1(x))
        return np.concatenate((b1, b2, b3, b4), axis=1)

Bottleneck arithmetic

First body block: 192 channels in, 64+128+32+32 = 256 out, spatial size unchanged.

The 5×5 branch, direct: 25 \cdot 192 \cdot 32 \approx 154k weights.

With a 16-channel 1×1 bottleneck first: 192 \cdot 16 + 25 \cdot 16 \cdot 32 \approx 16k, a 10× saving. This trick is in nearly every network since.

The whole network as data

Nine Inception blocks in three groups (2, 5, 2), pooling between groups. The hand-picked channel allocations are just a tuple; assembly is stem + body + head:

arch = (((64, (96, 128), (16, 32), 32), (128, (128, 192), (32, 96), 64)),
        ((192, (96, 208), (16, 48), 64), (160, (112, 224), (24, 64), 64),
         (128, (128, 256), (24, 64), 64), (112, (144, 288), (32, 64), 64),
         (256, (160, 320), (32, 128), 128)),
        ((256, (160, 320), (32, 128), 128), (384, (192, 384), (48, 128), 128)))

class GoogleNet(d2l.Classifier):
    def __init__(self, lr=0.1, num_classes=10):
        super().__init__()
        self.save_hyperparameters()
        pool = lambda: nn.MaxPool2D(pool_size=3, strides=2, padding=1)
        stem = nn.Sequential()
        stem.add(nn.Conv2D(64, kernel_size=7, strides=2, padding=3,
                           activation='relu'), pool(),
                 nn.Conv2D(64, kernel_size=1, activation='relu'),
                 nn.Conv2D(192, kernel_size=3, padding=1, activation='relu'),
                 pool())
        body = []
        for group in arch:
            blk = nn.Sequential()
            blk.add(*[Inception(*c) for c in group])
            body.append(blk)
        self.net = nn.Sequential()
        self.net.add(stem, body[0], pool(), body[1], pool(), body[2],
                     nn.GlobalAvgPool2D(), nn.Dense(num_classes))
        self.net.initialize(init.Xavier())

Shape check

GoogleNet().layer_summary((1, 1, 96, 96))

Sequential output shape:     (1, 192, 12, 12)
Sequential output shape:     (1, 480, 12, 12)
MaxPool2D output shape:  (1, 480, 6, 6)
Sequential output shape:     (1, 832, 6, 6)
MaxPool2D output shape:  (1, 832, 3, 3)
Sequential output shape:     (1, 1024, 3, 3)
GlobalAvgPool2D output shape:    (1, 1024, 1, 1)
Dense output shape:  (1, 10)

Cheaper than VGG (~7M vs. ~138M parameters) and more accurate: the start of deliberate cost–accuracy design.

What survived

• Blocks (VGG): everything is specified block by block.
• 1×1 convolution (NiN): the standard channel mixer.
• Global average pooling (NiN): the default head.
• Multi-branch (GoogLeNet): survives as grouped convolutions (ResNeXt) and train-time-only branches (RepVGG).
• Inception-style hand-tuned branch cocktails: extinct.

Next ingredient: normalization.