Blocks, Bottlenecks, and Branches: VGG, NiN, GoogLeNet

After AlexNet: organize the convolutions

AlexNet proved deep CNNs work, but gave no template: every layer was designed individually.

The next generation contributed one organizing idea each:

• VGG repeats identical blocks; a network becomes a tuple.
• NiN mixes channels with 1×1 convs and replaces the FC head with global average pooling.
• GoogLeNet runs branches of several filter sizes in parallel and concatenates.

All three ideas are still in every modern network.

VGG: regular blocks at scale

VGG (Simonyan & Zisserman, 2014) is AlexNet taken seriously: stack more layers, but make them regular blocks of 3×3 conv + ReLU, ending in a 2×2 max-pool.

From AlexNet’s hand-tuned layers to VGG’s repeated 3×3 blocks.

Receptive field arithmetic

Why 3×3 only? Stacking small kernels grows the visible patch without paying for a large kernel in one step. For stride 1:

r = 1 + \sum_{i=1}^{L} (k_i - 1).

Two 3×3 convolutions see 1 + 2 + 2 = 5 pixels across: the same receptive field as one 5×5 conv, with 18c^2 instead of 25c^2 weights and one extra ReLU. Deep-and-narrow beats shallow-and-wide.

The VGG block

A reusable subunit: num_convs consecutive Conv-ReLU pairs, then a 2×2 MaxPool:

from d2l import torch as d2l
import torch
from torch import nn
from torch.nn import functional as F

def vgg_block(num_convs, out_channels):
    layers = []
    for _ in range(num_convs):
        layers.append(nn.LazyConv2d(out_channels, kernel_size=3, padding=1))
        layers.append(nn.ReLU())
    layers.append(nn.MaxPool2d(kernel_size=2,stride=2))
    return nn.Sequential(*layers)

The VGG network

Five blocks at growing channel counts plus a 3-layer dense head. The “named architecture” is just a tuple of (n_convs, channels) pairs; a different tuple gives VGG-13/16/19:

class VGG(d2l.Classifier):
    def __init__(self, arch, lr=0.1, num_classes=10):
        super().__init__()
        self.save_hyperparameters()
        conv_blks = []
        for (num_convs, out_channels) in arch:
            conv_blks.append(vgg_block(num_convs, out_channels))
        self.net = nn.Sequential(
            *conv_blks, nn.Flatten(),
            nn.LazyLinear(4096), nn.ReLU(), nn.Dropout(0.5),
            nn.LazyLinear(4096), nn.ReLU(), nn.Dropout(0.5),
            nn.LazyLinear(num_classes))
        self.net.apply(d2l.init_cnn)

VGG-11 shape check

Each block halves the resolution; channels double until 512:

VGG(arch=((1, 64), (1, 128), (2, 256), (2, 512), (2, 512))).layer_summary(
    (1, 1, 224, 224))

Sequential output shape:     torch.Size([1, 64, 112, 112])
Sequential output shape:     torch.Size([1, 128, 56, 56])
Sequential output shape:     torch.Size([1, 256, 28, 28])
Sequential output shape:     torch.Size([1, 512, 14, 14])
Sequential output shape:     torch.Size([1, 512, 7, 7])
Flatten output shape:    torch.Size([1, 25088])
...
ReLU output shape:   torch.Size([1, 4096])
Dropout output shape:    torch.Size([1, 4096])
Linear output shape:     torch.Size([1, 4096])
ReLU output shape:   torch.Size([1, 4096])
Dropout output shape:    torch.Size([1, 4096])
Linear output shape:     torch.Size([1, 10])

Training a thin VGG

Full VGG-11 is heavy for a notebook, so we thin the channels (16/32/64/128/128) and train on Fashion-MNIST:

model = VGG(arch=((1, 16), (1, 32), (2, 64), (2, 128), (2, 128)), lr=0.01)
trainer = d2l.Trainer(max_epochs=10, num_gpus=1)
data = d2l.FashionMNIST(batch_size=128, resize=(224, 224))
model.apply_init([next(iter(data.get_dataloader(True)))[0]], d2l.init_cnn)
trainer.fit(model, data)

Same pipeline as AlexNet; the block design is what changed.

NiN: 1×1 convolutions and GAP

Network in network (Lin et al., 2013) attacks the FC head: VGG-11’s first dense layer alone needs ~400 MB in FP32.

• 1×1 convolutions: an MLP applied at every pixel; channel mixing and extra nonlinearity at zero spatial cost.
• Global average pooling: one number per channel replaces the giant dense classifier.

NiN vs. VGG: same body idea, radically different head.

The NiN block

A regular convolution followed by two 1×1 convolutions with ReLUs in between:

def nin_block(out_channels, kernel_size, strides, padding):
    return nn.Sequential(
        nn.LazyConv2d(out_channels, kernel_size, strides, padding), nn.ReLU(),
        nn.LazyConv2d(out_channels, kernel_size=1), nn.ReLU(),
        nn.LazyConv2d(out_channels, kernel_size=1), nn.ReLU())

The NiN model

Four NiN blocks (AlexNet’s kernel sizes), max-pool between them, and the last block emits num_classes channels. Then global average pooling. No fully connected layers at all.

class NiN(d2l.Classifier):
    def __init__(self, lr=0.1, num_classes=10):
        super().__init__()
        self.save_hyperparameters()
        self.net = nn.Sequential(
            nin_block(96, kernel_size=11, strides=4, padding=0),
            nn.MaxPool2d(3, stride=2),
            nin_block(256, kernel_size=5, strides=1, padding=2),
            nn.MaxPool2d(3, stride=2),
            nin_block(384, kernel_size=3, strides=1, padding=1),
            nn.MaxPool2d(3, stride=2),
            nn.Dropout(0.5),
            nin_block(num_classes, kernel_size=3, strides=1, padding=1),
            nn.AdaptiveAvgPool2d((1, 1)),
            nn.Flatten())
        self.net.apply(d2l.init_cnn)

NiN shape inspection

Spatial dims shrink, channels grow, and the final block already has one channel per class:

NiN().layer_summary((1, 1, 224, 224))

Sequential output shape:     torch.Size([1, 96, 54, 54])
MaxPool2d output shape:  torch.Size([1, 96, 26, 26])
Sequential output shape:     torch.Size([1, 256, 26, 26])
MaxPool2d output shape:  torch.Size([1, 256, 12, 12])
Sequential output shape:     torch.Size([1, 384, 12, 12])
MaxPool2d output shape:  torch.Size([1, 384, 5, 5])
Dropout output shape:    torch.Size([1, 384, 5, 5])
Sequential output shape:     torch.Size([1, 10, 5, 5])
AdaptiveAvgPool2d output shape:  torch.Size([1, 10, 1, 1])
Flatten output shape:    torch.Size([1, 10])

Training NiN

model = NiN(lr=0.05)
trainer = d2l.Trainer(max_epochs=10, num_gpus=1)
data = d2l.FashionMNIST(batch_size=128, resize=(224, 224))
model.apply_init([next(iter(data.get_dataloader(True)))[0]], d2l.init_cnn)
trainer.fit(model, data)

The averaging head costs nothing and does not hurt accuracy; that surprise made GAP the default head ever since.

GoogLeNet: go wide

GoogLeNet (Szegedy et al., 2015) won ImageNet 2014 with two lasting contributions:

• The stem / body / head decomposition of a network, still the universal vocabulary.
• The Inception block: don’t pick a filter size, run 1×1, 3×3, 5×5, and pooling in parallel and concatenate the channels.

Four branches, four scales, one shared input.

The Inception block in code

Four branches, channel-concatenated; 1×1 convs shrink channels before the costly 3×3 and 5×5:

class Inception(nn.Module):
    # c1--c4 are the number of output channels for each branch
    def __init__(self, c1, c2, c3, c4, **kwargs):
        super(Inception, self).__init__(**kwargs)
        # Branch 1
        self.b1_1 = nn.LazyConv2d(c1, kernel_size=1)
        # Branch 2
        self.b2_1 = nn.LazyConv2d(c2[0], kernel_size=1)
        self.b2_2 = nn.LazyConv2d(c2[1], kernel_size=3, padding=1)
        # Branch 3
        self.b3_1 = nn.LazyConv2d(c3[0], kernel_size=1)
        self.b3_2 = nn.LazyConv2d(c3[1], kernel_size=5, padding=2)
        # Branch 4
        self.b4_1 = nn.MaxPool2d(kernel_size=3, stride=1, padding=1)
        self.b4_2 = nn.LazyConv2d(c4, kernel_size=1)

    def forward(self, x):
        b1 = F.relu(self.b1_1(x))
        b2 = F.relu(self.b2_2(F.relu(self.b2_1(x))))
        b3 = F.relu(self.b3_2(F.relu(self.b3_1(x))))
        b4 = F.relu(self.b4_2(self.b4_1(x)))
        return torch.cat((b1, b2, b3, b4), dim=1)

Bottleneck arithmetic

First body block: 192 channels in, 64+128+32+32 = 256 out, spatial size unchanged.

The 5×5 branch, direct: 25 \cdot 192 \cdot 32 \approx 154k weights.

With a 16-channel 1×1 bottleneck first: 192 \cdot 16 + 25 \cdot 16 \cdot 32 \approx 16k, a 10× saving. This trick is in nearly every network since.

The whole network as data

Nine Inception blocks in three groups (2, 5, 2), pooling between groups. The hand-picked channel allocations are just a tuple; assembly is stem + body + head:

arch = (((64, (96, 128), (16, 32), 32), (128, (128, 192), (32, 96), 64)),
        ((192, (96, 208), (16, 48), 64), (160, (112, 224), (24, 64), 64),
         (128, (128, 256), (24, 64), 64), (112, (144, 288), (32, 64), 64),
         (256, (160, 320), (32, 128), 128)),
        ((256, (160, 320), (32, 128), 128), (384, (192, 384), (48, 128), 128)))

class GoogleNet(d2l.Classifier):
    def __init__(self, lr=0.1, num_classes=10):
        super().__init__()
        self.save_hyperparameters()
        pool = lambda: nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
        stem = nn.Sequential(
            nn.LazyConv2d(64, kernel_size=7, stride=2, padding=3), nn.ReLU(),
            pool(),
            nn.LazyConv2d(64, kernel_size=1), nn.ReLU(),
            nn.LazyConv2d(192, kernel_size=3, padding=1), nn.ReLU(), pool())
        body = [nn.Sequential(*[Inception(*c) for c in group])
                for group in arch]
        head = nn.Sequential(nn.AdaptiveAvgPool2d((1, 1)), nn.Flatten(),
                             nn.LazyLinear(num_classes))
        self.net = nn.Sequential(stem, body[0], pool(), body[1], pool(),
                                 body[2], head)
        self.net.apply(d2l.init_cnn)

Shape check

GoogleNet().layer_summary((1, 1, 96, 96))

Sequential output shape:     torch.Size([1, 192, 12, 12])
Sequential output shape:     torch.Size([1, 480, 12, 12])
MaxPool2d output shape:  torch.Size([1, 480, 6, 6])
Sequential output shape:     torch.Size([1, 832, 6, 6])
MaxPool2d output shape:  torch.Size([1, 832, 3, 3])
Sequential output shape:     torch.Size([1, 1024, 3, 3])
Sequential output shape:     torch.Size([1, 10])

Cheaper than VGG (~7M vs. ~138M parameters) and more accurate: the start of deliberate cost–accuracy design.

What survived

• Blocks (VGG): everything is specified block by block.
• 1×1 convolution (NiN): the standard channel mixer.
• Global average pooling (NiN): the default head.
• Multi-branch (GoogLeNet): survives as grouped convolutions (ResNeXt) and train-time-only branches (RepVGG).
• Inception-style hand-tuned branch cocktails: extinct.

Next ingredient: normalization.