Shape inspection

Deep Convolutional Neural Networks (AlexNet)

AlexNet: scale changes vision

AlexNet (Krizhevsky, Sutskever, Hinton — 2012) is what made deep learning the approach to vision. Won ImageNet by a huge margin and started the modern era.

AlexNet alongside the LeNet from a decade earlier.

What changed from LeNet

• Bigger — 8 layers, 60 M parameters, larger first-layer filters (11×11), deeper feature stack.
• ReLU activations (no more saturating sigmoids).
• Dropout in the dense head for regularization.
• GPUs, ImageNet (1.2 M images), and augmentation — the missing ingredients.

The architecture itself is straightforward; what changed was the scale.

The architecture in code

Five conv layers (11×11 → 5×5 → three 3×3) + max-pool, then three FC layers down to 1000 classes:

from d2l import mxnet as d2l
from mxnet import np, init, npx
from mxnet.gluon import nn
npx.set_np()

class AlexNet(d2l.Classifier):
    def __init__(self, lr=0.1, num_classes=10):
        super().__init__()
        self.save_hyperparameters()
        self.net = nn.Sequential()
        self.net.add(
            nn.Conv2D(96, kernel_size=11, strides=4, activation='relu'),
            nn.MaxPool2D(pool_size=3, strides=2),
            nn.Conv2D(256, kernel_size=5, padding=2, activation='relu'),
            nn.MaxPool2D(pool_size=3, strides=2),
            nn.Conv2D(384, kernel_size=3, padding=1, activation='relu'),
            nn.Conv2D(384, kernel_size=3, padding=1, activation='relu'),
            nn.Conv2D(256, kernel_size=3, padding=1, activation='relu'),
            nn.MaxPool2D(pool_size=3, strides=2),
            nn.Dense(4096, activation='relu'), nn.Dropout(0.5),
            nn.Dense(4096, activation='relu'), nn.Dropout(0.5),
            nn.Dense(num_classes))
        self.net.initialize(init.Xavier())

Walk a single 1×1×224×224 image through and print each block’s output shape — the feature pyramid going from 224×224×1 down to 6×6×256:

AlexNet().layer_summary((1, 1, 224, 224))

Training on Fashion-MNIST

For demonstration, upsample the 28×28 Fashion-MNIST images to the 224×224 input AlexNet expects, then train at lr=0.01:

model = AlexNet(lr=0.01)
data = d2l.FashionMNIST(batch_size=128, resize=(224, 224))
trainer = d2l.Trainer(max_epochs=10, num_gpus=1)
trainer.fit(model, data)

Trains slowly even on a GPU — AlexNet has ~10× the parameters of LeNet. The architecture’s lasting contribution: it proved that bigger is better when paired with enough data and compute.

Recap

• AlexNet = LeNet’s recipe at 8× the depth, massive parameter count, ReLU, Dropout, GPU training, on ImageNet.
• Validates the “deeper, bigger, more data” formula that drives the field for the next decade.
• The next handful of architectures (VGG, GoogLeNet, ResNet) are systematic refinements of this template.