The NiN model

Network in Network (NiN)

NiN: MLPs inside convolutions

Network-in-Network (Lin et al., 2014) introduces two ideas the rest of the field happily adopts:

• 1×1 convolutions as a lightweight “MLP per pixel” — adds nonlinearity and channel mixing without spatial cost.
• Global average pooling replaces the giant FC classifier head — huge parameter reduction.

NiN: regular conv followed by two 1×1 convs; ends in global average pool.

The NiN block

A regular conv followed by two 1×1 convs (with ReLU between) — the “MLP within a conv layer”:

import tensorflow as tf
from d2l import tensorflow as d2l

def nin_block(out_channels, kernel_size, strides, padding):
    return tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(out_channels, kernel_size, strides=strides,
                           padding=padding),
    tf.keras.layers.Activation('relu'),
    tf.keras.layers.Conv2D(out_channels, 1),
    tf.keras.layers.Activation('relu'),
    tf.keras.layers.Conv2D(out_channels, 1),
    tf.keras.layers.Activation('relu')])

Four NiN blocks at growing channel counts (96, 256, 384, num_classes), with max-pool downsampling between, then global average pooling + flatten → done. No FC layers.

class NiN(d2l.Classifier):
    def __init__(self, lr=0.1, num_classes=10):
        super().__init__()
        self.save_hyperparameters()
        self.net = tf.keras.models.Sequential([
            nin_block(96, kernel_size=11, strides=4, padding='valid'),
            tf.keras.layers.MaxPool2D(pool_size=3, strides=2),
            nin_block(256, kernel_size=5, strides=1, padding='same'),
            tf.keras.layers.MaxPool2D(pool_size=3, strides=2),
            nin_block(384, kernel_size=3, strides=1, padding='same'),
            tf.keras.layers.MaxPool2D(pool_size=3, strides=2),
            tf.keras.layers.Dropout(0.5),
            nin_block(num_classes, kernel_size=3, strides=1, padding='same'),
            tf.keras.layers.GlobalAvgPool2D(),
            tf.keras.layers.Flatten()])

Shape inspection

Walk a 1×1×224×224 input through; spatial dims shrink, channels grow until the final block produces num_classes channels:

NiN().layer_summary((1, 224, 224, 1))

Sequential output shape:     (1, 54, 54, 96)
MaxPooling2D output shape:   (1, 26, 26, 96)
Sequential output shape:     (1, 26, 26, 256)
MaxPooling2D output shape:   (1, 12, 12, 256)
Sequential output shape:     (1, 12, 12, 384)
MaxPooling2D output shape:   (1, 5, 5, 384)
Dropout output shape:    (1, 5, 5, 384)
Sequential output shape:     (1, 5, 5, 10)
GlobalAveragePooling2D output shape:     (1, 10)
Flatten output shape:    (1, 10)

Training

Same Trainer, slightly higher learning rate than the FC nets (no dense layer to overfit on small batches):

trainer = d2l.Trainer(max_epochs=10)
data = d2l.FashionMNIST(batch_size=128, resize=(224, 224))
with d2l.try_gpu():
    model = NiN(lr=0.05)
    trainer.fit(model, data)

The important comparison is parameter economy: accuracy comes from richer convolutional blocks, not a large fully connected head.

Recap

• NiN puts an MLP inside each conv block via two 1×1 convs.
• Global average pooling as the classifier head — one number per class per feature map, no FC layers needed.
• The 1×1 conv as channel-mixer becomes a foundational primitive in all later architectures.
• Despite never winning a major benchmark, NiN’s ideas are in every ConvNet that came after.