The NiN model

Network in Network (NiN)

NiN: MLPs inside convolutions

Network-in-Network (Lin et al., 2014) introduces two ideas the rest of the field happily adopts:

• 1×1 convolutions as a lightweight “MLP per pixel” — adds nonlinearity and channel mixing without spatial cost.
• Global average pooling replaces the giant FC classifier head — huge parameter reduction.

NiN: regular conv followed by two 1×1 convs; ends in global average pool.

The NiN block

A regular conv followed by two 1×1 convs (with ReLU between) — the “MLP within a conv layer”:

from d2l import jax as d2l
from flax import linen as nn
import jax
from jax import numpy as jnp

def nin_block(out_channels, kernel_size, strides, padding):
    return nn.Sequential([
        nn.Conv(out_channels, kernel_size, strides, padding),
        nn.relu,
        nn.Conv(out_channels, kernel_size=(1, 1)), nn.relu,
        nn.Conv(out_channels, kernel_size=(1, 1)), nn.relu])

Four NiN blocks at growing channel counts (96, 256, 384, num_classes), with max-pool downsampling between, then global average pooling + flatten → done. No FC layers.

import optax

class NiN(d2l.Classifier):
    lr: float = 0.1
    num_classes = 10
    training: bool = True

    def setup(self):
        self.net = nn.Sequential([
            nin_block(96, kernel_size=(11, 11), strides=(4, 4), padding=(0, 0)),
            lambda x: nn.max_pool(x, (3, 3), strides=(2, 2)),
            nin_block(256, kernel_size=(5, 5), strides=(1, 1), padding=(2, 2)),
            lambda x: nn.max_pool(x, (3, 3), strides=(2, 2)),
            nin_block(384, kernel_size=(3, 3), strides=(1, 1), padding=(1, 1)),
            lambda x: nn.max_pool(x, (3, 3), strides=(2, 2)),
            nn.Dropout(0.5, deterministic=not self.training),
            nin_block(self.num_classes, kernel_size=(3, 3), strides=(1, 1), padding=(1, 1)),
            lambda x: nn.avg_pool(x, window_shape=x.shape[1:3], strides=x.shape[1:3], padding='valid'),  # global avg pooling
            lambda x: x.reshape((x.shape[0], -1))  # flatten
        ])

    def configure_optimizers(self):
        return optax.sgd(self.lr)

Shape inspection

Walk a 1×1×224×224 input through; spatial dims shrink, channels grow until the final block produces num_classes channels:

NiN(training=False).layer_summary((1, 224, 224, 1))

Sequential output shape:     (1, 54, 54, 96)
function output shape:   (1, 26, 26, 96)
Sequential output shape:     (1, 26, 26, 256)
function output shape:   (1, 12, 12, 256)
Sequential output shape:     (1, 12, 12, 384)
function output shape:   (1, 5, 5, 384)
Dropout output shape:    (1, 5, 5, 384)
Sequential output shape:     (1, 5, 5, 10)
function output shape:   (1, 1, 1, 10)
function output shape:   (1, 10)

Training

Same Trainer, slightly higher learning rate than the FC nets (no dense layer to overfit on small batches):

model = NiN(lr=0.05)
trainer = d2l.Trainer(max_epochs=10, num_gpus=1)
data = d2l.FashionMNIST(batch_size=128, resize=(224, 224))
trainer.fit(model, data)

The important comparison is parameter economy: accuracy comes from richer convolutional blocks, not a large fully connected head.

Recap

• NiN puts an MLP inside each conv block via two 1×1 convs.
• Global average pooling as the classifier head — one number per class per feature map, no FC layers needed.
• The 1×1 conv as channel-mixer becomes a foundational primitive in all later architectures.
• Despite never winning a major benchmark, NiN’s ideas are in every ConvNet that came after.