3.2  The Image Classification Dataset

MNIST (LeCun et al. 1998) is a widely used image-classification benchmark with 70,000 handwritten-digit images (\(28 \times 28\) pixels, 10 classes). Many simple models now achieve high accuracy on it, which makes differences among model classes less visible.

We therefore use Fashion-MNIST (Xiao et al. 2017), a 2017 replacement with the same structure: 60,000 training and 10,000 test images, each containing \(28 \times 28\) grayscale pixels from one of 10 clothing categories. These categories provide a more discriminating comparison of the models developed in this chapter. ImageNet (Deng et al. 2009) supports larger-scale experiments with 1.2 million images and 1000 classes, but Fashion-MNIST keeps the examples interactive.

%matplotlib inline
import time
from d2l import torch as d2l
import torch
import torchvision
from torchvision import transforms

d2l.use_svg_display()
%matplotlib inline
import time
from d2l import tensorflow as d2l
import tensorflow as tf

d2l.use_svg_display()
%matplotlib inline
from d2l import jax as d2l
import jax
from jax import numpy as jnp
import numpy as np
import time
import tensorflow as tf

d2l.use_svg_display()
%matplotlib inline
import time
from d2l import mxnet as d2l
from mxnet import gluon, npx
from mxnet.gluon.data.vision import transforms
npx.set_np()

d2l.use_svg_display()

3.2.1 Loading the Dataset

The library provides a preprocessed version of Fashion-MNIST that we can download and read into memory.

class FashionMNIST(d2l.DataModule):
    """The Fashion-MNIST dataset."""
    def __init__(self, batch_size=64, resize=(28, 28)):
        super().__init__()
        self.save_hyperparameters()
        trans = transforms.Compose([transforms.Resize(resize),
                                    transforms.ToTensor()])
        self.train = torchvision.datasets.FashionMNIST(
            root=self.root, train=True, transform=trans, download=True)
        self.val = torchvision.datasets.FashionMNIST(
            root=self.root, train=False, transform=trans, download=True)
class FashionMNIST(d2l.DataModule):
    """The Fashion-MNIST dataset."""
    def __init__(self, batch_size=64, resize=(28, 28)):
        super().__init__()
        self.save_hyperparameters()
        self.train, self.val = tf.keras.datasets.fashion_mnist.load_data()
class FashionMNIST(d2l.DataModule):
    """The Fashion-MNIST dataset."""
    def __init__(self, batch_size=64, resize=(28, 28)):
        super().__init__()
        self.save_hyperparameters()
        self.train, self.val = tf.keras.datasets.fashion_mnist.load_data()
class FashionMNIST(d2l.DataModule):
    """The Fashion-MNIST dataset."""
    def __init__(self, batch_size=64, resize=(28, 28)):
        super().__init__()
        self.save_hyperparameters()
        trans = transforms.Compose([transforms.Resize(resize),
                                    transforms.ToTensor()])
        self.train = gluon.data.vision.FashionMNIST(
            train=True).transform_first(trans)
        self.val = gluon.data.vision.FashionMNIST(
            train=False).transform_first(trans)

Fashion-MNIST consists of images from 10 categories, each represented by 6000 images in the training dataset and by 1000 in the test dataset. A test dataset is used for evaluating model performance (it must not be used for training). Consequently the training set and the test set contain 60,000 and 10,000 images, respectively.

data = FashionMNIST(resize=(32, 32))
len(data.train), len(data.val)
(60000, 10000)
data = FashionMNIST(resize=(32, 32))
len(data.train[0]), len(data.val[0])
(60000, 10000)
data = FashionMNIST(resize=(32, 32))
len(data.train[0]), len(data.val[0])
(60000, 10000)
data = FashionMNIST(resize=(32, 32))
len(data.train), len(data.val)
(60000, 10000)

Fashion-MNIST images are natively \(28\times28\). We use resize=(32, 32) to match the spatial dimensions used by later convolution examples; interpolation changes the pixels and increases per-image storage and compute, but not the labels. Each image is delivered as a single-channel tensor of spatial size \(32 \times 32\). One subtlety matters here: where the channel axis lives. There are two conventions, channel-first \(c \times h \times w\) and channel-last \(h \times w \times c\). The get_dataloader method below produces the appropriate layout, which we confirm after constructing the loader. Here \(c = 1\) because the images are grayscale; most photographs have \(c = 3\).

The categories of Fashion-MNIST have human-understandable names. The following convenience method converts between numeric labels and their names.

@d2l.add_to_class(FashionMNIST)
def text_labels(self, indices):
    """Return text labels."""
    labels = ['t-shirt', 'trouser', 'pullover', 'dress', 'coat',
              'sandal', 'shirt', 'sneaker', 'bag', 'ankle boot']
    return [labels[int(i)] for i in indices]

3.2.2 Reading a Minibatch

We use the built-in data iterator to read the training and test sets. Recall that at each iteration, a data iterator reads a minibatch of data with size batch_size. We also randomly shuffle the examples for the training data iterator.

@d2l.add_to_class(FashionMNIST)
def get_dataloader(self, train):
    data = self.train if train else self.val
    return torch.utils.data.DataLoader(data, self.batch_size, shuffle=train,
                                       num_workers=self.num_workers)
@d2l.add_to_class(FashionMNIST)
def get_dataloader(self, train):
    data = self.train if train else self.val
    process = lambda X, y: (tf.expand_dims(X, axis=3) / 255,
                            tf.cast(y, dtype='int32'))
    resize_fn = lambda X, y: (tf.image.resize_with_pad(X, *self.resize), y)
    shuffle_buf = len(data[0]) if train else 1
    # `drop_remainder=train` keeps every training minibatch the same
    # shape so Keras `model.fit` / a `@tf.function`'d train-step compile
    # once and stop retracing for the smaller last batch (a major
    # speedup for HPO loops where a fresh model is fit per trial).
    return tf.data.Dataset.from_tensor_slices(process(*data)).shuffle(
        shuffle_buf).batch(self.batch_size,
                           drop_remainder=train).map(resize_fn)
@d2l.add_to_class(FashionMNIST)
def get_dataloader(self, train):
    data = self.train if train else self.val
    process = lambda X, y: (tf.expand_dims(X, axis=3) / 255,
                            tf.cast(y, dtype='int32'))
    resize_fn = lambda X, y: (tf.image.resize_with_pad(X, *self.resize), y)
    shuffle_buf = len(data[0]) if train else 1
    # `drop_remainder=train` keeps every training minibatch the same
    # shape, so JAX does not retrace the `@jax.jit`'d step function for
    # a smaller last batch.
    dataset = (tf.data.Dataset.from_tensor_slices(process(*data)).shuffle(
        shuffle_buf).batch(self.batch_size, drop_remainder=train).map(
            resize_fn))
    return d2l.TensorFlowDataLoader(dataset)
@d2l.add_to_class(FashionMNIST)
def get_dataloader(self, train):
    data = self.train if train else self.val
    return gluon.data.DataLoader(data, self.batch_size, shuffle=train,
                                 num_workers=self.num_workers)

We read one image to confirm the location of the channel axis.

X, y = next(iter(data.train_dataloader()))
X[0].shape  # channel-first: (channels, height, width)
torch.Size([1, 32, 32])
X, y = next(iter(data.train_dataloader()))
X[0].shape  # channel-last: (height, width, channels)
TensorShape([32, 32, 1])
X, y = next(iter(data.train_dataloader()))
X[0].shape  # channel-last: (height, width, channels)
(32, 32, 1)
X, y = next(iter(data.train_dataloader()))
X[0].shape  # channel-first: (channels, height, width)
(1, 32, 32)

Here is the same batch again, now at batch granularity: the first axis is the batch dimension (64 images per step by default), followed by the per-image shape we just confirmed, and the labels arrive as a matching vector of 64 integers.

X, y = next(iter(data.train_dataloader()))
print(X.shape, X.dtype, y.shape, y.dtype)
torch.Size([64, 1, 32, 32]) torch.float32 torch.Size([64]) torch.int64
(64, 32, 32, 1) <dtype: 'float32'> (64,) <dtype: 'int32'>
(64, 32, 32, 1) float32 (64,) int32
(64, 1, 32, 32) float32 (64,) int32

We time one full pass through the training set as a local loader check. The result depends on storage, worker count, framework, and hardware; it tells us whether loading is a bottleneck only for this run. If loading is slower than training on a target system, prefetching or additional loader workers may help.

tic = time.time()
for X, y in data.train_dataloader():
    continue
f'{time.time() - tic:.2f} sec'
'2.37 sec'
'0.71 sec'
'0.57 sec'
'3.03 sec'

3.2.3 Visualization

We will often be using the Fashion-MNIST dataset. The d2l library provides a convenience function show_images that lays out a list of images in a grid with optional per-image titles.

We visualize a minibatch before training. This check can reveal mislabeled examples, unexpected transformations, or shape errors early. Here are the images and their corresponding labels (in text) for the first few examples in the training dataset.

@d2l.add_to_class(FashionMNIST)
def visualize(self, batch, nrows=1, ncols=8, labels=None):
    X, y = batch
    if not labels:
        labels = self.text_labels(y)
    d2l.show_images(X.squeeze(1), nrows, ncols, titles=labels)
batch = next(iter(data.val_dataloader()))
data.visualize(batch)

@d2l.add_to_class(FashionMNIST)
def visualize(self, batch, nrows=1, ncols=8, labels=None):
    X, y = batch
    if not labels:
        labels = self.text_labels(y)
    d2l.show_images(tf.squeeze(X), nrows, ncols, titles=labels)
batch = next(iter(data.val_dataloader()))
data.visualize(batch)

@d2l.add_to_class(FashionMNIST)
def visualize(self, batch, nrows=1, ncols=8, labels=None):
    X, y = batch
    if not labels:
        labels = self.text_labels(y)
    d2l.show_images(jnp.squeeze(X), nrows, ncols, titles=labels)

batch = next(iter(data.val_dataloader()))
data.visualize(batch)

@d2l.add_to_class(FashionMNIST)
def visualize(self, batch, nrows=1, ncols=8, labels=None):
    X, y = batch
    if not labels:
        labels = self.text_labels(y)
    d2l.show_images(X.squeeze(1), nrows, ncols, titles=labels)
batch = next(iter(data.val_dataloader()))
data.visualize(batch)

We are now ready to work with the Fashion-MNIST dataset in the sections that follow.

3.2.4 Summary

Fashion-MNIST contains grayscale apparel images from 10 categories. A batch has a batch axis, two spatial axes, and one channel axis: PyTorch and MXNet use \((n,c,h,w)\), while TensorFlow and JAX use \((n,h,w,c)\). The visualization uses false color, but the underlying data have one grayscale channel. Loader throughput must be measured together with model computation on the target system; it is not guaranteed to be faster than training.

3.2.5 Exercises

  1. Time one full training epoch at batch_size of 1, 16, 64, 256, and 1024. Plot throughput (images per second) against batch_size. Why does throughput rise with batch size up to a point and then plateau?
  2. Set num_workers=0 (single-threaded loading) and compare against the default multi-worker setting. Under what conditions does increasing num_workers stop helping?
  3. PyTorch stores tensors in channel-first order \((c, h, w)\), while TensorFlow and JAX use channel-last \((h, w, c)\). Read the get_dataloader implementations for all four frameworks. Which step introduces the channel dimension, and where does the layout differ?