Pretraining BERT

With the model (last deck) and the data (deck before that), we can finally pretrain a small BERT end-to-end. This deck does it on a tiny scale: 2 layers, 128 hidden dim, 2 heads. The recipe scales to BERT-Base (12 layers, 768 dim, 12 heads) and BERT-Large by just changing the config.

Load pretraining data

Each batch supplies tokens, segment IDs, valid lengths, masked positions/labels, MLM weights, and NSP labels:

from d2l import jax as d2l
import jax
from jax import numpy as jnp
from flax import nnx
import optax
import numpy as np

Tiny BERT config

The notebook uses a deliberately small encoder so the full pretraining loop is runnable in class:

batch_size, max_len = 512, 64
# In JAX, train_iter is a callable returning a fresh iterator each call.
train_iter, vocab = d2l.load_data_wiki(batch_size, max_len)

Trainer setup

Initialize the optimizer/trainer for this tiny BERT. Scaling to BERT-Base changes only the data size, model width/depth, and compute budget:

net = d2l.BERTModel(len(vocab), num_hiddens=128,
                    ffn_num_hiddens=256, num_heads=2, num_blks=2, dropout=0.2)

Combined loss

Two heads, one combined loss:

\mathcal{L} = \mathcal{L}_\text{MLM} + \mathcal{L}_\text{NSP}.

MLM cross-entropy averaged over masked positions; NSP binary cross-entropy on the <cls> head:

def _get_batch_loss_bert(net, vocab_size, tokens_X,
                         segments_X, valid_lens_x,
                         pred_positions_X, mlm_weights_X,
                         mlm_Y, nsp_y):
    # Forward pass
    _, mlm_Y_hat, nsp_Y_hat = net(tokens_X, segments_X,
                                  valid_lens_x.reshape(-1),
                                  pred_positions_X)
    # Compute masked language model loss
    mlm_l = optax.softmax_cross_entropy_with_integer_labels(
        mlm_Y_hat.reshape(-1, vocab_size), mlm_Y.reshape(-1))
    mlm_l = (mlm_l * mlm_weights_X.reshape(-1)).sum() / (
        mlm_weights_X.sum() + 1e-8)
    # Compute next sentence prediction loss
    nsp_l = optax.softmax_cross_entropy_with_integer_labels(
        nsp_Y_hat, nsp_y).mean()
    l = mlm_l + nsp_l
    return l, (mlm_l, nsp_l)

Training loop

Standard SGD with warmup; on this tiny corpus a few hundred steps is enough to see both losses drop. MLM loss stays higher than NSP because it predicts a large vocabulary rather than a binary label:

def train_bert(train_iter, net, vocab_size, num_steps):
    optimizer = nnx.Optimizer(net, optax.adam(learning_rate=1e-4),
                              wrt=nnx.Param)

    @nnx.jit
    def train_step(net, optimizer, batch):
        def loss_fn(model):
            return _get_batch_loss_bert(model, vocab_size, *batch)
        (l, (mlm_l, nsp_l)), grads = nnx.value_and_grad(
            loss_fn, has_aux=True)(net)
        optimizer.update(net, grads)
        return l, mlm_l, nsp_l

    step, timer = 0, d2l.Timer()
    animator = d2l.Animator(xlabel='step', ylabel='loss',
                            xlim=[1, num_steps], legend=['mlm', 'nsp'])
    # Sum of masked language modeling losses, sum of next sentence prediction
    # losses, no. of sentence pairs, count
    metric = d2l.Accumulator(4)
    num_steps_reached = False
    while step < num_steps and not num_steps_reached:
        # train_iter is a callable: invoke it each epoch for a fresh iterator.
        for (tokens_X, segments_X, valid_lens_x, pred_positions_X,
            mlm_weights_X, mlm_Y, nsp_y) in train_iter():
            timer.start()
            batch = (tokens_X, segments_X, valid_lens_x, pred_positions_X,
                     mlm_weights_X, mlm_Y, nsp_y)
            _, mlm_l, nsp_l = train_step(net, optimizer, batch)
            metric.add(float(mlm_l), float(nsp_l), tokens_X.shape[0], 1)
            timer.stop()
            animator.add(step + 1,
                         (metric[0] / metric[3], metric[1] / metric[3]))
            step += 1
            if step == num_steps:
                num_steps_reached = True
                break

    print(f'MLM loss {metric[0] / metric[3]:.3f}, '
          f'NSP loss {metric[1] / metric[3]:.3f}')
    print(f'{metric[2] / timer.sum():.1f} sentence pairs/sec on '
          f'{str(jax.devices())}')
    return net

net = train_bert(train_iter, net, len(vocab), 50)

MLM loss 9.665, NSP loss 0.712
523.6 sentence pairs/sec on [CudaDevice(id=0)]

Using the trained encoder

After pretraining, the encoder is the useful part — turn token sequences into contextual representations. The pretraining heads can be discarded for most downstream tasks:

def get_bert_encoding(net, tokens_a, tokens_b=None):
    tokens, segments = d2l.get_tokens_and_segments(tokens_a, tokens_b)
    token_ids = jnp.array(vocab[tokens], dtype=jnp.int32)[None, :]
    segments = jnp.array(segments, dtype=jnp.int32)[None, :]
    valid_len = jnp.array([len(tokens)], dtype=jnp.float32)
    encoded_X, _, _ = nnx.view(net, deterministic=True)(
        token_ids, segments, valid_len)
    return encoded_X

Single sentence

“a crane is flying” → 6 hidden vectors (one per token, including <cls> and <sep>). Each is contextual — the representation of “crane” depends on its neighbors:

tokens_a = ['a', 'crane', 'is', 'flying']
encoded_text = get_bert_encoding(net, tokens_a)
# Tokens: '<cls>', 'a', 'crane', 'is', 'flying', '<sep>'
encoded_text_cls = encoded_text[:, 0, :]
encoded_text_crane = encoded_text[:, 2, :]
encoded_text.shape, encoded_text_cls.shape, encoded_text_crane[0][:3]

((1, 6, 128),
 (1, 128),
 Array([-0.48675025, -1.4178457 , -0.3135721 ], dtype=float32))

Sentence pair

“a crane driver came” / “he just left”. Same encoder, two-segment input — segment IDs distinguish the two halves inside the same sequence:

tokens_a, tokens_b = ['a', 'crane', 'driver', 'came'], ['he', 'just', 'left']
encoded_pair = get_bert_encoding(net, tokens_a, tokens_b)
# Tokens: '<cls>', 'a', 'crane', 'driver', 'came', '<sep>', 'he', 'just',
# 'left', '<sep>'
encoded_pair_cls = encoded_pair[:, 0, :]
encoded_pair_crane = encoded_pair[:, 2, :]
encoded_pair.shape, encoded_pair_cls.shape, encoded_pair_crane[0][:3]

((1, 10, 128),
 (1, 128),
 Array([-0.70371836, -0.90896857, -0.1139093 ], dtype=float32))

Recap

BERT pretraining is just two losses (MLM + NSP) optimized end-to-end on the encoder + heads.
Output of pretraining: a contextual token encoder.
For downstream tasks: load encoder weights, attach a small head, fine-tune. The next chapter does exactly this for sentiment classification, NLI, and SQuAD-style QA.