21  Pretraining

The reader has trained the GPT of Section 11.2 for an afternoon on one GPU and read its loss curve. A real pretraining run lasts days, is interrupted and resumed, spikes at hour six, runs its matrix multiplications in eight bits, is continued on new data after it finishes, and is read almost entirely from its logs. This chapter builds the parts of that run that an afternoon never exercises, and trains the reference model that later chapters shard, fine-tune, quantize and serve.