23  Parallelism Beyond One Node

The reader ran DDP on four GPUs in Section 13.6 and derived the ZeRO stages without executing them; an 8B model’s 128 GB of training state fits no card, so the state is cut up and the pieces must talk, across machines, for weeks, while the hardware fails under them.