Hardware

Where the Two Numbers Come From

The roofline had two free parameters: peak compute P and bandwidth \beta. This section is where they come from — and why P/\beta has climbed over the long run.

• bytes live in a hierarchy: each level out is bigger and slower by orders of magnitude
• compute is area, I/O is perimeter
• arithmetic is (energetically) free; moving operands is the budget

Properties here; buying advice lives in the Tools appendix.

The Bandwidth Ladder

Every chip boundary ≈ one order of magnitude. Keep bytes home.

Latency: Eight Orders of Magnitude

The highlighted rung — 5–15 µs to launch a kernel — is why the overhead regime exists.

Measured, Not Asserted

A big elementwise op is a bandwidth meter — bytes known, arithmetic negligible:

x = jax.random.normal(jax.random.PRNGKey(0), (256 * 1024 * 1024,))  # 1 GB
t = d2l.Benchmark(lambda: x * 2.0).time
print(f'measured {2 * x.size * 4 / t / 1e12:.2f} TB/s '
      f'(spec: about 1.0 TB/s)')

measured 0.91 TB/s (spec: about 1.0 TB/s)

Streaming kernels land within tens of percent of spec.

The Shoreline

Long run, per generation: compute ~4×, bandwidth ~2×, capacity ~1.7× — the ridge point climbs, the model starves a little more. Any single step can buck it: B200 and RTX 5090 both bought bandwidth faster, and their ridge points fell. Packaging picks which wall moves; HBM-on-interposer is the countermeasure.

The Format Ladder

Same 8-bit exponent (fp32/tf32/bf16) ⇒ same range: swapping costs precision, not magnitude. Every halving of storage wins twice: 2× FLOP/s and half the bytes — tf32 buys throughput only, it still stores 32 bits.

Our Box vs. a Datacenter Rack

No P2P on GeForce: every inter-GPU byte staged through host DRAM — PCIe-limited, tens of GB/s for a raw copy (a couple for NCCL busbw at its default fallback — §13.5 finds the stage), flat from 2 to 4 GPUs. An NVL72 gives each GPU 1.8 TB/s. That gap is the multi-GPU chapter.

Energy, and One Model at Both Ends

One DRAM read ≈ 500 fp32 multiplies.

Prefill: weight reuse ~ context length ⇒ typically compute-bound. Decode: every token reads all weights ⇒ tokens/s ≲ β / model bytes. One model, both ends of the roofline.