Hardware

Where the Two Numbers Come From

The roofline had two free parameters: peak compute P and bandwidth \beta. This section is where they come from — and why P/\beta has climbed over the long run.

• bytes live in a hierarchy: each level out is bigger and slower by orders of magnitude
• compute is area, I/O is perimeter
• arithmetic is (energetically) free; moving operands is the budget

Properties here; buying advice lives in the Tools appendix.

Bandwidth Across the Memory Hierarchy

Every chip boundary reduces bandwidth by roughly one order of magnitude. Keep frequently used data near the device.

Latency: Eight Orders of Magnitude

The highlighted range — 5–15 µs to launch a kernel — is why the overhead regime exists.

Measured Hardware Characteristics

A big elementwise op is a bandwidth meter — bytes known, arithmetic negligible:

x = torch.randn(256 * 1024 * 1024, device=d2l.try_gpu())  # 1 GB fp32
t = d2l.Benchmark(lambda: x * 2.0).time
print(f'measured {2 * x.numel() * 4 / t / 1e12:.2f} TB/s '
      f'(spec: about 1.0 TB/s)')

measured 0.92 TB/s (spec: about 1.0 TB/s)

Streaming kernels land within tens of percent of spec.

Compute and I/O Scaling

Over many generations, compute grows by about 4×, bandwidth by 2×, and capacity by 1.7×, so the ridge point tends to increase. Any single step can buck it: B200 and RTX 5090 both bought bandwidth faster, and their ridge points fell. Packaging picks which wall moves; HBM-on-interposer is the countermeasure.

Floating-Point Format Comparison

Same 8-bit exponent (fp32/tf32/bf16) ⇒ same range: swapping costs precision, not magnitude. Every halving of storage wins twice: 2× FLOP/s and half the bytes — tf32 buys throughput only, it still stores 32 bits.

Our Box vs. a Datacenter Rack

No P2P on GeForce: every inter-GPU byte staged through host DRAM — PCIe-limited, tens of GB/s for a raw copy (a couple for NCCL busbw at its default fallback — §13.5 finds the stage), flat from 2 to 4 GPUs. An NVL72 gives each GPU 1.8 TB/s. That gap is the multi-GPU chapter.

Energy, and One Model at Both Ends

One DRAM read ≈ 500 fp32 multiplies.

Prefill: weight reuse ~ context length ⇒ typically compute-bound. Decode: every token reads all weights ⇒ tokens/s ≲ β / model bytes. One model, both ends of the roofline.