from d2l import torch as d2l
import time
import torch
torch.set_float32_matmul_precision('high')13.2 Hardware
The previous section handed us a model with two free parameters: a machine delivers \(\min(P, I\beta)\) — peak arithmetic \(P\) if you feed it enough work per byte, bandwidth \(\beta\) times intensity if you cannot. This section explains where those two numbers come from, why their ratio — the ridge point — has climbed over the long run of hardware generations, and what else about the machine (latency, capacity, interconnect, energy) a practitioner needs in order to reason about performance before running anything. The aim is working knowledge, not a computer architecture course: enough to read a profiler, size a model, and predict which fix will pay. For the complementary question — what hardware should I buy — see the buyer’s guide in Section 29.4; this section is about why the machine behaves the way it does, whatever machine you have. For a proper treatment of computer architecture see (Hennessy and Patterson 2011).
One warning before the numbers. Hardware numbers decay: the specific gigabytes per second below were verified in mid-2026 and will drift; every generation roughly doubles something. What does not decay are the ratios and their trends — memory is orders of magnitude slower than arithmetic, every chip boundary costs roughly another order of magnitude, and compute grows faster than bandwidth, which grows faster than capacity. Learn the ladder shapes; look up the rungs when you need them.
from d2l import jax as d2l
import jax
from jax import numpy as jnp
import numpy as np
import time13.2.1 Where Bytes Live
Every byte your program touches lives somewhere in a hierarchy (Figure 13.2.1), and each level trades capacity for speed: small pools close to the arithmetic are fast; large pools far away are slow. On one end sit the GPU’s on-die memories — registers and caches totaling on the order of a tenth of a gigabyte, reachable at tens of terabytes per second. Then comes the GPU’s own DRAM — the high-bandwidth memory (HBM) of datacenter parts or the GDDR of consumer cards — tens to hundreds of gigabytes at one to eight terabytes per second. Host DRAM holds hundreds of gigabytes at a few hundred gigabytes per second; NVMe flash holds terabytes at about fourteen gigabytes per second; and network storage offers effectively unbounded capacity, at whatever your network card delivers.
Two ladders quantify the hierarchy, and both are worth staring at. Figure 13.2.2 ranks the pipes by bandwidth: from an NVMe drive’s 14 GB/s, through PCIe and socket DRAM, up to on-package HBM at 8 TB/s. The span is almost three orders of magnitude, and the rungs correspond to physical boundaries: on-package memory is fastest because it sits on the same package, millimeters from the die over thousands of short interposer wires; everything that crosses a connector, a board trace, or a cable pays. The rule of thumb: every chip boundary costs roughly an order of magnitude of bandwidth. Keep the bytes home.
Figure 13.2.3 ranks the same world by latency — how long before the first byte arrives — and spans eight orders of magnitude, from a one-nanosecond L1 hit to a sixty-millisecond WAN round trip. Note where the highlighted rung sits: launching a CUDA kernel costs 5–15 µs of latency, thousands of L2 hits’ worth, which is why the overhead regime of Section 13.1 exists at all and why this chapter keeps returning to “fewer, larger operations”.
Latency and bandwidth interact through access patterns. DRAM delivers its rated bandwidth only for burst reads: addressing a location costs on the order of a hundred nanoseconds, after which sequential transfers stream at full rate — so the first word of a random read is hundreds of times more expensive than the words that follow it. Hardware therefore rewards streaming through memory in order and punishes pointer-chasing; caches amortize latency only when access patterns are local and predictable. For dense tensors this is mostly handled for you — a matmul kernel is a carefully choreographed burst-read machine — but it is why sparse and gather-heavy workloads struggle to reach more than a fraction of rated bandwidth, and why shuffled dataset access is best done at the granularity of large blocks. (The classic multi-core pathology of false sharing — two CPU threads bouncing one cache line between cores — is relegated to the exercises.)
Let’s measure the one number from this subsection that matters most for the roofline: our card’s achievable memory bandwidth. A large elementwise scaling reads and writes every element once — bytes moved are known exactly, arithmetic is negligible — so timing it is measuring bandwidth:
x = torch.randn(256 * 1024 * 1024, device=d2l.try_gpu()) # 1 GB fp32
t = d2l.Benchmark(lambda: x * 2.0).time
print(f'measured {2 * x.numel() * 4 / t / 1e12:.2f} TB/s '
f'(spec: about 1.0 TB/s)')measured 0.80 TB/s (spec: about 1.0 TB/s)x = jax.random.normal(jax.random.PRNGKey(0), (256 * 1024 * 1024,)) # 1 GB
t = d2l.Benchmark(lambda: x * 2.0).time
print(f'measured {2 * x.size * 4 / t / 1e12:.2f} TB/s '
f'(spec: about 1.0 TB/s)')measured 0.91 TB/s (spec: about 1.0 TB/s)A well-shaped streaming kernel lands within tens of percent of the specification — the spec number is real, not marketing. Remember what this kernel is: the bandwidth-bound regime made flesh. Every elementwise operation in every network you train runs at this speed, no matter how little it computes.
13.2.2 Why Compute Outruns Bandwidth
The ridge point of Section 13.1 is not an accident of one product; it is geometry. Arithmetic units fill the interior of a die: double the die’s side length (or halve the feature size) and you get four times the compute, because compute scales with area. But the wires that leave the chip — the I/O drivers that talk to memory — live on the die’s edge, and the edge only doubles. Compute scales as \(n^2\), I/O as \(n\) (Figure 13.2.4). Chip designers call the edge “beachfront”, and there is never enough of it: with every generation, bytes-per-FLOP falls, and the model starves a little more.
HBM is the countermeasure: stack DRAM dies vertically and place the stack on a silicon interposer millimeters from the GPU die, so thousands of short, dense wires replace the few hundred long board traces a socketed DIMM gets. That is how a B200 reaches nearly 8 TB/s while a CPU socket manages 0.5 TB/s — not faster DRAM cells, but vastly more, vastly shorter wires. The second countermeasure is size: dies are manufactured against a hard lithographic ceiling (the reticle limit, about \(858\,\textrm{mm}^2\)), and flagship accelerators live at it — a B200 is two reticle-limit dies bridged into one logical GPU. When you cannot grow the die, you bridge dies; when you cannot bridge, you network — the interconnect story of Section 13.2.5 is the shoreline problem continued by other means.
The long-run trend to memorize, averaged over many hardware generations at fixed precision: compute grows about \(4\times\) per generation, bandwidth about \(2\times\), capacity about \(1.7\times\). Over the years the ridge point — compute over bandwidth — has therefore risen, and workloads that were compute-bound a decade ago are bandwidth-bound today. Do not expect every product step to obey the average, though: which roof moves faster in any single generation is a packaging and market decision, and it can go the other way. The H100\(\to\)B200 step grew compute and bandwidth almost in lockstep (both about \(2.3\times\) — the ridge point barely moved, in fact edging slightly down), and the consumer 4090$$5090 step bought bandwidth far faster than compute, dropping its ridge point by nearly a third — check the ridge row of Table 13.2.1 below. The shoreline sets the long-run pressure; engineering chooses, one generation at a time, which wall to push. Averaged over the decade, compute wins — which is why a chapter about performance is mostly a chapter about bytes, and why it will remain one.
13.2.3 The GPU
A GPU spends its area on thousands of simple arithmetic units rather than on the deep control logic of a CPU core. The units are organized into streaming multiprocessors (SMs) — our RTX 4090 has 128 of them — and each SM executes threads in lockstep groups of 32 called warps, swapping between many resident warps to hide memory latency. That is as much microarchitecture as this book needs: when a profiler reports “occupancy” it means resident warps available for latency-hiding; when a kernel is “too small for the machine” it means too few warps to keep 128 SMs busy — the launch-overhead regime again.
What makes GPUs deep learning machines is a further specialization: tensor cores, silicon shaped exactly like a small matrix multiply-accumulate. A scalar unit spends most of its energy on instruction handling per multiply; a tensor core amortizes that control over an entire \(16\times 16\)-ish tile, executing it as one instruction — the same insight as the TPU’s systolic array (Jouppi et al. 2017; Kung 1988). The consequence: matrix multiplications run an order of magnitude faster than anything else on the chip, provided the data comes in the formats the tensor cores speak. Those formats are the ladder in Figure 13.2.5.
Two rules organize the ladder. First, the exponent width sets the range, the mantissa width sets the precision. fp32, tf32, and bf16 all carry the same 8-bit exponent: they represent the same span of magnitudes, so swapping among them costs precision, not range — you give up decimal places, which deep learning mostly tolerates, rather than the ability to represent a large activation at all. fp16’s 5-bit exponent trades range for precision, which is why it needs the loss-scaling machinery we will meet in Section 13.4, and why bf16 has become the training default. The sub-byte formats — fp8 in two flavors (E4M3 for activations and weights, wider-range E5M2 for gradients, in the usual recipe rather than by any rule) and fp4 with just sixteen representable values — push the same trade to its limit and lean on per-block scaling factors to survive (Micikevicius et al. 2022). Second, every halving of storage wins twice: half the bits means double the peak FLOP/s (twice the tile fits in the same silicon) and half the bytes per operand — the format ladder climbs the roofline along both axes at once. tf32 is the exception that proves the byte rule: it is a tensor-core compute format that still stores 32 bits (the figure draws exactly this), so it buys throughput but saves nothing on memory traffic. The catch: each halving buys about \(2\times\) on real silicon, not the \(4\times\) the marketing arithmetic suggests, and the big jumps between generations come from new architectures, not formats alone.
The numbers-dense table, for orientation (conventions below it):
| H100 SXM | B200 SXM | RTX 4090 (ours) | RTX 5090 | |
|---|---|---|---|---|
| memory | 80 GB HBM3 | 180 GB HBM3e | 24 GB GDDR6X | 32 GB GDDR7 |
| bandwidth | 3.35 TB/s | 7.7 TB/s | 1.0 TB/s | 1.8 TB/s |
| bf16 dense | 989 TF | 2,250 TF | 165 TF | 210 TF |
| fp8 dense | 1,979 TF | 4,500 TF | 330 TF | 419 TF |
| fp4 dense | — | 9,000 TF | — | 1,676 TF |
| ridge (bf16) | ~295 FLOP/B | ~292 FLOP/B | ~165 FLOP/B | ~117 FLOP/B |
| power | 700 W | 1,000 W | 450 W | 575 W |
Conventions: dense throughput (no 2:4 sparsity), fp32 accumulation, boost clocks, vendor datasheets as of mid-2026. The B200 column is the shipping HGX B200 spec; the GB200 NVL72 variant of the same silicon runs 186 GB at 8 TB/s and up to 1,200 W. The 5090’s fp4 entry is \(4\times\) its fp8 entry, not \(2\times\), because GeForce parts run fp8/fp16 tensor math at half rate when accumulating in fp32 while fp4 carries no such penalty.
The same constraints produce the same shape at every vendor — high-bandwidth stacked memory next to a die full of matrix engines: AMD’s MI355X (288 GB HBM3e at 8 TB/s, 2,500 TF dense bf16), Google’s TPU v7 (192 GB at 7.4 TB/s, 2,307 TF), Amazon’s Trainium2 (96 GB at 2.9 TB/s, 667 TF). Different silicon, one design point; the roofline reasoning of this chapter applies unchanged to all of them.
One more distinction the table hides: training hardware must hold activations for the backward pass and accumulate gradients robustly (bf16 with fp32 accumulation as the floor — Section 13.4), while inference can run forward-only in the smallest format quality allows. That asymmetry, plus capacity — note that memory capacity grew the slowest of the three curves — shapes the memory anatomy we build in Section 13.4.
13.2.4 The CPU’s Role
The CPU in a deep learning machine is no longer where the FLOPs happen — one RTX 4090 out-multiplies a 64-core CPU by two orders of magnitude — but three jobs still live or die on it. First, the CPU is the orchestrator: every kernel the GPU runs was launched by a CPU thread at 5–15 µs apiece (Figure 13.2.3) — until the capture-and-replay of Section 13.3 takes the CPU out of that loop — which is exactly the Python-must-stay-ahead story of Section 13.1. Second, it runs the input pipeline: decoding JPEGs, tokenizing text, augmenting, batching — if those fall behind, the GPU starves no matter how fast it is (a dataloader-starved profile is the overhead regime with extra steps). Third, it feeds the bus: host-to-device copies cross PCIe, and how you stage the host memory matters. A regular (pageable) host tensor must first be copied into a locked staging buffer before DMA can move it; allocating the tensor in pinned (page-locked) memory skips that step and, as a bonus, allows the copy to proceed asynchronously alongside compute. How much the staging costs — and who controls it — differs between our two frameworks:
x_pageable = torch.randn(64 * 1024 * 1024) # 256 MB on the host
x_pinned = x_pageable.pin_memory()
gpu = d2l.try_gpu()
for desc, src in [('pageable', x_pageable), ('pinned', x_pinned)]:
t = d2l.Benchmark(lambda: src.to(gpu, non_blocking=True),
desc=f'{desc} H2D').time
print(f'{desc}: {x_pageable.numel() * 4 / t / 1e9:.1f} GB/s')pageable: 13.4 GB/s
pinned: 24.3 GB/s# JAX manages host staging internally; measure the achieved H2D rate.
x_host = np.random.randn(64 * 1024 * 1024).astype(np.float32) # 256 MB
t = d2l.Benchmark(lambda: jax.device_put(x_host), desc='H2D').time
print(f'device_put: {x_host.nbytes / t / 1e9:.1f} GB/s')device_put: 0.9 GB/sPinned memory roughly doubles the pageable rate, and the pinned figure sits near PCIe’s practical ceiling — tens of GB/s, a factor of about forty below the GPU’s own memory. This is why the standard input pipeline asks for both pieces at once: DataLoader(pin_memory=True) to stage batches in pinned memory, .to(device, non_blocking=True) to overlap the copy with compute.
The number deserves a hard look: the wire underneath is capable of tens of GB/s, yet jax.device_put from a pageable NumPy array lands far below the bus limit — around one gigabyte per second at our pin. The copy is not the cost; JAX’s transfer path (validation, layout, staging through its own host buffer) is, and that is itself the lesson: a transfer runs at the speed of its slowest stage, and the framework owns some of the stages. JAX programs sidestep the problem structurally rather than tuning it — keep arrays resident on the device, let the input pipeline overlap transfers with compute, and treat a per-step device_put of fresh host data as a red flag.
Either way the conclusion is structural: get data onto the device, keep it there, and overlap the unavoidable transfers with compute. This is also your first sighting of the number that will dominate the multi-GPU sections: everything that leaves the GPU moves at bus speed — at best — not memory speed.
13.2.5 Interconnects: Our Box as the Worked Example
When one GPU is not enough, GPUs must talk to each other, and the bandwidth ladder says this is where performance goes to die. Datacenter parts attack the problem with dedicated fabrics: NVLink gives each B200 1.8 TB/s to its peers — memory-class bandwidth, one rung below HBM — and an NVL72 rack wires 72 GPUs into a single 130 TB/s domain. Consumer parts get PCIe, and — the fact that shapes the rest of this chapter — the current GeForce generations (RTX 40 and 50 series) have peer-to-peer transfers disabled as market segmentation: two RTX 4090s in one box may not even talk PCIe-directly to each other. (It was not always so — the RTX 3090 still carried an NVLink bridge; its successors dropped it.)
Our build machine makes the consequence concrete (Figure 13.2.6). Its four RTX 4090s hang in pairs off two PCIe host bridges; with P2P disabled, every byte from one GPU to another is staged through host DRAM — up one PCIe link, through the CPU’s memory system, down another. Measured end to end (Section 13.5 does the measuring), a large GPU-to-GPU copy sustains roughly twenty GB/s — PCIe-limited, two orders of magnitude below an NVL72’s ~1.8 TB/s per GPU — and a collective library like NCCL, whose ring/tree chunking assumes a peer-to-peer fabric, extracts even less (a couple of GB/s of effective bus bandwidth) on this staged path — not because the wire slows down for NCCL, but because its P2P-less fallback moves bytes with a latency-bound mechanism by default; Section 13.5 diagnoses this and names the one-line workaround, which Section 13.6 weighs. Either way the staging path, not the link count, sets the ceiling, so the rate does not improve from two GPUs to four.
nvidia-smi topo: four RTX 4090s in pairs behind two PCIe host bridges, no P2P, no NVLink. The highlighted path is what every inter-GPU byte actually travels.
We treat this as a teaching instrument, not an embarrassment. Most readers’ multi-GPU machines look like ours, not like an NVL72; and a slow fabric makes the accounting of parallel training vivid — on this box, communication costs are impossible to ignore, so the cost model of Section 13.5 predicts real behavior instead of disappearing into NVLink headroom. Why datacenter fabrics exist will not need to be asserted; we will have measured it.
13.2.6 Energy: Why Moving Bytes Is the Budget
There is a final ladder underneath the other two. Figure 13.2.7 shows the energy cost of elementary operations at a fixed process node (Horowitz 2014): an 8-bit integer add costs 0.03 pJ, an fp32 multiply 3.7 pJ — and one 64-bit read from DRAM costs about 2,000 pJ. One memory access buys roughly five hundred multiplies. Arithmetic is nearly free; fetching operands is the budget, and a chip’s power limit is in the end an energy-per-second limit. This is the deepest version of the ridge-point story: the reason compute outruns bandwidth is not just wire count but joules — you can afford to place more multipliers, but you cannot afford to feed them from far away.
The practical corollaries are the fixes this chapter teaches: fusion (Section 13.3) saves energy, not just time, by keeping intermediates on-die; low-precision formats (Section 13.4) halve the bytes and hence nearly halve the energy per operand; recomputation trades cheap arithmetic for expensive memory traffic. When a technique in this chapter feels like a trick, check it against Figure 13.2.7 — nearly all of them are the same move: do more arithmetic per byte fetched.
13.2.7 Reading the Roofline: Two Workloads
Close the loop by reading one model’s two working regimes straight off the hardware numbers — the exercise this section exists to enable. Consider a language model of the kind built in Section 11.2, with \(N\) parameters in bf16 (\(2N\) bytes of weights), serving a prompt and then generating.
Prefill — processing the prompt — pushes the whole context through the network at once. Each weight fetched from memory is reused across every token in the context, so arithmetic intensity is on the order of the context length: hundreds to tens of thousands of FLOPs per byte, far above any ridge point. A long, well-batched prefill is therefore typically compute-bound: it runs at the tensor cores’ pace and benefits from every format halving. (A one-line prompt is not — intensity needs context to reuse each weight against.)
Decode — generating one token at a time — is the opposite. Producing a single token performs about \(2N\) FLOPs but must read all \(2N\) weight bytes (plus the KV cache of Section 11.3): intensity \(\approx 1\) FLOP/byte, the bottom of the roofline. Decode is bandwidth-bound, and its speed limit is arithmetic you can do on a napkin:
\[ \textrm{tokens/s} \;\lesssim\; \frac{\beta}{\textrm{bytes per token}} \;=\; \frac{\beta}{2N + \textrm{KV bytes}}. \]
An 8-billion-parameter model in bf16 reads at least 16 GB per generated token; on our 1 TB/s card that caps generation near sixty tokens per second regardless of compute, while the same card’s prefill chews through the prompt at tensor-core speed. One model, both ends of Figure 13.1.1 — and the reason batching multiple generation streams (which shares each weight read across streams, multiplying intensity by the batch size) is the central trick of inference serving. The systems that exploit this — batching schedulers, paged caches, speculative decoding — belong to the Language Models part; the economics fit in the equation above.
Rules of thumb worth carrying out of this section:
| quantity | magnitude | why it matters |
|---|---|---|
| kernel launch | 5–15 µs | small ops cannot feed the GPU (Section 13.3) |
| GPU memory bandwidth | 1–8 TB/s | the decode/elementwise speed limit |
| PCIe per direction | tens of GB/s | host↔︎device; get data on-device, keep it there |
| NVLink-class fabric | ~1.8 TB/s per GPU | why datacenter multi-GPU scales and PCIe boxes struggle |
| bytes/param, training with Adam | ~16–20 (mixed precision) | memory anatomy of Section 13.4 |
| bytes/token, decode | ~2 × params + KV | tokens/s ≤ bandwidth / this |
| DRAM read vs fp32 multiply | ~500× the energy | fuse, shrink formats, recompute |
13.2.8 Summary
- Memory is a hierarchy: each level away from the arithmetic holds orders of magnitude more bytes and delivers orders of magnitude fewer per second, with a latency ladder spanning eight decades. Bandwidth is earned by streaming; boundaries cost an order of magnitude each.
- Compute scales with die area, I/O with die perimeter. The long-run outcome is compute growing ~4× per generation against bandwidth’s ~2× and capacity’s ~1.7×, so the ridge point has risen over the years — though a single product step can move it either way (B200 and the RTX 5090 both nudged it down by buying bandwidth). HBM-on-interposer is a countermeasure, not a repeal.
- Tensor cores make matrix arithmetic an order of magnitude faster than everything else, in exchange for format discipline: same exponent = same range; every halving of width doubles FLOP/s and halves bytes.
- The CPU orchestrates (kernel launches), feeds (input pipeline), and stages (pinned-memory PCIe copies). Inter-GPU bandwidth on consumer boxes is host-staged and PCIe-limited — tens of GB/s for a raw copy, and only a couple of GB/s of NCCL busbw at its default fallback settings (Section 13.5 finds the binding stage) — one to three orders of magnitude below an NVLink domain, which is why datacenter fabrics exist.
- Energy explains everything twice: a DRAM access costs ~500 multiplies, so every technique in this chapter is a variation on “more arithmetic per byte fetched”.
- One model can live at both ends of the roofline: long-context prefill is typically compute-bound, single-stream decode is bandwidth-bound with tokens/s ≲ bandwidth / model bytes.
13.2.9 Exercises
- Compute the ridge point for each GPU in Table 13.2.1 at bf16 and at fp8. Which direction did it move from the H100 to the B200, and from the RTX 4090 to the RTX 5090? Reconcile what you find with the shoreline argument of Section 13.2.2: what did each step’s packaging buy, and why does the long-run direction still differ from these two steps?
- Take the GPT of Section 11.2 (count its parameters) and your own GPU’s specifications. Estimate (a) the prefill arithmetic intensity at context length 128 and (b) the decode tokens-per-second bound. Which regime is each in?
- Using Figure 13.2.7, estimate the energy spent on DRAM traffic alone in one epoch of the Fashion-MNIST training loop of Section 6.6 (count the bytes your activations and weights move). At $0.30/kWh, what does the epoch’s memory traffic cost?
- Measure the cache cliff on your GPU: time
x * 2.0for tensor sizes from 1 MB to 2 GB and plot achieved bandwidth against size. Where do the level transitions of Figure 13.2.1 appear? - Why is HBM placed on a silicon interposer rather than on the motherboard next to the CPU’s DRAM slots? Answer using the shoreline argument, then check: how many signal wires does a DIMM socket carry, versus an HBM stack’s interface?
- Burst versus random access: allocate a large tensor and compare summing it in natural order against summing it through a random permutation of indices. Explain the ratio using the address-setup-versus-streaming model of
- (False sharing.) Write a two-thread CPU program in which each thread increments its own counter, first with the counters adjacent in memory, then padded to separate cache lines. Measure, and explain why the “shared-nothing” version can be many times faster despite identical arithmetic.
- Your DataLoader delivers batches at 300 MB/s from NVMe while your GPU consumes them at 2 GB/s. Using the ladders in this section, list the three cheapest interventions in order of expected payoff.