{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=hardware-md-1 fw=all -->\n",
    "\n",
    "# Hardware\n",
    "\n",
    "The previous section handed us a model with two free parameters: a machine\n",
    "delivers $\\min(P, I\\beta)$ — peak arithmetic $P$ if you feed it enough work\n",
    "per byte, bandwidth $\\beta$ times intensity if you cannot. This section\n",
    "explains where those two numbers come from, why their ratio — the ridge\n",
    "point — has climbed over the long run of hardware generations, and what else about\n",
    "the machine (latency, capacity, interconnect, energy) a practitioner needs\n",
    "in order to reason about performance before running anything. The aim is\n",
    "working knowledge, not a computer architecture course: enough to read a\n",
    "profiler, size a model, and predict which fix will pay. For the\n",
    "complementary question — *what hardware should I buy* — see the buyer's\n",
    "guide in that section; this section is about why the\n",
    "machine behaves the way it does, whatever machine you have. For a proper\n",
    "treatment of computer architecture see [@Hennessy.Patterson.2011].\n",
    "\n",
    "One warning before the numbers. Hardware numbers decay: the specific\n",
    "gigabytes per second below were verified in mid-2026 and will drift; every\n",
    "generation roughly doubles something. What does not decay are the *ratios\n",
    "and their trends* — memory is orders of magnitude slower than arithmetic,\n",
    "every chip boundary costs roughly another order of magnitude, and compute\n",
    "grows faster than bandwidth, which grows faster than capacity. Learn the\n",
    "ladder shapes; look up the rungs when you need them."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "hardware"
   },
   "source": [
    "from d2l import jax as d2l\n",
    "import jax\n",
    "from jax import numpy as jnp\n",
    "import numpy as np\n",
    "import time"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=hardware-md-2 fw=all -->\n",
    "\n",
    "## Where Bytes Live\n",
    "\n",
    "Every byte your program touches lives somewhere in a hierarchy\n",
    "(the figure), and each level trades capacity for speed:\n",
    "small pools close to the arithmetic are fast; large pools far away are\n",
    "slow. On one end sit the GPU's on-die memories — registers and caches\n",
    "totaling on the order of a tenth of a gigabyte, reachable at tens of\n",
    "terabytes per second. Then comes the GPU's own DRAM — the *high-bandwidth\n",
    "memory* (HBM) of datacenter parts or the GDDR of consumer cards — tens to\n",
    "hundreds of gigabytes at one to eight terabytes per second. Host DRAM\n",
    "holds hundreds of gigabytes at a few hundred gigabytes per second; NVMe\n",
    "flash holds terabytes at about fourteen gigabytes per second; and network\n",
    "storage offers effectively unbounded capacity, at whatever your network\n",
    "card delivers.\n",
    "\n",
    "![The memory hierarchy in 2026. Each level downward holds orders of\n",
    "magnitude more bytes and delivers orders of magnitude fewer of them per\n",
    "second.](../img/mdl-perf-memory-hierarchy.svg)\n",
    "\n",
    "Two ladders quantify the hierarchy, and both are worth staring at.\n",
    "the figure ranks the *pipes* by bandwidth: from an\n",
    "NVMe drive's 14 GB/s, through PCIe and socket DRAM, up to on-package HBM\n",
    "at 8 TB/s. The span is almost three orders of magnitude, and the rungs\n",
    "correspond to physical boundaries: on-package memory is fastest because\n",
    "it sits on the same package, millimeters from the die over thousands of\n",
    "short interposer wires; everything that crosses a connector,\n",
    "a board trace, or a cable pays. The rule of thumb: **every chip boundary\n",
    "costs roughly an order of magnitude of bandwidth**. Keep the bytes home.\n",
    "\n",
    "![The 2026 bandwidth ladder. Our own GPU's GDDR6X sits two rungs below a\n",
    "B200's HBM3e; every boundary crossed costs roughly an order of\n",
    "magnitude.](../img/mdl-perf-bandwidth-ladder.svg)\n",
    "\n",
    "the figure ranks the same world by *latency* — how long\n",
    "before the first byte arrives — and spans eight orders of magnitude, from\n",
    "a one-nanosecond L1 hit to a sixty-millisecond WAN round trip. Note where\n",
    "the highlighted rung sits: launching a CUDA kernel costs 5–15 µs of\n",
    "latency, thousands of L2 hits' worth, which is why the overhead regime of\n",
    "that section exists at all and why this chapter keeps returning\n",
    "to \"fewer, larger operations\".\n",
    "\n",
    "![The 2026 latency ladder (log scale, nanoseconds). The kernel-launch rung\n",
    "is the one deep learning code trips over most often.](../img/mdl-perf-latency-ladder.svg)\n",
    "\n",
    "Latency and bandwidth interact through *access patterns*. DRAM delivers\n",
    "its rated bandwidth only for *burst* reads: addressing a location costs\n",
    "on the order of a hundred nanoseconds, after which sequential transfers\n",
    "stream at full rate — so the first word of a random read is hundreds of\n",
    "times more expensive than the words that follow it. Hardware therefore\n",
    "rewards streaming through memory in order and punishes pointer-chasing;\n",
    "caches amortize latency only when access patterns are local and\n",
    "predictable. For dense tensors this is mostly handled for you — a matmul\n",
    "kernel is a carefully choreographed burst-read machine — but it is why\n",
    "sparse and gather-heavy workloads struggle to reach more than a fraction\n",
    "of rated bandwidth, and why shuffled *dataset* access is best done at the\n",
    "granularity of large blocks. (The classic multi-core pathology of *false\n",
    "sharing* — two CPU threads bouncing one cache line between cores — is\n",
    "relegated to the exercises.)\n",
    "\n",
    "Let's measure the one number from this subsection that matters most for\n",
    "the roofline: our card's achievable memory bandwidth. A large elementwise\n",
    "scaling reads and writes every element once — bytes moved are known\n",
    "exactly, arithmetic is negligible — so timing it *is* measuring bandwidth:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "hardware-where-bytes-live"
   },
   "source": [
    "x = jax.random.normal(jax.random.PRNGKey(0), (256 * 1024 * 1024,))  # 1 GB\n",
    "t = d2l.Benchmark(lambda: x * 2.0).time\n",
    "print(f'measured {2 * x.size * 4 / t / 1e12:.2f} TB/s '\n",
    "      f'(spec: about 1.0 TB/s)')"
   ],
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "measured 0.91 TB/s (spec: about 1.0 TB/s)\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=hardware-md-3 fw=all -->\n",
    "\n",
    "A well-shaped streaming kernel lands within tens of percent of the\n",
    "specification — the spec number is real, not marketing. Remember what\n",
    "this kernel is: the bandwidth-bound regime made flesh. Every elementwise\n",
    "operation in every network you train runs at this speed, no matter how\n",
    "little it computes.\n",
    "\n",
    "## Why Compute Outruns Bandwidth\n",
    "\n",
    "The ridge point of that section is not an accident of one\n",
    "product; it is geometry. Arithmetic units fill the *interior* of a die:\n",
    "double the die's side length (or halve the feature size) and you get four\n",
    "times the compute, because compute scales with *area*. But the wires that\n",
    "leave the chip — the I/O drivers that talk to memory — live on the die's\n",
    "*edge*, and the edge only doubles. Compute scales as $n^2$, I/O as $n$\n",
    "(the figure). Chip designers call the edge \"beachfront\", and\n",
    "there is never enough of it: with every generation, bytes-per-FLOP falls,\n",
    "and the model starves a little more.\n",
    "\n",
    "![The shoreline problem: compute is area, I/O is perimeter. Scaling the\n",
    "die up buys compute four times faster than it buys\n",
    "bandwidth.](../img/mdl-perf-shoreline.svg)\n",
    "\n",
    "HBM is the countermeasure: stack DRAM dies vertically and place the stack\n",
    "on a silicon *interposer* millimeters from the GPU die, so thousands of\n",
    "short, dense wires replace the few hundred long board traces a socketed\n",
    "DIMM gets. That is how a B200 reaches nearly 8 TB/s while a CPU socket\n",
    "manages 0.5 TB/s — not faster DRAM cells, but vastly more, vastly shorter\n",
    "wires.\n",
    "The second countermeasure is size: dies are manufactured against a hard\n",
    "lithographic ceiling (the *reticle limit*, about $858\\,\\textrm{mm}^2$),\n",
    "and flagship accelerators live at it — a B200 is two reticle-limit dies\n",
    "bridged into one logical GPU. When you cannot grow the die, you bridge\n",
    "dies; when you cannot bridge, you network — the interconnect story of\n",
    "that section is the shoreline problem continued by\n",
    "other means.\n",
    "\n",
    "The long-run trend to memorize, averaged over many hardware generations\n",
    "at fixed precision: **compute grows about $4\\times$ per generation,\n",
    "bandwidth about $2\\times$, capacity about $1.7\\times$.** Over the years\n",
    "the ridge point — compute over bandwidth — has therefore *risen*, and\n",
    "workloads that were compute-bound a decade ago are bandwidth-bound today.\n",
    "Do not expect every product step to obey the average, though: which roof\n",
    "moves faster in any single generation is a packaging and market decision,\n",
    "and it can go the other way. The H100$\\to$B200 step grew compute and\n",
    "bandwidth almost in lockstep (both about $2.3\\times$ — the ridge point\n",
    "barely moved, in fact edging slightly *down*), and the consumer\n",
    "4090$\\to$5090 step bought bandwidth far faster than compute, dropping its\n",
    "ridge point by nearly a third — check the ridge row of\n",
    "the table below. The shoreline sets the long-run pressure;\n",
    "engineering chooses, one generation at a time, which wall to push.\n",
    "Averaged over the decade, compute wins — which is why a chapter about\n",
    "performance is mostly a chapter about bytes, and why it will remain one.\n",
    "\n",
    "## The GPU\n",
    "\n",
    "A GPU spends its area on thousands of simple arithmetic units rather than\n",
    "on the deep control logic of a CPU core. The units are organized into\n",
    "*streaming multiprocessors* (SMs) — our RTX 4090 has 128 of them — and\n",
    "each SM executes threads in lockstep groups of 32 called *warps*,\n",
    "swapping between many resident warps to hide memory latency. That is as\n",
    "much microarchitecture as this book needs: when a profiler reports\n",
    "\"occupancy\" it means resident warps available for latency-hiding; when a\n",
    "kernel is \"too small for the machine\" it means too few warps to keep 128\n",
    "SMs busy — the launch-overhead regime again.\n",
    "\n",
    "What makes GPUs *deep learning* machines is a further specialization:\n",
    "**tensor cores**, silicon shaped exactly like a small matrix\n",
    "multiply-accumulate. A scalar unit spends most of its energy on\n",
    "instruction handling per multiply; a tensor core amortizes that control\n",
    "over an entire $16\\times 16$-ish tile, executing it as one instruction —\n",
    "the same insight as the TPU's systolic array\n",
    "[@Jouppi.Young.Patil.ea.2017; @Kung.1988]. The consequence: matrix\n",
    "multiplications run an order of magnitude faster than anything else on\n",
    "the chip, *provided* the data comes in the formats the tensor cores\n",
    "speak. Those formats are the ladder in the figure.\n",
    "\n",
    "![The floating-point format ladder, bit-for-bit to scale. fp32, tf32, and\n",
    "bf16 share an 8-bit exponent and hence a dynamic range; each halving of\n",
    "width doubles peak arithmetic and halves the bytes\n",
    "moved.](../img/mdl-perf-float-formats.svg)\n",
    "\n",
    "Two rules organize the ladder. First, **the exponent width sets the\n",
    "range, the mantissa width sets the precision**. fp32, tf32, and bf16 all\n",
    "carry the same 8-bit exponent: they represent the same span of\n",
    "magnitudes, so swapping among them costs precision, not range — you give\n",
    "up decimal places, which deep learning mostly tolerates, rather than the\n",
    "ability to represent a large activation at all. fp16's 5-bit exponent\n",
    "trades range for precision, which is why it needs the loss-scaling\n",
    "machinery we will meet in that section, and why bf16\n",
    "has become the training default. The sub-byte formats — fp8 in two\n",
    "flavors (E4M3 for activations and weights, wider-range E5M2 for\n",
    "gradients, in the usual recipe rather than by any rule) and fp4 with just\n",
    "sixteen representable values — push the same trade to its limit and lean\n",
    "on per-block scaling factors to survive\n",
    "[@Micikevicius.Stosic.Burgess.ea.2022].\n",
    "Second, **every halving of storage wins twice**: half the bits means\n",
    "double the peak FLOP/s (twice the tile fits in the same silicon) *and*\n",
    "half the bytes per operand — the format ladder climbs the roofline along\n",
    "both axes at once. tf32 is the exception that proves the byte rule: it is\n",
    "a tensor-core *compute* format that still stores 32 bits (the figure\n",
    "draws exactly this), so it buys throughput but saves nothing on memory\n",
    "traffic. The catch: each halving buys about $2\\times$ on real silicon,\n",
    "not the $4\\times$ the marketing arithmetic suggests, and the big jumps\n",
    "between generations come from new architectures, not formats alone.\n",
    "\n",
    "The numbers-dense table, for orientation (conventions below it):\n",
    "\n",
    "| | H100 SXM | B200 SXM | RTX 4090 (ours) | RTX 5090 |\n",
    "|---|---|---|---|---|\n",
    "| memory | 80 GB HBM3 | 180 GB HBM3e | 24 GB GDDR6X | 32 GB GDDR7 |\n",
    "| bandwidth | 3.35 TB/s | 7.7 TB/s | 1.0 TB/s | 1.8 TB/s |\n",
    "| bf16 dense | 989 TF | 2,250 TF | 165 TF | 210 TF |\n",
    "| fp8 dense | 1,979 TF | 4,500 TF | 330 TF | 419 TF |\n",
    "| fp4 dense | — | 9,000 TF | — | 1,676 TF |\n",
    "| ridge (bf16) | ~295 FLOP/B | ~292 FLOP/B | ~165 FLOP/B | ~117 FLOP/B |\n",
    "| power | 700 W | 1,000 W | 450 W | 575 W |\n",
    "\n",
    "*Conventions: dense throughput (no 2:4 sparsity), fp32 accumulation, boost\n",
    "clocks, vendor datasheets as of mid-2026. The B200 column is the shipping\n",
    "HGX B200 spec; the GB200 NVL72 variant of the same silicon runs 186 GB at\n",
    "8 TB/s and up to 1,200 W. The 5090's fp4 entry is $4\\times$ its fp8 entry,\n",
    "not $2\\times$, because GeForce parts run fp8/fp16 tensor math at half rate\n",
    "when accumulating in fp32 while fp4 carries no such penalty.*\n",
    "\n",
    "The same constraints produce the same shape at every vendor —\n",
    "high-bandwidth stacked memory next to a die full of matrix engines: AMD's\n",
    "MI355X (288 GB HBM3e at 8 TB/s, 2,500 TF dense bf16), Google's TPU v7\n",
    "(192 GB at 7.4 TB/s, 2,307 TF), Amazon's Trainium2 (96 GB at 2.9 TB/s,\n",
    "667 TF). Different silicon, one design point; the roofline reasoning of\n",
    "this chapter applies unchanged to all of them.\n",
    "\n",
    "One more distinction the table hides: *training* hardware must hold\n",
    "activations for the backward pass and accumulate gradients robustly\n",
    "(bf16 with fp32 accumulation as the floor — that section),\n",
    "while *inference* can run forward-only in the smallest format quality\n",
    "allows. That asymmetry, plus capacity — note that memory capacity grew\n",
    "the *slowest* of the three curves — shapes the memory anatomy we build in\n",
    "that section.\n",
    "\n",
    "## The CPU's Role\n",
    "\n",
    "The CPU in a deep learning machine is no longer where the FLOPs happen —\n",
    "one RTX 4090 out-multiplies a 64-core CPU by two orders of magnitude —\n",
    "but three jobs still live or die on it. First, the CPU is the\n",
    "*orchestrator*: every kernel the GPU runs was launched by a CPU thread at\n",
    "5–15 µs apiece (the figure) — until the\n",
    "capture-and-replay of that section takes the CPU out of that\n",
    "loop — which is exactly the\n",
    "Python-must-stay-ahead story of that section. Second, it runs\n",
    "the *input pipeline*: decoding JPEGs, tokenizing text, augmenting,\n",
    "batching — if those fall behind, the GPU starves no matter how fast it\n",
    "is (a dataloader-starved profile is the overhead regime with extra\n",
    "steps). Third, it *feeds the bus*: host-to-device copies cross PCIe, and\n",
    "how you stage the host memory matters. A regular (pageable) host tensor\n",
    "must first be copied into a locked staging buffer before DMA can move it;\n",
    "allocating the tensor in *pinned* (page-locked) memory skips that step\n",
    "and, as a bonus, allows the copy to proceed asynchronously alongside\n",
    "compute. How much the staging costs — and who controls it — differs\n",
    "between our two frameworks:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "hardware-the-cpu-s-role"
   },
   "source": [
    "# JAX manages host staging internally; measure the achieved H2D rate.\n",
    "x_host = np.random.randn(64 * 1024 * 1024).astype(np.float32)  # 256 MB\n",
    "t = d2l.Benchmark(lambda: jax.device_put(x_host), desc='H2D').time\n",
    "print(f'device_put: {x_host.nbytes / t / 1e9:.1f} GB/s')"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "device_put: 0.9 GB/s\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=hardware-md-4 fw=jax -->\n",
    "\n",
    "The number deserves a hard look: the wire underneath is capable of tens\n",
    "of GB/s, yet `jax.device_put` from a pageable NumPy array lands far below\n",
    "the bus limit — around one gigabyte per second at our pin. The copy is\n",
    "not the cost; JAX's transfer path (validation, layout, staging through\n",
    "its own host buffer) is, and that is itself the lesson: a transfer runs\n",
    "at the speed of its slowest *stage*, and the framework owns some of the\n",
    "stages. JAX programs sidestep the problem structurally rather than tuning\n",
    "it — keep arrays resident on the device, let the input pipeline overlap\n",
    "transfers with compute, and treat a per-step `device_put` of fresh host\n",
    "data as a red flag."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=hardware-md-5 fw=all -->\n",
    "\n",
    "Either way the conclusion is structural: get data onto the device, keep\n",
    "it there, and overlap the unavoidable transfers with compute. This is\n",
    "also your first sighting of the number that will dominate the multi-GPU\n",
    "sections: *everything that leaves the GPU moves at bus speed — at best —\n",
    "not memory speed*.\n",
    "\n",
    "## Interconnects: Our Box as the Worked Example\n",
    "\n",
    "When one GPU is not enough, GPUs must talk to each other, and the\n",
    "bandwidth ladder says this is where performance goes to die. Datacenter\n",
    "parts attack the problem with dedicated fabrics: NVLink gives each B200\n",
    "1.8 TB/s to its peers — memory-class bandwidth, one rung below HBM — and\n",
    "an NVL72 rack wires 72 GPUs into a single 130 TB/s domain. Consumer\n",
    "parts get PCIe, and — the fact that shapes the rest of this chapter —\n",
    "the current GeForce generations (RTX 40 and 50 series) have *peer-to-peer\n",
    "transfers disabled* as market segmentation: two RTX 4090s in one box may\n",
    "not even talk PCIe-directly to each other. (It was not always so — the\n",
    "RTX 3090 still carried an NVLink bridge; its successors dropped it.)\n",
    "\n",
    "Our build machine makes the consequence concrete\n",
    "(the figure). Its four RTX 4090s hang in pairs off two\n",
    "PCIe host bridges; with P2P disabled, *every* byte from one GPU to\n",
    "another is staged through host DRAM — up one PCIe link, through the\n",
    "CPU's memory system, down another. Measured end to end\n",
    "(that section does the measuring), a large GPU-to-GPU copy\n",
    "sustains roughly twenty GB/s — PCIe-limited, two orders of magnitude\n",
    "below an NVL72's ~1.8 TB/s per GPU — and a collective library like NCCL,\n",
    "whose ring/tree chunking assumes a peer-to-peer fabric, extracts even less\n",
    "(a couple of GB/s of effective bus bandwidth) on this staged path — not\n",
    "because the wire slows down for NCCL, but because its P2P-less fallback\n",
    "moves bytes with a latency-bound mechanism by default;\n",
    "that section diagnoses this and names the one-line\n",
    "workaround, which that section weighs. Either\n",
    "way the staging path, not the link count, sets the ceiling, so the rate\n",
    "does *not* improve from two GPUs to four.\n",
    "\n",
    "![This book's build machine, per `nvidia-smi topo`: four RTX 4090s in\n",
    "pairs behind two PCIe host bridges, no P2P, no NVLink. The highlighted\n",
    "path is what every inter-GPU byte actually travels.](../img/mdl-perf-pcie-topology.svg)\n",
    "\n",
    "We treat this as a teaching instrument, not an embarrassment. Most\n",
    "readers' multi-GPU machines look like ours, not like an NVL72; and a slow\n",
    "fabric makes the *accounting* of parallel training vivid — on this box,\n",
    "communication costs are impossible to ignore, so the cost model of\n",
    "that section predicts real behavior instead of disappearing\n",
    "into NVLink headroom. Why datacenter fabrics exist will not need to be\n",
    "asserted; we will have measured it.\n",
    "\n",
    "## Energy: Why Moving Bytes Is the Budget\n",
    "\n",
    "There is a final ladder underneath the other two.\n",
    "the figure shows the energy cost of elementary\n",
    "operations at a fixed process node [@Horowitz.2014]: an 8-bit\n",
    "integer add costs 0.03 pJ, an fp32 multiply 3.7 pJ — and *one 64-bit\n",
    "read from DRAM costs about 2,000 pJ*. One memory access buys roughly\n",
    "five hundred multiplies. Arithmetic is nearly free; fetching operands is\n",
    "the budget, and a chip's power limit is in the end an energy-per-second\n",
    "limit. This is the deepest version of the ridge-point story: the reason\n",
    "compute outruns bandwidth is not just wire count but joules — you can\n",
    "afford to place more multipliers, but you cannot afford to feed them\n",
    "from far away.\n",
    "\n",
    "![Energy per operation (45 nm-class magnitudes). One DRAM access costs\n",
    "about five hundred fp32 multiplies.](../img/mdl-perf-energy-ladder.svg)\n",
    "\n",
    "The practical corollaries are the fixes this chapter teaches: fusion\n",
    "(that section) saves energy, not just time, by keeping\n",
    "intermediates on-die; low-precision formats\n",
    "(that section) halve the bytes and hence nearly halve\n",
    "the energy per operand; recomputation trades cheap arithmetic for\n",
    "expensive memory traffic. When a technique in this chapter feels like a\n",
    "trick, check it against the figure — nearly all of them\n",
    "are the same move: *do more arithmetic per byte fetched*.\n",
    "\n",
    "## Reading the Roofline: Two Workloads\n",
    "\n",
    "Close the loop by reading one model's two working regimes straight off\n",
    "the hardware numbers — the exercise this section exists to enable.\n",
    "Consider a language model of the kind built in\n",
    "that section, with $N$ parameters in bf16 ($2N$ bytes of weights),\n",
    "serving a prompt and then generating.\n",
    "\n",
    "**Prefill** — processing the prompt — pushes the whole context through\n",
    "the network at once. Each weight fetched from memory is reused across\n",
    "every token in the context, so arithmetic intensity is on the order of\n",
    "the context length: hundreds to tens of thousands of FLOPs per byte,\n",
    "far above any ridge point. A long, well-batched prefill is therefore\n",
    "typically *compute-bound*: it runs at the tensor cores' pace and benefits\n",
    "from every format halving. (A one-line prompt is not — intensity needs\n",
    "context to reuse each weight against.)\n",
    "\n",
    "**Decode** — generating one token at a time — is the opposite. Producing\n",
    "a single token performs about $2N$ FLOPs but must read all $2N$ weight\n",
    "bytes (plus the KV cache of that section): intensity $\\approx\n",
    "1$ FLOP/byte, the bottom of the roofline. Decode is *bandwidth-bound*,\n",
    "and its speed limit is arithmetic you can do on a napkin:\n",
    "\n",
    "$$\n",
    "\\textrm{tokens/s} \\;\\lesssim\\;\n",
    "\\frac{\\beta}{\\textrm{bytes per token}} \\;=\\;\n",
    "\\frac{\\beta}{2N + \\textrm{KV bytes}}.\n",
    "$$\n",
    "\n",
    "An 8-billion-parameter model in bf16 reads at least 16 GB per generated\n",
    "token; on our 1 TB/s card that caps generation near sixty tokens per\n",
    "second *regardless of compute*, while the same card's prefill chews\n",
    "through the prompt at tensor-core speed. One model, both ends of\n",
    "the figure — and the reason batching multiple generation\n",
    "streams (which shares each weight read across streams, multiplying\n",
    "intensity by the batch size) is the central trick of inference serving.\n",
    "The systems that exploit this — batching schedulers, paged caches,\n",
    "speculative decoding — belong to the Language Models part; the\n",
    "*economics* fit in the equation above.\n",
    "\n",
    "Rules of thumb worth carrying out of this section:\n",
    "\n",
    "| quantity | magnitude | why it matters |\n",
    "|---|---|---|\n",
    "| kernel launch | 5–15 µs | small ops cannot feed the GPU (that section) |\n",
    "| GPU memory bandwidth | 1–8 TB/s | the decode/elementwise speed limit |\n",
    "| PCIe per direction | tens of GB/s | host↔device; get data on-device, keep it there |\n",
    "| NVLink-class fabric | ~1.8 TB/s per GPU | why datacenter multi-GPU scales and PCIe boxes struggle |\n",
    "| bytes/param, training with Adam | ~16–20 (mixed precision) | memory anatomy of that section |\n",
    "| bytes/token, decode | ~2 × params + KV | tokens/s ≤ bandwidth / this |\n",
    "| DRAM read vs fp32 multiply | ~500× the energy | fuse, shrink formats, recompute |\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Memory is a hierarchy: each level away from the arithmetic holds orders\n",
    "  of magnitude more bytes and delivers orders of magnitude fewer per\n",
    "  second, with a latency ladder spanning eight decades. Bandwidth is\n",
    "  earned by streaming; boundaries cost an order of magnitude each.\n",
    "* Compute scales with die area, I/O with die perimeter. The long-run\n",
    "  outcome is compute growing ~4× per generation against bandwidth's ~2×\n",
    "  and capacity's ~1.7×, so the ridge point has risen over the years —\n",
    "  though a single product step can move it either way (B200 and the RTX\n",
    "  5090 both nudged it *down* by buying bandwidth). HBM-on-interposer is a\n",
    "  countermeasure, not a repeal.\n",
    "* Tensor cores make matrix arithmetic an order of magnitude faster than\n",
    "  everything else, in exchange for format discipline: same exponent =\n",
    "  same range; every halving of width doubles FLOP/s and halves bytes.\n",
    "* The CPU orchestrates (kernel launches), feeds (input pipeline), and\n",
    "  stages (pinned-memory PCIe copies). Inter-GPU bandwidth on consumer\n",
    "  boxes is host-staged and PCIe-limited — tens of GB/s for a raw copy,\n",
    "  and only a couple of GB/s of NCCL busbw at its default fallback\n",
    "  settings (that section finds the binding stage) — one to\n",
    "  three orders of magnitude below an NVLink domain, which is why\n",
    "  datacenter fabrics exist.\n",
    "* Energy explains everything twice: a DRAM access costs ~500 multiplies,\n",
    "  so every technique in this chapter is a variation on \"more arithmetic\n",
    "  per byte fetched\".\n",
    "* One model can live at both ends of the roofline: long-context prefill\n",
    "  is typically compute-bound, single-stream decode is bandwidth-bound\n",
    "  with tokens/s ≲ bandwidth / model bytes.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Compute the ridge point for each GPU in the table at\n",
    "   bf16 and at fp8. Which direction did it move from the H100 to the\n",
    "   B200, and from the RTX 4090 to the RTX 5090? Reconcile what you find\n",
    "   with the shoreline argument of that section: what did\n",
    "   each step's packaging buy, and why does the *long-run* direction still\n",
    "   differ from these two steps?\n",
    "1. Take the GPT of that section (count its parameters) and your own\n",
    "   GPU's specifications. Estimate (a) the prefill arithmetic intensity at\n",
    "   context length 128 and (b) the decode tokens-per-second bound. Which\n",
    "   regime is each in?\n",
    "1. Using the figure, estimate the energy spent on DRAM\n",
    "   traffic alone in one epoch of the Fashion-MNIST training loop of\n",
    "   that section (count the bytes your activations and weights\n",
    "   move). At $0.30/kWh, what does the epoch's memory traffic cost?\n",
    "1. Measure the cache cliff on your GPU: time `x * 2.0` for tensor sizes\n",
    "   from 1 MB to 2 GB and plot achieved bandwidth against size. Where do\n",
    "   the level transitions of the figure appear?\n",
    "1. Why is HBM placed on a silicon interposer rather than on the\n",
    "   motherboard next to the CPU's DRAM slots? Answer using the shoreline\n",
    "   argument, then check: how many signal wires does a DIMM socket carry,\n",
    "   versus an HBM stack's interface?\n",
    "1. Burst versus random access: allocate a large tensor and compare\n",
    "   summing it in natural order against summing it through a random\n",
    "   permutation of indices. Explain the ratio using the\n",
    "   address-setup-versus-streaming model of\n",
    "   that section.\n",
    "1. (False sharing.) Write a two-thread CPU program in which each thread\n",
    "   increments its own counter, first with the counters adjacent in\n",
    "   memory, then padded to separate cache lines. Measure, and explain why\n",
    "   the \"shared-nothing\" version can be many times faster despite\n",
    "   identical arithmetic.\n",
    "1. Your DataLoader delivers batches at 300 MB/s from NVMe while your GPU\n",
    "   consumes them at 2 GB/s. Using the ladders in this section, list the\n",
    "   three cheapest interventions in order of expected payoff."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-jax",
   "display_name": "d2l (JAX)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
