{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=hardware-md-1 fw=all -->\n",
    "\n",
    "# Hardware\n",
    "\n",
    "At some point most practitioners ask: should I buy a machine? This section\n",
    "is a buyers guide. Concrete hardware advice ages quickly — we quote prices\n",
    "as of July 2026 and expect them to be wrong in a year (indeed, 2026's\n",
    "memory shortage has GPUs *appreciating*, an anomaly worth remembering when\n",
    "you read anyone's price table, including ours). What does not age is the\n",
    "reasoning: figure out whether your workload is bound by memory capacity,\n",
    "memory bandwidth, or compute, and buy the binding constraint. The specific\n",
    "machines below are reference points for that reasoning as much as\n",
    "recommendations. Where that section explains *why* the machine\n",
    "behaves the way it does — the roofline, the memory hierarchy, the format\n",
    "ladder — this section is the complementary question of *what to\n",
    "purchase*.\n",
    "\n",
    "Two physical facts organize everything that follows. First, **capacity\n",
    "gates what runs at all**: training state is several times model size\n",
    "(that section), and an LLM that does not fit in memory\n",
    "does not run slowly — it does not run. Second, **bandwidth gates\n",
    "generation speed**: producing one token requires streaming essentially all\n",
    "active weights through the compute units, so\n",
    "\n",
    "$$\n",
    "\\textrm{decode tokens/s} \\lesssim\n",
    "\\frac{\\textrm{memory bandwidth}}{\\textrm{bytes of active weights}},\n",
    "$$\n",
    "\n",
    "while *prompt processing* (prefill) and *training* are parallel across\n",
    "tokens and lean on compute instead. Machines differ by an order of\n",
    "magnitude on each axis, in different directions — which is why \"what\n",
    "should I buy?\" has no single answer, but always the same analysis.\n",
    "\n",
    "## What Are You Buying For?\n",
    "\n",
    "Before comparing cards, estimate both sides of the constraint. For\n",
    "training, peak memory (the figure):\n",
    "\n",
    "![Peak training memory includes weights, gradients, optimizer state, and activations in addition to parameters.](../img/tools-memory-fit.svg)"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "hardware-memory-estimate"
   },
   "source": [
    "def training_memory_gib(parameters_billion, weight_bytes=2,\n",
    "                        gradient_bytes=2, optimizer_bytes=8,\n",
    "                        activation_gib=6, workspace_gib=2):\n",
    "    parameters = parameters_billion * 1e9\n",
    "    state = parameters * (weight_bytes + gradient_bytes + optimizer_bytes)\n",
    "    return state / 2**30 + activation_gib + workspace_gib\n",
    "\n",
    "for b in [0.1, 1, 7]:\n",
    "    print(f\"{b:>4}B params: {training_memory_gib(b):6.1f} GiB\")"
   ],
   "execution_count": 1,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      " 0.1B params:    9.1 GiB\n   1B params:   19.2 GiB\n   7B params:   86.2 GiB\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=hardware-md-2 fw=all -->\n",
    "\n",
    "A 7B full fine-tune wants ~90 GiB — no consumer card holds it — while\n",
    "LoRA (freeze the weights, train small adapters, quantize the base to\n",
    "4 bits) collapses the same job to under 8 GiB. That single technique is\n",
    "why a 16 GB card is a genuine training machine in 2026. For inference,\n",
    "apply the decode bound:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "hardware-decode-bound"
   },
   "source": [
    "bandwidth_gbs = {\"RTX 5070 Ti\": 896, \"RTX 3090 (used)\": 936,\n",
    "                 \"RTX 5090\": 1792, \"RTX PRO 6000\": 1792,\n",
    "                 \"DGX Spark\": 273, \"Strix Halo\": 256,\n",
    "                 \"M4 Max\": 546, \"M3 Ultra\": 819, \"H100 SXM\": 3350}\n",
    "active_gb = 4.5   # 8B model at 4-bit, including cache traffic\n",
    "for device, bw in sorted(bandwidth_gbs.items(), key=lambda kv: -kv[1]):\n",
    "    print(f\"{device:>16s}: <= {bw / active_gb:5.0f} tok/s\")"
   ],
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "        H100 SXM: <=   744 tok/s\n        RTX 5090: <=   398 tok/s\n    RTX PRO 6000: <=   398 tok/s\n RTX 3090 (used): <=   208 tok/s\n     RTX 5070 Ti: <=   199 tok/s\n        M3 Ultra: <=   182 tok/s\n          M4 Max: <=   121 tok/s\n       DGX Spark: <=    61 tok/s\n      Strix Halo: <=    57 tok/s\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=hardware-md-3 fw=all -->\n",
    "\n",
    "Realized speed is typically 50–80% of this bound, but the *ordering* it\n",
    "predicts matches measurement remarkably well — and it explains oddities\n",
    "like a \\$4,700 DGX Spark decoding no faster than a MacBook: both stream\n",
    "weights at roughly 273 GB/s.\n",
    "\n",
    "![Discrete GPUs pair modest capacity with very high bandwidth; unified-memory machines invert the trade. The gap in the upper right is where datacenter GPUs live.](../img/tools-hardware-menu.svg)\n",
    "\n",
    "the figure plots the July 2026 menu on those two\n",
    "axes. The machines split into two families — discrete GPUs (fast, small)\n",
    "and unified-memory systems (capacious, slow) — with the workstation-class\n",
    "RTX PRO 6000 bridging them at a price. Everything below walks this figure.\n",
    "\n",
    "## Training Boxes: Discrete NVIDIA GPUs\n",
    "\n",
    "For training and fine-tuning, a discrete NVIDIA card remains the default:\n",
    "highest bandwidth per dollar, mature software (every framework in this\n",
    "book, FlashAttention, quantization kernels) and the same CUDA stack you\n",
    "will meet on rented cloud machines.\n",
    "\n",
    "### RTX 5070 Ti: the Smallest Serious Trainer\n",
    "\n",
    "The RTX 5070 Ti (16 GB GDDR7 at 896 GB/s, 300 W, ~\\$900 in July 2026\n",
    "against a \\$749 list) is about the smallest card we would recommend for\n",
    "*training* rather than only inference. Sixteen gigabytes runs every\n",
    "notebook in this book with plenty of headroom, LoRA/QLoRA fine-tunes of\n",
    "7–8B models comfortably, and diffusion fine-tuning at 1024² — while a\n",
    "12 GB card starts excluding exactly the experiments you will want to try\n",
    "next. A complete quiet desktop around it (any 8-core CPU, 64 GB RAM, 1 TB\n",
    "NVMe, 850 W supply) lands near \\$2,000–2,500. Its natural competitor is\n",
    "the used RTX 3090 (24 GB at 936 GB/s, ~\\$1,000–1,400 used in July 2026):\n",
    "more memory and similar bandwidth, but 350 W, no FP8/FP4 tensor cores, no\n",
    "warranty, and — a 2026 curiosity — a price that has been rising.\n",
    "\n",
    "### RTX 5090: the Enthusiast Box\n",
    "\n",
    "The RTX 5090 (32 GB GDDR7 at 1.79 TB/s, 575 W, \\$3,000–4,300 street in\n",
    "July 2026 against a \\$1,999 list) is the fastest thing you can put in a\n",
    "desktop: datacenter-class bandwidth, Blackwell FP8/FP4 tensor cores, and\n",
    "enough memory for full fine-tunes of small models and LoRA on anything up\n",
    "to ~30B. It is also a *system-design problem*: budget a 1200 W power\n",
    "supply with a native 16-pin connector (measured transients approach\n",
    "660 W), a full tower with real airflow, the noise tolerance of a small\n",
    "space heater, and — on a 120 V circuit — awareness that the whole box\n",
    "under load approaches what the wall socket provides. A balanced build\n",
    "(the figure) pairs it with a current 16-core CPU,\n",
    "64–128 GB of RAM, and fast NVMe scratch for datasets; in the 2026 memory\n",
    "market that totals \\$6,500–10,000 depending mostly on the RAM.\n",
    "\n",
    "![A balanced workstation feeds the GPU, stores data and checkpoints, and sustains power and cooling under load.](../img/tools-workstation.svg)\n",
    "\n",
    "Two cards? Multi-GPU consumer rigs (classically 2–4 used 3090s) remain a\n",
    "beloved r/LocalLLaMA pattern for cheap VRAM aggregation, and data\n",
    "parallelism over PCIe works fine. Know what you are signing up for:\n",
    "consumer boards offer one full-bandwidth slot, so you want used\n",
    "Threadripper/EPYC platforms for lanes; power adds up (dual supplies are\n",
    "common); and no consumer card since the 3090 has NVLink, so\n",
    "communication-heavy tensor parallelism scales poorly. Renting an 8×GPU\n",
    "node for a day (that section) is the cheap way to learn\n",
    "whether your workload cares.\n",
    "\n",
    "## Local Inference: the Unified-Memory Class\n",
    "\n",
    "![Discrete GPUs copy between system RAM and VRAM; unified-memory systems share one address space, but capacity and bandwidth remain distinct constraints.](../img/tools-memory-path.svg)\n",
    "\n",
    "A second family of machines answers a different question: not \"how fast\n",
    "can I train?\" but \"how large a model can I *run* at home?\" These systems\n",
    "give the GPU direct access to a large pool of ordinary (LPDDR5X) memory —\n",
    "lots of capacity at a fraction of GDDR7/HBM bandwidth\n",
    "(the figure). The decode bound tells you exactly what\n",
    "to expect: big models run, none of them run fast, and mixture-of-experts\n",
    "models — which activate only a few billion parameters per token — are the\n",
    "loophole that makes the class shine. The July 2026 menu:\n",
    "\n",
    "<!-- tbl-caption:Unified-memory machines for local inference (July 2026) -->\n",
    "\n",
    "| Machine | Memory | Bandwidth | ≈ Price | Notes |\n",
    "|---|---|---|---|---|\n",
    "| AMD Strix Halo mini-PC | 128 GB | 256 GB/s | \\$2,000–2,500 | Framework, Beelink, HP et al.; ROCm still maturing, Vulkan works |\n",
    "| NVIDIA DGX Spark (GB10) | 128 GB | 273 GB/s | \\$3,500–4,700 | CUDA-native Blackwell; strong prefill; two units pair to 256 GB |\n",
    "| Mac Studio M4 Max | up to 128 GB | 546 GB/s | from \\$2,500 | MLX ecosystem; quiet, efficient |\n",
    "| Mac Studio M3 Ultra | up to 96 GB new | 819 GB/s | from \\$5,300 | 256/512 GB configs discontinued 2026; used units carry a premium |\n",
    "| MacBook Pro M5 Max | up to 128 GB | 614 GB/s | from ~\\$3,500 | the same class, portable |\n",
    "\n",
    "Measured reality (community `llama-bench` figures, mid-2026) matches the\n",
    "bandwidth ordering: the 120B-parameter MoE `gpt-oss` (~5B active per\n",
    "token) generates ~71 tok/s on an M3 Ultra, ~35 on a DGX Spark, ~40 on\n",
    "Strix Halo; a *dense* 70B at 4-bit limps at 5–12 tok/s on all of them.\n",
    "Where they differ sharply is prefill: the Spark's Blackwell tensor cores\n",
    "chew through long prompts several times faster than Strix Halo or Apple's\n",
    "GPU at the same decode speed — the compute-versus-bandwidth split of our\n",
    "opening formula, embodied in retail products. Choose by stack: CUDA\n",
    "development in miniature → DGX Spark; cheapest 128 GB and Linux → Strix\n",
    "Halo; the fastest local decode and a polished laptop-to-desktop path →\n",
    "Apple with MLX.\n",
    "\n",
    "## The Top End: Workstation Blackwell\n",
    "\n",
    "The RTX PRO 6000 Blackwell packs 96 GB of GDDR7 at the 5090's 1.79 TB/s —\n",
    "the upper-right bridge in the figure — for about\n",
    "\\$13,000 list in July 2026 (it launched at \\$8,565; the memory crisis did\n",
    "the rest). A 300 W Max-Q variant with identical memory exists precisely so\n",
    "that four of them fit in one chassis: a 2× build (~\\$30–35k) runs\n",
    "unquantized 70B inference and full 70B fine-tunes; a 4× Threadripper PRO\n",
    "workstation (~\\$60k turnkey) holds 384 GB — DeepSeek-class MoE territory\n",
    "at 4-bit. One caveat defines the class: no NVLink (dropped from\n",
    "workstation cards after Ampere), so inter-card traffic rides PCIe 5.0 at\n",
    "~64 GB/s per direction — nearly thirty times slower than each card's local\n",
    "memory. Data- and pipeline-parallel workloads shrug; tensor parallelism\n",
    "pays. Against this budget, always price the alternative: \\$60k rents\n",
    "roughly 20,000 H100-hours (the table), and the break-even\n",
    "needs sustained, private, or interactive use to clear:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "hardware-break-even"
   },
   "source": [
    "purchase_and_operation = 4200.0     # e.g. a 5090 box + 3y power\n",
    "rental_per_productive_hour = 2.50   # H100-class marketplace rate\n",
    "print(f\"{purchase_and_operation / rental_per_productive_hour:.0f} \"\n",
    "      \"productive hours to break even\")"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "1680 productive hours to break even\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=hardware-md-4 fw=all -->\n",
    "\n",
    "The relevant inputs are *productive* hours (a home box idles more than you\n",
    "think) — against which weigh privacy, zero queue time, and the real\n",
    "pedagogical value of hardware you can take apart.\n",
    "\n",
    "## Keeping Current\n",
    "\n",
    "Everything above will drift; the sources below stay good. For build\n",
    "advice and street prices, [r/LocalLLaMA](https://www.reddit.com/r/LocalLLaMA/)\n",
    "(hardware megathreads) and the\n",
    "[Level1Techs forum](https://forum.level1techs.com/) (multi-GPU build\n",
    "logs); for reviews of exactly this niche — Strix Halo boxes, DGX Spark,\n",
    "used servers — [ServeTheHome](https://www.servethehome.com/); for\n",
    "apples-to-apples numbers, the community benchmark threads in\n",
    "[llama.cpp discussions](https://github.com/ggml-org/llama.cpp/discussions)\n",
    "and [llm-tracker.info](https://llm-tracker.info/); and for the durable\n",
    "*reasoning* about GPU choice, Tim Dettmers' classic guide (frozen in 2023,\n",
    "still right about everything except the part numbers). One warning: GPU\n",
    "search results are now thick with machine-generated pages quoting invented\n",
    "benchmarks — trust datasheets, named reviewers, and numbers that pass the\n",
    "decode-bound sanity check above.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Buy the binding constraint: capacity gates what runs, bandwidth gates\n",
    "  decode speed, compute gates training and prefill — and the decode bound\n",
    "  (bandwidth ÷ active bytes) predicts generation speed to within a factor\n",
    "  of two.\n",
    "* Reference points, July 2026: RTX 5070 Ti (~\\$900, 16 GB) is the smallest\n",
    "  serious trainer; RTX 5090 (~\\$3–4k, 32 GB) is the enthusiast ceiling and\n",
    "  a system-design exercise; used 3090s remain the budget VRAM play.\n",
    "* Unified-memory machines (Strix Halo, DGX Spark, Apple silicon) trade\n",
    "  bandwidth for capacity: large MoE models run well, dense 70B crawls,\n",
    "  and prefill speed separates otherwise similar boxes.\n",
    "* The RTX PRO 6000 class (96 GB at full bandwidth, no NVLink) bridges the\n",
    "  two families for five figures; before spending it, price the rented\n",
    "  equivalent.\n",
    "* Date-stamp every price you rely on, and get current numbers from the\n",
    "  community sources above rather than from search-engine filler.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Compute the decode bound for a 30B-A3B mixture-of-experts model (3B\n",
    "   active parameters, 4-bit) on every machine in\n",
    "   the table. Which become interactive (>20 tok/s)?\n",
    "1. Estimate LoRA fine-tuning memory for an 8B model on a 16 GB card:\n",
    "   4-bit frozen base, BF16 adapters at 1% of parameters, Adam. Does it\n",
    "   fit, and what dominates?\n",
    "1. Spec a complete RTX 5070 Ti build at current local prices, then\n",
    "   compute its break-even in hours against a rented 4090 from\n",
    "   the table.\n",
    "1. Find this month's used 3090 price and recompute its \\$/GB against the\n",
    "   current 5070 Ti. Has the 2026 anomaly (used cards appreciating)\n",
    "   persisted?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-pytorch",
   "display_name": "d2l (PyTorch)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
