{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=model-serving-md-1 fw=all -->\n",
    "\n",
    "# Model Serving\n",
    "\n",
    "Training produces weights; serving turns them into something other people\n",
    "(or programs) can call. The engineering here is different in kind from\n",
    "training: latency distributions instead of loss curves, memory that scales\n",
    "with *users* instead of parameters, and a software landscape that\n",
    "consolidated, in the span of two years, around a handful of engines you\n",
    "should simply know by name. This section maps the landscape practically —\n",
    "what to run on a laptop, what to run behind an API, and the small set of\n",
    "ideas (KV-cache management, continuous batching, prefix reuse,\n",
    "quantization) that explain why those engines are fast.\n",
    "\n",
    "## Know Your Workload\n",
    "\n",
    "Two numbers govern interactive serving of autoregressive models: **time to\n",
    "first token** (TTFT), dominated by *prefill* — processing the prompt, a\n",
    "parallel, compute-bound pass — and **time per output token** (TPOT),\n",
    "dominated by *decode* — one token at a time, bandwidth-bound, as\n",
    "that section quantified. Throughput (tokens per second\n",
    "across all users) trades against both.\n",
    "\n",
    "![Prompt prefill is parallel and often compute-heavy; autoregressive decode advances one token per sequence and stresses memory movement.](../img/tools-prefill-decode.svg)\n",
    "\n",
    "Which numbers matter depends on the workload, and the workload picks the\n",
    "tool:\n",
    "\n",
    "<!-- tbl-caption:Serving workloads and sensible starting points (mid-2026) -->\n",
    "\n",
    "| Workload | Optimize for | Start with |\n",
    "|---|---|---|\n",
    "| Personal assistant on your machine | simplicity, privacy | Ollama / LM Studio (llama.cpp), MLX on Macs |\n",
    "| Offline batch scoring | throughput per dollar | vLLM offline mode |\n",
    "| One application's API | operability | vLLM behind a small authenticated proxy |\n",
    "| Multi-user / agentic service | goodput under an SLO | vLLM or SGLang, then NVIDIA Dynamo at scale |\n",
    "| Max throughput on fixed NVIDIA fleet | compiled kernels | TensorRT-LLM |\n",
    "\n",
    "\"Goodput\" — completed requests that also met their latency target — is the\n",
    "production metric; overload can raise raw throughput while goodput\n",
    "collapses.\n",
    "\n",
    "## Serving on Your Own Machine\n",
    "\n",
    "The local stack standardized around **llama.cpp** and its **GGUF** file\n",
    "format, which now accounts for the majority of quantized models published\n",
    "on Hugging Face — community quantizers upload GGUF conversions within\n",
    "hours of any release. You rarely invoke llama.cpp directly:\n",
    "\n",
    "* **Ollama** wraps it (plus an Apple-MLX backend) in a one-command\n",
    "  experience — `ollama run qwen3:8b` downloads, caches, and serves a model\n",
    "  with an OpenAI-compatible local API. The default for \"I just want a\n",
    "  model on my machine.\"\n",
    "* **LM Studio** offers the same engines behind a GUI, with per-layer GPU\n",
    "  offload control — the friendliest on-ramp for non-terminal users.\n",
    "* **MLX** is Apple's array framework; on M-series Macs `mlx-lm` is\n",
    "  typically the fastest option for models under ~14B, with thousands of\n",
    "  pre-converted models on the Hub.\n",
    "\n",
    "GGUF's quantization ladder is a vocabulary worth learning: `Q4_K_M`\n",
    "(~4.5 bits/weight) is the community default — roughly a quarter of the\n",
    "FP16 footprint at a few percent quality cost; `Q5_K_M` buys measurable\n",
    "stability for reasoning tasks; `Q8_0` is near-lossless at half of FP16.\n",
    "The rule of thumb follows the decode bound of\n",
    "that section: pick the largest model whose chosen quant\n",
    "fits your memory with room for the KV cache, then take the highest quant\n",
    "that still fits.\n",
    "\n",
    "## Serving as a Service\n",
    "\n",
    "### vLLM and SGLang\n",
    "\n",
    "For a GPU server exposed to real traffic, the open-source default is\n",
    "**vLLM**: continuous batching, paged KV-cache management, prefix caching,\n",
    "tensor parallelism, quantized-model support, and an OpenAI-compatible\n",
    "server in one command:\n",
    "\n",
    "```bash\n",
    "vllm serve Qwen/Qwen3-8B --max-model-len 32768\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=model-serving-md-2 fw=all -->\n",
    "\n",
    "**SGLang** is its closest peer, distinguished by *RadixAttention* — a\n",
    "radix-tree KV cache shared across requests, which shines when many\n",
    "requests share long prefixes (chat with a system prompt, RAG over the\n",
    "same documents, agent trees). High-volume shops, notably the DeepSeek\n",
    "ecosystem, favor it for exactly those workloads. Both projects are now\n",
    "community-governed, move monthly, and support NVIDIA plus (increasingly)\n",
    "AMD hardware; benchmarking *your* prompt mix on both is a one-afternoon\n",
    "exercise and the only comparison that matters.\n",
    "\n",
    "The NVIDIA-proprietary tier trades flexibility for peak numbers:\n",
    "**TensorRT-LLM** compiles a model into optimized engines (fastest steady-\n",
    "state serving of a fixed model, at real setup cost), while **Dynamo**\n",
    "orchestrates disaggregated prefill/decode and KV-aware routing across a\n",
    "fleet, wrapping vLLM, SGLang, or TensorRT-LLM as backends. These matter\n",
    "at datacenter scale; below it, they are complexity you do not need.\n",
    "\n",
    "### One Client Contract\n",
    "\n",
    "Nearly everything above speaks the OpenAI chat-completions API, which has\n",
    "become the de-facto client contract — the same application code targets a\n",
    "local Ollama, your vLLM server, or a commercial provider by changing one\n",
    "URL:\n",
    "\n",
    "```text\n",
    "from openai import OpenAI\n",
    "\n",
    "client = OpenAI(base_url=\"http://127.0.0.1:8000/v1\", api_key=\"local\")\n",
    "response = client.chat.completions.create(\n",
    "    model=\"Qwen/Qwen3-8B\",\n",
    "    messages=[{\"role\": \"user\", \"content\": \"Explain KV caching briefly.\"}],\n",
    "    temperature=0,\n",
    ")\n",
    "print(response.choices[0].message.content)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=model-serving-md-3 fw=all -->\n",
    "\n",
    "\"Compatible\" is not \"identical\": tokenization, sampling defaults,\n",
    "structured-output support, streaming behavior, and token accounting all\n",
    "vary. Keep a small conformance test and rerun it before swapping engines.\n",
    "\n",
    "## Why These Engines Are Fast\n",
    "\n",
    "Four ideas, composable and worth knowing even if you never implement\n",
    "them:\n",
    "\n",
    "**Continuous batching.** A static batch waits for its slowest member.\n",
    "Continuous batching admits new requests and retires finished ones between\n",
    "decode steps, keeping the GPU full:\n",
    "\n",
    "![Continuous batching reclaims finished slots and admits waiting requests instead of padding every sequence to the longest output.](../img/tools-continuous-batching.svg)\n",
    "\n",
    "The toy scheduler below captures the idea in ten lines; extend it with\n",
    "arrival times and a memory budget and you have reinvented the core of a\n",
    "serving engine:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "model-serving-scheduler"
   },
   "source": [
    "from collections import deque\n",
    "\n",
    "requests = deque([5, 2, 7, 3])  # output tokens requested\n",
    "capacity = 2\n",
    "active, timeline = [], []\n",
    "while requests or active:\n",
    "    while requests and len(active) < capacity:\n",
    "        active.append(requests.popleft())\n",
    "    timeline.append(tuple(active))\n",
    "    active = [remaining - 1 for remaining in active if remaining > 1]\n",
    "\n",
    "timeline"
   ],
   "execution_count": 1,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "[(5, 2), (4, 1), (3, 7), (2, 6), (1, 5), (4, 3), (3, 2), (2, 1), (1,)]"
     },
     "metadata": {},
     "execution_count": 1
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=model-serving-md-4 fw=all -->\n",
    "\n",
    "**Paged KV cache.** Each active sequence's key–value cache grows with its\n",
    "length; vLLM's PagedAttention allocates it in fixed-size blocks, like\n",
    "virtual memory, eliminating the fragmentation that once capped batch\n",
    "sizes. Capacity planning follows: GPU memory must hold weights *plus* KV\n",
    "for every concurrent sequence — the reason a 7B model on a 24 GB card can\n",
    "still refuse the fifty-first user.\n",
    "\n",
    "**Prefix caching.** When two requests share a tokenized prefix (the same\n",
    "system prompt, the same document), its KV state can be computed once and\n",
    "reused:\n",
    "\n",
    "![A prefix cache reuses KV state only when tokenized prefixes match, then computes each request's unique suffix.](../img/tools-prefix-cache.svg)\n",
    "\n",
    "This is also why commercial APIs price \"cached input tokens\" at ~10% of\n",
    "regular ones — structure your prompts (shared prefix first, variable\n",
    "suffix last) and the discount is automatic.\n",
    "\n",
    "**Speculative decoding and quantization.** A small draft model proposes\n",
    "several tokens; the target model verifies them in one parallel pass —\n",
    "2–3× decode speedups when acceptance is high, with provably unchanged\n",
    "output distribution. Quantization attacks the decode bound directly:\n",
    "fewer bytes per weight, more tokens per second. On datacenter GPUs the\n",
    "current ladder is FP8 (native since Hopper) and NVFP4 (Blackwell), with\n",
    "AWQ the workhorse for 4-bit weights in vLLM/SGLang deployments; GGUF\n",
    "quants play the same role locally.\n",
    "\n",
    "## Operating Notes\n",
    "\n",
    "A model behind a port is not yet a service. The short version of the\n",
    "production checklist: put authentication, TLS, and rate limits at a proxy\n",
    "in front of the engine (never expose the raw port); make admission\n",
    "control reject or queue *before* memory exhausts rather than OOM after;\n",
    "propagate client cancellations so abandoned generations stop burning\n",
    "compute; measure TTFT, TPOT, and end-to-end latency as p50/p95\n",
    "distributions under realistic arrival patterns, not averages under a\n",
    "closed loop; avoid logging raw prompts by default; and pin model,\n",
    "tokenizer, quantization, and engine versions together — a faster server\n",
    "that answers differently is a different system, so gate rollouts on task\n",
    "quality as well as latency.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Separate prefill (compute-bound, sets TTFT) from decode\n",
    "  (bandwidth-bound, sets TPOT), and pick tools by workload: Ollama/LM\n",
    "  Studio/MLX locally, vLLM or SGLang for services, TensorRT-LLM and\n",
    "  Dynamo at NVIDIA-fleet scale.\n",
    "* GGUF and its quant ladder rule local serving; AWQ/FP8/NVFP4 rule\n",
    "  servers; in both cases quantization converts bytes saved into tokens\n",
    "  per second.\n",
    "* Continuous batching, paged KV, prefix caching, and speculative decoding\n",
    "  are the four ideas behind every fast engine — and prefix structure is\n",
    "  something *you* control from the prompt side.\n",
    "* The OpenAI-compatible API is the portable client contract; verify\n",
    "  compatibility with your own conformance tests.\n",
    "* Capacity is weights plus per-user KV; production means admission\n",
    "  control, cancellation, percentile latencies, and pinned revisions.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Extend the toy scheduler with Poisson arrivals, a KV-memory budget,\n",
    "   and rejection. Compare first-come-first-served against\n",
    "   shortest-remaining-work on p95 TTFT.\n",
    "1. Serve the same 8B model through Ollama (Q4_K_M) and vLLM (AWQ) on\n",
    "   whatever hardware you have, and measure TTFT and TPOT at concurrency\n",
    "   1, 4, and 16. Which engine wins where, and why?\n",
    "1. Estimate KV-cache bytes per token for a model whose config you know\n",
    "   (layers × 2 × kv-heads × head-dim × bytes), then compute how many\n",
    "   8K-context users fit beside the weights on a 24 GB card.\n",
    "1. Design the cache-key policy for a service that reuses a long system\n",
    "   prompt across users. What must be in the key, and what is the privacy\n",
    "   obligation of caching at all?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-pytorch",
   "display_name": "d2l (PyTorch)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
