{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=cloud-instances-md-1 fw=all -->\n",
    "\n",
    "# Cloud Computing\n",
    "\n",
    "Sooner or later a workload outgrows the free tiers of\n",
    "that section: the dataset takes a day to preprocess, the\n",
    "model needs 80 GB of accelerator memory, or you want eight GPUs for an\n",
    "afternoon. Renting is the natural next step, and it has never been cheaper —\n",
    "the same H100 that rented for around \\$8 per hour during the 2023 shortage\n",
    "goes for \\$2–4 on specialist clouds in mid-2026, and a consumer RTX 4090\n",
    "can be had for well under a dollar an hour. For calibration: the notebooks\n",
    "in this book are kept below 8 GB of GPU memory, so *any* rentable GPU with\n",
    "16 GB runs everything here with room to spare. It is working with large\n",
    "language models — fine-tuning beyond LoRA, serving long contexts,\n",
    "pretraining anything — that pushes you up the memory ladder, and the price\n",
    "ladder with it.\n",
    "\n",
    "The hard part of cloud computing is not clicking **Create instance**. It is\n",
    "knowing what the market offers, picking a machine that fits the workload\n",
    "rather than the marketing page, and running it with the discipline that\n",
    "disposable compute demands: results leave the machine, and the machine gets\n",
    "deleted.\n",
    "\n",
    "## The Rental Market\n",
    "\n",
    "### Three Tiers of Provider\n",
    "\n",
    "![Cloud options trade managed integration and governance for price dispersion and operational responsibility.](../img/tools-cloud-spectrum.svg)\n",
    "\n",
    "* **Hyperscalers** — AWS, Google Cloud, Microsoft Azure — sell GPUs\n",
    "  embedded in a full cloud: identity management, virtual networks, managed\n",
    "  storage, and every adjacent service. You pay for that integration; their\n",
    "  on-demand GPU prices are consistently the highest, and the largest\n",
    "  instances often hide behind quota requests or sales conversations. They\n",
    "  are the right answer when the data already lives there, when compliance\n",
    "  matters, or when a training job is one part of a bigger system. Google\n",
    "  additionally rents TPUs (a v5e chip for about \\$1.20 per hour), the\n",
    "  natural target for the JAX code in this book.\n",
    "* **GPU specialists** — Lambda, CoreWeave, Crusoe, Nebius, Voltage Park,\n",
    "  Together, and others — do one thing: accelerators with fast interconnect\n",
    "  and ML-ready images. Self-serve H100s run \\$2–4.30 per hour (July 2026);\n",
    "  Lambda and Nebius are fully self-serve, while CoreWeave publishes prices\n",
    "  but onboards through sales. This tier is the sweet spot for serious\n",
    "  training runs that do not need a hyperscaler's ecosystem.\n",
    "* **Marketplaces** — Vast.ai, RunPod, TensorDock, Prime Intellect, and the\n",
    "  auction-style SF Compute — aggregate machines from many independent\n",
    "  operators, including consumer GPUs that the big clouds do not carry. This\n",
    "  is where compute is cheapest: an RTX 4090 for \\$0.30–0.60 per hour, an\n",
    "  H100 from about \\$1.50. The catch is variance: host reliability, disk\n",
    "  speed, and network quality differ per listing, and your code runs on a\n",
    "  stranger's machine — fine for coursework and public data, inappropriate\n",
    "  for anything sensitive unless the platform's vetted tier is used.\n",
    "\n",
    "### What Things Cost\n",
    "\n",
    "Prices move quickly — the long arc is downward, though 2026's capacity\n",
    "crunch pushed some rates back up — so treat the following July 2026\n",
    "snapshot as a calibration, not a catalog. The *ratios* between tiers,\n",
    "however, have been stable for years:\n",
    "\n",
    "<!-- tbl-caption:GPU rental snapshot, on-demand (July 2026) -->\n",
    "\n",
    "| Provider (tier) | Cheap GPU option | ≈ \\$/hr | 1× H100 80 GB ≈ \\$/hr |\n",
    "|---|---|---|---|\n",
    "| Vast.ai (marketplace) | RTX 4090 24 GB | 0.35 | 1.50–1.90 |\n",
    "| RunPod (marketplace) | RTX 4090 24 GB | 0.35–0.70 | 2.90 (vetted hosts) |\n",
    "| Prime Intellect (marketplace) | RTX 4090 24 GB | 0.32 | 1.49 |\n",
    "| Lambda (specialist) | A100 40 GB | 1.99 | 3.99–4.29 |\n",
    "| Nebius (specialist) | RTX PRO 6000 48 GB | 1.80 | 3.85 |\n",
    "| Voltage Park (specialist) | H100 (only SKU) | 1.99 | 1.99 |\n",
    "| AWS (hyperscaler) | L4 24 GB (g6) | 0.80 | 6.88 (p5) |\n",
    "| Azure (hyperscaler) | A10 24 GB | 1.43 | 6.98 |\n",
    "| Google Cloud (hyperscaler) | L4 24 GB | 0.70 | ≈ 11 (A3) |\n",
    "\n",
    "Three practical notes. First, the free money: new accounts get trial\n",
    "credits (Google Cloud \\$300, Azure \\$100–200 for students, AWS \\$1,000\n",
    "through its startup program), which comfortably covers every experiment in\n",
    "this book many times over. Second, multi-GPU nodes price linearly per GPU\n",
    "on most providers, but the interconnect does not: an 8×H100 machine with\n",
    "NVLink is a qualitatively different tool from eight PCIe cards, and\n",
    "communication-heavy training (that section) will feel the\n",
    "difference. Third, newer is not always cheaper per unit of work: B200-class\n",
    "instances (\\$6–14 per GPU-hour) only pay off when you exploit their memory\n",
    "and FP8/FP4 throughput.\n",
    "\n",
    "### Cost per Result, Not per Hour\n",
    "\n",
    "The hourly price is the most visible term of a larger sum: setup time,\n",
    "idle time while you debug, storage that keeps billing after the run, data\n",
    "egress, and — the term engineers habitually forget — your own time. A\n",
    "faster, pricier GPU frequently wins on cost per *completed* experiment.\n",
    "The little model below is worth re-running with your own assumptions;\n",
    "here it compares a cheap marketplace card against two datacenter GPUs for\n",
    "the same eight-hour (on the slowest card) job:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "cloud-instances-cost-model"
   },
   "source": [
    "import numpy as np\n",
    "\n",
    "gpu = [\"RTX 4090\", \"A100 80GB\", \"H100\"]\n",
    "gpu_per_hour = np.array([0.40, 1.50, 2.50])   # marketplace, July 2026\n",
    "relative_speed = np.array([1.0, 1.6, 3.0])    # measure for your workload!\n",
    "setup_hours = np.array([0.5, 0.5, 0.5])\n",
    "storage_and_egress = np.array([2.0, 2.0, 2.0])\n",
    "engineer_per_hour = 60.0\n",
    "\n",
    "wall_hours = 8.0 / relative_speed\n",
    "invoice = (wall_hours + setup_hours) * gpu_per_hour + storage_and_egress\n",
    "complete = invoice + setup_hours * engineer_per_hour\n",
    "for row in zip(gpu, np.round(wall_hours, 1), np.round(invoice, 2),\n",
    "               np.round(complete, 2)):\n",
    "    print(row)"
   ],
   "execution_count": 1,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "('RTX 4090', np.float64(8.0), np.float64(5.4), np.float64(35.4))\n('A100 80GB', np.float64(5.0), np.float64(10.25), np.float64(40.25))\n('H100', np.float64(2.7), np.float64(9.92), np.float64(39.92))\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=cloud-instances-md-2 fw=all -->\n",
    "\n",
    "The invoice favors the cheap card; the complete cost is nearly a tie, and\n",
    "if slow iteration means one extra debugging round, the fast card wins. This\n",
    "is also why an unreliable \\$0.30 host can be the most expensive machine you\n",
    "ever rent. For scale: a LoRA fine-tune of a 7B model is a few hours on one\n",
    "consumer GPU — \\$3–15 total on a marketplace — while pretraining even a\n",
    "small LLM from scratch is thousands of GPU-hours. Know which regime you\n",
    "are in before optimizing pennies.\n",
    "\n",
    "Two cost traps deserve their own warnings:\n",
    "\n",
    "* **Spot and interruptible capacity** is 50–90% off, and it is the right\n",
    "  default for any job that checkpoints and resumes cleanly\n",
    "  (that section). It is the wrong discount for an\n",
    "  interactive session or an uncheckpointed run — eviction notice can be as\n",
    "  short as a few seconds on marketplace spot tiers. Test recovery *before*\n",
    "  the long run, not during it.\n",
    "* **Egress fees.** Hyperscalers charge roughly \\$0.09–0.12 per GB to move\n",
    "  data out. Re-downloading a 140 GB checkpoint daily costs more per month\n",
    "  than many GPUs. Keep data and compute in the same region, and prefer\n",
    "  providers with free egress (most marketplaces) when your workflow moves\n",
    "  big artifacts around.\n",
    "\n",
    "## Working on a Rented Machine\n",
    "\n",
    "### Boot, Connect, Verify\n",
    "\n",
    "Start from the provider's current deep-learning image — drivers, CUDA, and\n",
    "container runtime preinstalled. Hand-installing CUDA on a bare OS image is\n",
    "an afternoon you never get back, and reproducing it next month is another.\n",
    "Connect with the SSH-tunnel pattern of that section\n",
    "(the same two commands work on every provider), or use VS Code Remote SSH.\n",
    "Run long jobs under `tmux` so an SSH disconnect does not kill training, and\n",
    "verify the machine before trusting it with a long run:\n",
    "\n",
    "```bash\n",
    "nvidia-smi                 # driver ok? expected GPU? memory free?\n",
    "df -h                      # scratch disk has room for data + checkpoints?\n",
    "python -c \"import torch; print(torch.cuda.get_device_name(0))\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=cloud-instances-md-3 fw=all -->\n",
    "\n",
    "A one-minute smoke test — one batch through the model, one checkpoint\n",
    "written and read back — catches most of what the console's green checkmark\n",
    "does not: broken drivers, full disks, read-only mounts, and datasets that\n",
    "stream too slowly to keep the GPU busy.\n",
    "\n",
    "### Compute Is Disposable, Results Are Not\n",
    "\n",
    "![Provision compute, connect securely, checkpoint to durable storage, and delete the VM; resume from that storage on a replacement machine.](../img/tools-cloud-lifecycle.svg)\n",
    "\n",
    "The workflow in the figure separates the lifetime\n",
    "of the machine from the lifetime of your work. Code lives in Git and is\n",
    "cloned onto the instance; data and checkpoints sync to durable object\n",
    "storage (S3, GCS, or the provider's volume product) on a schedule the job\n",
    "controls; the instance itself can then be preempted, crashed, or deleted\n",
    "without losing more than the last checkpoint interval. This is not merely a\n",
    "safety practice — it is what makes spot pricing and marketplace hosts\n",
    "usable at all.\n",
    "\n",
    "When the experiment ends, tear down *everything that bills*: the instance,\n",
    "its disks and snapshots, reserved IP addresses, and stale buckets. Set a\n",
    "billing alert on day one; a forgotten idle GPU costs the same as a busy\n",
    "one, and quota limits are the only thing standing between a leaked\n",
    "credential and a very large invoice.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Every notebook in this book fits in 8 GB of GPU memory, so the cheapest\n",
    "  rentable GPUs suffice; LLM-scale work is what climbs the price ladder.\n",
    "* The market has three tiers — hyperscalers, GPU specialists, and\n",
    "  marketplaces — with roughly a 4× price spread for the same GPU and a\n",
    "  matching spread in integration, reliability, and trust.\n",
    "* Compare cost per completed result, not per hour: include setup, idle\n",
    "  time, storage, egress, and your own time.\n",
    "* Spot capacity is the right default for checkpointed jobs and wrong for\n",
    "  everything else; egress fees punish moving large artifacts.\n",
    "* Treat instances as disposable: provider image, SSH tunnel, `tmux`,\n",
    "  checkpoints to durable storage, then delete every billable resource.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Price a fine-tuning job you care about on three providers from\n",
    "   the table using cost per completed run. State the date\n",
    "   and your speed assumptions, then check how prices have moved since this\n",
    "   table was written.\n",
    "1. Take whichever training notebook of this book you ran most recently and\n",
    "   measure its actual GPU memory high-water mark. Which entries of\n",
    "   the table could run it?\n",
    "1. Simulate an interruption: start a checkpointed training run, kill the\n",
    "   process mid-epoch, and resume on a fresh machine from object storage.\n",
    "   Time the recovery and identify what you forgot to save.\n",
    "1. Draw the trust boundary for (a) a marketplace host processing a public\n",
    "   dataset and (b) a hyperscaler processing medical records. What changes?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-pytorch",
   "display_name": "d2l (PyTorch)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
