{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=software-ecosystem-md-1 fw=all -->\n",
    "\n",
    "# Ecosystem\n",
    "\n",
    "Almost nothing in modern machine learning starts from an empty directory.\n",
    "You begin from someone else's work: a pretrained checkpoint, a curated\n",
    "dataset, a paper with released code, a benchmark that tells you what \"good\"\n",
    "currently means. Knowing *where to look* — and how to judge what you find —\n",
    "has quietly become a core skill, as load-bearing as knowing how to write a\n",
    "training loop. This section is the map we wish someone had handed us: the\n",
    "places that matter in 2026, the leaderboards worth trusting, how to stay\n",
    "current in a field that turns over monthly, and the handful of habits that\n",
    "keep borrowed artifacts from becoming liabilities.\n",
    "\n",
    "## Where to Find Things\n",
    "\n",
    "### Models\n",
    "\n",
    "The [Hugging Face Hub](https://huggingface.co/models) is the default: over\n",
    "two million public models as of spring 2026, with Git-style versioning,\n",
    "model cards, and integration into effectively every library in this book.\n",
    "Its scale cuts both ways — the top 200 repositories account for about half\n",
    "of all downloads, and most of the rest is abandoned experiments — so search\n",
    "by task, sort by downloads and recency, and read the model card before\n",
    "committing bandwidth. Complements worth knowing:\n",
    "\n",
    "* [ModelScope](https://modelscope.cn/), Alibaba's hub, is the primary home\n",
    "  of the Chinese open-model ecosystem (Qwen and friends, 170,000+ models) —\n",
    "  increasingly where open-weight state of the art appears first.\n",
    "* [Kaggle Models](https://www.kaggle.com/models) hosts curated weights\n",
    "  wired into Kaggle's competition and notebook infrastructure\n",
    "  (that section).\n",
    "* The [Ollama library](https://ollama.com/library) is a short, curated\n",
    "  menu of local-runtime models — less a discovery surface than a\n",
    "  convenience layer (that section).\n",
    "* [`timm`](https://huggingface.co/timm) remains the reference collection\n",
    "  of vision backbones; [Civitai](https://civitai.com/) dominates community\n",
    "  image-generation checkpoints and LoRAs.\n",
    "\n",
    "### Datasets\n",
    "\n",
    "[Hugging Face Datasets](https://huggingface.co/datasets) (half a million\n",
    "public datasets) and [Kaggle](https://www.kaggle.com/datasets) cover most\n",
    "supervised needs. For pretraining-scale text, the lineage runs from raw\n",
    "[Common Crawl](https://commoncrawl.org/) through filtered derivatives —\n",
    "[FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) (~15\n",
    "trillion tokens, plus a 1,000-language successor) has become the standard\n",
    "open baseline, the role C4 played in this book's era of BERT. For\n",
    "vision–language pairs, DataComp superseded LAION as the recommended\n",
    "starting point. Older and very large corpora sometimes live only on\n",
    "[Academic Torrents](https://academictorrents.com/). Whatever the source:\n",
    "datasets have versions, licenses, and documented failure modes exactly as\n",
    "models do, and the data card deserves the same read as the model card.\n",
    "\n",
    "### Papers and Code\n",
    "\n",
    "New work appears on [arXiv](https://arxiv.org/list/cs.LG/recent) first;\n",
    "the community's curated front page for ML is [Hugging Face\n",
    "Papers](https://huggingface.co/papers), which absorbed that role when\n",
    "Papers with Code — for years the standard index from papers to code and\n",
    "benchmarks — was shut down without notice in mid-2025. (Its historical\n",
    "leaderboard data survives only as a frozen archive; let that be a lesson\n",
    "about free infrastructure.) [Semantic Scholar](https://www.semanticscholar.org/)\n",
    "and [alphaXiv](https://alphaxiv.org/) help with search and discussion, and\n",
    "GitHub remains where the code actually is — a repository's issue tracker\n",
    "and commit recency tell you more about whether a method reproduces than\n",
    "the paper's abstract does.\n",
    "\n",
    "## Choosing a Model: Benchmarks and Leaderboards\n",
    "\n",
    "\"Which model should I use?\" changed answers a dozen times while this book\n",
    "was written; what stays stable is *how* to answer it. No single number\n",
    "survived contact with optimization — static benchmark suites become\n",
    "training data, and the once-canonical Open LLM Leaderboard was retired in\n",
    "2025 for exactly that reason. Practitioners now triangulate:\n",
    "\n",
    "* [LMArena](https://lmarena.ai/) (now Arena) — blind human pairwise\n",
    "  preference; hard to game, but measures \"pleasing an average user,\" which\n",
    "  may not be your task.\n",
    "* [LiveBench](https://livebench.ai/) — contamination-resistant by rotating\n",
    "  fresh questions monthly.\n",
    "* [SWE-bench Verified](https://www.swebench.com/) — the reference for\n",
    "  agentic coding against real GitHub issues; representative of the\n",
    "  benchmark-per-capability pattern (math, long context, safety all have\n",
    "  their own).\n",
    "* [Artificial Analysis](https://artificialanalysis.ai/) — the\n",
    "  quality/price/latency triangulation across hundreds of models and\n",
    "  providers; usually the first stop when cost matters.\n",
    "* [OpenRouter rankings](https://openrouter.ai/rankings) — revealed\n",
    "  preference by real token volume rather than scores; instructive\n",
    "  precisely where it disagrees with the quality leaderboards.\n",
    "\n",
    "Then the step that outranks all of the above: build a small evaluation set\n",
    "from *your* task — even fifty examples — and run the shortlist on it. The\n",
    "gap between leaderboard rank and performance on your distribution is\n",
    "routinely larger than the gap between adjacent leaderboard entries. The\n",
    "evaluation discipline this book has practiced throughout — held-out data,\n",
    "meaningful baselines, error analysis — applies to *choosing* models exactly\n",
    "as it does to training them.\n",
    "\n",
    "## Staying Current\n",
    "\n",
    "A field this fast rewards a deliberate information diet over doomscrolling.\n",
    "A workable minimal set, as of 2026:\n",
    "\n",
    "* [r/LocalLLaMA](https://www.reddit.com/r/LocalLLaMA/) — the town square\n",
    "  of open-weight ML; new releases, quantizations, and hardware reports\n",
    "  surface here first, usually with reproduction attempts attached.\n",
    "* [Hugging Face Papers](https://huggingface.co/papers) daily — a curated\n",
    "  dozen papers instead of arXiv's daily hundreds.\n",
    "* One good newsletter — Andrew Ng's *The Batch*, Jack Clark's *Import AI*,\n",
    "  or Sebastian Raschka's *Ahead of AI* — for the weekly synthesis.\n",
    "* For systems depth, the [GPU MODE](https://github.com/gpu-mode/lectures)\n",
    "  lecture series and community, and the\n",
    "  [EleutherAI Discord](https://www.eleuther.ai/community) for open\n",
    "  research.\n",
    "\n",
    "X/Twitter remains where labs announce and researchers argue; treat it as a\n",
    "discovery feed, not an archive. And the chapter-end\n",
    "resources of this book (that section) collect the\n",
    "durable long-form references.\n",
    "\n",
    "## Using What You Found\n",
    "\n",
    "### Pin the Identity\n",
    "\n",
    "![A reusable model artifact combines configuration, weights, preprocessing, documentation, and an immutable revision.](../img/tools-ecosystem-artifact.svg)\n",
    "\n",
    "A model is more than its weights: the tokenizer, preprocessing,\n",
    "configuration, license, and revision all determine whether you can\n",
    "reproduce a result (the figure). Repositories are\n",
    "Git repositories — `main` moves. Pin the commit:\n",
    "\n",
    "```text\n",
    "from huggingface_hub import snapshot_download\n",
    "\n",
    "path = snapshot_download(\n",
    "    repo_id=\"organization/model-name\",\n",
    "    revision=\"0123456789abcdef\",          # an immutable commit, not main\n",
    "    allow_patterns=[\"*.json\", \"*.safetensors\"],\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=software-ecosystem-md-2 fw=all -->\n",
    "\n",
    "A tiny manifest turns \"which model was that?\" from archaeology into a\n",
    "lookup — in production you would add file hashes, library versions, and an\n",
    "evaluation record:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "software-ecosystem-manifest"
   },
   "source": [
    "from dataclasses import asdict, dataclass\n",
    "import json\n",
    "\n",
    "@dataclass(frozen=True)\n",
    "class Artifact:\n",
    "    repository: str\n",
    "    revision: str\n",
    "    task: str\n",
    "    license: str\n",
    "    parent: str | None = None\n",
    "\n",
    "artifact = Artifact(repository=\"organization/model-name\",\n",
    "                    revision=\"0123456789abcdef\",\n",
    "                    task=\"text-generation\", license=\"apache-2.0\")\n",
    "print(json.dumps(asdict(artifact), indent=2))"
   ],
   "execution_count": 1,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "{\n  \"repository\": \"organization/model-name\",\n  \"revision\": \"0123456789abcdef\",\n  \"task\": \"text-generation\",\n  \"license\": \"apache-2.0\",\n  \"parent\": null\n}\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=software-ecosystem-md-3 fw=all -->\n",
    "\n",
    "The same pinning discipline applies to *derived* artifacts: a LoRA adapter\n",
    "without its base-model revision is incomplete, and a quantized conversion\n",
    "(GGUF, AWQ, ONNX — see that section) is a new artifact\n",
    "whose numerical fidelity someone should have checked against the source.\n",
    "\n",
    "### Trust and Licenses\n",
    "\n",
    "Downloaded models are software supply-chain inputs, and the ecosystem has\n",
    "real teeth:\n",
    "\n",
    "* Prefer **safetensors** — a pure tensor container. Legacy pickle-based\n",
    "  checkpoints can execute arbitrary code on load, and the Hub's scanners\n",
    "  are a mitigation, not a guarantee.\n",
    "* `trust_remote_code=True` runs Python from the repository on your\n",
    "  machine. Read and pin that code; use the flag only when the architecture\n",
    "  genuinely requires it.\n",
    "* \"Open\" spans a wide range of licenses: permissive Apache/MIT weights,\n",
    "  acceptable-use licenses with commercial thresholds, research-only\n",
    "  releases, and gated models whose terms you accept per account. The\n",
    "  license lives in the repository — record the one you actually accepted,\n",
    "  since repositories can relicense between revisions.\n",
    "* Every hub client caches aggressively (tens to hundreds of gigabytes in\n",
    "  `~/.cache/huggingface` is routine). Learn your cache tool's `scan` and\n",
    "  `delete` commands before your disk teaches you.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Discovery is a skill: Hugging Face is the center for models and\n",
    "  datasets, with ModelScope, Kaggle, Ollama's library, and Civitai as the\n",
    "  complements that matter; FineWeb-class corpora are the open pretraining\n",
    "  baseline.\n",
    "* Papers with Code is gone; arXiv plus Hugging Face Papers is the 2026\n",
    "  reading pipeline, with GitHub as the ground truth for whether code\n",
    "  exists and is alive.\n",
    "* No leaderboard is trusted alone: triangulate Arena, LiveBench,\n",
    "  task-specific benchmarks, and price/latency data — then decide on a\n",
    "  small evaluation you built from your own task.\n",
    "* Keep a deliberate information diet: one community, one curated paper\n",
    "  feed, one newsletter.\n",
    "* Pin revisions, prefer safetensors, read licenses, treat remote code as\n",
    "  code review, and manage your caches — borrowed artifacts are supply\n",
    "  chain, not just downloads.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Pick a task you care about and shortlist three models using at least\n",
    "   two leaderboards plus Artificial Analysis. Where do the rankings\n",
    "   disagree, and why might that be?\n",
    "1. Inspect a model repository of your choice and list every file needed\n",
    "   for offline inference. Which of them could execute code on your\n",
    "   machine?\n",
    "1. Build a 25-example evaluation set for a task you know well and run\n",
    "   your shortlist from Exercise 1 on it. Does your ranking match the\n",
    "   leaderboards'?\n",
    "1. Find the license of a popular open-weight model and determine: may you\n",
    "   deploy it commercially, fine-tune it, and redistribute the fine-tune?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-pytorch",
   "display_name": "d2l (PyTorch)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
