{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=synthetic-regression-data-md-1 fw=all -->\n",
    "\n",
    "# Synthetic Regression Data\n",
    "\n",
    "\n",
    "Before we can train a model we need data.\n",
    "Real datasets are what we ultimately care about,\n",
    "but they conflate three separate sources of failure:\n",
    "a misspecified model, a flawed optimization algorithm,\n",
    "and pathological data.\n",
    "When a method performs poorly on real data,\n",
    "all three explanations remain on the table at once.\n",
    "*Synthetic data* removes this ambiguity by construction.\n",
    "If we know the data-generating process exactly\n",
    "(the true weights $\\mathbf{w}^*$, the true bias $b^*$,\n",
    "and the noise distribution),\n",
    "then any *systematic* failure to recover them\n",
    "(beyond the irreducible noise) points to the algorithm\n",
    "or implementation.\n",
    "This is why synthetic datasets are the first test\n",
    "for any new learning method.\n",
    "We confirm that it solves a problem with a known answer\n",
    "before we ever hand it a real one."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "synthetic-regression-data"
   },
   "source": [
    "%matplotlib inline\n",
    "from d2l import torch as d2l\n",
    "import torch\n",
    "import random"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=synthetic-regression-data-md-2 fw=all -->\n",
    "\n",
    "## Generating the Dataset\n",
    "\n",
    "For this example, we will work in low dimension\n",
    "for succinctness.\n",
    "The following code snippet generates 1000 examples\n",
    "with 2-dimensional features drawn \n",
    "from a standard normal distribution.\n",
    "The resulting design matrix $\\mathbf{X}$\n",
    "belongs to $\\mathbb{R}^{1000 \\times 2}$. \n",
    "We generate each label by applying \n",
    "a *ground truth* linear function, \n",
    "corrupting them via additive noise $\\boldsymbol{\\epsilon}$, \n",
    "drawn independently and identically for each example:\n",
    "\n",
    "$$\\mathbf{y}= \\mathbf{X} \\mathbf{w}^* + b^* + \\boldsymbol{\\epsilon}.$$\n",
    "\n",
    "For convenience we assume that $\\boldsymbol{\\epsilon}$ is drawn \n",
    "from a normal distribution with mean $\\mu= 0$ \n",
    "and standard deviation $\\sigma = 0.01$.\n",
    "We put the generation code in the `__init__` method of a subclass\n",
    "of `d2l.DataModule` (introduced in that section),\n",
    "calling `save_hyperparameters()` so that every constructor argument\n",
    "(the parameters `w` and `b`, the noise level, the split sizes, and\n",
    "`batch_size`) is stored as an attribute and the dataset stays\n",
    "introspectable."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "synthetic-regression-data-generating-the-dataset-1"
   },
   "source": [
    "class SyntheticRegressionData(d2l.DataModule):\n",
    "    \"\"\"Synthetic data for linear regression.\"\"\"\n",
    "    def __init__(self, w, b, noise=0.01, num_train=1000, num_val=1000, \n",
    "                 batch_size=32):\n",
    "        super().__init__()\n",
    "        self.save_hyperparameters()\n",
    "        n = num_train + num_val\n",
    "        self.X = d2l.randn(n, len(w))\n",
    "        eps = d2l.randn(n, 1) * noise\n",
    "        self.y = d2l.matmul(self.X, d2l.reshape(w, (-1, 1))) + b + eps"
   ],
   "execution_count": 2,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=synthetic-regression-data-md-3 fw=all -->\n",
    "\n",
    "Below, we set the true parameters to $\\mathbf{w}^* = [2, -3.4]^\\top$ and $b^* = 4.2$.\n",
    "Later, we can check our estimated parameters against these *ground truth* values."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "synthetic-regression-data-generating-the-dataset-2"
   },
   "source": [
    "data = SyntheticRegressionData(w=d2l.tensor([2, -3.4]), b=4.2)"
   ],
   "execution_count": 3,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=synthetic-regression-data-md-4 fw=all -->\n",
    "\n",
    "Each row of `data.X` is a feature vector in $\\mathbb{R}^2$ and each row of `data.y` is a scalar label. Let's have a look at the first entry."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "synthetic-regression-data-generating-the-dataset-3"
   },
   "source": [
    "print('features:', data.X[0],'\\nlabel:', data.y[0])"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "features: tensor([-1.2309,  0.6609]) \nlabel: tensor([-0.5205])\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=synthetic-regression-data-md-5 fw=all -->\n",
    "\n",
    "## Reading the Dataset\n",
    "\n",
    "Training machine learning models often requires multiple passes over a dataset, \n",
    "grabbing one minibatch of examples at a time. \n",
    "This data is then used to update the model. \n",
    "To illustrate how this works, we \n",
    "implement the `get_dataloader` method, \n",
    "registering it in the `SyntheticRegressionData` class via `add_to_class` (introduced in that section).\n",
    "It takes a batch size, a matrix of features,\n",
    "and a vector of labels, and generates minibatches of size `batch_size`.\n",
    "As such, each minibatch consists of a tuple of features and labels. \n",
    "Note that we need to be mindful of whether we're in training or validation mode: \n",
    "in the former, we will want to read the data in random order, \n",
    "whereas for the latter, being able to read data in a pre-defined order \n",
    "may be important for debugging purposes."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "synthetic-regression-data-reading-the-dataset-1"
   },
   "source": [
    "@d2l.add_to_class(SyntheticRegressionData)\n",
    "def get_dataloader(self, train):\n",
    "    if train:\n",
    "        indices = list(range(0, self.num_train))\n",
    "        # The examples are read in random order\n",
    "        random.shuffle(indices)\n",
    "    else:\n",
    "        indices = list(range(self.num_train, self.num_train+self.num_val))\n",
    "    for i in range(0, len(indices), self.batch_size):\n",
    "        batch_indices = d2l.tensor(indices[i: i+self.batch_size])\n",
    "        yield self.X[batch_indices], self.y[batch_indices]"
   ],
   "execution_count": 5,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=synthetic-regression-data-md-6 fw=all -->\n",
    "\n",
    "To build some intuition, let's inspect the first minibatch of\n",
    "data. Each minibatch of features provides us with both its size and the dimensionality of input features.\n",
    "Likewise, our minibatch of labels will have a matching shape given by `batch_size`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "synthetic-regression-data-reading-the-dataset-2"
   },
   "source": [
    "X, y = next(iter(data.train_dataloader()))\n",
    "print('X shape:', X.shape, '\\ny shape:', y.shape)"
   ],
   "execution_count": 6,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "X shape: torch.Size([32, 2]) \ny shape: torch.Size([32, 1])\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=synthetic-regression-data-md-7 fw=all -->\n",
    "\n",
    "Iterating over `data.train_dataloader()` yields distinct minibatches\n",
    "until the dataset is exhausted (try it).\n",
    "Writing the loader by hand makes every step explicit,\n",
    "but it costs us in three ways:\n",
    "all of the data must fit in memory, the iteration is single-threaded\n",
    "Python looping over indices, and there is no prefetching to overlap\n",
    "data loading with computation on the previous batch.\n",
    "The data loaders built into a deep learning framework fix all three.\n",
    "They run several worker processes in parallel, prefetch the next batch\n",
    "while the current one trains, and stream from sources such as files,\n",
    "network streams, or generators that produce data on the fly.\n",
    "We now switch to the framework's built-in loader,\n",
    "which presents an identical interface to the caller.\n",
    "\n",
    "## Concise Implementation of the Data Loader\n",
    "\n",
    "Rather than writing our own iterator,\n",
    "we can call the existing API in a framework to load data.\n",
    "As before, we need a dataset with features `X` and labels `y`. \n",
    "Beyond that, we set `batch_size` in the built-in data loader \n",
    "and let it take care of shuffling examples  efficiently."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "synthetic-regression-data-concise-implementation-of-the-data-loader-1"
   },
   "source": [
    "@d2l.add_to_class(d2l.DataModule)\n",
    "def get_tensorloader(self, tensors, train, indices=slice(0, None)):\n",
    "    tensors = tuple(a[indices] for a in tensors)\n",
    "    dataset = torch.utils.data.TensorDataset(*tensors)\n",
    "    return torch.utils.data.DataLoader(dataset, self.batch_size,\n",
    "                                       shuffle=train)"
   ],
   "execution_count": 7,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "synthetic-regression-data-concise-implementation-of-the-data-loader-2"
   },
   "source": [
    "@d2l.add_to_class(SyntheticRegressionData)\n",
    "def get_dataloader(self, train):\n",
    "    i = slice(0, self.num_train) if train else slice(self.num_train, None)\n",
    "    return self.get_tensorloader((self.X, self.y), train, i)"
   ],
   "execution_count": 8,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=synthetic-regression-data-md-8 fw=all -->\n",
    "\n",
    "The new data loader behaves just like the previous one, except that it is more efficient and has some added functionality."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "synthetic-regression-data-concise-implementation-of-the-data-loader-3"
   },
   "source": [
    "X, y = next(iter(data.train_dataloader()))\n",
    "print('X shape:', X.shape, '\\ny shape:', y.shape)"
   ],
   "execution_count": 9,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "X shape: torch.Size([32, 2]) \ny shape: torch.Size([32, 1])\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=synthetic-regression-data-md-9 fw=all -->\n",
    "\n",
    "For instance, the data loader provided by the framework API \n",
    "supports the built-in `__len__` method, \n",
    "so we can query its length, \n",
    "i.e., the number of batches."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "synthetic-regression-data-concise-implementation-of-the-data-loader-4"
   },
   "source": [
    "len(data.train_dataloader())"
   ],
   "execution_count": 10,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "32"
     },
     "metadata": {},
     "execution_count": 10
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=synthetic-regression-data-md-10 fw=all -->\n",
    "\n",
    "With 1000 training examples and a batch size of 32, we expect\n",
    "$\\lceil 1000 / 32 \\rceil = 32$ batches: 31 full ones and a final\n",
    "partial batch of 8 examples.\n",
    "Note also that the built-in training loader *reshuffles* the examples at\n",
    "the start of every epoch, just as our hand-rolled loader drew a fresh\n",
    "random order on each call; exercise 8 of that section\n",
    "asks why this reshuffling matters.\n",
    "\n",
    "\n",
    "## Summary\n",
    "\n",
    "Synthetic data lets us check the recovered parameters against the truth\n",
    "we fixed: because we chose $\\mathbf{w}^*$ and $b^*$ ourselves, we can see\n",
    "after training whether the estimates agree, which makes such datasets\n",
    "the first place to validate any new algorithm.\n",
    "The `SyntheticRegressionData` class introduced here packages this\n",
    "data-generating process as a `DataModule` subclass, separating *where\n",
    "the batches come from* from *how a model consumes them*.\n",
    "Along the way we implemented the same `get_dataloader` protocol twice:\n",
    "a transparent hand-rolled iterator that is easy to read but loads\n",
    "everything in memory and loops in Python, and a framework-native loader\n",
    "that shuffles, prefetches, and parallelizes for us.\n",
    "The hand-rolled version is there to teach; the framework version is what\n",
    "we use from here on.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. When the number of examples is not divisible by the batch size, the loaders above keep the final partial batch. What does PyTorch's `drop_last` argument (and its TensorFlow counterpart, `batch(..., drop_remainder=...)`) do, and when would you want to enable it?\n",
    "1. Suppose that we want to generate a huge dataset, where both the size of the parameter vector `w` and the number of examples `num_examples` are large.\n",
    "    1. What happens if we cannot hold all data in memory?\n",
    "    1. How would you shuffle the data if it is held on disk? Your task is to design an *efficient* algorithm that does not require too many random reads or writes. Hint: [pseudorandom permutation generators](https://en.wikipedia.org/wiki/Pseudorandom_permutation) allow you to design a reshuffle without the need to store the permutation table explicitly [@Naor.Reingold.1999]. \n",
    "1. Implement a data generator that produces new data on the fly, every time the iterator is called. \n",
    "1. **(Reproducibility.)** How would you design a random data generator that\n",
    "   produces the *same* dataset every time it is called? Libraries differ here:\n",
    "   some expose a single global seed, while others thread an explicit random\n",
    "   state through every draw. For the threaded style, explain why fixing that\n",
    "   state up front makes the result reproducible, and why re-using the *same*\n",
    "   state for both $\\mathbf{X}$ and $\\boldsymbol{\\epsilon}$ (rather than\n",
    "   advancing it between the two draws) would be a bug.\n",
    "1. **(Signal-to-noise and recovery.)** Vary the noise standard deviation `noise`\n",
    "   over $\\{0.001, 0.01, 0.1, 0.5, 1.0\\}$. After fitting a linear model on each\n",
    "   dataset (using the code from that section or\n",
    "   that section), how closely does the estimate\n",
    "   $\\hat{\\mathbf{w}}$ match the true $\\mathbf{w}^* = [2, -3.4]^\\top$? Plot the\n",
    "   error $\\|\\hat{\\mathbf{w}} - \\mathbf{w}^*\\|_2$ as a function of $\\sigma$. How do\n",
    "   you expect it to scale with $\\sigma$ and with the number of training examples,\n",
    "   and does the experiment agree?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=synthetic-regression-data-md-11 fw=pytorch -->\n",
    "\n",
    "[Discussions](https://d2l.discourse.group/t/6663)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-pytorch",
   "display_name": "d2l (PyTorch)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
