{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=pandas-md-1 fw=all -->\n",
    "\n",
    "# Data Preprocessing\n",
    "\n",
    "So far we have worked with synthetic data\n",
    "that arrived in ready-made tensors.\n",
    "In the wild, data shows up as messy files\n",
    "in arbitrary formats, and the path from a raw file\n",
    "to a model-ready tensor is paved with decisions:\n",
    "which values are missing and how to fill them,\n",
    "how to turn categories into numbers,\n",
    "whether features need rescaling.\n",
    "Get these wrong and even a perfect model learns nothing useful.\n",
    "The *pandas* [library](https://pandas.pydata.org/) [@McKinney.2010]\n",
    "does the heavy lifting; this section is a crash course\n",
    "on the routines you will reach for most:\n",
    "enough to load, clean, and encode a tabular dataset,\n",
    "and to understand *why* each step matters.\n",
    "For the full story, see the official\n",
    "[tutorial](https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.html).\n",
    "the figure previews the steps in this section.\n",
    "\n",
    "![The preprocessing pipeline: a raw file is loaded into a DataFrame, cleaned and encoded, and finally converted into the tensors a model consumes.](../img/pandas-pipeline.svg)\n",
    "\n",
    "## Load and Inspect\n",
    "\n",
    "### Reading the Dataset\n",
    "\n",
    "Comma-separated values (CSV) files are ubiquitous\n",
    "for storing tabular (spreadsheet-like) data.\n",
    "Each line is one record of several comma-separated fields, e.g.,\n",
    "\"Albert Einstein,March 14 1879,Ulm,Federal polytechnic school\".\n",
    "To demonstrate, we write a small dataset of homes to\n",
    "`../data/house_tiny.csv`. Each row is a home;\n",
    "the columns are the number of rooms (`NumRooms`),\n",
    "the roof type (`RoofType`), the floor area in square feet (`Area`),\n",
    "and the sale price (`Price`). An empty field denotes a missing entry."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "pandas-reading-the-dataset-1"
   },
   "source": [
    "import os\n",
    "\n",
    "os.makedirs(os.path.join('..', 'data'), exist_ok=True)\n",
    "data_file = os.path.join('..', 'data', 'house_tiny.csv')\n",
    "with open(data_file, 'w') as f:\n",
    "    f.write('''NumRooms,RoofType,Area,Price\n",
    "3,Slate,1500,210000\n",
    ",Tile,2100,290000\n",
    "2,,850,127500\n",
    "4,Slate,1940,258000\n",
    ",,1200,168000\n",
    "3,Tile,1650,225000\n",
    "5,Slate,2600,375000\n",
    "2,,900,142000\n",
    ",Tile,1750,240000\n",
    "4,,2050,295000''')"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=pandas-md-2 fw=all -->\n",
    "\n",
    "Now we import `pandas` and load the dataset with `read_csv`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "pandas-reading-the-dataset-2"
   },
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv(data_file)\n",
    "data"
   ],
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "   NumRooms RoofType  Area   Price\n0       3.0    Slate  1500  210000\n1       NaN     Tile  2100  290000\n2       2.0      NaN   850  127500\n3       4.0    Slate  1940  258000\n4       NaN      NaN  1200  168000\n5       3.0     Tile  1650  225000\n6       5.0    Slate  2600  375000\n7       2.0      NaN   900  142000\n8       NaN     Tile  1750  240000\n9       4.0      NaN  2050  295000"
     },
     "metadata": {},
     "execution_count": 2
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=pandas-md-3 fw=all -->\n",
    "\n",
    "Notice that `pandas` has replaced the empty fields\n",
    "with a special `NaN` (*not a number*) marker.\n",
    "These are *missing values*, and dealing with them\n",
    "is one of the central chores of data preprocessing.\n",
    "\n",
    "### Knowing Your Data\n",
    "\n",
    "Before transforming anything, look at it.\n",
    "A few one-liners save hours of debugging downstream.\n",
    "The **shape** tells you how much data you have;\n",
    "the **dtypes** tell you how `pandas` interpreted each column:\n",
    "numeric (`int64`/`float64`) versus non-numeric strings (`str`/`object`,\n",
    "i.e. categorical or text); and **`describe`** summarizes the numeric columns,\n",
    "surfacing ranges and obvious outliers at a glance."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "pandas-knowing-your-data-1"
   },
   "source": [
    "data.dtypes"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "NumRooms    float64\nRoofType        str\nArea          int64\nPrice         int64\ndtype: object"
     },
     "metadata": {},
     "execution_count": 3
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=pandas-md-4 fw=all -->\n",
    "\n",
    "`NumRooms` came in as `float64` (not `int`) precisely because it\n",
    "contains a `NaN`; `RoofType` is a string column, our cue that it is\n",
    "categorical and will need encoding. The numeric summary:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "pandas-knowing-your-data-2"
   },
   "source": [
    "data.describe()"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "       NumRooms         Area          Price\ncount  7.000000    10.000000      10.000000\nmean   3.285714  1654.000000  233050.000000\nstd    1.112697   556.421104   76128.710455\nmin    2.000000   850.000000  127500.000000\n25%    2.500000  1275.000000  178500.000000\n50%    3.000000  1700.000000  232500.000000\n75%    4.000000  2022.500000  282000.000000\nmax    5.000000  2600.000000  375000.000000"
     },
     "metadata": {},
     "execution_count": 4
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=pandas-md-5 fw=all -->\n",
    "\n",
    "The three numeric columns live on very different scales:\n",
    "rooms in the single digits, area in the thousands,\n",
    "price in the hundred-thousands. Keep that in mind; it returns below.\n",
    "\n",
    "### Separating Inputs and Targets\n",
    "\n",
    "In supervised learning we predict a *target* from a set of *inputs*.\n",
    "The first step is to split the columns accordingly so that we only\n",
    "ever preprocess the inputs, never peeking at or transforming the target\n",
    "as if it were a feature. Here the last column, `Price`, is the target;\n",
    "the rest are inputs. We select by integer position with `iloc`\n",
    "(selecting by column *name* works too, and is often clearer):"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "pandas-separating-inputs-and-targets-1"
   },
   "source": [
    "inputs, targets = data.iloc[:, :3], data.iloc[:, 3]\n",
    "inputs"
   ],
   "execution_count": 5,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "   NumRooms RoofType  Area\n0       3.0    Slate  1500\n1       NaN     Tile  2100\n2       2.0      NaN   850\n3       4.0    Slate  1940\n4       NaN      NaN  1200\n5       3.0     Tile  1650\n6       5.0    Slate  2600\n7       2.0      NaN   900\n8       NaN     Tile  1750\n9       4.0      NaN  2050"
     },
     "metadata": {},
     "execution_count": 5
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=pandas-md-6 fw=all -->\n",
    "\n",
    "## Clean and Encode\n",
    "\n",
    "### Handling Missing Values\n",
    "\n",
    "Missing values are unavoidable in real datasets, and how we handle them\n",
    "can change what a model learns.\n",
    "First, *measure* the problem: how many values are missing, and where?"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "pandas-handling-missing-values-1"
   },
   "source": [
    "inputs.isna().sum()"
   ],
   "execution_count": 6,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "NumRooms    3\nRoofType    4\nArea        0\ndtype: int64"
     },
     "metadata": {},
     "execution_count": 6
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=pandas-md-7 fw=all -->\n",
    "\n",
    "There are three broad ways to respond, each with a cost:\n",
    "\n",
    "* **Deletion:** drop rows (or columns) that contain a `NaN`. Simple,\n",
    "  but it throws data away and can bias the result if values are not\n",
    "  missing at random.\n",
    "* **Imputation:** fill each `NaN` with an estimate (a column statistic,\n",
    "  or a model's prediction). Keeps every row, but injects assumptions and\n",
    "  shrinks the data's variance.\n",
    "* **Indicator:** add a boolean \"was-missing\" column so the model can\n",
    "  learn from the *fact* that a value was absent.\n",
    "\n",
    "Deletion is tempting but expensive on small data. Dropping every row\n",
    "with any missing entry here leaves only a fraction of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "pandas-handling-missing-values-2"
   },
   "source": [
    "len(inputs.dropna()), len(inputs)"
   ],
   "execution_count": 7,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "(4, 10)"
     },
     "metadata": {},
     "execution_count": 7
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=pandas-md-8 fw=all -->\n",
    "\n",
    "Only four of ten rows survive: too wasteful. So we **impute** instead.\n",
    "For a numerical column the simplest estimate is the column **mean** (use\n",
    "the median when the column is skewed). `fillna` applies it. We impute the\n",
    "numerical columns now and leave the categorical `RoofType` for the next\n",
    "section, where treating \"missing\" as its own category is the natural fix."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "pandas-handling-missing-values-3"
   },
   "source": [
    "inputs = inputs.fillna(inputs.mean(numeric_only=True))\n",
    "inputs"
   ],
   "execution_count": 8,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "   NumRooms RoofType  Area\n0  3.000000    Slate  1500\n1  3.285714     Tile  2100\n2  2.000000      NaN   850\n3  4.000000    Slate  1940\n4  3.285714      NaN  1200\n5  3.000000     Tile  1650\n6  5.000000    Slate  2600\n7  2.000000      NaN   900\n8  3.285714     Tile  1750\n9  4.000000      NaN  2050"
     },
     "metadata": {},
     "execution_count": 8
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=pandas-md-9 fw=all -->\n",
    "\n",
    "`NumRooms` and `Area` are now filled with their column means;\n",
    "`RoofType`'s missing entries remain, to be handled by encoding.\n",
    "\n",
    "### Encoding Categorical Features\n",
    "\n",
    "Models consume numbers, so the `RoofType` strings must be encoded.\n",
    "The standard choice for a *nominal* category (one with no inherent\n",
    "order) is **one-hot encoding**: one new 0/1 column per category. We\n",
    "avoid mapping categories to integers like `Slate=0, Tile=1`, because\n",
    "that invents a false ordering and magnitude the model would wrongly\n",
    "exploit. `pd.get_dummies` does the conversion; `dummy_na=True` adds a\n",
    "column for the missing category, which is exactly the *indicator*\n",
    "strategy from the previous section, applied to a categorical column:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "pandas-encoding-categorical-features-1"
   },
   "source": [
    "inputs = pd.get_dummies(inputs, dummy_na=True, dtype=float)\n",
    "inputs"
   ],
   "execution_count": 9,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "   NumRooms  Area  RoofType_Slate  RoofType_Tile  RoofType_nan\n0  3.000000  1500             1.0            0.0           0.0\n1  3.285714  2100             0.0            1.0           0.0\n2  2.000000   850             0.0            0.0           1.0\n3  4.000000  1940             1.0            0.0           0.0\n4  3.285714  1200             0.0            0.0           1.0\n5  3.000000  1650             0.0            1.0           0.0\n6  5.000000  2600             1.0            0.0           0.0\n7  2.000000   900             0.0            0.0           1.0\n8  3.285714  1750             0.0            1.0           0.0\n9  4.000000  2050             0.0            0.0           1.0"
     },
     "metadata": {},
     "execution_count": 9
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=pandas-md-10 fw=all -->\n",
    "\n",
    "`RoofType` has become three columns (`RoofType_Slate`, `RoofType_Tile`,\n",
    "and `RoofType_nan`); the\n",
    "already-imputed numerical columns pass through untouched. Every column is\n",
    "now numeric.\n",
    "\n",
    "One caveat: one-hot encoding a *high-cardinality* column (thousands of\n",
    "distinct values, or an identifier unique to each row) explodes the\n",
    "feature count and is rarely useful. Such columns call for embeddings,\n",
    "which we meet in that section.\n",
    "\n",
    "## Scale and Convert\n",
    "\n",
    "### Numerical Features\n",
    "\n",
    "Every entry is now numeric and complete. As promised when `describe`\n",
    "first surfaced it, though, the continuous columns\n",
    "still span very different scales (`NumRooms` ≈ 3, `Area` ≈ 1700). Left\n",
    "as-is, the large-magnitude feature dominates distances and gradients,\n",
    "making optimization *ill-conditioned*: gradient descent zig-zags along\n",
    "the large-scale direction instead of heading straight for the minimum\n",
    "(that section treats conditioning\n",
    "in depth). The standard fix is\n",
    "**standardization**: subtract the mean and divide by the standard\n",
    "deviation, so each continuous feature has roughly zero mean and unit\n",
    "variance. (We leave the one-hot columns alone; they are already 0/1.)"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "pandas-numerical-features-1"
   },
   "source": [
    "continuous = ['NumRooms', 'Area']\n",
    "inputs[continuous] = (inputs[continuous] - inputs[continuous].mean()) \\\n",
    "                     / inputs[continuous].std()\n",
    "inputs"
   ],
   "execution_count": 10,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "   NumRooms      Area  RoofType_Slate  RoofType_Tile  RoofType_nan\n0 -0.314485 -0.276769             1.0            0.0           0.0\n1  0.000000  0.801551             0.0            1.0           0.0\n2 -1.415185 -1.444949             0.0            0.0           1.0\n3  0.786214  0.513999             1.0            0.0           0.0\n4  0.000000 -0.815929             0.0            0.0           1.0\n5 -0.314485 -0.007189             0.0            1.0           0.0\n6  1.886913  1.700151             1.0            0.0           0.0\n7 -1.415185 -1.355089             0.0            0.0           1.0\n8  0.000000  0.172531             0.0            1.0           0.0\n9  0.786214  0.711691             0.0            0.0           1.0"
     },
     "metadata": {},
     "execution_count": 10
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=pandas-md-11 fw=all -->\n",
    "\n",
    "A subtlety to flag now and revisit later: the statistics used to impute\n",
    "(the column means earlier) and to standardize (mean and standard\n",
    "deviation) must be computed on the **training** data only, then reused\n",
    "for validation and test data. Computing them over the whole dataset\n",
    "leaks information about the test set into training and inflates your\n",
    "reported performance. We return to training/validation/test splits in\n",
    "that section and practice leak-free preprocessing\n",
    "on a real dataset in that section.\n",
    "\n",
    "### Conversion to Tensor Format\n",
    "\n",
    "Now that `inputs` and `targets` are entirely numeric, we can load them\n",
    "into tensors (recall that section). Pandas hands off to NumPy\n",
    "via `to_numpy`, and the tensor constructor wraps that array. Deep learning\n",
    "typically uses 32-bit floats; `to_numpy(dtype=float)` gives 64-bit, so\n",
    "in real pipelines you would cast down to `float32` to save memory and\n",
    "match the framework defaults."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "pandas-conversion-to-tensor-format"
   },
   "source": [
    "from jax import numpy as jnp\n",
    "\n",
    "X = jnp.array(inputs.to_numpy(dtype=float))\n",
    "y = jnp.array(targets.to_numpy(dtype=float))\n",
    "X, y"
   ],
   "execution_count": 11,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "(Array([[-0.31448546, -0.2767688 ,  1.        ,  0.        ,  0.        ],\n        [ 0.        ,  0.8015512 ,  0.        ,  1.        ,  0.        ],\n        [-1.4151845 , -1.4449488 ,  0.        ,  0.        ,  1.        ],\n        [ 0.78621364,  0.5139992 ,  1.        ,  0.        ,  0.        ],\n        [ 0.        , -0.8159288 ,  0.        ,  0.        ,  1.        ],\n        [-0.31448546, -0.0071888 ,  0.        ,  1.        ,  0.        ],\n        [ 1.8869127 ,  1.7001512 ,  1.        ,  0.        ,  0.        ],\n        [-1.4151845 , -1.3550888 ,  0.        ,  0.        ,  1.        ],\n        [ 0.        ,  0.1725312 ,  0.        ,  1.        ,  0.        ],\n        [ 0.78621364,  0.7116912 ,  0.        ,  0.        ,  1.        ]],      dtype=float32),\n Array([210000., 290000., 127500., 258000., 168000., 225000., 375000.,\n        142000., 240000., 295000.], dtype=float32))"
     },
     "metadata": {},
     "execution_count": 11
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=pandas-md-12 fw=all -->\n",
    "\n",
    "From here on we work with tensors, ready for gradients and GPUs.\n",
    "\n",
    "## Discussion\n",
    "\n",
    "You now know the core steps: inspect a dataset, separate inputs from\n",
    "targets, handle missing values, encode categoricals, scale numerics, and\n",
    "hand the result to a framework. You will pick up more data-processing\n",
    "skills in that section. This crash course kept things\n",
    "deliberately simple; real preprocessing is messier.\n",
    "\n",
    "Rather than a single CSV, a dataset may be spread across many files\n",
    "pulled from a relational database (customer addresses in one table,\n",
    "purchases in another) that must be joined first. Beyond categorical and\n",
    "numeric fields, practitioners face text, images, audio, and point\n",
    "clouds, each needing its own pipeline (we meet these in the computer\n",
    "vision and natural language processing chapters). For data that does\n",
    "not fit in memory, pandas is the wrong tool; columnar formats like\n",
    "Parquet and out-of-core engines take over. And throughout, watch data\n",
    "*quality*: outliers and recording errors must be caught before\n",
    "training, and plotting a column with a library such as\n",
    "[matplotlib](https://matplotlib.org/) is often the fastest way to\n",
    "spot them. For the transforms in this section (imputation, encoding,\n",
    "scaling) and for chaining them into reproducible pipelines, see\n",
    "`sklearn.impute` and `sklearn.preprocessing`.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Load a real dataset, e.g., Abalone from the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets), and inspect its properties. What fraction of values are missing? What fraction of the columns are numerical, categorical, or text?\n",
    "1. Select data columns by *name* rather than by integer position. The pandas docs on [indexing](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html) describe the options.\n",
    "1. Compare imputation strategies on `NumRooms`: mean, median, and adding a \"was-missing\" indicator column. Which assumptions does each make? When would you prefer one over another?\n",
    "1. We standardized using statistics from the whole dataset. Why is that a form of information leakage, and how should you compute the mean and standard deviation instead? What goes wrong if a feature has zero variance?\n",
    "1. How large a dataset could you load this way? Consider read time, in-memory representation, and processing. Try it on your laptop, then on a larger machine. What breaks first?\n",
    "1. How would you handle a categorical column with a very large number of categories? What if every label is unique: should you include the column at all?\n",
    "1. What alternatives to pandas can you think of? Consider [loading NumPy arrays from a file](https://numpy.org/doc/stable/reference/generated/numpy.load.html) and the image library [Pillow](https://python-pillow.org/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=pandas-md-13 fw=jax -->\n",
    "\n",
    "[Discussions](https://d2l.discourse.group/t/17967)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-jax",
   "display_name": "d2l (JAX)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
