{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=kaggle-house-price-md-1 fw=all -->\n",
    "\n",
    "# Predicting House Prices on Kaggle\n",
    "\n",
    "Now that we have introduced some basic tools\n",
    "for building and training deep networks\n",
    "and regularizing them with techniques including\n",
    "weight decay and dropout,\n",
    "we are ready to put all this knowledge into practice\n",
    "by participating in a Kaggle competition.\n",
    "The house price prediction competition\n",
    "is a great place to start.\n",
    "The data is fairly generic and does not exhibit exotic structure\n",
    "that might require specialized models (as audio or video might).\n",
    "This dataset, collected by @De-Cock.2011,\n",
    "covers house prices in Ames, Iowa from the period 2006--2010.\n",
    "It was assembled as a modern, larger alternative to the small classic\n",
    "teaching datasets (e.g. Boston Housing) that preceded it, with more\n",
    "examples and more features.\n",
    "\n",
    "\n",
    "In this section, we will walk you through details of\n",
    "data preprocessing, model design, and hyperparameter selection.\n",
    "We hope that through a hands-on approach,\n",
    "you will gain some intuitions that will guide you\n",
    "in your career as a data scientist."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "kaggle-house-price-predicting-house-prices-on-kaggle"
   },
   "source": [
    "%matplotlib inline\n",
    "from d2l import mxnet as d2l\n",
    "from mxnet import gluon, autograd, init, np, npx\n",
    "from mxnet.gluon import nn\n",
    "import pandas as pd\n",
    "\n",
    "npx.set_np()"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=kaggle-house-price-md-2 fw=all -->\n",
    "\n",
    "## Kaggle\n",
    "\n",
    "[Kaggle](https://www.kaggle.com) is a popular platform\n",
    "that hosts machine learning competitions.\n",
    "Each competition centers on a dataset and many\n",
    "are sponsored by stakeholders who offer prizes\n",
    "to the winning solutions.\n",
    "The platform helps users to interact\n",
    "via forums and shared code,\n",
    "fostering both collaboration and competition.\n",
    "While leaderboard chasing often spirals out of control,\n",
    "with researchers focusing myopically on preprocessing steps\n",
    "rather than asking fundamental questions,\n",
    "there is also value in the objectivity of a platform\n",
    "that facilitates direct quantitative comparisons\n",
    "among competing approaches as well as code sharing\n",
    "so that everyone can learn what did and did not work.\n",
    "If you want to participate in a Kaggle competition,\n",
    "you will first need to register for an account\n",
    "(see the figure).\n",
    "\n",
    "![The Kaggle website.](../img/kaggle.png)\n",
    "\n",
    "On the house price prediction competition page, as illustrated\n",
    "in the figure,\n",
    "you can find the dataset (under the \"Data\" tab),\n",
    "submit predictions, and see your ranking,\n",
    "The URL is right here:\n",
    "\n",
    "> https://www.kaggle.com/c/house-prices-advanced-regression-techniques\n",
    "\n",
    "![The house price prediction competition page.](../img/house-pricing.png)\n",
    "\n",
    "## Accessing and Reading the Dataset\n",
    "\n",
    "Note that the competition data is separated\n",
    "into training and test sets.\n",
    "Each record includes the property value of the house\n",
    "and attributes such as street type, year of construction,\n",
    "roof type, basement condition, etc.\n",
    "The features consist of various data types.\n",
    "For example, the year of construction\n",
    "is represented by an integer,\n",
    "the roof type by discrete categorical assignments,\n",
    "and other features by floating point numbers.\n",
    "And here is where reality complicates things:\n",
    "for some examples, some data is altogether missing\n",
    "with the missing value marked simply as \"na\".\n",
    "The price of each house is included\n",
    "for the training set only\n",
    "(it is a competition after all).\n",
    "We will want to partition the training set\n",
    "to create a validation set,\n",
    "but we only get to evaluate our models on the official test set\n",
    "after uploading predictions to Kaggle.\n",
    "The \"Data\" tab on the competition tab\n",
    "in the figure\n",
    "has links for downloading the data.\n",
    "\n",
    "To get started, we will read in and process the data\n",
    "using `pandas`, which we introduced in that section.\n",
    "For convenience, we use the `d2l.download` helper to fetch and cache\n",
    "the Kaggle housing dataset. It performs hash-checked caching: if a file\n",
    "corresponding to this dataset already exists in the cache directory and its\n",
    "SHA-1 matches `sha1_hash`, the cached copy is reused, avoiding redundant\n",
    "downloads (its implementation details live in the d2l library)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "kaggle-house-price-accessing-and-reading-the-dataset-1"
   },
   "source": [
    "class KaggleHouse(d2l.DataModule):\n",
    "    def __init__(self, batch_size, train=None, val=None):\n",
    "        super().__init__()\n",
    "        self.save_hyperparameters()\n",
    "        if self.train is None:\n",
    "            self.raw_train = pd.read_csv(d2l.download(\n",
    "                d2l.DATA_URL + 'kaggle_house_pred_train.csv', self.root,\n",
    "                sha1_hash='585e9cc93e70b39160e7921475f9bcd7d31219ce'))\n",
    "            self.raw_val = pd.read_csv(d2l.download(\n",
    "                d2l.DATA_URL + 'kaggle_house_pred_test.csv', self.root,\n",
    "                sha1_hash='fa19780a7b011d9b009e8bff8e99922a8ee2eb90'))"
   ],
   "execution_count": 2,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=kaggle-house-price-md-3 fw=all -->\n",
    "\n",
    "The training dataset includes 1460 examples,\n",
    "80 features, and one label, while the Kaggle *test* set\n",
    "contains 1459 examples and 80 features.\n",
    "(We store the test features in the `val` attribute\n",
    "because the data module treats any unlabeled held-out split uniformly;\n",
    "do not confuse it with the validation folds\n",
    "that we will carve out of the training data for cross-validation below.)"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "kaggle-house-price-accessing-and-reading-the-dataset-2"
   },
   "source": [
    "data = KaggleHouse(batch_size=64)\n",
    "print(data.raw_train.shape)\n",
    "print(data.raw_val.shape)"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "(1460, 81)\n(1459, 80)\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=kaggle-house-price-md-4 fw=all -->\n",
    "\n",
    "## Data Preprocessing\n",
    "\n",
    "Let's take a look at the first four and final two features\n",
    "as well as the label (SalePrice) from the first four examples."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "kaggle-house-price-data-preprocessing-1"
   },
   "source": [
    "print(data.raw_train.iloc[:4, [0, 1, 2, 3, -3, -2, -1]])"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "   Id  MSSubClass MSZoning  LotFrontage SaleType SaleCondition  SalePrice\n0   1          60       RL         65.0       WD        Normal     208500\n1   2          20       RL         80.0       WD        Normal     181500\n2   3          60       RL         68.0       WD        Normal     223500\n3   4          70       RL         60.0       WD       Abnorml     140000\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=kaggle-house-price-md-5 fw=all -->\n",
    "\n",
    "We can see that in each example, the first feature is the identifier.\n",
    "This lets us identify each record. We drop it here because the competition\n",
    "defines it as a row identifier rather than a measured house attribute.\n",
    "Identifiers are not harmless in every dataset: they can encode collection\n",
    "order, source, or time, so their semantics should be checked rather than\n",
    "discarded by rule.\n",
    "Furthermore, given a wide variety of data types,\n",
    "we will need to preprocess the data before we can start modeling.\n",
    "\n",
    "\n",
    "Let's start with the numerical features.\n",
    "First, we apply a heuristic,\n",
    "replacing all missing values\n",
    "by the corresponding feature's mean.\n",
    "Then, to put all features on a common scale,\n",
    "we *standardize* the data by\n",
    "rescaling features to zero mean and unit variance:\n",
    "\n",
    "$$x \\leftarrow \\frac{x - \\mu}{\\sigma},$$\n",
    "\n",
    "where $\\mu$ and $\\sigma$ denote the feature's mean and standard deviation.\n",
    "By the definition of mean and variance, the rescaled feature has\n",
    "$E\\!\\left[\\frac{x-\\mu}{\\sigma}\\right] = 0$ and\n",
    "$\\mathrm{Var}\\!\\left[\\frac{x-\\mu}{\\sigma}\\right] = 1$,\n",
    "so every column now lives on the same zero-mean, unit-variance scale.\n",
    "We compute $\\mu$ and $\\sigma$ from the *training* set only and\n",
    "apply the very same transformation to the test set. Using statistics that\n",
    "include the test data would let information about the test distribution\n",
    "seep into our preprocessing, optimistically biasing every evaluation we\n",
    "make afterwards. This pitfall, *test-set leakage*, is one of the most\n",
    "common ways a model looks better offline than it ever does in deployment.\n",
    "\n",
    "Intuitively, we standardize the data for three reasons.\n",
    "First, it proves convenient for optimization, putting all coordinates on a\n",
    "comparable scale. Second, because we do not know *a priori*\n",
    "which features will be relevant, we do not want to penalize coefficients\n",
    "assigned to one feature more than any other (a single scale lets weight\n",
    "decay treat them even-handedly). Third, it makes our mean-imputation step\n",
    "coherent: after standardization, filling a missing value with the column\n",
    "mean is exactly filling it with $0$, the same neutral value regardless of\n",
    "the feature's original units.\n",
    "\n",
    "Next we deal with discrete values.\n",
    "These include features such as \"MSZoning\".\n",
    "We replace them by a one-hot encoding\n",
    "in the same way that we earlier transformed\n",
    "multiclass labels into vectors (see that section).\n",
    "For instance, \"MSZoning\" assumes the values \"RL\" and \"RM\".\n",
    "Dropping the \"MSZoning\" feature,\n",
    "two new indicator features\n",
    "\"MSZoning_RL\" and \"MSZoning_RM\" are created with values being either 0 or 1.\n",
    "According to one-hot encoding,\n",
    "if the original value of \"MSZoning\" is \"RL\",\n",
    "then \"MSZoning_RL\" is 1 and \"MSZoning_RM\" is 0.\n",
    "The `pandas` package does this automatically for us.\n",
    "We also fit the categorical vocabulary on the training rows only. At\n",
    "validation or deployment time, a category unseen during fitting maps to all\n",
    "zeros for that feature's known indicators. Building the vocabulary from an\n",
    "unlabeled test batch is a transductive competition technique; it is unsuitable\n",
    "when future categories are unavailable during training."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "kaggle-house-price-data-preprocessing-2"
   },
   "source": [
    "@d2l.add_to_class(KaggleHouse)\n",
    "def preprocess(self):\n",
    "    self.train, self.val = fit_preprocess(self.raw_train, self.raw_val)\n",
    "\n",
    "def fit_preprocess(train_raw, other_raw):\n",
    "    \"\"\"Fit preprocessing on train_raw and apply it to both dataframes.\"\"\"\n",
    "    label = 'SalePrice'\n",
    "    train_X = train_raw.drop(columns=['Id', label])\n",
    "    other_X = other_raw.drop(columns=['Id', label], errors='ignore')\n",
    "    numeric = train_X.select_dtypes(include='number').columns\n",
    "    train_X[numeric] = train_X[numeric].astype(float)\n",
    "    other_X[numeric] = other_X[numeric].astype(float)\n",
    "    mean, std = train_X[numeric].mean(), train_X[numeric].std()\n",
    "    std = std.mask(std == 0, 1)\n",
    "    train_X.loc[:, numeric] = (train_X[numeric] - mean) / std\n",
    "    other_X.loc[:, numeric] = (other_X[numeric] - mean) / std\n",
    "    train_X.loc[:, numeric] = train_X[numeric].fillna(0)\n",
    "    other_X.loc[:, numeric] = other_X[numeric].fillna(0)\n",
    "    train_X = pd.get_dummies(train_X, dummy_na=True)\n",
    "    other_X = pd.get_dummies(other_X, dummy_na=True)\n",
    "    other_X = other_X.reindex(columns=train_X.columns, fill_value=0)\n",
    "    train_X[label] = train_raw[label].values\n",
    "    if label in other_raw:\n",
    "        other_X[label] = other_raw[label].values\n",
    "    return train_X, other_X"
   ],
   "execution_count": 5,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=kaggle-house-price-md-6 fw=all -->\n",
    "\n",
    "You can see that this conversion expands the 79 raw predictors into several\n",
    "hundred numeric columns; the exact count is determined by the categories\n",
    "observed in the training data."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "kaggle-house-price-data-preprocessing-3"
   },
   "source": [
    "data.preprocess()\n",
    "data.train.shape"
   ],
   "execution_count": 6,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "(1460, 331)"
     },
     "metadata": {},
     "execution_count": 6
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=kaggle-house-price-md-7 fw=all -->\n",
    "\n",
    "## Error Measure\n",
    "\n",
    "Before choosing a model, we need to decide what \"good\" means: the loss we train against and the metric we are scored on. So the choice of error measure comes first.\n",
    "\n",
    "With house prices, as with stock prices,\n",
    "we care about relative quantities\n",
    "more than absolute quantities.\n",
    "Thus we tend to care more about\n",
    "the relative error $\\frac{y - \\hat{y}}{y}$\n",
    "than about the absolute error $y - \\hat{y}$.\n",
    "For instance, if our prediction is off by \\$100,000\n",
    "when estimating the price of a house in rural Ohio,\n",
    "where the value of a typical house is \\$125,000,\n",
    "then we are probably doing a horrible job.\n",
    "On the other hand, if we err by this amount\n",
    "in Los Altos Hills, California,\n",
    "this might represent a stunningly accurate prediction\n",
    "(there, the median house price exceeds \\$4 million).\n",
    "\n",
    "One way to address this problem is to\n",
    "measure the discrepancy in the logarithm of the price estimates.\n",
    "In fact, this is also the official error measure\n",
    "used by the competition to evaluate the quality of submissions.\n",
    "After all, a small value $\\delta$ for $|\\log y - \\log \\hat{y}| \\leq \\delta$\n",
    "translates into $e^{-\\delta} \\leq \\frac{\\hat{y}}{y} \\leq e^\\delta$.\n",
    "This leads to the following root-mean-squared-error between the logarithm of the predicted price and the logarithm of the label price:\n",
    "\n",
    "$$\\sqrt{\\frac{1}{n}\\sum_{i=1}^n\\left(\\log y_i -\\log \\hat{y}_i\\right)^2}.$$"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "kaggle-house-price-error-measure"
   },
   "source": [
    "@d2l.add_to_class(KaggleHouse)\n",
    "def get_dataloader(self, train):\n",
    "    label = 'SalePrice'\n",
    "    data = self.train if train else self.val\n",
    "    if label not in data: return\n",
    "    get_tensor = lambda x: d2l.tensor(x.values.astype(float),\n",
    "                                      dtype=d2l.float32)\n",
    "    # Logarithm of prices \n",
    "    tensors = (get_tensor(data.drop(columns=[label])),  # X\n",
    "               d2l.reshape(d2l.log(get_tensor(data[label])), (-1, 1)))  # Y\n",
    "    return self.get_tensorloader(tensors, train)"
   ],
   "execution_count": 7,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=kaggle-house-price-md-8 fw=all -->\n",
    "\n",
    "## $K$-Fold Cross-Validation\n",
    "\n",
    "You might recall that we introduced cross-validation\n",
    "in that section, where we discussed how to deal\n",
    "with model selection.\n",
    "We will put this to good use to select the model design\n",
    "and to adjust the hyperparameters.\n",
    "The idea is shown in the figure: we partition the data into $K$\n",
    "folds whose sizes differ by at most one and run $K$ training rounds. In round $i$, fold $i$ is held out\n",
    "for validation and the model is trained on the remaining $K-1$ folds; our\n",
    "generalization estimate is the average of the $K$ validation scores. With\n",
    "only about $1500$ training examples here, this reuse of the data gives a far\n",
    "steadier estimate than any single train/validation split would.\n",
    "\n",
    "![In $K$-fold cross-validation with $K=5$, the data is partitioned into five equal folds. In each round $i$, fold $i$ is held out as the validation set (orange) and the model is trained on the remaining four folds (blue). The generalization estimate is the average of the five validation scores.](../img/mdl-mlp-kfold.svg)\n",
    "\n",
    "We first need a function that returns\n",
    "the $i^\\textrm{th}$ fold of the data\n",
    "in a $K$-fold cross-validation procedure.\n",
    "It proceeds by slicing out the $i^\\textrm{th}$ segment\n",
    "as validation data and returning the rest as training data.\n",
    "Note that this is not the most efficient way of handling data\n",
    "and we would definitely do something much smarter\n",
    "if our dataset was considerably larger.\n",
    "But this added complexity might obfuscate our code unnecessarily\n",
    "so we can safely omit it here owing to the simplicity of our problem."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "kaggle-house-price-k-fold-cross-validation-1"
   },
   "source": [
    "def k_fold_data(data, k):\n",
    "    rets = []\n",
    "    indices = data.raw_train.sample(frac=1, random_state=0).index.tolist()\n",
    "    base, remainder = divmod(len(indices), k)\n",
    "    start = 0\n",
    "    for j in range(k):\n",
    "        stop = start + base + (j < remainder)\n",
    "        val_idx = indices[start:stop]\n",
    "        raw_train = data.raw_train.drop(index=val_idx)\n",
    "        raw_val = data.raw_train.loc[val_idx]\n",
    "        train, val = fit_preprocess(raw_train, raw_val)\n",
    "        _, test = fit_preprocess(raw_train, data.raw_val)\n",
    "        fold = KaggleHouse(data.batch_size, train, val)\n",
    "        fold.test = test\n",
    "        rets.append(fold)\n",
    "        start = stop\n",
    "    return rets"
   ],
   "execution_count": 8,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=kaggle-house-price-md-9 fw=all -->\n",
    "\n",
    "The average validation error is returned\n",
    "when we train $K$ times in the $K$-fold cross-validation. We pass in a\n",
    "`model_fn` that builds a fresh model for each fold, so the *same*\n",
    "cross-validation loop can score a linear baseline or an MLP without change."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "kaggle-house-price-k-fold-cross-validation-2"
   },
   "source": [
    "def k_fold(trainer, data, k, model_fn):\n",
    "    val_loss, models = [], []\n",
    "    for i, data_fold in enumerate(k_fold_data(data, k)):\n",
    "        model = model_fn()\n",
    "        model.board.yscale='log'\n",
    "        if i != 0: model.board.display = False\n",
    "        trainer.fit(model, data_fold)\n",
    "        val_loss.append(float(model.board.data['val_loss'][-1].y))\n",
    "        models.append((model, data_fold.test))\n",
    "    print(f'average validation log mse = {sum(val_loss)/len(val_loss)}')\n",
    "    return models"
   ],
   "execution_count": 9,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=kaggle-house-price-md-10 fw=all -->\n",
    "\n",
    "## Model Selection\n",
    "\n",
    "We now have everything we need to compare model designs by their\n",
    "$K$-fold cross-validation score. Finding a good choice can take time,\n",
    "depending on how many variables one optimizes over.\n",
    "With a large enough dataset,\n",
    "and the normal sorts of hyperparameters,\n",
    "$K$-fold cross-validation tends to be\n",
    "reasonably resilient against multiple testing.\n",
    "However, if we try an unreasonably large number of options\n",
    "we might find that our validation\n",
    "performance is no longer representative of the true error.\n",
    "\n",
    "One caveat before we run the numbers. On *structured tabular data*\n",
    "like this dataset, gradient-boosted tree ensembles (XGBoost and LightGBM)\n",
    "typically outperform deep networks, including MLPs\n",
    "[@Grinsztajn.Oyallon.Varoquaux.2022; @Shwartz-Ziv.Armon.2022]. This\n",
    "does not diminish what we learn here, since the preprocessing pipeline,\n",
    "log-RMSE loss, and $K$-fold cross-validation apply to *any* model class.\n",
    "But it does set expectations about where neural networks shine\n",
    "(images, text, audio, and sequences) and where they currently do not\n",
    "(small-to-medium tabular data).\n",
    "\n",
    "We start with a linear model. It is a fast baseline that\n",
    "sanity-checks the pipeline. One subtlety is easy to get wrong: a baseline\n",
    "is only meaningful if it is *trained competently*. Plain minibatch SGD on\n",
    "these standardized features needs more than a handful of passes to converge,\n",
    "so we give the linear model a healthy $100$ epochs at learning rate $0.03$\n",
    "(a larger rate diverges on the log-price target). Trained this way the\n",
    "linear model already reaches a cross-validated log error well under $0.05$;\n",
    "stopping at the customary ten epochs would instead leave it badly underfit,\n",
    "which would flatter every model we compared against it."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "kaggle-house-price-model-selection-linear"
   },
   "source": [
    "trainer = d2l.Trainer(max_epochs=100)\n",
    "models = k_fold(trainer, data, k=5,\n",
    "                model_fn=lambda: d2l.LinearRegression(lr=0.03))"
   ],
   "execution_count": 10,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "average validation log mse = 0.05294514182955027\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"249.74375pt\" height=\"183.35625pt\" viewBox=\"0 0 249.74375 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2026-07-19T15:36:41.693248</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.10.8, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 183.35625 \nL 249.74375 183.35625 \nL 249.74375 0 \nL 0 0 \nz\n\" style=\"fill: #ffffff\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 37.7 145.8 \nL 233 145.8 \nL 233 7.2 \nL 37.7 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path id=\"mf4a06f2e88\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#mf4a06f2e88\" x=\"37.7\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <g transform=\"translate(34.51875 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use xlink:href=\"#mf4a06f2e88\" x=\"76.76\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 20 -->\n      <g transform=\"translate(70.3975 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-32\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use xlink:href=\"#mf4a06f2e88\" x=\"115.82\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 40 -->\n      <g transform=\"translate(109.4575 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-34\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#mf4a06f2e88\" x=\"154.88\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 60 -->\n      <g transform=\"translate(148.5175 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-36\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use xlink:href=\"#mf4a06f2e88\" x=\"193.94\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 80 -->\n      <g transform=\"translate(187.5775 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-38\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#mf4a06f2e88\" x=\"233\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 100 -->\n      <g transform=\"translate(223.45625 160.398438) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-31\" d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(127.246094 0)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- epoch -->\n     <g transform=\"translate(120.121875 174.076563) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \nL 1159 -1331 \nL 581 -1331 \nL 581 3500 \nL 1159 3500 \nL 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nz\nM 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \nL 3122 2828 \nQ 2878 2963 2633 3030 \nQ 2388 3097 2138 3097 \nQ 1578 3097 1268 2742 \nQ 959 2388 959 1747 \nQ 959 1106 1268 751 \nQ 1578 397 2138 397 \nQ 2388 397 2633 464 \nQ 2878 531 3122 666 \nL 3122 134 \nQ 2881 22 2623 -34 \nQ 2366 -91 2075 -91 \nQ 1284 -91 818 406 \nQ 353 903 353 1747 \nQ 353 2603 823 3093 \nQ 1294 3584 2113 3584 \nQ 2378 3584 2631 3529 \nQ 2884 3475 3122 3366 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-65\"/>\n      <use xlink:href=\"#DejaVuSans-70\" transform=\"translate(61.523438 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(125 0)\"/>\n      <use xlink:href=\"#DejaVuSans-63\" transform=\"translate(186.181641 0)\"/>\n      <use xlink:href=\"#DejaVuSans-68\" transform=\"translate(241.162109 0)\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_7\">\n      <defs>\n       <path id=\"m7646ba739e\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m7646ba739e\" x=\"37.7\" y=\"108.574246\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- $\\mathdefault{10^{-1}}$ -->\n      <g transform=\"translate(7.2 112.373465) scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \nL 4684 2272 \nL 4684 1741 \nL 678 1741 \nL 678 2272 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 38.965625) scale(0.7)\"/>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(186.855469 38.965625) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#m7646ba739e\" x=\"37.7\" y=\"71.035329\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <g transform=\"translate(13.1 74.834547) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_9\">\n      <g>\n       <use xlink:href=\"#m7646ba739e\" x=\"37.7\" y=\"33.496412\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- $\\mathdefault{10^{1}}$ -->\n      <g transform=\"translate(13.1 37.29563) scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625) scale(0.7)\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_10\">\n      <defs>\n       <path id=\"m4e485c351b\" d=\"M 0 0 \nL -2 0 \n\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"134.812823\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_11\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"128.202548\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_12\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"123.512483\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_13\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"119.874586\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_8\">\n     <g id=\"line2d_14\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"116.902208\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_9\">\n     <g id=\"line2d_15\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"114.389098\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_10\">\n     <g id=\"line2d_16\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"112.212143\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_11\">\n     <g id=\"line2d_17\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"110.291932\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_12\">\n     <g id=\"line2d_18\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"97.273906\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_13\">\n     <g id=\"line2d_19\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"90.663631\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_14\">\n     <g id=\"line2d_20\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"85.973566\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_15\">\n     <g id=\"line2d_21\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"82.335669\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_16\">\n     <g id=\"line2d_22\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"79.363291\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_17\">\n     <g id=\"line2d_23\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"76.850181\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_18\">\n     <g id=\"line2d_24\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"74.673226\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_19\">\n     <g id=\"line2d_25\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"72.753015\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_20\">\n     <g id=\"line2d_26\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"59.734989\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_21\">\n     <g id=\"line2d_27\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"53.124713\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_22\">\n     <g id=\"line2d_28\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"48.434649\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_23\">\n     <g id=\"line2d_29\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"44.796752\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_24\">\n     <g id=\"line2d_30\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"41.824373\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_25\">\n     <g id=\"line2d_31\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"39.311263\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_26\">\n     <g id=\"line2d_32\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"37.134309\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_27\">\n     <g id=\"line2d_33\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"35.214098\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_28\">\n     <g id=\"line2d_34\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"22.196072\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_29\">\n     <g id=\"line2d_35\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"15.585796\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_30\">\n     <g id=\"line2d_36\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"10.895732\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_31\">\n     <g id=\"line2d_37\">\n      <g>\n       <use xlink:href=\"#m4e485c351b\" x=\"37.7\" y=\"7.257835\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n      </g>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_38\">\n    <path d=\"M 38.111158 13.5 \nL 39.036263 73.221292 \nL 39.961368 87.162782 \nL 41.811579 96.608537 \nL 42.736684 95.318865 \nL 43.661789 100.069443 \nL 44.586895 98.271473 \nL 45.512 102.108041 \nL 46.437105 102.884065 \nL 47.362211 100.941232 \nL 48.287316 106.615693 \nL 49.212421 108.909504 \nL 50.137526 106.717447 \nL 51.062632 110.452293 \nL 51.987737 111.081776 \nL 52.912842 108.109597 \nL 53.837947 109.563397 \nL 54.763053 113.689049 \nL 56.613263 111.659936 \nL 57.538368 116.516667 \nL 58.463474 114.262087 \nL 59.388579 116.330339 \nL 60.313684 115.791216 \nL 61.238789 114.99036 \nL 62.163895 113.533484 \nL 63.089 115.957487 \nL 64.014105 116.147885 \nL 64.939211 119.40784 \nL 65.864316 119.659113 \nL 66.789421 117.548522 \nL 67.714526 118.383314 \nL 68.639632 120.974062 \nL 69.564737 120.515142 \nL 70.489842 121.085959 \nL 71.414947 118.744372 \nL 72.340053 120.054306 \nL 73.265158 120.323843 \nL 74.190263 124.784986 \nL 75.115368 118.457531 \nL 76.040474 125.587127 \nL 76.965579 124.05198 \nL 77.890684 121.984422 \nL 78.815789 122.701748 \nL 79.740895 123.568499 \nL 80.666 125.030559 \nL 81.591105 125.205677 \nL 82.516211 123.152338 \nL 83.441316 125.48057 \nL 84.366421 119.526435 \nL 85.291526 125.646828 \nL 86.216632 125.755445 \nL 88.066842 124.057536 \nL 88.991947 126.588412 \nL 89.917053 121.325502 \nL 90.842158 124.383549 \nL 91.767263 128.016214 \nL 92.692368 123.364242 \nL 93.617474 127.141872 \nL 94.542579 130.23005 \nL 95.467684 121.815738 \nL 96.392789 129.881303 \nL 97.317895 126.196183 \nL 98.243 128.491397 \nL 99.168105 127.04575 \nL 100.093211 121.410347 \nL 101.018316 128.359418 \nL 101.943421 127.728737 \nL 102.868526 127.57352 \nL 103.793632 131.701876 \nL 105.643842 126.566766 \nL 106.568947 128.760725 \nL 107.494053 127.801282 \nL 108.419158 130.892783 \nL 109.344263 127.659736 \nL 110.269368 127.957475 \nL 111.194474 129.527176 \nL 112.119579 128.105907 \nL 113.044684 132.455165 \nL 113.969789 126.506872 \nL 114.894895 127.00131 \nL 115.82 132.080845 \nL 117.670211 127.296572 \nL 118.595316 130.713257 \nL 119.520421 130.54475 \nL 120.445526 130.547956 \nL 121.370632 129.683371 \nL 122.295737 129.336618 \nL 123.220842 131.760967 \nL 124.145947 130.9949 \nL 125.071053 127.356157 \nL 125.996158 132.083188 \nL 126.921263 130.019745 \nL 127.846368 129.872306 \nL 128.771474 132.158059 \nL 129.696579 130.073148 \nL 130.621684 132.230819 \nL 131.546789 133.357812 \nL 132.471895 132.171933 \nL 133.397 129.172909 \nL 134.322105 131.812664 \nL 135.247211 132.193219 \nL 136.172316 133.971112 \nL 137.097421 128.819731 \nL 138.947632 133.357153 \nL 139.872737 133.354693 \nL 140.797842 131.034231 \nL 141.722947 129.673816 \nL 142.648053 133.266397 \nL 143.573158 132.177211 \nL 144.498263 133.302958 \nL 145.423368 135.08451 \nL 147.273579 131.194126 \nL 148.198684 130.9856 \nL 150.048895 136.462365 \nL 150.974 134.624484 \nL 151.899105 132.334583 \nL 152.824211 134.020541 \nL 153.749316 132.578426 \nL 154.674421 125.823714 \nL 155.599526 129.77703 \nL 156.524632 131.64259 \nL 158.374842 132.674298 \nL 159.299947 132.95356 \nL 160.225053 138.220354 \nL 161.150158 134.074457 \nL 162.075263 135.48048 \nL 163.000368 136.046379 \nL 163.925474 134.31348 \nL 164.850579 135.598099 \nL 165.775684 136.449111 \nL 166.700789 137.113309 \nL 168.551 130.378876 \nL 169.476105 136.309354 \nL 170.401211 136.906126 \nL 171.326316 135.020654 \nL 172.251421 134.748682 \nL 173.176526 132.879493 \nL 174.101632 133.487489 \nL 175.026737 136.361125 \nL 175.951842 133.650364 \nL 176.876947 131.3372 \nL 177.802053 137.747697 \nL 178.727158 135.303524 \nL 179.652263 136.491416 \nL 180.577368 136.712543 \nL 181.502474 137.825031 \nL 182.427579 133.724448 \nL 183.352684 137.053924 \nL 184.277789 131.979196 \nL 185.202895 132.512421 \nL 186.128 135.335977 \nL 187.053105 133.467392 \nL 187.978211 134.330139 \nL 188.903316 137.898743 \nL 189.828421 135.016338 \nL 190.753526 139.012008 \nL 191.678632 134.326751 \nL 192.603737 137.333892 \nL 193.528842 135.726103 \nL 194.453947 133.62684 \nL 195.379053 135.944469 \nL 196.304158 137.672082 \nL 198.154368 134.49306 \nL 199.079474 135.710415 \nL 200.004579 131.741016 \nL 200.929684 138.866664 \nL 201.854789 136.636558 \nL 202.779895 136.78754 \nL 203.705 137.753057 \nL 204.630105 133.675762 \nL 205.555211 131.858386 \nL 206.480316 137.419438 \nL 207.405421 136.684021 \nL 208.330526 137.583818 \nL 209.255632 134.182119 \nL 210.180737 136.873112 \nL 211.105842 135.72952 \nL 212.030947 138.571083 \nL 212.956053 135.371929 \nL 213.881158 139.123856 \nL 214.806263 135.388175 \nL 216.656474 136.940353 \nL 217.581579 133.991211 \nL 218.506684 139.5 \nL 219.431789 137.998879 \nL 220.356895 134.52364 \nL 221.282 138.644751 \nL 222.207105 133.618664 \nL 223.132211 136.809612 \nL 224.057316 139.067447 \nL 224.982421 133.29329 \nL 225.907526 137.927806 \nL 226.832632 138.121367 \nL 227.757737 139.27493 \nL 228.682842 133.776934 \nL 229.607947 138.151078 \nL 230.533053 138.506772 \nL 231.458158 137.467867 \nL 232.383263 137.700323 \nL 232.383263 137.700323 \n\" clip-path=\"url(#p185913426d)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"line2d_39\">\n    <path d=\"M 39.653 83.753527 \nL 41.606 92.114979 \nL 43.559 95.81396 \nL 45.512 95.856083 \nL 47.465 99.293051 \nL 49.418 101.928785 \nL 51.371 104.880505 \nL 53.324 104.757284 \nL 55.277 106.200224 \nL 57.23 109.431231 \nL 59.183 108.398204 \nL 61.136 105.443007 \nL 63.089 111.50692 \nL 65.042 110.71028 \nL 66.995 108.105657 \nL 68.948 110.935465 \nL 70.901 111.728106 \nL 72.854 111.799355 \nL 74.807 111.998119 \nL 76.76 111.107137 \nL 78.713 112.997614 \nL 80.666 112.651775 \nL 82.619 113.01966 \nL 84.572 112.898843 \nL 86.525 112.737616 \nL 88.478 113.438607 \nL 90.431 114.362689 \nL 92.384 112.0982 \nL 94.337 114.962229 \nL 96.29 115.197525 \nL 98.243 116.075691 \nL 100.196 110.123826 \nL 102.149 115.473375 \nL 104.102 113.847326 \nL 106.055 116.829791 \nL 108.008 117.887225 \nL 109.961 114.504774 \nL 111.914 117.623944 \nL 113.867 110.748262 \nL 115.82 116.713814 \nL 117.773 118.737859 \nL 119.726 116.573679 \nL 121.679 118.528256 \nL 123.632 119.260291 \nL 125.585 115.80052 \nL 127.538 120.431094 \nL 129.491 117.948949 \nL 131.444 115.135955 \nL 133.397 114.20285 \nL 135.35 116.375444 \nL 137.303 117.20929 \nL 139.256 119.298129 \nL 141.209 119.547682 \nL 143.162 117.65031 \nL 145.115 116.841299 \nL 147.068 115.662812 \nL 149.021 118.823256 \nL 150.974 118.175903 \nL 152.927 119.595091 \nL 154.88 114.886075 \nL 156.833 121.261433 \nL 158.786 118.649166 \nL 160.739 118.955454 \nL 162.692 119.273723 \nL 164.645 118.786256 \nL 166.598 118.136676 \nL 168.551 113.676649 \nL 170.504 118.247308 \nL 172.457 120.279581 \nL 174.41 118.533113 \nL 176.363 119.306174 \nL 178.316 118.449957 \nL 180.269 118.792954 \nL 182.222 118.34834 \nL 184.175 116.551576 \nL 186.128 121.496108 \nL 188.081 117.931934 \nL 190.034 118.688828 \nL 191.987 117.757686 \nL 193.94 117.298622 \nL 195.893 120.738924 \nL 197.846 119.267168 \nL 199.799 117.91396 \nL 201.752 119.929891 \nL 203.705 119.560949 \nL 205.658 110.36586 \nL 207.611 120.669056 \nL 209.564 119.431886 \nL 211.517 120.108447 \nL 213.47 120.773345 \nL 215.423 117.824594 \nL 217.376 116.086762 \nL 219.329 121.594699 \nL 221.282 120.226658 \nL 223.235 118.73146 \nL 225.188 118.278211 \nL 227.141 121.43097 \nL 229.094 121.078855 \nL 231.047 119.463872 \nL 233 116.600162 \n\" clip-path=\"url(#p185913426d)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #ff7f0e; stroke-width: 1.5\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 37.7 145.8 \nL 37.7 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 233 145.8 \nL 233 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 37.7 145.8 \nL 233 145.8 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 37.7 7.2 \nL 233 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 146.409375 45.1125 \nL 226 45.1125 \nQ 228 45.1125 228 43.1125 \nL 228 14.2 \nQ 228 12.2 226 12.2 \nL 146.409375 12.2 \nQ 144.409375 12.2 144.409375 14.2 \nL 144.409375 43.1125 \nQ 144.409375 45.1125 146.409375 45.1125 \nz\n\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n    </g>\n    <g id=\"line2d_40\">\n     <path d=\"M 148.409375 20.298438 \nL 158.409375 20.298438 \nL 168.409375 20.298438 \n\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n    </g>\n    <g id=\"text_11\">\n     <!-- train_loss -->\n     <g transform=\"translate(176.409375 23.798438) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \nL 3263 -1509 \nL -63 -1509 \nL -63 -1063 \nL 3263 -1063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use xlink:href=\"#DejaVuSans-72\" transform=\"translate(39.208984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(80.322266 0)\"/>\n      <use xlink:href=\"#DejaVuSans-69\" transform=\"translate(141.601562 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6e\" transform=\"translate(169.384766 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(232.763672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(282.763672 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(310.546875 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(371.728516 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(423.828125 0)\"/>\n     </g>\n    </g>\n    <g id=\"line2d_41\">\n     <path d=\"M 148.409375 35.254688 \nL 158.409375 35.254688 \nL 168.409375 35.254688 \n\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #ff7f0e; stroke-width: 1.5\"/>\n    </g>\n    <g id=\"text_12\">\n     <!-- val_loss -->\n     <g transform=\"translate(176.409375 38.754688) scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-76\" d=\"M 191 3500 \nL 800 3500 \nL 1894 563 \nL 2988 3500 \nL 3597 3500 \nL 2284 0 \nL 1503 0 \nL 191 3500 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-76\"/>\n      <use xlink:href=\"#DejaVuSans-61\" transform=\"translate(59.179688 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(120.458984 0)\"/>\n      <use xlink:href=\"#DejaVuSans-5f\" transform=\"translate(148.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" transform=\"translate(198.242188 0)\"/>\n      <use xlink:href=\"#DejaVuSans-6f\" transform=\"translate(226.025391 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(287.207031 0)\"/>\n      <use xlink:href=\"#DejaVuSans-73\" transform=\"translate(339.306641 0)\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p185913426d\">\n   <rect x=\"37.7\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n  </clipPath>\n </defs>\n</svg>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=kaggle-house-price-md-11 fw=all -->\n",
    "\n",
    "Notice that sometimes the number of training errors\n",
    "for a set of hyperparameters can be very low,\n",
    "even as the number of errors on $K$-fold cross-validation\n",
    "grows considerably higher.\n",
    "This indicates that we are overfitting.\n",
    "Throughout training you will want to monitor both numbers.\n",
    "Less overfitting might indicate that our data can support a more powerful model.\n",
    "Massive overfitting might suggest that we can gain\n",
    "by incorporating regularization techniques.\n",
    "\n",
    "##  Submitting Predictions on Kaggle\n",
    "\n",
    "Now that we know what a good choice of hyperparameters should be,\n",
    "we might \n",
    "calculate the average predictions \n",
    "on the test set\n",
    "by all the $K$ models.\n",
    "Since the models predict *log*-prices and the competition scores\n",
    "root-mean-squared *log* error,\n",
    "we average in log space before exponentiating:\n",
    "the mean of the log-predictions is the ensemble\n",
    "consistent with the metric\n",
    "(in price space it amounts to a geometric mean).\n",
    "\n",
    "Note what this *fold ensembling* is, though. Each of the $K$\n",
    "models saw only $(K-1)/K$ of the training data, and the \"average validation\n",
    "log mse\" we computed above estimates the error of a *single* such model, not\n",
    "of the ensemble we are about to submit, whose error the cross-validation\n",
    "score does not measure. The canonical alternative is to *refit* one model on\n",
    "all of the training data using the hyperparameters that cross-validation\n",
    "selected, so that the submitted model is a fresh draw of exactly the thing we\n",
    "scored. Fold ensembling is standard Kaggle practice: it is free (the $K$\n",
    "models are already trained) and the averaging usually buys a small variance\n",
    "reduction, so it tends to edge out the refit. But the refit is the more\n",
    "direct experiment, and you should choose between them deliberately.\n",
    "\n",
    "Saving the predictions in a csv file\n",
    "will simplify uploading the results to Kaggle.\n",
    "The following code will generate a file called `submission.csv`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "kaggle-house-price-submitting-predictions-on-kaggle"
   },
   "source": [
    "preds = [model(d2l.tensor(test.values.astype(float), dtype=d2l.float32))\n",
    "         for model, test in models]\n",
    "# Average the K log-price predictions in log space, then exponentiate.\n",
    "ensemble_preds = d2l.exp(d2l.reduce_mean(d2l.concat(preds, 1), 1))\n",
    "submission = pd.DataFrame({'Id':data.raw_val.Id,\n",
    "                           'SalePrice':d2l.numpy(ensemble_preds)})\n",
    "submission.to_csv('submission.csv', index=False)"
   ],
   "execution_count": 11,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=kaggle-house-price-md-12 fw=all -->\n",
    "\n",
    "Next, as demonstrated in the figure,\n",
    "we can submit our predictions on Kaggle\n",
    "and see how they compare with the actual house prices (labels)\n",
    "on the test set.\n",
    "The steps are quite simple:\n",
    "\n",
    "* Log in to the Kaggle website and visit the house price prediction competition page.\n",
    "* Click the “Submit Predictions” or “Late Submission” button.\n",
    "* Click the “Upload Submission File” button in the dashed box at the bottom of the page and select the prediction file you wish to upload.\n",
    "* Click the “Make Submission” button at the bottom of the page to view your results.\n",
    "\n",
    "![Submitting data to Kaggle.](../img/kaggle-submit2.png)\n",
    "\n",
    "## Summary and Discussion\n",
    "\n",
    "Real data is messy: a mix of numeric and categorical features, with missing\n",
    "values and wildly different scales. The preprocessing pipeline in this\n",
    "section (mean imputation, standardization with statistics fit on the training\n",
    "set only to avoid test-set leakage, and one-hot encoding of categoricals) is\n",
    "a sensible default that applies far beyond this competition. When the target\n",
    "spans an order of magnitude, predicting the *logarithm* of the price and\n",
    "scoring with root-mean-squared log error converts an asymmetric dollar-scale\n",
    "problem into one where a $10\\%$ error on a $\\$100{,}000$ house and on a\n",
    "$\\$1{,}000{,}000$ house are penalized equally, which is what we actually care\n",
    "about.\n",
    "\n",
    "$K$-fold cross-validation is useful for limited IID data such as this dataset:\n",
    "it spends $K$ training runs to reduce dependence on one validation split, and\n",
    "the same loop supports hyperparameter search. Random folds are invalid when\n",
    "rows are grouped, ordered in time, spatially dependent, or otherwise linked;\n",
    "those settings require a split that preserves the dependence structure.\n",
    "\n",
    "Two caveats belong in any account of this pipeline. First, none of it is\n",
    "specific to neural networks: the same preprocessing, loss design, and\n",
    "cross-validation loop work with any model class, including the\n",
    "gradient-boosted tree ensembles (XGBoost, LightGBM) that routinely beat deep\n",
    "networks on medium-sized tabular data\n",
    "[@Grinsztajn.Oyallon.Varoquaux.2022; @Shwartz-Ziv.Armon.2022]. Second,\n",
    "model capacity matters: adding hidden layers, tuning the dropout rate, and\n",
    "searching over learning rate and weight decay can improve substantially on\n",
    "the baseline shown here, which is exactly what the exercises ask you to do.\n",
    "\n",
    "Looking ahead, the moves we made here recur throughout supervised learning.\n",
    "Feature scaling and imputation reappear in nearly every tabular pipeline, and\n",
    "the competition recipe (download, preprocess, match the loss to the metric,\n",
    "cross-validate, refit, submit) generalizes directly: later chapters apply the\n",
    "same cross-validation discipline to image datasets, sequence tasks, and the\n",
    "fine-tuning of pretrained models.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Submit your predictions for this section to Kaggle. How good are they?\n",
    "1. Is it always a good idea to replace missing values by a mean? Hint: can you construct a situation where the values are not missing at random?\n",
    "1. Improve the score by tuning the hyperparameters through $K$-fold cross-validation.\n",
    "1. Improve the score by improving the model (e.g., layers, weight decay, and dropout).\n",
    "1. What happens if we do not standardize the continuous numerical features as we have done in this section?\n",
    "1. Swap the linear model for a gradient-boosted tree model (for example scikit-learn's `GradientBoostingRegressor`, or XGBoost or LightGBM if installed), trained on the same preprocessed features. How does its cross-validated log-RMSE compare? Why might tree ensembles have an edge on data like this?\n",
    "1. Revisit the preprocessing choices. How does median imputation compare to mean imputation? What changes if you encode high-cardinality categorical features with target encoding or a learned embedding instead of one-hot vectors? Target encoding uses labels, so implement it out of fold: for each training row, its encoded value must be computed without that row or its validation fold."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=kaggle-house-price-md-13 fw=mxnet -->\n",
    "\n",
    "[Discussions](https://d2l.discourse.group/t/106)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-mxnet",
   "display_name": "d2l (MXNet)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
