{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=natural-language-inference-and-dataset-md-1 fw=all -->\n",
    "\n",
    "# Natural Language Inference and the Dataset\n",
    "\n",
    "In that section, we discussed the problem of sentiment analysis.\n",
    "This task aims to classify a single text sequence into predefined categories,\n",
    "such as a set of sentiment polarities.\n",
    "However, when there is a need to decide whether one sentence can be inferred from another, \n",
    "or eliminate redundancy by identifying sentences that are semantically equivalent,\n",
    "knowing how to classify one text sequence is insufficient.\n",
    "Instead, we need to be able to reason over pairs of text sequences.\n",
    "\n",
    "\n",
    "## Natural Language Inference\n",
    "\n",
    "*Natural language inference* studies whether a *hypothesis*\n",
    "can be inferred from a *premise*, where both are a text sequence.\n",
    "In other words, natural language inference determines the logical relationship between a pair of text sequences.\n",
    "Such relationships usually fall into three types:\n",
    "\n",
    "* *Entailment*: the hypothesis can be inferred from the premise.\n",
    "* *Contradiction*: the negation of the hypothesis can be inferred from the premise.\n",
    "* *Neutral*: all the other cases.\n",
    "\n",
    "Natural language inference is also known as the recognizing textual entailment task.\n",
    "For example, the following pair will be labeled as *entailment* because \"showing affection\" in the hypothesis can be inferred from \"hugging one another\" in the premise.\n",
    "\n",
    "> Premise: Two women are hugging each other.\n",
    "\n",
    "> Hypothesis: Two women are showing affection.\n",
    "\n",
    "The following is an example of *contradiction* as \"running the coding example\" indicates \"not sleeping\" rather than \"sleeping\".\n",
    "\n",
    "> Premise: A man is running the coding example from Dive into Deep Learning.\n",
    "\n",
    "> Hypothesis: The man is sleeping.\n",
    "\n",
    "The third example shows a *neutral* relationship because neither \"famous\" nor \"not famous\" can be inferred from the fact that \"are performing for us\".\n",
    "\n",
    "> Premise: The musicians are performing for us.\n",
    "\n",
    "> Hypothesis: The musicians are famous.\n",
    "\n",
    "Natural language inference has been a central topic for understanding natural language.\n",
    "It enjoys wide applications ranging from\n",
    "information retrieval to open-domain question answering.\n",
    "To study this problem, we will begin by investigating a popular natural language inference benchmark dataset.\n",
    "\n",
    "\n",
    "## The Stanford Natural Language Inference (SNLI) Dataset\n",
    "\n",
    "Stanford Natural Language Inference (SNLI) Corpus is a collection of over 500000 labeled English sentence pairs [@Bowman.Angeli.Potts.ea.2015].\n",
    "We download and store the extracted SNLI dataset in the path `../data/snli_1.0`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "natural-language-inference-and-dataset-the-stanford-natural-language-inference-snli-dataset"
   },
   "source": [
    "from d2l import mxnet as d2l\n",
    "from mxnet import gluon, np, npx\n",
    "import os\n",
    "import re\n",
    "\n",
    "npx.set_np()\n",
    "\n",
    "\n",
    "d2l.DATA_HUB['SNLI'] = (\n",
    "    'https://nlp.stanford.edu/projects/snli/snli_1.0.zip',\n",
    "    '9fcde07509c7e87ec61c640c1b2753d9041758e4')\n",
    "\n",
    "data_dir = d2l.download_extract('SNLI')"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=natural-language-inference-and-dataset-md-2 fw=all -->\n",
    "\n",
    "### Reading the Dataset\n",
    "\n",
    "The original SNLI dataset contains much richer information than what we really need in our experiments. Thus, we define a function `read_snli` to only extract part of the dataset, then return lists of premises, hypotheses, and their labels."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "natural-language-inference-and-dataset-reading-the-dataset-1"
   },
   "source": [
    "\n",
    "def read_snli(data_dir, is_train):\n",
    "    \"\"\"Read the SNLI dataset into premises, hypotheses, and labels.\"\"\"\n",
    "    def extract_text(s):\n",
    "        # Remove information that will not be used by us\n",
    "        s = re.sub('\\\\(', '', s) \n",
    "        s = re.sub('\\\\)', '', s)\n",
    "        # Substitute two or more consecutive whitespace with space\n",
    "        s = re.sub('\\\\s{2,}', ' ', s)\n",
    "        return s.strip()\n",
    "    label_set = {'entailment': 0, 'contradiction': 1, 'neutral': 2}\n",
    "    file_name = os.path.join(data_dir, 'snli_1.0_train.txt'\n",
    "                             if is_train else 'snli_1.0_test.txt')\n",
    "    with open(file_name, 'r') as f:\n",
    "        rows = [row.split('\\t') for row in f.readlines()[1:]]\n",
    "    premises = [extract_text(row[1]) for row in rows if row[0] in label_set]\n",
    "    hypotheses = [extract_text(row[2]) for row in rows if row[0] in label_set]\n",
    "    labels = [label_set[row[0]] for row in rows if row[0] in label_set]\n",
    "    return premises, hypotheses, labels"
   ],
   "execution_count": 2,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=natural-language-inference-and-dataset-md-3 fw=all -->\n",
    "\n",
    "Now let's print the first 3 pairs of premise and hypothesis, as well as their labels (\"0\", \"1\", and \"2\" correspond to \"entailment\", \"contradiction\", and \"neutral\", respectively )."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "natural-language-inference-and-dataset-reading-the-dataset-2"
   },
   "source": [
    "train_data = read_snli(data_dir, is_train=True)\n",
    "for x0, x1, y in zip(train_data[0][:3], train_data[1][:3], train_data[2][:3]):\n",
    "    print('premise:', x0)\n",
    "    print('hypothesis:', x1)\n",
    "    print('label:', y)"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "premise: A person on a horse jumps over a broken down airplane .\nhypothesis: A person is training his horse for a competition .\nlabel: 2\npremise: A person on a horse jumps over a broken down airplane .\nhypothesis: A person is at a diner , ordering an omelette .\nlabel: 1\npremise: A person on a horse jumps over a broken down airplane .\nhypothesis: A person is outdoors , on a horse .\nlabel: 0\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=natural-language-inference-and-dataset-md-4 fw=all -->\n",
    "\n",
    "The training set has about 550000 pairs,\n",
    "and the testing set has about 10000 pairs.\n",
    "The following shows that \n",
    "the three labels \"entailment\", \"contradiction\", and \"neutral\" are balanced in \n",
    "both the training set and the testing set."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "natural-language-inference-and-dataset-reading-the-dataset-3"
   },
   "source": [
    "test_data = read_snli(data_dir, is_train=False)\n",
    "for data in [train_data, test_data]:\n",
    "    print([[row for row in data[2]].count(i) for i in range(3)])"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[183416, 183187, 182764]\n[3368, 3237, 3219]\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=natural-language-inference-and-dataset-md-5 fw=all -->\n",
    "\n",
    "### Defining a Class for Loading the Dataset\n",
    "\n",
    "Below we define a class for loading the SNLI dataset by inheriting from the `Dataset` class in Gluon. The argument `num_steps` in the class constructor specifies the length of a text sequence so that each minibatch of sequences will have the same shape. \n",
    "In other words,\n",
    "tokens after the first `num_steps` ones in longer sequence are trimmed, while special tokens “&lt;pad&gt;” will be appended to shorter sequences until their length becomes `num_steps`.\n",
    "By implementing the `__getitem__` function, we can arbitrarily access the premise, hypothesis, and label with the index `idx`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "natural-language-inference-and-dataset-defining-a-class-for-loading-the-dataset"
   },
   "source": [
    "\n",
    "class SNLIDataset(gluon.data.Dataset):\n",
    "    \"\"\"A customized dataset to load the SNLI dataset.\"\"\"\n",
    "    def __init__(self, dataset, num_steps, vocab=None):\n",
    "        self.num_steps = num_steps\n",
    "        all_premise_tokens = d2l.tokenize(dataset[0])\n",
    "        all_hypothesis_tokens = d2l.tokenize(dataset[1])\n",
    "        if vocab is None:\n",
    "            self.vocab = d2l.Vocab(all_premise_tokens + all_hypothesis_tokens,\n",
    "                                   min_freq=5, reserved_tokens=['<pad>'])\n",
    "        else:\n",
    "            self.vocab = vocab\n",
    "        self.premises = self._pad(all_premise_tokens)\n",
    "        self.hypotheses = self._pad(all_hypothesis_tokens)\n",
    "        self.labels = np.array(dataset[2])\n",
    "        print('read ' + str(len(self.premises)) + ' examples')\n",
    "\n",
    "    def _pad(self, lines):\n",
    "        return np.array([d2l.truncate_pad(\n",
    "            self.vocab[line], self.num_steps, self.vocab['<pad>'])\n",
    "                         for line in lines])\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return (self.premises[idx], self.hypotheses[idx]), self.labels[idx]\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.premises)"
   ],
   "execution_count": 5,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=natural-language-inference-and-dataset-md-6 fw=all -->\n",
    "\n",
    "### Putting It All Together\n",
    "\n",
    "Now we can invoke the `read_snli` function and the `SNLIDataset` class to download the SNLI dataset and return data-loader objects for both training and testing sets, together with the vocabulary of the training set.\n",
    "It is noteworthy that we must use the vocabulary constructed from the training set\n",
    "as that of the testing set. \n",
    "As a result, any new token from the testing set will be unknown to the model trained on the training set."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "natural-language-inference-and-dataset-putting-it-all-together-1"
   },
   "source": [
    "\n",
    "def load_data_snli(batch_size, num_steps=50):\n",
    "    \"\"\"Download the SNLI dataset and return data iterators and vocabulary.\"\"\"\n",
    "    num_workers = d2l.get_dataloader_workers()\n",
    "    data_dir = d2l.download_extract('SNLI')\n",
    "    train_data = read_snli(data_dir, True)\n",
    "    test_data = read_snli(data_dir, False)\n",
    "    train_set = SNLIDataset(train_data, num_steps)\n",
    "    test_set = SNLIDataset(test_data, num_steps, train_set.vocab)\n",
    "    train_iter = gluon.data.DataLoader(train_set, batch_size, shuffle=True,\n",
    "                                       num_workers=num_workers)\n",
    "    test_iter = gluon.data.DataLoader(test_set, batch_size, shuffle=False,\n",
    "                                      num_workers=num_workers)\n",
    "    return train_iter, test_iter, train_set.vocab"
   ],
   "execution_count": 6,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=natural-language-inference-and-dataset-md-7 fw=all -->\n",
    "\n",
    "Here we set the batch size to 128 and sequence length to 50,\n",
    "and invoke the `load_data_snli` function to get the data iterators and vocabulary.\n",
    "Then we print the vocabulary size."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "natural-language-inference-and-dataset-putting-it-all-together-2"
   },
   "source": [
    "train_iter, test_iter, vocab = load_data_snli(128, 50)\n",
    "len(vocab)"
   ],
   "execution_count": 7,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "read 549367 examples\nread 9824 examples\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "18678"
     },
     "metadata": {},
     "execution_count": 7
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=natural-language-inference-and-dataset-md-8 fw=all -->\n",
    "\n",
    "Now we print the shape of the first minibatch.\n",
    "Contrary to sentiment analysis,\n",
    "we have two inputs `X[0]` and `X[1]` representing pairs of premises and hypotheses."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "natural-language-inference-and-dataset-putting-it-all-together-3"
   },
   "source": [
    "for X, Y in train_iter:\n",
    "    print(X[0].shape)\n",
    "    print(X[1].shape)\n",
    "    print(Y.shape)\n",
    "    break"
   ],
   "execution_count": 8,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "(128, 50)\n(128, 50)\n(128,)\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "[14:03:31] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = stack\ninput storage types = [default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[14:03:31] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = stack\ninput storage types = [default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[14:03:31] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = stack\ninput storage types = [default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[14:03:31] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = stack\ninput storage types = [default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=natural-language-inference-and-dataset-md-9 fw=all -->\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Natural language inference studies whether a hypothesis can be inferred from a premise, where both are a text sequence.\n",
    "* In natural language inference, relationships between premises and hypotheses include entailment, contradiction, and neutral.\n",
    "* Stanford Natural Language Inference (SNLI) Corpus is a popular benchmark dataset of natural language inference.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Machine translation has long been evaluated based on superficial $n$-gram matching between an output translation and a ground-truth translation. Can you design a measure for evaluating machine translation results by using natural language inference?\n",
    "1. How can we change hyperparameters to reduce the vocabulary size?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=natural-language-inference-and-dataset-md-10 fw=mxnet -->\n",
    "\n",
    "[Discussions](https://d2l.discourse.group/t/394)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-mxnet",
   "display_name": "d2l (MXNet)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
