{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=movielens-md-1 fw=all -->\n",
    "\n",
    "#  The MovieLens Dataset\n",
    "\n",
    "There are a number of datasets that are available for recommendation research. Amongst them, the [MovieLens](https://movielens.org/) dataset is probably one of the more popular ones. MovieLens is a non-commercial web-based movie recommender system. It is created in 1997 and run by GroupLens, a research lab at the University of Minnesota, in order to gather movie rating data for research purposes.  MovieLens data has been critical for several research studies including personalized recommendation and social psychology.\n",
    "\n",
    "\n",
    "## Getting the Data\n",
    "\n",
    "\n",
    "The MovieLens dataset is hosted by the [GroupLens](https://grouplens.org/datasets/movielens/) website. Several versions are available. We will use the MovieLens 100K dataset [@Herlocker.Konstan.Borchers.ea.1999].  This dataset is comprised of $100,000$ ratings, ranging from 1 to 5 stars, from 943 users on 1682 movies. It has been cleaned up so that each user has rated at least 20 movies. Some simple demographic information such as age, gender, genres for the users and items are also available.  We can download the [ml-100k.zip](http://files.grouplens.org/datasets/movielens/ml-100k.zip) and extract the `u.data` file, which contains all the $100,000$ ratings in the csv format. There are many other files in the folder, a detailed description for each file can be found in the [README](http://files.grouplens.org/datasets/movielens/ml-100k-README.txt) file of the dataset.\n",
    "\n",
    "To begin with, let's import the packages required to run this section's experiments."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "movielens-getting-the-data-1"
   },
   "source": [
    "from d2l import mxnet as d2l\n",
    "from mxnet import gluon, np\n",
    "import os\n",
    "import pandas as pd\n",
    "import random"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=movielens-md-2 fw=all -->\n",
    "\n",
    "Then, we download the MovieLens 100k dataset and load the interactions as `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "movielens-getting-the-data-2"
   },
   "source": [
    "\n",
    "d2l.DATA_HUB['ml-100k'] = (\n",
    "    'https://files.grouplens.org/datasets/movielens/ml-100k.zip',\n",
    "    'cd4dcac4241c8a4ad7badc7ca635da8a69dddb83')\n",
    "\n",
    "\n",
    "def read_data_ml100k():\n",
    "    data_dir = d2l.download_extract('ml-100k')\n",
    "    names = ['user_id', 'item_id', 'rating', 'timestamp']\n",
    "    data = pd.read_csv(os.path.join(data_dir, 'u.data'), sep='\\t',\n",
    "                       names=names, engine='python')\n",
    "    num_users = data.user_id.unique().shape[0]\n",
    "    num_items = data.item_id.unique().shape[0]\n",
    "    return data, num_users, num_items"
   ],
   "execution_count": 2,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=movielens-md-3 fw=all -->\n",
    "\n",
    "## Statistics of the Dataset\n",
    "\n",
    "Let's load up the data and inspect the first five records manually. It is an effective way to learn the data structure and verify that they have been loaded properly."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "movielens-statistics-of-the-dataset-1"
   },
   "source": [
    "data, num_users, num_items = read_data_ml100k()\n",
    "sparsity = 1 - len(data) / (num_users * num_items)\n",
    "print(f'number of users: {num_users}, number of items: {num_items}')\n",
    "print(f'matrix sparsity: {sparsity:f}')\n",
    "print(data.head(5))"
   ],
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "number of users: 943, number of items: 1682\nmatrix sparsity: 0.936953\n   user_id  item_id  rating  timestamp\n0      196      242       3  881250949\n1      186      302       3  891717742\n2       22      377       1  878887116\n3      244       51       2  880606923\n4      166      346       1  886397596\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=movielens-md-4 fw=all -->\n",
    "\n",
    "We can see that each line consists of four columns, including \"user id\" 1-943, \"item id\" 1-1682, \"rating\" 1-5 and \"timestamp\". We can construct an interaction matrix of size $n \\times m$, where $n$ and $m$ are the number of users and the number of items respectively. This dataset only records the existing ratings, so we can also call it rating matrix and we will use interaction matrix and rating matrix interchangeably in case that the values of this matrix represent exact ratings. Most of the values in the rating matrix are unknown as users have not rated the majority of movies. We also show the sparsity of this dataset. The sparsity is defined as `1 - number of nonzero entries / ( number of users * number of items)`. Clearly, the interaction matrix is extremely sparse (i.e., sparsity = 93.695%). Real world datasets may suffer from a greater extent of sparsity and has been a long-standing challenge in building recommender systems. A viable solution is to use additional side information such as user/item features to alleviate the sparsity.\n",
    "\n",
    "We then plot the distribution of the count of different ratings. As expected, it appears to be a normal distribution, with most ratings centered at 3-4."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "movielens-statistics-of-the-dataset-2"
   },
   "source": [
    "d2l.plt.hist(data['rating'], bins=5, ec='black')\n",
    "d2l.plt.xlabel('Rating')\n",
    "d2l.plt.ylabel('Count')\n",
    "d2l.plt.title('Distribution of Ratings in MovieLens 100K')\n",
    "d2l.plt.show()"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "image/png": "[encoded data, 27704 characters]"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=movielens-md-5 fw=all -->\n",
    "\n",
    "## Splitting the dataset\n",
    "\n",
    "We split the dataset into training and test sets. The following function provides two split modes including `random` and `seq-aware`. In the `random` mode, the function splits the 100k interactions randomly without considering timestamp and uses the 90% of the data as training samples and the rest 10% as test samples by default. In the `seq-aware` mode, we leave out the item that a user rated most recently for test, and users' historical interactions as training set.  User historical interactions are sorted from oldest to newest based on timestamp. This mode will be used in the sequence-aware recommendation section."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "movielens-splitting-the-dataset"
   },
   "source": [
    "\n",
    "def split_data_ml100k(data, num_users, num_items,\n",
    "                      split_mode='random', test_ratio=0.1):\n",
    "    \"\"\"Split the dataset in random mode or seq-aware mode.\"\"\"\n",
    "    if split_mode == 'seq-aware':\n",
    "        train_items, test_items, train_list = {}, {}, []\n",
    "        for line in data.itertuples():\n",
    "            u, i, rating, time = line[1], line[2], line[3], line[4]\n",
    "            train_items.setdefault(u, []).append((u, i, rating, time))\n",
    "            if u not in test_items or test_items[u][-1] < time:\n",
    "                test_items[u] = (i, rating, time)\n",
    "        for u in range(1, num_users + 1):\n",
    "            train_list.extend(sorted(train_items[u], key=lambda k: k[3]))\n",
    "        test_data = [(key, *value) for key, value in test_items.items()]\n",
    "        # O(N) set-membership filter instead of O(N^2) list-membership.\n",
    "        test_set = set(test_data)\n",
    "        train_data = [item for item in train_list if item not in test_set]\n",
    "        train_data = pd.DataFrame(train_data)\n",
    "        test_data = pd.DataFrame(test_data)\n",
    "    else:\n",
    "        # Seed for deterministic splits across frameworks. Use Python's\n",
    "        # `random` module rather than `np.random.default_rng` because\n",
    "        # mxnet.numpy.random doesn't expose the modern numpy RNG API.\n",
    "        rng = random.Random(0)\n",
    "        mask = [rng.random() < 1 - test_ratio for _ in range(len(data))]\n",
    "        neg_mask = [not x for x in mask]\n",
    "        train_data, test_data = data[mask], data[neg_mask]\n",
    "    return train_data, test_data"
   ],
   "execution_count": 5,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=movielens-md-6 fw=all -->\n",
    "\n",
    "Note that it is good practice to use a validation set in practice, apart from only a test set. However, we omit that for the sake of brevity. In this case, our test set can be regarded as our held-out validation set.\n",
    "\n",
    "## Loading the data\n",
    "\n",
    "After dataset splitting, we will convert the training set and test set into lists and dictionaries/matrix for the sake of convenience. The following function reads the dataframe line by line and enumerates the index of users/items start from zero. The function then returns lists of users, items, ratings and a dictionary/matrix that records the interactions. We can specify the type of feedback to either `explicit` or `implicit`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "movielens-loading-the-data-1"
   },
   "source": [
    "\n",
    "def load_data_ml100k(data, num_users, num_items, feedback='explicit'):\n",
    "    users, items, scores = [], [], []\n",
    "    inter = np.zeros((num_items, num_users)) if feedback == 'explicit' else {}\n",
    "    for line in data.itertuples():\n",
    "        user_index, item_index = int(line[1] - 1), int(line[2] - 1)\n",
    "        score = int(line[3]) if feedback == 'explicit' else 1\n",
    "        users.append(user_index)\n",
    "        items.append(item_index)\n",
    "        scores.append(score)\n",
    "        if feedback == 'implicit':\n",
    "            inter.setdefault(user_index, []).append(item_index)\n",
    "        else:\n",
    "            inter[item_index, user_index] = score\n",
    "    return users, items, scores, inter"
   ],
   "execution_count": 6,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=movielens-md-7 fw=all -->\n",
    "\n",
    "Afterwards, we put the above steps together and it will be used in the next section. The results are wrapped with `Dataset` and `DataLoader`. We keep the partial last batch in both frameworks (`last_batch='keep'` for MXNet, `drop_last=False` for PyTorch) so that no training samples are silently dropped from each epoch, and orders are shuffled."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "movielens-loading-the-data-2"
   },
   "source": [
    "\n",
    "def split_and_load_ml100k(split_mode='seq-aware', feedback='explicit',\n",
    "                          test_ratio=0.1, batch_size=256):\n",
    "    data, num_users, num_items = read_data_ml100k()\n",
    "    train_data, test_data = split_data_ml100k(\n",
    "        data, num_users, num_items, split_mode, test_ratio)\n",
    "    train_u, train_i, train_r, _ = load_data_ml100k(\n",
    "        train_data, num_users, num_items, feedback)\n",
    "    test_u, test_i, test_r, _ = load_data_ml100k(\n",
    "        test_data, num_users, num_items, feedback)\n",
    "    train_set = gluon.data.ArrayDataset(\n",
    "        np.array(train_u), np.array(train_i), np.array(train_r))\n",
    "    test_set = gluon.data.ArrayDataset(\n",
    "        np.array(test_u), np.array(test_i), np.array(test_r))\n",
    "    train_iter = gluon.data.DataLoader(\n",
    "        train_set, shuffle=True, last_batch='keep',\n",
    "        batch_size=batch_size)\n",
    "    test_iter = gluon.data.DataLoader(\n",
    "        test_set, batch_size=batch_size)\n",
    "    return num_users, num_items, train_iter, test_iter"
   ],
   "execution_count": 7,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=movielens-md-8 fw=all -->\n",
    "\n",
    "## Summary\n",
    "\n",
    "* MovieLens datasets are widely used for recommendation research. They are publicly available and free to use.\n",
    "* We define functions to download and preprocess the MovieLens 100k dataset for further use in later sections.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "* What other similar recommendation datasets can you find?\n",
    "* Go through the [https://movielens.org/](https://movielens.org/) site for more information about MovieLens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=movielens-md-9 fw=mxnet -->\n",
    "\n",
    "[Discussions](https://d2l.discourse.group/t/399)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-mxnet",
   "display_name": "d2l (MXNet)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
