{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  enabled: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=semantic-segmentation-and-dataset-md-1 fw=all -->\n",
    "\n",
    "# Semantic Segmentation and the Dataset\n",
    "\n",
    "When discussing object detection tasks\n",
    "in that section to that section,\n",
    "rectangular bounding boxes\n",
    "are used to label and predict objects in images.\n",
    "This section will discuss the problem of *semantic segmentation*,\n",
    "which focuses on how to divide an image into regions belonging to different semantic classes.\n",
    "Different from object detection,\n",
    "semantic segmentation\n",
    "recognizes and understands\n",
    "what are in images in pixel level:\n",
    "its labeling and prediction of semantic regions are\n",
    "in pixel level.\n",
    "the figure shows the labels\n",
    "of the dog, cat, and background of the image in semantic segmentation.\n",
    "Compared with in object detection,\n",
    "the pixel-level borders labeled\n",
    "in semantic segmentation are obviously more fine-grained.\n",
    "\n",
    "\n",
    "![Labels of the dog, cat, and background of the image in semantic segmentation.](../img/segmentation.svg)\n",
    "\n",
    "\n",
    "## Image Segmentation and Instance Segmentation\n",
    "\n",
    "There are also two important tasks\n",
    "in the field of computer vision that are similar to semantic segmentation,\n",
    "namely image segmentation and instance segmentation.\n",
    "We will briefly\n",
    "distinguish them from semantic segmentation as follows.\n",
    "\n",
    "* *Image segmentation* divides an image into several constituent regions. The methods for this type of problem usually make use of the correlation between pixels in the image. It does not need label information about image pixels during training, and it cannot guarantee that the segmented regions will have the semantics that we hope to obtain during prediction. Taking the image in the figure as input, image segmentation may divide the dog into two regions: one covers the mouth and eyes which are mainly black, and the other covers the rest of the body which is mainly yellow.\n",
    "* *Instance segmentation* is also called *simultaneous detection and segmentation*. It studies how to recognize the pixel-level regions of each object instance in an image. Different from semantic segmentation, instance segmentation needs to distinguish not only semantics, but also different object instances. For example, if there are two dogs in the image, instance segmentation needs to distinguish which of the two dogs a pixel belongs to.\n",
    "\n",
    "\n",
    "## The Pascal VOC2012 Semantic Segmentation Dataset\n",
    "\n",
    "One of the most important semantic segmentation dataset\n",
    "is [Pascal VOC2012](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/).\n",
    "In the following,\n",
    "we will take a look at this dataset."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "semantic-segmentation-and-dataset-the-pascal-voc2012-semantic-segmentation-dataset-1"
   },
   "source": [
    "%matplotlib inline\n",
    "from d2l import mxnet as d2l\n",
    "from mxnet import gluon, image, np, npx\n",
    "import os\n",
    "\n",
    "npx.set_np()"
   ],
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=semantic-segmentation-and-dataset-md-2 fw=all -->\n",
    "\n",
    "The tar file of the dataset is about 2 GB,\n",
    "so it may take a while to download the file.\n",
    "The extracted dataset is located at `../data/VOCdevkit/VOC2012`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "semantic-segmentation-and-dataset-the-pascal-voc2012-semantic-segmentation-dataset-2"
   },
   "source": [
    "\n",
    "d2l.DATA_HUB['voc2012'] = (d2l.DATA_URL + 'VOCtrainval_11-May-2012.tar',\n",
    "                           '4e443f8a2eca6b1dac8a6c57641b67dd40621a49')\n",
    "\n",
    "voc_dir = d2l.download_extract('voc2012', 'VOCdevkit/VOC2012')"
   ],
   "execution_count": 2,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=semantic-segmentation-and-dataset-md-3 fw=all -->\n",
    "\n",
    "After entering the path `../data/VOCdevkit/VOC2012`,\n",
    "we can see the different components of the dataset.\n",
    "The `ImageSets/Segmentation` path contains text files\n",
    "that specify training and test samples,\n",
    "while the `JPEGImages` and `SegmentationClass` paths\n",
    "store the input image and label for each example, respectively.\n",
    "The label here is also in the image format,\n",
    "with the same size\n",
    "as its labeled input image.\n",
    "Besides,\n",
    "pixels with the same color in any label image belong to the same semantic class.\n",
    "The following defines the `read_voc_images` function to read all the input images and labels into the memory."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "semantic-segmentation-and-dataset-the-pascal-voc2012-semantic-segmentation-dataset-3"
   },
   "source": [
    "\n",
    "def read_voc_images(voc_dir, is_train=True):\n",
    "    \"\"\"Read all VOC feature and label images.\"\"\"\n",
    "    txt_fname = os.path.join(voc_dir, 'ImageSets', 'Segmentation',\n",
    "                             'train.txt' if is_train else 'val.txt')\n",
    "    with open(txt_fname, 'r') as f:\n",
    "        images = f.read().split()\n",
    "    features, labels = [], []\n",
    "    for i, fname in enumerate(images):\n",
    "        features.append(image.imread(os.path.join(\n",
    "            voc_dir, 'JPEGImages', f'{fname}.jpg')))\n",
    "        labels.append(image.imread(os.path.join(\n",
    "            voc_dir, 'SegmentationClass', f'{fname}.png')))\n",
    "    return features, labels\n",
    "\n",
    "train_features, train_labels = read_voc_images(voc_dir, True)"
   ],
   "execution_count": 3,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=semantic-segmentation-and-dataset-md-4 fw=all -->\n",
    "\n",
    "We draw the first five input images and their labels.\n",
    "In the label images, white and black represent borders and  background, respectively, while the other colors correspond to different classes."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "semantic-segmentation-and-dataset-the-pascal-voc2012-semantic-segmentation-dataset-4"
   },
   "source": [
    "n = 5\n",
    "imgs = train_features[:n] + train_labels[:n]\n",
    "d2l.show_images(imgs, 2, n);"
   ],
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "image/png": "[encoded data, 114468 characters]"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=semantic-segmentation-and-dataset-md-5 fw=all -->\n",
    "\n",
    "Next, we enumerate\n",
    "the RGB color values and class names\n",
    "for all the labels in this dataset."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "semantic-segmentation-and-dataset-the-pascal-voc2012-semantic-segmentation-dataset-5"
   },
   "source": [
    "\n",
    "VOC_COLORMAP = [[0, 0, 0], [128, 0, 0], [0, 128, 0], [128, 128, 0],\n",
    "                [0, 0, 128], [128, 0, 128], [0, 128, 128], [128, 128, 128],\n",
    "                [64, 0, 0], [192, 0, 0], [64, 128, 0], [192, 128, 0],\n",
    "                [64, 0, 128], [192, 0, 128], [64, 128, 128], [192, 128, 128],\n",
    "                [0, 64, 0], [128, 64, 0], [0, 192, 0], [128, 192, 0],\n",
    "                [0, 64, 128]]\n",
    "\n",
    "\n",
    "VOC_CLASSES = ['background', 'aeroplane', 'bicycle', 'bird', 'boat',\n",
    "               'bottle', 'bus', 'car', 'cat', 'chair', 'cow',\n",
    "               'diningtable', 'dog', 'horse', 'motorbike', 'person',\n",
    "               'potted plant', 'sheep', 'sofa', 'train', 'tv/monitor']"
   ],
   "execution_count": 5,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=semantic-segmentation-and-dataset-md-6 fw=all -->\n",
    "\n",
    "With the two constants defined above,\n",
    "we can conveniently\n",
    "find the class index for each pixel in a label.\n",
    "We define the `voc_colormap2label` function\n",
    "to build the mapping from the above RGB color values\n",
    "to class indices,\n",
    "and the `voc_label_indices` function\n",
    "to map any RGB values to their class indices in this Pascal VOC2012 dataset."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "semantic-segmentation-and-dataset-the-pascal-voc2012-semantic-segmentation-dataset-6"
   },
   "source": [
    "\n",
    "def voc_colormap2label():\n",
    "    \"\"\"Build the mapping from RGB to class indices for VOC labels.\"\"\"\n",
    "    colormap2label = np.zeros(256 ** 3)\n",
    "    for i, colormap in enumerate(VOC_COLORMAP):\n",
    "        colormap2label[\n",
    "            (colormap[0] * 256 + colormap[1]) * 256 + colormap[2]] = i\n",
    "    return colormap2label\n",
    "\n",
    "\n",
    "def voc_label_indices(colormap, colormap2label):\n",
    "    \"\"\"Map any RGB values in VOC labels to their class indices.\"\"\"\n",
    "    colormap = colormap.astype(np.int32)\n",
    "    idx = ((colormap[:, :, 0] * 256 + colormap[:, :, 1]) * 256\n",
    "           + colormap[:, :, 2])\n",
    "    return colormap2label[idx]"
   ],
   "execution_count": 6,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=semantic-segmentation-and-dataset-md-7 fw=all -->\n",
    "\n",
    "For example, in the first example image,\n",
    "the class index for the front part of the airplane is 1,\n",
    "while the background index is 0."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "semantic-segmentation-and-dataset-the-pascal-voc2012-semantic-segmentation-dataset-7"
   },
   "source": [
    "y = voc_label_indices(train_labels[0], voc_colormap2label())\n",
    "y[105:115, 130:140], VOC_CLASSES[1]"
   ],
   "execution_count": 7,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "(array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n        [0., 0., 0., 0., 0., 0., 0., 1., 1., 1.],\n        [0., 0., 0., 0., 0., 0., 1., 1., 1., 1.],\n        [0., 0., 0., 0., 0., 1., 1., 1., 1., 1.],\n        [0., 0., 0., 0., 0., 1., 1., 1., 1., 1.],\n        [0., 0., 0., 0., 1., 1., 1., 1., 1., 1.],\n        [0., 0., 0., 0., 0., 1., 1., 1., 1., 1.],\n        [0., 0., 0., 0., 0., 1., 1., 1., 1., 1.],\n        [0., 0., 0., 0., 0., 0., 1., 1., 1., 1.],\n        [0., 0., 0., 0., 0., 0., 0., 0., 1., 1.]]),\n 'aeroplane')"
     },
     "metadata": {},
     "execution_count": 7
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=semantic-segmentation-and-dataset-md-8 fw=all -->\n",
    "\n",
    "### Data Preprocessing\n",
    "\n",
    "In previous experiments\n",
    "such as in that section to that section,\n",
    "images are rescaled\n",
    "to fit the model's required input shape.\n",
    "However, in semantic segmentation,\n",
    "doing so\n",
    "requires rescaling the predicted pixel classes\n",
    "back to the original shape of the input image.\n",
    "Such rescaling may be inaccurate,\n",
    "especially for segmented regions with different classes. To avoid this issue,\n",
    "we crop the image to a *fixed* shape instead of rescaling. Specifically, using random cropping from image augmentation, we crop the same area of\n",
    "the input image and the label."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "semantic-segmentation-and-dataset-data-preprocessing-1"
   },
   "source": [
    "\n",
    "def voc_rand_crop(feature, label, height, width):\n",
    "    \"\"\"Randomly crop both feature and label images.\"\"\"\n",
    "    feature, rect = image.random_crop(feature, (width, height))\n",
    "    label = image.fixed_crop(label, *rect)\n",
    "    return feature, label"
   ],
   "execution_count": 8,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "semantic-segmentation-and-dataset-data-preprocessing-2"
   },
   "source": [
    "imgs = []\n",
    "for _ in range(n):\n",
    "    imgs += voc_rand_crop(train_features[0], train_labels[0], 200, 300)\n",
    "d2l.show_images(imgs[::2] + imgs[1::2], 2, n);"
   ],
   "execution_count": 9,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "image/png": "[encoded data, 107108 characters]"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=semantic-segmentation-and-dataset-md-9 fw=all -->\n",
    "\n",
    "### Custom Semantic Segmentation Dataset Class\n",
    "\n",
    "We define a custom semantic segmentation dataset class `VOCSegDataset` by inheriting the `Dataset` class provided by high-level APIs.\n",
    "By implementing the `__getitem__` function,\n",
    "we can arbitrarily access the input image indexed as `idx` in the dataset and the class index of each pixel in this image.\n",
    "Since some images in the dataset\n",
    "have a smaller size\n",
    "than the output size of random cropping,\n",
    "these examples are filtered out\n",
    "by a custom `filter` function.\n",
    "In addition, we also\n",
    "define the `normalize_image` function to\n",
    "standardize the values of the three RGB channels of input images."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "semantic-segmentation-and-dataset-custom-semantic-segmentation-dataset-class"
   },
   "source": [
    "\n",
    "class VOCSegDataset(gluon.data.Dataset):\n",
    "    \"\"\"A customized dataset to load the VOC dataset.\"\"\"\n",
    "    def __init__(self, is_train, crop_size, voc_dir):\n",
    "        self.rgb_mean = np.array([0.485, 0.456, 0.406])\n",
    "        self.rgb_std = np.array([0.229, 0.224, 0.225])\n",
    "        self.crop_size = crop_size\n",
    "        features, labels = read_voc_images(voc_dir, is_train=is_train)\n",
    "        self.features = [self.normalize_image(feature)\n",
    "                         for feature in self.filter(features)]\n",
    "        self.labels = self.filter(labels)\n",
    "        self.colormap2label = voc_colormap2label()\n",
    "        print('read ' + str(len(self.features)) + ' examples')\n",
    "\n",
    "    def normalize_image(self, img):\n",
    "        return (img.astype('float32') / 255 - self.rgb_mean) / self.rgb_std\n",
    "\n",
    "    def filter(self, imgs):\n",
    "        return [img for img in imgs if (\n",
    "            img.shape[0] >= self.crop_size[0] and\n",
    "            img.shape[1] >= self.crop_size[1])]\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        feature, label = voc_rand_crop(self.features[idx], self.labels[idx],\n",
    "                                       *self.crop_size)\n",
    "        return (feature.transpose(2, 0, 1),\n",
    "                voc_label_indices(label, self.colormap2label))\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.features)"
   ],
   "execution_count": 10,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=semantic-segmentation-and-dataset-md-10 fw=all -->\n",
    "\n",
    "### Reading the Dataset\n",
    "\n",
    "We use the custom `VOCSegDataset` class to\n",
    "create instances of the training set and test set, respectively.\n",
    "Suppose that\n",
    "we specify that the output shape of randomly cropped images is $320\\times 480$.\n",
    "Below we can view the number of examples\n",
    "that are retained in the training set and test set."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "semantic-segmentation-and-dataset-reading-the-dataset-1"
   },
   "source": [
    "crop_size = (320, 480)\n",
    "voc_train = VOCSegDataset(True, crop_size, voc_dir)\n",
    "voc_test = VOCSegDataset(False, crop_size, voc_dir)"
   ],
   "execution_count": 11,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "read 1114 examples\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "read 1078 examples\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=semantic-segmentation-and-dataset-md-11 fw=all -->\n",
    "\n",
    "Setting the batch size to 64,\n",
    "we define the data iterator for the training set.\n",
    "Let's print the shape of the first minibatch.\n",
    "Different from in image classification or object detection, labels here are three-dimensional tensors."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "semantic-segmentation-and-dataset-reading-the-dataset-2"
   },
   "source": [
    "batch_size = 64\n",
    "train_iter = gluon.data.DataLoader(voc_train, batch_size, shuffle=True,\n",
    "                                   last_batch='discard',\n",
    "                                   num_workers=d2l.get_dataloader_workers())\n",
    "for X, Y in train_iter:\n",
    "    print(X.shape)\n",
    "    print(Y.shape)\n",
    "    break"
   ],
   "execution_count": 12,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h[:521: \nStorage type fallback detected:\noperator = _npi_transpose\ninput storage types = [default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = _npi_transpose\ninput storage types = [default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = _npi_transpose\ninput storage types = [default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521[: \nStorage type fallback detected:\noperator = _npi_add\ninput storage types = [default, default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.15:21:57\n] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = _npi_add\ninput storage types = [default, default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = _npi_add\ninput storage types = [default, default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = _npx_reshape\ninput storage types = [default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = _npx_reshape\ninput storage types = [default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = stack\ninput storage types = [default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = stack\ninput storage types = [default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = _npx_reshape\ninput storage types = [default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = stack\ninput storage types = [default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = _npi_transpose\ninput storage types = [default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = _npi_add\ninput storage types = [default, default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = _npx_reshape\ninput storage types = [default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = stack\ninput storage types = [default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = stack\ninput storage types = [default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = stack\ninput storage types = [default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = stack\ninput storage types = [default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n[15:21:57] /home/smola/mxnet/src/imperative/./../common/../common/utils.h:521: \nStorage type fallback detected:\noperator = stack\ninput storage types = [default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, default, ]\noutput storage types = [default, ]\nparams = {}\ncontext.dev_mask = cpu\n WARNING:\nExecution of the operator above will fallback to the generic implementation (not utilizing kernels from oneDNN library) with default dense storage type. You are seeing this warning message because MXNET_ONEDNN_ENABLED flag is set to 0, in which case you can re-enable the default execution path by setting MXNET_ONEDNN_ENABLED back to 1, or the operator above is unable to process the given ndarrays with specified storage types, context and/or parameter, in which case temporary dense ndarrays are generated in order to execute the operator. The fallback does not affect the correctness of the programme. Using default storage type performance degradation might be observed. \nYou can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "(64, 3, 320, 480)\n(64, 320, 480)\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=semantic-segmentation-and-dataset-md-12 fw=all -->\n",
    "\n",
    "### Putting It All Together\n",
    "\n",
    "Finally, we define the following `load_data_voc` function\n",
    "to download and read the Pascal VOC2012 semantic segmentation dataset.\n",
    "It returns data iterators for both the training and test datasets."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "semantic-segmentation-and-dataset-putting-it-all-together"
   },
   "source": [
    "\n",
    "def load_data_voc(batch_size, crop_size):\n",
    "    \"\"\"Load the VOC semantic segmentation dataset.\"\"\"\n",
    "    voc_dir = d2l.download_extract('voc2012', os.path.join(\n",
    "        'VOCdevkit', 'VOC2012'))\n",
    "    num_workers = d2l.get_dataloader_workers()\n",
    "    train_iter = gluon.data.DataLoader(\n",
    "        VOCSegDataset(True, crop_size, voc_dir), batch_size,\n",
    "        shuffle=True, last_batch='discard', num_workers=num_workers)\n",
    "    test_iter = gluon.data.DataLoader(\n",
    "        VOCSegDataset(False, crop_size, voc_dir), batch_size,\n",
    "        last_batch='discard', num_workers=num_workers)\n",
    "    return train_iter, test_iter"
   ],
   "execution_count": 13,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=semantic-segmentation-and-dataset-md-13 fw=all -->\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Semantic segmentation recognizes and understands what are in an image in pixel level by dividing the image into regions belonging to different semantic classes.\n",
    "* One of the most important semantic segmentation dataset is Pascal VOC2012.\n",
    "* In semantic segmentation, since the input image and  label correspond one-to-one on the pixel, the input image is randomly cropped to a fixed shape rather than rescaled.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. How can semantic segmentation be applied in autonomous vehicles and medical image diagnostics? Can you think of other applications?\n",
    "1. Recall the descriptions of data augmentation in that section. Which of the image augmentation methods used in image classification would be infeasible to be applied in semantic segmentation?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- d2l:prose id=semantic-segmentation-and-dataset-md-14 fw=mxnet -->\n",
    "\n",
    "[Discussions](https://d2l.discourse.group/t/375)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "d2l-mxnet",
   "display_name": "d2l (MXNet)",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
