%matplotlib inline
from d2l import tensorflow as d2l
import tensorflow as tf
import numpy as np
import osSemantic Segmentation Data
Semantic segmentation assigns a class label to every pixel, not just to the image as a whole. Output shape = input shape; output channels = number of classes.
Two related tasks to keep distinct:
- Image segmentation — group pixels by similarity (no semantic labels). Pure clustering.
- Instance segmentation — like semantic, but separate instances of the same class get different labels (Mask R-CNN).
Semantic segmentation: pixel-level labels for dog, cat, background.
This deck sets up the PASCAL VOC 2012 dataset and the data plumbing for FCN training (next deck).
Reading images and labels
Inputs are RGB images; labels are RGB images too — the class is encoded in the color, not in a 1-channel id tensor:
def read_voc_images(voc_dir, is_train=True):
"""Read all VOC feature and label images."""
from PIL import Image
txt_fname = os.path.join(voc_dir, 'ImageSets', 'Segmentation',
'train.txt' if is_train else 'val.txt')
with open(txt_fname, 'r') as f:
images = f.read().split()
features, labels = [], []
for i, fname in enumerate(images):
features.append(np.array(Image.open(os.path.join(
voc_dir, 'JPEGImages', f'{fname}.jpg'))))
labels.append(np.array(Image.open(os.path.join(
voc_dir, 'SegmentationClass', f'{fname}.png')).convert('RGB')))
return features, labels
train_features, train_labels = read_voc_images(voc_dir, True)n = 5
imgs = train_features[:n] + train_labels[:n]
d2l.show_images(imgs, 2, n);
Crop, not resize
Standard image preprocessing resizes — but resizing the label would interpolate class IDs, which is meaningless. Use random crop on both image and label, with the same random window:
def voc_rand_crop(feature, label, height, width):
"""Randomly crop both feature and label images."""
# Use NumPy for the random crop so this is safe to run from worker
# threads (tf.image.random_crop holds graph-side state that breaks
# under tf.data parallel py_function calls).
H, W = feature.shape[0], feature.shape[1]
top = int(np.random.randint(0, H - height + 1))
left = int(np.random.randint(0, W - width + 1))
feat = feature[top:top + height, left:left + width, :]
lab = label[top:top + height, left:left + width, :]
return feat, labimgs = []
for _ in range(n):
imgs += voc_rand_crop(train_features[0], train_labels[0], 200, 300)
d2l.show_images(imgs[::2] + imgs[1::2], 2, n);