Transposed Convolution

A standard convolution + pooling stack reduces spatial resolution. For dense prediction (semantic segmentation, generative models, super-resolution) we need to go the other way — upsample features back to image resolution.

The standard tool: transposed convolution, also called “deconvolution” (a misnomer — it’s not a true inverse). Each input element broadcasts a full kernel into the output, contributions from neighbors get summed:

A 2 \times 2 transposed convolution: each input element scatters its kernel into the output.

Output shape grows: with stride 1, kernel k, no padding, n_{\text{out}} = n_{\text{in}} + k - 1. With stride s, multiplied accordingly.

From-scratch implementation

import jax
from jax import numpy as jnp
from flax import linen as nn
from d2l import jax as d2l
import numpy as np

def trans_conv(X, K):
    h, w = K.shape
    Y = d2l.zeros((X.shape[0] + h - 1, X.shape[1] + w - 1))
    for i in range(X.shape[0]):
        for j in range(X.shape[1]):
            Y = Y.at[i: i + h, j: j + w].add(X[i, j] * K)
    return Y

Verify on a small example

The hand-written implementation should match the framework operator. If the shape or values differ, the usual culprits are padding semantics or channel layout:

X = d2l.tensor([[0.0, 1.0], [2.0, 3.0]])
K = d2l.tensor([[0.0, 1.0], [2.0, 3.0]])
trans_conv(X, K)

Array([[ 0.,  0.,  1.],
       [ 0.,  4.,  6.],
       [ 4., 12.,  9.]], dtype=float32)

Same result via the framework op (PyTorch ConvTranspose2d, etc.):

X, K = X.reshape(1, 2, 2, 1), K.reshape(2, 2, 1, 1)
tconv = nn.ConvTranspose(1, kernel_size=(2, 2), use_bias=False)
params = tconv.init(jax.random.PRNGKey(0), X)
params = {**params, 'params': {'kernel': K}}
tconv.apply(params, X)

Array([[[[ 0.],
         [ 3.]],

        [[ 6.],
         [14.]]]], dtype=float32)

Padding, stride, channels

Padding here removes output rows/columns instead of adding them — it’s the inverse interpretation.

Stride > 1 inserts zeros between input elements before the scatter — that’s how transposed conv upsamples:

Stride-2 transposed conv: each input element’s kernel is placed at twice-spaced positions, then summed.

tconv = nn.ConvTranspose(1, kernel_size=(2, 2), padding='VALID', use_bias=False)
params = tconv.init(jax.random.PRNGKey(0), X)
params = {**params, 'params': {'kernel': K}}
# Apply then remove the outer border (equivalent to padding=1 in PyTorch)
out = tconv.apply(params, X)
out[:, 1:-1, 1:-1, :]

Array([[[[14.]]]], dtype=float32)

tconv = nn.ConvTranspose(1, kernel_size=(2, 2), strides=(2, 2), use_bias=False)
params = tconv.init(jax.random.PRNGKey(0), X)
params = {**params, 'params': {'kernel': K}}
tconv.apply(params, X)

Array([[[[0.],
         [0.],
         [3.],
         [2.]],

        [[0.],
...
         [6.]],

        [[2.],
         [0.],
         [3.],
         [0.]]]], dtype=float32)

Multi-channel works as expected: input channels reduce-add through the kernel, output channels stack in parallel:

# JAX uses channels-last format: (batch, height, width, channels)
X = jax.random.normal(jax.random.PRNGKey(0), (1, 16, 16, 10))
conv = nn.Conv(20, kernel_size=(5, 5), padding='SAME', strides=(3, 3))
tconv = nn.ConvTranspose(10, kernel_size=(5, 5), padding='SAME', strides=(3, 3))
params_conv = conv.init(jax.random.PRNGKey(1), X)
Y = conv.apply(params_conv, X)
params_tconv = tconv.init(jax.random.PRNGKey(2), Y)
tconv.apply(params_tconv, Y).shape == X.shape

False

Connection to matrix transposition

A standard convolution can be written as a sparse matrix multiplication \mathbf{y} = \mathbf{K}\mathbf{x} where \mathbf{K} encodes the kernel + stride + padding.

A transposed convolution multiplies by the transpose: \mathbf{x}' = \mathbf{K}^\top \mathbf{y}. That’s where the name comes from.

X = d2l.reshape(d2l.arange(9.0), (3, 3))
K = d2l.tensor([[1.0, 2.0], [3.0, 4.0]])
Y = d2l.corr2d(X, K)
Y

Array([[27., 37.],
       [57., 67.]], dtype=float32)

def kernel2matrix(K):
    k = jnp.zeros(5)
    k = k.at[:2].set(K[0, :])
    k = k.at[3:5].set(K[1, :])
    W = jnp.zeros((4, 9))
    W = W.at[0, :5].set(k)
    W = W.at[1, 1:6].set(k)
    W = W.at[2, 3:8].set(k)
    W = W.at[3, 4:].set(k)
    return W

W = kernel2matrix(K)
W

Array([[1., 2., 0., 3., 4., 0., 0., 0., 0.],
       [0., 1., 2., 0., 3., 4., 0., 0., 0.],
       [0., 0., 0., 1., 2., 0., 3., 4., 0.],
       [0., 0., 0., 0., 1., 2., 0., 3., 4.]], dtype=float32)

Matrix view (cont.)

Y == d2l.reshape(d2l.matmul(W, d2l.reshape(X, (-1, 1))), (2, 2))

Array([[ True,  True],
       [ True,  True]], dtype=bool)

Z = trans_conv(Y, K)
Z == d2l.reshape(d2l.matmul(d2l.transpose(W), d2l.reshape(Y, (-1, 1))), (3, 3))

Array([[ True,  True,  True],
       [ True,  True,  True],
       [ True,  True,  True]], dtype=bool)

Recap

Transposed conv = upsampling op; each input element scatters a full kernel into the output and overlapping contributions sum.
Stride > 1 inserts zeros between inputs → upsamples by s.
Mathematically the transpose of a normal convolution’s matrix form (hence the name).
Workhorse for FCN, U-Net, GAN generators, VAE decoders.
Modern alternative: bilinear upsample + 3×3 conv — avoids checkerboard artifacts that transposed conv can produce.