Verify multi-input convolution

Multiple Input and Multiple Output Channels

Channels turn filters into feature banks

Real images have channels: RGB has 3, a modern CNN’s deep feature map has hundreds (64 → 2048 is typical).

Going deeper, networks trade spatial resolution for channel depth — same information capacity, but representing kinds of features instead of places.

This deck:

  • Multiple input channels — kernels grow a 3rd axis.
  • Multiple output channels — stack many filters.
  • 1 \times 1 convolutions — pure channel mixing.

Multi-input-channel convolution

With c_i input channels, the kernel becomes c_i \times k_h \times k_w — a 2D filter per input channel. The output is the sum of per-channel cross-correlations:

Y = \sum_{c=1}^{c_i} X_c * K_c.

Two input channels: per-channel cross-correlation, then sum. (1{\cdot}1 + 2{\cdot}2 + 4{\cdot}3 + 5{\cdot}4) + (0{\cdot}0 + 1{\cdot}1 + 3{\cdot}2 + 4{\cdot}3) = 56.

Input channels in code

from d2l import tensorflow as d2l
import tensorflow as tf
def corr2d_multi_in(X, K):
    # Iterate through the 0th dimension (channel) of K first, then add them up
    return tf.reduce_sum([d2l.corr2d(x, k) for x, k in zip(X, K)], axis=0)

Verify against the figure — same numbers:

X = d2l.tensor([[[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]],
               [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]]])
K = d2l.tensor([[[0.0, 1.0], [2.0, 3.0]], [[1.0, 2.0], [3.0, 4.0]]])

corr2d_multi_in(X, K)
<tf.Tensor: shape=(2, 2), dtype=float32, numpy=
array([[ 56.,  72.],
       [104., 120.]], dtype=float32)>

Multi-output-channel convolution

Each output channel comes from its own set of input-channel filters. Stack c_o such filter sets to get a 4-D kernel of shape c_o \times c_i \times k_h \times k_w:

\mathbf{Y}_j = \sum_{c=1}^{c_i} \mathbf{X}_c * \mathbf{K}_{j, c} \quad\text{for}\quad j = 1, \ldots, c_o.

Intuition: each of the c_o output channels is a different combination of inputs, learned to detect a different feature. The network discovers an entire “feature dictionary” per layer.

Output channels in code

Apply the multi-input-channel function c_o times and stack the results along a new leading axis:

def corr2d_multi_in_out(X, K):
    # Iterate through the 0th dimension of K, and each time, perform
    # cross-correlation operations with input X. All of the results are
    # stacked together
    return d2l.stack([corr2d_multi_in(X, k) for k in K], 0)

Verify multi-output convolution

Build a 3-output-channel kernel by stacking three offset copies:

K = d2l.stack((K, K + 1, K + 2), 0)
K.shape
TensorShape([3, 2, 2, 2])
corr2d_multi_in_out(X, K)
<tf.Tensor: shape=(3, 2, 2), dtype=float32, numpy=
array([[[ 56.,  72.],
        [104., 120.]],

       [[ 76., 100.],
        [148., 172.]],

       [[ 96., 128.],
        [192., 224.]]], dtype=float32)>

Parameter count

A conv layer with c_o outputs, c_i inputs, and a k_h \times k_w kernel has

c_o \cdot c_i \cdot k_h \cdot k_w \;+\; c_o

learnable parameters. Standard sizes:

  • 3 → 64 channels, 3×3 kernel: 1.7k weights.
  • 256 → 256 channels, 3×3 kernel: 590k weights.
  • 512 → 2048 channels, 3×3 kernel: 9.4M weights.

Channel count drives parameter count quadratically. That’s why deeper layers in CNNs widen, but not too much.

The 1×1 convolution

A 1×1 kernel has no spatial structure — it doesn’t look at neighbors. So why use it?

Because it acts as a per-pixel fully connected layer across channels. At every spatial position, it computes a linear combination of the c_i input channels into the c_o output channels:

1×1 conv: 3 input channels × 2 output channels. Each output pixel = a 2×3 matrix-vector product on the input channel vector at that position.

1×1 conv as matmul

At each pixel, the 1×1 conv applies the same c_o \times c_i matrix to the input channel vector. Reshape the spatial axes out and it’s a single matrix multiply:

def corr2d_multi_in_out_1x1(X, K):
    c_i, h, w = X.shape
    c_o = K.shape[0]
    X = d2l.reshape(X, (c_i, h * w))
    K = d2l.reshape(K, (c_o, c_i))
    # Matrix multiplication in the fully connected layer
    Y = d2l.matmul(K, X)
    return d2l.reshape(Y, (c_o, h, w))
X = d2l.normal((3, 3, 3), 0, 1)
K = d2l.normal((2, 3, 1, 1), 0, 1)
Y1 = corr2d_multi_in_out_1x1(X, K)
Y2 = corr2d_multi_in_out(X, K)
assert float(d2l.reduce_sum(d2l.abs(Y1 - Y2))) < 1e-6

Why 1×1 convs everywhere

Modern architectures use them constantly:

  • Bottlenecks (ResNet) — squeeze channels with 1×1, do expensive 3×3 in the smaller space, expand back with 1×1. Cuts compute by \sim 4× for the same expressive power.
  • Pointwise convs (MobileNet) — depthwise 3×3 + pointwise 1×1 splits a regular conv into two cheaper pieces.
  • Squeeze-and-Excitation, attention heads — wherever you need cheap channel mixing without changing resolution.

Cost of channel depth

A h \times w image with k \times k kernel and c_i \to c_o channels takes

\mathcal{O}(h \cdot w \cdot k^2 \cdot c_i \cdot c_o)

operations. For a 256×256 image, 5×5 kernel, 128→128 channels: ~53 billion multiply-adds. That’s per layer; multiply by depth.

This is why - Convs benefit massively from GPU/TPU acceleration. - Channel reduction tricks (1×1 bottlenecks, depthwise separable, group conv, ResNeXt) get serious attention in efficiency-driven architectures.

Recap

  • Multi-input-channel: per-channel kernel, results summed across input channels.
  • Multi-output-channel: stack independent filter banks; one output channel per filter set.
  • Total weights: c_o c_i k_h k_w + c_o.
  • 1\times 1 convs = per-pixel fully connected layer across channels; the workhorse of modern CNN architecture (bottlenecks, pointwise convs).
  • Compute scales linearly with c_i \cdot c_o — the dominant cost in deep CNNs.