Multiple Input and Multiple Output Channels

Channels turn filters into feature banks

Real images have channels: RGB has 3, a modern CNN’s deep feature map has hundreds (64 → 2048 is typical).

Going deeper, networks trade spatial resolution for channel depth — same information capacity, but representing kinds of features instead of places.

This deck:

• Multiple input channels — kernels grow a 3rd axis.
• Multiple output channels — stack many filters.
• 1 \times 1 convolutions — pure channel mixing.

Multi-input-channel convolution

With c_i input channels, the kernel becomes c_i \times k_h \times k_w — a 2D filter per input channel. The output is the sum of per-channel cross-correlations:

Y = \sum_{c=1}^{c_i} X_c * K_c.

Two input channels: per-channel cross-correlation, then sum. (1{\cdot}1 + 2{\cdot}2 + 4{\cdot}3 + 5{\cdot}4) + (0{\cdot}0 + 1{\cdot}1 + 3{\cdot}2 + 4{\cdot}3) = 56.

Input channels in code

from d2l import jax as d2l
from flax import nnx
import jax
from jax import numpy as jnp

def corr2d_multi_in(X, K):
    # Iterate through the 0th dimension (channel) of K first, then add them up
    return sum(d2l.corr2d(x, k) for x, k in zip(X, K))

Verify multi-input convolution

Verify against the figure — same numbers:

X = d2l.tensor([[[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]],
               [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]]])
K = d2l.tensor([[[0.0, 1.0], [2.0, 3.0]], [[1.0, 2.0], [3.0, 4.0]]])

corr2d_multi_in(X, K)

Array([[ 56.,  72.],
       [104., 120.]], dtype=float32)

Multi-output-channel convolution

Each output channel comes from its own set of input-channel filters. Stack c_o such filter sets to get a 4-D kernel of shape c_o \times c_i \times k_h \times k_w:

\mathbf{Y}_j = \sum_{c=1}^{c_i} \mathbf{X}_c * \mathbf{K}_{j, c} \quad\text{for}\quad j = 1, \ldots, c_o.

Intuition: each of the c_o output channels is a different combination of inputs, learned to detect a different feature. The network discovers an entire “feature dictionary” per layer.

Output channels in code

Apply the multi-input-channel function c_o times and stack the results along a new leading axis:

def corr2d_multi_in_out(X, K):
    # Iterate through the 0th dimension of K, and each time, perform
    # cross-correlation operations with input X. All of the results are
    # stacked together
    return d2l.stack([corr2d_multi_in(X, k) for k in K], 0)

Verify multi-output convolution

Build a 3-output-channel kernel by stacking three offset copies:

K = d2l.stack((K, K + 1, K + 2), 0)
K.shape

(3, 2, 2, 2)

corr2d_multi_in_out(X, K)

Array([[[ 56.,  72.],
        [104., 120.]],

       [[ 76., 100.],
        [148., 172.]],

       [[ 96., 128.],
        [192., 224.]]], dtype=float32)

Parameter count

A conv layer with c_o outputs, c_i inputs, and a k_h \times k_w kernel has

c_o \cdot c_i \cdot k_h \cdot k_w \;+\; c_o

learnable parameters. Standard sizes:

• 3 → 64 channels, 3×3 kernel: 1.7k weights.
• 256 → 256 channels, 3×3 kernel: 590k weights.
• 512 → 2048 channels, 3×3 kernel: 9.4M weights.

Channel count drives parameter count quadratically. That’s why deeper layers in CNNs widen, but not too much.

The 1×1 convolution

A 1×1 kernel has no spatial structure — it doesn’t look at neighbors. So why use it?

Because it acts as a per-pixel fully connected layer across channels. At every spatial position, it computes a linear combination of the c_i input channels into the c_o output channels:

1×1 conv: 3 input channels × 2 output channels. Each output pixel = a 2×3 matrix-vector product on the input channel vector at that position.

1×1 conv as matmul

At each pixel, the 1×1 conv applies the same c_o \times c_i matrix to the input channel vector. Reshape the spatial axes out and it’s a single matrix multiply:

def corr2d_multi_in_out_1x1(X, K):
    c_i, h, w = X.shape
    c_o = K.shape[0]
    X = d2l.reshape(X, (c_i, h * w))
    K = d2l.reshape(K, (c_o, c_i))
    # Matrix multiplication in the fully connected layer
    Y = d2l.matmul(K, X)
    return d2l.reshape(Y, (c_o, h, w))

X = jax.random.normal(d2l.get_key(), (3, 3, 3))
K = jax.random.normal(d2l.get_key(), (2, 3, 1, 1))
Y1 = corr2d_multi_in_out_1x1(X, K)
Y2 = corr2d_multi_in_out(X, K)
assert float(d2l.reduce_sum(d2l.abs(Y1 - Y2))) < 1e-5

Why 1×1 convs everywhere

Modern architectures use them constantly:

• Bottlenecks (ResNet) — squeeze channels with 1×1, do expensive 3×3 in the smaller space, expand back with 1×1. Cuts compute by \sim 4× for the same expressive power.
• Pointwise convs (MobileNet) — depthwise 3×3 + pointwise 1×1 splits a regular conv into two cheaper pieces.
• Squeeze-and-Excitation, attention heads — wherever you need cheap channel mixing without changing resolution.

Grouped and depthwise convolutions

A dense conv connects every input channel to every output channel. Split the channels into g groups and convolve each group separately: parameters and compute drop by a factor of g (ResNeXt’s trick).

The extreme g = c_i = c_o is a depthwise convolution: one k \times k filter per channel, no channel mixing at all.

Dense conv mixes all input channels; depthwise filters each channel separately; pointwise 1×1 mixes them back.

Depthwise-separable convolutions

Factor spatial filtering from channel mixing: depthwise k \times k, then pointwise 1 \times 1 (MobileNet, Xception).

\frac{\text{separable cost}}{\text{dense cost}} = \frac{1}{c_o} + \frac{1}{k^2} \approx \frac{1}{9} \;\text{ for } k = 3.

Parameter counts confirm it, 147k vs. 17.5k:

c_i, c_o, k = 128, 128, 3
X = jax.random.normal(d2l.get_key(), (1, 32, 32, c_i))
dense = nnx.Conv(c_i, c_o, kernel_size=(k, k), padding='SAME',
                 use_bias=False, rngs=nnx.Rngs(d2l.get_key()))
depthwise = nnx.Conv(c_i, c_i, kernel_size=(k, k), padding='SAME',
                     feature_group_count=c_i, use_bias=False,
                     rngs=nnx.Rngs(d2l.get_key()))
pointwise = nnx.Conv(c_i, c_o, kernel_size=(1, 1), use_bias=False,
                     rngs=nnx.Rngs(d2l.get_key()))
Y = depthwise(X)
assert dense(X).shape == pointwise(Y).shape
size = lambda model: sum(p.size for p in jax.tree_util.tree_leaves(
    nnx.state(model, nnx.Param)))
p_dense = size(dense)
p_sep = size(depthwise) + size(pointwise)
p_dense, p_sep, p_dense / p_sep

(147456, 17536, 8.408759124087592)

Cost of channel depth

A h \times w image with k \times k kernel and c_i \to c_o channels takes

\mathcal{O}(h \cdot w \cdot k^2 \cdot c_i \cdot c_o)

operations. For a 256×256 image, 5×5 kernel, 128→128 channels: ~53 billion multiply-adds. That’s per layer; multiply by depth.

This is why - Convs benefit massively from GPU/TPU acceleration. - Channel reduction tricks (1×1 bottlenecks, depthwise separable, group conv, ResNeXt) get serious attention in efficiency-driven architectures.

Recap

• Multi-input-channel: per-channel kernel, results summed across input channels.
• Multi-output-channel: stack independent filter banks; one output channel per filter set.
• Total weights: c_o c_i k_h k_w + c_o.
• 1\times 1 convs = per-pixel fully connected layer across channels; the workhorse of modern CNN architecture (bottlenecks, pointwise convs).
• Compute scales linearly with c_i \cdot c_o — the dominant cost in deep CNNs.