Multiple Input and Multiple Output Channels

Channels turn filters into feature banks

Real images have channels: RGB has 3, while deep CNN feature maps may have hundreds or thousands.

Going deeper, networks often trade spatial resolution for channel depth, representing more feature types at fewer locations.

This deck:

  • Multiple input channels — kernels grow a 3rd axis.
  • Multiple output channels — stack many filters.
  • 1 \times 1 convolutions — pure channel mixing.

Multi-input-channel convolution

With c_i input channels, the kernel becomes c_i \times k_h \times k_w — a 2D filter per input channel. The output is the sum of per-channel cross-correlations:

Y = \sum_{c=1}^{c_i} X_c * K_c.

Two input channels: per-channel cross-correlation, then sum. (1{\cdot}1 + 2{\cdot}2 + 4{\cdot}3 + 5{\cdot}4) + (0{\cdot}0 + 1{\cdot}1 + 3{\cdot}2 + 4{\cdot}3) = 56.

Input channels in code

from d2l import jax as d2l
from flax import nnx
import jax
from jax import numpy as jnp
def corr2d_multi_in(X, K):
    # Iterate through the 0th dimension (channel) of K first, then add them up
    return sum(d2l.corr2d(x, k) for x, k in zip(X, K))

Verify multi-input convolution

Verify against the figure — same numbers:

X = d2l.tensor([[[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]],
               [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]]])
K = d2l.tensor([[[0.0, 1.0], [2.0, 3.0]], [[1.0, 2.0], [3.0, 4.0]]])

corr2d_multi_in(X, K)
Array([[ 56.,  72.],
       [104., 120.]], dtype=float32)

Multi-output-channel convolution

Each output channel comes from its own set of input-channel filters. Stack c_o such filter sets to get a 4-D kernel of shape c_o \times c_i \times k_h \times k_w:

\mathbf{Y}_j = \sum_{c=1}^{c_i} \mathbf{X}_c * \mathbf{K}_{j, c} \quad\text{for}\quad j = 1, \ldots, c_o.

Intuition: each of the c_o output channels is a different combination of inputs, learned to detect a different feature. Together, the channels form a learned feature representation.

Output channels in code

Apply the multi-input-channel function c_o times and stack the results along a new leading axis:

def corr2d_multi_in_out(X, K):
    # Iterate through the 0th dimension of K, and each time, perform
    # cross-correlation operations with input X. All of the results are
    # stacked together
    return d2l.stack([corr2d_multi_in(X, k) for k in K], 0)

Verify multi-output convolution

Build a 3-output-channel kernel by stacking three offset copies:

K = d2l.stack((K, K + 1, K + 2), 0)
K.shape
(3, 2, 2, 2)
corr2d_multi_in_out(X, K)
Array([[[ 56.,  72.],
        [104., 120.]],

       [[ 76., 100.],
        [148., 172.]],

       [[ 96., 128.],
        [192., 224.]]], dtype=float32)

Parameter count

A conv layer with c_o outputs, c_i inputs, and a k_h \times k_w kernel has

c_o \cdot c_i \cdot k_h \cdot k_w \;+\; c_o

learnable parameters. Standard sizes:

  • 3 → 64 channels, 3×3 kernel: 1.7k weights.
  • 256 → 256 channels, 3×3 kernel: 590k weights.
  • 512 → 2048 channels, 3×3 kernel: 9.4M weights.

If input and output channel counts grow together, parameter count grows quadratically, which constrains how quickly networks can widen.

The 1×1 convolution

A 1 \times 1 kernel has no spatial extent beyond its current position.

Because it acts as a per-pixel fully connected layer across channels. At every spatial position, it computes a linear combination of the c_i input channels into the c_o output channels:

1×1 conv: 3 input channels × 2 output channels. Each output pixel = a 2×3 matrix-vector product on the input channel vector at that position.

1×1 conv as matmul

At each pixel, the 1×1 conv applies the same c_o \times c_i matrix to the input channel vector. Reshape the spatial axes out and it’s a single matrix multiply:

def corr2d_multi_in_out_1x1(X, K):
    c_i, h, w = X.shape
    c_o = K.shape[0]
    X = d2l.reshape(X, (c_i, h * w))
    K = d2l.reshape(K, (c_o, c_i))
    # Matrix multiplication in the fully connected layer
    Y = d2l.matmul(K, X)
    return d2l.reshape(Y, (c_o, h, w))
X = jax.random.normal(d2l.get_key(), (3, 3, 3))
K = jax.random.normal(d2l.get_key(), (2, 3, 1, 1))
Y1 = corr2d_multi_in_out_1x1(X, K)
Y2 = corr2d_multi_in_out(X, K)
assert float(d2l.reduce_sum(d2l.abs(Y1 - Y2))) < 1e-5

Why 1×1 convs everywhere

Common uses include:

  • Bottlenecks (ResNet) — squeeze channels with 1×1, do expensive 3×3 in the smaller space, expand back with 1×1, reducing the cost of the spatial convolution. The exact saving depends on channel widths and can be \sim 4× overall.
  • Pointwise convs (MobileNet) — depthwise 3×3 + pointwise 1×1 splits a regular conv into two cheaper pieces.
  • Squeeze-and-Excitation, attention heads — wherever you need cheap channel mixing without changing resolution.

Grouped and depthwise convolutions

A dense conv connects every input channel to every output channel. Split the channels into g groups and convolve each group separately: parameters and compute drop by a factor of g (as in ResNeXt).

The extreme g = c_i = c_o is a depthwise convolution: one k \times k filter per channel, no channel mixing at all.

Dense conv mixes all input channels; depthwise filters each channel separately; pointwise 1×1 mixes them back.

Depthwise-separable convolutions

Factor spatial filtering from channel mixing: depthwise k \times k, then pointwise 1 \times 1 (MobileNet, Xception).

\frac{\text{separable cost}}{\text{dense cost}} = \frac{1}{c_o} + \frac{1}{k^2} \approx \frac{1}{9} \;\text{ for } k = 3.

Parameter counts confirm it, 147k vs. 17.5k:

c_i, c_o, k = 128, 128, 3
X = jax.random.normal(d2l.get_key(), (1, 32, 32, c_i))
dense = nnx.Conv(c_i, c_o, kernel_size=(k, k), padding='SAME',
                 use_bias=False, rngs=nnx.Rngs(d2l.get_key()))
depthwise = nnx.Conv(c_i, c_i, kernel_size=(k, k), padding='SAME',
                     feature_group_count=c_i, use_bias=False,
                     rngs=nnx.Rngs(d2l.get_key()))
pointwise = nnx.Conv(c_i, c_o, kernel_size=(1, 1), use_bias=False,
                     rngs=nnx.Rngs(d2l.get_key()))
Y = depthwise(X)
assert dense(X).shape == pointwise(Y).shape
size = lambda model: sum(p.size for p in jax.tree_util.tree_leaves(
    nnx.state(model, nnx.Param)))
p_dense = size(dense)
p_sep = size(depthwise) + size(pointwise)
p_dense, p_sep, p_dense / p_sep
(147456, 17536, 8.408759124087592)

Cost of channel depth

A h \times w image with k \times k kernel and c_i \to c_o channels takes

\mathcal{O}(h \cdot w \cdot k^2 \cdot c_i \cdot c_o)

operations. For a 256×256 image, 5×5 kernel, and 128→128 channels, counting multiplications and additions separately gives more than 53 billion operations for one layer.

Consequently:

  • Convolutions benefit from GPU and TPU acceleration.
  • Efficient architectures use 1×1 bottlenecks, depthwise-separable convolutions, and grouped convolutions to reduce channel costs.

Recap

  • Multi-input-channel: per-channel kernel, results summed across input channels.
  • Multi-output-channel: stack independent filter banks; one output channel per filter set.
  • Total weights: c_o c_i k_h k_w + c_o.
  • 1\times 1 convolutions are per-pixel fully connected layers across channels and are widely used in bottlenecks and pointwise convolutions.
  • Compute scales linearly with c_i \cdot c_o — the dominant cost in deep CNNs.