Parameter Management

Managing Parameters

A neural network is a tree of parameters — the weight matrices and bias vectors gradient descent updates. Training is one thing you do with them; this deck covers the others.

  • Inspection — debug, sanity-check init, view features.
  • Iteration — optimizers, weight decay, checkpointing all walk every parameter.
  • Sharing (“tying”) — make two layers refer to the same tensor (tied embeddings, autoencoders).

The parameter tree

A nested module is just a tree. Each module is a node; each parameter is a leaf:

net  (Sequential)
├─ 0: Linear      ├─ weight  (8, 4)
│                 └─ bias    (8,)
├─ 1: ReLU         (no params)
└─ 2: Linear      ├─ weight  (1, 8)
                  └─ bias    (1,)

Two access patterns:

  • By path: net[2].weight — direct.
  • By traversal: walk the tree, yield every leaf.

Frameworks give you both, plus serialization built on the same traversal.

A toy model

from mxnet import init, np, npx
from mxnet.gluon import nn
npx.set_np()
net = nn.Sequential()
net.add(nn.Dense(8, activation='relu'))
net.add(nn.Dense(1))
net.initialize()  # Use the default initialization method

X = np.random.uniform(size=(2, 4))
net(X).shape

Direct access

Index into a Sequential like a list; each layer exposes its parameters as attributes:

net[1].params

Two parameters per Linear layer — weight matrix and bias vector. The output object is a Parameter (PyTorch) or similar wrapper that carries the tensor + gradient + extra metadata.

Tensor inside the parameter

.data (PyTorch) unwraps the parameter to a plain tensor for inspection:

type(net[1].bias), net[1].bias.data()

.grad is the gradient buffer — populated by backward(), otherwise None. Useful for custom optimizers or diagnosing dead neurons:

net[1].weight.grad()

Recursive traversal

For everything-at-once, use named_parameters(). It walks the whole tree and yields (name, param) pairs at the leaves — names use dotted paths through the nesting:

net.collect_params()

This is the iterator optim.SGD(net.parameters(), …) consumes. It’s also what gets pickled when you save a checkpoint with state_dict(). Walk-tree-once, use many ways.

Parameter tying

Reuse the same module instance at multiple positions in your architecture, and the framework treats them as one parameter set — same memory, gradients accumulate across uses.

Common cases:

  • Tied embeddings: input embedding and output softmax projection in a language model share weights — saves |V| \cdot d parameters.
  • Autoencoders: decoder uses transposed encoder weights.
  • Recurrent layers: same kernel applied at every time step (the original tying mechanism).
net = nn.Sequential()
# We need to give the shared layer a name so that we can refer to its
# parameters
shared = nn.Dense(8, activation='relu')
shared_clone = nn.Dense(8, activation='relu')
shared_clone.share_parameters(shared.collect_params())
net.add(nn.Dense(8, activation='relu'),
        shared,
        shared_clone,
        nn.Dense(10))
net.initialize()

X = np.random.uniform(size=(2, 20))

net(X)
# Check whether the parameters are the same
print(net[1].weight.data()[0] == net[2].weight.data()[0])
net[1].weight.data()[0, 0] = 100
# Make sure that they are actually the same object rather than just having the
# same value
print(net[1].weight.data()[0] == net[2].weight.data()[0])

Modify net[2].weight and net[4].weight reflects the same change — they are the same tensor, not just equal.

Recap

  • A module is a tree; parameters live at the leaves.
  • Direct access: net[i].weight, .bias, .grad.
  • Recursive traversal: named_parameters() / state_dict() walks the whole tree.
  • Same iterator powers optimizers, weight decay, checkpointing.
  • Tied parameters = reuse the same module instance — gradients accumulate; one buffer in memory.