DeltaNet: Memory That Edits

Dive into Deep Learning · §12.5

DeltaNet
memory that edits: the overwrite failure · one SGD step per token · the WY trick · Gated DeltaNet · what a transition can compute

The write column, revisited

Last section’s family table: seven models, three lineages — and every write is an addition.

  • Decay manages crowding; it discounts by age.
  • But streams re-bind: the same ticker, a reassigned variable, a character who moves. Staleness is indexed by key.

After R writes to one key: \mathbf{S}^\top \mathbf{k} = \mathbf{v}^{(1)} + \cdots + \mathbf{v}^{(R)} — the current binding enjoys no privilege over the stale ones.

Overwriting, measured

8 orthonormal keys (zero cross-key interference by construction), R shuffled re-bindings per key, read back the latest:

writes/key   Hebbian   delta
         1     1.000   1.000
         2     0.510   1.000
         4     0.299   1.000
         8     0.217   1.000
  • Hebbian: recall \approx guessing among the R superposed values.
  • Erase-then-write: 1.000 at every R — the last write always wins.

Trained models hit the same ceiling

Same task, end-to-end training, identical parameter counts; only the write line differs:

writes/key   Hebbian   delta
         1     1.000   1.000
         2     0.537   1.000
         4     0.327   1.000
         6     0.268   1.000

Design constraint (learned the hard way): the write address (q, k, \beta) must come from the key token alone — give it the value token and a Hebbian model escapes by assigning every pair its own address.

The delta rule

Ask the state to be correct, not just aligned — one SGD step on the recall loss, inside the forward pass:

\ell_t(\mathbf{S}) = \tfrac{1}{2}\|\mathbf{S}^\top \mathbf{k}_t - \mathbf{v}_t\|^2, \qquad \mathbf{S}_t = \mathbf{S}_{t-1} - \beta_t\, \mathbf{k}_t (\mathbf{k}_t^\top \mathbf{S}_{t-1} - \mathbf{v}_t^\top)

  • Widrow–Hoff 1960 → fast weight programmers (Schlag et al. 2021) → parallel DeltaNet (Yang et al. 2024).
  • Hebbian write = SGD on -\langle \mathbf{S}^\top\mathbf{k}_t, \mathbf{v}_t\rangle: alignment, no feedback — that is the whole difference.

Unit key: stored value moves to (1-\beta_t)\mathbf{v}_{\textrm{old}} + \beta_t \mathbf{v}_t — \beta is a learned write strength (a per-token learning rate).

One rule, two readings

def delta_recurrent(Q, K, V, beta):
    """The delta rule in the error form: read, subtract, write, read out."""
    S = Q.new_zeros(Q.shape[-1], V.shape[-1])
    outputs = []
    for t in range(Q.shape[0]):
        error = V[t] - S.T @ K[t]                # What the memory got wrong
        S = S + beta[t] * K[t][:, None] * error[None, :]
        outputs.append(S.T @ Q[t])
    return torch.stack(outputs)

def delta_recurrent_matrix(Q, K, V, beta):
    """The same update as transition-then-write, for the family template."""
    S = Q.new_zeros(Q.shape[-1], V.shape[-1])
    I = torch.eye(Q.shape[-1], device=Q.device)
    outputs = []
    for t in range(Q.shape[0]):
        S = (I - beta[t] * torch.outer(K[t], K[t])) @ S \
            + beta[t] * torch.outer(K[t], V[t])
        outputs.append(S.T @ Q[t])
    return torch.stack(outputs)

torch.manual_seed(0)
d_k, d_v, T = 64, 64, 512
Q, K, V = (torch.randn(T, d, device=d2l.try_gpu()) for d in (d_k, d_k, d_v))
K = F.normalize(K, dim=-1)                       # Unit-norm keys, throughout
beta = torch.sigmoid(torch.randn(T, device=d2l.try_gpu()))
y_delta = delta_recurrent(Q, K, V, beta)
err = (y_delta - delta_recurrent_matrix(Q, K, V, beta)).abs().max() \
    / y_delta.abs().max()
print(f'error form vs matrix form: relative deviation {float(err):.2e}')
assert err < 1e-5                  # One rule, two readings
error form vs matrix form: relative deviation 1.88e-06
  • Error form: two matrix–vector products per token — the algorithm.
  • Matrix form: transition \mathbf{I} - \beta_t \mathbf{k}_t\mathbf{k}_t^\top + write — the family template.

Training it: the WY trick

Transitions are no longer diagonal — no segsum. But within a chunk the updates couple only through key overlaps:

(\mathbf{I} + \mathbf{A})\,\mathbf{U} = \mathrm{diag}(\boldsymbol{\beta})(\mathbf{V} - \mathbf{K}\mathbf{S}), \qquad A_{ts} = \beta_t\,\mathbf{k}_t^\top \mathbf{k}_s,\; s < t

One unit-triangular solve per chunk, against the actual right-hand side — \mathcal{O}(C^2 d_v), never form the inverse; then attention inside, recurrence across, as before.

chunk size   16: relative deviation 3.00e-07
chunk size   64: relative deviation 4.03e-07
chunk size  256: relative deviation 3.63e-07

Production kernels: the flash-linear-attention library — same algorithm.

Gating: the modern cell

DeltaNet preserves every unwritten row exactly — stale rows never fade. Add back a decay, decoupled from the write:

\mathbf{S}_t = \alpha_t\big(\mathbf{I} - \beta_t \mathbf{k}_t\mathbf{k}_t^\top\big)\mathbf{S}_{t-1} + \beta_t\,\mathbf{k}_t\mathbf{v}_t^\top

  • \beta_t = learning rate, \alpha_t = weight decay, applied decoupled — the gate plays AdamW’s decoupled-decay role (that one axis; no moments, no bias corrections).
  • \beta_t = 0: Mamba-2’s row. \alpha_t = 1: pure DeltaNet.
  • Shipped: Qwen3-Next (Gated DeltaNet), Kimi Linear (KDA), RWKV-7’s line.

On the scoreboard

Same capstone recipe as the Mamba row — The Time Machine, 50k windows, ten epochs:

          model  val ppl    params  s/epoch
 Gated DeltaNet     83.7   562,315      5.4
  • Lands in the strongest band of the chapter’s scoreboard at comparable parameter count.
  • At 1.3B scale: Gated DeltaNet beats Mamba-2 and DeltaNet — decay without delta trails on recall; delta without decay trails on modeling.

The generalized delta rule

RWKV-7 decouples every role the delta rule ties together:

\mathbf{S}_t = \big(\mathrm{diag}(\mathbf{w}_t) - (\mathbf{a}_t \odot \hat{\boldsymbol{\kappa}}_t)\,\hat{\boldsymbol{\kappa}}_t^\top\big)\,\mathbf{S}_{t-1} + \tilde{\mathbf{k}}_t\mathbf{v}_t^\top

  • Per-channel decay \mathbf{w}_t · removal key \hat{\boldsymbol{\kappa}}_t ≠ write key \tilde{\mathbf{k}}_t · per-channel write rate \mathbf{a}_t.
  • Tie them all → DeltaNet exactly (asserted in the notebook).

The family now has two parents: decay structure × write rule.

What the transition can compute: parity

Purely multiplicative recurrence \mathbf{h}_t = \mathbf{a}_t \odot \mathbf{h}_{t-1}; the only difference is the range of \mathbf{a}_t:

T= 8  sigmoid, a in (0,1): 0.372  0.375  0.650
T= 8  tanh,    a in (-1,1): 1.000  1.000  1.000
T=16  sigmoid, a in (0,1): 0.486  0.497  0.497
T=16  tanh,    a in (-1,1): 1.000  1.000  1.000
T=24  sigmoid, a in (0,1): 0.517  0.517  0.483
T=24  tanh,    a in (-1,1): 0.517  1.000  1.000
  • (0,1): chance in every run — a representational wall: the dynamics cannot flip a sign.
  • (-1,1): every seed solves it at T=8; longer T = an optimization horizon, failing seeds from T=16 on.

The reflection hiding in the delta rule

\mathbf{I} - \beta\,\mathbf{k}\mathbf{k}^\top has eigenvalue 1 - \beta along \mathbf{k}:

beta = 0.5: eigenvalues +0.5 (along k), +1.0 (elsewhere)
beta = 1.0: eigenvalues -0.0 (along k), +1.0 (elsewhere)
beta = 1.5: eigenvalues -0.5 (along k), +1.0 (elsewhere)
beta = 2.0: eigenvalues -1.0 (along k), +1.0 (elsewhere)
T =  8, beta = 1 (projection): parity accuracy 0.529
T =  8, beta = 2 (reflection): parity accuracy 1.000
T = 24, beta = 1 (projection): parity accuracy 0.518
T = 24, beta = 2 (reflection): parity accuracy 1.000
T = 64, beta = 1 (projection): parity accuracy 0.501
T = 64, beta = 2 (reflection): parity accuracy 1.000
  • \beta = 1: projection — erases, parity lost. \beta = 2: reflection — exact at T = 64, nothing optimized.
  • That one eigenvalue is the flag allow_neg_eigval: double the gate to 2\sigma(\cdot) \in (0, 2) (Grazzi et al. 2025) — open interval, so \beta = 2 itself is the boundary construction.

The ladder, and its ceiling

transition unlocks
scalar / diagonal (0,1) forgetting; parity out of reach
diagonal (-1,1) parity
Householder, \beta \in (0,2) targeted edits + reflections
product of n_h Householders group word problems (DeltaProduct)
dense nonlinear S_5 and beyond — at sequential cost
  • The \mathsf{TC}^0 ceiling (Merrill et al. 2024): diagonal SSMs and transformers share it — an “illusion of state.”

Still a fixed-size state; still short of a true recurrence. Both gaps → hybrids (two sections ahead). First: what if a layer is an optimizer, full stop — next section.