Training the airfoil model

Adadelta

Adadelta (Zeiler, 2012) takes RMSProp further: adapt per-parameter step magnitudes and remove the global learning rate entirely.

Keeps two EMAs — one over squared gradients, one over squared updates. The ratio of their square roots is dimensionally consistent — a “unitless” step size, so no separate \eta needed (in principle).

The update rule

\mathbf{s}_t = \rho \mathbf{s}_{t-1} + (1-\rho) \mathbf{g}_t^2,

\mathbf{u}_t = \rho \mathbf{u}_{t-1} + (1-\rho)(\Delta\mathbf{x}_t)^2,

where \mathbf{s}_t tracks squared gradients and \mathbf{u}_t tracks squared parameter updates. The actual step uses the previous update scale:

\Delta\mathbf{x}_t = -\frac{\sqrt{\mathbf{u}_{t-1}+\epsilon}} {\sqrt{\mathbf{s}_t+\epsilon}} \odot \mathbf{g}_t,

\mathbf{x}_t \leftarrow \mathbf{x}_{t-1} + \Delta\mathbf{x}_t.

In practice frameworks still expose a learning-rate hyper for fine-tuning.

From-scratch implementation

Two state buffers per parameter (s and delta):

%matplotlib inline
from d2l import tensorflow as d2l
import tensorflow as tf

def init_adadelta_states(feature_dim):
    s_w = tf.Variable(d2l.zeros((feature_dim, 1)))
    s_b = tf.Variable(d2l.zeros(1))
    delta_w = tf.Variable(d2l.zeros((feature_dim, 1)))
    delta_b = tf.Variable(d2l.zeros(1))
    return ((s_w, delta_w), (s_b, delta_b))

def adadelta(params, grads, states, hyperparams):
    rho, eps = hyperparams['rho'], 1e-5
    for p, (s, delta), grad in zip(params, states, grads):
        s[:].assign(rho * s + (1 - rho) * tf.math.square(grad))
        g = (tf.math.sqrt(delta + eps) / tf.math.sqrt(s + eps)) * grad
        p[:].assign(p - g)
        delta[:].assign(rho * delta + (1 - rho) * g * g)

data_iter, feature_dim = d2l.get_data_ch11(batch_size=10)
d2l.train_ch11(adadelta, init_adadelta_states(feature_dim),
               {'rho': 0.9}, data_iter, feature_dim);

loss: 0.243, 1.655 sec/epoch

The loss curve should be read as a scale-adaptation demo: Adadelta trains without the hand-picked global learning rate that earlier optimizers needed.

Concise: framework Adadelta

# adadelta is not converging at default learning rate
# but it is converging at lr = 5.0
trainer = tf.keras.optimizers.Adadelta
d2l.train_concise_ch11(trainer, {'learning_rate':5.0, 'rho': 0.9}, data_iter)

loss: 0.245, 1.653 sec/epoch

Recap

Two EMAs: squared gradients \mathbf{s}_t and squared updates \mathbf{u}_t.
Per-parameter step is the ratio \sqrt{\mathbf{u}_{t-1}}/\sqrt{\mathbf{s}_t}, dimensionally consistent — drops the explicit learning rate.
Less popular today than Adam, but a good case study in scale-invariant optimization design.