Temporal-Difference Learning and Exploration

Dive into Deep Learning · §14.4

Temporal differences, Q-learning and exploration
sampled backups · temporal-difference error · convergence · exploration

The Sampled Backup

Value iteration needs P inside one expectation. Replace it with the transition you just observed:

\delta = r + \gamma \max_{a'} Q(s', a') - Q(s, a), \qquad Q(s, a) \leftarrow Q(s, a) + \alpha\, \delta

  • \delta is the temporal-difference error: the sampled one-step target minus the current table estimate.
  • Bootstrap masked by terminated, never truncated.

:numref:fig_rl_backups, panel (d): one blue branch instead of the sum, the max at the next state kept.

What Actually Converges

The sampled least-squares objective is not the right justification:

L(Q) = E_\mu \big[ (Q - TQ)^2 \big] + \gamma^2\, E_\mu \big[ \mathrm{Var}_{s'} ( \max_{a'} Q(s', a') ) \big]

Q^* makes the first term zero, but the variance term can change the minimizer (double sampling). It vanishes for deterministic transitions.

The expected update satisfies E[\delta \mid s, a] = (TQ)(s, a) - Q(s, a), zero exactly at Q^*. A stochastic approximation of value iteration, convergent under Robbins-Monro steps.

The Update in Code

def epsilon_greedy(q, epsilon, rng):
    """Explore with probability epsilon, else act greedily on the values q."""
    if rng.random() < epsilon:
        return int(rng.integers(len(q)))
    # Random tie-breaking is load-bearing: np.argmax would always return
    # action 0 on a zero-initialized table, and an agent that only ever
    # proposes *left* on this lake never finds the goal.
    return int(rng.choice(np.flatnonzero(q == q.max())))
def q_learning(seed, Q, visits, env, num_episodes,
               alpha=lambda n: 1 / (1 + 0.1 * n)):
    """Tabular Q-learning; updates Q in place, yields each episode's return."""
    rng = np.random.default_rng(seed)
    epsilon = linear_schedule(1.0, 0.05, num_episodes // 2)
    env.reset(seed=seed)
    for episode in range(num_episodes):
        s, done, ret = env.reset()[0], False, 0.0
        while not done:
            a = epsilon_greedy(Q[s], epsilon(episode), rng)
            s_next, r, terminated, truncated, _ = env.step(a)
            visits[s, a] += 1
            delta = r + gamma * (1 - terminated) * Q[s_next].max() - Q[s, a]
            Q[s, a] += alpha(visits[s, a]) * delta
            s, done, ret = s_next, terminated or truncated, ret + r
        yield ret

Graded Against Dynamic Programming

This finite example retains the exact MDP for evaluation.

max_s |V_Qhat(s) - V*(s)| per seed: [0.019 0.012 0.021 0.008 0.006]
success rate: learned greedy 71.2% to 74.1% over 5 seeds; pi* 73.6%
pi* forced to explore at epsilon = 0.05: 54.1%
median environment steps: 95569

The table is within hundredths of V^*, and the greedy policy agrees with \pi^* within sampling error. With \epsilon=0.05, continuing exploration reduces the measured success rate of the behavior policy to 54\%.

Step Sizes and Finite Budgets

Q*(s0, <) = 0.180
alpha = 0.9 (constant): final estimates [0.056 0.221 0.206 0.265 0.236]
alpha =  1/(1 + 0.1 n): final estimates [0.19  0.186 0.195 0.184 0.18 ]
alpha =      1/(1 + n): final estimates [0.003 0.024 0.008 0.009 0.002]
  • constant 0.9: estimates continue to fluctuate (0.06 to 0.27)
  • 1/(1 + 0.1 n): converged, leaning slightly high
  • 1/(1 + n): satisfies Robbins–Monro but leaves all five seeds below 0.025 at this budget

Exploration Measured by Regret

A bandit is an MDP with one state. Regret assigns each pull the mean-reward gap from the best arm.

Greedy 824 · fixed \epsilon 117 (linear regret) · annealed 69 · UCB 37 · Thompson 32.

Optimism and Uncertainty

a_t = \mathrm{argmax}_a \big[ \hat{\mu}(a) + \kappa \sqrt{\log t / n(a)} \big]

Per-arm exploration decreases as observations accumulate: logarithmic regret under the stated UCB conditions, whereas fixed \epsilon has linear regret (proved at \kappa = \sqrt 2; play each arm once first). Thompson: sample a Beta posterior, play the argmax.

Online and offline uncertainty. Online exploration adds a confidence radius that decreases with counts; offline pessimism (:numref:sec_offline) subtracts an uncertainty penalty because no new samples can correct errors.

Which Policy Is Being Learned

  • The \max_{a'} ignores the action the behavior took: off-policy. Learn about the greedy policy from data collected by any policy.
  • One symbol away: SARSA bootstraps on the action taken, learning the behavior’s value, \epsilon floor and all.
  • The \max also produces positive bias here: four of five final estimates exceed the true 0.180, and none are below it. Maximization bias, repaired in :numref:sec_dqn.

Recap

  • TD error :eqref:eq_td_error: the one-step residual; reused by every algorithm ahead.
  • Convergence follows from the expected update’s fixed point, not from minimizing the sampled squared objective (double sampling).
  • Robbins-Monro is necessary for the guarantee; budgets decide between schedules that both pass.
  • Online feedback: selecting an overestimated action provides data that can correct its estimate; this mechanism is absent offline.
  • Fixed \epsilon has linear regret; UCB and Thompson sampling reduce exploration as uncertainty decreases. The uncertainty term changes sign in :numref:sec_offline.
  • The first \max defines a greedy off-policy target; the same \max can also introduce maximization bias.