class ReplayBuffer:
"""A ring of transitions in preallocated numpy; sample() returns a Batch."""
def __init__(self, capacity, obs_dim):
self.obs = np.zeros((capacity, obs_dim), np.float32)
self.act = np.zeros(capacity, np.int64)
self.rew = np.zeros(capacity, np.float32)
self.next_obs = np.zeros((capacity, obs_dim), np.float32)
self.term = np.zeros(capacity, np.float32)
self.capacity, self.size, self.ptr = capacity, 0, 0
def add(self, obs, act, rew, next_obs, term):
i = self.ptr
self.obs[i], self.act[i], self.rew[i] = obs, act, rew
self.next_obs[i], self.term[i] = next_obs, term
self.ptr, self.size = (i + 1) % self.capacity, min(self.size + 1,
self.capacity)
def __len__(self):
return self.size
def sample(self, batch_size, rng):
i = rng.integers(self.size, size=batch_size)
return d2l.Batch(self.obs[i], self.act[i], self.rew[i],
self.next_obs[i], self.term[i],
np.array([batch_size]))The Deadly Triad
Function approximation + bootstrapping + off-policy data :cite:Sutton.Barto.2018,vanHasselt.Doron.Strub.ea.2018. Every region is an algorithm already taught:
Baird’s Counterexample
Baird’s construction has seven states and zero true value everywhere, which the parameterization can represent with w=0. Even deterministic expected updates diverge under uniform off-policy weighting:
sup norm of w after 0, 500, 1000 sweeps: 10, 77, 335
the value the weights claim for state 7: 677, true value 0The parameters diverge exponentially for every positive constant step size. The failure belongs to the expected update operator, not to sampling noise :cite:Baird.1995.
Replay and Target Networks
Target-Network Ablation
With a target network, every seed reaches returns in the hundreds, although performance remains variable. Without it, every seed in both implementations falls to near-minimal return and its value estimates exceed 10^8. In this ablation, removing the target network therefore causes clear divergence.
Converging Values, Churning Policy
continuing-task ceiling: 100; the no-target arm ends at 1e+08, 1e+08, 1e+08At the probe state, the value estimate approaches the correct scale rather than diverging; a single probe, however, cannot certify the whole value function. Small changes in action values can still change the greedy policy, and fixed-budget evaluations range from 90 to 500 across seeds and implementations. Because the update bootstraps through the time limit, it optimizes a continuing objective whose start-state value is bounded by 1/(1-\gamma)=100. Estimates above this bound are necessarily overestimates.
Maximization Bias
single estimator, E[max of the estimates]: 1.031
select with one, evaluate with the other: -0.006In this example, applying \max to noisy estimates introduces approximately one unit of positive bias :cite:Thrun.Schwartz.1993. Using independent estimates for selection and evaluation removes the bias :cite:vanHasselt.2010. The hard maximum is the \beta\to0 limit of the soft backup in :numref:sec_regularized.
The Double DQN Target
Select with the online network, evaluate with the frozen one :cite:Hasselt.Guez.Silver.2016:
def q_step_double(qnet, target, opt, batch):
"""q_step with selection split from evaluation, eq_double_dqn."""
sel = q_values(qnet, batch.next_obs).argmax(-1)
fit_q(qnet, opt, batch, batch.td_target(
lambda s: q_values(target, s)[np.arange(len(s)), sel], gamma))
DQN: final value estimate at s0, per seed: [93.8 98.3 93. ]
Double DQN: final value estimate at s0, per seed: [92.6 95.1 91. ]Most seeds finish with values a few points lower under Double DQN. This small difference is consistent with the two-action experiment in panel (b), where maximization has limited scope to select extreme errors. On Atari, with as many as 18 actions, the same modification produces substantially larger effects.