15  Deep Reinforcement Learning

Chapter 14 introduced tabular value methods, policy gradients, and neural function approximation. The resulting REINFORCE agent waits for complete episodes, uses each on-policy batch once, and places no explicit constraint on the change in its policy. Deep reinforcement learning addresses these limitations while also introducing new sources of instability: bootstrapped targets depend on learned predictions, shared parameters couple updates across states, and reused data may have been collected by an older policy.

The chapter develops these issues through seven examples. Actor–critic methods replace complete returns with bootstrapped value estimates and lead to \(n\)-step returns and generalized advantage estimation. Trust regions and PPO control changes in the policy while permitting limited reuse of on-policy data. Regularized policy optimization studies learned rewards and KL penalties. DQN combines Q-learning with replay and target networks; SAC extends the same off-policy machinery to continuous actions. Offline reinforcement learning considers a fixed dataset with no further interaction. The final section formulates language-model generation as a decision process and identifies which preceding methods remain necessary in that special case.

Figure 14.2 locates these methods according to what they learn and which data they use. Because the implementations reuse objects introduced in Chapter 14, Table 15.1 lists those objects and the additions made in this chapter.

Table 15.1: The objects this chapter’s code uses without rebuilding, and where each was built. The rows below the marked line join the library in this chapter.
object what it does built in
TabularMDP the model as arrays: kernel, reward, discount, and one Bellman backup sweep Section 14.1
value_iteration, policy_evaluation the exact sweeps to \(V^*\) and to \(V^\pi\), this chapter’s yardsticks Section 14.2
evaluate the mean return of a policy over fresh evaluation episodes Section 14.2
ActorCritic, .tabular the policy-plus-value container with its optimizers, and its table form Section 14.3
ActorCritic.mlp the same container with one-hidden-layer network heads Section 14.7
policy_step one ascent step on advantage-weighted log-probabilities Section 14.3
fit_value regression passes of \(\hat{V}\) toward a supplied target Section 14.7
Batch, rollout the trajectory container and the collection loop that fills it Section 14.5
Batch.backward_scan, Batch.reward_to_go the discounted backward scan, and the reward-to-go it computes Section 14.6
normalize per-batch standardization of the policy weight Section 14.6
run_seeds one training generator run across seeds into a single array Section 14.6
epsilon_greedy, linear_schedule the exploration rule and its annealing Section 14.4
plot_curves, show_grid seed-band learning curves; gridworld values and policies as panels Section 30.9
added in this chapter
Batch.td_target the one-step bootstrapped target \(r + \gamma (1 - \textrm{terminated})\, \hat{V}(s')\) Section 15.1
Batch.gae generalized advantage estimation: the backward scan run on TD errors Section 15.1
ppo_epochs clipped-surrogate reuse epochs on a frozen batch, diagnostics returned as data Section 15.2
ReplayBuffer a ring of transitions whose sample scrambles time into a Batch Section 15.4
offline_q Q-learning swept over a fixed dataset, with optional pessimism \(\kappa/\sqrt{n}\) Section 15.6

Training loss alone is often insufficient to diagnose a reinforcement-learning run. A value estimate can diverge while its regression loss decreases, and a poor policy can continue to generate apparently regular curves. Table 15.2 collects the additional measurements used in this chapter and indicates how they should be interpreted.

Table 15.2: Diagnostics for the reinforcement-learning experiments in this chapter. The reported ranges are task-specific; the qualitative patterns are more general.
diagnostic healthy reading measured in
the ratio \(\rho_t\) on the first reuse epoch exactly \(1\), by construction; anything else is a bug in the frozen log-probabilities Section 15.2
approximate KL within a batch small and front-loaded: it increases for a few epochs, then flattens; continued growth indicates excessive policy change Section 15.2
fraction of ratios outside the clip band about one check in twenty; several times that means the reuse or the step size is too aggressive Section 15.2
policy entropy a slow decrease, about \(0.65\) to \(0.25\) nats over a CartPole run; a rapid decrease toward zero indicates policy saturation Section 15.2
weight-only effective sample size of the reused batch ratio concentration: weights nearly flat through the reuse epochs; concentration to half or less says stop reusing; blind to advantages, dependence, and state staleness Section 15.2
correlation of the TD error with the Monte Carlo advantage about \(0.3\) early, falling toward zero as episodes lengthen, consistent with the Monte Carlo weight dissolving into tail noise; a descriptive diagnostic, not a critic certificate Section 15.1
pre-clip gradient norm, and how often the clip binds stable norms, the clip binding on a bounded fraction of updates: about a third for the bootstrapped weight and none for Monte Carlo, at matched normalization; descriptive of where variance lives, not proof of signal Section 15.1
streak lengths at the ceiling long runs of consecutive perfect batches, a run-specific stability visualization; arriving without resting points at the estimator’s noise, not the policy Section 15.1
value estimate at the start state below the ceiling of the objective the update defines, \(1/(1-\gamma) = 100\) on CartPole’s continuing formulation, by tens of points at most; above the line is overestimation, growing without bound is divergence Section 15.4
best against final trailing window, across seeds descriptive statistics of the curve only: the final window depends on when training stops, while the retrospective best window is affected by optimistic selection; report a predeclared fixed-budget evaluation instead Section 15.4, and the boxed reading rule of Section 14.6
the entropy trace of a stochastic continuous policy descends from its wide start and ends within about a nat of the autotuning target \(-\dim \mathcal{A}\); an entropy that rises without bound, or disagrees with a one-line quadrature check of the density, indicates an incorrect squashing-density calculation Section 15.5
predicted value against evaluated return predictions should not exceed the maximum achievable return; a prediction above a computable optimum establishes overestimation without requiring a baseline Section 15.6, and the twin-against-single calibration of Section 15.5

Most experiments use CartPole and several random seeds. Section 15.3 and Section 15.6 use tabular gridworlds because their optimal values can be computed exactly, and Section 15.7 uses deterministic string concatenation rather than a simulator. The chapter does not cover multi-agent, meta-, hierarchical, goal-conditioned, partially observable, or model-based reinforcement learning. Large-scale RLHF, DPO, and additional GRPO variants are treated in the Language Models part.

Resources and Further Reading

The resources in Chapter 14 cover the general theory and courses. The following references focus on implementations and empirical evaluation.