Reinforcement Learning for Sequence Generation

Dive into Deep Learning · §15.7

Sequences are trajectories
text generation as an MDP · response factorization · terminal rewards · group baselines and KL penalties

The Sequence-Generation MDP

language modeling reinforcement learning
prompt x start state s_0 \sim \mu_0
token y_t action a_t
prefix (x, y_{<t}) state s_t
response y trajectory \tau
EOS terminal state
next-token softmax the policy \pi_\theta
generate rollout

Transitions are concatenation: deterministic, known, probability one. All randomness comes from policy sampling; the reward is terminal.

Sequence and Token Gradients

\nabla_\theta \log \pi_\theta(y \mid x) = \sum_{t=1}^{T} \nabla_\theta \log \pi_\theta(y_t \mid x, y_{<t})

  • chain rule, no transition factors to drop: three-line proof
  • terminal reward \Rightarrow one response-level weight r(x, y) - b(x) on every token’s score (rather than the prefix-dependent A(s_t, a_t))
  • one structured action per episode: the contextual bandit of :numref:sec_qlearning; multi-turn interaction restores sequential dependence

Simplifications for Sequence Generation

Deterministic concatenation and terminal reward remove the need for transition learning and intermediate Bellman targets; policy optimization components remain applicable.

The Group-Mean Baseline

Sample K responses per prompt, standardize within the group, one step on the log-probs. Same-group centering is biased: its expectation is (K-1)/K of the gradient (leave-one-out is exact). Prediction: at K = 1 the shrinkage reaches zero and the update vanishes identically.

K =  1: success of a sampled response 0.062
K =  2: success of a sampled response 0.500
K =  4: success of a sampled response 0.500
K =  8: success of a sampled response 0.500
K = 32: success of a sampled response 0.500

0.062 is the unchanged reference score. At K=1, including the sample in its own baseline makes the reward-gradient update identically zero.

Reward-Model Exploitation

A grader that searches for the answer accepts a response listing every candidate. :eqref:eq_kl_optimum predicts that this response is favored when $< $ reward gap / reference log-odds = 1/4.

beta = 0.0: sloppy 1.00, gold 0.50, hedge 0.50
beta = 0.1: sloppy 0.99, gold 0.50, hedge 0.49
beta = 0.2: sloppy 0.60, gold 0.48, hedge 0.12
beta = 0.3: sloppy 0.42, gold 0.39, hedge 0.04
beta = 0.5: sloppy 0.25, gold 0.24, hedge 0.01

At \beta=0, the approximate score is perfect while the exact score is 0.5. Above the predicted threshold, the multi-answer response is suppressed. The penalty limits rather than prevents exploitation.

Components of GRPO

  • group-standardized advantage: :numref:sec_baselines’s :eqref:eq_pg_normalized, group mean as baseline (biased by self-inclusion; LOO is the exact variant), no value network
  • a few epochs per group: token-level clipped ratios :eqref:eq_ppo_clip
  • \beta\, D_{\textrm{KL}} to the frozen reference as a separate estimator: :eqref:eq_kl_objective
  • both KLs at once: clip against the previous iterate, penalty against the frozen reference
  • the example uses one step per group, no clipping, a KL term folded into the reward, and one-token responses

Read :eqref:eq_kl_optimum backwards and preferences fit the policy directly: DPO :cite:Rafailov.Sharma.Mitchell.ea.2023.

Notation for Later Chapters

Reward is learned (Bradley-Terry, :numref:sec_regularized) or checked (a verifier: RLVR :cite:DeepSeekAI.2025). Both can omit aspects of the intended objective, as do \hat{Q} in :numref:sec_offline, r_\phi in :numref:sec_regularized, and the approximate grader here.

:numref:tab_rl_notation_contract: x, y, \tau, \mu_0, \pi_{\textrm{ref}}, \hat{G}_t, A, K, \rho_t, \epsilon, \beta, \delta_t, w: inherited verbatim by the Language Models part.

Where To Go Next

  • search: MCTS is a policy-improvement operator, distilled back into the network; AlphaZero \to MuZero \to world models; the same pattern as test-time reasoning
  • continuous control: same objectives, gains from normalization and scale
  • omitted with pointers: multi-agent, distributional, meta, hierarchical, POMDPs
  • next read: the RLHF book :cite:Lambert.2026

Recap

  • prompt = start state, token = action, prefix = state, response = trajectory; transitions are concatenation
  • one gradient, two views; terminal reward puts one response-level weight r - b(x) on every token
  • at K=1, the reward-gradient update is zero because of self-inclusion; the verifier example matches the predicted threshold $= $ reward gap / reference log-odds
  • policy-gradient estimation, clipping, and KL regularization carry directly to the Language Models part