25  Preference Optimization and Reinforcement Learning

The reader derived GRPO and DPO on a toy verifier task in Advanced and saw the KL-regularized optimum in closed form; on a real language model the reward is a second network that can be wrong, sampling takes most of every step, four copies of the model compete for one GPU, and the policy learns whatever the reward left unsaid. This chapter runs the methods the reader already knows on Qwen3-0.6B and measures the symptoms that decide which variant is usable.