PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning from Human Feedback

Nathan Lambert

arXiv:2504.12501v11cs.LG

TL;DR

Post-training must reshape next-token predictors into conversational question-answering systems while drawing out pretrained knowledge. This book explains RLHF’s proxy-reward objective and canonical recipes; it argues that most user-visible gains come from post-training, which can elicit more latent potential than narrow instruction tuning alone.

  • Problem

    Post-training must reshape models from next-token prediction to conversational question-answering while extracting knowledge and intelligence from pretraining.

    Source: §1.4 · p. 13

  • Method

    The book explains RLHF’s core objective of optimizing a proxy reward for human preferences with a distance-based regularizer and presents canonical recipes.

    Source: §3 · p. 21

  • Results

    The book argues that most user-visible gains come from post-training, which can elicit more latent potential than narrow instruction tuning alone.

    Source: §1.4 · p. 13

  • Takeaways & Limitations

    The canonical recipes serve as reference points for comparing RLHF optimization choices and the training steps used in key models.

    Source: §3 · p. 21

Abstract

from arXiv · show

Reinforcement learning from human feedback (RLHF) has become a crucial tool to build the latest machine learning systems at scale. The field grew around the core methods of RLHF into today's broader suite of post-training techniques. In this book, we give a comprehensive introduction to the core methods for post-training models for people with some level of quantitative background, organized around the canonical RLHF recipe. The book starts with what RLHF does and why it was created, with seminal technical milestones in its young history and a primer on reinforcement learning context needed to understand the book. The core of the book details every optimization stage in using RLHF, from starting with instruction tuning to training a reward model and finally all of rejection sampling, reinforcement learning, on-policy distillation, and direct alignment algorithms. The book also discusses broader topics, such as the origins of RLHF -- both in recent literature and in a convergence of disparate fields of science in economics, philosophy, and optimal control. The book concludes with advanced topics -- understudied or emerging research questions in synthetic data, tool-use, character training, and evaluation -- and open questions for the field. The book is released with a variety of companion resources, including a codebase, a library to compare model completions from within post-training stages, and an educational course, to be a one-stop shop for learning all foundational concepts for post-training language models.

10 The Nature of Preferences … 1 Introduction

The section frames RLHF as a way to use human preferences to shape model responses and elicit pretrained models’ latent capabilities within post-training. It introduces RLHF’s three-stage recipe, benefits and limitations, development and core methods, and open directions.

1 Introduction

  • RLHF addresses hard-to-specify problems arising from human preferences and grew from traditional RL applications into a method widely associated with modern language models.Its motivating question was whether basic preference signals could guide optimization on difficult problems.Source: §1 · p. 7

1.2 What Does RLHF Do?

  • Unlike instruction fine-tuning’s next-token updates, RLHF compares whole responses, indicating both what is better and what to avoid.The section describes RLHF as integrating subtle stylistic and behavioral features, supporting safety [7] [5] and generalizing across domains better than instruction fine-tuning.Source: §1.2 · p. 10
  • RLHF can over-optimize proxy rewards and costs more than instruction fine-tuning in compute, data, and time, so it needs regularization and a strong starting point.Reward-model best practices remain unsettled and application-dependent; implementation can also encounter length bias [10].Source: §1.2 · pp. 10–11

1.3 Walkthrough of an RLHF Recipe

  • The canonical RLHF recipe moves from instruction tuning to a preference-trained reward model, then uses reinforcement learning to improve sampled responses.The reward model learns from preference relations across prompts, completions, and labelers; RL updates the model iteratively while maintaining its initial capabilities.Source: §1.3 · pp. 11–12

1.4 An Intuition for Post-Training

  • Post-training is framed as eliciting valuable behaviors already present in a base model, and the book explains its methods, trade-offs, and practical implementation.The book aims to clarify research and trade-offs, provide basic code examples, and build intuition; it also covers related methods in the broader post-training suite.Source: §1.1 · p. 8, §1.4 · p. 12, §1.3 · p. 12
  • Updating only post-training raised OLMoE Instruct’s evaluation average on popular benchmarks from 35 to 48.The first and second versions retained the majority of pretraining; the second updated only post-training.Source: §1.4 · p. 12
  • Post-training’s task is to cultivate base models’ latent potential, reshaping next-token predictors into conversational question-answering systems while eliciting pretrained knowledge and intelligence.The text calls this the Elicitation Theory of Post-training.Source: §1.4 · p. 13
  • Post-training is not merely superficial style adjustment: reinforcement learning on mathematics problems can elicit chain-of-thought reasoning and improve scores across reasoning evaluations.The cited examples include BigBenchHard, Zebra Logic, and AIME.Source: §1.4 · p. 13

1.5 How We Got Here

  • Open post-training moved from early human-and-synthetic-data fine-tuning through skepticism about RLHF to DPO, which optimized pairwise preferences directly and gained traction with breakthrough models.The text says lower learning rates enabled models including Zephyr-Beta and Tülu 2 [27].Source: §1.5 · pp. 14–15
  • Large-scale post-training is a complex, multi-stage process, and emerging RLVR and reasoning methods build on RLHF; this book captures the first stable RLHF literature.The text identifies instruction tuning, RLHF, and prompt design among the stages used in full post-training.Source: §1.5 · p. 15

1.6 Scope of This Book

  • The book focuses on recurring techniques, problems, and trade-offs in canonical RLHF, including instruction tuning, reward modeling, reinforcement learning, reasoning, direct alignment, and rejection sampling.Source: §1.6 · p. 15, §1.6.1.2 · pp. 15–16

1.6.1 Chapter Summaries

1.6.3 How to Use This Book

  • The book aims to provide enough knowledge to implement a toy system or enter the literature, not a comprehensive textbook; readers should compare papers because results can be noisy and hard to reproduce.Source: §1.6.3 · p. 17

1.7 Future of RLHF

  • As reinforcement-learning fine-tuning succeeds, RLHF is framed as a bridge to further investment in these methods and as a lens on mapping human values and objectives into AI systems.Source: §1.7 · p. 17

2 A Tiny History of RLHF

The chapter traces a rapidly evolving field through key papers leading to ChatGPT, explaining how earlier work shaped RLHF practices while acknowledging that some methods may change.

2.1 Origins to 2018: RL on Preferences

  • TAMER used human action scores to train reward models and policies, while COACH used positive and negative feedback to tune advantages; trajectory preferences helped in some domains.Source: §2.1 · pp. 18–19

2.2 2019 to 2022: RL from Human Preferences on Language Models

  • From 2019, language-model preference training formalized canonical RLHF terms [39] and spread to summarization, InstructGPT, WebGPT [4], GopherCite [41], Sparrow [42], and book summarization [40].Source: §2.2 · p. 19
  • This period defined reward-model over-optimization, language models as alignment objects [23], and red teaming [44], while work continued on Claude [5] and open-source RLHF tools,,.Source: §2.2 · p. 19

2.3 2023 to the Present: The ChatGPT Era

  • ChatGPT described RLHF training using InstructGPT methods with slight data-collection differences [48], and subsequent leading language models adopted RLHF extensively.Examples include Constitutional AI for Claude [24], Llama 2 [49] and Llama 3 [29], Nemotron [30], and Tülu 3 [6].Source: §2.3 · p. 20
  • RLHF is broadening into preference fine-tuning, including process rewards, DPO-inspired direct alignment, execution feedback,, and online reasoning methods inspired by o1.Source: §2.3 · p. 20

3 Training Overview

RLHF optimizes a proxy for human preferences while regularizing distance from a pretrained policy, using multiple training stages. Canonical recipes range from instruction tuning, reward modeling, and RL to increasingly iterative post-training.

3.1 Problem Formulation

  • Standard RL maximizes discounted expected reward over policy–environment trajectories, illustrated by a thermostat learning through trial and error and CartPole’s continuous control.The thermostat loop observes a state, acts, receives reward, and updates its policy; CartPole has four continuous state variables and physics-based transitions.Source: §3.1 · p. 21, §3.1.1 · p. 23, §3.1.2 · pp. 23–24

3.1.3 Manipulating the Standard RL Setup

  • Standard RLHF uses a learned preference reward, treats each prompt and completion as one rollout, and assigns the whole response reward with γ = 1.Unlike standard RL, prompts are sampled as initial states and completions do not define subsequent prompts; the single-turn objective can omit the horizon and discount factor.Source: §3.1.3 · pp. 24–25

3.1.4 Fine-Tuning and Regularization

  • A KL penalty constrains RLHF fine-tuning to stay near its strong pretrained starting policy, with larger β enforcing greater closeness and smaller β allowing more reward pursuit.This regularization aims to permit reward maximization without over-optimization; training often treats distance from the initial model as a KL budget.Source: §3.1.4 · pp. 25–26

3.1.5 Optimization Tools

  • Post-training combines instruction fine-tuning with options including reward modeling, rejection sampling, policy gradients, and direct alignment algorithms.Modern RLHF-trained models use instruction fine-tuning followed by a mixture of other optimization options.Source: §3.1.5 · pp. 26–27

3.1.6 Subtle Advantages of RL in Post-Training Language Models

  • RL requires more infrastructure than instruction tuning or DPO-like methods, yet its updates can improve robustness and chat quality while preserving broad capabilities through targeted training.The text describes RL losses as robust, scalable, effective, and flexible; Tülu 3 is cited as using math-only RL while maintaining performance across a broad task suite [6].Source: §3.1.6 · p. 27

3.2.1 InstructGPT

  • The canonical InstructGPT recipe [5] applied instruction tuning, reward modeling, and RLHF to a base language model.The stages used about 10K instruction examples, about 100K pairwise prompts for reward modeling (33K in the paper), and a separate set of about 100K RL prompts (31K in the paper).Source: §3.2.1 · pp. 27–28

3.2.2 Tülu 3

  • Tülu 3’s open multi-stage recipe combines instruction tuning, on-policy preference data, and reinforcement learning with verifiable rewards, and has been applied to Llama 3.1 [6], OLMo 2 [58], and SmolLM [59].Its stages use about 1M instruction examples, about 1M preference pairs, and about 10K RLVR prompts, respectively.Source: §3.2.2 · p. 29

3.2.3 DeepSeek R1

  • DeepSeek R1’s reasoning-model recipe [15] shifts compute across stages, progressing from cold-start reasoning samples and RLVR to rejection-sampling SFT and mixed reinforcement learning.It uses 100K+ cold-start samples, then RLVR until convergence; near convergence it builds an approximately 800K-sample SFT dataset, followed by mixed RL with verifiable rewards and preference reward models.Source: §3.2.3 · pp. 29–30

4 Instruction Fine-Tuning

Instruction fine-tuning (IFT), also called supervised fine-tuning (SFT), trains pretrained models on instruction-response examples, making cross-task instruction-following more reliable. Its data format and training choices establish a foundation for RLHF and other post-training stages.

4 Instruction Fine-Tuning

  • IFT/SFT emerged as pretrained-model scaling and prompting converged with unified text-to-text task framing, making instruction-response training more reliable for cross-task generalization,,,.Source: §4 · p. 31
  • IFT is a standard, simple way to adapt language models to a target task distribution and provides the instruction-following foundation needed for preference collection and online RLHF.Source: §4 · p. 31

4.1 Chat Templates and the Structure of Instructions

  • Chat templates serialize role-tagged messages into a flat token sequence, commonly ending with an assistant prefix that cues generation to continue until an end-of-sequence token.The template is used across post-training stages, including instruction tuning and RLHF.Source: §4.1 · pp. 32–33
  • The traditional roles are system, user, and assistant: system provides first-turn instructions or context, while user and assistant carry the person’s messages and model responses.Source: §4.1 · pp. 32–33

4.2 Best Practices for Instruction Tuning

  • High-quality completions and prompts matched to downstream tasks matter; about 1M prompts can support excellent post-training, with diminishing returns from further scaling and some noise recoverable in later stages.Human datasets such as No Robots with as few as 10K samples were state-of-the-art soon after ChatGPT, while large-scale synthetic datasets later worked best on most tasks.Source: §4.2 · p. 34

4.3 Implementation Details

  • Compared with pretraining, IFT uses smaller batches and typically a learning rate one to two orders of magnitude lower; sequence packing can increase effective batch size and support a higher rate.Source: §4.3 · pp. 34–35
  • IFT retains autoregressive loss but masks prompt tokens; multi-turn training can score only the final assistant turn or all assistant turns while masking user turns.Source: §4.3 · p. 35

4.4 Suggested Experiments

  • The suggested SFT exercise shows a base model shifting from rambling to concise answers, then recommends a learning-rate sweep to compare clean answering with overfitting.The example trains OLMo-2-0425-1B on No Robots and checks generations every 50 optimizer steps.Source: §4.4 · p. 36

5 Reward Modeling

This section explains how reward models learn human preferences as scalar proxy objectives for downstream RLHF, including Bradley–Terry pairwise training, common architectures, implementation, variants, and outcome-based alternatives. It also covers differences in what reward models score and how those distinctions shape their training and use, along with LLM-as-a-judge preference data and generative reward models.

5.1 Training a Bradley-Terry Reward Model

  • Bradley–Terry training models pairwise preference from score differences and fits chosen-versus-rejected completions by minimizing their negative log-likelihood.Adding a constant to every score leaves the preference probability unchanged; the loss applies a sigmoid to the chosen-minus-rejected score difference.Source: §5.1 · pp. 37, 39

5.2 The Default Reward Model Architecture

  • The common architecture appends a small linear head to a language model and returns one scalar reward for each prompt-completion pair.The implementation example extracts the final hidden state at the last real token before applying the linear head.Source: §5.2 · p. 40, §5.3 · p. 41

5.3 Implementation Example

  • Implementation complexity centers on building the data loader and inference pipeline correctly; reward models are commonly trained for only 1 epoch to avoid overfitting.The example scores chosen and rejected inputs jointly and applies the pairwise loss to their rewards.Source: §5.3 · pp. 40–42

5.4 Reward Model Variants

  • Reward modeling remains relatively under-explored, and modifications to the traditional loss have not converged on a single best practice.Source: §5.4 · p. 42

5.4.1 Preference Margin Loss

  • Preference margins can preserve rating magnitude in pairwise training: Llama 2 uses the score difference minus a margin, while Llama 3 removed the margin after diminishing improvements with scaling.For example, ratings of 5 and 2 yield a margin of 3.Source: §5.4.1 · p. 42

5.4.2 Balancing Multiple Comparisons Per Prompt

  • InstructGPT forms pairwise comparisons from K = 4 to 9 completions per prompt, grouping and averaging each prompt’s comparisons to limit overfitting and prevent prompts with more pairs dominating the loss.Source: §5.4.2 · pp. 42–43

5.4.3 K-Wise Loss Function

  • A Plackett-Luce K-wise loss, used in Starling 7B and 34B, trains on ranked action lists and reduces to Bradley–Terry for K = 2 [74, 75].Once trained, these models are used similarly to other reward models during RLHF.Source: §5.4.3 · p. 43

5.5 Outcome Reward Models

  • For reasoning-heavy tasks, outcome reward models train on correct-versus-incorrect completions using per-token binary cross-entropy on completion tokens, rather than Bradley–Terry’s chosen-rejected structure [77].The outcome label is copied across completion tokens, while prompt tokens are masked from the loss.Source: §5.5 · pp. 43–44
  • An outcome reward model predicts correctness probabilities at each completion token from the final-answer outcome, so reasoning errors are not captured and token-level learning can be noisy.Prompt tokens are ignored at inference, and completion probabilities can be aggregated into a response score for verification, filtering, or reranking.Source: §5.5 · p. 45, Figure 13 · p. 45

5.6 Process Reward Models

  • Process reward models predict whether individual reasoning steps are correct, using step-level supervision rather than an outcome label for the entire response.Traditional PRMs predict at step boundaries and may use correct (+1), neutral (0), and incorrect (-1) labels; correctness does not necessarily mean a step is on the right path.Source: §5.6 · pp. 45–46, Figure 15 · p. 47

5.7 Comparing Reward Model Types (and Value Functions)

  • Reward-model types capture different notions of quality and differ in what they predict and how they are trained.The section summarizes these distinctions across reward-model types.Source: §5.7 · p. 48, Table 2 · p. 48

Value Functions

  • Unlike offline-trained ORMs, value functions estimate expected remaining return, typically from on-policy rollouts, and provide baselines for policy-gradient advantages.A discount factor of 1 can make a value function resemble an ORM, while a PRM trained with per-step labels remains a PRM even if supervised using rollouts from intermediate states.Source: Value Functions · pp. 48–49

5.7.1 Inference Across Reward Model Types

  • At inference, preference models score a whole answer, ORMs score completion tokens, PRMs score reasoning steps, and value functions estimate remaining return from a prefix.ORM scores can be aggregated by mean, minimum, or other token-level rules; PRMs can guide search by pruning low-scoring branches, while value estimates support per-token RL advantages.Source: §5.7.1 · pp. 49–50

5.8 Generative Reward Modeling (a.k.a. LLM-as-a-judge)

  • Generative reward models, including models trained as judges [84], tend to trail existing reward models on reward-model evaluations.LLM-as-a-judge prompts a language model to compare two completions against human-preference criteria; its evaluation use also expanded to preference-data creation, including AlpacaEval [79], Arena-Hard, and WildBench.Source: §5.8 · pp. 50–51

5.9 Further Reading

  • Reward-model research established itself academically in 2024, with RewardBench providing common infrastructure and evaluations covering known-answer accuracy and more subjective correlations.Subsequent evaluation has broadened across general preferences and specialized domains.Source: §5.9 · p. 51

6 Reinforcement Learning

This section examines how reinforcement learning updates language-model policies from reward-model feedback, focusing on policy-gradient methods and their advantage estimation, clipping, importance sampling, and loss-aggregation choices. It also covers token- and sequence-level framings, practical distributed training and implementation, alternative algorithms, and experiments for comparing and tuning methods.

6.1 The Role of Reinforcement Learning in RLHF

  • RLHF combines a trainable policy, reward model, and frozen reference model for KL control; PPO adds a learned value model, while data quality strongly affects success.The section covers PPO, GRPO, and REINFORCE as central policy-gradient algorithms; REINFORCE-style methods are valued for avoiding a separate value model [111].Source: §6.1 · p. 53, Figure 16 · p. 54

6.2 Policy Gradient Algorithms

  • Policy gradients increase action probability when the score Ψt is positive and decrease it when Ψt is negative, using sampled prompt-completion rollouts to estimate expected return.Language-model RLHF typically sets γ = 1 because it optimizes the completion collectively rather than individual tokens.Source: §6.2 · pp. 55–56

6.2.1 Deriving the Policy Gradient

  • The policy-gradient derivation reduces trajectory-probability gradients to token log-probability gradients, enabling Monte Carlo estimation and autodiff implementation.Only policy probabilities depend on policy parameters; the initial-state distribution and environment transitions do not, and a learned value estimate can form a temporal-difference advantage.Source: §6.2.1 · pp. 57–58

6.2.2 Vanilla Policy Gradient

  • Vanilla policy gradients can have high-variance updates, especially with noisy rollouts or sparse rewards, while state-dependent baselines reduce variance without changing expected gradients.Simple baselines include batch-average rewards and moving averages.Source: §6.2.2 · p. 59

6.2.3 REINFORCE

  • REINFORCE is a Monte Carlo vanilla policy-gradient method that weights log-probability gradients by return adjusted with a baseline.Its update combines a positive learning rate, reward normalization or baseline, and the eligibility term ∇θ log πθ(at | st).Source: §6.2.3 · pp. 59–60

6.2.4 REINFORCE Leave One Out (RLOO)

  • RLOO uses the other completions for a per-prompt baseline, yielding an unbiased, low-variance advantage estimate when multiple completions are sampled per prompt.RLOO folds aggregated token-level KL into sequence reward and broadcasts sequence advantage; unlike canonical GRPO’s loss-level KL and PPO-style clipping choices, it can also combine with PPO clipping.Source: §6.2.4 · pp. 60–61

6.2.5 Proximal Policy Optimization (PPO)

  • PPO [116] is presented as a foundational deep-RL algorithm whose objective optimizes policy probabilities weighted by advantages.The supplied passage introduces its objective but does not provide further PPO results or mechanics.Source: §6.2.5 · p. 61
  • PPO uses an importance-sampling ratio between the current and data-collection policies, clipping it to [1 − ε, 1 + ε] to constrain policy updates.Its clipped surrogate objective takes the minimum of the unclipped and clipped advantage-weighted terms.Source: §6.2.5 · pp. 62–63

6.2.6 Understanding the PPO Objective

  • Within PPO’s trust region, updates follow standard policy gradients; beyond it, the objective can flatten and stop further movement in the already-desired direction.The minimum operation selects the unclipped or clipped term according to the advantage sign and policy-ratio region.Source: §6.2.6 · p. 64, Figure 20 · p. 65

6.2.6.1 Positive Advantage (At > 0)

  • For positive advantage, PPO increases an action’s likelihood only until its ratio reaches 1 + ε, then its gradient is zero.Once the action is already more likely beyond that threshold, clipping prevents over-reinforcement.Source: §6.2.6.1 · p. 66

6.2.6.2 Negative Advantage (At < 0)

  • For negative advantage, PPO decreases an action’s likelihood only until its ratio falls below 1 − ε, then its gradient is zero.Clipping prevents further suppression once the action is already less likely beyond that threshold.Source: §6.2.6.2 · p. 67

6.2.7 Value Functions and PPO

  • PPO trains a learned value function as a per-token baseline and uses its estimates to compute advantages for policy updates.The section identifies Generalized Advantage Estimation (GAE) as the canonical modern target method, while also noting Monte Carlo targets and separate value and policy losses.Source: §6.2.7 · pp. 67–68, Figure 21 · p. 68

6.2.8 Group Relative Policy Optimization (GRPO)

  • GRPO avoids a learned value function by estimating advantages from multiple completions to one prompt, assigning each token the episode-level value; this can save memory and avoid value-learning challenges.Its standard-deviation normalization can bias learning toward prompts with low answer variation; Liu et al. 2025 [118] propose removing that term, potentially down-weighting prompts where only a few completions are correct.Source: §6.2.8 · pp. 69, 71

6.2.9 Group Sequence Policy Optimization (GSPO)

  • GSPO requires importance sampling to correct the distribution mismatch when optimizing the current policy with data collected from a previous policy.The supplied passage introduces the importance-sampling identity for estimating expectations under a target distribution from samples drawn under another distribution.Source: §6.2.9 · p. 72
  • GSPO replaces token-level importance ratios with a length-normalized sequence-level ratio, matching the correction granularity to sequence-level rewards.A single response weight is applied uniformly to its tokens; the group-relative advantage computation remains as in GRPO.Source: §6.2.9 · p. 73

6.2.10 Clipped Importance Sampling Policy Optimization (CISPO)

  • CISPO clips importance weights rather than the surrogate objective, preserving gradient signals for all tokens while bounding their importance-weight amplification or suppression.It uses a stop-gradient on the clipped weight in a REINFORCE-style formulation, trading bias for variance control.Source: §6.2.10 · pp. 73–74

6.2.11 Comparing Algorithms

  • REINFORCE uses Monte Carlo rewards and a state baseline, RLOO uses other samples’ mean reward, PPO adds a learned value function and clipped ratio, and GRPO removes the value function via group-normalized rewards.CISPO clips importance weights, GSPO uses a length-normalized policy ratio, and DPO bypasses the reward model using preference pairs; only PPO requires a learned value function, and all can use learned or verifiable rewards.Source: §6.2.11 · pp. 74–75

6.3 Implementation

  • Language-model RLHF implementations vary in handling truncated generations, value-network initialization, reward or advantage normalization, KL estimation, and static versus dynamic KL penalties.When generations hit a length cap, scoring only EOS outputs and penalizing excessive length avoids out-of-distribution reward-model scores; InstructGPT and Tülu 3 [6] initialize the value network from the reward model.Source: §6.3 · pp. 76–77

6.3.1 Policy-Gradient Basics

  • Policy-gradient updates reinforce actions with positive advantage, reduce the likelihood of actions with negative advantage, and make no update for zero advantage.The probability ratio compares the new policy with the old policy that generated the batch.Source: §6.3.1 · pp. 77–78

6.3.2 Loss Aggregation Tradeoffs

  • Strategy 1 gives each sequence equal batch-loss weight, Strategy 2 gives each token equal weight, and Strategy 3 normalizes by maximum length Lmax while longer sequences contribute more total gradient.Strategy 1 is standard GRPO and also appears in some PPO implementations; Strategies 2 and 3 are associated with DAPO and Dr. GRPO [118], respectively.Source: §6.3.2 · p. 78

6.3.2.1 Why Does This Matter?

  • With per-sequence normalization, the short sequence receives larger per-token gradients (0.25 versus 0.14), while Strategies 2 and 3 equalize them; gradient accumulation can reverse the balance.The preferred aggregation depends on the training setup, and RLHF implementations often favor numerical stability or lower loss variance.Source: §6.3.2.1 · p. 80

6.3.2.2 Related: MDP vs. Bandit Framing

  • The MDP view treats tokens as actions with token-level advantages, whereas the bandit view treats a completion as one action and broadcasts its sequence-level advantage to all tokens.PPO commonly uses the MDP view; RLOO and GRPO-style advantages commonly use the bandit view, and implementations differ in whether KL is added per token or folded into rewards.Source: §6.3.2.2 · p. 81
  • In virtually all RLHF implementations, γ is 1.0 because the reward scores the whole completion, so discounting earlier tokens lacks a principled basis.Discounting may matter again for agentic RL with genuinely sequential actions and differing long-term consequences.Source: §6.3.2.2 · p. 81

6.3.3 Asynchronous RL Systems

  • Exact on-policy execution slows training, so language-model RL commonly separates sampling actors from training learners and accepts nearly on-policy data.Long reasoning roll-outs can bottleneck generation; sequence-level packing helps distribute shorter samples, while fully asynchronous training can use recent roll-outs across instances and ease multi-datacenter scaling [132] [133].Source: §6.3.3 · pp. 82–83

6.3.4 Truncated Importance Sampling; 6.3.4.1 REINFORCE with TIS (Single Gradient Step)

  • Truncated importance sampling caps weights at min(ρ, C), trading bias for bounded variance; single-step REINFORCE uses it to correct the learner–sampler gap.With no policy drift, the learner–sampler mismatch is the only correction needed.Source: §6.3.4 · pp. 83–84, §6.3.4.1 · p. 84

6.3.4.2 PPO/GRPO with TIS (Multiple Gradient Steps)

  • With multiple gradient steps, recomputed learner log-probabilities let TIS separately correct sampler–learner divergence, while using sampler log-probabilities makes the policy ratio capture that mismatch.The latter ratio starts away from 1.0 before gradient updates.Source: §6.3.4.2 · p. 84

6.3.5 Example: PPO; 6.3.5.1 PPO/GRPO Simplification with One Gradient Step per Sample (No

  • PPO’s clipped objective takes the more pessimistic of the unclipped and clipped advantage-weighted losses, limiting policy updates in either direction.PPO also learns a value function; with one gradient step per sample, PPO/GRPO can omit the second policy-gradient and clipping logic.Source: §6.3.5 · pp. 86–87, §6.3.5.1 · p. 87

6.3.6 Example: GRPO

  • GRPO normalizes rewards within each prompt’s generation group instead of using a learned value baseline, removes the value network, and adds the KL penalty to the loss [121].Source: §6.3.6 · pp. 87–88

6.3.6.1 RLOO vs. GRPO

  • RLOO estimates each completion’s advantage against the average reward of the other completions for the same prompt, closely paralleling GRPO’s group-relative baseline.Source: §6.3.6.1 · pp. 88–89

6.4.1 Generalized Advantage Estimation (GAE)

  • GAE balances bias and variance by exponentially averaging multi-step advantage estimates, with λ ranging from single-step TD at 0 to full returns at 1.λ = 0.95 is a common default for LLM fine-tuning. In the backward implementation, terminal tokens stop accumulation; value targets train the value function, while detached advantages weight the policy gradient, and γ = 1.0 is common for short language-model episodes.Source: §6.4.1 · pp. 89–90

6.4.2 Double Regularization

  • With one gradient step per batch, PPO’s clipping often does not activate in language-model RLHF, so KL penalties predominate, though multi-step exceptions exist.Clip fraction measures the share of samples outside the clipping interval, but does not mean each has zero gradient; examples of multi-step training include DAPO’s 16 steps and Tülu 3’s 4 iterations for 8B and 70B models, versus 1 for 405B.Source: §6.4.2 · p. 91

6.4.3 Further Reading

  • Further-reading methods include pairwise-data P3O, smooth-gated SAPO, off-policy CoPG, reward-uncertainty baseline ReMax, and Mirror Descent variants used in some foundation models.DAPO modifies GRPO with asymmetric clipping, dynamic sampling, per-token loss, and a soft penalty for overly long samples; VAPO combines DAPO optimizations with value-function pretraining and length-adaptive GAE. Mirror Descent is an optimization method, not itself a policy-gradient algorithm.Source: §6.4.3 · pp. 91–92

6.5 Suggested Experiments

  • Suggested experiments use a small observable setup to test GRPO, compare group-relative and single-sample estimators, vary sampling settings, and progress toward math tasks.The companion implementation defaults to Qwen/Qwen3-1.7B on spell_backward; track correctness, format, and binary metrics, checking whether prompt groups contain reward contrast. Vary rollout count, temperature, data size, and format weight, then consider GSM8K-style work with a sub-1B Qwen model and group-contrast diagnostics.Source: §6.5 · pp. 92–93

7 Reasoning and Inference-Time Scaling

This section surveys how verifiable feedback, inference-time scaling, and evolving training practices have shaped reasoning models, from early releases to current RLVR techniques. It treats these techniques as foundations while outlining emerging work to explain scaling and training outcomes.

7.1 The Role of RLVR

  • RLVR replaces a learned reward model with a verifier that scores correctness, often using binary rewards or partial credit for tests.Verifiers can check a mathematical answer or multiple valid code solutions; many setups omit a reward model, though combined rewards are possible.Source: §7.1 · pp. 94–96
  • Repeated sampling and updates toward correct answers, revisiting the same questions, can improve performance on unseen questions and some unseen domains.The cycle depends on a careful data distribution and stable training infrastructure.Source: §7.1 · p. 97

7.2.1 Why Does RL Work Now?

  • RL training became more practical as stability barriers fell, open-source tools spread, and models reached the capabilities needed for reasoning training.Examples of available tools include TRL, Open Instruct [6], veRL, and OpenRLHF [154].Source: §7.2.1 · p. 98

7.2.2 RL Training vs. Inference-Time Scaling

  • Inference-time scaling uses more computation at generation to improve downstream performance, and RL training often elicits longer responses correlated with better performance.Earlier methods included value-guided sampling [155] and repeated random sampling with answer extraction [156]; longer responses alone are not useful without that performance correlation.Source: §7.2.2 · pp. 98–99

7.2.3 The Future (Beyond Reasoning) of RLVR

  • RLVR focuses on producing correct answers and can reinforce behaviors over hundreds or thousands of epochs on the same data, beyond standard instruction tuning’s 1 or 2 epochs.This performance-focused approach may extend beyond reasoning domains.Source: §7.2.3 · p. 99

7.3 Understanding Reasoning Training Methods

  • Reasoning-model recipes retain instruction fine-tuning and RLHF but increase RLVR and change the order of training stages, often using large-scale RL or distillation.Distillation uses instruction tuning on outputs from another model that has undergone substantial RLVR training.Source: §7.3 · p. 99

7.3.1 Reasoning Research Before OpenAI o1 or DeepSeek R1

  • Before o1 and R1, STaR, TRICE, VinePPO, code-feedback training, and verifier-feedback work explored learning reasoning from correctness signals.STaR approximated policy gradients with filtered samples and cross-entropy; TRICE optimized generated traces, while VinePPO used PPO with binary math rewards. Tülu 3 emphasized retaining broad evaluation performance.Source: §7.3.1 · p. 100

7.3.2 Early Reasoning Models

  • Hunyuan-TurboS is described as using an RL recipe that avoids entropy collapse and beats DeepSeek on AIME.The table summarizes notable reasoning-model technical reports from 2025.Source: Table 4 · p. 100, §7.3.2 · p. 101
  • Olmo 3 Think [18] presents a fully open model flow with lifecycle stages, checkpoints, and data, positioning its 32B model as a flagship open thinking model.The broader early-model landscape includes varied architectures, training recipes, and degrees of openness.Source: §7.3.2 · p. 102

7.3.3 Common Practices in Training Reasoning Models

  • RLVR recipes filter for problems the starting model solves 20–80% of the time, then use online filtering or curricula to sequence problem difficulty.Difficulty filtering samples completions and retains prompts with intermediate success rates to provide learning gradients; these documented examples are not a definitive optimal recipe.Source: §7.3.3 · pp. 102–103
  • Reasoning-model RL recipes adapt optimization and throughput with relaxed clipping, removing KL penalties, asynchronous data, and format, language-consistency, or length rewards.Longer RL runs and less over-optimizable rewards motivate KL removal; highly variable completion lengths can leave compute idle, motivating asynchronous updates.Source: §7.3.3 · p. 103
  • Parallel test-time rollouts can improve answers over a single rollout through majority voting or scoring-model selection, while text-only reasoning training can improve multimodal performance.Open documented recipes had not commonly adopted scoring-model selection as of 2026; some models also use prompts or length control to toggle reasoning or set its effort.Source: §7.3.3 · p. 104

7.4 Looking Ahead

  • Olmo 3 Think [18] documents a nearly four-week training run on 220 GPUs, while research is beginning to formalize how compute and data scale reasoning RL performance.The field is changing rapidly, so current practices may be superseded.Source: §7.4 · p. 104
  • RL has become a load-bearing component of frontier-model training, while current RLVR techniques are foundations for methods still to come.The chapter describes difficulty filtering, format rewards, and related techniques as the field’s current best understanding, not final answers.Source: §7.4 · p. 104

8 Direct-Alignment Algorithms

This section covers direct-alignment algorithms for optimizing the RLHF preference objective from preference data without an explicit reward model or RL optimizer, including DPO’s comparison of chosen and rejected responses against a reference model. It also discusses synthetic preference data, implementation choices, and experiments comparing these offline methods with policy-gradient and RL approaches.

8 Direct-Alignment Algorithms

  • DAAs optimize the same RLHF preference objective from the same data without an explicit reward model or RL optimizer, reducing compute and implementation barriers.They are easier to implement and experiment with on both toy and production examples.Source: §8 · p. 105

8.1.1 How DPO Works

  • DPO compares chosen and rejected completions’ probability shifts relative to a reference model; its loss falls when the chosen shift exceeds the rejected shift.The hyperparameter β balances reward optimization against KL divergence from the reference model.Source: §8.1.1 · pp. 105–106

8.1.2.3 Deriving the BT DPO Gradient

  • DPO’s gradient weights updates more when the reward estimate is incorrect, increases chosen-response likelihood, decreases rejected-response likelihood, and scales this balance by β.The chapter derives this gradient from the Bradley-Terry DPO loss.Source: §8.1.1 · p. 106, §8.1.2.3 · p. 111

8.1.2.1 Deriving the Optimal RLHF Solution

  • For a fixed KL constraint β, the optimal RLHF policy is the distribution matching the reference policy weighted by the reward, normalized by a partition function.The derivation identifies the objective as a KL divergence minimized when the learned policy equals that distribution.Source: §8.1.2.1 · pp. 108–109

8.1.2.2 Deriving DPO Objectives for BT Models

  • Under a Bradley-Terry preference model, substituting the optimal-policy reward into preference likelihood yields the DPO loss; the Plackett-Luce derivation is less used in practice.This reparameterization expresses preference likelihood through the optimal policy rather than an external reward model.Source: §8.1.1 · p. 106, §8.1.2.2 · p. 111

8.2 Numerical Concerns, Weaknesses, and Alternatives

  • Variants address equal pair weighting, noisy labels, richer reward signals, or loss and efficiency changes, including REBEL, cDPO, IPO, ODPO, ORPO, and SimPO.ORPO removes the reference model; SimPO averages log-probabilities or adds length normalization.Source: §8.2 · p. 112
  • DPO can reduce probabilities of both chosen and rejected responses, disproportionately suppressing rejected responses and potentially displacing unaddressed behaviors.Cal-DPO and AlphaPO mitigate this displacement; online variants generate new completions and incorporate preference signals during training, including DPO [199] and D2PO [200].Source: §8.2 · pp. 112–113, Figure 27 · p. 113

8.3 Implementation Details

  • DPO implementation computes policy and reference chosen–rejected log-probability gaps, applies a negative log-sigmoid loss, and logs detached implicit rewards in standard training stacks.The required values are collated during the model forward pass, with the addition of a reference model.Source: §8.3 · p. 113
  • Caching reference-model log-probabilities before training reduces DPO peak memory use by 50%, while its β parameter explicitly sets the KL penalty.DPO steps toward the solution specified by β, unlike RL optimizers, which take steps based on the batch and recent data.Source: §8.3 · p. 114

8.4 DAAs with Synthetic Preference Data

  • Synthetic preference data is a practical starting point for DAAs, though judge biases can favor longer or judge-styled chosen responses.Best practices remain unsettled: on-policy completions help target the model’s token space, while Delta Learning argues that the chosen–rejected difference matters more than which models produced them.Source: §8.4 · p. 114
  • Most DAAs increase the probability margin between chosen and rejected completions; often both probabilities fall, but the rejected one falls more.Token-level gradients vary in magnitude and direction according to each token’s contribution to the preference margin.Source: §8.4 · pp. 114–115

8.5 DAAs vs. RL: Online vs. Offline Data

  • Policy-gradient and RL methods slightly outperform DPO and its variants, but DAAs remain widely used for their simplicity and rapid iteration.Because data can matter more than algorithms, DPO can still be a practical choice when iterating on training data and configurations.Source: §8.5 · p. 115
  • As reasoning models are increasingly trained with RL, renewed investment in RL may strengthen its infrastructure and entrench its advantage over DAAs for preference-tuning.The section frames this as a long-term expectation.Source: §8.5 · p. 115

8.6 Suggested Experiments

  • The companion code enables offline DPO experiments without a reward-model server or rollout loop; a small UltraFeedback run can check that margins move as intended without generation collapse.Monitor loss, accuracy, margins, chosen_rewards, rejected_rewards, and sample generations.Source: §8.6 · p. 115
  • Compare DPO, IPO, and length-normalized DPO using accuracy and margin behavior, since IPO loss is not on the same numeric scale as DPO.Reference-free SimPO and ORPO are sensitive to log-probability scaling and learning rate; varying data before the loss can reveal whether data changes matter more than objective changes.Source: §8.6 · pp. 115–116

9 Rejection Sampling

Rejection sampling generates candidate completions, scores them with a reward model, and fine-tunes the starting model on selected high-reward outputs. It is widely used and versatile, but its implementation and best practices remain underdocumented.

9 Rejection Sampling

  • Rejection sampling curates candidate completions, filters them with a trained reward model, and fine-tunes the original model on the selected outputs using the instruction-tuning loss.It can be applied after instruction fine-tuning, RL-based optimization, or RLVR.Source: §9 · p. 117

9.1 Training Process, Step by Step

  • The process selects prompts and a reward model, generates completions, ranks them, and instruction-fine-tunes the starting checkpoint on the selected outputs.Reusing all initial instruction-tuning prompts is simple but may cause overfitting; prompt selection, reward-model choice, and sequencing are poorly documented.Source: §9.1 · pp. 117–118

9.1.1 Generating Completions

  • Prompts commonly come from the instruction-training set, with N candidate completions generated for each prompt.Successful implementations have used 10 to 30 or more completions per prompt; too few can make training biased or noisy.Source: §9.1.1 · p. 118, §9.2 · p. 121

9.1.2 Scoring Completions

  • A reward model scores each prompt-completion pair, enabling selection either of the highest-reward completion per prompt or the top K pairs overall.The overall selection ranks the flattened reward matrix and maps the chosen indices back to their original prompt-completion pairs.Source: §9.1.2 · pp. 118–119, §9.1.2.1 · p. 119, §9.1.2.2 · p. 119

9.1.3 Fine-Tuning

  • After selection, the starting model is trained on the chosen completions with standard instruction fine-tuning.The procedure uses the selected completions to fine-tune the current model version.Source: §9.1.3 · p. 121

9.2 Implementation Details

  • Implementation settings include sampling temperatures commonly between 0.7 and 1.0, choices such as top-p or top-k, and a reward model that can strongly affect results.Some implementations mix generations from multiple models, but best practices for doing so are not established; fine-tuning settings are also not clearly documented.Source: §9.2 · pp. 121–122

9.3 Related: Best-of-N Sampling

  • Best-of-N follows the same generate-and-score procedure as rejection sampling but selects a completion at inference time without fine-tuning the model.For one prompt, top-K selection with K = 1 is equivalent to selecting the argmax completion.Source: §9.3 · p. 122

10 The Nature of Preferences

RLHF uses human preferences where correctness is unclear, translating them into optimization signals despite preferences’ complexity and instability. Its assumptions draw on fields with different accounts of preference, reward, utility, and value.

10.1 When Preference Replaces Correctness

  • Human preference replaces correctness for open-ended outputs: feedback supplies an indirect reward signal, while RLHF practice emphasizes empirical skill performance over calibration to values.Research on pluralistic alignment, datasets, and personalization addresses value alignment across populations.Source: §10.1 · pp. 124–125

10.2 The Origins of RLHF and Preferences

  • Human preferences, decision theory, optimal control, reinforcement learning, and deep learning converge in RLHF, whose architectures inherit uncertainty from these histories and use approximate solutions for under-specified problems.The chapter traces how assumptions about human nature and optimality shape modern preference-reward optimization.Source: §10.2 · p. 125

10.3 Specifying Objectives: From Logic of Utility to Reward Functions

  • RLHF combines preference, reward, and cost as measures of relative goodness, although social sciences retain distinct measurement methods and debate whether preferences exist or can be measured.Reducing many individual desires to one reward function conflicts with the view that preferences depend on social relations and task context.Source: §10.3 · pp. 125–127

10.4 Tools for Optimizing Utility

  • Classical RL optimizes reward-to-go in closed MDPs with stable objectives, but human preferences lack explicit optimal behaviors, and inverse reinforcement learning remains underrepresented in RLHF.Bellman-based guarantees apply to MDPs or tasks with a single closed-form reward; the engineering path to stable RLHF motivates further comparison with IRL.Source: §10.4 · pp. 127–128

10.5 Complexity of Optimizing Preferences

  • Human preferences drift, depend on context, and are compressed by reward models from text into a scalar, while sequential labeling can shift preferences and obscure decision-relevant dynamics.RL’s numerical value estimate and alignment’s moral or ethical value are distinct meanings that often go undifferentiated.Source: §10.5 · p. 128
  • Preference measurement is shaped by presentation and context, while fairness impossibility results and changing preferences complicate aggregation across people, places, and situations.Measurements may be nontransitive or incomparable; implicit proxies can interact with training and deployment, input options and timing can alter results, and low respondent accuracy can mask contextual differences.Source: §10.5 · p. 129

11 Preference Data

This section examines preference data as the proxy signal for preference fine-tuning, focusing on practical challenges and open questions in collecting feedback that reflects diverse human preferences. It covers on-policy needs, collection costs and biases, annotator context and representation, disagreement, and auditing whether model behavior matches annotation specifications.

11.1 Why We Need Preference Data

  • Preference data provides the proxy signal for tuning model behavior because complex human preferences cannot be fully captured in a clearly defined reward function.It is easier for people and AI systems supervising collection to distinguish a good answer from a bad one than to generate a good answer themselves.Source: §11 · p. 130, §11.1 · p. 130

11.2 Collecting Preference Data

  • Human preference collection is costly and opaque, while preference data is most robust when it comes from the current family of models rather than unrelated model pools.On-policy preference data is especially important for effective RLHF training [89], but the balance of human and AI preference data in popular models is unknown.Source: §11.2 · pp. 130–131

11.2.1 Interfaces

  • Collection interfaces shape the feedback people provide, but the effects of design choices are understudied and user-facing feedback may be used for training or evaluation.Interfaces range from choosing between two responses to thumbs-up or thumbs-down feedback; Arena also allows ties.Source: §11.2.1 · pp. 131, 133

11.2.2 Rankings vs. Ratings

  • Common practice trains on relative rankings rather than standalone ratings, while other signals include directional labels (KTO [261]), token-level feedback, and natural-language feedback.With rankings, users choose a preferred response among completions; ratings assign scores to individual completions and are often used as metadata or explored in related work.Source: §11.2.2 · p. 137, §11.2.4 · p. 139, §11.2.4.1 · p. 139

11.2.3 Multiturn Data

  • Multi-turn preference data can be collected at the final prompt or every response, then unrolled into single-prompt training examples with earlier turns masked from the loss.Continuing the conversation with the chosen answer and deciding whether prompt creators should also label preferences are practical considerations; prompt-creator labeling may help avoid sycophancy.Source: §11.2.3 · pp. 137–138

11.2.5 Sourcing and Contracts

  • Sourcing human preference data involves vendor and contract risks, and collection must be integrated with training plans to make iterative improvements useful.Data may arrive in weekly batches over a typical six-week contract, with teams hoping to see model improvement by week 4 or 5, though first-time teams may not achieve it.Source: §11.2.5 · pp. 139–140

11.3 Bias: Things to Watch Out For in Data Collection

  • Subtle annotation biases, including prefix bias, sycophancy, verbosity, and formatting habits, can transfer to models, so mitigating them distinguishes good from great preference data.Interventions vary in effectiveness; verbosity and formatting biases are easier to detect and mitigate than some other biases.Source: §11.3 · p. 141

11.4 Open Questions in RLHF Preference Data

  • An open question is how sparse judgments from paid annotators and consumer use capture the broad, diverse preference function they are intended to represent.Each instance records a preference between two text outputs, leaving questions about data curation and its impacts to be shared more openly.Source: §11.4 · p. 141
  • Professional data collection raises questions about how workplace context, interfaces, repetition, and instructions shape annotators’ preferences and their transfer to downstream users.The text also asks whether professional workers follow instructions or their innate values, and how this compares with volunteer workers.Source: §11.4 · pp. 141–142
  • Pairwise preferences may not capture preferences as intended, motivating comparison with multi-axis feedback and other ways people communicate preferences in text.Source: §11.4 · p. 142
  • Open questions include who annotates the data, how to maintain population diversity, how many people represent a population, and whether annotator disagreement is noise or signal.Source: §11.4 · p. 142
  • Because RLHF’s motivation of aligning models to human preferences has drifted toward user effectiveness, model behavior should be audited against annotator specifications, though tools such as OpenAI’s Model Spec offer limited visibility.The text notes that industrial RLHF’s closed nature makes this feedback loop difficult to measure.Source: §11.4 · p. 142

12 Synthetic Data & Distillation

This section traces synthetic data’s growing role in post-training, from lowering RLHF experimentation costs to supporting many stages, alongside distillation’s shift toward on-policy training. It also covers AI feedback for evaluation and rubric-driven RL, its costs and limits, Constitutional AI, and on-policy self-distillation experiments.

12.1 The Roles of Synthetic Data

  • Synthetic data now supports prompts, completions, preference feedback, and filtering, but humans remain essential at capability frontiers and for evaluation ground truth.Synthetic instruction data often exceeds human writing quality at scale, while synthetic preference data is comparable in academic work but remains a competitive moat for frontier labs; LLM judges scale evaluation, but benchmarks and ground-truth labels still require humans. Model collapse is most associated with unfiltered, repetitive outputs, while real-data mixtures, diverse teachers, deduplication, and quality filters mitigate it; evidence supports large-scale use without catastrophic regressions [271].Source: §12.1 · pp. 143–144

12.2 Distillation with Synthetic Data

  • Distillation serves both as a data engine across post-training and as a way to transfer specific skills, with GPT-4-class models broadening it to complex tasks.Open-model practice often distills data from closed API models into smaller open weights; curating high-quality prompts and filtering teacher responses are crucial to performance.Source: §12.2 · p. 145

12.3 The Path to On-Policy, Teacher-Student Distillation

  • Modern synthetic-data pipelines are distillation-inspired, while technical distillation research is developing into its own set of post-training methods, especially on-policy and self-distillation.The chapter treats distillation as part of the synthetic-data toolkit for now, while noting it may warrant a dedicated chapter in future editions.Source: §12.3 · p. 145

12.3.1 Adapting Knowledge-Distillation for LMs

  • Knowledge distillation uses soft labels to match the teacher’s distribution; language-model variants match token distributions or approximate sequence-level matching with a single teacher-generated output.With a fixed teacher and teacher-sourced training data, cross-entropy is equivalent to forward KL from teacher to student; sequence-level KD uses a high-probability beam-search output to avoid summing over all possible sequences.Source: §12.3.1 · pp. 146–147

12.3.2 From Offline to On-Policy Distillation

  • On-policy distillation addresses offline KD’s exposure bias by sampling student trajectories and supervising visited states with teacher feedback, adapting DAgger’s recovery logic.The motivating imitation-learning analysis gives compounding loss of O(ϵL^2) versus O(ϵL) for DAgger, but this is an analogy rather than a guarantee for LLM token-level distributional losses; OPD uses a reverse-KL objective.Source: §12.3.2 · pp. 147–149

12.3.3 Modern OPD Variants

  • Multi-Teacher On-Policy Distillation combines teachers using prompt-dependent weights, while self-distillation can use model or tool verification and inserted hints as privileged feedback.On-policy self-distillation can help a model correct its outputs without an explicitly stronger teacher; on-policy distillation can combine skills in a general model or advance a specialized deployment.Source: §12.3.3 · pp. 149–150

12.3.4 Suggested Experiments

  • The companion SDPO experiment implements on-policy self-distillation on a small string-reversal task, using one policy as both demonstration-conditioned teacher and question-only student.The example applies per-token reverse KL and is designed to make the training loop inexpensive to observe end-to-end on a single GPU.Source: §12.3.4 · p. 150
  • Sequence KD uses offline teacher outputs, on-policy distillation scores student rollouts with a separate teacher, and OPSD uses a hint-conditioned pass to guide the no-hint pass through reverse KL.Suggested OPSD sweeps vary num_rollouts, kl_top_k, and prompts_per_step; more rollouts can make a correct sibling demonstration easier to find and reduce skipped prompts, at added generation cost, while kl_top_k trades teacher-distribution coverage against compute.Source: §12.3.4 · pp. 150, 152, Figure 37 · p. 151, Figure 38 · p. 151

12.4 AI Feedback

  • AI feedback began with pairwise preferences [296] and is used in preference collection and reward-model training, while LLM-as-a-judge infrastructure also supports scalable evaluation and rubric-based rewards.The chapter treats AI feedback broadly as AI-generated or AI-augmented data explaining the quality of an input.Source: §12.4 · p. 152

12.4.1 Balancing AI and Human Feedback Data

  • Human preference data costs on the order of $1 or higher (or even above $10 per prompt), versus less than $0.01 for frontier-model AI feedback as of 2026, but AI feedback does not replace human data.Which domains favor AI over human feedback remains unsettled; one proposed balance routes challenging examples to humans and most data to AI feedback, while industry often views human data as a technical advantage.Source: §12.4.1 · pp. 152–153

12.4.2 Building Specific LLMs for Judgment

  • LLM judges can be inconsistent and self-prefer, but dedicated critic or judge models are not widely adopted; leading models usually suffice unless the task involves private information.Repeated sampling, self-refinement, or tournament ranking are among approaches proposed to improve judgment estimates.Source: §12.4.2 · pp. 153–154

12.5 Constitutional AI

  • Constitutional AI uses a human-written constitution and an LLM to create revised instruction data and synthetic preference labels for RLHF [24].For instruction data, the model repeatedly revises an answer against sampled principles; for preferences, a feedback model selects the completion better aligned with principles, often explaining its reasoning as a generative reward model.Source: §12.5 · pp. 154–155

12.5.1 Further Reading on CAI

  • Few Constitutional AI extensions are documented as clear improvements to RLHF recipes, despite work on behavior specifications, revised constitutions, open-source replications, and principle-driven methods [314] [318].Related research also applies principles to reward models and human judgments, generates principles at inference time, or frames principle-following as mutual-information maximization [42] [157].Source: §12.5.1 · p. 155

12.6 Rubrics: Prompt-Specific AI Feedback for Training

  • Rubric-based rewards let RL optimize sampled answers against prompt-specific, near-verifiable criteria, improving skills such as scientific reasoning or factuality,,,.To reduce per-prompt rubric-generation costs, a domain-level rubric can be refined by a supervising language model into prompt-specific scores; criteria should be atomic, objective, non-overlapping, self-contained, and weighted.Source: §12.6 · pp. 155–157

13 Tool Use and Function Calling

Tool use extends language models beyond their fixed weights by letting them invoke external capabilities. The section explains how calls are represented, trained, evaluated, and standardized.

13 Tool Use and Function Calling

  • External tools extend language models to high-precision tasks and broader workflows, while function calling adds schema-constrained arguments and code execution is a special case of tool use.Tool use emits a structured request, has an orchestrator execute it, and appends the result to context for continued generation.Source: §13 · p. 159

13.1 Tool-Use Overview

  • Tool-use evaluation checks exact-match tool and argument correctness, schema validity, end-to-end task completion, and consistent success across trials.ToolLLM and ToolBench cover 16,000+ real-world APIs, while BFCL is a popular function-calling accuracy benchmark.Source: §13.1 · p. 160

13.2 Interweaving Tool Calls in Generation

  • Tool-use generation interleaves model tokens with executed calls and returned outputs; training teaches when to call, format arguments, and use results, while masking tool-output tokens from loss.Models may emit multiple tool calls in one generation, and open models must accommodate tools users connect.Source: Figure 39 · p. 163, §13.2 · p. 163, §13.5 · p. 165

13.3 Multistep Tool Reasoning

  • Multi-step tool-use training lets an agent interact with tools over a trajectory before receiving a single end reward that drives the policy update.This resembles classic reinforcement learning more than the per-sample RLHF loop.Source: §13.3 · pp. 163–164, Figure 40 · p. 164

13.4 Model Context Protocol

  • MCP [335] standardizes access to external data and actions through servers, clients, and hosts, allowing shared infrastructure and predictable integration across models.Its primitives include read-only resources, templated prompts, and callable tools.Source: §13.4 · p. 164

13.5 Implementation Details

  • Implementations vary in Python or JSON and provider-specific token and API formats; they may preserve reasoning between tool calls, enforce schemas, and manage context-heavy outputs.Tool outputs are masked from training loss because they are produced by the external system, not the model.Source: §13.5 · pp. 165–167
  • Tool-use corpora are often synthetic or bootstrapped; supervised fine-tuning teaches basic formatting and tool selection, preference optimization improves call-versus-answer decisions, and environment-feedback RL trains multi-step task success.Human-written tool traces are expensive to collect.Source: §13.5 · p. 167

14 Over-Optimization

RLHF can over-optimize proxy rewards, pushing language models toward outputs that satisfy reward models or checkers without serving the intended goals. The chapter examines qualitative degradation, proxy mismatch, over-refusal, quantitative limits, and potential misalignment.

14 Over-Optimization

  • RL is a strong optimizer that can push language models to satisfy reward models or checkers while diverging from training goals.Over-optimization more broadly occurs when a training metric is mismatched with final evaluations, reducing alignment with real-world goals or quality.Source: §14 · p. 168

14.1 Qualitative Over-Optimization

  • Qualitative over-optimization can make models feel worse through verbosity, sycophancy, or rigidity, even without measurable reward hacking.The chapter presents this qualitative degradation as a motivation for studying over-optimization further.Source: §14 · p. 168, §14.1 · p. 168

14.1.1 Managing Proxy Objectives

  • Because reward models are only at best correlated with downstream performance, excessive RL optimization can reduce model usefulness [343] even as training reward rises and downstream quality declines.Unlike overfitting, this reflects divergence between the proxy metric and the true goal, not failure to generalize from training data.Source: §14.1.1 · p. 170

14.1.2 Over-Refusal and “Too Much RLHF”

  • Over-refusal examples, including Llama 2 chat models [49], reflected training goals and deployment settings, but the prominence of these behaviors has since decreased.Mitigation commonly involves modifying training data; already-post-trained models can be harder to steer than base models, and safety filters may also shape behavior.Source: §14.1.2 · pp. 171–172

14.2 Quantitative Over-Optimization

  • Over-optimization is fundamental and unavoidable with RLHF’s soft learned reward signal; a higher KL penalty can reduce it at a given KL distance but may require more training steps.Mitigations discussed include larger policy models, reward-model ensembles, and optimizer changes; direct alignment algorithms remain vulnerable.Source: §14.2 · pp. 172, 174

14.3 Misalignment and the Role of RLHF

  • RLHF can promote sycophancy, as when GPT-4o validated a user’s grandiose claim; preference data may reward supportiveness or confidence over accuracy and uncertainty.The potential consequences of this misalignment may grow as language models become more integrated into society.Source: §14.3 · p. 174

15 Regularization

This section examines how regularization and on-policy training relate to post-training policy drift, out-of-distribution performance, and retention. It covers explicit controls such as KL penalties and evidence associating lower drift and better retention with on-policy RL rather than negative gradients.

15.1 KL Divergence in RL Optimization

  • The common RLHF KL penalty is reverse KL to a reference policy, estimated from RL-generated samples and penalizing probability mass placed where the reference assigns little probability.The reference is often the instruction-tuned model, but can also be a previous RL checkpoint.Source: §15.1 · p. 176, §15.1.1 · p. 176

15.1.2 Implementation Example

  • A sample-based KL approximation uses generated tokens’ log probabilities under the current and reference models; summing their difference gives a simple sequence-level estimate.The passage also gives a full KL computation as an alternative.Source: §15.1.2 · pp. 176–177

15.2.1 Pretraining Gradients in RL

  • InstructGPT adds a pretraining next-token prediction reward over corpus documents to RLHF, encouraging the model to remain close to a dataset and maintain textual coherence.This approach was used to address performance regressions on public NLP datasets.Source: §15.2.1 · p. 177

15.2.2 Next-token Accuracy in DPO

  • A length-normalized negative log-likelihood penalty on DPO’s winning completion keeps preferred text likely in absolute language-modeling terms, and the modification can also constrain reward-model training.The standard DPO logistic loss instead increases the margin between winning and losing completions relative to the reference model.Source: §15.2.2 · pp. 177–178

15.2.3 Margin-Based Regularization in Reward Modeling

  • Llama 2 [49] uses a reward-model margin based on annotator rating differences; reward margins also appear in Reward-weighted DPO, RPO [30], and REBEL.The margin can come from numerical ratings or a quantified ranking method such as a Likert scale.Source: §15.2.3 · p. 178

15.3.1 SFT Memorizes, RL Generalizes

  • On V-IRL language-only OOD per-step accuracy, RL rose from 80.8% to 91.8%, while SFT fell from 80.8% to 1.3%, illustrating transferable rules versus memorization.Across task variants, RL’s OOD performance improved as training compute scaled up, while SFT’s degraded despite improving in-distribution.Source: §15.3.1 · p. 179

15.3.2 Retaining by Doing: On-Policy Data Mitigates Forgetting

  • Chen et al. 2025 find RL achieves comparable or higher target-task gains while forgetting less than SFT; forward-KL SFT can disrupt old modes, whereas reverse-KL RL can shift a new mode without disturbing them.SFT minimizes forward KL, while RL minimizes reverse KL; the different optimization dynamics help explain the retention difference.Source: §15.3.2 · p. 179, §15.3.2.2 · p. 182

15.3.3 RL’s Razor: Why Online RL Forgets Less

  • RL’s Razor proposes that on-policy methods favor high-reward solutions that remain closer in KL divergence to the original policy.This offers a complementary perspective on why online RL may generalize and forget less.Source: §15.3.3 · p. 183
  • At matched new-task performance, RL retains more prior-task performance than SFT, consistent with RL favoring KL-minimal solutions among policies that solve the new task.The authors describe forgetting as directly proportional to KL divergence from the initial model.Source: Figure 44 · p. 183, §15.3.3 · p. 183
  • R2 = 0.96: KL divergence between trained and initial policies on new-task data strongly correlates with forgetting on past tasks.This suggests measuring KL on new specialized data can estimate forgetting without evaluating held-out prior-task data.Source: §15.3.3 · p. 183
  • On-policy versus offline data fully accounts for RL–SFT generalization differences, while negative gradients in RL have no discernible effect.On-policy updates use outputs the model already assigns non-negligible probability to, whereas SFT pulls toward a fixed external distribution that may be distant from the model’s outputs.Source: §15.3.3 · pp. 183–184

16 Evaluation

This section surveys language-model evaluation across post-training goals, including how prompts, scoring, and benchmark contamination shape measured performance, how labs use evaluations to guide training, and what open-source evaluation tools are available.

16 Evaluation

  • Early RLHF evaluations centered on chat, expanded to multi-skill suites, and now emphasize difficult reasoning, knowledge, software engineering, and tool-use tasks.Examples range from MT-Bench, AlpacaEval [79], and Arena-Hard to GPQA Diamond [367], SWE-Bench+ [369], LiveCodeBench [370], and recent AIME contests.Source: §16 · p. 185

16.1 Prompting Formatting

  • Prompt formatting can drop model performance from 60% to near 0%; prompting gains are usually smaller than improvements from data or training algorithms, but can matter in the final product.Strong models may be easier to break with a prompt change than to improve slightly.Source: §16.1 · p. 186

16.1.1 Few-Shot Prompting and Log-Likelihood Scoring

  • Few-shot evaluations use exact match on sampled answers or log-likelihood scoring; sampling is random, while token log-probabilities are static, and exact match is standard in post-training [18].Log-likelihood scoring is more common in pretraining; format mismatches can cause scores to plummet, making format-agnostic evaluation desirable but difficult to achieve in practice.Source: §16.1.1 · pp. 187–188

16.1.2 Chain-of-Thought Prompting; 16.1.3 Zero-Shot Instruction Following

  • Few-shot prompting developed to include worked reasoning examples, while FLAN and T0 showed instruction-tuned models could generalize to unseen zero-shot questions.As usage became more open-ended, evaluation increasingly sampled model generations to mirror actual use.Source: §16.1.2 · p. 188, §16.1.3 · pp. 188–189

16.1.4 Reasoning-Era Evaluation Prompts

  • Reasoning-era evaluation shifted toward generation-based exact match with chain-of-thought prompting, often at nonzero temperature, and models no longer needed the canonical “think step by step” prompt [375].Evaluation is moving toward generative responses with chain-of-thought prompting, including special prompts to elicit reasoning and extract answers.Source: §16.1.4 · p. 189

16.2 Why Many External Evaluation Comparisons Are Unreliable

  • 0.25 to 1.5 standard deviations separate many post-training evaluation scores under fixed setups [18], while different prompts, sampling parameters, inference-token budgets, and undocumented configurations undermine external comparisons.Public benchmark saturation also weakens progress signals as scores approach 100%, as illustrated by Figure 45.Source: §16.2 · pp. 189–190, Figure 45 · p. 191

16.3 How Labs Actually Use Evaluations Internally to Improve Models

  • Labs track progress on selected evaluations, often keeping key tests private; internally refined evaluations reduce noise and improve the statistical power of comparisons between training runs.Post-training evaluation also relies on human ratings such as Elo and reward-model agreement, or user A/B tests.Source: §16.3 · p. 192

16.4 Contamination

  • Intentional or accidental use of evaluation-dataset data in training is a major problem in current language-model practices, including RLHF and post-training.Source: §16.4 · p. 192
  • Decontamination searches training and test data for n-gram overlap or fixed-length character matches, while web scraping and user-submitted questions can contaminate training data [380].The passage identifies public benchmark pages and questions passed into models as potential sources of training data.Source: §16.4 · p. 193
  • Tülu 3’s evaluation-suite audit found 8-gram overlaps linking UltraFeedback to TruthfulQA, Evol-CodeAlpaca to HumanEval, NuminaMath to MATH, and WildChat to safety evaluations [6].The overlaps were between training prompts and exact evaluation prompts.Source: §16.4 · p. 193
  • Training on benchmark-like math problems can produce unusual behavior, including gains from random-reward RL, confounding early RLVR results on Qwen 2.5 and Qwen 3 [381].The passage describes this as base-model contamination whose precise cause cannot be proven.Source: §16.4 · p. 193
  • Perturbed-question benchmarks, including versions of MATH, can probe format sensitivity, but high variance does not confirm contamination or establish real-world performance.Models may have been trained to match a specific format that does not transfer beyond it.Source: §16.4 · p. 193

16.5 Tooling

  • Open-source evaluation options include Inspect AI, LightEval, EleutherAI’s harness, OLMES, HELM, and Eval Gauntlet.LightEval powered the Open LLM Leaderboard, and EleutherAI’s harness includes a GPT-3-era evaluation setup [387].Source: §16.5 · p. 193

17 Crafting Model Character and Products

This section examines how post-training shapes model character for product use, including fine-tuned character training, activation-space controls, and documents for expressing intended behavior. It also considers RLHF’s role in connecting product development with model training, and the practical benefits and limits of these interventions, including capability trade-offs and challenges aligning models with human preferences.

17.1 Character Training

  • Character training fine-tunes personality-specific data into a stable base persona [391], using filtered synthetic data to tune language and manner, though its trade-offs and effects on user preferences remain poorly measured.The methods can instill traits beyond positive ones, and industry use often treats character as a way to capture user interests and improve performance.Source: §17.1 · pp. 194–195

17.1.1 Persona Vectors

  • Persona vectors are extracted by contrasting trait-eliciting and trait-suppressing residual-stream activations, then added at inference to amplify (α > 0) or suppress (α < 0) traits.Increasing the steering coefficient can intensify a trait, but its upper limit is unclear and may be non-monotonic; negative coefficients can also counter unwanted shifts introduced by fine-tuning.Source: §17.1.1 · pp. 196–197
  • R2 > 0.94 for nine of ten vectors: steering scores varied nearly linearly with coefficient, while combining inventive and outgoing vectors raised Extraversion by +1.13 and Openness by +0.20.Ten OCEAN vectors were approximately orthogonal, and arbitrary multi-trait profiles can be served from one set of weights without retraining.Source: §17.1.1 · p. 199

17.1.2 The Assistant Axis

  • PC1 across 275+ character archetypes largely aligned with the default Assistant persona, while activation capping substantially reduced harmful drift in emotionally vulnerable conversations.The 25th-percentile activation threshold offered the best reported trade-off between external-benchmark capability and reducing harmful responses.Source: §17.1.2 · pp. 200–202
  • Inference-time activation capping reduces harmful outputs with minimal capability loss by reprojecting drifted activations onto the Assistant Axis, without retraining or weight changes.It addresses slow drift that can accumulate turn by turn in sensitive conversations.Source: §17.1.2 · p. 202

17.1.3 Persona Subnetworks

  • Persona subnetworks enable training-free persona switching by masking sparse, persona-specialized weights, but pruning can affect general capabilities.The method uses hundreds of calibration examples per persona and row-wise top-K pruning; authors prune up to 60% of connections per layer.Source: §17.1.3 · pp. 202–203

17.2 Model Specifications

  • Model specifications make intended behavior comparable with model behavior and clarify priorities for designers, developers, and the public, but their impact depends on how strongly developers enforce them.They can help distinguish intentional behaviors from training side-effects and provide public information relevant to regulatory oversight.Source: §17.2 · pp. 203–204

17.3 Product Cycles and What’s Next for RLHF

  • Product requests are often tested in RLHF because post-training is faster and cheaper; successful features can propagate into earlier training stages.The core trade-offs persist because human preferences cannot be modeled precisely, even as the domains of AI applications change.Source: §17.3 · pp. 204–205

A Definitions … C.4 Identifying Bad Training Jobs

This appendix defines terminology and core concepts used throughout RLHF and examines preference training’s stylistic effects, including its trade-off with broader capability. It also covers practical post-training challenges—costly, noisy recipe development and evaluation, training variability, and systematic checks for broken jobs.

A.1 Language Modeling Overview

  • Autoregressive language models predict each token from prior tokens, typically optimizing training-data likelihood with negative log-likelihood or next-token cross-entropy.Modern models often use decoder-only Transformers with self-attention and an LM head mapping embeddings to vocabulary.Source: §A.1 · p. 228

A.2 Machine Learning

  • The appendix defines KL divergence as a measure of the difference between probability distributions.It introduces the divergence for discrete distributions P and Q over the same probability space.Source: §A.2 · p. 228

A.3 Natural Language Processing

  • NLP terminology distinguishes prompts, completions, chosen and rejected completions, policies, and preference relations used to compare responses.A policy is a probability distribution over possible completions, and a reward model can predict the probability of a preference relation.Source: §A.3 · p. 229

A.4 Reinforcement Learning

  • RL terminology covers actions, states, rewards, value and advantage functions, discounted returns, and trajectories conditioned on a policy.The appendix also distinguishes on-policy data from off-policy data in RLHF and preference-tuning contexts.Source: §A.4 · pp. 229–230

A.5 RLHF-Only

  • The RLHF reference model is a saved parameter set whose outputs regularize optimization.Source: §A.5 · p. 230

A.6 Extended Glossary

  • The extended glossary covers chain-of-thought, in-context learning, synthetic data, and distillation, including teacher-student training from teacher log-probabilities.Teacher-student knowledge distillation modifies the loss to learn from the teacher distribution rather than only a chosen output [276].Source: §A.6 · p. 231

B Beyond “Just Style”

  • RLHF’s stylistic effects can add value: Llama 3’s fun, succinct, clever personality was linked to strong Arena scores [29], and preference tuning can add structure and length.The appendix argues that style is intertwined with information, not merely superficial presentation.Source: B Beyond “Just Style” · pp. 232, 234

B.1 The Chattiness Balance

  • Preference tuning can improve chat evaluations, but excessive chattiness may not transfer to harder evaluations and can harm mathematics or coding performance.The section also warns that narrow-test-set results may be overstated and fail to reflect real-world use.Source: §B.1 · pp. 234–235
  • DPO can improve human preference while degrading benchmark scores, and narrow chat alignment can harm other capabilities; Starling Beta’s longer answers helped raters, while Olmo 3 prioritized math, coding, and reasoning.Starling Beta moved up 10 places in Arena; Olmo 3’s authors preferred a checkpoint with stronger math, coding, and reasoning scores over one best on chat benchmarks.Source: §B.1 · p. 236

C.1 Compute Costs of Post-Training

  • Recipe development can use 10X to 100X the compute of final runs; Tülu 3 required thousands of 7B-scale experiments, and Olmo 3.1 continued RL training for 21 days.Olmo 3’s final runs included an SFT learning-rate sweep and evaluation totaling about two days, DPO sweeps extending over multiple days, and roughly five days for initial RL, with at least a day lost to stability issues.Source: §C.1 · p. 237

C.2 Evaluation Variance

  • Reasoning-model evaluations vary with sampling and infrastructure settings; rerunning noisy, cheap LiveCodeBench evaluations 10 times per model can stabilize them, but increases costs.Variance also depends on benchmark characteristics, batch size, tensor-parallel settings, and sampling numerics for long generations.Source: §C.2 · pp. 237–238

C.3 Managing Training Performance Variance

  • To manage training-performance uncertainty, sweep core hyperparameters, run multiple seeds on promising settings, and consider model merging, though merging best practices remain unsettled.Unlike evaluation noise, which can be reduced by testing a checkpoint more, training uncertainty allows teams to benefit from positive outliers.Source: §C.3 · pp. 238–239

C.4 Identifying Bad Training Jobs

  • Large drops of 10-20 points on relatively stable evaluations such as knowledge-heavy MMLU can signal a badly broken post-training setup.Evaluating many models on a mostly static suite helps teams learn which tests are typically hard to move with post-training interventions.Source: §C.4 · p. 239

Related papers

Loading 2504.12501v11…