PaperlayerView exact revision on arXiv

Source-linked AI summary

All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning

Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, J. Andrew Bagnell

arXiv:2503.01067v2cs.LG

TL;DR

The paper examines why complex two-stage online fine-tuning often outperforms direct offline likelihood optimization despite reward-model information loss and no information gain from on-policy sampling. Through theoretical and empirical analysis, it finds the most support for a generation-verification explanation: learning a relatively simple verifier restricts downstream policy search to policies optimal for that verifier.

  • Problem

    The paper asks why two-stage online fine-tuning is often preferred over direct offline likelihood optimization when the goal is to maximize data likelihood.

  • Method

    The paper scrutinizes hypotheses about reinforcement learning’s value in preference fine-tuning using theoretical and empirical analyses of online and offline procedures.

  • Results

    Under idealized assumptions with matched policy and reward-model function classes, online and offline fine-tuning have the same optima, while evidence favors a generation-verification explanation for observed performance differences.

  • Takeaways & Limitations

    When verification is simpler than generation, two-stage online fine-tuning reduces the end-to-end search to policies optimal for relatively simple verifiers.

  • Takeaways & Limitations

    Traditional online-RL arguments are difficult to apply directly to foundation-model post-training because language models cannot obviously recover from mistakes by deleting tokens and reward models may be as large as policies.

Abstract

from arXiv · show

From a first-principles perspective, it may seem odd that the strongest results in foundation model fine-tuning (FT) are achieved via a relatively complex, two-stage training procedure. Specifically, one first trains a reward model (RM) on some dataset (e.g., human preferences) before using it to provide online feedback as part of a downstream reinforcement learning (RL) procedure, rather than directly optimizing the policy parameters on said dataset via offline maximum likelihood estimation. In fact, from an information-theoretic perspective, we can only lose information via passing through a reward model and cannot create any new information via on-policy sampling. To explain this discrepancy, we scrutinize several hypotheses on the value of RL in FT through both theoretical and empirical lenses. Of the hypotheses considered, we find the most support for the explanation that on problems with a generation-verification gap, (1) it is relatively easy to learn the relatively simple RM (verifier) from the preference data. Then, (2) the downstream RL procedure only returns policies (generators) that are optimal for such relatively simple verifiers. Thus, end-to-end, two-stage online FT only has to search over a reduced subset of the full space of policies, requiring less data than offline FT.

1 INTRODUCTION

The paper asks why online, two-stage preference fine-tuning often outperforms direct offline likelihood optimization despite passing through a reward model and gaining no information from on-policy sampling. It studies this question theoretically and empirically, finding support for a generation-verification explanation in which learning a simple verifier reduces the policy search space.

  • Motivation: Offline methods directly optimize policy parameters by regularized maximum likelihood, whereas online methods first fit a reward model and then use it for downstream reinforcement learning.This contrast motivates the paper’s central question about the value of the two-stage procedure.
  • Motivation: Online, two-stage fine-tuning has robustly outperformed simpler offline approaches across academic, industry, and open-source comparisons.The paper also notes that recent complex-reasoning models continue to use on-policy RL rather than offline MLE.
  • Research question: The paper asks what two-stage online fine-tuning contributes when the objective is simply to maximize data likelihood.The question is sharpened by the data processing inequality: passing through a reward model can lose information, while on-policy sampling cannot create it.
  • Evidence: Under idealized assumptions, online and offline preference fine-tuning have equivalent optima when policies and reward models use the same function class, regardless of preference-sample coverage.This result is established using information geometry.
  • Evidence: The paper finds evidence against explanations based solely on regularization, on-policy computational benefits, or broader reward-model training data, while noting these factors are not conclusively ruled out.The authors therefore examine an alternative generation-verification hypothesis.
  • Core explanation: For problems with a generation-verification gap, online fine-tuning can learn a relatively simple reward model and search only among policies optimal for such verifiers.This reduces the end-to-end policy search from the full policy class to a subset of policies associated with relatively simple verifiers.

2 ON THE INFORMATION GEOMETRY OF FT

The paper represents fine-tuning as KL projections through policy and reward-model classes. Under idealized assumptions and equivalent policy and reward classes, online and offline methods reduce to the same likelihood-maximization solution.

  • Policy and reward-model classes: A policy can also define a local reward model through the sum of its token log probabilities, linking policy and reward-model representations.This isomorphism makes offline preference fine-tuning a classification problem over local reward models.
  • A unified objective for fine-tuning: Fine-tuning objectives combine data likelihood with reference-policy regularization, represented by forward and reverse KL terms.The forward KL measures likelihood of preference-data samples under the policy, while reverse KL keeps on-policy completion probabilities near the reference policy.
  • Maximum likelihood in PFT: Offline methods fit policies directly by maximum likelihood, whereas online methods first fit reward models and then optimize policies against learned rewards.The online procedure performs forward-KL maximum likelihood over reward models before a reverse-KL policy projection.
  • Soft RL as projection: Soft reinforcement learning is a reverse-KL projection from the soft-optimal distribution induced by a learned reward onto policy-induced trajectory distributions.This characterizes the second stage of online PFT as projection back into the policy class.
  • Equivalences with isomorphic classes: When reward and policy classes are isomorphic, exact optimization makes online RLHF equivalent to offline MLE, and with reference regularization equivalent to DPO.The equivalence holds regardless of how the same reward functions are represented.
  • Implication: The theory therefore predicts no discernible benefit from on-policy sampling under its assumptions, motivating empirical analysis of why practice differs.The paper explicitly contrasts this idealized conclusion with observed empirical performance.

3 ON THE VALUE OF REINFORCEMENT LEARNING IN FINE-TUNING

Controlled experiments compare online and offline preference fine-tuning while holding major training factors constant. Online methods outperform offline methods despite the theoretical equivalence, motivating tests of alternative explanations for the gap.

  • Experimental setup: The experiments evaluate preference fine-tuning for summarization on tl;dr using Pythia models and GPT-4o winrates against human references.The study reports winrates for trained models evaluated against human-generated summaries.
  • Controlling confounders: Online and offline comparisons use the same DPO loss, preference data, and SFT checkpoint to reduce confounding from objectives and initialization.The main procedural difference is the training data supplied to the final DPO stage.
  • Online DPO: Online DPO samples 25 completions per prompt, ranks them with a reward model, and uses the highest- and lowest-ranked completions for DPO training.The method regularizes to the policy that generated the sampled data.
  • Empirical gap: Online DPO significantly outperforms offline DPO despite matched starting points and equally imperfect optimization conditions.The result contradicts the idealized theoretical equivalence while reducing gradient-step count and optimization quality as explanations.
  • Additional observation: Online DPO can further improve an offline-DPO policy without additional human feedback, suggesting that the reward model extracts more usable structure from the same data.This observation is reported as more performance remaining in reward models than in policies trained on the same human data.
  • Broader pattern: Online PFT’s advantage appears across preference fine-tuning, supervised fine-tuning, and reasoning settings, suggesting the phenomenon is not limited to one PFT benchmark.The section presents this as an analogy across reported prior results.
  • Alternative hypotheses: The appendix tests six hypotheses and finds evidence against explanations based solely on sampling, reference regularization, optimization, data quantity, or OOD generalization.The authors report that online PFT remains stronger across sampling distributions, labelers, and model sizes, while global reward models appear easier to learn than local ones.
  • Scope of the analysis: The paper notes that the DPO loss removes reward-model-output variance as an explanation because methods observe only binarized labels.This scope condition follows from the particular loss choice used in the comparison.

4 GENERATION-VERIFICATION GAPS IN FINE-TUNING

The paper proposes H6: when verification is simpler than generation, online PFT learns a restricted policy class through a reward model and RL, reducing the statistical burden relative to offline FT. Theory and experiments support this generation-verification account, while also showing that online PFT trades additional computation for preference data and may lose its advantage when the gap disappears.

  • H6: Online PFT is Proper Policy Learning: H6 frames online PFT as proper policy learning over Π(Rsim) ⊂ Π, whereas offline FT learns improperly over the full policy space Π.The restricted space contains policies optimal for reward models in Rsim.
  • Theoretical account: Theorem 4.1 states that, when the RL-based reverse KL projection is without loss, RLHF recovers the MLE over the constrained policy space Π(Rsim).The two-stage procedure provides a practical way to enforce this constraint.
  • Empirical evidence: Experiments suggest summarization verifiers are simpler than generators: substantially smaller reward models match policy-sized models on BoN, while larger reward models provide no discernible gain.This supports a generation-verification gap in which the verifier is well-approximated by a shallower circuit.
  • Empirical evidence: H6 correctly predicts the reported experimental pattern, including persistent online-offline gaps under prompt augmentation and altered sample or label distributions.The account attributes this pattern to an unchanged relative complexity of generation versus verification.
  • Scope and unresolved mechanism: The advantage is scope-dependent: tasks without a generation-verification gap show similar data requirements for learning rewards and optimal policies, while the mechanism of on-policy benefit remains unclear informationally.On-policy samples are described as redundant because reward-model labels derive from existing preference data and the base model.
  • Statistical-computational trade-off: Online PFT trades preference data for computation: RL computes a policy from a simpler reward model instead of directly fitting a potentially more complex Q-function.The second stage requires no additional preference samples, but incurs computational cost.

A.4 H4: GLOBAL RMS CAN BE TRAINED ON MORE DATA.

The paper tests whether global reward models outperform local models because they can exploit broader training data, but finds little evidence that this explains the online–offline gap. Narrowing the preference data does not eliminate online DPO’s advantage, and policies and reward models appear similarly able to use wider data.

  • Motivation: Global reward models are trained on broader data distributions than the preference datasets used for offline fine-tuning, motivating H4.The paper asks whether this reflects an intrinsic advantage of global reward models in exploiting diverse data.
  • Test: H4 predicts that concentrating preference data around SFT samples should shrink the online–offline fine-tuning gap.The test uses SFT-generated samples with winners selected by gpt-4o.
  • Results: Online DPO still significantly improves performance over offline DPO on the narrow, on-policy dataset, contrary to H4.The comparison uses the third and first bars in Figure 9, respectively.
  • Experimental control: The experiments use equal amounts of data for local and global reward-model training, despite a potential O(H) computational speedup for global models.Thus, the reported comparison does not test whether global models can process more data under a fixed computation budget.
  • Conclusion: The authors find no evidence that policies and reward models differ in their ability to exploit wider data distributions for the problems studied.They acknowledge that more and more diverse data could still improve reward models.

A.5 H5: GLOBAL RMS GENERALIZE BETTER OOD.

The paper examines whether global reward models generalize better out of distribution than local or DPO reward models. Global models show stronger in-distribution validation likelihood and corresponding BoN performance, but the root cause of this difference remains unresolved.

  • Motivation: Prior work suggests that online fine-tuning benefits when reward-model optima lie outside the preference-data support, implicitly assuming reward models generalize better OOD than policies.The paper notes that reward-model over-optimization and confounded comparisons leave this explanation open.
  • Experimental design: The experiments remove reference-policy regularization by training unregularized local reward models, called Local.This isolates locality from the reference-policy confounder and corresponds to the analyzed MLE policy.
  • Evaluation: Global, local, and DPO reward models are compared using validation likelihood and Best-Of-N performance across SFT and offline DPO samples.Higher N tests reward models on a wider range of generations.
  • In-distribution results: Global reward models perform better in-distribution: token-wise decomposition worsens validation likelihood, and regularization further harms local-model likelihood across model sizes.Figure 10 reports these effects across three seeds.
  • OOD results: Higher in-distribution validation likelihood perfectly correlates with BoN performance for sufficiently large N.This pattern appears when evaluating samples from both SFT and offline DPO policies.
  • Interpretation: The findings do not establish why global reward models have better in-distribution behavior, leaving the explanation for their OOD advantage incomplete.An external global reward model nevertheless ranks learner samples more accurately than the internal DPO reward model across the evaluated settings.

B.1 PROOF OF LEMMA 2.1

Lemma B.1 characterizes minimum-relative-entropy RL as a trajectory-level reverse-KL projection. The soft-optimal policy is obtained by projecting onto a mixture involving the reward model and reference-policy trajectory distributions.

  • Definitions: For distributions P and Q over trajectories, the paper defines their trajectory-level mixture P · Q.This mixture is the object used in the reverse-KL projection formulation.
  • Lemma statement: The lemma defines the optimal policy as the policy minimizing reverse KL to the trajectory-level mixture of the reward-weighted optimal-policy and reference-policy distributions.The notation identifies this policy as the soft-optimal policy computed from the global reward model.
  • Interpretation: The resulting policy is the soft-optimal policy obtained by solving the minimum-relative-entropy RL objective.The construction expresses RL as a projection at the trajectory level.

B.2 PROOF OF THEOREM 2.2

The proof establishes that the offline MLE policy and its corresponding MLE reward model induce the same set of minima, then uses the reverse-KL characterization to show that RLHF has the same optima. Thus, under the theorem’s assumptions, the RLHF and MLE solution sets coincide.

  • Setup: The proof selects specific minima of the MLE policy and reward-model objectives to compare their solution sets.These minima are denoted by ˜πmle and ˜rmle.
  • MLE correspondence: Because the policy and reward model use the same functional minimized over corresponding function classes, their MLE solution sets satisfy ˆrmle = R(ˆπmle).Each MLE policy minimum has a corresponding reward-model minimum with identical values on every trajectory.
  • RLHF characterization: Each RLHF optimum can be rewritten as the minimizer of a reverse-KL projection using the characterization from Lemma 2.1.This connects the theorem’s RLHF objective to the MLE solution set.
  • Conclusion: The proof concludes that the RLHF and MLE policy optima are identical: ˆπrlhf = ˆπmle.The argument repeats the correspondence for every MLE reward-model minimum.

B.3 PROOF OF THEOREM 2.3

The proof establishes that, when reward and policy representations cover the same reward-function class, RLHF and DPO have the same set of optima.

  • B.3 PROOF OF THEOREM 2.3: The proof identifies equivalent MLE reward and policy minima through the mapping R(Π), establishing a correspondence between their solution sets.Each selected policy minimum has a reward minimum with identical values on every trajectory, and vice versa.
  • B.3 PROOF OF THEOREM 2.3: Every DPO optimum corresponds to an MLE policy optimum after substituting its equivalent MLE reward representation.
  • B.3 PROOF OF THEOREM 2.3: Each RLHF optimum can be expressed as a reverse-KL projection using the corresponding MLE reward, allowing the proof to identify it with a DPO optimum.

B.4 PROOF OF THEOREM 4.1

The proof shows that under the theorem’s realizability condition, RLHF optima are contained in the optima obtained from the simpler verifier class, with full equivalence requiring a stronger assumption.

  • B.4 PROOF OF THEOREM 4.1: The proof begins by representing RLHF optima through the relationship established by Lemma 2.1.
  • B.4 PROOF OF THEOREM 4.1: Because the simpler reward class is contained in the full reward class, every RLHF optimum lies in the policy set associated with the simpler verifiers.
  • B.4 PROOF OF THEOREM 4.1: The contradiction argument uses the definition of a BT likelihood maximizer over the restricted policy class to rule out an RLHF optimum outside the simulated optimum set.
  • B.4 PROOF OF THEOREM 4.1: The realizability assumption requires the policy class to contain policies soft-optimal for simple verifiers, but not necessarily policies soft-optimal for every verifier.
  • B.4 PROOF OF THEOREM 4.1: The project uses TL;DR preference data, Pythia models, REBEL code, and VLLM for inference, with code, models, and data planned for release.

C.1 DATASET DETAILS

The dataset details describe the standard summarization setup, including its data splits, prompting procedures, synthetic preference generation, and online-DPO sampling process.

  • C.1 DATASET DETAILS: The standard summarization experiments use specified dataset splits, prompt formats, and maximum generation lengths for TL;DR tasks.
  • C.1 DATASET DETAILS: The workflow first fine-tunes Pythia base models on SFT data, then applies offline or online DPO using preference-training data.
  • C.1 DATASET DETAILS: GPT-4o generates and ranks candidate summaries for the GPT-label training set, while two-word summaries use a five-token limit and a dedicated prompt.
  • C.1 DATASET DETAILS: Online DPO samples 25 completions from an SFT or offline-DPO policy, ranks them with a reward model, and uses the highest- and lowest-ranked outputs as preference pairs.
  • C.1 DATASET DETAILS: The experiments use Pythia models at 1.4B, 2.8B, and 6.9B parameters and convert policy models into global reward models with a linear output layer.

C.3 TRAINING DETAILS

The training details specify optimization, evaluation, prompting, sampling, and likelihood procedures for global and local reward models and DPO variants.

  • C.3 TRAINING DETAILS: The paper trains global reward models with logistic regression and trains policies with SFT and online or offline DPO, using AdamW throughout.
  • C.3 TRAINING DETAILS: Offline DPO uses the SFT policy as reference, whereas online DPO uses the policy that generated its training data and rescales logits by 10.
  • C.3 TRAINING DETAILS: Winrate and ROUGE-L evaluations use temperature 0.01, while best-of-N evaluations use temperature 0.1.
  • C.3 TRAINING DETAILS: Winrate evaluation uses GPT-4o to compare summaries according to accuracy, coverage, coherence, and relevance to important points.
  • C.3 TRAINING DETAILS: The evaluation samples 600 prompts for the reported policy assessments and uses ROUGE-L F1 against preferred validation completions.
  • C.3 TRAINING DETAILS: Likelihood evaluation uses logistic loss for global reward models, β-scaled reference-policy terms for DPO, and β scaling alone for local reward models.
Loading 2503.01067v2…