PaperlayerView exact revision on arXiv

Source-linked AI summary

Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker

arXiv:2402.14740v2cs.LG

TL;DR

PPO is a canonical RLHF approach, but its computational cost and optimization complexity leave open whether simpler RL can preserve performance. The paper revisits preference optimization with REINFORCE and RLOO, reporting that they outperform PPO and that RLOO also beats DPO and RAFT. It concludes that these simpler online RL variants can perform strongly in LLM preference training.

  • Problem

    PPO is widely used for RLHF, but its computational cost and optimization complexity accompany a lack of consensus on the best way to align models with human preferences.

    Source: §1 · p. 2

  • Method

    The paper revisits sequence-level preference optimization with REINFORCE and RLOO, using multiple online samples to form an unbiased baseline for policy-gradient updates.

    Source: §2.3 · p. 6

  • Results

    3.2% to 20.3% win-rate improvement over PPO for Vanilla Policy Gradient REINFORCE across dataset/base-model pairings; RLOO also outperforms PPO, DPO, and RAFT across datasets and models.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The findings support simple REINFORCE-style optimization for LLM preference training, with RLOO combining strong baseline performance and robustness relative to iterative fine-tuning methods such as RAFT.

    Source: §6 · p. 15

    The study does not examine reward-model over-optimization, where proxy-reward optimization diverges from the gold reward objective.

    Source: §7 · p. 15

Abstract

from arXiv · show

AI alignment in the shape of Reinforcement Learning from Human Feedback (RLHF) is increasingly treated as a crucial ingredient for high performance large language models. Proximal Policy Optimization (PPO) has been positioned by recent literature as the canonical method for the RL part of RLHF. However, it involves both high computational cost and sensitive hyperparameter tuning. We posit that most of the motivational principles that led to the development of PPO are less of a practical concern in RLHF and advocate for a less computationally expensive method that preserves and even increases performance. We revisit the formulation of alignment from human preferences in the context of RL. Keeping simplicity as a guiding principle, we show that many components of PPO are unnecessary in an RLHF context and that far simpler REINFORCE-style optimization variants outperform both PPO and newly proposed "RL-free" methods such as DPO and RAFT. Our work suggests that careful adaptation to LLMs alignment characteristics enables benefiting from online RL optimization at low cost.

1 Introduction

The introduction questions PPO’s cost and optimization complexity for RLHF and asks whether simpler online RL can preserve performance. It motivates sequence-level REINFORCE methods and reports stronger results than PPO and other baselines.

1 Introduction

  • PPO’s computational cost and sensitive, complex optimization motivate seeking a simpler RL approach rather than removing RL components as DPO, IPO, and iterative fine-tuning methods do.Source: §1 · p. 2
  • Figure 1 reports that λ = 1.0 Vanilla PG performs best among the displayed λ settings, while removing PPO clipping and loss normalization does not degrade performance.Source: Figure 1 · p. 3
  • Modeling each full generation as one action from the prompt is presented as more appropriate and efficient than modeling partial sequences when rewards apply only to complete generations.Source: §1 · pp. 3–4
  • The paper explores REINFORCE and its multi-sample extension RLOO for sequence-level optimization, contrasting full-trajectory returns and unbiased baselines with PPO’s bootstrapped value estimates.Source: §1 · p. 3
  • Vanilla Policy Gradient REINFORCE beats PPO by 3.2% to 20.3% in win-rate across dataset/base-model pairings, while RLOO outperforms PPO, DPO, and RAFT across datasets and models.Source: §1 · p. 4
  • RLOO is reported as more robust than RAFT to noise and KL-penalty degree, alongside analysis of fluency and diversity.Source: §1 · p. 4

2 Background

The section outlines RLHF’s reward-model pipeline and contrasts token-level PPO with sequence-level REINFORCE methods. It also introduces RLOO and preference-training alternatives including RAFT and DPO.

2 Background

  • RLHF first instruction-tunes a pretrained language model, trains a reward model on preferred versus rejected completions, then optimizes the policy using online reward feedback.The reward model is trained as a binary classifier on preference pairs; the SFT model often initializes both reward model and policy.Source: §2 · p. 4

2.1 PPO

  • PPO treats each generated token as an action and partial sequence as a state, using a clipped token-level objective and an advantage estimate based on GAE.The prompt determines the initial state; only the end-of-sequence token receives the reward-model output, combined with KL penalty.Source: §2.1 · p. 5

2.2 REINFORCE

  • REINFORCE treats the full completion as one action, directly optimizing its sequence-level KL-shaped return with an unbiased Monte Carlo estimator.A moving average of rewards is a simple, parameter-free baseline choice.Source: §2.2 · pp. 5–6

2.3 REINFORCE Leave-One-Out (RLOO)

  • RLOO uses the other k−1 online samples’ rewards as an unbiased baseline for each sample, reducing variance while using all samples in the policy-gradient estimate.This per-prompt, per-step baseline can be more effective than a moving average, at the cost of additional sampling time.Source: §2.3 · p. 6

2.4 Alternatives to RL in Preference Training

  • RAFT ranks k completions and trains on only the top-ranked one, whereas DPO skips reward modeling and directly optimizes the policy from preference pairs.RLOO instead uses all samples to construct a baseline and multi-sample Monte Carlo estimate.Source: §2.4 · p. 7

3 From PPO to REINFORCE

In the tested RLHF setting, Vanilla PG’s unbiased estimator achieves the strongest reward among the tested GAE variants, while removing clipping does not meaningfully harm learning. The section argues that strong pretrained initialization and terminal-only rewards make simpler sequence-level optimization suitable.

3.1 Revisiting the need for low-variance estimators

  • Actor-critic methods use value-function bootstrapping to lower variance, but this introduces bias that risks optimizing biased rewards.The paper notes that prior reports of REINFORCE’s high variance came from random or weak initialization, unlike warm-started pretrained LLMs.Source: §3.1 · p. 7
  • Strong pretrained initialization and prompt conditioning concentrate probability mass on a few tokens, which the authors argue makes high-variance updates less concerning in RLHF.They therefore argue that further variance reduction at the cost of bias is not worthwhile; GAE λ controls that bias-variance trade-off, with λ=1 yielding the return estimator used by Vanilla PG.Source: §3.1 · pp. 7–8
  • Reward decreases monotonically as GAE λ falls, with unbiased Vanilla PG performing best among the tested variants.The authors interpret this as evidence that reducing variance at the cost of bias needlessly introduces bias in this RLHF setting.Source: §3.1 · p. 8

3.2 Clipping is Rarely Necessary in RLHF

  • Clipping is active in fewer than 5% of batch cases on average, and removing clipping does not meaningfully impact learning across the tested dataset and model pairings.Removing clipping and the policy ratio with λ=1 reduces the PPO loss to Vanilla PG and gives a slight performance boost.Source: §3.2 · pp. 8–9

3.3 Modeling Partial Completions is Not Necessary

  • Because the reward is assigned only to the end-of-sequence token, the authors argue that modeling partial completions is unnecessary and that the full generation can be treated as one action.The sequence-level formulation is described as a bandit problem with prompt-defined initial state and a terminal state reached after generation.Source: §3.3 · p. 9

4 Experimental Setup

The experiments use TL;DR Summarize and Anthropic Helpful and Harmless Dialogue, primarily with Pythia-6.9B and also Llama-7B for an Anthropic-HH model-quality ablation. Evaluation measures reward optimization, simulated preference win-rates, and fluency and diversity proxies.

4.1 Training Details

  • Experiments use TL;DR Summarize, with 116k training instructions and 93k preference pairs, and Anthropic-HH, with 112k training preference pairs.For Anthropic-HH, prompts and preferred responses from binary comparisons supply the SFT stage; preference training uses the same prompts.Source: §4.1 · pp. 9–10
  • Pythia-6.9B is the pretrained base model for both datasets, with Llama-7B additionally tested on Anthropic-HH to ablate pretrained-model quality.The policy and reward model are initialized from the corresponding SFT checkpoint unless otherwise noted; context length is 512 tokens for SFT and reward-model training.Source: §4.1 · p. 9

4.2 Evaluation

  • Online methods are evaluated on 1,000 test samples using training-reward averages and GPT-4-proxy simulated win-rates against SFT references for TL;DR or preferred completions for HH.Evaluation uses greedy sampling unless otherwise noted.Source: §4.2 · p. 11
  • Fluency and diversity proxies include perplexity on preferred test completions, average completion length, and average n-gram diversity.These metrics are reported because RLHF fine-tuning is associated with an alignment tax involving reduced diversity and fluency.Source: Alignment Tax · p. 11

5 Results and Discussion

The experiments find that full-generation methods outperform partial-completion methods on reward optimization, while RLOO outperforms RAFT on win-rates and robustness tests. RLOO also maintains fluency and diversity while achieving higher reward scores and win-rates.

5.1 Reward Optimization

  • Methods that model full generations consistently outperform Vanilla PG and PPO, which model each token as an action, on reward optimization.Full-generation methods also require loading one fewer model copy by eliminating the learned baseline or value network used by the token-level methods.Source: §5.1 · p. 11
  • With the same sampling budget, RLOO consistently outperforms RAFT, and RLOO with k = 2 often matches or exceeds RAFT with k = 4.RLOO uses all generated samples, whereas RAFT fine-tunes only on the top-ranked sample; Figure 4 compares rewards by normalized online-sample use.Source: §5.1 · p. 11, Figure 4 · p. 12

5.2 Simulated win-rates

  • RLOO with k = 4 achieves the highest win-rates, outperforming PPO by 10.3, 14.5, and 32.1 for TL;DR, HH (Pythia), and HH (Llama), respectively.Table 1 reports final win-rates on held-out prompts for checkpoints with the highest test reward.Source: Table 1 · p. 12, §5.2 · p. 12
  • Across three dataset–model pairings, RLOO averages win-rates of 61.3 and 61.9 at k = 2 and k = 4, versus RAFT’s 56.1 and 59.5.RLOO’s largest reported gain over RAFT is 9.9 on HH with k = 2 and Pythia-based models.Source: Table 1 · p. 12, §5.2 · p. 13

5.2.1 Alignment Tax

  • RLOO, RAFT, and REINFORCE with baseline achieve lower perplexity than PPO and Vanilla PG, while RLOO and REINFORCE with baseline maintain fluency and diversity alongside higher reward scores and win-rates.Diversity-1 is similar across methods; Diversity-2 tends to decrease slightly for methods with higher reward optimization.Source: §5.2.1 · p. 13, Table 2 · p. 13
  • For the same k, RLOO has slightly lower reward variance than RAFT, while REINFORCE with baseline has 27% less variance than Vanilla PG despite comparable or better reward optimization and win-rates.Vanilla PG has the highest reward variance.Source: §5.2.1 · pp. 13–14

5.2.2 Robustness

  • At higher KL penalties (β = {0.25, 0.5, 1.0}), RAFT optimizes reward less effectively than RLOO and deviates further from the reference policy.At β = 0.1, both methods converge to equal KL distances, but RLOO achieves higher reward.Source: §5.2.2 · p. 14, Figure 5 · p. 14
  • With reward noise σ = {3.0, 5.0}, RAFT’s training reward drops more sharply than RLOO’s, as noise affects the relative rankings of its samples.RLOO maintains relatively robust reward optimization under noisy reward signals.Source: §5.2.2 · p. 15, Figure 6 · p. 15

6 Conclusion

The paper argues that RLHF's strong policy initialization and prompt conditioning make simpler REINFORCE-style optimization viable. Its experiments report that Vanilla PG beats PPO and RLOO outperforms PPO, DPO, and RAFT across datasets and models.

6 Conclusion

  • Vanilla policy-gradient REINFORCE outperforms PPO in the RLHF setting, despite its historical association with high variance in traditional Deep-RL.The paper attributes the setting's reduced variance concern to strong initialization and prompt conditioning.Source: §6 · p. 15
  • The paper argues that strong policy initialization, combined with prompt conditioning, alleviates historical concerns about high variance and large action spaces in RLHF.Source: §6 · p. 15
  • RLOO outperforms PPO, DPO, and RAFT across all datasets and models, while using online samples more effectively than RAFT and showing robustness to noise and KL-penalty degree.The paper characterizes RLOO as a multi-sample extension of REINFORCE.Source: §6 · p. 15

7 Limitations

The paper identifies three limitations: it does not study reward-model over-optimization, does not evaluate LOO in its single-token-action setting, and relies on simulated rather than human win-rates.

7 Limitations

  • Reward-model over-optimization—the divergence of proxy-reward optimization from the gold-reward objective—is not studied and is left for future work.The paper notes that this issue has also not been studied for iterative fine-tuning methods such as RAFT.Source: §7 · p. 15
  • The work does not explore the LOO baseline in a single-token-action framework with partial sequences and intermediary rewards.Its experiments instead treat rewards as applying to full sequences.Source: §7 · p. 16
  • The evaluation uses win-rates simulated by an LLM without measuring their correlation with final human preferences.Source: §7 · p. 16

A Effective Conditioning

The conditioning analysis finds that probability mass concentrates sharply after the first generated token and normalized entropy drops most at that step. The section also relates RLOO's k=2 loss to contrastive loss.

A Effective Conditioning

  • About 60% of probability mass lies on the most probable token at each step, and more than 90% lies in the top 16; the largest concentration jump follows the first token.Figure 7 plots normalized entropy on the left and mass in the top 1, 16, 32, and 64 tokens on the right, relative to a 32k-token vocabulary.Source: A Effective Conditioning · p. 24, Figure 7 · p. 24
  • Normalized entropy drops most after the first generated token, then rises only slightly and remains low through generation.The authors interpret this pattern as evidence that the first conditioning is the most impactful.Source: A Effective Conditioning · p. 24, Figure 7 · p. 24

B RLOO’s Connection to Contrastive Loss

C Training Details

The training details specify dataset preprocessing, SFT and reward-model training, and preference-training configurations for TL;DR and Anthropic-HH. The evaluation prompts ask GPT-4 to compare summaries or chatbot responses and select a preference.

C Training Details

  • Prompts longer than 448 tokens for TL;DR or 348 tokens for HH are filtered to reduce generations without EOS; HH SFT uses preferred comparison responses, while TL;DR uses its dedicated SFT split.Source: C Training Details · p. 25
  • Pythia SFT runs for two epochs at an initial 2e-5 learning rate, while Llama SFT runs for one epoch; reward-model training runs one epoch at 1e-5.SFT and reward-model training use cosine decay with a 0.03 warm-up ratio.Source: C Training Details · p. 25
  • TL;DR preference training runs 600 steps with rollout and step batch sizes of 512 and 256, respectively, using β=0.03.Source: C Training Details · p. 25
  • For Anthropic-HH, Pythia models train for 393 steps using the TL;DR batch-size configuration; Llama models use 2048 rollout and step batch sizes over 2 epochs, with β=0.10 unless noted.Both datasets use SFT-stage prompts for online generations and a constant 1e-6 learning rate with a linear warm-up of 3% of total steps.Source: C Training Details · p. 26

D GPT-4 Evaluation Prompts

  • The TL;DR GPT-4 prompt requests a one-sentence summary comparison followed by a choice; the HH prompt similarly asks which response is more helpful and permits “Neither.”Source: D GPT-4 Evaluation Prompts · p. 26

E.1 TL;DR Summarize (Pythia)

The post asks for an affordable caffeine supplement, describing a switch from frequent Diet Dr. Pepper to mostly water and headaches without morning caffeine. Responses preserve the request but vary in how many details they retain about the gummies and the poster’s circumstances.

E.1 TL;DR Summarize (Pythia)

  • The poster wants caffeine without spending daily calories on coffee, after switching from 5–6 Diet Dr. Peppers a day to mostly water and experiencing headaches without morning caffeine.They report feeling better when drinking water and sometimes having a Diet Dr. Pepper.Source: §E.1 · p. 27
  • The gummies provide 90 mg of caffeine per serving, require four gummies daily, and cost about $20 monthly, but the poster asks whether there is a better option.The original post also says one serving has 20 calories and that the poster has taken them for a few days without headaches.Source: §E.1 · p. 27
  • The generated summaries vary in detail: some retain the gummies’ name, caffeine amount, or price, while others reduce the post to a general request for an alternative.The REINFORCE w/ B. version retains the calorie concern and 90 mg amount, while the PPO version keeps the monthly cost.Source: §E.1 · pp. 27–28

E.2 HH (Llama)

The responses explain pain from stretching too hard mainly as muscle soreness or strain, sometimes mentioning injury. Several recommend starting gently and increasing the stretch gradually, while some add possible causes or circumstances.

E.2 HH (Llama)

  • Responses describe stretching too hard as potentially causing muscle soreness, strain, or injury.Some characterize the strain as minor or soreness as temporary; the PPO response simply mentions soreness or injury.Source: §E.2 · p. 28
  • Other explanations mention muscle strain and inflammation, stress on muscles and connective tissue, or greater risk in fatigued or unaccustomed muscles.The RAFT (k=4) response says injury from stretching too far is possible but less common.Source: §E.2 · p. 28
  • Several responses recommend beginning with a light or gentle stretch and increasing it gradually to avoid overdoing it.The RLOO (k=2) response says soreness may go away as muscles get used to the stretch.Source: §E.2 · p. 28

Related papers

Loading 2402.14740v2…