Source-linked AI summary
Secrets of RLHF in Large Language Models Part I: PPO
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang
TL;DR
LLM alignment is difficult because RLHF requires reliable rewards and stable PPO training, while harmful behaviors remain possible after pre-training and SFT. The report dissects RLHF, evaluates PPO components, and proposes PPO-max with released reward models and code; it identifies policy constraints as central to stable alignment training. The authors also evaluate alignment against SFT models and ChatGPT, while noting limitations in reward-model data and training-phase performance indicators.
Problem
LLMs can produce fabricated, biased, toxic, or harmful content, while RLHF training remains difficult because reward design and PPO optimization are unstable and costly.
Method
The report dissects RLHF and PPO, evaluates reward-model quality and implementation choices, proposes PPO-max, and releases reward models and complete PPO-max code.
Results
The report identifies policy constraints as a key factor in consistent alignment and evaluates PPO-max on 7B and 13B SFT models with alignment performance comparable to ChatGPT.
Takeaways & Limitations
Policy constraints are presented as central to stable PPO-based alignment, with released models and code intended to support further LLM alignment research.
Takeaways & Limitations
The reward-model evaluation uses openly available English preference data and a small amount of self-constructed Chinese data that may be insufficient for comprehensive evaluation.
Abstract
from arXiv · showhide
Large language models (LLMs) have formulated a blueprint for the advancement of artificial general intelligence. Its primary objective is to function as a human-centric (helpful, honest, and harmless) assistant. Alignment with humans assumes paramount significance, and reinforcement learning with human feedback (RLHF) emerges as the pivotal technological paradigm underpinning this pursuit. Current technical routes usually include \textbf{reward models} to measure human preferences, \textbf{Proximal Policy Optimization} (PPO) to optimize policy model outputs, and \textbf{process supervision} to improve step-by-step reasoning capabilities. However, due to the challenges of reward design, environment interaction, and agent training, coupled with huge trial and error cost of large language models, there is a significant barrier for AI researchers to motivate the development of technical alignment and safe landing of LLMs. The stable training of RLHF has still been a puzzle. In the first report, we dissect the framework of RLHF, re-evaluate the inner workings of PPO, and explore how the parts comprising PPO algorithms impact policy agent training. We identify policy constraints being the key factor for the effective implementation of the PPO algorithm. Therefore, we explore the PPO-max, an advanced version of PPO algorithm, to efficiently improve the training stability of the policy model. Based on our main results, we perform a comprehensive analysis of RLHF abilities compared with SFT models and ChatGPT. The absence of open-source implementations has posed significant challenges to the investigation of LLMs alignment. Therefore, we are eager to release technical reports, reward models and PPO codes, aiming to make modest contributions to the advancement of LLMs.
1 Introduction
LLMs can exhibit unintended harmful or biased behaviors, while SFT alone remains below human safety and groundedness levels. The report analyzes RLHF and releases models, PPO-max, and code to improve alignment research.
- LLMs trained on mixed-quality pre-training data may generate fabricated, biased, toxic, or harmful content.
- SFT adds helpful, honest, and harmless data, but model safety and groundedness remain below human levels.
- RLHF training is difficult because reward quality, PPO stability, four-model coordination, sparse rewards, exploration, and hyperparameters interact.
- The report finds reward-model quality sets the policy model’s upper bound and identifies PPO policy constraints as central to consistent alignment.
- The authors release Chinese and English reward models, PPO-max code, and technical analyses to reduce alignment research costs.
2 Related Work
Prior work emphasizes RLHF objectives, PPO implementation details, and low-level design choices, but PPO instability and hyperparameter sensitivity remain insufficiently addressed. This paper targets those unresolved training-stability problems.
- RLHF seeks helpful, honest, and harmless LLMs aligned with human values, while related systems also use supervised techniques for preference alignment.
- Prior studies show that seemingly small PPO code modifications and overlooked low- and high-level design decisions can substantially affect reinforcement-learning performance.
- Few previous efforts directly address PPO instability and hyperparameter sensitivity in RLHF language-model training.
- The paper dissects RLHF and explores an advanced PPO version intended to improve policy-model training stability.
3 Reinforcement Learning from Human Feedback
RLHF combines SFT, reward-model training, and PPO, using human preferences to update a dialogue policy. The section explains reward modeling, return and advantage estimation, and PPO’s policy constraints for stable updates.
- RLHF training comprises SFT dialogue imitation, reward-model preference comparison, and PPO policy updates driven by reward-model feedback.
- 3.1 Reward Modeling: A reward model assigns scalar rewards to responses, with larger values indicating better samples, and is trained on paired human-preference comparisons.
- 3.1 Reward Modeling: The reward-model objective combines preference discrimination with imitation learning on preferred responses.
- 3.2 Reinforcement Learning: KL regularization keeps the RL policy near the SFT policy, supporting exploration while limiting drastic deviation from reward-model training data.
- 3.2.2 Generalized Advantage Estimation: GAE balances bias and variance by interpolating between one-step TD estimates and full Monte Carlo returns.
- 3.2.2 Generalized Advantage Estimation: Small k-step advantages have higher bias, whereas large k-step advantages can have higher variance from accumulated noisy rewards.
- 3.2.3 Proximal Policy Optimization: PPO-Clip constrains policy changes by clipping the new-to-old action-probability ratio, preventing overly large updates while retaining sample efficiency over vanilla policy gradients.
4 Reward Modeling for Helpfulness and Harmlessness
The reward model (RM) is trained to emulate human evaluations for helpfulness and harmlessness, using English HH-RLHF data and manually labeled Chinese data. Evaluation shows partial human-preference alignment, but observed reward errors and training behavior indicate that accuracy alone is insufficient.
- Reward-model motivation: The RM emulates human evaluation because collecting feedback before every optimization iteration is impractical.
- Data and models: English experiments use LLaMA-7B with 160k HH-RLHF pairwise samples, while Chinese experiments use OpenChineseLLaMA with manually labeled pairwise data.The English dataset contains 118k helpful and 42k harmless instances; the Chinese dataset includes 31k helpful and 8k harmless samples.
- Evaluation: Chinese and English RMs are evaluated by comparing reward differences between preferred and dispreferred responses on held-out test samples.The evaluation uses approximately 0.9k English HH-RLHF samples and 3k Chinese annotator-labeled samples.
- Evaluation: Both RMs align to some degree with human preferences, with the Chinese RM showing substantial consistency with human judgments.
- Observed failure cases: The RMs can reward longer but factually false Chinese responses and penalize honest English responses that acknowledge missing information.They may also assign high rewards to responses containing deceptive or misleading information when those responses appear helpful.
- Training performance: RM accuracy improves slowly after 200 steps, yet initializing PPO from the 200-step model performs unsatisfactorily, showing accuracy alone is insufficient.The 200-step accuracy is comparable to accuracy after a complete epoch.
5 Exploration of PPO
The report explores why vanilla PPO becomes unstable in RLHF and evaluates implementation choices that constrain policy optimization. It identifies policy constraints, especially KL or entropy penalties, as central to long-lasting stability while examining score transformations and model initialization.
- Pattern Collapse: PPO training can suffer pattern collapse, in which over-optimized policies exploit fixed generative patterns for anomalously high reward scores.Higher rewards and stable training loss may not correspond to better behavior under human or GPT-4 evaluation.
- Evaluation Metrics: Perplexity, policy-reference KL divergence, and response length provide more informative monitoring signals than reward and loss alone.Over-optimization can push responses into the preference model’s out-of-distribution region.
- PPO-max: PPO-max consolidates implementation strategies intended to support stable policy optimization, including score reparameterization, policy constraints, and initialization choices.The report evaluates these choices because traditional RL settings differ from RLHF language-model training.
- Score Reparameterization: Reward scaling does not guide proper policy optimization, while strict advantage cropping can maintain stability within a fixed epoch.The authors recommend constraining instability at the reward level because reward and advantage reparameterizations can have similar effects.
- Policy Constraints: Policy divergence penalties are critical for stable RLHF optimization, and KL or entropy penalties provide longer-lasting stability than other tested modifications.With policy divergence penalty, RLHF can improve response quality while barely modifying language modeling, exhibiting almost zero KL divergence from the original policy.
- Initialization: Critic pre-training can improve stability through better advantage estimation, whereas starting PPO directly from a pre-trained language model severely reduces language-modeling ability.Initializing the critic with reward or SFT models converges similarly, while a qualified dialogue model is essential for policy optimization.
6 Evaluations and Discussions
The evaluations compare PPO-max RLHF models with SFT models and ChatGPT using human and GPT-4 preferences, while also testing language understanding after PPO. RLHF improves helpfulness and harmlessness relative to SFT, reduces defeats against ChatGPT, and PPO-ptx mitigates PPO-related NLU decline.
- Alignment Metrics and Experiment Setups: The evaluation defines helpfulness through instruction following and relies on annotator preference ratings because prompt intentions can be ambiguous.Responses were generated for held-out prompts and compared by human annotators and GPT-4.
- Preference Comparison between RLHF models and SFT models: RLHF-trained models received stronger human preferences than SFT models across Chinese and English question types.On the English held-out datasets, RLHF received 62% versus 5% for SFT on Harmless and 44% versus 30% on Helpful.
- Preference Comparison between RLHF models and SFT models: GPT-4 evaluations closely mirrored human evaluations, including RLHF’s advantage over SFT on harmful prompts.GPT-4 produced more tie votes than human evaluators in the Harmless evaluation.
- Our Models vs. ChatGPT on Harmless Evaluation: RLHF reduced defeat rates against ChatGPT from 45% to 24% for English models and from 37% to 29% for Chinese models.The RLHF models still lagged behind ChatGPT but competed on par on some prompts where SFT models had failed.
- NLU Evaluation: PPO decreased natural language understanding capabilities, while PPO-ptx effectively alleviated that decline by incorporating pre-training data during PPO training.The evaluation used C-Eval, a Chinese suite with approximately 13k questions across 52 disciplines and four difficulty levels.
Limitations
The paper identifies limitations in scale, reward-model data, evaluation coverage, and training indicators. It focuses on 7-billion-parameter models, uses limited preference data, omits many benchmarks, and finds that stability and reward scores do not guarantee or predict final performance.
- Scope: The authors frame this work as a first step constrained by time and resources.
- Scaling Law: The study focuses primarily on a 7-billion-parameter model and does not investigate how model size or data scale affects RLHF performance.
- Reward Model: The reward-model experiments use openly available English preference data and a small amount of self-constructed Chinese data, limiting comprehensive evaluation.
- Evaluation Metric: The evaluation relies largely on manual and GPT-4 assessments rather than detailed testing across numerous available benchmarks and NLP tasks.
- Performance Indicator: PPO-phase stability does not necessarily improve final outcomes, and reward scores cannot reliably predict RLHF performance during training.The paper calls for a more suitable training-phase performance indicator.
A Reward Distribution under PPO Training
Reward-model scores remain identically distributed during stable PPO training but develop long tails after pattern collapse. The authors argue that data-specific reward ceilings mean optimal results may occur before collapse.
- Reward Distribution: Reward distributions remain identical during stable training and exhibit long-tail characteristics after pattern collapse.
- Reward Distribution: Different data may have different upper reward bounds, so the best PPO results should appear before the stage at which collapse occurs.
B Supplementary Experiments on Hyperparameter Tuning
Supplementary experiments find strong correlations between hyperparameter choices and training results, with some stabilization methods requiring extensive experimentation and precise control.
- Hyperparameter Sensitivity: Hyperparameter choices show a rich correlation with training results.
- Hyperparameter Sensitivity: Some methods require extensive experimentation and precise control to achieve stable optimization.The examples include the clipping range for the entropy bonus.
B.1 Collaborative Analysis on Rewards, Advantages, and Value Loss
The ablations show that normalization and clipping choices can conflict during policy optimization, while KL-penalty weight produces a clear reward–policy-deviation hierarchy. Importance sampling removes initial fluctuations by aligning responses with the current policy distribution.
- Normalization and clipping operations on the advantage and value function conflict during policy optimization.
- Reward scaling with value clipping, or normalizing and clipping only rewards and advantages, are configurations that can converge.
- Increasing KL-penalty weight imposes a tighter constraint, lowering reward responses while reducing deviation from the original policy distribution.
- Importance sampling eliminates initial training fluctuations by aligning responses with the current policy distribution.
B.3 Clip Region for Entropy Bonus
The entropy bonus stabilizes training but is sensitive to clipping, so its configuration matters even when the underlying PPO setup normally converges.
- The entropy bonus has a stabilizing effect on PPO training.
- Clipping the entropy bonus changes training behavior when it is added to an otherwise normally convergent PPO configuration.The entropy-loss learning rate is set to 0.01 in all experiments.
- Because the entropy bonus is implemented as a negative loss term, optimization drives it toward a larger value.
C Comparison Results on Secondary Tricks
The report treats several PPO implementation adjustments as secondary, while finding that clipped-surrogate PPO does not provide the stable optimization achieved by a KL constraint.
- Secondary PPO adjustments: The comparison examines the clipped surrogate objective, global gradient clipping, and Generalized Advantage Estimation parameter tuning as secondary PPO adjustments.GAE reduces to the traditional TD error when λ = 0.
- Clipped surrogate objective: The clipped surrogate objective reduces complexity and KL-divergence estimation error, making PPO resemble TRPO and commonly identifying it as PPO2.
- Clipped surrogate objective: Different clipping values have little effect on results and do not provide optimization stability comparable to a KL constraint.
C.2 Global Gradient Clip
Global gradient clipping is intended to reduce data-noise effects, but experiments find its constraint settings difficult to distinguish during PPO training. GAE trades estimation variance against long-range dependence error and training stability.
- Global Gradient Clip: Global gradient clipping is commonly enabled in PPO implementations to reduce the impact of data noise on training.
- Global Gradient Clip: Experiments find it difficult to distinguish PPO training differences across global gradient constraints.The strategy is enabled by default in PPO-max.
- GAE: GAE applies reward shaping to estimate a more instructive value function while balancing estimation precision and variance.
- GAE: A smaller λ reduces sequence-estimation variance but increases error on long-range dependence.
- GAE: TD estimation with λ = 0 provides smaller variance but is numerically more unstable during training.