Source-linked AI summary
OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, Weikai Fang, Xianyu, Yu Cao, Haotian Xu, Yiming Liu
TL;DR
RLHF and RLVR can advance alignment and reasoning, but training faces inference bottlenecks and frameworks can impose complexity barriers. OpenRLHF combines Ray, vLLM, DeepSpeed, and Transformers in an open-source framework; in long-CoT RLVR, it reports 1.22×–1.68× speedups over verl.
Problem
RLHF and RLVR training faces computational challenges, while existing frameworks can impose complexity and accessibility barriers for newcomers and academic researchers.
Source: §1 · pp. 1–2Method
OpenRLHF is an open-source RLHF and RLVR framework built on Ray, vLLM, DeepSpeed, and HuggingFace Transformers, with Ray-coordinated GPU roles and asynchronous engine communication.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 5Results
1.22× to 1.68× speedups: OpenRLHF trains faster than verl across tested long-CoT RLVR configurations and model sizes.
Source: §4.1 · p. 5Takeaways & Limitations
OpenRLHF is publicly released to foster openness and accelerate RLHF and RLVR research, with reported adoption by leading institutions and companies.
Source: §1 · p. 2OpenRLHF currently focuses on language models and does not support Vision-Language Models or other multimodal architectures.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Large Language Models (LLMs) fine-tuned via Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR) significantly improve the alignment of human-AI values, further raising the upper bound of AI capabilities, particularly in reasoning-intensive, long-context Chain-of-Thought (CoT) tasks. However, existing frameworks commonly face challenges such as inference bottlenecks and complexity barriers, which restrict their accessibility to newcomers. To bridge this gap, we introduce \textbf{OpenRLHF}, a user-friendly, scalable, and easy-to-learn open-source RLHF framework built upon Ray, vLLM, DeepSpeed, and HuggingFace Transformers, featuring a simplified design, clear code structure, and comprehensive documentation to facilitate entry for researchers and practitioners. Experimental results show that OpenRLHF achieves superior training efficiency, with speedups ranging from 1.22x to 1.68x across different model sizes, compared to state-of-the-art frameworks. Additionally, it requires significantly fewer lines of code for implementation. OpenRLHF is publicly available at https://github.com/OpenRLHF/OpenRLHF, and has already been adopted by leading institutions to accelerate RLHF research and learning.
1 Introduction
RLHF and RLVR have advanced alignment and reasoning, but their training faces computational demands and frameworks can impose steep learning curves. OpenRLHF is introduced as an open-source framework intended to balance performance, scalability, and accessibility.
1 Introduction
- Inference can account for over 90% of RLHF or RLVR runtime, motivating frameworks that reduce inference overhead and simplify distributed training workflows.The stated challenge is especially associated with PPO workflows that generate thousands of tokens per inference step.Source: §1 · pp. 1–2
- Existing frameworks span accessible open-source systems and optimized industrial solutions, but specialized designs can bring complexity and accessibility barriers for newcomers and academic researchers.The introduction names DeepSpeed-Chat, TRL, ColossalChat, Nemo-aligner, ChatLearn, and verl [12].Source: §1 · p. 2
- OpenRLHF is a simple, high-performance, open-source framework for RLHF and RLVR, built on Ray, vLLM, DeepSpeed, and HuggingFace Transformers.Its contributions include Ray-based orchestration, 3D parallelism, accelerated CoT inference, and asynchronous dataflow.Source: §1 · p. 2
- OpenRLHF has been adopted by institutions and companies including CMU, MIT, Microsoft, and HKUST, and is publicly released to foster openness and accelerate research.The paper presents the framework as suitable for academic experimentation and practical deployment.Source: §1 · p. 2
2 Related Work
Prior work develops RLHF and RLVR methods and specialized frameworks, but the literature describes a persistent tension between optimization and accessibility. OpenRLHF is positioned to address this gap.
2 Related Work
- RLHF trains a reward model from human preferences and optimizes a language model with reinforcement learning, while RLVR uses automatically verifiable signals such as mathematical verification or code execution.PPO remains dominant due to stability, while DPO has gained attention for computational efficiency [15] [18] [19].Source: §2 · pp. 2–3
- RLHF and RLVR require substantial computational resources and distributed training strategies, prompting work on efficient inference, memory-efficient training, and distributed orchestration.The paper cites efficient inference engines [20], memory-efficient strategies [21], and orchestration frameworks [7].Source: §2 · p. 3
- Existing frameworks often trade performance against accessibility, with specialized industrial systems requiring engineering expertise while accessible implementations can lack orchestration or inference optimization.The review also notes that static resource allocation can limit utilization and adaptability.Source: §2 · p. 3
3 Design of OpenRLHF
OpenRLHF separates response generation from training and uses Ray to coordinate distributed RLHF workflows. It combines model-parallel strategies, vLLM inference, and asynchronous engine communication to support scalable training.
3.1 Overview: Ray-based RLHF architecture
- Ray schedules GPU roles that separate response generation in rollout engines from log-probability computation and model training in ZeRO engines.The overview also describes vLLM for low-memory rollout and DeepSpeed for tensor, data, and sequence parallelism.Source: §3.1 · pp. 3–4, Figure 1 · p. 3
- Flexible model slicing, weight transfer, and Ray scheduling support changes between model-parallel modes and asynchronous training, reducing implementation complexity.The workflow instantiates Hugging Face models with DeepSpeed ZeRO, AutoTP, and Ring-Attention, then transfers slices to vLLM through AutoTP and AutoPP.Source: §3.1 · p. 4
3.2 Distributed and Efficient System Design
- DeepSpeed ZeRO can determine and apply tensor-parallel injection policies at runtime for HuggingFace models, avoiding manual per-model configuration.The paper contrasts this with industrial architectures that require users to specify policies for transformer layers and attention outputs.Source: §3.2 · p. 4
- Ring attention distributes long-sequence attention computation across GPUs while minimizing memory use and communication overhead.The paper presents it alongside automatic tensor parallelism and data parallelism for long-CoT training.Source: §3.2 · p. 4
- vLLM supports high-throughput, memory-efficient sample generation and frequent weight updates; PagedAttention reduces memory waste to less than 4%.PagedAttention also supports memory sharing for advanced sampling, reducing memory use by up to 55%.Source: §3.2 · p. 4
- Asynchronous message passing lets rollout, actor, and remote engines process data independently, reducing idle time when CoT generation varies in length and cost.The design is described as supporting dynamic agent RL workflows and extension to scalable agent RL training.Source: §3.2 · p. 5
4 Experiments
Across the reported evaluations, OpenRLHF trains faster than the compared frameworks in long-CoT RLVR, GSM8K RLVR, and PPO RLHF settings. The paper also compares codebase size and functionality as aspects of usability.
4.1 Performance Comparison
- 1.22× to 1.68× speedups: OpenRLHF trains faster than verl across all tested long-CoT RLVR configurations.The evaluation uses 1.5B, 7B, and 14B models with generation lengths from 1K to 8K tokens; speedups become more pronounced with larger models and longer contexts.Source: §4.1 · p. 5
- 3.1× speedup: OpenRLHF completes one GSM8K GRPO training epoch in 1,657 seconds, versus 5,189 seconds for optimized TRL.The comparison uses identical hyperparameters and hardware.Source: §4.1 · p. 6
- 3.6× speedup: OpenRLHF completes a 1,024-prompt PPO epoch in 236.8 seconds, versus 855.09 seconds for DeepSpeed-Chat.The paper attributes the gain primarily to vLLM token generation and Ray distributed execution.Source: §4.1 · p. 6
4.2 Usability Comparison
- 8,523 lines of code make OpenRLHF the second-most concise framework in the comparison, below TRL at 19,071 and verl at 32,325.The paper describes support for SFT, DPO, reward models, and process reward models alongside its modular, documented architecture.Source: §4.2 · p. 6, Figure 2 · p. 6
5 Limitations
OpenRLHF has limitations in performance, resourcing, model scope, and reliance on upstream systems.
5 Limitations
- The framework may not match highly specialized industrial systems that benefit from dedicated engineering teams and extensive resources.As a community-driven open-source project without dedicated economic support, it may also be slower to integrate cutting-edge features.Source: §5 · p. 7
- OpenRLHF focuses primarily on language models and does not support vision-language or other multimodal architectures.This limits its applicability to multimodal AI alignment research.Source: §5 · p. 7
- Its modular design depends on Ray, vLLM, and DeepSpeed, whose updates may require maintenance or introduce compatibility issues.Source: §5 · p. 7
6 Conclusion
OpenRLHF combines distributed computing, inference optimization, memory efficiency, sequence parallelism, and asynchronous dataflow in an open-source RLHF and RLVR framework. The conclusion describes its adoption and influence as evidence of practical reach and a contribution to broader participation in RLHF research.
6 Conclusion
- OpenRLHF combines Ray, vLLM, DeepSpeed ZeRO, and ring attention with a Ray-based architecture, 3D parallelism, accelerated CoT inference, and asynchronous dataflow.Source: §6 · p. 7
- The framework’s adoption across academic courses and major technology companies includes research use and production deployments.Source: §6 · p. 7
- The conclusion presents OpenRLHF’s influence on later frameworks and its role in democratizing RLHF research as contributions to the field.Source: §6 · p. 7
A Full Contributors
The contributor list credits individuals across integrations, core training implementations, and supporting work including documentation, testing, and writing.
A Full Contributors
- Ray integration is credited to Jian Hu, Xibin Wu, and Songlin Jiang; vLLM integration also credits Kaichao You of the vLLM Team.Source: A Full Contributors · p. 10
- DeepSpeed integration credits Jian Hu, Xibin Wu, Songlin Jiang, Bin Chen, and Yiming Liu, while ring attention credits Zilin Zhu, Zhibo Zhou, gzpan, and Jian Hu.Source: A Full Contributors · p. 10
- The list separately credits PPO, GRPO, KL control, RLVR and RLHF experiment design, and dynamic sampling contributors.It also names contributors to asynchronous agentic RL, single-controller architecture, and dynamic batching.Source: A Full Contributors · p. 10
- Documentation is credited to Jian Hu; testing and bug reports to Weikai Fang, Wei Shen, and Weixun Wang; and writing and presentations to Wei Shen, Jason Klein Liu, Weixun Wang, Yu Cao, and Jian Hu.Source: A Full Contributors · p. 10
B Broad Impact and Adoption
OpenRLHF is described as adopted across academic and industrial settings, used as a foundation for specialized frameworks, and influential in later framework development. The section also attributes broader participation in RLHF research to its accessibility.
B Broad Impact and Adoption
- The CMU Advanced Natural Language Processing course adopted OpenRLHF as a core teaching framework in Spring 2025.The passage describes this as demonstrating educational value and accessibility for students new to RLHF.Source: B Broad Impact and Adoption · p. 10
- OpenRLHF has been adopted by technology companies and research institutions, with reported use spanning research prototyping to production-scale RLHF workloads.The listed adopters include Google, ByteDance, Baidu, NVIDIA, Tencent, China Telecom, Vivo, UC Berkeley’s Starling Team, Meituan, and HKUST.Source: B Broad Impact and Adoption · p. 10
- LMM-R1, MARTI, and MM-EUREKA are presented as specialized frameworks built on OpenRLHF’s modular, extensible architecture.Their applications include multimodal reinforcement learning, advanced reasoning, and multimodal applications.Source: B Broad Impact and Adoption · p. 11
- The passage reports that verl, Alibaba ROLL, SLIME, and Open-Reasoner-Zero acknowledged OpenRLHF’s design contributions in documentation or publications.The acknowledged contributions include distributed architecture, Ray-based orchestration, and integration strategies.Source: B Broad Impact and Adoption · p. 11
- OpenRLHF is described as lowering entry barriers for academic researchers and smaller organizations, broadening participation in RLHF research.The passage connects this accessibility with participation by communities that might lack resources for complex distributed training setups.Source: B Broad Impact and Adoption · p. 11
C PPO Workflow Design
OpenRLHF’s PPO workflow iteratively generates rollouts, computes rewards and advantages, and optimizes the policy. Its modular engines coordinate these stages while supporting stable, customizable training.
C PPO Workflow Design
- The PPO workflow iterates through rollout generation, reward computation, GAE-based advantage estimation, and policy optimization until convergence.Each epoch processes multiple batches of rollout data.Source: C PPO Workflow Design · pp. 11–12
- The Rollout Engine uses vLLM to generate responses from sampled prompts and records action log-probabilities and attention masks for subsequent processing.Continuous batching, KV-cache management, and PagedAttention support throughput during inference.Source: C PPO Workflow Design · pp. 11–12
- The reward model scores each prompt-response pair, while the frozen reference policy supplies log-probabilities and the critic estimates values for advantage computation.These computations can be parallelized across GPUs and batched together.Source: C PPO Workflow Design · p. 12
- GAE derives advantages from temporal-difference residuals and computes discounted returns; KL penalties adjust rewards to limit policy deviation from the reference.The adjusted reward is r′_t = r_t − βKL[π_θ||π_ref].Source: C PPO Workflow Design · p. 12
- The ZeRO Engine updates the policy with PPO’s clipped objective and combines policy and critic losses with an entropy bonus.Early stopping based on KL-divergence thresholds and performance metrics supports stable training, while modularity permits techniques such as gradient clipping and advantage normalization.Source: C PPO Workflow Design · p. 12
- Ray coordinates the engines across potentially separate inference- and training-optimized GPU clusters, overlapping stages asynchronously to improve throughput.Ray scheduling handles data transfer, synchronization, and fault tolerance.Source: C PPO Workflow Design · p. 12