Source-linked AI summary
Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, Aaron Courville
TL;DR
Online, on-policy RLHF synchronously generates and learns from the current model’s outputs, making the performant paradigm computationally inefficient; asynchronous RLHF’s off-policy regime remains underexplored. The paper separates generation and training across GPUs, generating samples online while learning asynchronously from samples produced by previous model iterations. Asynchronous RLHF achieves the same performance as synchronous state-of-the-art methods ∼25% faster with 2.8B models, and on math trains 68% faster while matching or exceeding state-of-the-art finetuning numbers.
Problem
Method
The paper separates generation and training across GPUs, generating samples online while learning asynchronously from samples produced by previous model iterations.
Source: §1 · p. 1, §3.1 · p. 4Results
Asynchronous RLHF achieves the same performance as synchronous state-of-the-art methods ∼25% faster with 2.8B models, and on math trains 68% faster while matching or exceeding state-of-the-art finetuning numbers.
Source: §1 · p. 2Takeaways & Limitations
Online DPO is the most robust tested loss to off-policy data, and robustness increases with policy-model size.
Source: §1 · p. 2Naively keeping training and generation model representations on each GPU is infeasible memory-wise, while interleaving backends incurs overhead and reduces available memory.
Source: §3 · p. 3
Abstract
from arXiv · showhide
The dominant paradigm for RLHF is online and on-policy RL: synchronously generating from the large language model (LLM) policy, labelling with a reward model, and learning using feedback on the LLM's own outputs. While performant, this paradigm is computationally inefficient. Inspired by classical deep RL literature, we propose separating generation and learning in RLHF. This enables asynchronous generation of new samples while simultaneously training on old samples, leading to faster training and more compute-optimal scaling. However, asynchronous training relies on an underexplored regime, online but off-policy RLHF: learning on samples from previous iterations of our model which give a worse training signal. We tackle the fundamental challenge in this regime: how much off-policyness can we tolerate for asynchronous training to speed up learning but maintain performance? Among several RLHF algorithms we test, online DPO is found to be most robust to off-policy data, and robustness increases with the scale of the policy model. We study further compute optimizations for asynchronous RLHF but find that they come at a performance cost, giving rise to a trade-off. We verify the scalability of asynchronous RLHF by training a general-purpose chatbot from LLaMA 3.1 8B on an instruction-following task ~40% faster than a synchronous run while matching final performance. Finally, we extend our results to math and reasoning to demonstrate asynchronous RL can finetune Rho 1B on GSM8k ~70% faster while matching synchronous accuracy.
1 INTRODUCTION
The paper proposes asynchronous RLHF, separating generation from learning to reduce compute time while training on previous iterations’ feedback. It evaluates off-policy robustness and reports faster training with matched performance across TLDR, instruction-following, and math tasks.
1 INTRODUCTION
- The paper addresses online RLHF’s computational inefficiency by generating responses online and learning off-policy from previous iterations’ feedback.It presents asynchronous RLHF as an initial step focused on the TLDR benchmark.Source: §1 · pp. 1–2
- Online DPO is the most robust tested RLHF loss to off-policy data, and its robustness increases with policy-model size.The paper also reports that RLHF performance generally degrades as off-policyness increases.Source: §1 · p. 2
- ∼25% faster: at 2.8B, asynchronous RLHF matches synchronous performance, with speed improvements increasing with model scale.On TLDR, asynchronous off-policy RLHF matches synchronous on-policy win-rate across model scales.Source: §1 · p. 2, Figure 1 · p. 2
- Further compute optimizations can reach nearly the same performance ∼250% faster with 2.8B models in the authors’ setup.The paper frames these optimizations as performance/speed trade-offs.Source: §1 · p. 2
- ∼40% faster: asynchronous RLHF achieves equal GPT-4-measured performance to a synchronous approach when training an instruction-following chatbot.This result is reported for LLaMA 3.1 8B finetuned on No Robots.Source: §1 · p. 2
- 68% faster: asynchronous RL on math matches or exceeds state-of-the-art finetuning numbers.The result is reported for finetuning Rho 1B on Grade School Math problems.Source: §1 · p. 2
2 BACKGROUND
The background introduces RLHF’s preference-learning pipeline and the main RLHF algorithms, then connects asynchronous RLHF to prior asynchronous deep RL and separate LLM training and generation systems.
2.1 REINFORCEMENT LEARNING FROM HUMAN FEEDBACK
- RLHF aligns models with human or synthetic preferences through reward-model feedback and RL optimization, while a KL loss helps preserve pretrained abilities and limit reward overoptimization.The standard pipeline gathers preferred and non-preferred responses, learns a reward model, and optimizes the model against its reward.Source: §2.1 · pp. 2–3
- PPO uses an actor-critic framework, RLOO estimates a baseline from multiple samples, and Online DPO ranks two online continuations with a reward model.Online DPO then optimizes the direct preference optimization objective.Source: §2.1 · p. 3
2.2 ASYNCHRONOUS DEEP RL
- Asynchronous deep RL parallelizes actors and learners to improve throughput, but introduces off-policy data from slightly outdated policies.The passage describes asynchronous RLHF as severely under-explored relative to this deep RL work.Source: §2.2 · p. 3
2.3 EFFICIENT LLM TRAINING AND GENERATION
- LLM training and generation use distinct efficiency techniques and increasingly diverged backends, including separate state-of-the-art libraries.Examples include sharding and faster backpropagation for training, versus KV-cache management, continuous batching, and speculative decoding for generation.Source: §2.3 · p. 3
3 ASYNCHRONOUS OFF-POLICY RLHF
The section separates generation and training across GPUs to improve compute efficiency, accepting off-policy learning on samples from earlier policy iterations. It tests how off-policyness affects RLHF methods and applies the findings to asynchronous training.
3 ASYNCHRONOUS OFF-POLICY RLHF
- Synchronous RLHF is inefficient because training libraries generate more slowly than inference libraries; vllm is 12× faster than transformers in the reported 7B test.The comparison generated 1024 batches of 128 tokens.Source: §3 · pp. 3–4
3.1 OFF-POLICY RLHF
- Asynchronous RLHF separates generation and training onto different GPUs, enabling simultaneous work but requiring training on samples from an earlier policy iteration.The proposed approach uses inference libraries for generation and training libraries for learning.Source: §3.1 · p. 4, Figure 2 · p. 4
- The controlled TLDR setup uses a gold reward model to evaluate win-rate and estimates drift from the SFT initialization using perplexity.The setup uses 410m Pythia policies and runs methods for 256 steps with mini-batches of 512.Source: §3.1 · pp. 4–5
3.2 OFF-POLICY WIN-RATE AND KL
- As PPO updates become more off-policy, gold win-rate declines; off-policyness slows progress along the win-rate–KL frontier without changing that frontier.The reported decline is logarithmic, with N = 1 and N = 2 performing similarly.Source: §3.2 · p. 5, Figure 3 · p. 5
3.3 ROBUSTNESS OF RLHF LOSSES TO OFF-POLICYNESS
- Online DPO is more robust to off-policy learning than PPO, RLOO, and Best-of-2 SFT, retaining learning at high off-policyness.The authors suggest its contrastive loss may explain the robustness.Source: §3.3 · pp. 5–6, Figure 4 · p. 6
3.4 SCALING MODEL SIZE WITH OFF-POLICY RLHF
- Larger policy models improve off-policy robustness, whereas larger reward models reduce overoptimization but do not improve robustness.With a 410m reward model, the 2.8B policy's worst point at N = 64 remains close to the optimal area; the 410m reward model achieves the highest win-rate at N = 64 when policy size is held at 410m.Source: §3.4 · pp. 6–7, Figure 5 · p. 6
3.5 SCALING ASYNCHRONOUS OFF-POLICY RLHF
- Across model scales, one-step off-policy asynchronous RLHF matches synchronous RLHF's final win-rate-versus-KL performance and is 25% faster at 2.8B.The setup uses separate GPUs for generation and Online DPO training.Source: §3.5 · p. 7
4 OPTIMIZING ASYNCHRONOUS RLHF
The section addresses idle compute caused by mismatched generation and training speeds by testing extra updates in generation-bound settings and extra sampling in training-bound settings. Both approaches improve speed or data efficiency, with higher KL as a trade-off.
4 OPTIMIZING ASYNCHRONOUS RLHF
- Asynchronous training can leave GPUs idle when generation and training speeds differ, motivating separate strategies for generation-bound and training-bound settings.The naive setup assumes generation and training take approximately similar amounts of time.Source: §4 · p. 7
4.1 GENERATION-BOUND RLHF
- For generation-bound runs, the method performs T = 1, 2, or 3 updates on each generated mini-batch to use extra training compute.The experiment generates N = 1 mini-batch.Source: §4.1 · p. 7, Figure 7 · p. 8
- In generation-bound runs, multiple updates per generated batch improve win-rate per sample at 410m and 1B, but achieve the same win-rate at higher KL.The approach uses extra training GPU cycles for repeated updates on the same batch.Source: §4.1 · pp. 7–8, Figure 7 · p. 8
4.2 TRAINING-BOUND RLHF
- For training-bound runs, the method generates K = 4 completions instead of two and trains on only the highest- and lowest-rewarded completions.The authors halve the learning rate and training steps; the reward margin is approximately 2× larger than with K = 2.Source: §4.2 · p. 8
- In training-bound runs, K = 4 sampling trains 2.5× faster than synchronous training, but achieving the same win-rate comes at higher KL.The difference in KL decreases with scale but remains visible at 2.8B.Source: §4.2 · p. 8
5 SCALING ASYNCHRONOUS RLHF
The section tests asynchronous RLHF on an 8B instruction-following chatbot and on GSM8k reasoning. In both settings, asynchronous training matches synchronous performance while reducing training time.
5.1 GENERAL-PURPOSE CHATBOT
- The chatbot preference dataset used three sampled completions plus a human reference per prompt, with GPT-4o judging all six pairs.The reward model was trained from the LLaMA 3.1 SFT checkpoint.Source: §5.1 · p. 8
- 38% faster training with the same GPT-4o win rate of 57.2% on the No Robots test set was achieved by asynchronous Online DPO on LLaMA 3.1 8B.The table caption reports matching KL and final win rate against human-written responses.Source: Table 1 · p. 9, §5.1 · p. 9
5.2 MATH AND REASONING
- For GSM8k, each question received four sampled completions and a binary exact-match reward, with final accuracy evaluated by pass@1.Training used approximately 128k prompts and 512k episodes.Source: §5.2 · p. 9
- 68% faster training with the same pass@1 and KL was achieved by asynchronous Online DPO on Rho 1B for GSM8k.The synchronous Online DPO baseline slightly improved over the existing synchronous PPO baseline.Source: §5.2 · p. 9, Table 2 · p. 10
- The authors suggest asynchronous RL may be especially suited to reasoning tasks that do not use a reward model.They describe efficiency in this setting as optimizing LLM generation and training.Source: §5.2 · p. 10
6 RELATED WORK
Prior work contrasts efficient offline preference optimization with online methods that rely on a model’s own generations. This paper focuses on online RLHF using off-policy data and examines algorithm and scale effects.
6 RELATED WORK
- Offline preference optimization avoids online generation but has been reported to underperform online methods, which use the model’s own generations to support high reward and retained pretrained capabilities.The cited work includes DPO and its followups (Rafailov et al., 2023; Tang et al., 2024b; Rafailov et al., 2024).Source: §6 · p. 10
- Prior work on off-policy online RLHF was limited, and existing findings suggested that more off-policy data decreases performance.The paper extends this direction by comparing methods and examining scale effects.Source: §6 · p. 10
- Asynchronous RLHF is presented as complementary to engineering approaches that improve synchronous RLHF generation, reward inference, and training.Separating generation and learning may also reduce some synchronous engineering challenges.Source: §6 · p. 10
7 CONCLUSION
The conclusion presents asynchronous RLHF as a viable approach that can improve efficiency while maintaining performance, with speed-performance trade-offs in some optimizations. The paper also provides model-training details and reports released code, checkpoints, and datasets.
7 CONCLUSION
- Asynchronous RLHF is presented as a first step toward improving efficiency while maintaining performance, with further speed-performance trade-offs possible.The authors note synchronous RLHF libraries are currently well-optimized and likely outperform their setup.Source: §7 · p. 10
REPRODUCIBILITY STATEMENT
- The paper reports that its code, baseline checkpoints, and training datasets are open-sourced or released, and that one-step asynchronous RLHF was integrated into Open-Instruct and used for Tulu 3.The reproducibility statement points to the project repository for details.Source: REPRODUCIBILITY STATEMENT · p. 11
A.1 TLDR SUMMARIZATION
- The TLDR experiments use Pythia 410M, 1B, and 2.8B models and relabel completion pairs with a 6.7B reward model before RLHF training.Table 3 reports win-rate and perplexity after supervised finetuning and before RLHF.Source: §A.1 · p. 18, Table 3 · p. 18
A.2 NO ROBOTS INSTRUCTION-FOLLOWING
The appendix describes the No Robots instruction-following setup, reports asynchronous and synchronous Online DPO results, and discusses runtime overheads. It also reports a comparison of asynchronous and synchronous PPO.
A.2 NO ROBOTS INSTRUCTION-FOLLOWING
- The synthetic preference data comes from three sampled demonstrations plus the human reference per prompt, paired and judged by GPT-4o.The reward model is trained from the LLaMA 3.1 SFT checkpoint.Source: §A.2 · pp. 18–19
- The async Online DPO checkpoint achieved the same GPT-4o win rate as the synchronous checkpoints, while Figure 9 reports similar reward and KL during training.Both approaches trained for 100,000 episodes on eight H100s, with one GPU reserved for generation.Source: §A.2 · pp. 19–20, Figure 9 · p. 20
- Runtime reached 151 minutes rather than the estimated ideal 128 minutes because generation and training took longer than their synchronous timings.The authors identify Python thread blocking and communication between generation and training as possible sources of slowdown.Source: §A.2 · p. 20
- Passing updated policy parameters to generation requires a synchronous GPU call that can slow training.The authors suggest reducing synchronization frequency as one possible mitigation.Source: §A.2 · p. 20
- Asynchronous PPO nearly matched synchronous PPO performance while training faster, though both runs showed a KL spike that may reflect PPO instability.The PPO setup lowered the KL coefficient to 0.01; the authors report that PPO performed worse than Online DPO under GPT-4o evaluation.Source: §A.2 · p. 21
A.3 GSM8K
On GSM8k, asynchronous Online DPO reaches nearly the same train score as synchronous Online DPO while training faster. The section also compares Online DPO with RLOO and PPO and explains the training setup and asynchronous timing.
A.3 GSM8K
- Online DPO samples four completions per prompt and trains on the best and worst as its DPO pair, reducing training time by discarding two samples.The authors report that four completions matched eight for RLOO; preliminary RLOO experiments using the best and worst two performed worse.Source: §A.3 · pp. 21–22
- 68% faster than Sync, Async Online DPO reaches a nearly identical train score on GSM8k.Table 11 compares final pass@1, a heuristic KL measure, and training time; its caption says asynchronous Online DPO achieves the same results 68% faster.Source: Figure 11 · p. 22, Table 11 · p. 23
- Online DPO outperforms RLOO and a well-tuned PPO baseline on GSM8k, while VinePPO achieves higher pass@1 with much more compute time.The authors describe Online DPO as a strong baseline, not state-of-the-art for GSM8k.Source: §A.3 · p. 23
- In the synchronous GSM8k setup, generation and training alternate; asynchronous training overlaps them but adds step-time overhead.The reported synchronous step took 25.5 seconds on average; the asynchronous step took 15.1 seconds, including 2.2 seconds of overhead.Source: §A.3 · p. 23, Figure 12 · p. 23
B OFF-POLICY RLOO
The authors propose Proximal RLOO for off-policy learning, using an importance-sampling ratio and PPO-style clipping. In their comparison, it remains robust as data becomes more off-policy, unlike CoPG.
B OFF-POLICY RLOO
- The authors sought an RLOO formulation robust to off-policy data because they found the existing formulation’s claimed robustness unsupported empirically and theoretically.They note that a formulation without a reference to πold does not explain how it handles off-policy data.Source: B OFF-POLICY RLOO · p. 24
- Proximal RLOO incorporates an importance-sampling ratio between the current policy and the policy that generated the data, then clips the ratio within ϵ of 1 for stability.The ratio is rθ(y1) = πθ(y1|x) / πold(y1|x).Source: B OFF-POLICY RLOO · p. 25
- As data becomes more off-policy (N = 16), CoPG performance drops to 0, while Proximal RLOO remains robust.Figure 13 presents the comparison of the two methods.Source: B OFF-POLICY RLOO · p. 25, Figure 13 · p. 25
C.1 TRAINING LIBRARIES ARE INEFFICIENT FOR GENERATION
Training libraries are inefficient for generation, particularly at larger model scales. The paper argues that effective synchronous RLHF must integrate efficient generation libraries.
C.1 TRAINING LIBRARIES ARE INEFFICIENT FOR GENERATION
- Generating with a training library becomes infeasible at larger scales, while vLLM is faster across the model sizes compared.Figure 14 compares generation time for 128 tokens from 512 prompts, each 512 tokens long, across Pythia 410m to 6.7B.Source: §C.1 · p. 25, Figure 14 · p. 26
- The authors argue that synchronous RLHF must integrate efficient generation libraries to avoid relying on inefficient generation from training libraries.They note that vLLM is known to be substantially better than training-oriented alternatives.Source: §C.1 · p. 25
C.2 INTEGRATING GENERATION INTO SYNCHRONOUS RLHF TRAINING IS DIFFICULT
The paper describes the costs and maintenance challenges of integrating generation into synchronous RLHF, notes that such systems can already pipeline some models asynchronously, and gives a Cleanba-style asynchronous algorithm.
C.2 INTEGRATING GENERATION INTO SYNCHRONOUS RLHF TRAINING IS DIFFICULT
- Combining efficient training and generation backends in synchronous RLHF requires converting models between backends and can reduce training memory and increase generation overhead.Keeping the TensorRT-LLM engine in GPU memory leaves less memory for training, while resharding or increased communication adds generation overhead.Source: §C.2 · p. 26
C.3 MAINTAINING GENERATION IN SYNCHRONOUS RLHF IS VERY DIFFICULT
- Keeping synchronous generation integrations current is difficult as backend libraries change, whereas asynchronous RLHF can use generation libraries as stand-alone processes and integrate updates mostly frictionlessly.The paper gives NeMo-Aligner and TensorRT-LLM version updates as a case study.Source: §C.3 · pp. 26–27
C.4 SYNCHRONOUS RLHF IS ALREADY PARTIALLY ASYNCHRONOUS
- Synchronous RLHF systems can already run policy and reference models separately from critic and reward models, with those servers communicating asynchronously for pipelining.A fully asynchronous NeMo-Aligner would add a third server for policy generation.Source: §C.4 · p. 27
D ASYNCHRONOUS ALGORITHM
- The Cleanba-style algorithm sends prior prompts and completions to training while sending current parameters and new prompts to generation, then updates from the previous batch’s rewards.Training and generation run asynchronously, so the update is off-policy.Source: D ASYNCHRONOUS ALGORITHM · p. 27