Source-linked AI summary
AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, Yi Wu
TL;DR
Large-scale RL for language reasoning needs massive rollout parallelism, while synchronous systems waste computation waiting for variable-length outputs. AReaL decouples generation and training with asynchronous workers, staleness control, and a modified PPO objective. Experiments show faster training with matched or improved performance, though throughput gains can diminish for shorter contexts or overly restrictive staleness limits.
Problem
Large-scale LLM RL requires massive parallelization, but synchronous systems wait for the longest rollout and underutilize GPUs.
Method
AReaL asynchronously decouples rollout generation from training, controls data staleness, and uses a staleness-enhanced PPO objective with system-level optimizations.
Results
AReaL delivers higher throughput and improved or matched accuracy than synchronous systems across mathematical reasoning and code generation tasks.
Takeaways & Limitations
AReaL provides a practical foundation for efficient, stable, and scalable asynchronous RL training for language reasoning models.
Takeaways & Limitations
Throughput gains can be smaller for shorter contexts when generation cannot match training throughput, and very small staleness limits can slow generation around long trajectories.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has become a dominant paradigm for training large language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs requires massive parallelization and poses an urgent need for efficient training systems. Most existing large-scale RL systems for LLMs are synchronous, alternating generation and training in a batch setting where rollouts in each training batch are generated by the same model. This approach stabilizes RL training but suffers from severe system-level inefficiency: generation must wait until the longest output in the batch is completed before model updates, resulting in GPU underutilization. We present AReaL, a fully asynchronous RL system that completely decouples generation from training. Rollout workers in AReaL continuously generate new outputs without waiting, while training workers update the model whenever a batch of data is collected. AReaL also incorporates a collection of system-level optimizations, leading to substantially higher GPU utilization. To stabilize RL training, AReaL balances the workload of rollout and training workers to control data staleness, and adopts a staleness-enhanced PPO variant to better handle outdated training samples. Extensive experiments on math and code reasoning benchmarks show that AReaL achieves up to 2.77$\times$ training speedup compared to synchronous systems with the same number of GPUs and matched or improved final performance. The code of AReaL is available at https://github.com/inclusionAI/AReaL/.
1 Introduction
Large-scale RL for language reasoning requires massive parallelization, but synchronous systems waste computation by waiting for variable-length rollouts. AReaL decouples generation and training asynchronously, improving throughput while maintaining or improving accuracy.
- Motivation: RL training for language reasoning requires thousands of rollout outputs and must handle tens of thousands of thinking tokens per prompt.These demands create an urgent need for efficient large-scale training systems.
- Motivation: Synchronous RL waits for the longest output in each batch, causing severe inefficiency as generation and training workloads vary.The changing output lengths of reasoning models produce idle time and wasted computation.
- Approach: AReaL fully decouples generation from training, allowing rollout workers to stream outputs while trainers update whenever a batch is available.The system synchronizes updated weights and uses staleness-aware mechanisms for samples from older policy versions.
- Results: Up to 2.57× higher training throughput and linear scaling efficiency up to 512 GPUs were achieved versus state-of-the-art synchronous systems.The evaluation covered mathematical reasoning and code generation with models up to 32B parameters, alongside improved solution accuracy.
2 Related Work
Prior work established asynchronous RL mainly in games or short-context and limited-overlap LLM settings. AReaL extends this direction with a more flexible staleness–speed trade-off and an algorithm compatible with interruptible generation.
- Asynchronous RL: Asynchronous RL architectures and algorithms have achieved notable success in game applications.
- Asynchronous RL: Earlier asynchronous LLM approaches typically targeted short-context settings or only one- or two-step generation–training overlap.
- Asynchronous RL: AReaL extends prior work by providing a more flexible trade-off between data staleness and training speed.
- Novelty: AReaL dynamically interrupts generation while buffering unfinished sequences, preserving consistent PPO batch sizes without a fixed length budget.Its asynchronous algorithm tolerates higher data staleness and remains compatible with interruptible generation.
3 Background
LLM RL treats token generation as sequential decision-making and commonly optimizes a final-action reward with PPO. Synchronous execution creates underutilization and poor scaling because all devices alternate between generation and training.
- RL Formulation and PPO: In the MDP formulation, each action is a vocabulary token, while states contain the question and previously generated tokens.
- RL Formulation and PPO: PPO uses an importance ratio between current and old policies together with an estimated advantage to constrain policy updates.
- RL Formulation and PPO: The reward is rule-based, nonzero only at the final action for answer correctness, with γ = 1.
- Synchronous RL Limitations: Synchronous RL alternates generation and training, waiting for the longest sequence and underutilizing inference devices because decoding lengths differ across GPUs.
- Synchronous RL Limitations: Synchronous systems scale poorly because distributing generation across devices reduces per-GPU decoding batch size and creates a memory-IO-bound regime.Additional devices therefore fail to improve throughput effectively.
4 System Architecture
AReaL separates rollout and training across GPU clusters, connecting workers through reward evaluation, replay buffering, and weight updates. Its asynchronous pipeline improves utilization but requires controls for stale and mixed-version trajectories.
- System Overview: AReaL separates generation and training across GPU clusters to improve hardware efficiency, scalability, and workflow flexibility.
- System Overview: The architecture includes rollout workers, a reward service, trainer workers, and a rollout controller coordinating data and parameter flow.
- Rollout Management: Interruptible rollout workers load new weights, discard old KV caches, recompute them, and continue unfinished sequences.Interruptions can create trajectories composed of segments from different model versions.
- Data and Parameter Flow: The controller sends generated responses for reward evaluation, stores rewarded trajectories in replay, and triggers rollout weight updates after training.This pipeline is designed to keep generation and training resources continuously utilized.
- Algorithmic Challenges: Asynchronous batches contain data from multiple policy versions, creating distribution gaps that can degrade learning, especially for long trajectories.
- Algorithmic Challenges: Mixed-version trajectories violate standard PPO’s assumption that all actions come from a single old policy.
5 Addressing the Algorithmic Challenges in AREAL
AREAL addresses asynchronous RL challenges by controlling data staleness and using a decoupled PPO objective that supports interrupted, mixed-policy generation while stabilizing updates.
- 5.1 Staleness-Aware Training: η limits the maximum permitted staleness of training samples, with η = 0 reducing AREAL to synchronous RL.The system enforces this limit by dynamically controlling generation-request throughput.
- 5.1 Staleness-Aware Training: The rollout controller prioritizes older buffered trajectories and rejects generation requests that would violate the staleness constraint.
- 5.1 Staleness-Aware Training: Large η improves throughput in practice, whereas small η can slow generation when extremely long trajectories are being produced.This trade-off motivates an enhanced algorithm that can use more stale data effectively.
- 5.2 Decoupled PPO Objective: The decoupled PPO objective separates the behavior policy used for sampling from the proximal policy that regularizes updates.Importance sampling is applied to sampled trajectories for asynchronous RL training.
- 5.2 Decoupled PPO Objective: Using a recent proximal policy keeps updates near a high-quality target, stabilizing asynchronous training and avoiding attraction toward outdated policies.The objective also permits mixed policy versions within a batch while preserving equivalence to a single behavior policy for interrupted generation.
6 Implementation
AREAL improves implementation efficiency by overlapping generation with CPU work and packing variable-length sequences efficiently under memory constraints.
- 6 Implementation: AREAL combines SGLang for generation serving, Megatron-Core for training, and SLURM for resource scheduling.The implementation uses Python and PyTorch within the ReaLHF framework.
- 6 Implementation: AREAL separates GPU computation from reward computation and TCP data transfer, overlapping both CPU operations with subsequent generation requests.Asyncio coroutines allow multiple rollout requests to run concurrently without mutual blocking waits.
- 6 Implementation: Padding-free sequence packing and dynamic allocation balance tokens across micro-batches under fixed memory constraints.This maximizes GPU memory utilization while reducing the number of forward-backward passes.
7 Experiments
The experiments evaluate AReaL across models, tasks, scaling settings, and ablations, showing substantial efficiency gains while maintaining or improving performance. Ablations support controlled staleness, the decoupled PPO objective, and system optimizations as key design choices.
- End-to-End Comparison: AReaL matches or exceeds baseline performance while reducing end-to-end training time by 2.77× compared with synchronous systems.The comparison covers mathematical reasoning and code generation settings.
- Scalability: AReaL achieves up to 2.5× speedup and approximately linear scaling as device count increases, whereas the synchronous baseline typically scales poorly.The comparison spans different model sizes and context lengths.
- Scalability: For smaller context lengths, AReaL’s advantage can diminish when generation throughput cannot keep pace with training consumption.Longer generation lengths are more robust because asynchronous, interruptible generation can hide them in the critical path.
- Algorithm Ablations: Moderate staleness with the decoupled PPO objective preserves final evaluation performance while accelerating training by over 2×.Unbounded staleness remains inferior to the zero-staleness oracle, so staleness must be constrained.
- System Ablations: Dynamic batching improves throughput by an average of 30% across model sizes.The dynamic approach uses a 32,768-token budget per micro-batch, compared with 32 standard micro-batches.
- System Ablations: Interruptible generation increases generation throughput by 12% for 1.5B models and 17% for 7B models on 4 nodes.Without it, the controller waits for the longest response.
8 Conclusion
AREAL is a fully asynchronous RL system for efficient large-scale training, combining architectural and algorithmic innovations with improved hardware efficiency, sample efficiency, and scalability.
- AREAL fully decouples generation and training for efficient large-scale RL training.
- Staleness-aware training and a decoupled PPO objective support efficient and stable PPO training in asynchronous environments.
- Experiments demonstrate superior hardware efficiency, sample efficiency, and scalability compared to existing synchronous RL systems.
- AREAL provides a starting point for reliably scaling RL training.
NeurIPS Paper Checklist
The checklist reports that the paper’s claims, reproducibility information, experimental details, and theoretical assumptions are adequately disclosed, while statistical significance reporting is incomplete.
- The paper’s abstract and introduction accurately reflect its contributions and scope.
- Theoretical results include stated assumptions and complete proofs in Proposition 1 and Section 3.
- The paper provides information needed to reproduce its main experimental results, including models, datasets, and hyperparameters.
- The paper reports open-source code, datasets, and models with sufficient instructions for faithful reproduction.
- The paper specifies training and test details and provides sufficient computer-resource information for understanding and reproducing experiments.
- Large-scale end-to-end experiments omit error bars and report single trials under the same fixed random seed.
B.2 Dataset Details
The experiments use open-source math and code datasets, compare training systems with dynamic batching, and evaluate trained models on math benchmarks.
- Dataset Details: Math training uses open-source DeepScaleR data, while code training uses the dataset released by DeepCoder.
- Dataset Details: Dynamic batching partitions sequences into balanced micro-batches whose total length does not exceed capacity C.
- Dataset Details: The batching algorithm sorts sequences by descending length and assigns each sequence to a fitting micro-batch with the fewest sequences.
- Dataset Details: Throughput experiments use verl with SGLang or vLLM for generation and PyTorch FSDP for training.
- Dataset Details: Models trained with AREAL are evaluated on additional math and coding benchmarks presented in Tables 4 and 5.
C.2 Generalization Across Model Architectures
Additional experiments examine AREAL across model families, staleness levels, and RL algorithms, while identifying broader algorithmic scope as future work.
- C.2 Generalization Across Model Architectures: Experiments with DeepSeek-Distilled-Llama-8B evaluate generalization across math benchmarks.
- C.2 Generalization Across Model Architectures: The results demonstrate that AREAL generalizes effectively across different model families.
- C.4 Staleness-Throughput Trade-off with Different RL Algorithms: Small-scale experiments test staleness values using DeepSeek-Distilled-Qwen-1.5B with 8k context length and batch size 64×16 on 8 GPUs.
- C.4 Staleness-Throughput Trade-off with Different RL Algorithms: RLOO exhibits slightly better tolerance to asynchronous training than vanilla PPO in local 8-GPU experiments.
- C.4 Staleness-Throughput Trade-off with Different RL Algorithms: AREAL modifies PPO/GRPO because importance sampling supports asynchronous off-policy training, leaving other off-policy algorithms for future investigation.
- D Proof of Proposition 1: Interrupted generation can be represented as sampling from a behavior policy constructed from the sequence of policies used during generation.
E Limitations and Future Work
The evaluation and system design leave several areas for future work, including device allocation, dynamic workload balancing, and broader interaction settings.
- E Limitations and Future Work: The inference-to-training device ratio could be optimized for specific training setups and dynamically adjusted during training.The authors particularly note that context lengths typically increase when fine-tuning pre-trained base models.
- E Limitations and Future Work: The evaluation focuses on single-step mathematical and coding tasks rather than multi-turn interactions or agentic scenarios.The architecture is described as not inherently limited to the evaluated domains, but those settings remain for future exploration.
- E Limitations and Future Work: Future research should explore AREAL in multi-turn and agentic scenarios.