Source-linked AI summary
AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
Zhenyu Han, Ansheng You, Haibo Wang, Kui Luo, Guang Yang, Wenqi Shi, Menglong Chen, Sicheng Zhang, Zeshun Lan, Chunshi Deng, Huazhong Ji, Wenjie Liu, Yu Huang, Yixiang Zhang, Chenyi Pan, Jing Wang, Xin Huang, Chunsheng Li, Jianping Wu
TL;DR
LLM post-training RL must coordinate multiple tasks, while existing designs contend with task coordination and dataflow challenges. AsyncFlow combines streamed data management with a producer-consumer workflow; it improves throughput over the baseline and scales with superior linearity.
Problem
RL post-training involves several models and tasks, while existing frameworks face coordination challenges, including predefined data dependencies in task-separated setups.
Source: §1 · p. 1, §9 · p. 12Method
AsyncFlow combines TransferQueue for streamed, dynamically routed data with a producer-consumer workflow allowing one-step asynchronization between actor rollout and actor update.
Source: §1 · p. 2, §9 · p. 12Results
AsyncFlow achieves significant throughput improvements over the baseline and superior linearity when scaling to larger clusters.
Source: §9 · p. 13Takeaways & Limitations
Its service-oriented interface provides modular adapters for diverse training and inference engines and a unified algorithm entry point.
Source: §3 · p. 4Sub-step asynchronization remains future work and is conditioned on generated responses sufficiently supporting downstream tasks.
Source: §9 · p. 13
Abstract
from arXiv · showhide
Reinforcement learning (RL) has become a pivotal technology in the post-training phase of large language models (LLMs). Traditional task-collocated RL frameworks suffer from significant scalability bottlenecks, while task-separated RL frameworks face challenges in managing complex dataflows and resolving resource idling. Furthermore, most existing frameworks are tightly coupled with LLM training or inference engines, making them difficult to support custom-designed engines. To address these challenges, we propose AsyncFlow, an asynchronous streaming RL framework tailored for efficient post-training. Specifically, we introduce a distributed data storage and transfer module that provides panoramic data management and fine-grained scheduling capabilities in a fully streamed manner. This architecture inherently enables automated pipeline overlapping among RL tasks and dynamic load-balancing. Moreover, we propose an asynchronous producer-consumer workflow, which is engineered to minimize computational idleness by strategically deferring the parameter update process within staleness thresholds. Finally, the core capabilities of AsyncFlow are architecturally decoupled from underlying training and inference engines and encapsulated by service-oriented user interfaces, offering a modular and customizable user experience. Extensive experiments demonstrate an average throughput of 1.59x compared to the state-of-the-art baseline. The architecture presented in this work provides actionable insights for designing next-generation RL training systems.
1 Introduction
LLM post-training brings multiple models and tasks into one workflow, complicating efficient and scalable system design. AsyncFlow addresses these challenges with streaming data management, asynchronous execution, and engine-independent interfaces.
1 Introduction
- PPO post-training coordinates six tasks across several models, making efficient and scalable system design challenging.The tasks are actor rollout, reference inference, critic inference, reward inference, actor update, and critic update.Source: §1 · p. 1
- Task-collocated frameworks incur memory overhead, resharding latency, and underutilization when differently sized models share resources.They run one task at a time on the same devices; model parameters may also need costly offloading.Source: §1 · pp. 1–2
- Task-separated frameworks can leave resources idle while dependent tasks wait for actor rollout and can complicate cross-task dataflow.Nested parallelism further complicates dataflow between minimal instances.Source: §1 · p. 2
- TransferQueue dynamically dispatches inference responses to downstream RL tasks, enabling streamed dataflow, automated load-balancing, and pipeline overlap.This avoids explicitly defining cross-instance data dependency chains.Source: §1 · p. 2
- AsyncFlow defers rollout parameter updates within controllable staleness to reduce idle time in its producer-consumer workflow.The introduction describes this workflow as balancing training efficiency and convergence.Source: §1 · p. 2
- Service-oriented interfaces decouple AsyncFlow from specific training and inference engines and support customized backends.The stated aim is to accommodate research flexibility and industrial deployment requirements.Source: §1 · p. 2
- 2.03× maximum throughput improvement over the baseline in large-scale clusters is reported for AsyncFlow.The comparison is against a state-of-the-art RL post-training framework.Source: §1 · p. 2
2 Motivation
LLM post-training combines distinct computational phases and interdependent tasks. Task-separated execution can address scaling and deployment pressures, but it introduces dataflow, synchronization, and interface challenges.
2.1.1 Computational characteristic of RL post-training
- Rollout generates responses, inference evaluates them, and update optimizes model weights; rollout is memory-bound while inference and update are computation-bound.In PPO, rollout responses feed the other models, whose inference outputs are needed by the update tasks.Source: §2.1.1 · p. 3
2.1.2 Architecture taxonomy of RL frameworks
- Task-collocated execution shares devices across tasks in time, whereas task-separated execution places tasks on distinct devices through spatial allocation.Task-separated allocation may be partial or full.Source: §2.1.2 · p. 3
2.2 Why we need task-separated frameworks
- Task-separated design can improve scaling efficiency and deployment costs as post-training resources grow and task workloads differ.Collocated rollout and update phases share devices despite their different computational characteristics, while task-separated architectures can reuse existing clusters and leverage hardware suited to each task.Source: §2.2 · p. 3
- Task-collocated transitions incur model offloading and onloading costs, and actor rollout-to-update transitions require resharding for different parallelization strategies.These overheads become more dominant as model size grows and are difficult to overlap with other computation.Source: §2.2 · p. 3
2.3 Challenges for task-separated frameworks
- Task-separated resources can remain underutilized because task dependencies limit parallelism, making fine-grained sample-level data management important for reducing pipeline bubbles.The data-management design is more complex than a centralized global-batch buffer.Source: §2.3 · p. 4
- Strict rollout-update synchronization creates pipeline bubbles, while task separation also complicates unified control, algorithm development, and customized backend support.Asynchronous off-policy approaches may improve efficiency, but limiting their effect on convergence remains a challenge.Source: §2.3 · p. 4
3 AsyncFlow Overview
AsyncFlow combines streaming data management with an asynchronous workflow to address task coordination and resource idling in RL post-training. It also provides service-oriented interfaces for algorithm users and backend integration.
3 AsyncFlow Overview
- TransferQueue dynamically schedules complex post-training dependencies with sample-level data management, load balancing, and gateway routing.The gateway decouples explicit data dependencies across RL tasks and supports a divide-and-conquer approach to algorithm-controller design.Source: §3 · p. 4
- AsyncFlow proposes asynchronous off-policy execution that delays parameter updates within a staleness threshold to reduce pipeline bubbles across iterations.Its producer-consumer prototype lets some rollout instances update parameters while others serve downstream tasks.Source: §3 · p. 4
- A service-oriented interface separates algorithm logic from execution engines through modular backend adapters and provides a unified algorithm entry point.The design also offers service-oriented APIs for integration with existing training and inference clusters.Source: §3 · p. 4
4 TransferQueue: An Asynchronous Streaming Data Management Module
TransferQueue provides streamed, sample-level data management across RL tasks, pairing distributed storage with a controller that tracks data status and schedules requests. Its interfaces let tasks retrieve needed data independently, supporting overlapping work and simplifying workflow design.
4.1 Architecture Overview
- TransferQueue separates control and data planes, tracks sample status, and routes data between RL tasks to support fine-grained scheduling and overlap.Its distributed units mitigate potential I/O and network bottlenecks; the design also removes manual cross-process dataflow configuration and lets tasks manage dependencies autonomously.Source: §4.1 · p. 5
4.2 Interaction Interface
- Algorithm controllers request sample metadata, while distributed workers retrieve data and write results directly to storage units.Consumption records prevent duplicate access within an RL task; storage updates notify controllers when data becomes ready.Source: §4.2 · p. 5
4.3 Data Plane: Distributed Data Storage
- The 2D data structure stores complete samples as rows and task-specific components as columns, allowing tasks to request only required fields.Global indices address samples across storage units, which distribute entries to balance storage and communication overhead.Source: §4.3 · p. 6, Figure 6 · p. 6
4.4 Control Plane: Panoramic Data Admin
- Controllers select samples whose requested fields are ready and unconsumed, then use load balancing to choose data and return its metadata.For DP groups, the controller keeps sample counts consistent while minimizing token-count variation; for PP groups, it relaxes the sample-count constraint and uses Karmarkar–Karp [9].Source: §4.4 · pp. 6–7, Figure 7 · p. 6
5 Producer-Consumer-Based Asynchronous Workflow Optimization
AsyncFlow reduces idle time in task-separated RL through sample-level pipeline overlap and asynchronous parameter updates. Its workflow combines delayed updates, overlapping communication, and resource planning while maintaining constraints on parameter staleness.
5 Producer-Consumer-Based Asynchronous Workflow Optimization
- Task-separated RL can leave hardware idle across dependent tasks; AsyncFlow combines pipeline overlap, delayed updates, and resource planning to reduce those bubbles.The streaming dataloader overlaps RL tasks, while one-step asynchronization is designed to balance efficiency with convergence stability.Source: §5 · p. 7
5.1 Sample-Level Overlapping across Tasks
- Fine-grained samples scheduled through a streaming dataloader let RL tasks overlap without manually rescheduling data streams.The approach is described as extensible to RL algorithms with varying tasks.Source: §5.1 · p. 7, Figure 8 · p. 7
5.2.1 Basic asynchronous RL algorithm
- Strict synchronization creates warm-up and cool-down bubbles, while larger rollout batches permit stale responses but version differences remain bounded by convergence constraints.One-step asynchronization is reported not to cause significant performance degradation or convergence issues; performance drops logarithmically beyond that threshold [15] [42].Source: §5.2.1 · pp. 7–8
5.2.2 Delayed parameter update mechanism
- The delayed-update mechanism lets rollout continue with old weights during an update, then loads new weights after generation to reduce exposed synchronization to host-to-device transfer.The paper also describes sub-step asynchrony using sequential parameter updates across rollout instances, but leaves that mechanism’s implementation for future work.Source: §5.2.2 · p. 8, Figure 9 · p. 8
5.2.3 Parameter update overlapping
- WeightSender and WeightReceiver support synchronous and asynchronous updates; asynchronous host-network transfer can overlap computation without stalling ongoing tasks.The synchronous mode uses HCCL links across Ascend NPUs to shorten exposed transmission time.Source: §5.2.3 · p. 8
5.3 Task Resource Planning
- A graph-based resource planner searches configurations under resource constraints using simulated computation and communication times to minimize end-to-end workflow time.Its hybrid cost model combines fast analytical estimates to narrow the search with more accurate, slower profiling.Source: §5.3 · p. 9
6 Service-Oriented User Interface
AsyncFlow separates algorithm-facing controls from backend execution through a hierarchical service-oriented interface. This division is intended to support rapid algorithm iteration alongside flexible deployment across training and inference engines.
6 Service-Oriented User Interface
- A hierarchical interface exposes high-level algorithm APIs while backend adapters decouple algorithm logic from execution engines.Researchers use the high-level layer for algorithm design, while infrastructure engineers focus on backend implementation.Source: §6 · p. 9
6.1 User-Level Interface
- The Trainer class centralizes the workflow and lets researchers modify core RL algorithms, while APIs initialize engines, manage data, and notify weight synchronization.These controls are designed to initiate post-training with minimal configuration.Source: §6.1 · p. 9
6.2 Backend-Level Interface
- The Adapter class integrates varied training and inference backends, including FSDP, DeepSpeed, and vLLM, while supporting custom-designed frameworks.Engine changes can be made without disrupting workflow integrity.Source: §6.2 · p. 9
7 Evaluation
AsyncFlow is evaluated on Qwen2.5 models using GRPO and compared primarily with verl across NPU cluster sizes. It reports higher throughput, gains from its dataflow and workflow optimizations, and comparable reward convergence for asynchronous and synchronous runs.
7.1 Experiment Setup
- The evaluation uses Qwen2.5-7B and 32B models, GRPO, the DeepScaleR dataset, and Ascend clusters; verl is the primary comparison baseline.The authors exclude OpenRLHF results because its NVIDIA-optimized implementation performed suboptimally out of the box on Ascend NPUs.Source: §7.1 · pp. 9–10
7.2 Overall Performance Analysis
- 1.59× average throughput over verl across 32–512 NPUs; scaling linearity is 0.65 for 7B and 0.88 for 32B over 16× cluster growth.The comparison covers Qwen2.5-7B and Qwen2.5-32B; Figure 11 reports average throughput over 10–20 iterations.Source: §7.2 · p. 10, Figure 11 · p. 10
7.3 Ablation Studies
- 2.01× throughput over the baseline with TransferQueue, followed by a further 36.3% gain from asynchronous workflow optimization over the TransferQueue-enabled baseline.The workflow bundle includes delayed parameter updates, overlap techniques, and task resource allocation.Source: §7.3 · p. 11, Table 1 · p. 11
7.4 Load-Balancing Strategies
- 2.91× and 1.81× throughput over baseline for reference inference and actor update with PP load balancing; adding DP balancing raises these to 3.03× and 1.85×.Table 2 summarizes normalized throughput for the 7B model on 16 NPUs.Source: §7.4 · p. 11, Table 2 · p. 11
7.5 Optimized Workflow of AsyncFlow
- Actor rollout uses DP=64 and actor update DP=20; their nearly contiguous, overlapping execution indicates little inter-task idle time in the illustrated workflow.Figure 12 depicts a 32B model on 512 NPUs across iterations 0–3.Source: §7.5 · p. 11, Figure 12 · p. 11
7.6 Stability of Asynchronous RL Algorithm
- Asynchronous and synchronous runs show negligible reward differences and comparable convergence, while asynchronous response lengths fluctuate less after initial exploration.Both modes ultimately converge to stable response lengths in the 7B, 16-NPU evaluation.Source: §7.6 · pp. 11–12, Figure 13 · p. 12
8 Related Work
Related work distinguishes task-collocated from task-separated post-training frameworks, reviews RLHF algorithm variants, and describes system optimizations that overlap rollout with downstream work.
8.1 LLM Post-Training Frameworks
- Task-collocated frameworks include TRL, DeepSpeed-Chat, NeMo-Aligner, RLHFuse, and verl, while task-separated frameworks allocate distinct resources to rollout and update.The related-work section notes that collocated frameworks can achieve high utilization at small scale, while task-separated frameworks show competitiveness at large scale and specialize resources for inference and training.Source: §8.1 · p. 12
8.2 RLHF Algorithms
- PPO coordinates actor, reference, reward, and critic models; GRPO removes the critic by estimating advantages from group-normalized rewards, while simplification may introduce stability issues.The section presents GRPO and DAPO as variants that eliminate specific components to reduce computational overhead.Source: §8.2 · p. 12
8.3 LLM Post-Training System Optimization
- RLHFuse, StreamRL, and k1.5 support rollout overlap through inter-stage fusion, response-length prediction, and partial rollout, respectively.Because post-training rollout lacks a strict latency requirement, the section identifies broader scheduling strategies as an opportunity to maximize system throughput.Source: §8.3 · p. 12
9 Conclusion
AsyncFlow is a task-separated framework for modular, scalable post-training that combines streamed data management with asynchronous producer-consumer execution. Experiments report throughput improvements and superior scaling linearity, while finer-grained rollout asynchrony remains future work.
9 Conclusion
- TransferQueue streams and routes data dependencies across training and inference instances, while distributed storage and panoramic management address single-controller routing bottlenecks.The module supports automated load balancing and pipeline overlap among RL tasks.Source: §9 · p. 12
- A producer-consumer workflow permits one-step asynchronization between actor rollout and update, overlapping parameter updates with computation to minimize hardware idling while maintaining algorithmic integrity.The workflow is designed to balance training efficiency and convergence stability.Source: §9 · p. 12
- Service-oriented interfaces provide two-level abstraction for algorithm workflow and backend engines, bridging theoretical research and industrial deployment.The framework is designed to be modular and scalable, particularly in large-scale clusters.Source: §9 · p. 12
- AsyncFlow achieves significant throughput improvements over the baseline and exhibits superior linearity when scaling to larger clusters.The conclusion characterizes the framework as offering a modular and scalable post-training capability, especially in large-scale clusters.Source: §9 · pp. 12–13
- Future work proposes staggering parameter updates across inference instances, conditional on generated responses sufficiently supporting downstream tasks, to reduce exposed transition time and enable sub-step asynchrony.The proposed workflow places the staleness threshold below one training step.Source: §9 · p. 13
APPENDIX
The appendix gives code examples for manually binding OpenRLHF task dataflows, using TransferQueue, and abstracting backend interfaces; it also illustrates a sub-step asynchrony workflow identified for future work.
APPENDIX
- Figure A1 illustrates sequentially enabling parameter updates across rollout instances for sub-step asynchrony, whose implementation is left for future work.Source: Figure A1 · p. 16
- The OpenRLHF example manually binds dataflow relationships across data-parallel groups, and the caption notes that this implementation was replaced in recent releases.The excerpt includes actor-to-critic binding and a conditional critic-training assignment.Source: APPENDIX · p. 16
- The TransferQueue example initializes the queue in rollout code and writes generated responses to it.The cited code snippets show queue initialization and response collection.Source: APPENDIX · p. 17
- Backend-interface examples show MindSpeedAdapter and VLLMAdapter implementing RLAdapter-level abstractions, including compute_log_prob.The code labels compute_log_prob as an abstraction for an RL task.Source: APPENDIX · p. 17