Source-linked AI summary
AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs
Haizhong Zheng, Yizhuo Di, Jiahui Wang, Shuowei Jin, Xueshen Liu, Yongji Wu, Z. Morley Mao, Ion Stoica, Jiawei Zhao, Beidi Chen
TL;DR
Agentic RL needs to coordinate multi-policy workloads across elastic, heterogeneous, and cross-region resources, while existing trainer-centered systems require feature-specific engineering. AstraFlow decouples rollout, dataflow, and training into autonomous components, and achieves comparable or better accuracy with up to 2.7× faster multi-policy training. Its scope is limited by workload-dependent benefits, controlled or simulated deployments, and assumptions about dataflow representation.
Problem
Agentic RL must handle dynamic execution, multi-policy coordination, and diverse compute environments, but existing systems lack clean abstractions for composing these capabilities.
Method
AstraFlow uses autonomous dataflow, Rollout-as-a-Service, and trainer components connected through stable data and weight interfaces.
Results
AstraFlow supports multi-policy training, elastic scaling, heterogeneous cross-region execution, and composable data algorithms without feature-specific system code, with up to 2.7× faster training.
Takeaways & Limitations
A dataflow-oriented architecture provides one system design for composing agentic RL workloads and diverse execution resources.
Takeaways & Limitations
Benefits may depend on workload characteristics, some experiments use controlled or simulated deployments, and the system assumes tasks, metadata, and rewards fit the dataflow layer.
Abstract
from arXiv · showhide
Reinforcement learning (RL) is increasingly used to improve the reasoning, coding, and tool-use capabilities of large language models, but agentic RL remains prohibitively expensive. Scaling RL to agentic LLMs requires supporting complex workloads, including multi-policy collaborative training, while efficiently using elastic, heterogeneous, and cross-region compute resources. Existing LLM RL systems support some of these capabilities, but each new extension often requires dedicated system engineering. This burden arises from trainer-centered control architectures and the lack of principled abstractions for RL system components. To address these limitations, we propose AstraFlow, a dataflow-oriented RL system that replaces conventional trainer-centered control with principled component abstractions. In AstraFlow, rollout services, dataflow management, and training are decoupled into autonomous components, enabling the system to natively support complex multi-policy agentic RL workloads and efficiently exploit diverse compute resources. We evaluate AstraFlow across math, code, search, and AgentBench workloads, showing that the same system supports multi-policy training, elastic scaling, heterogeneous cross-region execution, and composable data algorithms without system-level code changes. In multi-policy collaborative training, AstraFlow achieves comparable or better accuracy than existing RL systems while speeding up training time by 2.7x.
1 Introduction
Agentic RL must handle dynamic, multi-policy workloads across diverse compute environments, but existing trainer-centered systems are rigid and costly to extend. AstraFlow addresses this with dataflow-oriented abstractions and demonstrates broad flexibility, including faster multi-policy training.
- Agentic RL must support dynamic execution, multi-policy coordination, and elastic, heterogeneous compute environments.
- Existing systems center control on trainers, making multi-policy coordination and extensibility difficult.
- AstraFlow separates dataflow, rollout services, and trainers into autonomous components connected through shared data and stable interfaces.
- AstraFlow evaluates multi-policy collaborative RL, system flexibility, and composable data algorithms across its supported workloads.
- 2.7× speedup in training is achieved for multi-policy collaborative RL while maintaining comparable or better accuracy than an existing multi-agent RL system.
2 Related Work
LLM RL systems are expanding from conventional training toward data-centric and agentic workloads, but existing pipeline organizations impose coordination and scalability constraints.
- RL improves LLM reasoning, code generation, and tool use through advances in objectives, rewards, asynchronous training, and data-centric algorithms.
- LLM RL pipelines typically alternate rollout, which generates trajectories and rewards, with training, which updates the policy.
- Colocated synchronous systems place rollout and training on the same GPU pool and alternate between trajectory generation and optimization.
3 Dataflow-Oriented RL for Agentic LLMs
AstraFlow replaces trainer-centered orchestration with autonomous dataflow, rollout, and training abstractions that coordinate through stable interfaces. This design supports composable data policies, elastic rollout services, heterogeneous execution, and multi-policy training without pipeline rewrites.
- Motivation: From Trainer-Centered Control to Dataflow-Oriented Coordination: Existing disaggregated systems separate compute placement but retain trainer-centered control over scheduling, data selection, replay, staleness, and weight synchronization.
- Motivation: From Trainer-Centered Control to Dataflow-Oriented Coordination: AstraFlow gives rollout services, trainers, and the dataflow layer autonomous control loops connected through minimal data and weight interfaces.
- Dataflow Layer Abstraction: The dataflow layer coordinates prompts, trajectories, metadata, and batches between rollout services and trainers.
- Dataflow Layer Abstraction: Dataflow policies support selective rollout, curriculum scheduling, filtering, dynamic sampling, replay, data mixing, and staleness correction without modifying trainers or rollout implementations.
- Rollout-as-a-Service (RaaS) Abstraction: RaaS nodes consume tasks, execute agent workflows, return trajectories, and refresh weights through a substitutable rollout interface.
- Rollout-as-a-Service (RaaS) Abstraction: Elastic rollout capacity is achieved by adding or removing RaaS nodes connected to the same dataflow and weight-transfer interfaces.
- Trainer Abstraction and Weight Transfer: Trainers consume batches, update policies, and publish weights through a common abstraction that supports multiple trainers and independent policy weight streams.
- Trainer Abstraction and Weight Transfer: Weight transfer stores versions and supports full-model, sparse, and version-aware refresh while keeping delivery outside the trainer’s critical path.
4 Evaluation: Applications of AstraFlow
AstraFlow is evaluated across multi-policy collaboration, system flexibility, and data-algorithm composition. Across these settings, it supports diverse workflows and compute environments while improving efficiency without feature-specific system changes.
- Multi-Policy Collaborative Training: AstraFlow evaluates multi-policy collaboration through math solver–verifier, code solver–selector, and code solver–test-case generation workflows.Users specify role order, context passing, and reward assignment rather than modifying the system pipeline.
- Multi-Policy Collaborative Training: 2.7× faster iteration time is achieved in the direct math comparison while maintaining comparable or better accuracy than the matched system.AstraFlow reduces iteration time from 212.64s to 77.65s and improves average accuracy from 51.1% to 56.5%.
- Multi-Policy Collaborative Training: 34.55% average code accuracy is achieved by the stronger multi-agent workflow, versus 30.29% for the matched single-policy Solver baseline.Both code workflows improve over their matched single-policy baselines across the evaluated benchmarks.
- System Flexibility: Rollout auto-scaling reduces total GPU-hours to 312.0, about 13% below both fixed-pool baselines, with comparable accuracy and nearly the same wall-clock time as 11 GPUs.The auto-scaling controller releases rollout capacity when it is not needed and scales up when trainer waiting rises.
- System Flexibility: 67.6 average accuracy is reached on the five-benchmark math suite in cross-region training, comparable to a homogeneous local baseline despite constrained networking.Delta sparsity reduces per-iteration transfer from approximately 28 GB to roughly 1.5 GB, while periodic full synchronizations remain amortized.
- Data Algorithm Flexibility: Data algorithms become composable plug-ins: dynamic sampling improves final accuracy at roughly 3.5× generation cost, while GRESO and buffer replay reach baseline-level accuracy with fewer rollouts.The dataflow layer composes prompt selection, rollout filtering, and trajectory replay without system-wide rewrites.
5 Conclusion
AstraFlow is presented as a dataflow-oriented RL training system whose abstractions replace trainer-centered control. Its dataflow layer, rollout service, and trainer interact through stable interfaces.
- Figure 9 evaluates math accuracy against generated rollouts for dataflow-layer data algorithms.
- AstraFlow is a dataflow-oriented RL training system for agentic LLMs.
- The system replaces trainer-centered control with dataflow, rollout-service, and trainer abstractions that interact through stable interfaces.
Limitations
AstraFlow focuses on infrastructure abstractions rather than a new RL optimization algorithm, and its evaluation does not cover all agentic RL settings. Observed benefits may depend on workload and deployment characteristics.
- AstraFlow addresses system abstractions for agentic RL workloads rather than proposing a new RL optimization algorithm.
- The evaluation does not exhaust settings such as long-horizon web agents, robotics environments, or safety-critical interactive systems.
- Observed benefits may depend on rollout latency, trainer throughput, network bandwidth, weight-transfer frequency, and asynchronous data availability.
- Some experiments use controlled or simulated deployment settings, and AstraFlow assumes relevant tasks, metadata, and rewards can be represented through the dataflow layer.
- Model quality, robustness, and safety still depend on the underlying data, rewards, policies, and evaluation protocol.
A Experimental Settings
The appendix provides a self-contained, per-experiment account of the experimental setup. Shared evaluation choices are fixed across experiments, while subsections specify experiment-specific details.
- The appendix organizes experimental setup per experiment so each subsection is self-contained and reproducible.
- Shared setup covers model cards, evaluation suites, decoding, and reward functions across experiments.
- Experiment-specific subsections specify training data, algorithm hyperparameters, hardware and topology, baselines, and additional knobs.
- Each benchmark uses 4 independent generations per question and reports mean per-question accuracy averaged over the benchmark.
- Results are reported from the checkpoint with the best benchmark-average accuracy, evaluated at a fixed cadence during training.
A.2 Multi-policy Math (Section 4.1, Table 2)
The multi-policy math experiment trains separate Solver and Verifier policies and compares AstraFlow with a reproduced synchronous baseline under matched hardware timing. Evaluation uses four math benchmarks and reports pass@1(avg@4).
- Setup: Separate Solver and Verifier policies are initialized from Qwen3-8B and trained jointly in the actor-and-verify workflow.Training uses the DAPO RL math set with prompts capped at 2,000 tokens.
- Setup: The run uses M2PO with normalized rewards and advantages, a fixed KL penalty coefficient of 10^-3, AdamW, and 256-sample batches.The cited configuration also specifies a 5e−6 learning rate, eight rollouts per prompt, and four PPO mini-batches per iteration.
- Timing: 212.64 s for verl and 77.65 s for AstraFlow are measured per iteration on the same 8×H100 node.The verl pipeline executes rollout and training sequentially, whereas AstraFlow reports trainer-step time with rollout overlap absorbed by the dataflow layer.
- Evaluation: Accuracy is evaluated on AIME24, AIME25, MATH500, and Minerva Math every 25 training iterations using pass@1(avg@4).Four generations are sampled per question at temperature 0.6.
A.5 Heterogeneous and Cross-region Training (Section 4.2.2)
The cross-region experiment compares a homogeneous local baseline with a heterogeneous deployment spanning local and remote H200 rollout pools. It evaluates math accuracy under this distributed topology.
- Results: The homogeneous local baseline records accuracy 68.0, compared with 67.6 for the cross-region run.The local result is used as the baseline for the cross-region comparison.
- Topology: The cross-region deployment uses a 4-GPU FSDP trainer and three 4-GPU SGLang RaaS pools across three H200 nodes.One rollout pool is co-located with the trainer, while two are remote.
- Evaluation: The evaluation covers AIME24, AIME25, AMC, MATH500, and Minerva Math using pass@1(avg@4).Each benchmark is evaluated every 50 training iterations with four generations per question at temperature 0.6.
A.7 Performance vs. AReaL (Section 4.2.3, Table 5)
The AReaL comparison holds model initialization, data, training budget, and algorithm hyperparameters constant while varying only the framework implementation. It measures benchmark accuracy and training efficiency.
- Comparison setup: AstraFlow is compared against AReaL on Qwen3-1.7B and Qwen3-8B math jobs trained on the DeepScaler RL math set.Prompts are filtered to at most 2,000 tokens.
- Comparison setup: Only the framework implementation differs between AstraFlow and AReaL under the matched comparison setup.Initialization, training data, budget, and algorithm hyperparameters are shared.
- Metrics: Evaluation reports AIME24, AIME25, AMC, MATH500, and Minerva Math accuracy alongside median per-iteration training time and Time/1M tok.Efficiency values are averaged after the warmup phase.
A.8 Data-algorithm Flexibility (Section 4.3, Figure 9)
The data-algorithm experiment composes filtering, replay, and GRESO through dataflow-layer hooks and examines accuracy against cumulative rollout cost. All variants share the same prompt stream and training setup.
- Setup: The experiment truncates the DeepScaler math set to 8,000 prompts so GRESO variants can make multiple passes within 800 iterations.The authors note that this makes Figure 9 accuracy a few points below the corresponding full-set result.
- Data algorithms: Vanilla, DS + Replay, and DS + Replay + GRESO intervene at different dataflow points through filtering, replay, and pre-rollout curation.GRESO adapts per-bucket prompt submission probabilities toward easy and hard zero-variance targets.
- Evaluation: Figure 9 plots average pass@1(avg@4) across five math benchmarks against cumulative generated rollouts.The plot exposes the rollout-cost / accuracy trade-off across data algorithms.
B.1 Accuracy of Agentic-Task Runs
These runs report trained-policy accuracy on three agentic tasks using a shared evaluation protocol, alongside per-iteration delta-sparsity measurements. Search includes four open-domain question-answering suites with an additional per-benchmark breakdown.
- Evaluation protocol: Accuracy uses four generations per question at temperature 0.6, averaged across generations, with the checkpoint having the best benchmark-average score reported.The protocol is applied to Qwen2.5-7B-Instruct runs on three agentic tasks.
- Task coverage: Search (ASearcher) is evaluated on four open-domain QA suites and includes a per-benchmark accuracy breakdown.Results are reported as pass@1(avg@4) percentages at each run’s best-average checkpoint.
- Delta-sparsity analysis: Delta-sparsity curves are averaged step-wise across overlapping sub-runs, smoothed with a 15-iteration centered moving average, and limited to the first 500 iterations.Figure 10 varies model scale on math and task and learning rate on Qwen2.5-7B-Instruct agentic tasks.
- Delta-sparsity analysis: The Search run at learning rate 5e−6 is a clear outlier, while rerunning it at 1e−6 aligns it with AlfWorld and WebShop.The comparison attributes the gap to learning-rate magnitude rather than task type.