Source-linked AI summary
MegaFlow: Large-Scale Distributed Orchestration System for the Agentic Era
Lei Zhang, Mouxiang Chen, Ruisheng Cao, Jiawei Chen, Fan Zhou, Yiheng Xu, Jiaxi Yang, Zeyao Ma, Liang Chen, Changwei Luo, Kai Zhang, Fan Yan, KaShun Shum, Jiajun Zhang, Zeyu Cui, Feng Hu, Junyang Lin, Binyuan Hui, Min Yang
TL;DR
Complex agentic tasks require infrastructure that can coordinate large-scale agent-environment interactions beyond existing open-source support. MegaFlow addresses this gap with three independently scalable services connected by unified interfaces, achieving large-scale deployment performance including 32% cost reduction and consistent scaling to 10,000 concurrent tasks. The paper concludes that its orchestration architecture provides a production-ready foundation while identifying multi-environment dependencies as future work.
Problem
Existing infrastructure does not effectively support the large-scale training and evaluation coordination required for complex agentic tasks such as software engineering and computer use.
Method
MegaFlow decouples agent training into Model, Agent, and Environment Services connected through unified interfaces for scheduling, resource allocation, and task management.
Results
32% cost reduction and consistent performance scaling to 10,000 concurrent tasks were demonstrated using over 130,000 production task records.
Takeaways & Limitations
MegaFlow provides a production-ready foundation for large-scale agent training research by addressing infrastructure bottlenecks through distributed orchestration.
Takeaways & Limitations
Future work must address multi-environment agent tasks with complex service dependencies, alongside dynamic execution-mode switching and multi-cloud deployment.
Abstract
from arXiv · showhide
The rapid development of interactive and autonomous AI systems signals our entry into the agentic era. Training and evaluating agents on complex agentic tasks such as software engineering and computer use requires not only efficient model computation but also sophisticated infrastructure capable of coordinating vast agent-environment interactions. However, no open-source infrastructure can effectively support large-scale training and evaluation on such complex agentic tasks. To address this challenge, we present MegaFlow, a large-scale distributed orchestration system that enables efficient scheduling, resource allocation, and fine-grained task management for agent-environment workloads. MegaFlow abstracts agent training infrastructure into three independent services (Model Service, Agent Service, and Environment Service) that interact through unified interfaces, enabling independent scaling and flexible resource allocation across diverse agent-environment configurations. In our agent training deployments, MegaFlow successfully orchestrates tens of thousands of concurrent agent tasks while maintaining high system stability and achieving efficient resource utilization. By enabling such large-scale agent training, MegaFlow addresses a critical infrastructure gap in the emerging agentic AI landscape.
1 INTRODUCTION
MegaFlow addresses the infrastructure gap in large-scale training and evaluation of agents on complex tasks by decoupling orchestration into independently scalable services. Its distributed design targets coordination, security, storage, and throughput constraints while demonstrating large-scale deployment performance.
- Motivation: Existing infrastructure cannot effectively orchestrate the massive concurrent agent-environment interactions required for complex multi-step agentic tasks.The core challenge is coordination of dynamic, interdependent processes rather than model computation scalability alone.
- System contribution: MegaFlow separates agent training infrastructure into Model, Agent, and Environment Services connected through unified interfaces.The services respectively support model computation, agent coordination and experience management, and interactive environment execution.
- Security and isolation: MegaFlow migrates containerized workloads to elastic cloud compute services for secure, isolated execution without specialized cluster configurations or compromised security frameworks.This addresses incompatibilities between agent training requirements and existing cluster security policies.
- Storage scalability: On-demand container image provisioning through cloud registries makes storage elastic and usage-based instead of requiring massive local storage.High-bandwidth internal network access supports provisioning as training demand changes.
- Throughput: Thousands of lightweight instances replace reliance on high-specification machines, improving resource utilization and avoiding availability constraints limiting centralized methods to hundreds of concurrent tasks.The approach is designed for distributed orchestration of large numbers of agent tasks.
- Evaluation: 32% cost reduction and consistent scaling to tens of thousands of concurrent tasks were demonstrated, with production validation across over 2 million agent training executions.The evaluation validates MegaFlow’s distributed orchestration and resource-allocation design at deployment scale.
2 MEGAFLOW
MegaFlow is organized as three independently scalable services coordinated through unified APIs and cloud-native infrastructure. Its scheduling, execution, resource, environment, and monitoring mechanisms support elastic, fault-tolerant orchestration of agent workloads.
- Service architecture: The Model, Agent, and Environment Services separately handle model computation, agent execution and experience feedback, and containerized task execution.This separation assigns specialized responsibilities across the architecture.
- Orchestration: Unified APIs coordinate request intake, environment provisioning, event-driven progress monitoring, and downstream result collection across the three services.Cloud-native services provide elastic compute, real-time monitoring, and distributed storage.
- System properties: The architecture enables elastic scaling, event-driven fault tolerance, intelligent scheduling, and independent optimization of isolated services.These properties arise from dynamic resource allocation, event-driven monitoring, and service separation.
- Resource strategy: MegaFlow uses standardized small instances to improve elasticity and cost optimization compared with few-large-instance designs.The strategy supports rapid resource provisioning and deallocation for containerized agent workloads.
- Execution model: Ephemeral execution prioritizes task isolation, while persistent execution reuses compute resources for efficiency according to task characteristics.The hybrid model balances reliability and resource utilization.
- Coordination: Event-driven coordination and distributed state management replace polling-heavy coordination while supporting consistent resource allocation and task scheduling.Direct API calls supplement event notifications with detailed execution information.
- Task scheduling: The asynchronous FIFO scheduler provisions dedicated instances for ephemeral tasks and reuses persistent instance pools for sustained tasks.Containerization maintains isolation for persistent workloads while pool-based allocation improves reuse.
- Resource management: Concurrency control combines user API-rate limits, distributed semaphores tied to compute capacity, and administrative quotas for fair resource sharing.The three-tier mechanism prevents downstream bottlenecks, capacity overrun, and resource abuse.
3 EVALUATION
MegaFlow is evaluated on large-scale software-engineering agent training workloads, where distributed orchestration improves scalability, cost efficiency, utilization predictability, and latency. The evaluation also identifies coordination and local resource constraints as key bottlenecks and validates selectable execution modes.
- Performance and Scalability: MegaFlow maintains approximately 100-minute execution times from 1 to 10,000 tasks, while centralized methods degrade from 100 to 110 minutes.Distributed per-task resources avoid network congestion during image pulls and initialization competition.
- Performance and Scalability: 10,000 concurrent tasks are provisioned by MegaFlow, whereas centralized execution is limited to 2,000 by instance availability.MegaFlow uses standardized instances rather than a maximum of 40 high-specification instances.
- Cost Efficiency: 32% cost reduction at 2,000 tasks lowers cost from 1,470 USD to 1,005 USD, with larger-scale advantages increasing.The distributed approach also removes resource-availability constraints that limit traditional scaling.
- Utilization Pattern Analysis: MegaFlow sustains 5-10% CPU and approximately 12% memory utilization, contrasting with centralized peaks of 25% CPU and 50% memory.Stable utilization and narrow confidence intervals support more predictable capacity planning and allocation.
- Latency Breakdown Analysis: Persistent MegaFlow execution achieves approximately 75-minute latency, compared with 90 minutes for ephemeral execution and 110 minutes centrally.Persistent mode benefits from environment reuse, while ephemeral mode provides stronger isolation at moderate overhead.
- Environment Startup Scaling: Centralized startup time rises from 1 to 13 minutes at 1,000 concurrent tasks, while persistent MegaFlow remains below 1 minute through environment reuse.The analysis attributes the centralized increase primarily to local bandwidth and resource contention rather than cloud-service limits.
- Discussion: The evaluation identifies coordination overhead, cloud-versus-local constraints, and execution-mode choice as infrastructure design considerations.MegaFlow’s evaluation covers over 130,000 production task records and supports broad agent-framework compatibility.
- Discussion: Future work should address multi-environment tasks with complex service dependencies, dynamic execution-mode switching, and multi-cloud deployment.Kubernetes-like orchestration is proposed for dependency management.
4 RELATED WORK
Related systems provide general orchestration, machine-learning infrastructure, multi-agent coordination, or distributed model training, but are not tailored to large-scale interactive agent execution. MegaFlow differentiates itself by decoupling model serving, agent coordination, and environment provisioning.
- Distributed Container Orchestration: Traditional container orchestrators manage distributed workloads but are not optimized for agent-training characteristics such as rapid environment creation.Their abstractions include resource allocation and service discovery, while agent workloads require specialized execution behavior.
- Cloud-Native AI Infrastructure: Kubeflow, MLflow, and Ray primarily target traditional machine-learning pipelines rather than dynamic, containerized interactive agent training.The supplied passage highlights dynamic environment creation and complex agent execution contexts as distinguishing requirements.
- Multi-Agent System Infrastructure: Multi-agent infrastructure research emphasizes coordination algorithms, communication protocols, and small-scale or single-agent interactions.The passage identifies thousands of concurrent tasks across distributed environments as an unresolved infrastructure challenge.
- Large-Scale AI Training Systems: Horovod, FairScale, and Megatron-LM coordinate large AI workloads but focus on synchronous model training rather than asynchronous agent execution.Their tightly coupled architectures are poorly suited to loosely coupled agent-environment interactions.
- MegaFlow: MegaFlow decouples model serving, agent coordination, and environment provisioning, enabling independent scaling and optimization for agent-training workloads.This specialized three-service architecture is the paper’s stated distinction from existing approaches.
5 CONCLUSION
MegaFlow addresses agent-training infrastructure scalability through a three-service architecture that decouples model serving, agent coordination, and environment provisioning. Evaluation on over 130,000 production task records reports 32% lower cost and consistent scaling to 10,000 concurrent tasks.
- Conclusion: MegaFlow decouples Model Service, Agent Service, and Environment Service to address scalability challenges in agent-training infrastructure.The architecture uses unified interfaces across the three services.
- Conclusion: 32% cost reduction and consistent performance scaling to 10,000 concurrent tasks are demonstrated using over 130,000 production task records.The comparison is against traditional centralized approaches.
- Conclusion: MegaFlow provides a production-ready foundation for large-scale agent-training research and supports development of sophisticated AI agents at scale.This conclusion follows the paper’s stated infrastructure outcome.
A DEFINITIONS
MegaFlow defines agent tasks as interactive six-tuples and decomposes agentic systems into three modular services with unified interfaces. Its formalization covers environments, instructions, goals, states, actions, transitions, trajectories, and terminal rewards.
- The three-service architecture separates Model Service, Agent Service, and Environment Service behind well-defined interfaces.The separation supports independent scaling and optimization while preserving coordination interfaces.
- An agent task is an interactive problem-solving environment represented as a six-tuple.
- The task tuple includes an environment specification, task description, goal criteria, state space, action space, and transition function.The transition function maps states and actions to subsequent states and captures deterministic or stochastic environment dynamics.
- Execution produces a state-action trajectory, and the final reward is computed from the complete trajectory upon task completion.Termination occurs through explicit completion decisions or a maximum step limit.
B ADDITIONAL RELATED WORK
The additional related work surveys agent systems and benchmarks for software engineering and computer-use tasks. It positions MegaFlow alongside frameworks, datasets, and interactive evaluation environments spanning these domains.
- Software Engineering Agent Systems: Software engineering agents use language models for interactive code editing, testing, command-line interaction, web browsing, and multi-agent coordination.
- Software Engineering Benchmarks and Datasets: Software engineering benchmarks evolved from real-world GitHub issues and fixes toward interactive, multimodal, multilingual, and multi-agent settings.
- Computer Use and Browser Automation: Computer-use research includes realistic web environments, web navigation tasks, operating-system interaction benchmarks, and browser automation agents.
C AGENT FRAMEWORK COMPATIBILITY
MegaFlow provides broad compatibility across agent frameworks and software engineering datasets through a unified API abstraction. This supports specialized component delegation while leaving agent-specific logic to researchers’ preferred frameworks.
- MegaFlow’s compatibility matrix covers major agent frameworks and software engineering datasets.The unified API abstraction enables integration across all evaluated benchmark suites.
- Supported agent implementations include SWE-Agent, OpenHands, Mini-SWE-Agent, Qwen Code, and Claude Code.
- The architecture delegates agent-specific logic to existing frameworks while MegaFlow handles orchestration and coordination.Standardized interfaces let researchers retain their preferred agent frameworks during distributed execution.
D REINFORCEMENT LEARNING FOR AGENT TRAINING
MegaFlow is used to orchestrate large-scale agentic reinforcement learning for coding agents in realistic software engineering environments. Its distributed execution model supports high-cost environments requiring compilation, testing, and verification.
- MegaFlow enables end-to-end reinforcement learning of coding agents directly inside realistic software engineering environments.
- The distributed execution model scales software engineering environments that require real compilation, testing, and verification.The passage contrasts this capability with conventional reinforcement learning training infrastructure.
D.1 TRAINING SETUP
The training setup combines diverse SWE-style environments with GSPO and multiple coding-agent frameworks, while evaluating training dynamics across two model scales.
- D.1 TRAINING SETUP: The RL corpus combines SWE-Gym, Multi-SWE-RL, SWE-rebench, and internally synthesized SWE-style repair environments.Before training, the corpus includes 2,438 SWE-Gym, 21,336 SWE-rebench, 4,723 Multi-SWE-RL, and 30,274 synthesized environments.
- D.1 TRAINING SETUP: Training uses Group Sequence Policy Optimization with OpenHands, SWE-agent, mini-SWE-agent, Qwen Code, and Claude Code.The frameworks support heterogeneous agent designs.
- D.1 TRAINING SETUP: MegaFlow orchestrates 1024 parallel SWE environments during reinforcement-learning training.
- D.1 TRAINING SETUP: Figure 6 compares a 235 billion parameters MoE model with a 30 billion parameters MoE model across training steps 0–100 on SWE-bench Verified.Scores use the OpenHands scaffold.
- D.1 TRAINING SETUP: Training uses a sampling temperature of 1.0, a 4096-token maximum response per turn, and a fixed −0.5 penalty after 100 rounds without explicit termination.The learning rate is 1e−6, with positive and negative reward clipping thresholds of 4e−4 and 2e−4.
D.2 TRAINING RESULTS
On SWE-bench Verified, both models improve during reinforcement-learning training, while the larger model remains substantially ahead; the rollout system is reported as stable and high-throughput.
- D.2 TRAINING RESULTS: Both models show consistent improvement during reinforcement-learning training, with the larger model achieving substantially higher scores throughout.The evaluation uses SWE-bench Verified and the OpenHands scaffold.