PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolving Agent Teams

Shiyi Cao, Ziming Mao, Dacheng Li, Joseph E. Gonzalez, Ion Stoica

arXiv:2610.36848v1cs.AI

TL;DR

Strong coding agents often converge to local optima on challenging GPU kernels. Evolving Agent Teams uses a supervisor and persistent file system to adapt team organization during optimization. It outperforms strong coding-agent baselines across four CUDA tasks and, with additional budget, exceeds the production FlashInfer MLA decode kernel’s mean speedup.

  • Problem

    Strong coding agents can make rapid initial progress on GPU kernels but converge to local optima well below expert-written baselines.

    Source: §1 · p. 2

  • Method

    Evolving Agent Teams uses a central supervisor to diagnose plateaus and restructure agent roles, grounded in a persistent file system of accumulated experience.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Across four CUDA optimization tasks, Evolving Agent Teams consistently outperforms strong coding-agent baselines under a matched token budget; with additional budget, it reaches 1.21× mean speedup over the production FlashInfer kernel and is faster on 36 of 47 workloads.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The evaluation spans Flash Attention 4, Gated Delta Net, MLA paged decode, and fused GEMM+AllReduce across Blackwell B200 and Hopper H100.

    Source: §1 · p. 2

    Each main setting was run twice and each Section 4.7 setting once, limiting statistical comparisons.

    Source: §5 · p. 10

Abstract

from arXiv · show

The rapid progress of software engineering agents drives interest in applying them to open-ended problems such as GPU kernel optimization. However, we find that existing coding agents converge to local optima on real-world, expert-written kernels (e.g., FlashInfer). We introduce Evolving Agent Teams, a multi-agent framework that escapes such local optima. It features (1) a central supervisor that diagnoses each local optimum and restructures a team of agents; and (2) a persistent file system that comprises a learning document, an evolution history, and per-team artifacts, grounding restructuring with accumulated experience. Under a matched token budget, Evolving Agent Teams consistently outperforms strong coding-agent baselines. With an extended token budget, Evolving Agent Teams generates a CUDA kernel that reaches 1.2x mean speedup over the production FlashInfer MLA paged decode kernel from scratch and wins on 36 of 47 production workloads from FlashInfer-Bench (0.86-0.99x on the remaining 11). It also demonstrates superior performance on tasks such as KDA prefill on AMD MI350X (3.01x over vLLM) and NVIDIA B200 (3.44x over FlashKDA), MoE expert-offload serving on Apple silicon (30.3 tokens/s decode for Qwen3.8-Flash-Next in MLX 4-bit on a 48 GB M4 Pro), and RoboDojo robot manipulation (62.1 on held-out seeds vs. 55.3 for the best leaderboard entry). Code is available at https://github.com/caoshiyi/EAT.

1 Introduction

Strong coding agents can make early progress on GPU kernels but often settle far below expert-written baselines. Evolving Agent Teams adapts its organization as optimization proceeds and outperforms strong baselines across four CUDA tasks.

1 Introduction

  • Strong coding agents often converge to local optima far below expert-written kernels after initially rapid progress.They may spend most of their budget tuning easy-to-implement but architecturally limited designs rather than revisiting harder alternatives.Source: §1 · p. 2
  • On MLA decode, Codex reached 28% of expert performance and a fixed four-role team reached 55%, illustrating the limits of stronger single agents and static teams.The kernel optimization process shifts from correctness and primitive debugging to profiling and architectural redesign, requiring different expertise and coordination.Source: §1 · p. 2
  • A central supervisor diagnoses plateaus and restructures agent roles, coordination, and knowledge management using accumulated experience in a persistent file system.The system can add specialists, profiling roles, or parallel search directions as needed.Source: §1 · p. 2
  • On MLA decode, the system explored over 100 team configurations and successively added WGMMA matrix multiplication, PV accumulation, and L2 prefetching.These architectural advances substantially outperformed prior single-agent and static-team approaches.Source: §1 · p. 2
  • With additional budget, Evolving Agent Teams reaches 1.21× mean speedup over the production FlashInfer kernel, faster on 36 of 47 workloads.Under a matched token budget, it reaches 0.74× the expert-written kernel, compared with 0.51× for Claude Code.Source: §1 · p. 2

2 Related Work

Prior agent frameworks often fix roles or optimize an agent or artifact within a set process. Evolving Agent Teams instead adapts the team and search process over time, retaining global experience across restructurings.

2 Related Work

  • Unlike frameworks that assign fixed software-engineering roles, Evolving Agent Teams changes roles and mandates as productive work shifts across multi-day optimization.Its supervisor also maintains a persistent global file system across team restructurings.Source: §2 · pp. 2–3
  • DYLAN selects roles from a hand-crafted prompt pool, while AGENTVERSE regenerates groups without preserving global context across recompositions.Evolving Agent Teams retains persistent context while restructuring.Source: §2 · p. 3
  • Reflexion and Voyager improve an agent within a fixed role, while population-based systems use program mutation; Evolving Agent Teams instead sequentially adapts team architecture through diagnosis.It uses no population or crossover and optimizes the team to support different stages of the optimization process.Source: §2 · p. 3
  • Meta-Harness optimizes the surrounding harness, whereas Evolving Agent Teams adapts the search process used to discover high-performance kernels.Inference-time scaling improves reasoning within a fixed search procedure, while this framework adapts the procedure itself.Source: §2 · p. 3

3 Method

Evolving Agent Teams co-adapts the agent-team configuration and the program being optimized, using a supervisor and persistent shared records to guide successive teams. Its MLA decode case study illustrates how changes to roles, coordination, and knowledge management respond to observed failures and produce further optimization gains.

3.2 Evolving Agent Teams

  • Evolving Agent Teams couples a supervisor that designs successive teams with configured coding agents that execute the optimization, while a persistent file system connects both levels.The system optimizes a program under correctness constraints and a finite token budget; existing approaches generally leave team configuration fixed.Source: §3.1 · pp. 3–4, §3.2 · p. 4
  • The supervisor can invent focused roles and choose pipelined or parallel coordination in response to observed successes and failures.Examples include profiling scientists, challengers, primitive-study agents, and knowledge integrators; the team can isolate debugging from main-branch optimization.Source: §3.2 · p. 4
  • A shared file system preserves learnings, per-team artifacts, and evolution history so later teams inherit prior failures and successful strategies.Agents use shell, compiler, and profiling tools, and their outputs become inputs to others according to the coordination plan.Source: §3.2 · p. 5

3.3 MLA Decode: A Case Study in Team Evolution

  • The first validated correct WGMMA kernel followed a redesign that added a disciplinarian role and persistent learnings after an agent abandoned WGMMA and skipped profiling.The trajectory records a change from 0.15 to 0.065 alongside the first correct WGMMA kernel.Source: §3.3 · p. 5
  • After repeated failed P@V WGMMA attempts, the supervisor split debugging and optimization across isolated parallel branches, and the optimizer advanced from 0.34 to 0.49 (+44%).The debugging branch confirmed the P@V attempt was a dead end, while the parallel pattern let the optimizer continue.Source: §3.3 · p. 5
  • A knowledge integrator and shared synthesis document recovered a prior WGMMA recommendation, enabled the first successful standalone P@V WGMMA test, and preceded a change from 0.50 to 0.51.Earlier teams had ignored the recommendation because it was not in shared learnings; the synthesis directly enabled V9’s later integration milestone.Source: §3.3 · p. 5
  • After V9 integrated P@V WGMMA with registers, the supervisor called for fresh profiling because the kernel’s bottleneck structure had changed.The subsequent trajectory reports a change from 0.50 to 0.66 (+32%) after re-profiling.Source: §3.3 · p. 6

4 Experiments

Across the four CUDA kernel tasks, Evolving Agent Teams outperforms strong coding-agent baselines under matched budgets and continues improving with additional budget. Further evaluations cover an already-optimized kernel, ablations, kernel-specific systems, and new hardware and domains.

4.1 Experimental Setup

  • The evaluation covers four GPU kernel optimization tasks across two architectures: FA4 targets the attention forward pass; GDN computes a prefill pass; MLA implements paged decode; and G+AR fuses GEMM with inter-GPU AllReduce.For each task, speedup is the geometric mean of baseline latency divided by optimized-kernel latency across workloads, relative to the production baseline. All settings are run twice, and the best result from each pair is reported.Source: §4.1 · p. 6

4.2 Budget-Controlled Progress Curves

  • 1.21× mean speedup over the production FlashInfer MLA decode kernel with additional budget; under matched budgets, Evolving Agent Teams outperforms Claude Code on all four tasks and matches or exceeds it on nearly every workload.The single exception is MLA decode in the workload-level comparison.Source: §4.2 · pp. 7–8, Figure 3 · p. 7

4.3 Scaling with Additional Budget

  • 4.06× mean speedup for GDN prefill across 21 team versions; G+AR reaches 1.56× over the cuBLAS+NCCL baseline, near ThunderKittens at 1.54×.The GDN worst workload improves to 0.40×, with 37 of 100 workloads below baseline; G+AR progresses from a two-phase kernel to a custom WGMMA kernel fused with NVLink multicast.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 Optimizing an Already-Optimized Implementation

  • 1.71× speedup over Triton on the default Flash-KMeans workload; the CUDA WGMMA kernel achieves up to 1.5× lower latency on compute-bound configurations and matches Triton in memory-bound regimes.The system rewrites an expert-optimized Triton assignment kernel in CUDA, using the Triton source as its reference.Source: §4.4 · pp. 8–9, Figure 5 · p. 9

4.5 Ablation Studies

  • On MLA decode, the evolving team reaches 0.74× versus 0.31× for the fixed four-role team and 0.28× for Codex; the fixed team cannot adapt its roles as the optimization landscape shifts.The comparison attributes the fixed-team gap to restructuring and coordination, including missing knowledge integration and challenge roles.Source: §4.5 · p. 9, Figure 6 · p. 9

4.6 Comparison with Kernel-Specific Systems

  • Under a matched $800 inference budget, Evolving Agent Teams outperforms K-SEARCH and CUDAFORGE in both runs, producing WGMMA-based kernels while they converge to WMMA and scalar/SIMT implementations.The comparison covers 47 production workloads on H100, and all kernels pass all 47 correctness tests.Source: §4.6 · pp. 9–10, Table 2 · p. 9

4.7 Surpassing State-of-the-Art Systems

  • The team reports 3.01× speedup over vLLM on AMD MI350X and 3.44× over FlashKDA on B200 for KDA prefill, plus 62.1 on held-out RoboDojo seeds versus 55.3 for the best leaderboard entry.Other cases include a Metal port serving recorded coding-agent sessions 3.32× faster on an M4 Pro; these new-setting runs are single-run evaluations.Source: §4.7 · p. 10, Figure 7 · p. 10

5 Conclusion

The paper concludes that evolving team organization helps coding agents escape local optima, with results across kernel optimization and other tasks. Its evaluation is limited by few runs and leaves cost-performance Pareto analysis and controlled studies of noisier domains for future work.

5 Conclusion

  • Evolving Agent Teams addresses coding agents’ tendency to plateau below expert-written baselines by having a supervisor diagnose plateaus and restructure roles using a persistent file system.The conclusion reports matched-budget gains across four CUDA tasks and a 1.21× mean speedup over the production FlashInfer MLA decode kernel with additional budget.Source: §5 · p. 10
  • The reported results extend beyond kernel optimization to KDA prefill, Apple-silicon MoE serving, and robot manipulation.The conclusion describes these as demonstrations of superior performance on those tasks.Source: §5 · p. 10
  • Each main setting is run twice and each Section 4.7 setting once, limiting statistical comparisons; cost-performance Pareto analysis and controlled studies of noisier domains remain future work.The paper identifies these as limitations and future work.Source: §5 · p. 10

A Observed Roles Invented by the Supervisor

The supervisor invents roles in response to diagnosed needs rather than selecting from a predefined role set.

A Observed Roles Invented by the Supervisor

B Case Study: Why the Fixed Four-Role Baseline Starts Slowly

The fixed four-role team reached its first correctness-passing MLA decode commit substantially later than the single agent. Its delay stemmed from a layout bug whose relevant specification and code were split across roles, while the team’s structure discouraged cross-agent inspection.

B Case Study: Why the Fixed Four-Role Baseline Starts Slowly

  • A low-level SMEM-layout mismatch delayed the fixed four-role team’s first correctness-passing MLA decode commit relative to the single agent.WGMMA on SM90 requires 8 × 64 atom-tiled shared memory, but the Implementer used row-major tiles, corrupting QK output.Source: B Case Study: Why the Fixed Four-Role… · pp. 14–15
  • The layout requirement and actual SMEM declaration were held by different agents, and neither inspected the Implementer’s source despite having read access.The Hardware Researcher held the requirement; the Implementer held the declaration. A single agent with both specification and code in context would have identified the discrepancy on inspection.Source: B Case Study: Why the Fixed Four-Role… · p. 15
  • Role charters, reactive spawn prompts, and prose-only messaging contributed to the debugging stall across role boundaries.The authors describe the failure as structural rather than capability-driven; Evolving Agent Teams responds by adding a Knowledge Integrator that mandates cross-agent artifact reads.Source: B Case Study: Why the Fixed Four-Role… · p. 15

C Details of the Additional Settings

The additional settings cover KDA prefill on AMD and NVIDIA GPUs, MoE expert-offload serving on Apple silicon, and RoboDojo manipulation. These are case studies with varied evaluation setups, including a run that began from an existing kernel and a robot policy assessed on held-out seeds.

Additional-setting evaluation

  • The four additional-setting runs used the main experiments’ supervisor prompt and harness, with no predefined roles; they are case studies rather than controlled comparisons.Costs are booked agent and supervisor API spend at published per-token prices.Source: C Details of the Additional Settings · p. 15

KDA prefill on AMD MI350X

  • 3.01× was the final KDA prefill score on AMD MI350X, the arithmetic mean of five re-measurements against vLLM’s production kernels.The workloads ranged from 1K–64K tokens; the team added a slow-decay check after finding that fast-decaying gates could let state-handoff bugs pass.Source: C Details of the Additional Settings · p. 15

KDA prefill on B200

  • 3.44× was the KDA prefill score over FlashKDA on B200 across six workloads.The run began by porting an existing kernel, so it measures improvement from a strong starting point rather than from scratch.Source: C Details of the Additional Settings · p. 15

MoE expert-offload serving on Apple silicon

  • 27.3 tokens/s was the final batch-1 decode rate for Qwen3.8-Flash-Next on a 48 GB M4 Pro, versus 13.4 for the naive engine.The task retained the model and math while streaming experts that did not fit in memory from SSD; candidates failing correctness, memory, swap-growth, or load-time gates scored zero.Source: C Details of the Additional Settings · pp. 15–16

RoboDojo

  • 62.1 was the RoboDojo policy’s held-out score, compared with 62.6 on development tasks.The score averages five task-dimension means; seeds 0 and 2 were held out and evaluated only by the operator, while agents developed on seed 1.Source: C Details of the Additional Settings · p. 16

D Matched-Budget Solo Ablations

Matched-spend solo comparisons favored the team on MI350X KDA prefill, while the optimized-start offload comparison was close at $200. Over the same 20-hour window, the team continued improving and finished with the fastest engine on each reported offload workload.

KDA prefill on AMD MI350X

MoE expert-offload serving, from an optimized start

Loading 2610.36848v1…