PaperlayerView exact revision on arXiv

Source-linked AI summary

InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?

Yangshen Deng, Jiaqi Ruan, Yeqi Huang, Nick Lau, Guangming Sheng, Luo Mai

arXiv:2610.36589v1cs.AIcs.DC

TL;DR

Existing benchmarks cover narrow kernel or single-GPU tasks and may misjudge patches because tests and prompts are incomplete. InfraDev benchmarks 36 real infrastructure changes and jointly synthesizes hidden probes and prompts. Across six frontier models, the best resolves 38.9% of tasks, and an agent skill improves a model’s resolved rate on Megatron-LM tasks.

  • Problem

    Existing benchmarks cover mainly individual kernels or single-GPU tasks, while PR tests and descriptions can fail to check important requirements or state needed interfaces.

    Source: §1 · p. 1, §3.1.2 · p. 4

  • Method

    InfraDev builds 36 tasks from real pull requests and jointly synthesizes hidden probes and prompts, alongside a shared environment for dependencies and artifacts.

    Source: §1 · p. 2

  • Results

    38.9% of InfraDev tasks are resolved by the best of six frontier agents, and hidden probes reject 32% of submissions that pass all official tests.

    Source: §1 · p. 2, §4.3 · p. 7

  • Takeaways & Limitations

    The agent skill raises GPT-5.6 Sol’s resolved rate on Megatron-LM tasks from 42.1% to 52.6%.

    Source: §1 · p. 2

    Each agent run can occupy an eight-GPU node for up to 2.5 hours, and PR curation still depends on domain experts selecting tasks by hand.

    Source: §5 · p. 9

Abstract

from arXiv · show

Coding agents are beginning to develop the production-grade AI infrastructure that trains and serves large models, yet existing benchmarks cover only single kernels or single-GPU tasks, and grade with tests and prompts that misjudge both correct and incorrect patches. We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning. Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs. To grade such changes, we jointly synthesize hidden probes, which enforce the requirements the original tests leave unchecked, and a prompt that states every interface the probes call without revealing the solution. We also design a shared execution environment that reuses dependencies and artifacts across tasks, cutting storage by 52-57% and build time by 13-14% on Megatron-LM and vLLM. Across six frontier models, the best resolves only 38.9% of the tasks; the probes reject 32% of the submissions that pass every official test, and with the original pull request descriptions as prompts, no model resolves a single task. Failure analysis shows that structural and state errors dominate and that agents' own tests rarely check memory, time, or traffic. Turning these findings into an agent skill raises the resolved rate of a frontier model on the Megatron-LM tasks by 10.5%.

1 Introduction

InfraDev targets a gap in evaluating agents on real AI infrastructure: existing benchmarks and PR-based grading miss multi-file, multi-GPU work and can accept incorrect patches or reject correct ones. It combines a 36-task benchmark with jointly synthesized tests and prompts, a shared execution environment, and analyses of agent failures and improvement.

1 Introduction

  • Existing evaluations mostly cover individual kernels or single-GPU optimization, while PR tests and descriptions can miss numerical or behavioral requirements and omit interfaces needed for correct implementations.Across the studied PRs, 24 have no numerical-result check, and 13 of 15 performance optimizations have no test of the promised saving.Source: §1 · pp. 1–2
  • InfraDev’s 36 tasks reimplement merged pull requests to Megatron-LM, vLLM, and verl, averaging 17.4 files and 1,818 lines changed, with 20 requiring multiple GPUs.The tasks cover training, inference, and reinforcement learning.Source: §1 · p. 2
  • Hidden probes enforce requirements missed by official tests, while synthesized prompts state every probed behavior and interface without revealing the solution.The probes reject 32% of submissions that pass all official tests; with original PR descriptions as prompts, no model resolves a task.Source: §1 · p. 2
  • The shared execution environment cuts storage by 52–57% and build time by 13–14% on Megatron-LM and vLLM compared with one image per task.Source: §1 · p. 2
  • The best of six frontier agents resolves only 38.9% of tasks, while analyses find structural errors most common, state errors escaping official tests, and self-written tests rarely checking resource savings.The agent skill raises GPT-5.6 Sol’s Megatron-LM resolved rate from 42.1% to 52.6%.Source: §1 · p. 2

2 Related Work

Prior benchmarks evaluate kernels, inference optimization, or ML engineering rather than agents changing production AI infrastructure across multi-file, multi-GPU systems. InfraDev differs in its pull-request task unit and jointly addresses test coverage, prompt completeness, and execution-resource reuse.

2 Related Work

  • InfraDev evaluates reimplementation of merged pull requests in production training, inference, and reinforcement-learning frameworks, unlike benchmarks that use fixed-target optimization tasks or have agents use rather than change infrastructure.Source: §2 · pp. 2–3
  • Earlier repository-level grading commonly relies on PR tests; prior work strengthens tests or enriches prompts separately, whereas InfraDev jointly uses adversarial patches, hidden probes, and prompts listing required interfaces.Source: §2 · p. 3
  • InfraDev’s skill encodes three failure classes and targets skipped checks such as callers of changed functions and promised resource savings, unlike general agent-improvement approaches.Source: §2 · p. 3

3 InfraDev

InfraDev evaluates coding agents on real, multi-file AI-infrastructure changes and combines official tests with jointly designed hidden probes and prompts. Its evaluation uses a shared execution environment to reuse dependencies and artifacts across tasks.

3 InfraDev

  • InfraDev reimplements real pull requests in a multi-GPU sandbox and grades agent patches with both official tests and hidden probes.The task prompt states what the tests check without revealing the solution.Source: §3 · p. 3

3.1.1 PR Curation

  • The benchmark curates 36 merged pull requests from Megatron-LM, verl, and vLLM, classifying changes as features, performance improvements, refactors, or bug fixes.Each task uses the pre-PR snapshot as its starting point and the developers’ change as the golden patch.Source: §3.1.1 · p. 4, Table 1 · p. 4

3.1.2 Joint Synthesis of Tests and Prompt

  • Hidden-probe synthesis targets numerical and behavioral requirements that official tests leave unchecked, using adversarial patches that pass those tests but violate the PR’s promises.Prompts state required interfaces and behaviors without giving the solution; probes are checked against the golden patch and adversarial patches.Source: §3.1.2 · pp. 4–5

3.2 Evaluation

  • Patches are graded with official tests and hidden probes; a task is resolved when its patch passes at least as many test cases as the golden patch.Missing or empty patches score zero, while harness faults are tracked separately from model failures.Source: §3.2 · p. 5

3.3 Execution Environment

  • The shared execution environment reuses packages and artifacts independently, avoiding repeated dependency builds and downloads across tasks.Tasks with identical locked dependencies share an environment, and containers mount a shared Hugging Face cache.Source: §3.3 · pp. 5–6

4 Experiment

Across five experiments sections, InfraDev finds that agents resolve few infrastructure tasks, while hidden probes expose failures missed by official tests. The analysis identifies recurring failure patterns and tests a reusable skill for improving agents.

4.1 Setup

  • Agents ran once per task, with up to 2.5 hours on nodes containing eight H800 or H20 GPUs.Each grading or inference run had exclusive use of a node, and golden calibration used the same environment and GPU type.Source: §4.1 · p. 6

4.2 Results on InfraDev

  • Resolved rates range from 25.0% for GLM 5.3 to 38.9% for GPT-6 Sol; inference is hardest, with every model resolving only one of ten tasks.No model leads across all categories: GPT-6 Sol leads on training, while HY4 preview and Kimi K3 tie on RL.Source: §4.2 · p. 6

4.3 Ablation Study

  • Hidden probes lower every model’s resolved rate by 11–19 percentage points and reject 32% of submissions accepted by official tests.Figure 2 compares resolved rates on the 36 tasks with and without hidden probes.Source: §4.3 · p. 7, Figure 2 · p. 7
  • With original PR descriptions and hidden probes, neither GPT-6 Sol nor DeepSeek V4 Pro resolves a Megatron-LM task.Figure 3 compares resolved rates on the 19 Megatron-LM tasks under original-description prompting.Source: §4.3 · p. 7, Figure 3 · p. 7
  • The shared execution environment cuts storage by 57% on Megatron-LM and 52% on vLLM, and build time by 14% and 13%, respectively.Against one Docker image per task, storage savings grow as more tasks share the environment.Source: §4.3 · p. 7

4.4 Failure Analysis

  • Structural errors affect 48% of submissions; numerical errors stem from wrong algorithms, while condition-specific behavioral errors occur in 60% of CUDA-graph-task submissions versus 15% on other tasks.State errors often escape official tests: hidden probes alone catch them in 15 of 35 submissions with a state error. Figures 4 and 5 show the error classes and condition-specific errors by task.Source: §4.4 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
  • Fewer than 2% of agents’ 5,532 test assertions check resource usage, and only one of 11 traced runs failing a resource check wrote any resource assertion.Figure 6 classifies what the agents’ own tests check.Source: §4.4 · p. 8, Figure 6 · p. 8

4.5 Improving Agents with Reusable Skills

  • The reusable skill raises GPT-5.6 Sol’s resolved rate on Megatron-LM tasks from 42.1% to 52.6%, a gain of 10.5 percentage points.The skill asks agents to check callers and state, compare computations with a naive reference, and test resource requirements and CUDA-graph behavior.Source: Figure 7 · p. 8, §4.5 · p. 9

5 Conclusion and Future Work

The paper finds that current agents remain unreliable on real infrastructure tasks, while jointly written hidden tests and prompts provide a more reliable grading approach. It identifies recurring failure modes and future scaling challenges for InfraDev.

5 Conclusion and Future Work

  • The best frontier agent resolves fewer than two in five tasks, and most failures are shared across agents.The paper also finds that grading is reliable only when hidden tests and the prompt are written together.Source: §5 · p. 9
  • Agents most often write code that does not fit the surrounding program, and rarely test promised memory, time, or traffic savings.When agents compute wrong values, the paper attributes errors more often to the algorithm than to precision.Source: §5 · p. 9
  • The proposed agent skill targets these weaknesses and improves the resolved rate.Scaling InfraDev remains constrained by costly multi-GPU execution and manual PR curation.Source: §5 · p. 9

A Details of Test and Prompt Synthesis

The appendix details how probes are designed and accepted: they check numerical or behavioral requirements and must accept the golden patch while rejecting known incorrect alternatives.

A.1 Hidden-Probe Synthesis

  • Numerical probes check outputs against references, invariances, hand-derived values, or properties; behavioral probes check promised effects such as communication or memory savings.Examples include cached-state comparisons, chunk-size invariance, activation-offload bytes, and elements received by each rank.Source: §A.1 · p. 13
  • Every probe must reject the pre-PR snapshot, the hollow patch, and each adversarial patch.The hollow patch preserves the golden patch's interfaces but makes every PR-added function raise an exception.Source: §A.1 · p. 13
  • Probe acceptance runs use the same GPU count as scoring, and a PR is left without probes if they fail to pass within a fixed number of rounds.The acceptance procedure checks the golden patch, pre-PR snapshot, hollow patch, and adversarial patches.Source: §A.1 · p. 13
  • The acceptance check requires every probe to pass on the golden patch in three consecutive runs.A golden-patch pass counts only when no case is skipped.Source: §A.1 · p. 13
  • The acceptance check also requires adversarial patches to score no worse than the golden patch on official tests.This verifies that the adversarial patches can pass the tests they are meant to fool.Source: §A.1 · p. 13
  • A probe set is accepted only when it passes all acceptance conditions: golden-patch success, rejection of incorrect patches, and adversarial patches that pass official tests.The check returns true only when all three conditions hold.Source: §A.1 · p. 13

B Task Prompt Template

The task prompt template describes the required change while withholding its solution, and supplies the interface and execution details needed to implement and validate it. It directs agents to run the code and test it on the grading hardware.

B Task Prompt Template

  • Figure 8 gives the specification, Figure 9 requires an autonomous run without looking up the original PR, Figure 10 gives the interface contract, and Figure 11 states execution rules.The interface contract lists names, signatures, shapes, and configuration knobs called by tests, but not algorithms.Source: B Task Prompt Template · p. 13, Figure 8 · p. 15, Figure 9 · p. 15
  • The rewritten task description states what the change does, why it matters, its scope, and the definition of done, while withholding algorithms, formulas, exact numeric behavior, and call signatures.The exact import and call surface pinned by tests appears separately in the interface contract.Source: B Task Prompt Template · pp. 13–14
  • The template covers the task’s what and why, before-and-after data flow, change scope, required behaviors, and decisions left to the agent.It also identifies regression behavior that must remain intact and the tests’ high-level coverage.Source: B Task Prompt Template · p. 14
  • The prompt requires agents to execute the code; matching the interface contract without running it is not sufficient.A syntax or indentation error in a shared module can prevent test collection and cause the entire suite to score zero.Source: B Task Prompt Template · p. 14
  • The first validation gate is clean imports and test collection, which agents must rerun after edits to shared files.The template directs agents to stop and fix errors at this gate before continuing.Source: B Task Prompt Template · pp. 14, 16
  • The prompt requires validation inside the real repository test image, with the attached GPUs and mounted test data.Agents must confirm that imports resolve to the working tree they are editing.Source: B Task Prompt Template · p. 16
  • After single-process forward-and-backward checks, agents must run distributed checks at the grader’s rank count because single-rank runs do not establish parallel behavior.The prompt specifies per-test timeouts for the multi-rank run.Source: B Task Prompt Template · p. 16

C Detailed Performance Tables

The section’s tables cover resolved rates under two test suites, per-repository results, and token-use components; the supplied passages describe table scope but give no cell values.

C Detailed Performance Tables

D Failure Analysis: Error Types and Labelling

The analysis examines failing cases across 180 submissions and classifies errors by how patches fail to meet task requirements. It distinguishes structural, behavioral, and numerical failures and explains how submissions are counted and labelled.

D Failure Analysis: Error Types and Labelling

  • 1,754 failing test cases were analyzed across 180 submissions from five models, grouped by task, test file, error signature, and label.A failing case is one that the golden patch passes but the agent’s patch does not.Source: D Failure Analysis: Error Types and… · p. 19
  • Structural errors include execution or API failures, mismatched tensor shape, layout, or dtype, and mishandled persistent state or CUDA-graph lifecycle.These errors prevent code from fitting the surrounding program or its expectations.Source: D Failure Analysis: Error Types and… · p. 19, Table 7 · p. 20
  • Behavioral errors compute correct values but miss resource requirements or fail under particular conditions, such as a mode, boundary input, fallback, or error path.Resource requirements can concern memory, speed, communication traffic, or liveness.Source: D Failure Analysis: Error Types and… · p. 19
  • Numerical errors occur when code runs and fits the program but computes incorrect values, either from a wrong algorithm or insufficient precision despite correct logic.Precision-only errors are assigned only when the same patch passes an exact-logic check.Source: D Failure Analysis: Error Types and… · p. 19
  • Table 7 counts submissions once per error type, allows multiple types per submission, and labels non-exception failures by the property checked by the assertion.Exceptions and collection failures are classified as structural.Source: Table 7 · p. 20
Loading 2610.36589v1…