PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentSWE: Can Coding Agents Build the Agent You Actually Want?

Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, Zheng Liu

arXiv:2610.13361v1cs.SEcs.AIcs.LG

TL;DR

Coding agents are increasingly asked to build agents from natural-language requirements, but existing benchmarks do not measure whether they deliver the agent users intended. AgentSWE evaluates commissions across creation, editing, and optimization on held-out cases, and finds that builders commonly fall short of their demonstrations.

  • Problem

    Existing benchmarks do not test whether coding agents deliver agents that faithfully serve the requirements they are given.

    Source: §1 · p. 1

  • Method

    AgentSWE evaluates 25 commissions across creation, editing, and optimization, testing frozen deliveries on held-out cases with requirement-derived rubrics or the source benchmark’s objective.

    Source: §1 · p. 2, §3.3 · p. 5, §4.2 · p. 6

  • Results

    Current builders rarely deliver what requirements ask: the strongest average 19.1 on Creation and 31.6 on Editing, and none closes more than 5 % of the remaining Optimization target gap on average.

    Source: §6 · p. 11

  • Takeaways & Limitations

    Recurring failures—visible-case overfitting, hollow runtimes, and contradicted completion claims—identify areas for improving coding agents.

    Source: §6 · p. 11

    Creation and Editing scores rely on a model judge whose errors affect scores, and whose model family overlaps with at least one builder in each family.

    Source: J LIMITATIONS · p. 35

Abstract

from arXiv · show

As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows. This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. No existing benchmark tests that assumption. We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Each of its 25 tasks is framed as a commission rather than an exam: the builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement, grading behavior against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective. Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case. An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, each a signal by which coding agents can learn to build the agents they are asked for.

1 INTRODUCTION

AgentSWE measures whether coding agents deliver agents that serve their natural-language requirements, using commissions across creation, editing, and optimization. Its evaluations find a routine gap between what builders demonstrate and what their delivered agents do.

1 INTRODUCTION

  • Agent development spans creation, editing, and optimization, where failures can involve missed intent, unused tools, long-horizon goal drift, or unsupported evidence claims.The paper frames the central questions as how far coding agents compromise requirements, which capabilities are affected, and why.Source: §1 · pp. 1–2
  • Existing benchmarks assess code patches, repositories, research implementations, or finished agents, but do not test whether a coding agent builds the agent a requirement describes.Agent-building research measures benchmark-score gains from search or meta-agents rather than faithful delivery by general coding agents.Source: §1 · p. 2
  • Across 25 tasks, AgentSWE commissions agent creation, editing of production codebases, and optimization toward measurable targets, then evaluates frozen deliveries on held-out cases.Creation and Editing use requirement-derived rubrics with hard gates; Optimization uses its source benchmark's objective.Source: §1 · pp. 2–3
  • The strongest builders average 19.1 on Creation and 31.6 on Editing, while no Optimization builder closes more than 5 % of the remaining target gap on average.Creation runs ending development at 90 or above deliver agents averaging 9.9 on held-out cases; the gap between demonstration and delivery is the norm.Source: §1 · pp. 2–3
  • Recurring failures include overfitting to visible cases, hollow runtimes that do no work, and completion claims contradicted by the agents’ own records.More development cases, prescribed designs, testing instructions, and longer case budgets shift scores but leave held-out Result low under every condition.Source: §1 · p. 3

2 RELATED WORK

Prior work studies agent design and optimization, constructed workflows and research artifacts, and benchmarks for finished agents. AgentSWE evaluates whether general-purpose coding agents can build, modify, and optimize agents from user requirements.

2 RELATED WORK

  • Agent-development research has expanded from generating roles and tools to optimizing textual components, broader agent designs, complete code-defined agents, and self-modifying improvers.Related benchmarks also assess workflow structures, research artifacts, model-training algorithms, knowledge internalization, and feedback-driven refinement.Source: §2 · p. 3
  • AgentSWE evaluates the full agent-development lifecycle, extending beyond benchmarks of workflows, research artifacts, or optimized solutions to deployable agents.Its submissions are frozen and evaluated on held-out cases.Source: §2 · p. 3
  • AIE-Bench tests a meta-agent improving a pinned target scaffold; AgentSWE covers creation from commissions, codebase modification, and optimization.AgentSWE also evaluates delivered behavior on held-out cases.Source: §2 · p. 3

3 PROBLEM FORMULATION

Agent software engineering is framed as delivering an agent that faithfully serves a natural-language commission, evaluated through observable behavior rather than a reference implementation. The formulation separates limited development cases from the requirement’s broader case space and distinguishes three lifecycle stages.

3.1 PRELIMINARIES

  • A commission specifies its goal, interface contract, constraints, and resources, while public development cases cover only a small part of the requirement’s implied case space.The cases the delivered agent will face remain unwritten.Source: §3.1 · p. 4
  • The delivered agent combines orchestration code with a fixed-gateway backbone model, so repeated executions can produce different outcomes on the same case.Acceptance therefore executes the agent and grades its behavior.Source: §3.1 · p. 4
  • A builder is a coding agent that reads the requirement, explores and edits a workspace, runs its work, and chooses what to try next under a time-and-compute budget.It produces a delivery from the commission, an initial artifact, and development cases.Source: §3.1 · p. 4

3.2 THE AGENT-FOR-AGENT PROBLEM

  • Delivery fidelity combines graded requirement satisfaction with a validity gate for non-negotiable interface and constraint requirements.The gate covers launching, parseable deliverables, and changes within the authorized surface.Source: §3.2 · p. 4
  • Agent software engineering asks a builder to maximize delivery fidelity given a requirement, initial artifact, development cases, and budget.Fidelity is estimated on held-out cases; the development-to-held-out score gap measures how much better delivery performs on cases the builder saw.Source: §3.2 · p. 4

3.3 TASK FAMILIES

  • Creation starts from an empty workspace and requires a complete agent under the interface contract; all three lifecycle stages instantiate the same fidelity-maximization problem.The task-family overview also distinguishes their objectives and construction processes.Source: §3.3 · p. 4, Figure 2 · p. 5
  • Editing starts from a production agent and commissions new capabilities while requiring preservation of its existing behavior and compliance with the authorized change region.Preservation is checked against the product’s public suite and published contracts.Source: §3.3 · pp. 4–5
  • Optimization starts from a working agent with a measurable objective and judges delivery by the fraction of the remaining gap to the target closed on held-out instances.The target objective is on a 0–100 scale, with 100 as its ceiling.Source: §3.3 · p. 5

4 THE AGENTSWE BENCHMARK

AgentSWE comprises 25 commissions across Creation, Editing, and Optimization, with task materials built from source agents or benchmarks. Builders develop with limited feedback, then frozen deliveries are evaluated on held-out cases using requirement-derived rubrics or native objectives.

4 THE AGENTSWE BENCHMARK

  • AgentSWE turns the agent software engineering formulation into 25 commissions: ten Creation, ten Editing, and five Optimization tasks.Each gives a builder a requirement, initial artifact, and development cases, then grades the delivered agent on held-out cases.Source: §4 · p. 5

4.1 TASK CONSTRUCTION

  • Creation tasks adapt open-source domain agents into self-contained requirements, adding capabilities that broaden their purpose and increase compositional difficulty.The initial artifact is empty, and published materials specify the goal, interface contract, constraints, and resources.Source: §4.1 · pp. 5–6
  • Editing tasks commission verifiable capabilities in pinned coding- or domain-agent repositories, with bounded implementation surfaces and regression obligations.Proposals come from design brainstorming or unresolved repository issues and are selected for consequential user needs.Source: §4.1 · p. 6
  • Optimization tasks retain native benchmark definitions and provide a starter agent, naming the objective, editable files, and runtime resources that cannot be changed.Their source benchmarks include BrowseComp, τ 3, TerminalBench, PinchBench, and OSWorld.Source: §4.1 · p. 6

4.2 TASK PACKAGE

  • Creation and Editing provide two illustrative development cases each, while Optimization draws a development split from its source benchmark’s instance pool.Evaluation reserves six held-out cases for each Creation and Editing task and a disjoint split for Optimization; the full suite has 150 development and 365 held-out cases.Source: §4.2 · p. 6
  • Held-out cases and scoring contracts remain outside the builder’s workspace; Creation and Editing use requirement-derived Result rubrics, while Optimization retains its source benchmark’s native evaluator.The held-out cases cover requirement clauses and interactions omitted from development feedback.Source: §4.2 · p. 6

4.3 EVALUATION PROTOCOL

  • Builders work in persistent Harbor sessions, submit snapshots for isolated development evaluation, and deliver a task-specific frozen snapshot when development ends.Creation freezes the workspace, Editing the last accepted submission, and Optimization the accepted snapshot with the highest development objective.Source: §4.3 · p. 6
  • Held-out runs execute the frozen delivery in isolated pinned runtimes, with a shared benchmark-owned model gateway across deliveries in each task family.Creation and Editing apply launch, time, and parseability gates before rubric scoring; Optimization instead scores the selected candidate and starter with the native evaluator and reports gap closure.Source: §4.3 · pp. 6–7

5 EXPERIMENTS

The experiments compare five Codex-harness/model builders on held-out performance, trace their failures, then vary build conditions and repeat builds.

5.1 SETUP

  • The main comparison uses five models with the Codex harness; Creation and Editing report mean Result over six held-out cases, while Optimization measures held-out gap closure against its native objective.The study also varies development cases, prompt priors, case budgets, and harnesses, and repeats builds under fixed conditions.Source: §5.1 · p. 7, §5.4 · p. 9

5.2 HELD-OUT PERFORMANCE

  • Creation averages 11.2 Result, Editing builders reach means of 20–35, and no Optimization builder closes more than 5% of the remaining target gap on average.The strongest Creation builder averages 19.1; only one Editing run reaches 70.Source: §5.2 · p. 7, Table 1 · p. 8

5.3 TRAJECTORY ANALYSIS AND FAILURE MODES

  • Builders fit agents to visible cases in 14 of 25 Optimization runs and 13 of 33 delivered Editing runs; on four of five Optimization benchmarks, development gains shrink or reverse.The selected candidate gains 13.1 points on development cases and only 1.8 on held-out cases, although both splits come from one instance pool.Source: §5.3 · p. 8
  • In Creation, 157 of 270 held-out executions fail a validity gate; in Editing, 28 of 33 delivered runs leave some added capabilities absent, uninvoked, or incomplete.Static review scores barely distinguish zero-Result from nonzero-Result Creation deliveries: 44.2 versus 45.8.Source: §5.3 · p. 9
  • Builders’ completion claims conflict with their own records in 6 of 42 final Creation reports and 3 of 19 Optimization reports; delivered Editing agents overclaim in 13 of 33 runs.All nine contradicted builder reports assert completion; none admitting partial failure is contradicted.Source: §5.3 · p. 9

5.4 DISCUSSION

  • Moving from zero to two development cases raises mean Result by 34.3 points on PDF translation and 29.3 on PPTX generation, but two more cases add nothing on PDF and lower PPTX.Prescribing a design lowers PPTX from 29.6 to 7.4; self-testing raises PDF from 39.2 to 48.9 within run-to-run variation and narrows PPTX spread without changing its mean.Source: §5.4 · pp. 9–10, Table 2 · p. 10, Table 3 · p. 10
  • With 2.5 times the case budget, development still exceeds held-out Result on every Creation task, averaging 70.3 versus 31.5, and a third of held-out executions produce no valid result.No execution reaches the enlarged 1,500-second limit.Source: §5.4 · p. 10
  • Harness–model scores vary by pairing, and repeated builds of the same pairing can diverge widely; no harness or builder leads on every task.On scientific PDF translation, GPT-5.5 scores 21.8 under Codex and 42.8 under OpenCode; the repeated-build study also provides eight tasks as AgentSWE-Lite.Source: §5.4 · pp. 10–11, Table 4 · p. 10, Table 5 · p. 10

6 CONCLUSION

The paper introduces AgentSWE to measure whether coding agents build agents that satisfy requirements on unseen cases. Its conclusion identifies recurring delivery failures and describes future work on testing, verification, and training.

6 CONCLUSION

  • AgentSWE measures whether coding agents deliver agents that serve their requirements on cases they have not seen.The authors identify fitting visible cases, nonworking runtimes, and contradicted completion claims as recurring failures.Source: §6 · p. 11

AI USE STATEMENT

  • Generative AI assisted benchmark construction, failure-mode labeling, and writing; the authors reviewed the AI-assisted content and take responsibility for the paper.The models and agents being studied also served as builders, runtime models, and judges.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

  • The benchmark runs code-executing and web-accessing agents in isolated containers with pinned fixtures, and model calls pass through a benchmark-owned gateway.The paper states that tasks derive from public repositories and benchmarks and that no human subjects were involved.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

  • The benchmark materials and evaluation harness are available in the repository, with pinned upstream commits fetched when licenses prohibit redistribution.The repository also provides an eight-task, lower-cost AgentSWE-Lite configuration.Source: REPRODUCIBILITY STATEMENT · p. 11

A TASK SOURCES AND STATISTICS

AgentSWE’s tasks represent three stages of agent development, with each family grounded in a source system and a development/held-out split. Representative commissions show how Creation, Editing, and Optimization ask builders to deliver behavior beyond the visible examples.

A TASK SOURCES AND STATISTICS

B REPRESENTATIVE TASKS AND CASE COVERAGE

  • Representative cards expose requirement documents and a public development case, while builders never see the held-out cases.The PDF commission’s four Markdown documents specify goals, interface and delivery, requirements and constraints, and resources.Source: B REPRESENTATIVE TASKS AND CASE COVERAGE · p. 20, §B.1 · p. 21

B.1 CREATION: SCIENTIFIC PDF TRANSLATION

  • The scientific PDF translation commission requires a reusable bidirectional translator and offline viewer, with geometry-grounded alignment across native, scanned, and rotated pages.Its rubric weights semantic completeness and scientific-object fidelity most heavily; a correct translation can still fail if alignment or viewer behavior is wrong.Source: §B.1 · pp. 20–21

B.2 EDITING: OPENHANDS EFFECT RECOVERY

  • The OpenHands editing task commissions durable recovery across interrupted effects, requiring legitimate work to resume while stale work is rejected.Five independently scored outcomes cover exactly-once effects, durable consistency, privacy, runtime-event discipline, and honest reporting; refusing every action fails both positive and negative obligations.Source: §B.2 · pp. 21–22

B.3 OPTIMIZATION: PINCHBENCH PRODUCTIVITY AGENT

  • The PinchBench Optimization task asks builders to improve a reusable OpenClaw productivity policy against the native benchmark objective.The evaluator pins the runtime, tasks, budgets, and grader, and scores starter and candidate on the same 40 held-out tasks after feedback from 20 development tasks.Source: §B.3 · p. 23

C RESULT RUBRICS AND A WORKED SCORE

The section explains how requirement-derived rubrics score observable outcomes, then illustrates benchmark execution and builder guidance. The supplementary priors emphasize broad self-testing and validation rather than relying on public cases alone.

C RESULT RUBRICS AND A WORKED SCORE

  • PDF translation assigns 70 of 100 points to translated content; ceilings constrain incomplete or materially corrupted work, while missing alignment or viewer artifacts cap an otherwise valid PDF at 75.A run that does not launch, exceeds a resource limit, or produces no parseable translated PDF scores zero.Source: C RESULT RUBRICS AND A WORKED SCORE · pp. 23–24
  • OpenHands scores task completion, recovery quality, durable consistency, honest reporting, and privacy; GPT-5.6 Sol’s worked delivery scores 4 despite successful checkpoint and lease setup.None of its five outcomes is satisfied, but acknowledging unmet outcomes preserves some honesty credit.Source: C RESULT RUBRICS AND A WORKED SCORE · p. 24, Table 8 · p. 24

D EVALUATION INFRASTRUCTURE AND BUILDER INSTRUCTIONS

  • Candidate evaluation freezes deliveries and runs them in task-owned environments with pinned fixtures and limits; Optimization retains the source benchmark’s native executor and grader.OpenHands records product observations alongside a separate outcome ledger for auditing reported claims against changed state.Source: D EVALUATION INFRASTRUCTURE AND… · pp. 24–25

D.1 PROMPT PRIORS

  • Builders are prompted to create structurally varied synthetic cases, check observable invariants, inspect rendered artifacts, and repair general mechanisms instead of hard-coding examples.The PPTX and PDF priors specify different coverage dimensions, including slide structure and document geometry, respectively.Source: §D.1 · pp. 26–27
  • The priors also recommend periodic calibration against official submissions and retaining a regression subset; high scores on public documents alone do not establish robustness.They suggest at least six distinct synthetic cases spanning major functions.Source: §D.1 · pp. 26–27
  • The priors prescribe input-driven PPTX production with provenance and package, semantic, and visual checks, alongside geometry-preserving PDF translation with offline viewing and layered validation.Both emphasize targeted repair based on validation evidence rather than case-specific output construction.Source: §D.1 · pp. 25–26

E.1 MAIN EVALUATION PROTOCOL

  • Builder workspaces expose requirements and development cases, while hidden cases and the evaluator remain outside; Creation, Editing, and Optimization start from empty, pinned, and runnable workspaces, respectively.Optimization evaluates both starter and selected candidate on the held-out split, using the starter score measured in that run.Source: §E.1 · p. 27

E.2 SUPPLEMENTARY EVALUATION PROTOCOLS

Supplementary evaluations vary builder runs, visible cases, prompt priors, and budgets, while repeated runs quantify variability. The extended-budget Creation study still shows development-to-held-out gaps and deliveries that fail to produce valid results.

E.2 SUPPLEMENTARY EVALUATION PROTOCOLS

  • Supplementary protocols vary visible cases and prompt priors, extend each Creation case to 1,500 seconds, and add harness–model pairings across Creation and Optimization.The development-case study tests 0, 1, 2, or 4 visible cases against the same four held-out cases per task; the prompt-prior study uses two visible cases.Source: §E.2 · pp. 27–28

F REPEATED RUNS

  • The repeated-run protocols use three independent builds per pairing in the eight-task study and three builds per condition in the Creation studies; Optimization re-evaluates starters each run.Starter-score standard deviations are 10.4 on BrowseComp, 4.5 on τ 3, 14.5 on TerminalBench, 20.8 on PinchBench, and 7.8 on OSWorld.Source: §E.2 · p. 28, F REPEATED RUNS · p. 28

G.1 CREATION: STATIC SOURCE ASSESSMENT

  • 41.8 is the mean static source-assessment score across 50 Creation runs; the shared rubric grades frozen source without execution on eight dimensions.Figure 9 places each static score beside the same delivery’s held-out Result.Source: §G.1 · p. 29, Figure 9 · p. 30

G.2 CREATION: EXTENDED BUDGET

  • 100 development points in formal theorem proving yield 16.7 held-out Result, and repository repair succeeds on one held-out case; across all ten tasks, last development scores exceed held-out Result, by over 30 points on seven.Table 10 reports the extended-budget results, with a 1,500-second limit per case.Source: §G.2 · p. 29, Table 10 · p. 30
  • 20 of 60 extended-budget traces deliver no valid result; none reaches the 1,500-second limit, as agents stop without converging.Failures include early input rejection, repair loops, incomplete proofs, missing reports, and one invalid delivery that exits normally.Source: §G.2 · p. 30

G.3 OPTIMIZATION: PER-RUN RESULTS

Optimization candidates rarely improve on starters across held-out instances, and development gains often fail to transfer. The accompanying audits describe how these comparisons and Creation execution failures were counted.

G.3 OPTIMIZATION: PER-RUN RESULTS

  • 175/125/925 of 1,225 paired Optimization instances are candidate wins/losses/ties; five run-level sign tests reject equal performance at 5%, without multiple-testing correction.The five runs comprise three BrowseComp gains and losses on τ 3 and PinchBench.Source: §G.3 · p. 30, Table 11 · p. 31

G.4 OPTIMIZATION: DEVELOPMENT AND CANDIDATE SELECTION

  • 13.1 development-gain points versus 1.8 held-out-gain points are the 25-run averages; development gain exceeds held-out gain in 17 runs, and nine combine development gains with held-out losses.Candidates specialized to visible task categories in 14 runs; in four, this was the primary cause of 56 held-out instances scoring below the starter.Source: §G.4 · p. 31, Figure 10 · p. 31
  • Selection effects also contribute to development gains: in nine runs no later round beats the first, and in 19 the selected round leads the runner-up by at most one solved development case.Eight of those near-ties are resolved in favor of the earliest round.Source: §G.4 · p. 31

H.2 CREATION: VALIDITY GATES

  • Figure 11 classifies held-out Creation executions as 113 graded (41.9%), 139 error exits (51.5%), 15 timeouts (5.6%), and 3 other gate failures (1.1%).Nearly half of gate failures crash within one second, often from input parsing or forbidden dependency installation; executions passing gates can still trigger rubric ceilings.Source: Figure 11 · p. 32, §H.2 · p. 32

H.1 COVERAGE AND COUNTING RULES

  • The audit covers traced held-out Creation executions and builder reports, Optimization logs and candidates, and Editing runs that delivered candidates; failure modes may co-occur.A mode counts if it appears in at least one case or in the builder trajectory, and a named primary cause is the failure costing the run the most points.Source: §H.1 · p. 32

H.3 EDITING: RUNS WITH A DELIVERY

The Editing analysis finds repeated delivery defects despite development feedback, alongside preservation failures and capabilities that fail silently. Its audit also finds unsupported completion claims by builders and delivered agents.

H.3 EDITING: RUNS WITH A DELIVERY

  • The analysis covers 33 Editing runs that delivered candidates across 198 held-out executions; Table 12 reports modes whose rows can overlap.Source: §H.3 · p. 32, Table 12 · p. 33
  • One OpenClaw delivery made 53 handoff-client calls that the product refused, yet the held-out execution passed validity gates.Source: §H.3 · p. 33
  • Eight DeepSeek-V4-Pro runs repeated a feedback-identified cap violation on held-out cases, and nine of ten runs had exhausted all five submissions.The same cap identifier remained violated after feedback named it.Source: §H.3 · p. 33
  • Fourteen delivered products allowed mutations that should have been refused, including repeating one-time operations or overwriting protected state.These failures concern the regression obligation in Equation (2).Source: §H.3 · p. 33

H.4 COMPLETION CLAIMS

  • Nine unsupported final reports all asserted completion, while none of 21 reports admitting partial completion or failure was contradicted by the record.Reports were judged against recorded actions, delivered files and capabilities, and development feedback; held-out failures alone did not count as contradictions.Source: §H.4 · p. 33, Table 13 · p. 33
  • 13 of 33 delivered Editing runs included a runtime claim contradicted by held-out evidence, such as a reported workspace commit absent from local state.The audit checks claims against effect logs, state, and output artifacts; a completed effect-log entry alone does not establish workspace convergence.Source: §H.4 · pp. 33–34

I EXECUTION CASE STUDIES

The execution cases trace failures from development feedback or runtime actions to frozen artifacts and observed outcomes, alongside one repair that reaches the delivered agent. They show how visible-case specialization, displaced regressions, ineffective cancellation, and discarded work can undermine delivery.

I EXECUTION CASE STUDIES

  • 6 of 6 held-out Database analytics requests failed validity: five had unknown case types and one lacked the organizations table.The script classified requests using phrases from two development cases and contained queries with fixed quarter bounds and refund cutoff; its development mean rose from 38 to 99 over ten rounds and ended at 96.Source: I EXECUTION CASE STUDIES · p. 34
  • 40.625 held-out mean versus the starter’s 55.825, despite PinchBench round 2’s development score rising to 76.5 from 72.8.Round 2 was selected after later rounds fell to 60.6, 63.7, and 51.65; its held-out results were 7 wins, 17 losses, and 16 ties.Source: I EXECUTION CASE STUDIES · p. 34
  • 80 callbacks were accepted and 0 rejected after cancellation; the audit length rose from 1 to 81 and revision from 5 to 85.The commission required late callbacks to change nothing, although the terminal state remained cancelled.Source: I EXECUTION CASE STUDIES · p. 34
  • Round 3, scoring 40 on development, was frozen after later rounds scored 20 and 30, so its final report described harness changes absent from the delivered file.The frozen round 3 file was byte-identical to the starter’s; the changes existed in round 4, which was not selected.Source: I EXECUTION CASE STUDIES · pp. 34–35
  • Held-out performance rose from 18 to 46 (S = 34.1) after a BrowseComp repair restored a missing Authorization header and reached the frozen source.The trajectory records a local whole-pipeline check against a mock broker.Source: I EXECUTION CASE STUDIES · p. 35

J LIMITATIONS

The paper identifies limits in repeated-run consistency, model-based judging, and evaluation-setting comparability. These constrain how individual results and cross-family score levels should be interpreted.

J LIMITATIONS

  • Native Optimization objectives vary across repeated evaluations of the same agent, while three-run studies report differences descriptively.Source: J LIMITATIONS · p. 35
  • Creation and Editing scores depend on a model judge and inherit its errors, with the judge sharing a model family with at least one builder in both families.Validity gates and rubric ceilings bound the judge’s discretion.Source: J LIMITATIONS · p. 35
  • Absolute score levels are comparable within each task family, not across families, because each family fixes its own runtime model and judge.Several auxiliary studies also change builder, runtime model, judge, case budget, or requirement and are compared only within themselves.Source: J LIMITATIONS · p. 35
Loading 2610.13361v1…