PaperlayerView exact revision on arXiv

Source-linked AI summary

What the Specification Did Not Say: Specification-Oracle Gaps in Test-Graded Coding-Agent Benchmarks

Fei Ding

arXiv:2610.30635v1cs.CL

TL;DR

Coding-agent benchmarks can grade natural-language tasks with upstream tests that depend on unstated implementation details, and milestone chains add recurring dependencies and inconsistencies. This paper formalises and audits these gaps across five languages, finding that affected SWE-Milestone units fail more often than comparable units while leaderboard rankings remain robust to test removal. It also finds that gap prevalence follows benchmark construction and releases audit corrections and specification addenda.

  • Problem

    Upstream tests can depend on names, paths, or literals absent from task specifications, so a task’s grading may require reproducing implementation details beyond its stated behavior.

    Source: §1 · p. 1

  • Method

    The paper formalises specification–oracle gaps, models milestone-chain inconsistencies, and builds a static five-language audit of name-, path-, and literal-level dependencies in scored tests.

    Source: §1 · pp. 1–2

  • Results

    25 of 98 SWE-Milestone milestones contain an unstated, non-inferable name or path dependency, and affected units fail 1.67 times as often as comparable units.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across five benchmarks, gap prevalence follows how task text is made, while the harness determines how much gaps affect scores; the paper releases corrected filters and specification addenda.

    Source: §9 · p. 8

    The in-depth claims cover one benchmark of seven repositories; the four-benchmark comparison is exploratory because review units and sizes differ.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Coding-agent benchmarks often describe a task in natural language but grade it with tests taken from the upstream project. Passing those tests can require information the description never states: a constant's name, a module path, an exact message. We call this a specification-oracle gap: where it exists, a benchmark also measures whether an agent reproduces upstream implementation details. We formalise the gap, including the inconsistencies that chains of dependent tasks add, and build a static audit for five languages that classifies every name-, path- and literal-level dependency of every scored test (held-out precision 89.2%, recall 92.9%). On SWE-Milestone (98 graded milestones, 35 published runs), 25 milestones contain a test that depends on an unstated, non-inferable name or path. Build units with such a dependency fail in 80.6% of runs, against 48.4% for comparable units without one, and withholding a single hidden name from the reference stops its unit's tests from compiling in 126 of 128 cases. The leaderboard's ranking is robust to removing the affected tests, but they shift scores within repositories and favour agents that reproduce upstream names. Across four further benchmarks, prevalence appears to follow construction: a third or more of the tasks are affected where specifications are release notes or LLM summaries, almost none where tasks are screened or written for their tests; how much a gap costs depends on how the harness grades. We release the audit, corrected filter lists and specification addenda.

1 Introduction

Long-horizon coding-agent benchmarks pair natural-language specifications with upstream tests, which can require unstated implementation details and dependencies inherited across milestone chains. This paper formalises and audits those gaps, measures their effects on SWE-Milestone and other benchmarks, and releases corrections.

1 Introduction

  • The specification–oracle gap arises when tests require names, paths, or literals the task does not state, so correct behavior alone may not pass.Long-horizon chains add recurring dependencies and inconsistencies, including tests that no agent could pass.Source: §1 · p. 1
  • The paper defines a taxonomy separating unstated, non-inferable dependencies from chain inconsistencies and models their blast radius.The taxonomy is designed for milestone benchmarks, where a hidden name can affect a whole build unit and recur in later milestones.Source: §1 · p. 1
  • The five-language static audit classifies name-, path-, and literal-level dependencies for every required test, with held-out precision of 89.2% and recall of 92.9%.It runs in minutes without model calls.Source: §1 · pp. 1–2
  • 25 of 98 SWE-Milestone milestones contain a test depending on an unstated, non-inferable name or path; affected units fail 1.67 times as often as comparable units.Removing affected tests leaves leaderboard rankings within the range produced by random removals of the same size, but shifts scores within the most affected repository.Source: §1 · p. 2
  • Across four further benchmarks, gap prevalence follows how task text is constructed, while the grading harness determines how much a gap costs.The comparison uses public data only.Source: §1 · p. 2
  • The study releases corrected filter lists and specification addenda, alongside recommendations for building test-graded benchmarks.Its in-depth claims concern one benchmark of seven repositories; the four-benchmark comparison shows the audit transfers and prevalence depends on construction.Source: §1 · p. 2

2 Background: SWE-Milestone

SWE-Milestone reconstructs repository release histories as dependent milestones and grades agents against tests drawn from each milestone’s upstream gold state. Its end-to-end leaderboard aggregates milestone scores across repositories.

2 Background: SWE-Milestone

  • SWE-Milestone v1.0.2 covers seven repositories in five languages, with milestones organized as a dependency graph of upstream commits.In end-to-end mode, agents start from a trial base and implement milestones in one working tree, choosing among milestones whose prerequisites are complete.Source: §2 · p. 2
  • For each milestone, the harness copies upstream gold tests onto the agent’s code and grades required tests classified as new behavior or pass-to-pass preservation.Per-milestone filter lists remove tests maintainers found invalid; pass-to-none tests are not graded.Source: §2 · p. 2
  • The leaderboard averages milestone scores within each repository and then across repositories; its best published configuration resolves 29.75% of 98 graded milestones.A milestone is resolved only when all required tests pass.Source: §2 · p. 2

3 Motivating Example

The go-zero M019 example shows how tests can require names absent from an otherwise specific behavioral request. Because the tests compile with the package, a naming mismatch blocks the unit’s required tests and can recur in later milestones.

3 Motivating Example

  • 84 required M019 tests are lost if either hidden name, valueSSE or headerAccept, is missing, although the specification states the SSE behavior.The same dependency recurs in five other milestones, each inheriting between 88 and 108 tests.Source: §3 · p. 2, Figure 1 · p. 3
  • 32 of 34 runs implement the requested bypass, but the 23 that use another value name or inline the string pass none of the 84 tests.Of nine runs that name valueSSE, four pass all 84 tests; the other five lack a further name.Source: §3 · p. 2
  • These name mismatches can decide comparisons between individual runs while remaining invisible in an aggregate score.The figure’s test tiles and milestone links are schematic.Source: §3 · p. 3, Figure 1 · p. 3

4 Specification–Oracle Gaps

A specification–oracle gap occurs when a scored test depends on an artefact not fixed by the task, starting code, or inferable conventions. In milestone chains, gaps can recur or arise from inconsistent reference states; the audit classifies dependencies and maps their impact.

4.1 Definition

  • A test has a specification–oracle gap when it depends on a name, path, or literal that the task, trial base, and inferable conventions do not fix.Such a test can fail even when the agent implements the specified behaviour correctly; name or path gaps can block a whole build unit, while literal gaps affect individual tests.Source: §4.1 · p. 3

4.2 Where gaps come from

  • Gold-state diffs classify dependencies as under-specification or chain inconsistency; order conflicts arise when milestones change shared APIs incompatibly.A test may reference a name absent from its own milestone’s gold state, leaving even the reference implementation unable to pass it.Source: §4.2 · pp. 3–4
  • A dependency is not counted as hard when conventions, systematic naming, or a matching interface make it inferable from the trial base or specification.Specifications for a milestone can also name artefacts through their own text or the specifications of its ancestors.Source: §4.1 · p. 3, §4.2 · p. 4

4.3 Implementation

  • The static audit extracts names, paths, and literals, diffs gold states, classifies dependencies, and maps required tests to build units.It uses one extractor per language and a language-independent core.Source: §4.3 · p. 4

5 Experimental Setup

The experiments assess gap prevalence, effects on failures and scores, audit accuracy and cost, and prevalence in other benchmarks. They use public benchmark data, published runs, reference-state re-scoring, and reviewed audit cases.

Experimental questions

  • The study asks five questions about gap prevalence, failure effects, score and ranking changes, audit accuracy and cost, and gaps in differently constructed benchmarks.The audit examines whether dependencies are unstated and non-inferable, not merely whether agents fail.Source: §5 · p. 4

Subjects and runs

  • The SWE-Milestone analysis covers seven repositories, 98 graded milestones, 580,039 required tests, and 35 published end-to-end runs.The runs include 27 models; the authors’ replay matches the 28 leaderboard runs within 0.20 points.Source: §5 · p. 4

Realisation

  • Realisation compares blocked units with new-test units without new dependencies within the same milestones, then checks failure logs and renames hidden Go identifiers in reference code.A unit is a milestone–build-unit pair; it fails when a required test fails or is missing.Source: §5 · p. 4

Re-scoring and other benchmarks

  • Re-scoring removes tests affected by hidden dependencies, then compares scores and rankings with the original and with matched random removals.Other-benchmark analyses use public results from SWE-EVO, SWE-Chain, SWE-bench Verified, and DeepSWE.Source: §5 · pp. 4–5

6 Results

The audit finds hard hidden dependencies across SWE-Milestone, and units containing them fail more often than comparable new-test units. Removing affected tests leaves overall rankings largely unchanged, while effects vary by repository and benchmark construction.

6.1 RQ1: Prevalence

  • 25 of 98 milestones contain a required test with a hard hidden dependency, mostly ancestor-introduced names; in go-zero, two constants block a package across six milestones.The blocked package contains 84 to 108 required tests per milestone; hard dependencies affect 0.6% of required tests overall.Source: §6.1 · p. 5

6.2 RQ2: Realisation

  • 80.6% of blocked unit-run pairs fail, versus 48.4% for new-test units without new dependencies; renaming one hidden identifier stops reference tests compiling in 126 of 128 Go cases.Blocked units fail 1.67 times as often as comparable units, and the counterfactual failures are 99.5% of those predicted by the audit.Source: §6.2 · p. 6

6.3 RQ3: Impact on scores and rankings

  • Removing affected tests leaves rankings close to the published ranking, but go-zero’s mean score rises from 38.33 to 42.08 and mean resolve rate from 4.35% to 5.71%.For the full variant, Kendall’s τ-b is 0.984 by score and 0.942 by resolve rate; these changes fall within random-removal comparisons.Source: §6.3 · p. 6, Table 3 · p. 6

6.4 RQ4: Precision and cost

  • The held-out audit reaches 89.2% precision and 92.9% recall; a cold-cache audit of seven repositories takes 28.1 minutes without model calls.The recall estimate comes from the second held-out set; precision is measured over 65 flagged units across two held-out rounds.Source: §6.4 · pp. 6–7

6.5 RQ5: Other benchmarks

  • A third or more of tasks are flagged in benchmarks built from release notes or LLM summaries, while hidden names are almost absent in screened or test-written tasks.The cross-benchmark comparison is exploratory because review units and sizes differ.Source: §6.5 · p. 7
  • In SWE-Chain, flagged steps fail only slightly more often than others (lift 1.12); its harness isolates failed imports and reports them to agents.The study finds 45% reproduction of hard dependencies with evaluation evidence, versus 79% of inferable ones, with no sign of a training-cutoff relationship.Source: §6.5 · p. 7, §6.6 · p. 7

7 A Corrected Benchmark and Recommendations

The authors provide corrected test-filter lists and specification addenda for affected milestones, alongside recommendations for benchmark construction and grading. The addenda state required interfaces while retaining the tests.

Corrected filter lists

  • Corrected filter lists waive tests in units blocked by hard hidden dependencies and tests whose reference lacks an available name.The full variant also waives explained tests that no published run passes; the lists contain 3,321 conservative or 3,964 full entries.Source: §7 · p. 7

Specification addenda

  • Specification addenda state the interface conventions required by tests without describing implementation, preserving tests while removing the need to guess.The authors provide 153 addenda across 25 affected milestones; evaluating them requires new agent runs.Source: §7 · pp. 7–8

Recommendations

  • The recommendations are to audit dependencies, replay reference solutions in multiple milestone orders, isolate build units, report unresolved names, and publish per-test results.When tests reach into internals, benchmarks should state required names, paths, and messages or exclude those tests.Source: §7 · p. 8

8 Related Work

Prior benchmark audits mainly examine tests that accept incorrect code; this paper instead studies tests that demand details absent from task descriptions. It extends interface-specification practices to mined, long-horizon benchmarks.

Validity of coding benchmarks

  • Earlier audits examine under-specified tasks, weak tests, issue–patch misalignment, and benchmark re-annotation, mostly for single tasks or tests that accept too much.The paper positions its focus as the converse: tests demanding information the task does not state.Source: §8 · p. 8

Telling the agent what the tests need

  • Some benchmarks provide task interfaces, component signatures, specification documents, or task-specific verifiers; this paper adds specification addenda to a mined long-horizon benchmark.The audit identifies where those addenda are needed.Source: §8 · p. 8

Long-horizon and multilingual benchmarks

  • Long-horizon benchmarks evaluate sequences of changes, while multilingual benchmarks extend issue resolution beyond Python and live benchmarks add fresh tasks.Source: §8 · p. 8

Benchmarks that measure the wrong thing

  • The paper frames specification–oracle gaps as a generative analogue of NLP annotation artefacts: the grader rewards information absent from the input.Source: §8 · p. 8

9 Conclusion

The paper audits specification–oracle gaps in test-graded coding benchmarks, finding concentrated milestone and build-unit effects alongside robust overall leaderboard ranking. It releases audit materials and corrections.

9 Conclusion

  • Across five benchmarks, gap prevalence follows task construction, and a dependency audit runs in minutes without model calls.The paper releases the audit, corrected filter lists, and specification addenda.Source: §9 · p. 8
  • In SWE-Milestone, a quarter of graded milestones contain a hidden dependency, and affected build units fail markedly more often than comparable units.Source: §9 · p. 8
  • Removing affected tests preserves leaderboard ranking, but per-repository scores shift; the ranking is coarser than it appears.Source: §9 · p. 8

Limitations

The authors identify limits in measuring realisation, auditing dependencies, and generalising prevalence across benchmarks; addenda have not yet been evaluated with new agent runs.

Limitations

  • Realisation is observational: blocked and comparison units may differ in difficulty, and the reference counterfactual covers Go only.Source: Limitations · p. 9
  • Three audit rules and RQ4 fixes were set on development data; DeepSWE is not an out-of-sample test.Source: Limitations · p. 9
  • Lexical SRS matching can over-report or under-report dependencies, while name-level extractors do not type-check and miss some method and external-library cases.Source: Limitations · p. 9
  • Behavioural gaps without new names rely on a text heuristic, while dependencies outside the rules are reported only as name-level upper bounds.Runtime-computed test identifiers are matched only when the auditor reads the helper that builds them.Source: Limitations · p. 9
  • Reviews involve judgement that can differ, and prevalence may vary with language, test style, and benchmark construction.Source: Limitations · p. 9
  • The memorisation analysis is correlational, and the effect of specification addenda has not been measured with new agent runs.Source: Limitations · p. 9

Ethics Statement

The appendices document the study’s data and auditing procedures, regression and agreement analyses, explanations for missed tests, and examples of benchmark grading effects.

Ethics Statement; A Units and Extractors per Language

  • The study uses public benchmark data and runs, involves no human-subjects research or agent runs, and reports author-checked labels and numbers; the audit extractor design is language-specific.The held-out labels involve one author and a hired software expert; public artefacts are used for evaluation research.Source: Ethics Statement · p. 9, A Units and Extractors per Language · p. 11

B Regression Details

  • The regression models unit-run failure using hard-block status, required-test count, and repository fixed effects across 3,379 pairs; setting the blocking indicator to zero lowers predicted failure probability by 31.9 points on average.Without the run marked as leaked, the odds ratio is 4.75 (2.99–7.54).Source: B Regression Details · p. 11

C Agreement Between Runs

  • Same-model run pairs have median κ 0.81 (95% interval 0.66–0.94), versus 0.63 (0.50–0.75) for cross-model pairs; the difference interval spans zero.Source: C Agreement Between Runs · p. 11

D Tests No Run Passes

  • Of 6,167 required tests no published run passes, 5,338 are attributed to one nushell milestone no run could build; 97 remain unexplained.Source: D Tests No Run Passes · p. 11

E Why the Generator Leaves Names Out

  • The specification generator instructs writers to omit implementation hints and names, while its test-aligned pass can skip names inferred from an unseen diff or tests grouped under one error signature.The authors say these rules are likely to omit names that decide whether whole units build.Source: E Why the Generator Leaves Names Out · pp. 11–12

F Harness Design

  • SWE-Chain supplies missing modules, placeholder imports, and one repair round; SWE-Milestone instead grafts reference tests and provides no such feedback.Source: F Harness Design · p. 12

G Reproduced Hidden Names in Detail

  • Runs reproduce 45.2% of determined hard-item pairs and 79.0% of inferable ones; the analysis finds no sign that upstream-code exposure explains reproduction.The exposure analysis is correlational and has little power.Source: G Reproduced Hidden Names in Detail · p. 12

Tests the benchmark silently dropped (go-zero,

  • A nushell milestone’s reference tests depend on a field introduced by an earlier milestone, leaving all 5,339 required tests without a published result; other go-zero tests were dropped when the reference lacked required names.Those go-zero tests were classified as pass-to-none and disappeared from the required set.Source: Tests the benchmark silently dropped… · pp. 12–13

I Reviews of the Other Benchmarks

The review protocol checks whether flagged dependencies are unstated, non-inferable requirements of scored tests. Because review units and sampling differ, the cross-benchmark precision figures are exploratory rather than directly comparable.

I Reviews of the Other Benchmarks

  • Reviewers confirmed a flag only when a scored test required an exact name, module path, or text absent from the task and not determined by the base or convention.SWE-Milestone was reviewed per build unit by two blind annotators; other benchmarks were reviewed per flagged item, with sampled items checked against task text, tests, and reference changes.Source: I Reviews of the Other Benchmarks · p. 13
  • Because units, reviewers, and sample sizes differ, benchmark precisions are not comparable and Table 4 is exploratory evidence about construction, not a controlled comparison.Intervals are wide where few items were reviewed.Source: I Reviews of the Other Benchmarks · p. 13
  • The review table reports tasks, flags, reviewed items, confirmed flags, and precision with Wilson intervals for each benchmark.SWE-EVO and SWE-Chain use stratified random samples; Verified and DeepSWE review all flags.Source: Table 6 · p. 13, I Reviews of the Other Benchmarks · p. 13

Related papers

Loading 2610.30635v1…