PaperlayerView exact revision on arXiv

Source-linked AI summary

Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks

Yuanzhe Liu, Bastian Lipka, Avraham Shinnar, Jatin Ganhotra, Yao Ma, Martin Hirzel

arXiv:2610.22760v1cs.AIcs.SE

TL;DR

Repository-level coding benchmarks can leave test-required names unspecified, creating guesswork, while reference-derived hints risk disclosing implementation choices. The paper compares hint selection and projection policies, finding that resolution gains can accompany greater disclosure. It motivates combining DefUse with Declaration and assessing hints by both disclosure and performance.

  • Problem

    Hidden identifiers required by tests can make a behaviorally correct patch fail because agents cannot infer the test-facing names from the task or repository.

    Source: §1 · pp. 1–2

  • Method

    The paper separates identifier selection from projection and proposes DefUse to select code–test interface names and Declaration to reveal their structural attributes.

    Source: §1 · p. 2

  • Results

    Across five benchmarks, hint policies affect measured task resolution and the reference-derived information available to agents, with performance gains potentially accompanying greater disclosure.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Benchmark designers should assess hints by what they disclose as well as by the performance gains they produce.

    Source: §5 · p. 10

    Small benchmark subsets and a single agent harness limit generalizability, while serving variation and rubric-dependent LLM judgments add uncertainty.

    Source: §5 · p. 10

Abstract

from arXiv · show

Benchmarks drive AI progress, but are hampered by task under-specification and over-specification. Task under-specification causes guesswork, where solutions have to make things up, or worse, resort to contamination or reward hacking. Task over-specification causes spoilers, where solutions can bypass some of the harder steps the benchmark was supposed to measure. Since future-looking benchmarks should be difficult, over-specification reduces a benchmark's shelflife of community adoption. Repository-level coding benchmarks have helped the field progress but experience unnecessary churn due to both over- and under-specification. For instance, SWE-bench Pro, FEA-bench, Doc2Feat-bench, and Multi-SWE-bench all tackle under-specification by augmenting task statements with hints derived from ground truth. We show that leaking ground-truth information into the task statement causes over-specification in some of these benchmarks. We experiment with various specification levels, trying to thread the needle between spoilers and guesswork. Based on these results, we recommend techniques for deriving parsimonious hints. We will release our hints so the community can obtain more reliable signals from popular benchmarks.

1 INTRODUCTION

Repository-level coding benchmarks can leave hidden test-facing names unspecified, forcing agents to guess, while reference-derived hints can disclose implementation choices the benchmark intends agents to make. The paper separates hint selection from projection and proposes policies to clarify required interfaces while limiting spoilers.

1 INTRODUCTION

  • Hidden names required by tests can make a behaviorally correct implementation fail solely because it chose a different name.The paper identifies this ambiguity as especially relevant to feature-addition benchmarks, where new names may be absent from the existing repository.Source: §1 · pp. 1–2
  • The paper frames hint design as separating identifier selection from projection, enabling controlled comparisons of which entities and attributes are disclosed.Selection chooses program entities; projection determines whether details such as names, signatures, locations, or semantic descriptions are revealed.Source: §1 · p. 2
  • DefUse selects identifiers through which code and tests interact, while Declaration reveals interface attributes without semantic descriptions or implementation guidance.Together, the policies aim to reduce guesswork about required identifiers while limiting spoilers about implementation choices.Source: §1 · p. 2
  • Across five benchmarks, hint policies affect measured task resolution and disclosed implementation information, motivating evaluation beyond solve rates alone.The authors motivate combining DefUse selection with Declaration projection to clarify test-facing interfaces while limiting implementation guidance.Source: §1 · p. 2

2 RELATED WORK

Prior work documents several sources of benchmark validity concerns, including underspecified tasks, inadequate tests, disclosed solutions, and contamination. Related research also addresses missing requirements and naming ambiguity; this paper distinguishes identifier selection from projection to study hint disclosures.

2 RELATED WORK

  • Benchmark audits report underspecified tasks, tests enforcing unstated details, disclosed solutions, inadequate tests, and contamination concerns.This study complements those audits by examining how reference-derived hints change the information available to agents during evaluation.Source: §2 · p. 3
  • Existing audits also find that passing benchmark tests does not guarantee patch correctness, while memorization raises contamination concerns.LiveCodeBench and SWE-rebench seek to mitigate contamination by collecting new tasks.Source: §2 · p. 3
  • Prior work studies task content, hidden naming contracts, reference-derived hints, agent workflows, clarification, and information-removal experiments.This paper separates identifier selection from projection to examine which disclosures reduce interface guesswork and which supply implementation spoilers.Source: §2 · p. 3

3 METHODOLOGY

The methodology decomposes hint construction into identifier selection and identifier projection so each can be compared under controlled conditions. It proposes DefUse to select code–test interface names and Declaration to reveal their structural declarations without semantic descriptions.

3 METHODOLOGY

  • Separating identifier selection from projection supports controlled comparisons by fixing one decision while varying the other.Selection determines which program entities are disclosed; projection determines which attributes of each selected entity are revealed.Source: §3 · p. 4

3.1 IDENTIFIER SELECTION: DEFUSE

  • DefUse selects newly introduced identifiers defined in one patch and used in the other, capturing names through which code and tests communicate.It includes both code-defined names used by tests and test-defined names used by code; the paper argues selected names can cause false failures if hidden, while discarded names can be changed freely without affecting test outcomes.Source: §3.1 · p. 4

3.2 IDENTIFIER PROJECTION: DECLARATION

  • Declaration reveals each selected identifier’s name, kind, declaring-file path, new-file status, and callable signature, stopping before semantic descriptions.The authors characterize semantic attributes as describing behavior and mechanism, whereas structural attributes carry no solution content in their assessment.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments vary hint selection and projection separately, then test hint attributes and assess their disclosure of reference-solution information. They compare task resolution across Python and Java benchmarks to examine the rationale for combining DefUse selection with Declaration projection.

4.1 IMPLEMENTATION DETAILS

  • The pipeline extracts identifiers from reference and test patches, renders hints from selection and projection rules, and reproduces released Doc2Feat-bench and FEA-Bench hints with minor bug fixes.SWE-bench Pro hints were implemented by extracting desired interfaces and generating requirements and interface descriptions with Claude Haiku 4.5.Source: §4.1 · p. 5

4.2 SETTINGS

  • Evaluation uses official Docker evaluators, counts tasks resolved only when all fail-to-pass and pass-to-pass tests pass, and reports resolve rate across five benchmarks.The models use one trajectory per instance; the selection and projection subsets vary in size, and DefUse can be empty when tests reference no new identifier.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 IDENTIFIER HINT SELECTION

  • For GPT 5.6 Luna on SWE-bench Pro, DefUse discloses 1.2 identifiers per instance and resolves 91 of 131 tasks (69.5%), versus 4.8–17.2 identifiers and 70.2–72.5% for broader selections.For open-source models, broader selection helps more on SWE-bench Pro, but its benefit varies by benchmark; DefUse sometimes matches or exceeds broader policies.Source: §4.3 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7

4.4 IDENTIFIER HINT PROJECTION

  • Declaration improves resolution over the empty baseline for all three models on SWE-bench Pro, while omitting the semantic descriptions included in Pro.DeepSeek rises from 10.3% to 29.4%, MiniMax from 2.9% to 14.7%, and GPT 5.6 Luna from 55.9% to 73.5%; Java results provide supporting evidence.Source: §4.4 · pp. 7–8, Table 6 · p. 8, Table 7 · p. 8

4.5 HINT ATTRIBUTE ABLATION

  • On SWE-bench Pro, adding semantic descriptions to Declaration raises DeepSeek V4 Flash resolution from 29.4% to 72.1%, but single-trajectory variation limits attribution to hint content alone.The semantic configurations also use a different rendering from Pro in the projection comparison, so their absolute rates should not be directly compared with Table 6.Source: Table 5 · p. 7, §4.5 · p. 9
  • Names improve resolution over the empty baseline, Declaration adds a further gain, and entity type alone adds no observed improvement; structural and semantic attributes can provide complementary benefits.The ablation evaluates DeepSeek V4 Flash on the 68-instance SWE-bench Pro subset with DefUse selection.Source: Table 5 · p. 7, §4.5 · pp. 8–9

4.6 HINT LEAKAGE JUDGE

  • Structural and semantic attributes have similar average necessity scores, but semantic descriptions receive greater disclosure and guidance scores; names and signatures have the highest hint value margins.Strong semantics scores higher than weak semantics on disclosure and guidance, consistent with descriptions conveying implementation mechanisms.Source: §4.6 · p. 9, Table 8 · p. 9

5 CONCLUSIONS AND LIMITATIONS

The paper frames identifier hints as affecting both task resolution and the reference-derived information agents receive, and recommends assessing disclosure alongside performance. It motivates combining DefUse selection with Declaration projection while noting limits on experimental generalizability.

5 CONCLUSIONS AND LIMITATIONS

  • The authors motivate combining DefUse selection with Declaration projection to clarify test-facing interfaces while limiting implementation guidance.They argue benchmark designers should assess what hints disclose as well as the performance gains they produce.Source: §5 · p. 10
  • Hint policies affect both measured task resolution and the reference-derived information available to agents.Source: §5 · p. 10
  • Small benchmark subsets, one agent harness, one trajectory per instance, serving variation, and rubric-dependent LLM judgments limit generalizability and add uncertainty.The reported experiments represent approximately $2,300 in inference value, including unbilled MiniMax usage valued at public API rates.Source: §5 · p. 10

A.1 SELECTION

For Doc2Feat and FEA Bench Brief, the authors adapted the original hint-generation code to extract hints on multiple benchmarks. For SWE Bench Pro, they used an LLM to approximate hints created by human engineers.

A.1 SELECTION

  • The authors adapted the Doc2Feat and FEA Bench Brief code so their pipelines could extract hints on various benchmarks.Source: §A.1 · p. 12
  • Because SWE Bench Pro uses human engineers, its hint-generation pipeline could not be completely replicated, so an LLM approximated the detailed hint text.Source: §A.1 · p. 12
  • For SWE Bench Pro, the authors identified changed identifiers, extracted identifier information, and used Claude Haiku 4.5 to generate interface descriptions and requirements.On 266 Python instances, the generated descriptions and requirements had reported cosine similarities of 0.8978 and 0.8405, respectively.Source: §A.1 · p. 12

A.2 PROJECTION

The projection pipeline extracts identifiers from code and test patches, selects identifiers through communication between those patches, and varies how much information each hint reveals. The compared projections range from names alone to descriptions of behavior and constraints.

A.2 PROJECTION

  • The pipeline extracts identifiers from code and test patches, obtains information from their abstract syntax trees, selects DefUse identifiers through communication between the patches, and enriches semantics with Claude Haiku 4.5.Source: §A.2 · p. 12
  • Declaration reveals each identifier’s name, entity type, file path, new-file flag, and signature, nesting class members under their class without natural-language text.Source: §A.2 · p. 12
  • Declaration+Doc adds a one-sentence description of what an entity is and does, while Pro adds a two-to-four-sentence account of behavior, edge cases, and test-implied constraints.The weak and strong semantics correspond to SWE-bench Pro interface descriptions and requirements, respectively.Source: §A.2 · p. 12

B IDENTIFIER SELECTION UNDER NAME ONLY PROJECTION

With hints restricted to names, selection-policy effects vary across models and benchmarks: broader selection can help but does not consistently improve resolution. The comparison uses shared instances where Doc2Feat and FEA-Bench Brief yield non-empty identifier sets.

B IDENTIFIER SELECTION UNDER NAME ONLY PROJECTION

C HINT LEAKAGE JUDGE DETAILS

The judge rates hint attributes on necessity, disclosure, and behavioral guidance, combining these dimensions in the heuristic HVM. The assessment checks selected judgments against repository evidence and reports uncertainty that varies by attribute.

  • Necessity scores how recoverable an attribute is without the reference solution, while HVM subtracts the higher of disclosure and guidance scores from necessity.HVM is a heuristic, not a calibrated measure of information quantity.Source: C HINT LEAKAGE JUDGE DETAILS · pp. 12–13
  • The judge assessed 787 identifier–attribute pairs across 160 identifiers on 68 SWE-bench Pro instances, with necessity judged without patches and disclosure and guidance judged with both patches.Each call showed the identifier’s other attributes; GPT 5.6 Luna returned category labels, with medians of three samples at temperature 1 and instance-clustered bootstrap uncertainty.Source: C HINT LEAKAGE JUDGE DETAILS · p. 13
  • Repository evidence raised correct membership distinctions from 18 to all 31 checked identifiers; unanimity was 94% for signatures and 41–42% for semantic descriptions.The membership check validates only a limited aspect of necessity, not the full rubric.Source: C HINT LEAKAGE JUDGE DETAILS · p. 13
  • Disclosure ranges from interface identity and addressing through observable contract and implementation mechanism to transcription; guidance separately ranges from declarative to prescriptive.The rubric distinguishes disclosure from guidance: a path can orient an agent without describing an implementation.Source: C HINT LEAKAGE JUDGE DETAILS · p. 15

Related papers

Loading 2610.22760v1…