PaperlayerView exact revision on arXiv

Source-linked AI summary

Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer

Md Shohel Arman, Igor Molybog

arXiv:2609.31587v1cs.SEcs.AIcs.CL

TL;DR

The paper asks whether natural-language documentation can help coding agents resolve issues when source code does not fit in context. It builds a roundtrip benchmark and optimizer for descriptions, but finds that documentation does not improve issue resolution when source is present; its value lies in substituting for unread code.

  • Problem

    For large repositories, loading the entire codebase into an agent’s context can be prohibitively expensive or impossible, motivating the question of whether high-fidelity descriptions can supply missing intent and contracts.

    Source: §1 · p. 1

  • Method

    The authors score descriptions by regenerating code from them and running the original tests, use that benchmark to optimize a description-writing prompt, and test downstream issue resolution.

    Source: §1 · p. 1, §4 · p. 4, §6.1 · p. 6

  • Results

    With source present, neither static compact descriptions nor retrieved past-task context improves issue resolution over the issue alone across two model families and ten repositories.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Documentation of this kind helps when it substitutes for source that does not fit in context, but is redundant when the agent can already read the code.

    Source: §8 · p. 8

    The roundtrip fixture set is small—eleven fixtures plus three hand-built codebases—from one language and project family, so benchmark-side numbers characterize the method rather than provide powered measurements.

    Source: §8 · p. 8

Abstract

from arXiv · show

We investigate whether natural-language documentation helps coding agents resolve software issues, and we build the tools to construct and evaluate it. We introduce a roundtrip benchmark that scores code descriptions by whether code regenerated from them passes the original tests, and show that completeness, not length, drives a description's fidelity. Using the benchmark as an optimization signal, we discover a description-writing prompt that reaches full fidelity and generalizes to unseen files. We then test the hypothesis that motivated the work: that better documentation helps an agent resolve real repository issues. Across two model families and ten repositories, and against a positive control confirming that our evaluation can detect a genuine improvement, we find that it does not. When the source is present, neither static compact documentation nor retrieved context beats the issue alone. We report this negative result together with the benchmark and the optimizer, and we characterize the boundary at which documentation helps.

1 Introduction

The paper asks whether high-fidelity documentation can stand in for source code that agents cannot fit in context, and builds ways to evaluate and optimize such descriptions. Its benchmark and optimizer succeed, but the expected issue-resolution benefit disappears when the source is available.

1 Introduction

  • With source withheld, optimized descriptions raise mean test-pass fraction from 0.08 to 0.71, but with source present neither static compact descriptions nor retrieved context beat the issue alone.The source-present null holds across two model families and ten repositories, with a positive control showing the evaluation can detect a genuine improvement; longer documentation can make matters slightly worse.Source: §1 · p. 1
  • The authors find that completeness, not length, primarily drives description fidelity, and the optimizer converges on descriptions that are both complete and compact.They focus on evaluating and optimizing descriptions independently of an agent’s internal machinery.Source: §1 · p. 1
  • The paper introduces a roundtrip benchmark that scores descriptions by whether code regenerated from them passes the original tests, alongside a compression measure and memorization controls.The benchmark is also used to optimize the description writer; the resulting prompt reaches full fidelity and generalizes to held-out files.Source: §1 · p. 1

2 Background

The paper connects roundtrip evaluation, execution-based code benchmarks, documentation compression, and automatic prompt optimization. Its distinction is using regeneration fidelity not only to evaluate descriptions but also to improve the system that writes them.

Roundtrip and self-consistency evaluation

  • Prior roundtrip work evaluates natural-language description and code regeneration against unit tests, while this paper turns the roundtrip score into an optimization target for the description writer.Related work includes Allamanis, Panthaplackel, and Yin (2024), IdentityChain (Min et al. 2024), and invertibility analyses (Maveli, Vergari, and Cohen 2026).Source: §2 · p. 2

Execution-based code benchmarks

  • Execution-based benchmarks test code against real test suites, and this paper uses SWE-ContextBench to ask whether static file documentation helps when source is already available.SWE-bench evaluates repository issue resolution with fail-to-pass and pass-to-pass tests in isolated containers (Jimenez et al. 2024).Source: §2 · p. 2

Code documentation and compression

  • Earlier work studies documentation generation and compression, while these experiments find that uplift tracks completeness and verbosity beyond completeness adds nothing.The cited lines include shorter docstrings retaining code-generation performance (Yang et al. 2024b) and prompt compression with little loss (Jiang et al. 2023).Source: §2 · p. 2

Prompt and harness optimization

  • The optimization loop applies automatic prompt-optimization approaches to documentation quality, scoring candidate prompts by regeneration fidelity.The paper situates the loop among candidate generation, score-guided refinement, prompt compilation, prompt evolution, and textual-feedback methods.Source: §2 · p. 2

3 When Documentation Helps an Agent

The experiments find that documentation helps when it supplies intent or a contract absent from the code, while completeness matters more than length. The study uses agent tasks designed to isolate information supplied by descriptions from coding ability.

3.1 Experimental Setup

  • The experimental setup uses temperature-zero generation and reports independent attempts passing each condition’s associated oracle.Flash-lite is used in the initial comparison so success depends on information supplied by the description rather than raw capability.Source: §3.1 · p. 2, §3.2 · p. 2

3.2 Documentation Helps Only Beyond the Code

  • Descriptions supply a needed contract when the code does not expose it, but restating readable code adds nothing.In three hand-built codebases, adding a description of an access-control contract changed the weak agent from zero of three attempts to three of three on each codebase; a real 423-line file succeeded with or without a description.Source: §3.2 · p. 2

3.3 What Determines a Description’s Value

  • Descriptions containing one, three, or five of five required facts yield one, three, or five correct answers, respectively; a description with none yields none.The comprehension task asks the agent to recover exact documented key–value pairs, so success depends on whether the description carries the facts.Source: §3.3 · p. 3
  • A complete description gives equal uplift at 38 and 664 words, and the broader confusable-key tests likewise find no effect of length once completeness is achieved.Table 2 summarizes pass counts for descriptions with and without the description, varying exposed contracts, fact coverage, and verbosity.Source: §3.3 · p. 3, Table 2 · p. 3

4 The Roundtrip Benchmark

The roundtrip benchmark regenerates code from a natural-language description and scores it against the original tests. It uses single-file fixtures, controls for possible memorization, and measures description compactness relative to code size.

4 The Roundtrip Benchmark

  • The benchmark uses eleven single-module fixtures from SWE-bench Verified plus three hand-built codebases, with each fixture checked against its own original oracle tests.The SWE-bench filter retains single-source-file instances with named oracle tests and a passing baseline; the small fixture set is drawn from one language and project family.Source: §4 · p. 3
  • The benchmark addresses memorization by drawing fixtures from SWE-bench Verified and monitoring regeneration for verbatim recitation of source.No verbatim recitation was observed on the benchmark fixtures; widely mirrored files that prompt refusal are excluded from fixture selection.Source: §4 · pp. 3–4
  • The pipeline describes a source file, regenerates its implementation from the description and scaffold alone, then evaluates it against the original unit tests.The original tests are excluded from the description stage, and the regenerate stage has no access to the original source.Source: §4 · p. 4
  • Alongside fidelity, the benchmark measures description compactness relative to the code, using the same subword tokenizer for both counts.Density is reported for the per-file protocol; whole-package regeneration is kept separate because its effective density is not directly comparable.Source: §4 · p. 4

5 Benchmark Results

Across eleven Verified fixtures, description density tracks roundtrip fidelity, and regeneration failures recur in omissions and invalid modules. OpenWiki documentation regenerates less faithfully than the paper’s descriptions and can report behavior opposite to a task’s required contract.

5 Benchmark Results

  • Eleven Verified fixtures remain unsaturated: one regenerates perfectly on every run, several earn stable partial credit, and four fail outright.Fidelity separates stronger descriptions from weaker ones.Source: §5 · p. 4
  • Failures include omitted output strings, imports, or invariants, as well as parse or import defects that dominate zero-fidelity core modules.An invalid module can prevent any tests from running, while leaf modules may earn partial credit test by test.Source: §5 · p. 4, Table 4 · p. 4
  • 0.88 Pearson correlation links description density to mean fidelity; all four zero-fidelity fixtures are below 0.45 density, while density of 0.9 or higher corresponds to fidelity of 0.33 or higher.Figure 1 plots fidelity against description density across the eleven fixtures.Source: Figure 1 · p. 5, §5 · p. 5

5.1 Comparison to an External Documentation Tool

  • Across three sampled fixtures spanning high, middle, and low fidelity, OpenWiki documentation regenerates code less faithfully than the paper’s descriptions.OpenWiki omits test-dependent internals, including a private helper and a class attribute; reference documentation and reconstruction descriptions serve different objectives.Source: Table 5 · p. 5, §5.1 · p. 5
  • On deny-override, OpenWiki reports the observed input-order behavior rather than the required priority contract, leaving the agent where no documentation does.The task requires priority evaluation with deny overriding allow at equal priority.Source: Table 6 · p. 5, §5.1 · p. 5

6 Optimizing the Description Writer

The authors optimize the describe-stage prompt against a roundtrip benchmark that rewards fidelity while lightly penalizing length. The discovered prompt reaches full training fidelity, shortens descriptions, and raises held-out fidelity from 0.5 to 1.0.

6.1 Method

  • The optimizer proposes describe prompts, scores their descriptions on fixtures, and keeps improvements using a fidelity objective with a small length penalty.It optimizes on three fixtures and evaluates the discovered prompt on two held-out fixtures.Source: §6.1 · p. 6, Figure 2 · p. 6

6.2 Results

  • The loop reaches 1.0 fidelity across training fixtures, then cuts description length while keeping fidelity complete.Three independent runs with different proposer temperatures reached full training fidelity and preserved it out of sample.Source: §6.2 · p. 6
  • 1.0 vs 0.5: On held-out fixtures, the discovered prompt raises mean fidelity from the baseline while using fewer words.The held-out fixtures were never used during optimization.Source: Figure 3 · p. 6, §6.2 · p. 6
  • The discovered prompt specifies imports, exact module-level constant values, signatures and defaults, plus conditions for exceptions and return paths.These instructions align with failure modes identified in Table 4, although the optimizer had no access to that hand analysis.Source: §6.2 · p. 6

7 Transfer to Issue Resolution

When source is present, neither compact descriptions nor retrieved context beats the issue alone in the reported comparisons. Descriptions help when they substitute for withheld source, while editing without loading a file shows a narrower benefit when code is much larger than its description.

7.1 Source withheld: documentation substitutes for code

  • With source withheld, mean test-pass fraction rises from 0.08 for issue-only to 0.21 with baseline descriptions and 0.71 with optimized descriptions.Optimized descriptions are best on ten of eleven fixtures; failures coincide with near-zero roundtrip fidelity.Source: §7.1 · p. 7, Table 7 · p. 7

7.2 Source present: documentation stops helping

  • With source present, the optimized description resolves 11 of 43 paired Kind-B fixtures versus 14 for issue-only; on django, Gemini resolves 8 of 15 with compact descriptions and 4 with full-length ones, versus 9 issue-only.The local-model comparison also reports that descriptions sometimes prompt file rewrites that break tasks issue-only resolves.Source: §7.2 · p. 7
  • On SWE-ContextBench django tasks, the positive control resolves 15 versus 14 issue-only, while compact descriptions resolve 16 versus 15; the one-task advantage is indistinguishable from zero.The control and compact-description comparisons use separate runs with slightly different paired task sets.Source: §7.2 · p. 7
  • On 58 paired tasks, issue-only resolves 33, retrieved context 30, and compact descriptions 29; neither documentation condition significantly improves resolution.The differences are not significant, and documentation conditions produce no edit slightly more often.Source: §7.2 · p. 7, Table 8 · p. 8

7.3 Editing without loading the file

  • For single-file editing without loading the whole file, descriptions do not improve resolution overall and cost more tokens, but lambdify resolves only with a description and at fewer tokens than loading the file.The authors characterize this benefit as occurring when code is much larger than its description.Source: §7.3 · p. 8

8 Discussion

The discussion draws a boundary: descriptions help when they supply code the agent cannot see, but are redundant and sometimes distracting when source is available. The authors characterize their benchmark and optimizer results as promising within a limited fixture scope, while identifying repository-scale evaluation as a next step.

Discussion

  • Documentation acts as compression for code outside context; inside it, redundant descriptions can distract agents into rewriting rather than patching.The authors say their standard, well-specified issues already provide needed information in the issue and source.Source: §8 · p. 8
  • Description length did not predict the penalty in these data, so the authors treat distraction from redundant content, rather than length alone, as the operative mechanism.They contrast this with prior findings that long inputs can degrade reliability.Source: §8 · p. 8
  • The roundtrip benchmark uses eleven fixtures plus three hand-built codebases from one language and project family, so its numbers characterize the method rather than provide powered measurements.The authors also note that the optimization loop uses few fixtures, few iterations, and a single proposer.Source: §8 · p. 8
  • The authors identify establishing the same documentation boundary at repository scale, where whole codebases do not fit, as a natural next step.Source: §8 · p. 9

A Testing Whether Length Hurts

With the same five facts, compact and verbose descriptions perform equally across model sizes; in a harder task, the apparent length penalty disappears once the verbose version includes the required identifiers. Fixture selection then applies explicit, reproducible filters to define the benchmark set.

A Testing Whether Length Hurts

  • 38-word and 664-word descriptions carrying the same five facts give identical uplift, from Flash-lite down to a 4B model.The experiment measures the fraction of independent attempts that recover all checked values.Source: Table 9 · p. 10, A Testing Whether Length Hurts · p. 10
  • The first harder-task comparison was confounded: its verbose description omitted exact key strings, so it lacked information required to recover the checked values.The task used 28 settings, including deliberately confusable key families, and checked ten values.Source: A Testing Whether Length Hurts · p. 10
  • The fair verbose description passes as well as the compact one on every tested model, showing that the apparent length penalty was a completeness effect.The verbose version that first failed omitted exact key strings; when those were included, even amid long filler and confusable keys, performance matched the compact version.Source: A Testing Whether Length Hurts · pp. 10–11

B Fixture Selection Criteria

  • The scripted single-file and named-oracle filters admit 429 of 2,294 SWE-bench Verified instances before further project, era, and executability constraints narrow the fixture set.The selection criteria also require bounded subpackage size, self-contained tests, and an executable baseline.Source: B Fixture Selection Criteria · p. 11

Results

The source-present SWE-ContextBench Lite evaluation compares issue-only, retrieved-context, and compact-description conditions using the official harness. The discovered description prompt encodes specific implementation details, while the base prompts define the describe, regenerate, and optimization stages.

Results

  • The source-present protocol compares issue-only, retrieved-context, and compact-description conditions, scoring scorable edits with the official harness on fail-to-pass and pass-to-pass tests.The compact description comes from the pre-fix file and is compressed to roughly eighty words; a gold-patch run recovers full resolution.Source: Results · p. 11
  • Issue-only resolution varies by repository, from matplotlib at eight of nine to sympy at six of twenty-one; ten repositories and 68 of 99 tasks are represented.The head-to-head comparisons use the same task set, controlling for this repository-level variation.Source: Results · p. 11, Table 11 · p. 12

D The Discovered Describe Prompt

  • The discovered prompt specifies exact imports, literal module constants, signatures, defaults, and conditions for exceptions and return paths.These instructions map to failure modes identified in the paper; the full prompt appears in the supplementary material.Source: D The Discovered Describe Prompt · p. 12

E Base Prompts

  • The base describe prompt requests a precise behavioral specification, the regenerate prompt implements it from a scaffold, and the proposer rewrites the describe prompt using measured fidelity and length.The proposer is sampled at temperature 0.4, while describe and regenerate scoring use temperature zero.Source: E Base Prompts · p. 12

F OpenWiki Baseline Details

OpenWiki documentation can omit implementation details needed by tests, while the paper’s example description specifies internals and regenerates its file at full fidelity.

F OpenWiki Baseline Details

  • OpenWiki’s human-oriented documentation omits details such as a private helper or relied-on class attribute, causing regenerated modules to fail import; another runnable result misses two tests.The comparison replaces the describe-stage output while keeping the remaining regeneration and evaluation stages identical.Source: F OpenWiki Baseline Details · p. 13
  • On the deny-override fixture, OpenWiki documents the code’s order-dependent behavior rather than the required priority-and-deny-override contract, and the agent fails as it does without documentation.The example distinguishes documentation of observed behavior from documentation of the contract the task requires.Source: F OpenWiki Baseline Details · p. 13

G An Example Description

  • 1.00 fidelity: the contains example specifies exact argument positions, an error string, and property-filtering logic that human-facing documentation typically omits.The example description regenerates the file and passes its tests.Source: G An Example Description · p. 13
Loading 2609.31587v1…