Source-linked AI summary
Experimental Settings in LLM-Based Program Repair: A Study of Inputs, Tool Access, Feedback, and Validation
Xushu Dai, Yicheng Cai, Nanqing Luo, Pei-Yu Tseng
TL;DR
A benchmark and repair score alone do not fully specify the task an APR system performed, limiting reconstruction and comparison of reported results. The paper introduces a result-level protocol and machine-readable schema, and finds that shared benchmark identity is insufficient to establish that systems faced the same repair task or support model-level comparisons. The framework makes task conditions and comparison scope explicit.
Problem
A benchmark name and score alone do not reliably reconstruct a system’s observations, permitted actions, or division of repair responsibility, making reproducibility and comparison scope unclear.
Source: §1 · p. 2Method
The paper analyzes reported Defects4J and SWE-bench settings and specifies each result through a protocol and schema that separate task conditions from system configuration.
Source: §1 · p. 2, §4.2 · p. 5Results
Shared benchmark identity is insufficient to identify the same repair task, and score differences alone do not establish that one model or repair method is better when task conditions differ.
Source: §7 · pp. 9–10Takeaways & Limitations
The framework makes reported task conditions explicit so readers can determine which comparisons a result supports.
Source: §1 · p. 2Because corpus eligibility required minimum reporting detail, its missingness counts may understate missingness among otherwise eligible studies.
Source: §5.3 · p. 8
Abstract
from arXiv · showhide
Evaluations of automated program repair (APR) systems commonly report the benchmark, the number of repaired defects, and the tests used for final patch validation, but these items no longer fully specify the repair task presented to a system. Recent LLM-based systems differ in the information supplied before repair, the repository and testing operations permitted during repair, and the feedback returned after unsuccessful attempts, allowing the same benchmark to instantiate substantially different repair tasks ranging from localized patch generation to repository-level diagnosis and iterative repair. We present a framework for explicitly specifying the experimental settings associated with reported APR results. We analyze reported experimental settings from systems evaluated on Defects4J and SWE-bench and characterize each result by its task unit, fault-localization assumptions, initial input, tool access, repair-time feedback, final validation, and resource budget. Our analysis shows that benchmark identity alone is insufficient to reconstruct the evaluated task or determine the appropriate scope of comparison across reported repair rates. We therefore introduce a machine-readable schema for specifying each experimental setting to improve reproducibility and make the scope of cross-system comparisons explicit.
1 Introduction
A benchmark name and repair rate do not specify the information, operations, and responsibilities involved in an APR task. The paper proposes a result-level protocol and machine-readable schema to make those conditions explicit and clarify comparison scope.
1 Introduction
- Benchmark identity alone does not determine the repair task: studies can expose different inputs, assign different workflow responsibilities, and provide different operations and feedback.The same benchmark can support localized patch generation or repository-level fault diagnosis, with tests either reserved for final validation or returned during repair.Source: §1 · p. 2
- The proposed protocol separates benchmark artifacts, initial input, permitted actions and runtime observations, and final validation to make each reported task contract reconstructable.It distinguishes an operation a system may invoke from the information returned by that operation, such as test outcomes or error messages.Source: §1 · pp. 1–2
- The paper analyzes 10 experimental settings from seven studies and introduces a machine-readable schema covering localization, input, tools, feedback, validation, and budget.It also offers guidance on when repair rates support complete-system comparisons and when model- or method-level claims require additional controls.Source: §1 · p. 2
- A benchmark supplies program and evaluation artifacts, while the experimental setting determines what is available during repair and which tests are reserved for final evaluation.This distinction underlies the paper’s framework for specifying repair settings.Source: Figure 1 · p. 3
2 Study design
The study uses a bounded, maximum-variation corpus to describe repair-setting differences, not to estimate how common particular practices are. Its coding rules treat unreported information as unclear and compare task differences through task-defining conditions.
2 Study design
- The study covers source-code editing for reported failures or repository issues, excluding pure bug detection, decompilation, and unconstrained code generation.The corpus is limited to seven studies and 10 settings on Defects4J or SWE-bench.Source: §2 · p. 3, §2.1 · pp. 3–4
2.1 Research questions and corpus scope
- The three research questions concern evaluation setups, task differences within shared benchmarks, and which setting details are clearly reported or remain unclear.Maximum-variation sampling captures design variation rather than estimating practice prevalence; eligibility requires enough detail to code localization, initial evidence, and final validation.Source: §2.1 · pp. 3–4
2.2 Coding procedure and reproducibility
- Coding treats an artifact as model-visible only when the study establishes its inclusion in the prompt or workspace or its return through a permitted operation.Missing information is coded as unclear rather than inferred, and records represent separately reported settings rather than individual runs or bugs.Source: §2.2 · p. 4
2.3 Task-difference analysis
- Two records on the same benchmark family count as different repair tasks when task units or task-defining parts of T differ, but configuration differences alone do not suffice.The compared parts of T are initial exposure, localization, permitted actions, runtime observations, and final validation.Source: §2.3 · p. 4
3 Related work
Prior APR work recognizes many individual evaluation factors, while this paper’s contribution is to encode their joint evidence flow. It also distinguishes its evaluation-specification focus from work on prompt fact selection, agent methods, and broader coding-agent comparisons.
3 Related work
- Prior APR work records factors including benchmark variants, metrics, base models, localization assumptions, input/context, repository-level repair, leakage, and test-time scaling.The paper’s contribution is a result-level encoding that separates artifact presence from exposure and distinguishes initial exposure, runtime observations, and final validation.Source: §3 · p. 4
- The framework specifies what evidence a reported system could access and what comparisons its score supports, unlike work focused on selecting prompt facts or using execution evidence within a repair method.It specializes the broader concern that coding-agent harnesses can confound comparisons by model name alone to APR’s benchmark-to-evidence mapping, localization, actions, feedback, and validation.Source: §3 · p. 4
4 From a benchmark to an experimental repair task
The paper separates benchmark artifacts from what a system initially sees, can observe through actions, and receives only at final evaluation. It formalizes those task conditions separately from system configuration and uses that distinction to bound comparisons.
4.1 Four evaluation dimensions
- Benchmark artifacts do not imply model visibility; initial input includes prompt and workspace contents, runtime observations come through permitted actions, and evaluator-only oracles remain distinct from diagnostic input.A repository checkout can expose files beyond those named in the initial prompt, while held-out tests or a PoC should not be silently counted as repair-time evidence.Source: §4.1 · p. 5
4.2 Repair evaluation protocol
- The repair evaluation protocol T represents environment, initial input, localization, allowed actions, runtime observations, and the evaluator-only final validation oracle.A test file may exist in the environment without appearing in initial input or any permitted observation.Source: §4.2 · p. 5
- System configuration S records model provenance, policies, budget, and run controls separately from the task’s permitted action interface.Separating T from S avoids defining every model comparison as a different task.Source: §4.2 · p. 5
- Tests and other evidence may be available during diagnosis, reserved for post-patch validation, or partly hidden; each choice must be reported for its scientific question.A test that returns feedback during repair belongs to runtime observations even if a related held-out test remains in final validation.Source: §4.2 · p. 6
- A shared benchmark and final test suite provide a common outcome measure, but direct comparison of system components requires matching relevant task and configuration factors or a controlled ablation.An ablation supports claims only for the reported implementation and does not replace specifying other results’ task contracts.Source: §4.2 · p. 6
5 Empirical results
Across the studied corpus, Defects4J settings assign different combinations of localization, repair-time feedback, and workspace access, spanning localized patch generation through repository-level diagnosis and repair. Reported descriptions also leave some workspace permissions and resource bounds unresolved.
5.1 RQ1: Variation in reported experimental settings
- Five of 10 settings use oracle localization, one uses a reported localization tool, one derives localization from issue or test information, and three derive it through repository actions.Six settings explicitly describe execution or tool output returned for later repair decisions; three start with a repository workspace, and three name a benchmark-harness test patch as the final oracle.Source: §5.1 · p. 6
- The same benchmark family evaluates localized patch generation, repair with repeated test feedback, and repository-level diagnosis and repair.The corpus includes Defects4J settings with oracle localization, Ochiai-based localization, and repository-level fault localization.Source: §5.1 · p. 6
5.2 RQ2: Different repair tasks constructed from the same benchmark
- AlphaRepair’s two records hold system family and reported budget constant while changing localization from an oracle location to tool-derived suspicious locations.A difference between these records reflects the localization condition as well as patch generation, not a clean estimate of an underlying model difference.Source: §5.2 · p. 7
- ChatRepair feeds compilation or test failures from earlier attempts into later turns, while its single-line and single-function settings differ in task unit and repair budget.Those budgets are 200 versus 100 attempts; ThinkRepair also permits testing information to be appended during interaction, making tests diagnostic evidence as well as validation.Source: §5.2 · p. 7
- RepairAgent lets the model choose repository tools and incorporates their results into its prompt, whereas Agentless reports fixed stages without autonomous tool use.AutoCodeRover and SWE-agent also expose repository exploration through action interfaces, so complete-system results combine different information-acquisition and control arrangements.Source: §5.2 · p. 7
5.3 RQ3: What remains unresolved within the selected corpus
- In six of 10 records, reported descriptions do not establish workspace permissions well enough to determine whether arbitrary repository artifacts could be read.Four records omit a common resource bound; these counts are conditional on a detail-selected corpus and may understate missingness among eligible studies.Source: §5.3 · pp. 7–8
6 Discussion and implications
A common benchmark and final test support comparison of complete systems, but do not by themselves justify attributing score differences to a model or repair method. Interpreting results also requires reporting localization, feedback, sampling, budget, contamination risks, and a machine-readable evidence manifest.
6.1 A common final test supports a system-level comparison
- With the same benchmark instances and final tests, repair counts compare complete systems under their reported settings, not necessarily their base models or methods.Differences in localization, input, tools, feedback, sampling, or budget limit stronger attribution.Source: §6.1 · p. 8
6.2 Localization is information
- Naming the localization regime as oracle, generated, search-based, or absent distinguishes how much debugging responsibility the system receives.A supplied buggy method, suspicious line, or changed file removes a substantial part of the debugging problem; repository tools and test failures can also provide localization evidence.Source: §6.2 · p. 8
6.3 Validation feedback can become diagnosis feedback
- Compilation and test output becomes diagnosis feedback when an iterative system uses it to choose subsequent edits.Reports should specify feedback granularity, accessible test identities, validation rounds, and whether feedback comes from public or hidden tests.Source: §6.3 · p. 8
6.4 Sampling and budget change the measured capability
- Pass@k measures success across k sampled candidates, not the quality of one repair trajectory, and varies with sampling and validation choices.Agentic comparisons should account for tool calls, context growth, runtime, and monetary cost; RepairAgent reports time, tokens, and cost [1].Source: §6.4 · p. 8
6.5 Contamination and temporal validity
- Public benchmark artifacts may occur in training or retrieval data, especially for small, long-lived benchmarks and unrestricted web-enabled agents.Temporal splits, decontamination searches, held-out repositories, post-training tasks, and transparent model release dates improve evidence.Source: §6.5 · p. 9
6.6.1 Publish an evidence manifest
- A concise evidence manifest can report initial context, permitted actions, evaluator-only tests, localization, and budget for each headline result.Serialize task and system separately, identify the schema version, benchmark, and environment, and preserve value sources; distinguish permitted actions from actions used in a trace, and leave missing information unclear.Source: §6.6.1 · p. 9
7 Conclusion
In the studied corpus, one benchmark family can support materially different repair tasks, and reported descriptions leave some setting details unresolved. The framework makes experimental conditions explicit so complete-system comparisons can be interpreted without treating score differences alone as evidence of model or method superiority.
7 Conclusion
- In the studied corpus, one benchmark family includes oracle, tool-derived, and agent-derived localization, while six settings feed execution or tool observations into later repair decisions.Descriptions frequently leave workspace permissions or a common budget unresolved.Source: §7 · p. 9
- When localization, input, feedback, tools, sampling, or budget differs, score differences alone do not establish that one model or method is better.A controlled ablation can establish the effect of one factor within an implementation, while a common specification helps interpret independently reported tasks.Source: §7 · pp. 9–10