Source-linked AI summary
Does Fault Localization Beat a Fresh Attempt? A Placebo-Controlled Study of Test-Guided Code Repair
Anik Jha
TL;DR
The paper asks whether test-guided fault localization improves code repair beyond the benefits of a small edit or an additional model sample. Using blind resampling, localized infilling, and a random-span placebo, it finds that blind resampling decisively outperforms localized infilling, while the pooled location effect remains suggestive rather than established.
Problem
The study separates whether localized repair succeeds because tests identify useful code, because the edit is small, or because an additional model sample helps without using the failure signal.
Method
The paper compares blind whole-solution resampling, spectrum-based suspect-span infilling, and same-length disjoint random-span infilling across three models, benchmarks, and 488 failing candidates, with a declared fourth-model replication.
Results
Blind resampling beats localized infilling at matched attempt counts (3:40, p = 3.0 x 10^-9), while localized infilling leads the pooled random-span placebo comparison (11:1, Holm-adjusted p = .019) without resolving in any individual model under the primary analysis.
Takeaways & Limitations
Localization is rarely available, and within the tested 24-32B models a fresh attempt is more effective than targeted infilling despite the latter using far fewer generated tokens per attempt.
Takeaways & Limitations
The conclusions condition on localizable failures from one greedy baseline and function- or class-level Python tasks, excluding parse failures, hidden-only failures, diffuse spectra, repository repair, and multi-hunk editing.
Abstract
from arXiv · showhide
Fault localization can focus a code model's repair on the statements a failing test implicates, but a targeted edit may succeed merely because it is small, and a second model call may succeed without using the failure at all. We separate these explanations with three arms applied to the same failed candidate: blind whole-solution resampling, spectrum-based localization followed by suspect-span infilling, and same-length infilling at a disjoint random code span. Across three frozen 26-32B models, three benchmarks and 488 failing candidates, plus a separately declared 24B fourth model from a third family, three results follow. First, localization is rarely available: only 9.0% of failing candidates expose a failing public test with a usable spectrum. Second, among the 177 candidates localizable from a strong suite, localized infilling loses decisively to blind resampling at a matched attempt count (3:40, p = 3.0 x 10^-9), opposite to our hypothesis; the loss replicates in a third family at -11.3 points (95% CI [-16.6, -6.8]), and widening the edit does not rescue it. Third, against the random-span placebo localized infilling leads pooled (11:1, Holm-adjusted p = .019), but that lead resolves in no individual model under the analysis our shipped plan designates primary (best Holm p = .087), so we report the location effect as suggestive rather than established. Re-pricing attempts as tokens narrows but does not overturn this: a span attempt spends 21.7 generated tokens against 371.1, yet 16 localized attempts reach 6.8% while one blind attempt already reaches 10.1%. Infilling reproduces the removed span verbatim in 48.9% of attempts, which is why more budget does not help. We restrict every localization conclusion to the 24-32B models tested.
1 Introduction
The study separates the value of test-derived localization from the benefits of small edits and additional sampling using blind resampling, localized infilling, and a random-span placebo. Localization is usually unavailable, and where available, localized infilling loses to blind resampling while showing only suggestive gains over the placebo.
- Study question: Three arms compare blind whole-solution resampling, spectrum-based suspect-span infilling, and same-length disjoint random-span infilling on the same failed candidate.The controls distinguish the value of the chosen location from the benefit of editing a small span or simply trying again.
- Main findings: 9.0% of failing candidates expose a failing public test with a usable spectrum, making localization usually unavailable.The audit covers 488 failures.
- Main findings: 3:40 paired outcomes favor blind resampling over localized infilling at matched attempts (p = 3.0 x 10^-9), reversing the hypothesis.The loss also replicates in a fourth model from a third family at -11.3 points with 95% CI [-16.6, -6.8].
- Main findings: 11:1 pooled paired outcomes favor localized infilling over the random-span placebo (Holm-adjusted p = .019), but the location effect is only suggestive.No individual model resolves the placebo comparison under the primary analysis.
- Contribution: The contribution is an evaluative two-control protocol, a localizability audit, attempt-and-token budget analysis, and safeguards rather than a new repair algorithm.The paper makes no formal-verification or correctness-guarantee claim.
2 Related work
Prior work studies blind retries, oracle- or predictor-based localization, trajectory intervention, and infilling, but does not isolate predicted test-derived location value from retry benefits with matched controls. This paper positions its comparison as complementary to repository-level and other intervention settings.
- Blind retry: Blind resampling can match or outperform feedback-conditioned self-repair in frozen small code models, motivating an untargeted regeneration control.Those studies regenerate entire solutions and do not test targeted spans at larger model scale.
- Localization: Oracle localization and predictor-based repository localization report repair benefits, but their settings do not provide the present matched-attempt blind and random-location controls.Published contrasts include ground-truth locations, structured repository localization, and predicted file locations.
- Targeted intervention: Luan et al. report 20.15% repair for symptom-driven intervention versus 6.90% for unguided rerunning, opposite to this paper’s localized-infill ordering.Their intervention replays a trajectory prefix and regenerates downstream rather than replacing a localized code span.
- Infilling: Infilling studies establish the primitive’s viability and report benefits from suffix context, but they lack a random-location arm or matched attempt budget.The compared settings also differ in model era, patch counting, and whether bug location is assumed.
3 Experimental design
The experiment conditions all three arms on the same localizable failures, applies matched stochastic attempts across frozen models and benchmarks, and evaluates both attempt-level and unlock outcomes. It distinguishes a strong-suite upper bound from the public signal available to deployed loops.
- Study population: Three-arm analyses include only dead candidates with a failing localizing-suite test and line coverage; nonlocalizable failures are excluded from resampling outcomes.The study separately replays every dead candidate to audit this conditioning decision.
- Study population: The strong suite raises localizable candidates from 44 to 177, but uses augmented tests unavailable to deployed loops and is reported as an upper bound.The public signal uses only HumanEval and MBPP original cases.
- Localization: Ochiai SBFL scores lines using failing and passing coverage, and the localized span is the contiguous range spanning all maximally tied lines.Candidates are excluded when the span exceeds 60% of source lines or lacks a disjoint same-length code-only placebo.
- Arms and sampling: K = 16 stochastic samples per arm use temperature 0.8 and nucleus probability 0.95.Blind resampling generates complete solutions; localized and placebo arms replace localized or disjoint same-length executable spans.
- Budget: Attempts are matched, but measured output averages 21.7 generated tokens for span replacement versus 371.1 for whole-solution resampling.Token-priced analyses therefore report a different budget currency from the primary attempt-matched comparison.
- Capability probe: The capability probe on smaller Qwen models yields no usable location comparison because 65.5% of spliced programs do not parse.It is treated as a boundary on the technique rather than as a fourth model analysis.
- Scoring: Per-attempt success is primary, while exact McNemar/binomial unlock testing is secondary; per-model tests use task-clustered bootstrap and Holm correction.All arms are generated in one harness pass on the same candidates and decoding rules.
4 Results
Test-guided localization was available for only a small minority of failures. On the strong-signal subset, localized infilling beat the random-span placebo only in pooled analysis but lost decisively to blind resampling, with compute and infilling-repetition patterns explaining why additional span attempts did not compensate.
- 4.1 How often can test-guided repair fire at all?: 9.0% of 488 candidates were localizable from public tests, versus 177 candidates (36.3%) under the strong suite.Public-test failures often lacked any failing visible test, while strong-suite localization was frequently blocked by diffuse suspiciousness ties.
- 4.1 How often can test-guided repair fire at all?: A deployed loop could act on only 9.0% of failures, and the strong suite adds bugs that survive every public test.The paper therefore treats public-test localization as a biased subset and the strong suite as an upper bound on available signal.
- 4.2 Does the location matter, and does it beat starting over?: 11:1 pooled discordance favored localized infilling over the random-span placebo, but no individual model resolved after Holm correction (best Holm p = .087).The pooled difference was 4.3% versus 1.1%, or +3.2 points, but the analysis plans do not treat models as exchangeable confirmatory replicates.
- 4.3 Attempts are the wrong unit of cost: 21.7 generated tokens per span attempt versus 371.1 per whole-solution attempt shifted the token crossover but did not overturn blind resampling’s advantage.One blind attempt reached 10.1%, while 16 localized attempts reached 6.8% in total; the 372-token threshold is population-specific and excludes prompt cost.
- 4.4 Why the span arms saturate, and whether more room helps: 48.9% of localized infilling attempts reproduced the removed span verbatim, versus 10.2% for blind resampling, limiting edit diversity and gains from extra attempts.Localized infilling produced 0.23 distinct programs per attempt, compared with 0.83 for resampling.
5 Implementation findings and threats to validity
The study identifies multiplicity and external-validity constraints that narrow how its repair comparisons should be interpreted.
- Multiplicity: p = 1.2 × 10^-8 for resampling and p = .019 for placebo comparisons after Holm correction across four pooled comparisons.Per-model confirmatory analyses resolve only the resampling comparison, in two of three checkpoints, under both planned tests.
- External validity: The analysis conditions on localizable failures from one greedy baseline per model and function- or class-level Python tasks.It therefore does not describe parse failures, hidden-only failures, diffuse spectra, repository repair, or multi-hunk editing.
- External validity: The Qwen interface rerun leaves the resampling loss negative in both models, at -4.9 and -9.6 points.This tests the prompted path used by Gemma against the native FIM interface used by Qwen models.
6 Broader impact
The paper argues for simpler repair baselines and cautions that weak tests and surgical-looking patches can create security and correctness risks.
- Broader impact: A fresh-attempt baseline can prevent deployment of a complicated repair loop when fresh attempts perform similarly.The practical consequence is reduced engineering and inference cost.
- Broader impact: Weak test suites can allow test-guided generation to produce plausible but incorrect or insecure code.The paper recommends sandboxing, review, and testing against independent oracles.
7 Reproducibility
The reproducibility workflow ties reported quantities to shipped result files and reruns documented commands in a fresh extraction.
- Reproducibility: Every table, figure, and headline quantity is emitted from shipped result files rather than transcribed.The README maps each quantity to its file, and missing inputs fail the build.
- Reproducibility: The build reruns every documented command in a fresh extraction while tracing the emitter’s file reads.This connects generated paper outputs to the files used to produce them.
A The capability probe
The capability probe finds that sub-4B models produce many invalid splices, so it bounds the technique rather than measuring capability moderation.
- The capability probe: 71 of 263 tasks were unlocked by blind resampling, versus 1 by localized infilling and 0 by the placebo.The probe produced 256 localizable pairs but no usable comparison between the two span arms.
- The capability probe: 65.5% of small-model spliced programs do not parse, compared with 2.5% for deployment-scale models.Only 34.5% of small-model splices are valid Python, preventing the placebo contrast from carrying a usable comparison.
- The capability probe: The authors report the probe as a boundary on the technique, not as a capability moderator.The experiment does not separate inability to use a localized gap from inability to follow the raw fill-in-the-middle format.
- The capability probe: The probe establishes that blind-resampling results extend below 7B parameters with a large margin.This conclusion is reported as consistent with earlier small-model findings.
B Implementation defects
The study found and corrected implementation defects affecting execution limits, infilling termination, and truncation handling.
- Controls and localization: Comment and docstring spans formed weak placebos, so the final sampler excluded them and required disjoint equal-length spans.The procedure also addressed ties among statements with identical spectra by spanning the full contiguous maximum-suspiciousness region.
- Execution safeguards: A flat five-second test timeout was unreliable because a correct solution needed about 5.1 seconds under the study hardware.The revised EvalPlus-compatible bound uses the larger of one second and four times the reference runtime; memory remained separately relevant.
- Infilling extraction: Missing FIM stop-token registration caused generations to overrun the intended span and splice unparseable text into 13.4% of localized attempts.The fix reads terminators from each tokenizer and adds truncation detection.
C Widening the localized edit
Widening the localized edit reduced no-op regeneration but did not overturn the method’s loss to blind resampling; adaptivity added little beyond width.
- C Widening the localized edit: Every widened variant still lost to blind resampling on the 155 candidates where all variants remained valid.The larger-edit comparison included static widening, adaptive widening, and a same-size placebo.
- C Widening the localized edit: A +4.9-point Qwen3.6 gain over narrow editing was matched exactly by always editing wider, while adaptivity added at most +0.6 points.The decomposition therefore attributes the movement to width rather than adaptive localization.
- C Widening the localized edit: Conditioning on attempts that changed the program narrowed the gap but did not close or reverse the negative comparison.This addresses whether degenerate no-op attempts alone explain the result.
E Pre-specified decision rule and artifact map
The pre-specified decision rule required localized infilling to beat both controls and replicate across model families; the reported evidence satisfies replication only in the direction of loss.
- E Pre-specified decision rule and artifact map: The win bar required at least a five-point placebo margin, Holm-adjusted p < .05, superiority over blind resampling, and replication across two model families.The rule was called pre-specified rather than preregistered because it was not externally registered.
- E Pre-specified decision rule and artifact map: The pooled Table 4 tests find strong-signal resampling p = 1.2 × 10^-8 and placebo p = .019, but neither public-signal comparison survives correction.The paper treats these pooled comparisons as sensitivity summaries; Table 1 is primary.
- E Pre-specified decision rule and artifact map: Replication is satisfied only in the inverted direction: the loss holds across two Qwen checkpoints and a later third-family replication.No confirmatory follow-on or positive-mechanism claim is reported.
- E Pre-specified decision rule and artifact map: The public analysis record distinguishes the kickoff unlock rule from the primary per-model success-rate comparison under task-clustered bootstrap and Holm correction.The record treats pooled task–model-pair tests as sensitivity analyses rather than confirmatory evidence.
- E Pre-specified decision rule and artifact map: The released supplement maps Table 4 and every figure to underlying result files and statistics summaries, while preserving exact localized and placebo spans.Placebo spans depend on process-derived seeds recoverable from released checkpoints.
F The prompted pseudo-FIM interface ablation
The prompted pseudo-FIM ablation isolates interface effects on the same Qwen candidates and comparator, finding small changes that attenuate but do not reverse the localization loss.
- F The prompted pseudo-FIM interface ablation: The ablation holds candidates, spans, funnel, and blind comparator fixed while changing only native FIM tokens to a marked-gap chat prompt.Both Qwen models contribute 142 paired candidates under a two-model Holm family.
- F The prompted pseudo-FIM interface ablation: The prompted path had a 5.3% localized splice parse-failure rate versus 2.5% natively, with a 49.7% no-op rate essentially matching native execution.This made the interface comparison usable rather than a capability-probe failure.
- F The prompted pseudo-FIM interface ablation: The Qwen2.5-Coder loss narrowed enough to stop resolving under correction, but the interface did not reverse either model’s sign or explain Gemma’s different behavior.A prompted-path placebo comparison resolved in one model, but it was exploratory rather than evidence for a location effect.