PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Coding Agents Still Need a Debugger? Frontier-Model-Based Fault Localization at the Accuracy Ceiling

Christoph Daxerer, Johann Glock, Christian Macho, Rudolf Ramler, Martin Pinzger, Josef Pichler

arXiv:2610.36015v1cs.SEcs.AI

TL;DR

The study asks how accurately frontier agents localize faults from source code alone across behavior-preserving rewrites, and what debugger assistance adds. Across three tool-use settings, agents localized nearly all faults without tools, while debugger assistance removed many remaining errors at higher token cost.

  • Problem

    It remains unclear how accurately frontier agents localize faults from source alone across behavior-preserving rewrites, and what debugger assistance adds; prior repair studies conflate localization with patch generation.

    Source: §I · p. 2

  • Method

    The study seeds single-line faults into reference programs, rewrites faulty versions with behavior-preserving transformations, and tests localization across representations and three tool-use settings.

    Source: §III · p. 3, A. Localization Cases · p. 4

  • Results

    Without tools, agents localized 98.2–99.9% of original and 96.7–99.9% of transformed faults; encouraged debugging removed 63% and 61% of Luna 6’s and Qwen 3.8’s remaining transformed errors.

    Source: §I · p. 2

  • Takeaways & Limitations

    With assistance optional, agents used the interactive debugger in only 0.2–6.2% of transformed episodes yet retained most accuracy benefit, often preferring execution probes.

    Source: §VII · p. 11

    The findings concern short Python programs for precisely specified algorithmic problems and do not generalize directly to multi-file repositories, incomplete specifications, or multi-statement defects.

    Source: §VI · p. 10

Abstract

from arXiv · show

Background: To localize a fault, a coding agent can reason from source code alone or use debugger assistance by running the program and stepping through it in an interactive debugger. While earlier agents struggled with static execution tracing and debugger operation, frontier LLMs have improved at both. Objective: We evaluate how accurately coding agents localize faults without tools across original and transformed programs, how much debugger assistance improves accuracy and at what cost, and how agents use it when optional. Methods: We seed 610 single-line faults into 144 LeetCodeDataset solutions and rewrite each faulty program with five recipes of behavior-preserving transformations. Agents based on Sol 6.1, Luna 6, and Qwen 3.8 localize each fault with no, optional, or encouraged debugger assistance. Results: Without tools, the agents localized 98.2 to 99.9% of faults in original and 96.7 to 99.9% in transformed programs, losing accuracy mainly where recipes restructured loops and conditions. Encouraged assistance removed 63% and 61% of Luna 6's and Qwen 3.8's remaining transformed errors (+2.1 and +1.3 percentage points) at 14.7 to 27.0 times the token use, while Sol 6.1 stayed at 99.9%. With optional assistance, Luna 6 and Qwen 3.8 still gained +1.5 and +1.3 points, though the agents used the interactive debugger in only 0.2 to 6.2% of runs. Conclusions: On short, precisely specified programs, current coding agents localize nearly all faults from source code alone, and debugger assistance removes most errors that remain. Given the choice, agents executed the program far more often than they stepped through it, yet kept most of the benefit.

I. INTRODUCTION

The paper asks whether frontier coding agents can localize faults from source code alone, how behavior-preserving transformations affect that ability, and whether debugger assistance improves it. In controlled experiments, agents approached the accuracy ceiling without tools, while optional assistance retained gains with little interactive-debugger use.

I. INTRODUCTION

  • Prior work leaves open how frontier models localize faults under behavior-preserving transformations and how optional versus encouraged debugger use changes accuracy and cost.End-to-end repair can confound localization with patch generation, while prior studies have not separated optional from encouraged tool use or compared their costs.Source: §I · p. 2
  • Across 610 seeded faults and five transformation recipes, Sol 6.1, Luna 6, and Qwen 3.8 were evaluated with no, optional, or encouraged runtime tools.The study reports 65,376 agent runs on untransformed and transformed programs.Source: §I · p. 2
  • Without tools, agents localized 98.2–99.9% of faults in untransformed programs and 96.7% (Luna 6), 97.9% (Qwen 3.8), and 99.9% (Sol 6.1) in transformed programs.Encouraged assistance raised transformed accuracy to 98.8% for Luna 6 (+2.1 points) and 99.2% for Qwen 3.8 (+1.3 points), removing 63% and 61% of their remaining errors; Sol 6.1 stayed at 99.9%.Source: §I · p. 2
  • With optional assistance, Luna 6 and Qwen 3.8 gained +1.5 and +1.3 points, although agents used the interactive debugger in only 0.2–6.2% of transformed-program episodes.Agents instead ran execution probes or answered from source code alone; encouragement added only +0.0 to +0.6 points over optional access.Source: §I · p. 2
  • The study measures accuracy gains, error reduction, and resource costs of encouraged assistance, and compares discretionary tool use with encouraged use.The introduction identifies these as separate contributions alongside static localization across transformations.Source: §I · p. 3

II. RELATED WORK

Prior LLM debugging work uses fixed execution pipelines or agent-controlled tools, while transformation studies test robustness or reduce benchmark familiarity. This study isolates fault localization and compares no, optional, and encouraged assistance across original and transformed programs.

II. RELATED WORK

  • Most LLM fault-localization pipelines use fixed execution rules to filter candidates, and prior methods generally rank methods rather than lines or statements.Existing approaches include failing-test coverage, spectrum or information-retrieval rankings, semantic constraints, and agent-inserted print statements.Source: §II · p. 3
  • Transformation research rewrites code to test robustness or reduce training overlap, but prior evaluations did not isolate matched-pair fault localization across five recipes and three tool settings.Prior work also reports varied effects of renaming and structural or lexical obfuscation on repair and code reasoning.Source: §II · p. 3
  • Runtime-evidence systems range from fixed output or trace feedback to agent-issued debugger commands, but prior studies assess repair or explanation under a single tool configuration.The paper contrasts those evaluations with its comparison of no, optional, and encouraged debugger assistance for fault localization.Source: §II · p. 3

III. STUDY DESIGN

The study crosses untransformed and behavior-preserving transformed programs with three tool-use settings to measure localization accuracy, episode cost, and tool use. It frames the design around static accuracy, debugger effects, and discretionary use.

III. STUDY DESIGN

  • Each localization case pairs a faulty program with its task description, and agents localize it under no-tools, discretionary-tools, or encouraged-tools settings in both program representations.The six conditions are A–C for untransformed programs and D–F for transformed programs; an episode is one agent run on a case under a setting.Source: §III · p. 3
  • The study asks how accurately agents localize faults from source alone, how transformations change accuracy, and how much encouraged assistance improves accuracy and costs.A third question asks how often agents choose assistance when optional and how that affects accuracy relative to the other settings.Source: §III · pp. 3–4
  • Table I characterizes admitted programs by counts, LOC and CC medians, transformed-program changes, and preserved four-token-sequence share.The caption specifies interquartile ranges for LOC and CC and defines Kept as the median share of preserved four-token sequences.Source: Table I · p. 4

A. Localization Cases

The study constructs verified matched faulty programs from LeetCode solutions, applies behavior-preserving rewrites, and evaluates agents under controlled interaction settings. The localization cases and agent-interaction protocol define what is tested and how agents respond.

A. Localization Cases

  • The source programs come from 150 LeetCodeDataset problems, sampled equally across Easy, Medium, and Hard difficulty groups.The dataset pairs problem descriptions with verified Python reference solutions and executable tests; self-contained programs support checks of behavior preservation and matching failures.Source: A. Localization Cases · p. 4
  • Each seeded fault is a single edit to one expression on one line, drawn from five fault types: off-by-one, wrong arithmetic, wrong logic, wrong variable, and static return.A randomly chosen edit must fail at least one test; a fault type is skipped when no qualifying edit exists.Source: A. Localization Cases · p. 4
  • Five fixed transformation recipes range from variable renaming and distractors to loop and condition restructuring, and matched pairs are admitted only after behavior and fault checks.Admission requires the transformed reference to pass, the transformed faulty program to reproduce the same test failure, and restoring the mutated expression to make it pass.Source: A. Localization Cases · p. 4
  • Admission retained 144 reference programs, 610 seeded faults, and 3,022 transformed programs; median LOC changes ranged from +0% to +450% across recipes.Median preserved four-token sequences ranged from 37% for Maximal mix to 71% for Distractors.Source: A. Localization Cases · p. 4

B. Agent Interaction

  • Each task gives agents the problem description, faulty program with numbered lines, and shared supporting code, then requests the single most likely affected line.Evaluation tests are withheld, and answers are submitted through report_bug.Source: B. Agent Interaction · p. 4
  • Discretionary and encouraged settings provide execution probes and an interactive debugger; only encouraged settings append an instruction to gather runtime evidence.Prompts otherwise remain identical across settings; the study runs Sol 6.1, Luna 6, and Qwen 3.8 with two episodes per model, case, and setting.Source: B. Agent Interaction · p. 5

C. Measurements

The measurements define fault-localization accuracy at statement and exact-line granularity, and record runtime-tool use and episode cost.

Localization

  • Statement-level grading accepts any line in the faulty statement, including when a transformation splits it across multiple lines.The faulty statement was split across lines in 47% of transformed cases.Source: C. Measurements · p. 5
  • Line-level grading accepts only the exact line containing the mutated expression.Source: C. Measurements · p. 5
  • Top-1 accuracy is the share of episodes that localize the fault successfully; episodes without an answer count as unsuccessful.Source: C. Measurements · p. 5

Tool use and cost

  • The study records tokens, wall-clock time, and monetary cost per episode, including cached-input rates in provider list-price estimates.Self-hosted Qwen 3.8 cost is estimated using DeepInfra list prices.Source: C. Measurements · p. 5

D. Analysis

The analysis compares localization accuracy across tool settings and representations, examines tool use and cost, and quantifies uncertainty with program-level resampling.

RQ1 and RQ2

  • For each case, the analysis defines Top-1 accuracy separately under no-tools, discretionary-tools, and encouraged-tools settings.Source: D. Analysis · p. 5
  • The no-tools change G averages matched-pair transformed-minus-untransformed accuracy differences, while ∆ and ∆′ average encouraged-minus-no-tools effects on untransformed and transformed programs.Source: D. Analysis · p. 5
  • RQ2 also measures the share of no-tools errors removed, the interaction I = ∆′ − ∆, and encouraged-to-no-tools ratios for tokens, time, and cost.Source: D. Analysis · p. 5

RQ3

  • RQ3 compares discretionary tool use, accuracy, and episode cost across settings, and tests whether tool use is associated with larger assistance effects.It examines probe runs and debugger actions separately.Source: D. Analysis · pp. 5–6
  • RQ3 ranks case-level tool use from 0 to 1 and fits ordinary least squares, with the slope estimating the fitted assistance-effect difference between the lowest and highest tool-use ranks.Ranks prevent a few cases with many debugger actions from dominating the fit.Source: D. Analysis · p. 6

Uncertainty

  • The 95% percentile intervals use 2,000 nonparametric cluster-bootstrap resamples of whole reference programs, stratified by difficulty group.Each resample retains associated cases and episodes to capture program-level sampling variation and within-case episode variance.Source: D. Analysis · p. 6
  • RQ3 slope intervals are reported only after checks for defined bootstrap estimates and placebo-interval calibration; otherwise results are marked with a dagger or dash.The placebo validation uses 1,000 cluster-level datasets.Source: D. Analysis · p. 6

IV. RESULTS

Without tools, the agents localized nearly all faults, but transformations that rewrote control flow produced the clearest accuracy losses, often through errors on doubly negated conditions.

A. RQ1: Static Localization and Transformation Sensitivity

  • The weakest untransformed fault types were off-by-one and wrong-logic for Luna 6 and Qwen 3.8, while transformed static-return faults were especially difficult for Qwen 3.8.Luna 6 accuracy on off-by-one and wrong-logic fell to 93.8% and 94.9%; Qwen 3.8 static-return accuracy fell from 99.3% to 97.0%.Source: A. RQ1: Static Localization and… · p. 6
  • Control flow and Maximal mix lowered no-tools accuracy by 2.1–3.8 points for Luna 6 and 2.1–2.4 points for Qwen 3.8, while other recipes changed accuracy by −1.0 to +0.1 points.The two recipes restructure loops and wrap conditions in double negations.Source: A. RQ1: Static Localization and… · p. 6
  • 68.2–71.4% of new errors on Control flow and Maximal mix named doubly negated conditions, which formed only 4.4–7.8% of lines in those programs.Source: A. RQ1: Static Localization and… · p. 6
  • 98.2–99.9% of faults were localized in untransformed programs and 96.7–99.9% in transformed programs; transformation effects were −1.5 points for Luna 6, −0.9 for Qwen 3.8, and +0.0 for Sol 6.1.Source: A. RQ1: Static Localization and… · p. 6

B. RQ2: Debugger Effect

Encouraged debugger assistance improved transformed-program accuracy for Luna 6 and Qwen 3.8, closing nearly all of their static transformation gap, but increased resource use substantially.

B. RQ2: Debugger Effect

  • For both Luna 6 and Qwen 3.8, ∆′ was positive across all five transformation recipes and all five fault types; Luna 6’s 95% intervals were strictly above zero in 9 of 10 subsets.Source: B. RQ2: Debugger Effect · p. 6
  • +2.1 points for Luna 6 and +1.3 for Qwen 3.8 on transformed programs removed 63% and 61% of no-tools errors, bringing encouraged accuracy within −0.1 to +0.1 points across representations; Sol 6.1 stayed at 99.9%.Source: B. RQ2: Debugger Effect · pp. 6–7, Table II · p. 7
  • +3.1 and +4.4 points for Luna 6 and +2.8 and +2.1 for Qwen 3.8 were the largest recipe-level gains, on Control flow and Maximal mix; gains also varied by fault type.Across fault types, gains were largest on wrong-logic and off-by-one faults, and assistance removed 50–87% of no-tools errors.Source: B. RQ2: Debugger Effect · p. 7
  • On transformed programs, encouraged assistance multiplied token use by 14.7–27.0, wall time by 5.3–7.7, and cost by 3.4–5.0 compared with no-tools.Sol 6.1 used 18.4–19.8 times as many tokens without an accuracy gain.Source: B. RQ2: Debugger Effect · p. 7

C. RQ3: Discretionary Use

Optional tool access improved transformed-program accuracy for Luna 6 and Qwen 3.8, while agents rarely used the interactive debugger. Probe use was associated with greater assistance gains for some cases, and optional access achieved gains near those of encouraged use at lower token cost.

C. RQ3: Discretionary Use

  • Debugger actions occurred in only 0.2 to 6.2% of transformed discretionary episodes, while agents often ran probes or answered without tools.Luna 6 called no tool in 87.9% of transformed episodes; Sol 6.1 and Qwen 3.8 commonly ran probes without debugger actions.Source: C. RQ3: Discretionary Use · pp. 7–9, Table IV · p. 8
  • Qwen 3.8’s transformed-program probe-rank slope was +5.3 points, compared with +1.9 points on untransformed programs.Luna 6’s transformed probe-rank estimate was +15.0 points but lacked an interval; debugger-action-rank slopes were descriptive and based on few cases.Source: C. RQ3: Discretionary Use · pp. 8–9, Table V · p. 9
  • +1.5 points for Luna 6 and +1.3 for Qwen 3.8: discretionary tools raised transformed-program accuracy over no-tools, nearly matching encouraged assistance at lower token use.Luna 6 used 3.3 thousand tokens under discretionary-tools versus 55.0 thousand when encouraged; Qwen 3.8 used 38.3 versus 73.5 thousand.Source: C. RQ3: Discretionary Use · pp. 8–9, Table III · p. 8

D. Robustness

The robustness checks found that statement-level grading accepted many answers that diagnosed the seeded fault despite naming a different line. In discussion, the authors attribute transformation-related errors chiefly to particular syntactic rewrites and conclude that optional runtime assistance can retain much of its accuracy benefit at lower cost.

D. Robustness

  • Line-level grading lowered transformed no-tools accuracy to 89.4% for Luna 6 and 96.4% for Qwen 3.8, although audited statement-only answers usually identified the seeded fault.The audit verified the seeded fault in 133 of 135 statement-only answers and 59 of 60 line-level successes.Source: D. Robustness · p. 9

V. DISCUSSION

  • 98.2 to 99.9% of faults in untransformed programs and 96.7 to 99.9% in transformed programs were localized without tools, and debugger use removed many remaining errors.Encouraged debugger use removed 63% and 61% of Luna 6’s and Qwen 3.8’s remaining transformed errors, while Sol 6.1 stayed at 99.9%.Source: §V · p. 9
  • Transformation-related accuracy loss tracked syntactic structure more than surface-code change, with doubly negated conditions prominent among new errors.The authors recommend checking whether errors cluster on specific syntax before attributing transformed-code losses to reduced training-data familiarity.Source: §V · p. 9
  • Optional assistance retained most accuracy gains while interactive debugger actions remained rare, but tool-use comparisons are confounded by agents’ choices and extra model turns.The authors say isolating program execution from interactive debugging requires a dedicated execution-only setting.Source: §V · pp. 9–10
  • Tool frequency did not track accuracy benefit: Luna 6 retained +1.5 of its +2.1-point gain at 3.3 thousand tokens, while Qwen 3.8 matched its encouraged gain at roughly half the tokens.The authors conclude that optional rather than forced multi-turn debugging is more cost-effective on self-contained code, and that scaffolding cannot substitute for a stronger base model.Source: §V · p. 10
  • Statement-level and line-level grading differed where transformations split faulty statements across lines, so evaluations should track the full faulty-statement span.Strict line-level grading would count some valid diagnoses as failures and could make model comparisons depend on which fragment was named.Source: §V · p. 10

VI. THREATS TO VALIDITY

The evaluation’s conclusions are bounded by how transformations, assistance effects, and discretionary use are measured, as well as by the studied program setting.

a) Comparing representations

  • Four-token overlap between program versions does not measure overlap with training data, and transformed programs remain semantic duplicates of public solutions.Residual familiarity can attenuate effects attributed to unfamiliarity.Source: §VI · p. 10
  • The unchanged problem description can let agents recognize familiar LeetCode problems, making transformation sensitivity an imperfect measure of unfamiliarity.The authors therefore interpret G and I as sensitivity to transformation recipes rather than unfamiliarity alone.Source: §VI · p. 10

b) Measuring the assistance effect

  • Encouraged-tools combines execution, interactive debugging, explicit tool-use instructions, and additional model calls, so its measured gain is not attributable to debugging alone.The reported deltas capture the combined effect of these additions versus no-tools.Source: §VI · p. 10

c) Discretionary use

  • The discretionary-use slope measures association between tool use and per-case assistance effects, not why an agent chose a tool.With two episodes per case and setting, noise may attenuate the slope toward zero, so the authors draw no case-level conclusions.Source: §VI · p. 10

d) Generalization

  • The study covers short Python programs for precisely specified algorithmic problems, not multi-file repositories, incomplete specifications, or multi-statement defects.The authors note that interactive debugging has shown larger gains in those settings.Source: §VI · p. 10

e) Reproducibility

  • Commercial API updates can change model behavior, while the estimated dollar cost for self-hosted Qwen 3.8 relies on an external pricing proxy.The authors also report token counts and wall time for each condition.Source: §VI · p. 10

VII. CONCLUSION

Across a large evaluation of single-line faults, agents localized nearly all faults without tools, and encouraged assistance removed most remaining transformed-program errors for two models. The replication package provides the study materials and analysis scripts.

VII. CONCLUSION

  • 98.2–99.9% of untransformed and 96.7–99.9% of transformed faults were localized without tools; assistance removed 63% and 61% of two models’ remaining transformed errors.Encouraged assistance yielded +2.1 and +1.3 percentage points, at 14.7–27.0 times the token use; optional access retained most accuracy benefit with rare debugger stepping.Source: §VII · pp. 10–11

DATA AVAILABILITY

  • The replication package provides the pipeline, programs, prompts, episode traces, and analysis scripts.It is available as reference.Source: DATA AVAILABILITY · p. 11
Loading 2610.36015v1…