PaperlayerView exact revision on arXiv

Source-linked AI summary

Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?

Wang Bill Zhu, Miaosen Chai, Shangshang Wang, Yejia Liu, Song Bian, Honghua Dong, Willie Neiswanger, Robin Jia

arXiv:2604.17338v4cs.SEcs.CL

TL;DR

Existing unit-test-only benchmarks do not distinguish targeted debugging from full regeneration or track partial repair of multi-bug programs. PDB converts coding datasets into precision-aware debugging benchmarks, and evaluations show frontier models can pass tests while remaining imprecise in their edits.

  • Problem

    Existing debugging benchmarks rely on unit tests, which do not distinguish minimal targeted fixes from regeneration or measure partial repair of multi-bug programs.

    Source: §1 · p. 2

  • Method

    PDB converts coding datasets into debugging benchmarks by synthesizing verified atomic bugs, composing multi-bug programs, and evaluating patches with edit-level precision and bug-level recall.

    Source: §1 · p. 2

  • Results

    >76% unit-test pass rates but ≤45% edit precision are reported for GPT-5.1-Codex and DeepSeek-V3.2-Thinking.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that improving debugging requires rethinking both evaluation and post-training objectives.

    Source: §7 · p. 10

Abstract

from arXiv · show

Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during debugging. To evaluate how far LLMs are from precise debugging, we introduce the Precise Debugging Benchmark (PDB) framework, which automatically converts any coding dataset into a debugging benchmark with precision-aware evaluation. PDB generates buggy programs by synthesizing verified atomic bugs and composing them into multi-bug programs. We define two novel metrics, edit-level precision and bug-level recall, which measures how many necessary edits are made and how many bugs are resolved. We release two evaluation benchmarks: PDB-Single-Hard on single-line bugs, and PDB-Multi on multi-line bugs. Experiments show that frontier models, such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking, achieve unit-test pass rates above 76% but exhibit precision below 45%, even when explicitly instructed to perform minimal debugging. Finally, we show that iterative and agentic debugging strategies do not substantially improve precision or recall, highlighting the need to rethink post-training pipelines for coding models.

1 Introduction

PDB addresses a mismatch between unit-test evaluation and precise debugging, where passing tests can conceal extensive rewrites or unresolved bugs. Its experiments show that strong test performance can coexist with low edit precision.

1 Introduction

  • Real-world software work centers on debugging and maintenance, yet frontier LLMs may rewrite large portions of buggy programs instead of making precise, intent-preserving edits.The paper illustrates this over-editing with GPT-5.2 rewriting a binary-search solution.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • Existing unit-test-only benchmarks score full rewrites and minimal fixes equally, obscure partial repairs, and rely on public debugging data vulnerable to contamination and costly manual construction.Source: §1 · p. 2
  • PDB converts coding datasets into debugging benchmarks by synthesizing verified atomic bugs and composing independent multi-bug programs, then evaluates patches with edit-level precision and bug-level recall.The framework rewards targeted fixes and penalizes unnecessary modifications.Source: §1 · p. 2
  • GPT-5.1-Codex and DeepSeek-V3.2-Thinking exceed 76% unit-test pass rates but reach ≤45% edit precision, while Qwen3-Coder-480B scores 70% and 66%, respectively.This ranking inversion persists on multi-line and repository-level PDB-WILD; iterative and agentic strategies can improve unit-test performance but do not meaningfully improve precision or recall.Source: §1 · p. 2
  • The authors construct 5,751-example PDB-SINGLE and 484-example PDB-WILD benchmarks from BigCodeBench, LiveCodeBench, and SWE-smith.Source: §1 · p. 2

2 Precise Debugging Setup

Precise debugging evaluates proposed edits rather than only whether the final program passes tests. It defines edit-level precision and bug-level recall to distinguish necessary fixes from unnecessary changes and track bugs resolved.

2 Precise Debugging Setup

  • Conventional pass/fail unit-test evaluation cannot penalize unnecessary edits or rewrites, so the setup evaluates the specific edits a system proposes.A debugging system maps a buggy program and task description to a revision; the conventional function returns 1 only if the revision passes all tests.Source: §2 · p. 2
  • A minimal correction is the smallest set of line edits that makes a buggy program pass its unit tests; a bug is atomic when such edits occupy contiguous lines.The setup also calls two bugs independent when the minimal corrections for their composed program are the pairwise union of the individual corrections.Source: §2 · pp. 2–3
  • For each bug, semantic correctness is tested by replacing its ground-truth edits with the closest predicted edits and checking whether the resulting pseudo-revision passes unit tests.The ground-truth edits are decomposed into contiguous, non-overlapping bug edit sets; precision averages over predicted edits, while recall averages over bugs.Source: §2 · p. 3
  • The setup relaxes strict minimality with an epsilon tolerance, allowing up to |Ei| + ϵ edited lines for each bug.Because semantically correct predicted edits may still contain regeneration, essentialU searches their subsets for minimal edits that preserve semantic correctness.Source: §2 · pp. 3–4

3 Generation and Evaluation Pipeline

PDB has two stages: it generates and verifies atomic bugs, composes them into multi-bug programs, then evaluates automated debugging systems with unit-test accuracy, edit-level precision, and bug-level recall.

3.1 PDB generation

  • PDB generation spans five ODC defect categories and creates single-line or contiguous multi-line bugs, retaining only injected programs that fail unit tests.Single-line bugs use insertion, deletion, or substitution on editable lines; multi-line instances are kept only when reverting any strict subset of modified lines still fails tests. The pipeline composes spaced, distinct block edits into multi-bug programs and subsamples to limit over-representation of task-program pairs.Source: §3.1 · p. 4

3.2 PDB evaluation

  • Evaluation prompts single-pass LLMs or LLM-based agents to debug a buggy program with its task description and optional unit tests or error feedback.It reports Pass@1 unit-test score alongside precision and recall, micro-averaging first within each bug count and then across bug counts.Source: §3.2 · p. 5

4 Evaluation Sets

The evaluation sets apply the PDB generation pipeline to single-line and contiguous multi-line bugs, drawing tasks from coding benchmarks spanning API use, competitive programming, and repository-level engineering.

Evaluation sets

  • The task sources are BigCodeBench [Zhuo et al., 2024], LiveCodeBench [Jain et al., 2024], and SWE-smith [Yang et al., 2025a], while four frontier LLMs generate bugs.The bug-generation pool includes GPT-5.1-Codex, Claude-Sonnet-4.5, Gemini-2.5-Pro [Comanici et al., 2025], and Claude-Opus-4.7 for SWE-smith-specific generation.Source: §4 · p. 5

PDB-SINGLE

  • 5,751 challenging examples make up the final PDB-SINGLE benchmark, after filtering the initial 7,591-example set for examples easy for at least 7 of 9 models.Models were prompted for minimal edits; an example was considered easy only if it achieved perfect precision, recall, and unit-test score for at least 7 models, using precision tolerance ϵ = 2.Source: §4 · p. 5

PDB-WILD

  • 484 examples make up PDB-WILD: 256 use multi-line blocks in BigCodeBench and LiveCodeBench programs, and 228 come from sampled SWE-smith repositories.The benchmark uses maximum block sizes of 4 lines for coding-benchmark programs and 30 lines for repository files, with tolerance ϵ = 1 because multi-line blocks cannot strictly guarantee atomicity.Source: §4 · p. 5

Evaluation sets

  • As bug counts increase, precision generally correlates negatively with unit-test score, while recall behavior depends on the dataset.Figure 4 shows this relationship on PDB-SINGLE subsets from BigCodeBench and LiveCodeBench.Source: Figure 4 · p. 6

5 Experiment Results

Across single-line and multi-line benchmarks, current models and agents still struggle with precise, edit-aware debugging: functional success often comes with unnecessary edits, and feedback or prompt constraints do not eliminate regeneration.

5.1 PDB-SINGLE overview

  • No frontier model exceeds 72% edit-level precision on PDB-SINGLE, even when instructed to make minimal edits, despite some models achieving strong debugging performance.The results distinguish targeted debugging from pass-driven regeneration, which unit-test pass rates alone do not reveal.Source: §5.1 · p. 6
  • Claude-Sonnet-4.5 and Gemini-2.5-Pro debug correctly on more than 75% of cases with precision above 71% and recall above 81%, while pass-oriented models have precision at or below 48%.The results identify distinct strategies: some models are weak but relatively precise, while others identify buggy regions but struggle to produce correct, precise fixes.Source: §5.1 · p. 6
  • As injected bugs increase from 1 to 4, unit-test scores fall and edit-level precision declines, while recall varies by less than 5% on BigCodeBench but rises with unit-test scores on LiveCodeBench.More bugs increase the chance of modifying a necessary line but also increase unnecessary edits; recall reflects per-bug difficulty and varies by dataset.Source: §5.1 · p. 6

5.2 Iterative and agentic debugging

  • Iterative and agentic debugging improve unit-test scores and recall but usually leave precision unchanged or lower; test and execution feedback often triggers broad rewrites rather than fault localization.Agentic debugging often underperforms iterative debugging in precision, and Claude-Code reaches only about 50% precision despite feedback access.Source: §5.2 · p. 7, Figure 5 · p. 7

5.3 Multi-line bug extension

  • On PDB-WILD, Claude-Sonnet-4.7 and Gemini-3.1-Pro exceed 69% precision, while GPT-5.5 scores above 71% on unit tests, 74% recall, and below 55% precision.The precision gap persists for multi-line and repo-level bugs, and increasing bug granularity does not close it.Source: §5.3 · pp. 7–8, Table 1 · p. 7

5.4 Analysis of prompting and data generation

  • Freeform prompting lowers precision and recall across models: Gemini-2.5-Pro loses 40 percentage points in precision, and GPT-5.1-Codex falls below 20% precision.Even the strongest models achieve less than 60% precision under freeform prompts; minimal-debug instructions reduce over-editing but do not fundamentally change model behavior.Source: §5.4 · p. 8, Figure 6 · p. 8
  • Rewriting ground-truth solutions improves edit-level precision by 2.8–3.5% on average, while cross-model bug generation lowers unit-test pass rates by up to 1.4%.These findings suggest contamination may marginally affect performance but does not explain pervasive regeneration; GPT-5.1-Codex-generated bugs are easiest to debug and Claude-Sonnet-4.5-generated bugs hardest across metrics.Source: §5.4 · p. 8, Table 2 · p. 8, Table 3 · p. 8

5.5 Metric verification and error analysis

  • Among passing cases with imperfect precision or recall, 66.8% modify correct code blocks; precision agrees strongly with expert judgments (Spearman ρ = 0.72), and bug-level recall is accurate for over 97.5% of data.In passing cases, 83.5% have full recall but imperfect precision; 70% of passing cases with imperfect recall are functionally correct but evade recall detection through over-editing or structural rewrites.Source: §5.5 · p. 9

6 Related Works

Prior work has developed agentic debugging systems and benchmarks covering varied debugging settings, but unit-test evaluation can reward broad rewrites as much as targeted fixes. PDB adds edit-level precision and bug-level recall to expose this limitation and align evaluation with real-world debugging.

6 Related Works

7 Discussion … B Additional Experiments

PDB exposes a gap between passing tests and precise debugging, and its precision-aware metrics and data-generation pipeline are proposed as tools for both evaluation and training. The related discussion contrasts PDB with binary repair benchmarks, describes its defect taxonomy, and outlines additional experiments on robustness and transfer.

7 Discussion

  • Frontier LLMs often pass unit tests while relying on solution regeneration rather than targeted edits, exposing a gap between functional correctness and fault localization.PDB measures this gap with edit-level precision and bug-level recall, which unit-test-only evaluation misses.Source: §7 · p. 9
  • PDB’s precision-aware metrics can provide fault-localization and edit-minimality reward signals, while its plug-and-play pipeline enables scalable debugging-data construction.The paper presents PDB as infrastructure for closing the training loop in self-improving coding models.Source: §7 · p. 10

A Additional Related Works

  • Targeted rewriting experiments found memorization was not the root cause of regenerator-style debugging, testing the hypothesis of Zhang et al. [2025].Source: A Additional Related Works · p. 15
  • Prior debugging work spans unsupervised repair [Yasunaga and Liang, 2021], multitask code-text learning [Phan et al., 2021], and agent-based systems [Lee et al., 2025, Bouzenia et al., 2025, Yang et al., 2025b].Source: A Additional Related Works · p. 15
  • Existing debugging benchmarks’ binary test outcomes cannot distinguish targeted fixes from regenerated solutions, whereas PDB evaluates repair precision, minimality, and human-like qualities.The passage frames root-cause understanding, rather than wholesale replacement, as central to real-world debugging.Source: A Additional Related Works · p. 15
  • The bug-injection taxonomy retains five code-level ODC categories, excluding Documentation and Function because they cannot be expressed in line-level static examples.The retained categories—Assignment, Checking, Algorithm, Build/Package/Merge, and Timing/Serialization—cover concrete bug patterns used to condition the generator and broaden defect-family coverage.Source: A Additional Related Works · pp. 15–16, Table 4 · p. 16

B Additional Experiments

  • PDB-SINGLE’s model ranking persists on the unfiltered PDB-SINGLE-FULL pool; further experiments ablate code length, defect category, and rewriting, then test transfer to human-curated and contemporaneous-generator benchmarks.Source: B Additional Experiments · p. 17

B.1 Results on the full PDB-SINGLE-FULL pool … D Examples of Debugging Categories

Across expanded, length-scaled, category-specific, rewritten, real-world, and multi-line evaluations, high unit-test success can coexist with weak edit precision. The appendix also formalizes edit matching and presents examples of debugging failure modes.

B.1 Results on the full PDB-SINGLE-FULL pool

  • On PDB-SINGLE-FULL, Claude-Sonnet-4.5 and Gemini-2.5-Pro lead precision at 78.1% and 77.9%, while GPT-5.1-Codex is lowest at 50.3%, preserving filtered-pool ordering.As bug count increases, precision falls while unit-test pass rate decays more slowly; recall stays within a 5% band on BigCodeBench and rises with bug count on LiveCodeBench, with these patterns more pronounced on the full pool.Source: §B.1 · pp. 17–18, Figure 7 · p. 17

B.2 Buggy code length scaling

  • As buggy programs grow longer, precision, recall, and unit-test pass rate all decline together, consistent with longer code inviting more spurious edits as well as complicating localization.The authors argue that the precision drop is not merely a consequence of harder tasks, since recall and unit score decline alongside precision.Source: §B.2 · p. 18, Figure 8 · p. 18

B.3 Defect-category breakdown

  • Eight of nine models show a Build/Package/Merge recall spike, often 10–20 percentage points above other categories; Gemini-2.5-Pro is the exception, with roughly 70% recall across categories.The authors suggest API-misuse and dependency-version faults may be easier because their fixes are overrepresented in pretraining material.Source: §B.3 · pp. 18–19, Figure 9 · p. 18

B.4 Source rewriting and cross-model bug generation

  • Rewriting raises precision by 2.8–3.5% on average, while cross-model bug generation lowers unit-test pass rate by up to 1.4%; the authors conclude contamination is at most a secondary contributor to the precision gap.Strong-baseline-precision models gain least from rewriting, while the weakest-precision models gain less from rewriting than they lose under cross-model generation.Source: §B.4 · p. 19, Figure 10 · p. 19

B.5 Real-world debugging on DEBUGBENCH

  • On 40 one-bug DEBUGBENCH examples [Tian et al., 2024], GPT-5.1-Codex has the highest unit-test pass rate (90.0%) but the lowest precision (61.9%), while Claude-Sonnet-4.5 and Gemini-2.5-Pro approach 80% precision.This recall–precision dissociation also appears on a benchmark not constructed by the authors, suggesting it is not specific to synthetic bug injection.Source: §B.5 · pp. 19–20

B.6 Multi-line bugs and post-release models on PDB-WILD

  • On PDB-WILD, Claude-Opus-4.7 and Gemini-3.1-Pro lead near 70% precision, while GPT-5.5 scores 54.8% precision and GPT-5.1-Codex 30.8% despite a 62.8% unit-test pass rate.The results support robustness of the precision gap to bug-generator choice and seed-code exposure; the authors also note PDB can be regenerated with newer frontier models.Source: §B.6 · p. 20

C Algorithm on Precision and Recall

  • The strict matcher aligns predicted and ground-truth edits through exact line matches, structural containment, then equal-size content matches, forming a partial bijection whose unmatched blocks count against precision or recall.The ϵ-tolerant variant can excuse bounded extra edits only when removing them preserves unit-test behavior.Source: C Algorithm on Precision and Recall · p. 21

D Examples of Debugging Categories

  • The examples distinguish patches that pass tests but have imperfect precision or recall from patches that fail tests despite partially correct edits; caption percentages use 240 manually inspected patches.Source: D Examples of Debugging Categories · p. 24

D.1 Failure modes when unit tests pass

Among unit-test-passing patches, failures split into precision loss despite full recall and recall loss despite passing tests; the dominant precision problem is rewriting correct code, while recall loss includes reference ambiguity and a small injection-coupling artifact.

D.1 Failure modes when unit tests pass

  • 9.8% of recall=1, precision<1 cases add a redundant defensive guard on top of an already-correct fix, reducing precision without affecting tests.The extra check is harmless but makes the patch a strict superset of the ground-truth edit.Source: Figure 11 · p. 24, §D.1 · p. 25
  • 66.8% of recall=1, precision<1 cases involve edits to unrelated, already-correct code alongside the necessary fix.Typical changes include renaming variables, reordering statements, or adjusting unrelated control flow, reflecting regeneration-style behavior rather than local amendment.Source: Figure 12 · p. 25, §D.1 · p. 25
  • From-scratch reimplementations can pass tests yet score near-zero precision because their edits do not align with any ground-truth block.This is the most extreme precision failure and effectively bypasses the debugging task.Source: §D.1 · p. 25
  • Under 2% of inspected patches fix an additional real seed-solution bug, showing that low precision can sometimes reflect useful repairs omitted from ground truth.The authors treat these cases as a soft upper bound on optimizing precision without losing useful repairs.Source: §D.1 · p. 25
  • Structurally mismatched but test-correct fixes dominate recall loss, motivating mapϵ, which allows semantically validated relocations.The relaxed matcher absorbs much of the gap between predicted edits and ground-truth blocks.Source: §D.1 · p. 25
  • 20% of recall-loss cases involve equally targeted alternative fixes that the single-reference metric penalizes despite no over-editing.The authors identify multi-reference annotations as a future direction for addressing this ambiguity.Source: §D.1 · p. 25
  • 1.65% overall prevalence of injection coupling indicates the generator's bug-independence assumption holds for most cases, with the metric accurate on >97.5% of evaluation data.Two locally minimal injected bugs can become one coupled defect, which a single predicted edit resolves while the matcher counts one original bug as missing.Source: §D.1 · p. 26

D.2 Failure modes when unit tests fail · E Prompt templates

Among patches that fail unit tests, the benchmark distinguishes under-repair from regressive repair; the appendix also provides prompt templates for debugging and benchmark construction.

D.2 Failure modes when unit tests fail

  • Together, under-repair and regressive repair show why unit-test pass rates alone omit information captured by precision and recall.The two categories separate patches that miss fixes from patches that resolve all injected bugs but introduce new defects.Source: §D.2 · p. 26
  • 31.4% of failing cases are under-repairs: the model makes only necessary edits, but leaves at least one injected bug unresolved.These failures are recall-bound rather than precision-bound and tend to grow with bug count k.Source: §D.2 · p. 26, Figure 18 · p. 32
  • 39.2% of failing cases are regressive repairs: all injected bugs are fixed, but new defects make the unit tests fail.This is the largest single failure category in this regime and reflects regeneration that breaks code exercised by the test suite.Source: §D.2 · p. 26, Figure 19 · p. 33

E Prompt templates

  • Benchmark-construction templates include prompts for bug injection and solution rewriting.The bug-injection prompt is shown in Figure 20 and the solution-rewriting prompt in Figure 31.Source: Figure 20 · p. 34, Figure 31 · p. 42
  • The prompt appendix includes minimal debugging templates with buggy code, unit tests, execution feedback, or both.It also shows a response format containing buggy_solution and instructs models not to edit or add comments.Source: Figure 21 · p. 35, Figure 22 · p. 35, Figure 23 · p. 36
  • Free-form debugging templates vary whether they include unit tests, execution feedback, or both, and the appendix provides external API templates for minimal and free-form debugging.These are represented by Figures 25–30.Source: Figure 25 · p. 37, Figure 26 · p. 38, Figure 27 · p. 38

F Checklist Information

The checklist identifies risks that PDB and PDB-SINGLE could enable malicious fault insertion or data poisoning, and records licensing for evaluated models and benchmarks alongside the study’s use of LLMs.

Risks of PDB pipeline

  • PDB’s bug-generation pipeline could be repurposed to inject plausible faults at scale, degrade software reliability, burden maintainers, or seed faulty code into shared repositories.Its minimal-surface-change bug injection may also be used to create synthetic buggy-code corpora that silently bias model behavior or contaminate deployment and benchmark datasets.Source: F Checklist Information · p. 43

Risks of PDB-SINGLE

  • PDB-SINGLE could support misuse if used to train models for localized fault insertion in settings where code integrity matters.The benchmark comprises challenging instances derived from LiveCodeBench and BigCodeBench; training models to both repair and introduce faults is a possible extension of its intended role.Source: F Checklist Information · p. 43

Licensing landscape of evaluated models

  • Evaluated models include proprietary API-only services, MIT- and Apache-licensed open weights, and Modified MIT Kimi models with visible-attribution requirements.GPT-5.1-Codex, Claude-Sonnet-4.5, Gemini-2.5-Pro, and Grok-Code-Fast are API-only; DeepSeek-V3.2 variants use MIT, Qwen3-Coder-480B uses Apache 2.0, and Kimi-K2 models use Modified MIT, requiring attribution above 100 million monthly active users or $20 million monthly revenue.Source: F Checklist Information · p. 43

Licensing landscape of datasets

  • BigCodeBench uses Apache License 2.0, while LiveCodeBench uses MIT for its codebase and CC-BY 4.0 for dataset artifacts.Source: F Checklist Information · p. 43

Use of LLM

  • The study uses LLMs to generate and debug buggy programs in experiments and to improve writing fluency and correct grammar.Source: F Checklist Information · p. 43

G Limitation

PDB currently targets Python, evaluates rather than trains models, and its precision and recall metrics may miss some correct, semantically equivalent fixes.

G Limitation

  • PDB’s prompts and data generation target Python, limiting immediate use elsewhere, though adapting it mainly requires new in-context examples and language-specific sub-categories because ODC categories are language-agnostic.The framework itself need not be redesigned to support other languages.Source: G Limitation · p. 44
  • PDB supports benchmark construction and evaluation but has not been used to train or post-train models; future work could test precision-aware objectives in learning pipelines.Suggested approaches include supervised fine-tuning, reinforcement learning, and self-improvement, with the goal of reducing over-editing while preserving functional correctness.Source: G Limitation · p. 44
  • Edit-level precision and bug-level recall may miss correct, semantically equivalent fixes, and reliably evaluating semantic correctness of code edits remains open.The authors suggest flexible evaluation mechanisms such as LLM-as-a-judge to address these edge cases.Source: G Limitation · p. 44

Related papers

Loading 2604.17338v4…