Source-linked AI summary
Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Wang Bill Zhu, Miaosen Chai, Shangshang Wang, Yejia Liu, Song Bian, Honghua Dong, Willie Neiswanger, Robin Jia
TL;DR
Debugging requires localized, minimal edits, but existing evaluations mainly reward final test passing and can therefore miss regeneration-heavy behavior. PDB converts coding datasets into precision-aware debugging benchmarks, and its results show a substantial gap between functional correctness and precise fault localization.
Problem
Existing debugging benchmarks rely on unit-test outcomes and do not measure whether models localize faults or make minimal edits.
Method
PDB synthesizes verified atomic bugs, composes independent multi-bug programs, and evaluates revisions with edit-level precision and bug-level recall.
Results
Frontier models can exceed 76% unit-test pass rates while achieving edit precision of 45% or less, and iterative or agentic strategies do not substantially improve precision or recall.
Takeaways & Limitations
PDB reveals debugging behavior that unit-test-only evaluation misses and supports precision-aware evaluation and post-training objectives.
Abstract
from arXiv · showhide
Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during debugging. To evaluate how far LLMs are from precise debugging, we introduce the Precise Debugging Benchmark (PDB) framework, which automatically converts any coding dataset into a debugging benchmark with precision-aware evaluation. PDB generates buggy programs by synthesizing verified atomic bugs and composing them into multi-bug programs. We define two novel metrics, edit-level precision and bug-level recall, which measures how many necessary edits are made and how many bugs are resolved. We release two evaluation benchmarks: PDB-Single-Hard on single-line bugs, and PDB-Multi on multi-line bugs. Experiments show that frontier models, such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking, achieve unit-test pass rates above 76% but exhibit precision below 45%, even when explicitly instructed to perform minimal debugging. Finally, we show that iterative and agentic debugging strategies do not substantially improve precision or recall, highlighting the need to rethink post-training pipelines for coding models.
1 Introduction
Frontier LLMs often regenerate large portions of buggy programs rather than applying precise, intent-preserving edits. PDB addresses evaluation gaps by measuring targeted repair behavior beyond unit-test correctness.
- Motivation: Frontier LLMs often rewrite large portions or entire programs during debugging, even when the result passes tests.Such regeneration is costly and risky for realistic codebases.
- Problem: Existing benchmarks reward regeneration, hard-coded outputs, and minimal fixes equally because they rely primarily on unit-test outcomes.They also cannot distinguish partial repair from no repair in multi-bug programs.
- Contribution: PDB converts coding datasets into debugging benchmarks by synthesizing verified atomic bugs, composing independent multi-bug programs, and evaluating precision and recall.Its metrics reward targeted fixes and penalize unnecessary modifications.
- Results: GPT-5.1-Codex and DeepSeek-V3.2-Thinking exceed 76% unit-test pass rates but achieve edit precision of 45% or less.Qwen3-Coder-480B provides a contrasting profile with 70% unit-test pass rate and 66% precision.
2 Precise Debugging Setup
The precise debugging setup evaluates revisions at the edit and bug levels rather than only by final program correctness. It formalizes minimal corrections, atomic independent bugs, semantic correctness, and tolerance for limited extra edits.
- Conventional evaluation: Conventional debugging evaluation returns a binary unit-test result for the revised program, without analyzing the proposed edits.A program receives 1 only if it passes all tests in the suite.
- Precise evaluation: The precise setup shifts evaluation from program-level correctness to the specific edits proposed by the model.This enables penalties for unnecessary edits and distinguishes partial repairs from entirely incorrect solutions.
- Bug structure: Minimal corrections are the smallest edit sets that make a buggy program pass its unit tests.The setup also defines atomic bugs through contiguous correction lines and independent bugs through pairwise-compatible corrections.
- Per-bug evaluation: For each bug, PDB maps predicted edits to the corresponding ground-truth edit region and tests a pseudo-revision for semantic correctness.This isolates whether each bug is resolved while retaining the other ground-truth corrections.
- Metrics: Precision averages over predicted edits, whereas recall averages over the k bugs.An epsilon tolerance allows up to |E_i| + epsilon edited lines per bug to discourage regeneration without requiring strict minimality.
- Redundancy control: The essentialU function searches predicted-edit subsets for the minimal edits that preserve semantic correctness and resolve each bug.It removes redundancy from semantically correct revisions that still contain regeneration.
3 Generation and Evaluation Pipeline
PDB generates verified atomic bugs from existing coding tasks, composes them into controlled multi-bug programs, and evaluates debugging systems with both functional and edit-aware metrics.
- Pipeline: PDB generation first synthesizes atomic bugs from existing coding datasets, verifies them with unit tests, and records their edit sets.The evaluation stage prompts systems to revise the buggy programs and measures unit-test accuracy, precision, and recall.
- Atomic bugs: Single-line bug generation samples insertion, deletion, or substitution operations across editable lines and defect categories.Rule-based filtering excludes lines that are unsafe to delete or edit.
- Multi-line bugs: Multi-line generation samples contiguous blocks with selected defect categories and filters out violations of atomicity.Block size B is sampled from [2, B_max].
- Bug composition: Bug composition samples distinct block edits and enforces a stride constraint to encourage independence among composed bugs.The process repeats across target bug counts and records the resulting multi-bug programs.
- Evaluation: Systems are evaluated as either single-pass LLMs or LLM-based agents, optionally using unit tests and error feedback.Reported metrics are micro-averaged across examples after averaging within bug counts.
4 Evaluation Sets
PDB releases evaluation sets spanning single-line, multi-line, and repository-level debugging tasks from three coding benchmarks. The benchmarks support analysis across increasing numbers of injected bugs.
- Source benchmarks: PDB draws tasks from BigCodeBench, LiveCodeBench, and SWE-smith to cover API-heavy, competitive-programming, and repository-level settings.The pipeline uses disjoint or selected repository subsets for broader software-engineering coverage.
- PDB-SINGLE: PDB-SINGLE generates 20 single-line bugs per task, composes up to 100 variants with at most 4 independent bugs, and yields 7,591 initial examples.A stride of 3 lines separates composed blocks.
- PDB-WILD: PDB-WILD uses contiguous multi-line blocks with maximum size 4, up to 3 blocks per program, and repository samples for realistic software engineering.The multi-line benchmark uses a stride of 5 lines.
- Bug-count analysis: As bug counts increase, precision generally correlates negatively with unit-test score, while recall varies by dataset.Figure 4 compares BigCodeBench and LiveCodeBench subsets across bug counts.
5 Experiment Results
PDB evaluation reveals a substantial gap between functional success and precise, edit-aware debugging: models often pass tests while over-editing or regenerating code. Iterative, agentic, prompting, and data-generation analyses show that this gap persists across settings and bug granularities.
- No frontier model exceeds 72% edit-level precision, even when explicitly instructed to perform minimal debugging.
- Claude-Sonnet-4.5 and Gemini-2.5-Pro exceed 75% unit score, 71% precision, and 81% recall, whereas GPT-5.1-Codex and DeepSeek variants use regeneration-heavy strategies with precision at most 48%.
- Increasing injected bugs lowers unit-test scores but produces an inverse trend in edit-level precision because over-editing becomes more likely.
- Iterative and agentic debugging improve unit-test scores and recall, but precision usually remains unchanged or degrades relative to single-shot debugging.
- Agentic systems often treat tests and error messages as coarse success signals that trigger broad rewrites rather than supporting fault localization.
- Freeform prompting substantially reduces edit-level precision and bug-level recall across models, while rewriting references improves precision by 2.8–3.5% on average.
6 Related Works
Existing debugging benchmarks largely equate passing unit tests with success, rewarding wholesale rewrites and minimal fixes alike. PDB instead evaluates the precision and completeness of individual repairs, aligning assessment more closely with real-world debugging.
- Debugging benchmarks span mined bug-fixing commits, multilingual repair tasks, live contests, code editing, automated bug fixing, and broader debugging scenarios.
- Existing benchmarks predominantly rely on unit-test evaluation, which rewards large rewrites and minimal targeted fixes equally.
- PDB introduces edit-level precision and bug-level recall to expose behaviors that unit-test-only evaluation misses.
7 Discussion
The paper argues that frontier models remain far from precise debugging and that progress requires changing both evaluation and post-training objectives. PDB is positioned as both a benchmark and infrastructure for precision-aware training signals.
- Frontier LLMs often pass unit tests while relying on solution regeneration rather than targeted edits.
- PDB-SINGLE and PDB-WILD motivate rethinking evaluation and post-training objectives for debugging.
- PDB’s precision-aware metrics can provide reward signals focused on fault localization and edit minimality.
- The framework’s plug-and-play pipeline enables scalable, controlled construction of debugging data from existing coding datasets.
- 6 Related Works: Related work includes code-generation models, debugging frameworks, debugging benchmarks, and defect taxonomies such as ODC.
B.1 Results on the full PDB-SINGLE-FULL pool
On the full PDB-SINGLE-FULL pool, model rankings and precision deficits remain stable, while increasing bug counts and program length expose systematic over-editing and declining performance across benchmarks.
- Full-pool comparison: All three metrics are 4–8% higher than on PDB-SINGLE, while the model ordering is preserved.Claude-Sonnet-4.5 and Gemini-2.5-Pro retain the highest precision, whereas GPT-5.1-Codex remains weakest on precision despite competitive unit-test performance.
- Full-pool comparison: 50.3% precision remains GPT-5.1-Codex’s score on PDB-SINGLE-FULL despite a competitive unit-test pass rate.The stable ranking across full and filtered pools indicates that the reported precision gap is not an artifact of the difficulty subset.
- Bug-count effects: Each additional bug monotonically lowers precision, while unit-test pass rate declines more slowly and recall remains source-dependent.Recall stays within a 5% band across bug counts on BigCodeBench but increases with bug count on LiveCodeBench.
- Program-length effects: Longer buggy programs simultaneously reduce precision, recall, and unit-test pass rate.The metrics move together as models must localize faults among more candidate lines and edit without disturbing surrounding code.
- Defect categories: Eight of nine models show a pronounced recall spike for Build/Package/Merge defects, often 10–20 percentage points above other categories.Gemini-2.5-Pro is the exception, with recall roughly uniform at ∼70% across categories.
- Rewrite ablation: Rewriting raises precision by 2.8–3.5% on average, while cross-model bug generation lowers unit pass rate by up to 1.4%.The aggregate pattern holds for every model individually, although the magnitude varies.
- Transfer beyond synthetic bugs: The precision deficit persists on DEBUGBENCH, where GPT-5.1-Codex reaches 90.0% unit-test pass rate but 61.9% precision.The same recall–precision dissociation appears beyond synthetically injected bugs, on a benchmark built from real bug-fixing commits.
D.1 Failure modes when unit tests pass
When unit tests pass, models can still lose precision through redundant checks, edits to correct code, or complete rewrites. Some low-precision repairs also fix latent defects, while recall losses can reflect matching ambiguity or dataset-construction artefacts.
- Precision-loss patterns: 9.8% of recall=1, precision<1 cases add redundant guard checks after an already-correct fix.The extra defensive check is irrelevant to unit tests but counts as a spurious edit.
- Precision-loss patterns: 66.8% of recall=1, precision<1 cases involve additional modifications to already-correct code.These edits include renaming variables, reordering statements, or changing unrelated control flow.
- Precision-loss patterns: Under 2% of inspected patches fix an additional latent defect in the seed solution, making low precision informative when ground truth is incomplete.Such repairs suggest perfect precision is not always desirable when useful repairs extend beyond injected bugs.
- Recall-loss patterns: 20% of recall<1 cases involve multiple equally minimal fixes, while bug-composition artefacts account for 1.65% overall.The former motivates multi-reference annotations; the latter reflects occasional coupling between composed injections.
D.2 Failure modes when unit tests fail
When unit tests fail, models either leave some bugs unrepaired or introduce regressions despite resolving the injected bugs. These cases separate recall-bound targeted repair from regeneration-induced failures.
- Under-repair: Under-repair failures are recall-bound and tend to increase with the bug count k.The model applies precise edits but leaves at least one ground-truth bug untouched.
- Regressive repair: Regressive repair is the largest single failure category in this regime.It shares the regeneration mechanism associated with precision losses from rewriting correct code.
- Under-repair: 31.4% of failing cases are under-repairs that fix only a subset of bugs without extraneous edits.They have precision = 1 but recall < 1, so the unit tests fail.
- Regressive repair: 39.2% of failing cases are regressive repairs that resolve all injected bugs but introduce new defects elsewhere.These patches have recall = 1, yet their new defects cause unit-test failure.
F Checklist Information
The checklist identifies risks associated with generating realistic buggy code and with the governance of the evaluated models and datasets.
- Risks: PDB could be repurposed to automate malicious bug injection at scale.The pipeline can generate plausible faulty code with minimal surface changes, potentially degrading software reliability.
- Risks: Structured buggy-program and solution pairs could lower the cost of creating synthetic corpora for data poisoning or capability shaping.Outside the intended research context, such data could bias training or contaminate downstream datasets.
- Risks: PDB-SINGLE’s use for training debugging or code-editing models creates risks depending on how the capability is used and framed.Its intended role includes training models that can repair and introduce faults under different objectives.
- Licensing: The evaluated proprietary models are accessible exclusively through commercial APIs and governed by restrictive terms of service.The passage contrasts this proprietary tier with open-weight ecosystems.
- Licensing: BigCodeBench uses Apache License 2.0, while LiveCodeBench uses MIT licensing for its codebase and CC-BY 4.0 for dataset artifacts.The licensing arrangements differ across the evaluated datasets.
G Limitation
The paper’s scope is limited by its Python-focused data generation, lack of model training with PDB signals, and imperfect semantic evaluation of equivalent fixes.
- Scope: PDB’s current prompts and data-generation procedures target Python programs, limiting immediate applicability to other languages.The authors state that adapting the framework mainly requires language-specific examples and subcategories.
- Training scope: The work constructs and evaluates the benchmark but does not train or posttrain models using PDB signals.Testing precision-aware objectives in supervised fine-tuning, reinforcement learning, or self-improvement is left for future work.
- Evaluation: Edit-level precision and bug-level recall may miss some correct semantically equivalent fixes.The authors identify reliable semantic evaluation of code edits as an open problem.