Source-linked AI summary
Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiency
Junchi Liu, Ali Bigdeli, Roya Daneshi, Atu Ambala, Sudipto Ghosh, Fabio Santos
TL;DR
LLM-based APR has been studied mainly through overall repair effectiveness, leaving the roles of bug complexity, fault localization, and cost-efficiency insufficiently explored. This study evaluates two APR techniques across three LLM series and finds that complex bugs and imprecise localization reduce effectiveness, while advanced models and reasoning modes create repair-effectiveness and cost-efficiency trade-offs.
Problem
Prior LLM-based APR studies provide insufficient evidence about how bug complexity, fault localization, reasoning settings, and computational cost shape repair performance.
Method
The study evaluates two LLM-based APR techniques across three LLM series using multidimensional effectiveness and cost-efficiency metrics with statistical tests.
Results
Structurally complex bugs and imprecise fault localization generally reduce repair effectiveness, while LLMs exhibit clear trade-offs between repair effectiveness and cost-efficiency.
Takeaways & Limitations
Practical APR evaluation should balance repair capability, robustness to localization and complexity, and computational cost.
Takeaways & Limitations
Because the study uses only AtCoder C++ programs and evaluates two APR techniques with three LLM series, its findings may not generalize fully to industrial software or definitive state-of-the-art comparisons.
Abstract
from arXiv · showhide
Background: Software bugs remain a critical challenge in development, necessitating effective Automated Program Repair (APR) techniques. While Large Language Model (LLM)-based APR systems have shown promise, prior studies primarily focus on overall repair effectiveness. The effects of bug complexity, fault localization, reasoning settings, and repair cost-effectiveness remain insufficiently explored. Aims: This study presents a comprehensive empirical analysis of LLM-based APR, focusing on how repair performance is shaped by bug complexity, fault localization, reasoning settings, and costs. Method: We evaluate two APR techniques (ChatRepair and CodeCorrector) using three LLMs (DeepSeek, GPT, and Llama), and examine their performance across diverse levels of bug complexity and localization strategies through a multi-dimensional empirical framework and statistical analysis. Results: Although structurally complex bugs and imprecise fault localization make repair more challenging, LLM-based APR techniques still achieve competitive repair effectiveness. Imprecise fault localization can substantially enlarge the performance gap between APR techniques. Furthermore, higher-cost LLMs and stronger reasoning settings do not consistently yield better cost-efficiency, revealing a nontrivial trade-off between repair effectiveness and computational cost. Conclusions: Over 50% of moderately complex bugs can be repaired by low-cost LLM-based APR techniques. The repair effectiveness gap between APR techniques becomes larger as fault localization becomes less precise. GPT-5 repairs 7 and 39 more complex bugs than DeepSeek-V4-pro and DeepSeek-V3.2, respectively; whereas the total repair cost of DeepSeek-V3.2 shows the best cost-efficiency performance.
1 Introduction
This introduction motivates a multidimensional evaluation of LLM-based automated program repair that considers bug complexity, fault localization, repair effectiveness, consistency, and computational cost. It argues that practical APR assessment must examine both repair capability and the cost implications of LLM choices and reasoning settings.
- Motivation: LLM-based APR has shown strong benchmark performance, but simple datasets such as QuixBugs and IntroClass may be insufficiently challenging.These datasets are primarily composed of simple, single-hunk bugs, and many APR techniques achieve near-perfect performance on them.
- Motivation: Commercial LLM APIs and repeated candidate-patch generation create substantial computational costs, especially under reasoning-intensive settings.Different LLM choices and reasoning settings can therefore produce significant cost discrepancies, which prior APR evaluations often overlook.
- Study design: The study evaluates two LLM-based APR techniques across three LLMs and multiple reasoning settings using repair, consistency, Top −k, and cost-efficiency metrics with statistical tests.The evaluation counts compilable patches, plausible patches, and repaired bugs among its multidimensional measures.
- Findings: Repair performance varies with bug complexity, yet remains relatively considerable for structurally complex bugs, while fault localization improves repair effectiveness and patch consistency.The simplest bugs are consistently easiest to fix, but performance remains comparatively stable as complexity increases and more precise localization generally helps.
- Contributions: The study systematically compares accurate, vague, and absent fault localization while quantifying trade-offs between repair effectiveness and computational cost.Its contributions include standardized cost-efficiency metrics, extensible experimentation infrastructure, and practical guidance for selecting LLMs and reasoning strategies.
2 Background and Related Work
Prior APR research relies heavily on Defects4J and effectiveness-oriented metrics, limiting evaluation diversity, generalizability, statistical reliability, and cost-efficiency assessment. This study addresses these gaps by analyzing LLM-based APR across real-world algorithmic bugs, complexity and fault-localization settings, and cost-efficiency dimensions.
- Benchmark limitations: Defects4J dominates APR evaluation, but reliance on one benchmark can reduce bug-type diversity and weaken assessment of generalizability.DEFECTS4J-TRANS and MuBench extend the benchmark landscape, yet still primarily contain software-engineering–oriented bugs.
- Evaluation limitations: The lack of statistical testing in most benchmarks reduces the reliability of empirical APR findings.
- Study scope: The study collects real-world AtCoder bugs, categorizes them by complexity using Git, srcML, and GumTreeDiff, and evaluates LLM-based APR across bug-complexity and fault-localization settings.The evaluation also includes cost-efficiency and statistical analysis.
- LLM-based APR: LLM-based APR has expanded with advances in code reasoning and generation, with ChatRepair serving as a representative iterative repair baseline.ChatRepair generates and refines patches using feedback from failed and successful repair attempts; the study also selects CodeCorrector as a recent strong-performing method.
- Evaluation limitations: Existing APR metrics emphasize repair effectiveness through repaired-bug counts, Top −k, compilable patches, plausible patches, and consistency, but omit cost-efficiency.The consistency metric measures the extent of changes between patches and buggy programs.
3 Research Design
The study uses an isolated, multi-component infrastructure and a controlled AtCoder-based C++ benchmark for LLM-based APR. Its design varies bug complexity, fault-localization strategies, LLM reasoning settings, and computational cost through a three-stage experimental framework.
- Experimental infrastructure: The infrastructure combines an isolated execution environment, distributed computing resources, a dataset, APR techniques, LLMs, tools, and Python scripts.Git, srcML, GumTreeDiff, and the AtCoder Library support differencing, structural analysis, compilation, and data processing.
- Dataset construction: The benchmark contains 1,914 AtCoder C++ submissions paired into incorrect programs and correct patches, with 640 small-scope bugs retained after applying a five-fixed-line threshold.The threshold was selected because larger fixes made APR considerably more difficult and computationally expensive.
- Bug complexity: Bug complexity is measured through four hunk-line categories—SHSL, SHML, MHSL, and MHML—and AST edit-operation diversity based on insert, delete, relabel, and move operations.Git provides hunk and fixed-line information, while srcML and GumTreeDiff provide AST-level structural differences.
- Experimental factors: The experiments compare buggy line-level fault localization, buggy hunk-level fault localization, and no fault localization across multiple LLM reasoning configurations.DeepSeek and GPT expose reasoning controls, whereas Llama has no explicit mechanism for directly controlling reasoning settings.
- Experimental procedure: The study adopts a three-stage experimental design to balance experimental comprehensiveness and computational cost.The framework deploys APR systems with multiple LLM APIs and reasoning settings, then evaluates complexity, fault localization, and cost-efficiency effects.
4 Results and Empirical Analysis
LLM-based APR effectiveness depends on bug complexity, fault-localization precision, and LLM choice and reasoning settings, with cost-efficiency varying substantially across models. Simpler bugs and precise localization generally improve repair outcomes, while advanced models and reasoning incur considerable costs without uniformly proportional gains.
- LLM Effectiveness and Cost: DeepSeek-V3.2-chat repairs more than twice as many bugs as GPT-4o-mini in both APR frameworks, while Llama-3.1-8B shows limited repair effectiveness.CodeCorrector repairs 295 versus 126 bugs, and ChatRepair repairs 292 versus 140 bugs; Llama-3.1-8B repairs 11 and 50 bugs, respectively.
- LLM Effectiveness and Cost: 47.62 bugs per dollar makes DeepSeek-V3.2-chat more cost-efficient than GPT-4o-mini’s 17.86 under CodeCorrector.Under ChatRepair, DeepSeek-V3.2-chat achieves 71.43 repaired bugs per dollar versus GPT-4o-mini’s 20.41.
- Bug Complexity: 78.5% and 75.2% are the highest repair rates for SHSL and single-operation-type bugs, whereas four-operation-type bugs are the most challenging.Moderate-complexity categories exceed 50% repair rates, and repair effectiveness generally decreases as bug complexity increases.
- Fault Localization: 58 more bugs under hunk-level fault localization and 93 more bugs without fault localization show ChatRepair’s growing advantage over CodeCorrector as localization becomes less precise.The techniques achieve nearly identical repair effectiveness under line-level localization, with CodeCorrector repairing three more bugs.
- Fault Localization: 0.85 versus 0.54 shows ChatRepair’s higher structural consistency than CodeCorrector without fault localization, while line-level localization achieves the highest consistency overall.More precise fault localization improves both repair effectiveness and patch consistency.
- LLM Selection and Reasoning: 1,565.56 repair attempts per dollar makes DeepSeek-V3.2-chat far cheaper than DeepSeek-V4-pro’s 68.29 and GPT-5’s 11.19 in non-reasoning mode.GPT-5 repairs 175 bugs without reasoning and only 3 additional bugs with high reasoning, whereas DeepSeek-V3.2 and DeepSeek-V4-pro repair 64 and 81 additional bugs, respectively.
5 Discussion
Bug complexity and fault-localization precision materially shape LLM-based APR effectiveness, while advanced models and reasoning modes create a trade-off between repair gains and computational cost. Practical APR therefore requires robustness to imprecise localization and careful control of repair rounds.
- Impacts of bug complexity: Structurally complex bugs with multiple buggy lines or diverse AST edit operations are more challenging to repair than simple bugs.The discussion recommends emphasizing complex repair scenarios in future APR evaluations because they better reflect practical challenges.
- Impacts of fault localization: Precise fault localization improves repair effectiveness and produces patches more consistent with the original program logic.Such consistency can facilitate understanding, validation, and maintenance of repaired programs.
- Impacts of fault localization: Existing LLM-based APR methods are affected differently as fault-localization information becomes less precise, making robustness to imprecise localization important for practical deployment.Because real-world fault localization is often noisy and incomplete, stable performance under reduced precision is a practical requirement.
- The Trade-off Between Repair Effectiveness and Cost-efficiency: More advanced LLMs and reasoning modes increase computational cost while often improving repair effectiveness, creating a trade-off between the two objectives.The discussion identifies GPT-5 as achieving the highest repair effectiveness and DeepSeek-V3.2-chat as providing the most cost-efficient repair performance.
- The Trade-off Between Repair Effectiveness and Cost-efficiency: Marginal repair efficiency decreases as repair rounds increase, so selecting an appropriate maximum number of rounds is important for balancing repair effectiveness and cost-efficiency.The cost impact is especially relevant for advanced LLMs such as GPT5 and DeepSeek-V4-pro in reasoning mode.
6 Threats to Validity
The study’s validity is constrained by limited dataset and model coverage, possible pretraining exposure, parameter and API nondeterminism, and reliance on test-case passing as a proxy for semantic correctness.
- External Validity: The AtCoder-only, C++-only dataset may not generalize to industrial software with larger codebases, complex dependencies, and practical engineering constraints.The evaluation also covers only two APR techniques and three LLM series.
- External Validity: Different LLM knowledge cutoffs and possible DeepSeek pretraining exposure to benchmark solutions or related code patterns may affect validity.Benchmark data were collected after 2025-10-01, after the GPT and Llama cutoff dates, while DeepSeek’s cutoff is undisclosed.
- Internal Validity: Fixed temperature, nonmaximum reasoning settings, and nondeterministic API outputs may cause repair-performance variation across configurations and runs.High reasoning-effort settings were used instead of maximum settings for GPT-5, DeepSeek-V3.2-reasoner, and DeepSeek-V4-pro because of substantial time costs.
- Construct Validity: Test-case passing may not establish semantic correctness because incomplete tests can allow hidden defects or overfitting in plausible patches.Repaired bugs are determined mainly by whether generated patches pass all collected test cases.
7 Conclusion and Future Work
The study finds that bug complexity and imprecise fault localization generally reduce LLM-based APR effectiveness, although well-designed techniques remain effective under challenging conditions. It also highlights trade-offs between repair capability and cost-efficiency and identifies directions for improving benchmarks, efficiency evaluation, model coverage, and agent-based APR.
- Conclusion: Structurally complex bugs and imprecise fault localization generally reduce repair effectiveness, but well-designed LLM-based APR techniques remain considerably effective.The study evaluates complexity using hunk-line-level and AST-operation-level perspectives across accurate, vague, and absent fault localization.
- Conclusion: Different LLMs exhibit trade-offs between repair effectiveness and cost-efficiency, making balanced capability, robustness, and cost important for practical APR.The conclusion frames these dimensions as jointly relevant rather than treating repair effectiveness alone as sufficient.
- Future Work: Future work will develop benchmarks that automatically collect, filter, and categorize bug complexity.This aims to make complexity assessment more systematic for APR evaluation.
- Future Work: Future evaluations will incorporate API response latency and overall inference time when measuring LLM-based APR efficiency.These practical efficiency factors are identified as additions to current evaluation criteria.
- Future Work: The study will examine additional LLM families, including Gemini and Claude, and further investigate agent-based APR frameworks.These directions extend model coverage and explore agent-based repair techniques.
40:18 REFERENCES · 40:20 REFERENCES
The references section cites recent work on generative artificial intelligence and code-similarity evaluation using abstract syntax tree edit distance.
- 40:20 REFERENCES: Reference 42 presents a 2025 systematic review of generative artificial intelligence and its applications.The work appears in Multimedia Tools and Applications, volume 84, issue 21, pages 23661–23700.
- 40:20 REFERENCES: Reference 43 revisits code-similarity evaluation with abstract syntax tree edit distance.It is listed in the Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics.
- 40:20 REFERENCES: The listed references span artificial intelligence applications and program-analysis-oriented code-similarity evaluation.Both works are included among the paper’s cited literature.