Source-linked AI summary
A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization
Sungmin Kang, Gabin An, Shin Yoo
TL;DR
Fault localization remains a substantial debugging task, and existing techniques often omit rationales for their suggested locations. AutoFL uses an LLM with repository-navigation function calls to generate both fault locations and bug explanations. Across Java and Python benchmarks, it improved method-level acc@1 by up to 233.3% over baselines, while developers generally liked its natural-language explanations and preferred a few high-quality explanations.
Problem
Fault localization consumes substantial debugging effort, while existing techniques often fail to provide rationales that developers consider important for adoption.
Method
AutoFL prompts an LLM to navigate large repositories through function calls, generate an explanation of the bug, and suggest a fault location.
Results
AutoFL improved method-level acc@1 by up to 233.3% over baselines and produced accurate explanations for 56.7% of bugs, while developers generally supported its explanations.
Takeaways & Limitations
Developers valued natural-language explanations but preferred receiving a few explanations rather than many, indicating that explanation quality and selection matter for use.
Takeaways & Limitations
Developer feedback identified confusion and wasted time when confident explanations contained inaccurate fix suggestions, patches, or fault locations.
Abstract
from arXiv · showhide
Fault Localization (FL), in which a developer seeks to identify which part of the code is malfunctioning and needs to be fixed, is a recurring challenge in debugging. To reduce developer burden, many automated FL techniques have been proposed. However, prior work has noted that existing techniques fail to provide rationales for the suggested locations, hindering developer adoption of these techniques. With this in mind, we propose AutoFL, a Large Language Model (LLM)-based FL technique that generates an explanation of the bug along with a suggested fault location. AutoFL prompts an LLM to use function calls to navigate a repository, so that it can effectively localize faults over a large software repository and overcome the limit of the LLM context length. Extensive experiments on 798 real-world bugs in Java and Python reveal AutoFL improves method-level acc@1 by up to 233.3% over baselines. Furthermore, developers were interviewed on their impression of AutoFL-generated explanations, showing that developers generally liked the natural language explanations of AutoFL, and that they preferred reading a few, high-quality explanations instead of many.
1 INTRODUCTION
Fault localization consumes substantial debugging effort, yet existing techniques often lack accessible rationales that practitioners consider important. AutoFL addresses this gap by using LLMs to explore repositories, explain bug causes, and suggest fault locations, with evaluations covering localization performance and explanation quality.
- 66% of debugging time was reported as being spent on bug diagnosis, which includes fault localization.
- Almost 90% of developers considered rationale generation important for fault-localization techniques, while few techniques explain why a location is implicated.
- AutoFL uses LLMs to navigate large repositories through function calls, then explains how a bug occurred and why the suggested location is suspicious.
- 19.7% and 166.7% improvements in method-level acc@1 over Ochiai SBFL were reported on Defects4J and BugsInPy with GPT-3.5, while GPT-4 reached 233.3% on BugsInPy.
- Manual evaluation found accurate explanations for 56.7% of bugs, while developers generally valued natural-language explanations but preferred a few explanations over many.
- Preliminary experiments found a positive correlation between explanation correctness and the correctness of tests and patches produced with those explanations.
2 BACKGROUND
The background situates AutoFL within LLM tool use and automated fault localization. Existing FL families have practical artifact requirements and commonly lack rationales, motivating repository exploration through specialized functions.
- LLM tool use lets models request functions that return information needed to complete a task, with function calling producing structured requests and arguments.
- AutoFL’s diagram depicts ordered component interactions and limits function interactions to a predetermined N.
- Spectrum-based, information-retrieval-based, and mutation-based fault localization are common technique families.
- Spectrum-based techniques require coverage from passing and failing tests, which can be computationally costly for large enterprise software.
- Most fault-localization techniques lack rationales, limiting their reliability and practicality during debugging.
3 APPROACH
AutoFL uses LLMs to localize bugs at method level while generating explanations of how failures occurred. It enables repository exploration through custom functions and separates root-cause analysis from fault-location prediction.
- Approach: AutoFL localizes bugs and generates explanations linking the root cause to the observed failure.The method is designed to help developers understand both the bug and the reasoning behind a suggested location.
- Approach: AutoFL targets method-level fault localization, a granularity favored by developers and commonly required by LLM-based debugging techniques.The approach addresses repository-scale code exploration with custom-designed functions.
- Approach: The approach uses a two-stage prompting process: first infer the failure’s root cause, then predict the likely culprit methods.The supplied system prompt frames the LLM as a debugging assistant that explains how the bug occurred.
<HANDCRAFTED ROOT CAUSE ANALYSIS EXAMPLE>
AutoFL guides an LLM through repository exploration before requesting culprit methods, producing explanations and ranked fault locations. Its repeated predictions are aggregated into method scores, while confidence and inspected-method frequency further support ranking and use.
- LLM-guided fault localization: AutoFL prompts an LLM to explain the logical flow from fault to failure before predicting culprit methods.The system message asks for a step-by-step explanation, and the workflow separates exploratory debugging from final fault-location prediction.
- Initial debugging context: The initial prompt combines the failing test, a minimized test snippet, an error message, and a repository-focused stack trace.AutoFL also encourages the LLM to call get_covered_classes so it can inspect relevant repository information.
- Two-stage workflow: During Stage 1, the LLM can request up to N function interactions or generate a user-facing bug explanation before fault-location prediction.AutoFL executes requested functions, appends their results to the message history, and then requires an immediate method-signature response in the next stage.
- Worked example: For Lang-48, four function calls led AutoFL to inspect covered methods and identify erroneous equals usage for BigDecimal comparison.The example connects repository navigation with a root-cause explanation involving reference-based rather than value-based comparison.
- Prediction aggregation: Method scores average 1/n contributions across R runs, then rank methods in descending score order.For method B, five runs produce (0.5 + 1.0 + 1.0 + 0.5 + 0.0)/5 = 0.6.
- Ranking completion: Covered methods with score 0 are appended to the ranking, with ties resolved by failing-test coverage and function-interaction frequency.This keeps methods relevant to the failure in the output even when they were absent from final predictions.
4 EXPERIMENTAL SETTINGS
The evaluation uses 798 bugs across Java and Python benchmarks and examines localization accuracy, confidence alignment, explanation quality, and developer experience. It combines benchmark comparisons, manual explanation assessment, and a study with 16 professional developers.
- Research questions: RQ1 measures fault-localization accuracy with method-level acc@k and compares AutoFL with existing techniques, SmartFL, and Test-LLM.The SmartFL comparison covers 222 Defects4J bugs because Closure is excluded.
- Research questions: RQ2 tests whether AutoFL confidence aligns with FL performance using Spearman correlations against P@1, RR, and AP.These metrics respectively capture top-rank correctness, reciprocal rank, and average precision across faulty-method ranks.
- Research questions: RQ3 manually evaluates 300 explanations from 60 randomly selected Defects4J bugs for accuracy, imprecision, concision, and usefulness.Each explanation is rated as a whole rather than in fragments.
- Research questions: RQ4 studies practical reactions by asking 16 professional developers to debug two moderately difficult BugsInPy bugs for about one hour.Participants receive failing-test information, AutoFL results, and 10 explanations from GPT-3.5 and GPT-4.
- Datasets: The study evaluates AutoFL on 798 bugs from 21 open-source projects using Defects4J v1.0 and BugsInPy.Defects4J contributes 353 active Java bugs, while BugsInPy demonstrates adaptation to Python.
- AutoFL configuration: Each bug is run through AutoFL five times with gpt-3.5-turbo-0613 and at most 10 function interactions.The run count reflects diminishing returns beyond five, while the interaction limit avoids context-length problems observed with N=20.
5 RESULTS
AutoFL generally matches or exceeds standalone fault-localization techniques across Java and Python benchmarks, while confidence values correlate with localization quality. Explanation quality is mixed: accurate explanations can be obtained, but failures arise from repository complexity and developers may be confused by incorrect recommendations.
- RQ1: FL Efficacy: AutoFL-GPT3.5 outperforms SBFL and MBFL on Defects4J acc@1 and surpasses SmartFL on the evaluated subset.AutoFL also outperforms SmartFL on the same dataset in the reported comparison.
- RQ1: FL Efficacy: 166.7% and 233.3% method-level acc@1 improvements over Ochiai are reported for AutoFL-GPT3.5 and AutoFL-GPT4 on BugsInPy, respectively.AutoFL-GPT4 consistently performs better than GPT-3.5 relative to baselines up to acc@5.
- RQ1: FL Efficacy: Aggregating multiple AutoFL runs substantially improves performance, while one GPT-4 run can outperform five aggregated GPT-3.5 runs.The comparison is reported for both Defects4J and BugsInPy.
- RQ1: FL Efficacy: AutoFL averages 16.4 seconds per run and 87.24 seconds for five runs, compared with 112 seconds reported for SBFL.The system averages 5.37 function calls per run, and call-chain lengths vary.
- RQ2: Confidence: AutoFL confidence values have statistically significant positive correlations with Precision@1, Reciprocal Rank, and Average Precision on both benchmarks.The authors conclude that confidence can help filter potentially inaccurate results.
- RQ3: Explanation Quality: Custom test helpers caused 53.8% of explanation failures, while other failures involved the function-call budget or excessively long buggy methods.The authors suggest self-contained tests and further project-specific information could help address these limitations.
- RQ3: Explanation Quality: About 20% of individual explanations accurately describe bug causes, while at least one accurate explanation is generated for 56.7% of bugs.Filtering by confidence modestly improves helpful explanation qualities with negligible increase in imprecision.
- RQ4: Developer Feedback: Developers valued explanations and suggested fixes, but inaccurate recommendations sometimes confused them or wasted their time.Twelve of sixteen developers wanted explanations, while two reported confusion from insufficient patches or fault locations.
6 FUTURE DIRECTIONS: BETTER QUALITY ESTIMATION FOR AUTOFL
The paper investigates whether explanation-quality predictors can identify helpful AutoFL explanations and improve fault-localization confidence. Dynamic features derived from downstream tasks show promise, whereas purely LLM-based evaluation remains unreliable.
- Predictor Design: The evaluation defines Test Score through generated failing and passing tests, while APR Score uses an explanation and buggy code to assess patch generation.Test Score measures the ratio of generated tests behaving as expected; the supplied passage truncates the APR Score description.
- Predicting Explanation Quality: Dynamic features such as Test Score and APR score significantly correlate with positive explanation qualities, especially Accurate rather than Wrong.The analysis uses Spearman correlations between predictors and manually evaluated explanation qualities.
- Predicting Explanation Quality: GPTuseful correlates significantly with every explanation quality, including Wrong, suggesting GPT-3.5 may rate detailed explanations as useful.Length shows a similar pattern and correlates with GPTuseful at 0.49.
- Using Dynamic Scores: The score-boosting method incorporates Test Score or APR score into method scores to refine AutoFL’s confidence estimates.The boost is applied using the corresponding dynamic score for each explanation.
- Using Dynamic Scores: AUC increased by up to 5.4% using APR score when predicting Precision@1 on Defects4J.Confidence values incorporating dynamic scores were more accurate estimators than the original confidence score.
- Future Directions: Future work should further investigate dynamic evaluation because it may identify both helpful explanations and accurate fault localization.The paper characterizes these findings as preliminary.
7 THREATS TO VALIDITY
The validity discussion identifies variability from external LLM execution conditions and random generation, along with limitations in developer-feedback realism.
- Construct Validity: Developer feedback came from testing AutoFL on pandas rather than participants’ work projects, so impressions may not fully reflect real-world debugging.The paper attributes this choice to security reasons.
- Internal Validity: AutoFL’s computation time varies with OpenAI server conditions and the random nature of LLMs.The authors averaged time cost over multiple runs to mitigate this variability.
- Internal Validity: Human error was possible when validating explanations, so two authors independently assessed them and resolved disagreements.This procedure was used to address validation inconsistency.
- Internal Validity: Potential training-data leakage from bug-fixing commits remains a concern, although comparison with a same-model non-function-calling baseline suggests memorization was not the sole explanation.The comparison involved the Test-GPT3.5 baseline in RQ1.
8 CONCLUSION
The conclusion presents AutoFL as an explainable fault-localization technique requiring only a single failing test and generating explanations alongside localization. Its results show strong standalone fault-localization performance, mixed explanation quality, developer demand for structured concise explanations, and promise for dynamic quality estimation.
- Conclusion: AutoFL requires only a single failing test, has reasonable runtime, and generates explanations intended to ease practitioner adoption.The conclusion identifies explanation generation as a critical strength.
- Conclusion: AutoFL outperformed the compared baselines as a standalone fault-localization tool.The conclusion characterizes its fault-localization performance as strong.
- Conclusion: 56.7% of bugs received an accurate AutoFL explanation under manual evaluation.The conclusion presents this as evidence that explanation quality requires nuanced evaluation.
- Conclusion: Developers requested structured explanations covering test intention, bug manifestation, and bug fixes, and were willing to read only a few explanations.This preference motivates selecting high-quality explanations.
- Conclusion: Dynamic features show promise for automatically identifying high-quality explanations, but further research is necessary.The conclusion frames these findings as preliminary.