RGFL: Reasoning Guided Fault Localization for Automated Program Repair Using Large Language Models

1 Introduction

• LLM context limits make whole-repository input inefficient, as irrelevant details can distract models from a bug’s root cause.The largest SWE-bench Verified repository is nearly 12M tokens, and the SymPy repository also exceeds current context limits.Source: §1 · pp. 1–2
• Existing LLM localization approaches rely on textual matching or retrieval without explicitly reasoning about why code relates to a bug.The paper’s motivating example contrasts superficially related selections with locating the fault-inducing HTML.write method.Source: §1 · p. 2, Figure 1 · p. 2

1.1 Research Questions

• The study asks whether reasoning improves localization in Agentless, generalizes across datasets and languages, and improves repair success against other agents.The questions cover file- and element-level localization, baseline comparisons, and end-to-end repair.Source: §1.1 · p. 4

1.2 Contributions

• RGFL generates bug-specific explanations for candidate files and elements, then uses them to rank candidates.Its contributions also include embedding-based file ranking for comparison with LLM ranking.Source: §1.2 · p. 4
• RGFL is integrated into Agentless to assess localization’s effect on repair while retaining the existing line-localization and patch-generation steps.The contribution description also reports a systematic ablation and counterfactual study of file- and element-level contributions.Source: §1.2 · p. 4
• The evaluation covers SWE-bench Verified, Lite, and Java, while counterfactual injections of ground-truth information estimate localization-stage repair upper bounds.The authors also introduce a file-, element-, line-, and repair-stage error taxonomy.Source: §1.2 · p. 4

2.1 Automated Program Repair (APR)

• APR pipelines separate fault localization, patch generation, and validation, while end-to-end systems generate patches without explicit localization.Modularity permits targeted improvement of individual pipeline stages.Source: §2.1 · p. 4

2.2 Fault Localization Techniques and the Role of LLMs

• Traditional localization methods use test coverage, lexical or semantic similarity, or learned code representations, but may miss rich bug-report–code relationships.Ambiguous reports can require deeper reasoning than surface-level similarity or rigid structural patterns provide.Source: §2.2 · p. 5

2.3 LLM-based APR Agents

• LLM-based APR includes tool-using end-to-end agents and pipeline systems such as Agentless, which hierarchically localizes files, functions, and edit locations.Pipeline approaches separate localization from repair, enabling targeted stage improvements.Source: §2.3 · p. 5
• Existing RAG-based localization retrieves or scores candidates using similarity, graph proximity, or exploration, but typically lacks isolated per-candidate reasoning.RGFL fills this gap by generating individual candidate rationales and using them as ranking signals within a pipeline compatible with existing repair stages.Source: §2.3 · p. 6

3 Methodology

• The method studies explicit LLM reasoning as a way to improve localization accuracy and downstream repair in modular APR systems.It builds on Agentless, whose modular pipeline allows localization components to be modified independently.Source: §3 · pp. 6–7

3.1 Reasoning-based Localization Strategy

• File-level reasoning is limited to Agentless-retrieved candidates to control the cost of evaluating potentially more than 1,000 files.Agentless combines LLM-selected files with embedding-retrieved files; the method generates explanations for those candidates and ranks them with LLM and embedding approaches.Source: §3.1 · pp. 7–9
• For each selected file, RGFL explains the functionality of its functions, classes, and global variables, then ranks those elements against the bug report.The approach reasons about each candidate in isolation rather than asking the model to select from a large set at once.Source: §3.1 · pp. 7, 9
• RGFL leaves line localization and repair unchanged because isolated-line reasoning lacks context and prompting separately for every line would be costly.Keeping these stages unchanged isolates the effect of the file- and element-level modifications.Source: §3.1 · p. 9
• After localization, the system supplies top-ranked code to the repair model, generates candidate patches, and validates them against the regression suite.The highest-ranked passing patch is selected as the output.Source: §3.1 · pp. 9–10

4.1.1 Dataset.

• Verified contains 500 manually reviewed solvable instances, Lite contains 300 instances, and SWE-bench Java contains 91 Java issue instances.The Java subset includes repository snapshots, bug descriptions, and human-authored patches and tests.Source: §4.1.1 · p. 10

4.1.2 LLM.

• Gemini 2.5 Pro, Claude 4 Sonnet, and o4-mini were evaluated for file reasoning, with the best-performing model selected for later ranking and repair comparisons under cost constraints.The selected model was used for element localization and to guide the repair phase.Source: §4.1.2 · pp. 10–11

4.2 Evaluation metrics

• File localization is assessed with Hit@k, Recall@k, and MRR, element localization with Exact Match, and repair with Resolved Rate.Hit@k checks whether a relevant file appears in the top k; Exact Match requires the predicted element set to match the ground truth exactly.Source: §4.2 · pp. 11–12

4.3 Experimental setup

• RGFL integrates file and element localization into Agentless, then performs line localization and patch generation; its average cost is about $4.4 per sample, 3.7× Agentless.The pipeline uses reasoning-based element localization within the top three files.Source: §4.3 · p. 12

4.4 Research Questions

• The research questions test whether LLM-generated reasoning improves file and element localization and whether it can improve automated program repair.The broader evaluation also considers localization across datasets and languages.Source: §4.4 · p. 12

• File retrieval uses code-based similarity to produce candidates, then re-ranks them using LLM-generated reasoning relevant to the bug report.The approach combines source code and its associated reasoning to identify relevant files.Source: element localization performance in a… · p. 12
• The file re-ranking comparison includes embedding bug reports and generated reasoning for cosine similarity, or asking an LLM to judge reasoning relevance directly.The direct LLM strategy is motivated as a way to address potential weaknesses of embedding-only methods.Source: element localization performance in a… · p. 12
• Element ranking is compared across Agentless-only localization, RGFL file localization with Agentless elements, and RGFL at both stages, with an upper bound estimating gains beyond file identification.Gemini 2.5 Pro is used for this experiment, and the upper bound is based on file-localization results.Source: element localization performance in a… · pp. 12–13

• RGFL is compared with BM25 retrieval and agents including Agentless, OpenHands, AutoCodeRover, and OrcaLoca on SWE-bench Verified, Lite, and Java.The datasets are used to examine generalizability across programming languages.Source: languages, and how does it compare to… · p. 13
• The comparisons separately assess file localization and element localization across datasets and languages, using the baselines’ best-reported LLMs against RGFL with Gemini 2.5 Pro for element localization.The stated scope includes Java as well as Python.Source: languages, and how does it compare to… · p. 13

• The experiments replace file localization alone or both file and element localization in Agentless while keeping line localization and patch generation unchanged.This design isolates the impact of the localization stages on repair outcomes against the original Agentless system.Source: repairs compared to baseline agents? · p. 13
• Unresolved cases are categorized as file-, element-, line-, or repair-level misses, distinguishing localization failures from unsuccessful repair after finding the buggy location.Source: repairs compared to baseline agents? · p. 13
• Counterfactual upper bounds inject ground truth at file, element, or line level to estimate how much of the remaining repair gap is attributable to localization versus patch generation.Source: repairs compared to baseline agents? · p. 13

RQ1.1

• On SWE-bench Verified, RGFL raises Gemini 2.5 Pro’s file Hit@1 from 71.4% to 85% and MRR from 81.8% to 88.8%.Hit@5 converges around 93–94% as Agentless already retrieves the correct file(s) within that range.Source: §5 · p. 13, Table 2 · p. 14
• Embedding reranking yields Hit@1 of 64.4% with Gemini-embedding-001, 60.8% with Voyage 3.5, and 52% with text-embedding-3-small; the paper attributes its weakness to missed fine-grained task signals.The authors adopt LLM-based ranking, which applies deeper contextual understanding to generated reasoning.Source: §5 · p. 14

RQ1.2

• Claude 4 Sonnet and Gemini 2.5 Pro differ by only ≈0.1% on average across file Hit@k and Recall@k, while Gemini’s element reasoning costs approximately $400 versus Claude’s $2,000.The authors select Gemini as the more cost-effective backbone for scaling experiments without sacrificing performance.Source: §5 · pp. 14–15
• On SWE-bench Verified, reasoning at both stages raises element Exact Match to 69%, versus 36% with Agentless file and element localization and 41% with RGFL files alone.The respective file-coverage upper bounds are 70% for Agentless files and 80% for RGFL files.Source: §5 · p. 15, Table 3 · p. 15

RQ2.2

• On SWE-bench Verified, RGFL achieves 69% element Exact Match versus OpenHands’ 64%; on Java it reaches 46% versus 38%, while Lite results are 68% versus OpenHands’ 71%.All approaches use the top-3 files from file localization.Source: §5 · p. 16

5 Experiment Results

• Across SWE-bench Verified, Lite, and Java, the method achieves the best file-level Hit@3, Recall@3, and MRR, though OrcaLoca sometimes leads at Hit@1.It attains the highest element-level Exact Match on Verified and Java and remains competitive on Lite.Source: Table 4 · p. 17, Table 5 · p. 17, §5 · p. 18
• 291/500 (58.2%) SWE-bench Verified instances were resolved with reasoning-based file and element localization, up from 258 (51.6%) for Agentless and 279 (55.8%) with file localization alone.The study reports 258/500 (52%) for Agentless versus 49% for OpenHands when both use Gemini 2.5 Pro.Source: Table 6 · p. 18, §5 · p. 18

5.4 Discussion: Counterfactual ablation

• Among 209 unresolved Verified instances, wrong files occurred in 28 cases (13%), wrong elements in 111 (53%), and wrong lines in 175 (84%); 21 failed despite correct localization.Error categories are not mutually exclusive.Source: §5.4 · p. 19, Figure 5 · p. 20
• Replacing wrong files with ground truth resolved 14 of 28 cases (50%); then correcting elements resolved 29 of 111 (≈26%), while correcting lines resolved 33 of 175 (≈19%).File correction had the largest per-case impact, whereas line errors were most frequent but yielded the smallest recovery when corrected in isolation.Source: §5.4 · p. 19
• In 70 cases with correct files but incorrect elements that remained unresolved after ground-truth element injection, providing only ground-truth files repaired 9 cases.The authors suggest that precise element guidance can sometimes reduce repair power, while correct-file context alone can suffice.Source: §5.4 · p. 19
• A Django example showed that file-level context produced the correct method with its alias parameter, whereas emphasizing ground-truth elements led to a patch missing that parameter.The authors conclude that narrowing context to elements can confuse the model and that the best way to present fine-grained localization remains unclear.Source: §5.4 · p. 19
• The authors identify finer-grained localization and how to present it without overly limiting model exploration as issues needing further research.They describe element- and line-level localization as more challenging than file-level localization.Source: §5.4 · p. 19

6 Threats to Validity

• The experiments cover Python and Java, so results may not fully generalize to ecosystems such as C++, JavaScript/TypeScript, or Rust.The authors recommend additional languages and repositories to assess cross-language applicability.Source: §6 · p. 20
• The later-stage backend was Gemini 2.5 Pro, selected for its experimental accuracy-cost balance, which may not hold for other or future models.The baselines also use the best-reported LLMs at the time, which may not be the current best-performing models.Source: §6 · p. 20
• Patch validation uses only the repositories’ original regression tests and does not add reproduction tests.The authors identify this as standard SWE-bench practice.Source: §6 · p. 20
• Repair performance is reported only on SWE-bench Verified, limiting the scope of conclusions about RGFL’s downstream repair impact.The authors say extending repair evaluation to SWE-bench Lite and SWE-bench Java would strengthen the evidence.Source: §6 · p. 20

7 Conclusion and Future Work

• RGFL’s LLM-based reasoning framework targets improved file- and element-level fault localization in automated program repair.It uses LLMs to interpret bug reports and explain code functionality, then uses those explanations to guide localization.Source: §7 · pp. 20–21
• Across SWE-bench evaluation, reasoning at both localization levels consistently improves results over strong baselines and increases the number of bugs fully resolved.The conclusion reports significant end-to-end repair gains but gives no numerical value in these passages.Source: §7 · p. 21
• The broader idea of LLM-based reasoning may extend to test generation, code summarization, specification mining, code review, and vulnerability detection.The authors connect these tasks to understanding both code and intent.Source: §7 · p. 21