Source-linked AI summary
LocAgent: Graph-Guided LLM Agents for Code Localization
Zhaoling Chen, Xiangru Tang, Gangda Deng, Fang Wu, Jialong Wu, Zhiwei Jiang, Viktor Prasanna, Arman Cohan, Xingyao Wang
TL;DR
Code localization must connect natural-language issue descriptions to relevant code in large repositories despite implicit dependencies and limited model context. LocAgent represents codebases as lightweight directed heterogeneous graphs and equips LLM agents for multi-hop exploration. Fine-tuned Qwen models achieve competitive localization performance while reducing API costs, though evaluation remains mainly limited to Python and current training data.
Problem
Code localization remains difficult because existing methods struggle to navigate large, evolving codebases and trace dependencies from issue symptoms to affected components.
Method
LocAgent builds directed heterogeneous graph indexes that unify code structures, dependencies, and contents, then supports LLM-agent exploration through multi-hop navigation tools.
Results
Fine-tuned Qwen models achieve performance comparable to Claude-3.5 while reducing API cost by over 80%, from $0.66 to $0.09 per example.
Takeaways & Limitations
The framework enables efficient repository-level exploration and high-accuracy code localization with substantially lower costs than larger proprietary models.
Takeaways & Limitations
The evaluation focuses primarily on Python codebases, and the fine-tuning data relies heavily on trajectories generated by Claude-3.5 and Qwen2.5-32B.
Abstract
from arXiv · showhide
Code localization--identifying precisely where in a codebase changes need to be made--is a fundamental yet challenging task in software maintenance. Existing approaches struggle to efficiently navigate complex codebases when identifying relevant code sections. The challenge lies in bridging natural language problem descriptions with the appropriate code elements, often requiring reasoning across hierarchical structures and multiple dependencies. We introduce LocAgent, a framework that addresses code localization through graph-based representation. By parsing codebases into directed heterogeneous graphs, LocAgent creates a lightweight representation that captures code structures (files, classes, functions) and their dependencies (imports, invocations, inheritance), enabling LLM agents to effectively search and locate relevant entities through powerful multi-hop reasoning. Experimental results on real-world benchmarks demonstrate that our approach significantly enhances accuracy in code localization. Notably, our method with the fine-tuned Qwen-2.5-Coder-Instruct-32B model achieves comparable results to SOTA proprietary models at greatly reduced cost (approximately 86% reduction), reaching up to 92.7% accuracy on file-level localization while improving downstream GitHub issue resolution success rates by 12% for multiple attempts (Pass@10). Our code is available at https://github.com/gersteinlab/LocAgent.
1 Introduction
Code localization requires connecting natural-language issues to code components across structural and dependency relationships, while existing retrieval and agent approaches struggle with evolving, large repositories and implicit dependencies. LocAgent addresses this with graph-based indexing, agent tools for multi-hop exploration, and LOC-BENCH for broader evaluation.
- Motivation: Code localization links natural-language issue descriptions to code snippets requiring modification, demanding reasoning over both programming-language and codebase structure.Poor localization can produce incomplete fixes, new bugs, and longer development cycles.
- Challenges: Existing methods face repository-scale navigation and dependency challenges because vector indexes require continual updates, large-context models cannot process whole codebases, and issues may describe symptoms rather than causes.Traditional retrieval struggles when affected components are implicit, such as shared utilities omitted from an issue description.
- LocAgent: LocAgent builds directed heterogeneous graph indexes that unify code structures, dependencies, and contents, enabling agents to traverse multi-hop relationships during localization.The representation is lightweight and supports efficient entity search while preserving structural information.
- Evaluation: LOC-BENCH broadens evaluation beyond bug fixing by covering feature requests, security patches, and performance optimizations while using newer repository examples to mitigate contamination concerns.The benchmark also includes tooling for continuous updates as models and training cutoffs evolve.
- LocAgent: Unified exploration tools let LLM agents navigate code dependencies systematically, including cases where the target code is not explicitly mentioned in the issue.The tools guide autonomous repository exploration according to contextual needs.
- Results: Fine-tuned open-source models reduce code-localization cost by 86% while maintaining competitive performance.The introduction reports comparable performance to state-of-the-art proprietary models, with API cost decreasing from $0.66 to $0.09 per example.
2 Related Work
Prior code-localization methods use retrieval, LLM reasoning, agentic traversal, or graph representations, but differ in navigation complexity, efficiency, and dependency coverage. LOCAGENT combines a comprehensive repository graph with unified retrieval tools designed for LLM consumption.
- Retrieval Methods: Traditional retrieval methods rank code snippets through lexical or semantic matching, using sparse or dense representations.Sparse retrievers offer robustness to domain adaptation, while dense retrievers use embeddings for semantic search.
- LLM and Agent-Based Methods: LLM-based methods directly process issue queries and code, while agent-based systems add multi-step repository traversal through commands and specialized interfaces.Examples include Agentless, OpenHands, SWE-Agent, and MoatlessTools.
- Graph-Based Methods: Graph-based approaches capture relationships among code components through ranking, subgraph retrieval, graph databases, or guided exploration.RepoUnderstander uses MCTS, whereas other methods employ graph ranking, ego-networks, or database queries.
- LOCAGENT: LOCAGENT provides a more comprehensive and unified repository representation with efficient retrieval tools specifically designed for LLM consumption.Its framework combines graph-based code representation with agent-guided searches and sparse indexing.
3 The LOCAGENT Framework
LOCAGENT parses Python repositories into a heterogeneous dependency graph, builds sparse indexes, and equips agents with tools for autonomous multi-hop code exploration. It also fine-tunes open-source models to support practical local deployment.
- Framework Overview: LOCAGENT locates relevant code at file, class, function, or line granularity for diverse GitHub issue types.The framework includes graph-based indexing, agent-based search, and model fine-tuning or distillation components.
- Graph-Based Representation: The code representation is a directed heterogeneous graph whose node types are directory, file, class, and function, with contain, import, invoke, and inherit relations.The graph is designed to capture both explicit and implicit code relationships for multi-hop reasoning.
- Graph-Based Representation: LOCAGENT focuses on Python repositories and uses functions as the smallest indexed granularity and retrieval document.Functions are extracted recursively from Python files using abstract syntax trees.
- Sparse Indexing: Entity indexes map names and identifiers to graph nodes, while BM25 and code-chunk indexes support imperfect keyword matches and references outside entity names.These indexes combine exact name lookup, fuzzy keyword search, and coverage for terms such as global variables.
- Sparse Indexing: Graph dependencies connect modules across directory boundaries, helping identify components related through invocation or inheritance.This captures functional proximity that may be absent from directory-based navigation.
- Agent-Based Search: Agents autonomously search and navigate repositories through entity search, graph traversal, retrieval, and consistency-based ranking tools.Type-aware BFS supports configurable directions and hops, while repeated rankings are aggregated into confidence scores.
- Model Adaptation: Fine-tuning open-source models targets lower API costs and local deployment using successful reasoning trajectories generated by Claude-3.5 and Qwen2.5-32B.The collected trajectories contain cases where issues were correctly localized.
4 LOC-BENCH: A New Benchmark for Code Localization
LOC-BENCH is introduced to address contamination and scope limitations in existing code-localization benchmarks. It uses recent Python-repository issues spanning bug, feature, security, and performance scenarios.
- Existing Benchmarks: SWE-Bench contains GitHub issues and resolving patches, and its subset SWE-Bench-Lite has been adapted for code localization.Localization targets are treated as the patched files and functions.
- Existing Benchmarks: Existing datasets risk contamination from overlap with repositories or issues used during modern model pre-training and were not designed specifically for localization.SWE-Bench primarily evaluates end-to-end bug fixing, with localization serving as only one component.
- LOC-BENCH: LOC-BENCH collects up-to-date issues from Python repositories and covers bug reports, feature requests, security issues, and performance issues.The design aims to reduce pre-training bias and broaden evaluation scenarios.
- LOC-BENCH: Bug reports include GitHub issues created after October 2024, while additional security and performance cases are found through keyword searches.The dataset excludes patches modifying more than five Python files or ten functions.
5 Experiments
LOCAGENT is evaluated against retrieval-based and agent-based baselines on SWE-Bench-Lite and Loc-Bench, including localization accuracy, difficulty robustness, efficiency, ablations, and downstream issue resolution. Results show strong localization performance, competitive fine-tuned open-source models, lower costs, and benefits from graph traversal, sparse indexing, and multi-hop exploration.
- Evaluation setup: LOCAGENT is evaluated on SWE-Bench-Lite and Loc-Bench for localization accuracy, task-category variation, component contributions, and downstream software maintenance.The evaluation compares graph-based tooling with existing methods and examines fine-tuned open-source models as cost-effective alternatives.
- Localization results: LOCAGENT achieves the best results across file-, module-, and function-level localization among the compared methods.Agent-based methods consistently outperform other approaches, while Agentless is limited by its narrow repository scope.
- Difficulty analysis: Performance decreases as graph-based difficulty increases, but agent-based methods are more robust, with LOCAGENT remaining competitive across difficulty levels.Difficulty is measured by shortest-hop distance between functions mentioned in an issue and patched functions.
- Open-source models: Fine-tuned Qwen2.5-32B achieves performance comparable to Claude-3.5, while fine-tuned Qwen2.5-7B is on par with GPT-4o.The fine-tuned 32B model outperforms nearly all baselines, including methods using larger models.
- Efficiency analysis: Qwen2.5-7B(ft) costs $0.05 per example, and fine-tuned Qwen models provide more efficient alternatives than competing agent-based methods.LOCAGENT’s tool outputs remain token-efficient even when more interaction rounds are required with Claude-3.5.
- Ablation study: Removing tools, graph traversal, multi-hop exploration, or sparse index components reduces localization accuracy, especially for module- and function-level targets.Fixing exploration to one hop particularly harms function-level accuracy, while removing SearchEntity substantially lowers performance across metrics.
- Task categories and downstream results: LOCAGENT outperforms other methods in almost all task categories, although performance is lower outside bug reports, likely reflecting the training-data distribution.Better localization accuracy is associated with significantly improved GitHub issue-solving success rates.
6 Conclusion
LOCAGENT structures codebases as graphs to support efficient repository-level exploration by LLM agents. With fine-tuned open-source models, it achieves high localization accuracy while reducing costs and improving downstream tasks.
- LOCAGENT enhances code localization by structuring codebases as graphs for efficient repository-level exploration by LLM agents.
- Fine-tuned open-source models provide high localization accuracy at significantly lower cost than larger proprietary models.
- LOCAGENT improves downstream software maintenance tasks by identifying relevant code components.
Limitations
The study identifies limitations concerning model and fine-tuning coverage, downstream-task evaluation, training-trajectory diversity, and programming-language generalizability. It also examines graph-output formats, finding that tree-based representations improve localization performance while entity attributes can add noise.
- Model and fine-tuning scope: The study primarily fine-tunes Qwen-2.5-Coder models and does not compare broader base-model or fine-tuning choices.Suggested extensions include CodeLlama, Mistral, Yi, full fine-tuning, and other parameter-efficient methods.
- Downstream evaluation: Downstream evaluation of LocAgent extends mainly to bug repair, leaving refactoring, feature addition, security patching, and performance optimization for future study.The authors state that these applications were only partially explored.
- Training-data scope: Fine-tuning relies heavily on trajectories generated by Claude-3.5 and fine-tuned Qwen2.5-32B models, potentially limiting training-data diversity.The authors propose broader examples across models, tasks, and repositories, alongside analysis of dataset composition and filtering.
- Evaluation scope: The current evaluation focuses primarily on Python codebases and uses accuracy and NDCG, leaving cross-language and more nuanced localization-quality evaluation open.The authors propose extending support across programming languages and language paradigms.
- Graph-output design: Tree-based TraverseGraph subgraphs achieve the best overall localization performance, whereas adding entity attributes consistently underperforms, suggesting that extra node information can be noisy.The comparison evaluates multiple graph-output formats on challenging SWE-Bench-Lite samples using Claude-3.5.
A.2 Implementation
LocAgent adapts API interaction to the LLM’s tool-calling capabilities and returns compact graph-retrieval outputs. Its outputs include multiple formats, including fold, preview, full code, and tree-based subgraph representations.
- API interaction: Tool-capable LLMs receive JSON tool definitions, while Qwen receives API descriptions and expected outputs in the prompt.Locally executed retrieval requests return results to the agent after searching the built code graph.
- Output formats: Retrieval outputs use fold, preview, and full-code formats to provide necessary context while limiting lengthy, noisy agent inputs.Figure 6 illustrates format selection based on the number of matched entities and file size.
- Graph output: Figure 7 shows a truncated example of the expanded tree-based format returned for TraverseGraph output subgraphs.The figure presents the format used to expose graph structure to the agent.
B.1 Dataset construction details
The dataset construction collects diverse Python GitHub issues and derives approximate localization targets from patched files or functions. Performance and security issues are identified using keyword search and GPT-4o classification.
- Example collection: Examples are collected from popular Python GitHub repositories, with performance and security issues retrieved through GitHub Search API keywords.GPT-4o-2024-0513 classifies issues using their descriptions.
- Ground-truth locations: Patched files or functions in the original codebase serve as ground-truth locations for each issue.These locations are treated as approximate targets because a valid fix may occur elsewhere.
- Ground-truth locations: Documents, import statements, and comments edited by patches are excluded from localization targets because they do not directly affect code functionality or execution.The exclusion applies when extracting affected files or functions from patches.
C.1.1 Baselines Implementation
The evaluation implements embedding baselines at the function level with flat indexing and measures task difficulty through shortest-hop distances between issue-mentioned and patched functions in the code graph.
- Embedding baselines: Embedding-based baselines represent each function separately, appending its containing file and class context before constructing a flat index.The evaluated standard implementations do not separately embed files or classes.
- Evaluation execution: OpenHands and SWE-agent evaluations are parallelized with OpenHands’s remote runtime using version 0.12.0.The specified version was released on October 31, 2024.
- Difficulty measurement: Task difficulty is measured as the average shortest hop distance between predicted nodes extracted from issue descriptions and target nodes linked from patches.GPT-4o extracts potential function names, while the code-graph dictionary links them to nodes.
- Difficulty measurement: Difficulty groups use floor-rounded δ values, and samples without any valid extracted function names are excluded from stratified analysis.The shortest-path distance d(c, t) is computed between predicted and target graph nodes.
C.1.3 Training details.
The training details describe fine-tuning Qwen-2.5-Coder-Instruct variants on SWE-Bench samples using LoRA, while related passages identify prompt-history and ranking-quality materials.
- Training details: Qwen-2.5-Coder-Instruct 7B and 32B were fine-tuned on 768 SWE-Bench training samples using LoRA.The training set included 447 Claude-3.5-generated samples and additional samples generated iteratively with fine-tuned Qwen2.5-32B.
- Ranking evaluation: Table 11 compares the ranking quality of different methods using NDCG scores.
- Training details: Fine-tuning used 5 epochs, a maximum token setting of 128k, and a learning rate of 2 × 10−4.
- Prompt template: The agent-history section presents the prompt template used to construct the agent’s history.