Source-linked AI summary
OrcaLoca: An LLM Agent Framework for Software Issue Localization
Zhongming Yu, Hejia Zhang, Yujie Zhao, Hanxian Huang, Matrix Yao, Ke Ding, Jishen Zhao
TL;DR
Software issue localization remains difficult because agents must explore large repositories while balancing search completeness against concise, relevant context. OrcaLoca integrates priority-based scheduling, relevance-scored action decomposition, and distance-aware context pruning. It achieves a 65.33% function match rate on SWE-bench Lite and improves an open-source framework’s final resolution rate by 6.33 percentage points through patch-generation integration.
Problem
Software issue localization remains challenging because existing approaches struggle to balance context granularity, search-space completeness, stable action planning, and effective context management.
Method
OrcaLoca combines priority-based scheduling for LLM-guided actions, relevance-scored action decomposition, and distance-aware pruning of searched context.
Results
65.33% function match rate on SWE-bench Lite establishes OrcaLoca as a new open-source SOTA, while patch-generation integration improves final resolution by 6.33 percentage points.
Takeaways & Limitations
OrcaLoca provides an open-source software issue-localization framework that improves localization accuracy and raises final resolution when integrated with patch generation.
Takeaways & Limitations
Effective and robust exploration while balancing context granularity and search-space coverage remains a significant challenge.
Abstract
from arXiv · showhide
Recent developments in Large Language Model (LLM) agents are revolutionizing Autonomous Software Engineering (ASE), enabling automated coding, problem fixes, and feature improvements. However, localization -- precisely identifying software problems by navigating to relevant code sections -- remains a significant challenge. Current approaches often yield suboptimal results due to a lack of effective integration between LLM agents and precise code search mechanisms. This paper introduces OrcaLoca, an LLM agent framework that improves accuracy for software issue localization by integrating priority-based scheduling for LLM-guided action, action decomposition with relevance scoring, and distance-aware context pruning. Experimental results demonstrate that OrcaLoca becomes the new open-source state-of-the-art (SOTA) in function match rate (65.33%) on SWE-bench Lite. It also improves the final resolved rate of an open-source framework by 6.33 percentage points through its patch generation integration.
1. Introduction
Software issue localization is crucial but difficult in autonomous software engineering because repositories are large, requirements are imprecise, and existing exploration methods struggle to balance search completeness with concise, relevant context. OrcaLoca addresses these challenges with scheduled actions, relevance-scored decomposition, and distance-aware context pruning.
- Localization precisely identifies and navigates to relevant code for resolving software engineering problems, yet remains underexplored in ASE.
- 53.5% of issues achieve a correct function match on average across submitted SWE-bench Lite agent solutions.
- Existing LLM-guided exploration can produce unstable, redundant searches, while graph-based scheduling can confine searches to neighboring nodes.
- Concise contexts risk omitting critical details, whereas fully detailed search spaces introduce noise, redundancy, and irrelevant exploration paths.
- OrcaLoca combines priority-based scheduling, relevance-scored action decomposition, and distance-aware context pruning to improve codebase exploration and focus.
2. Related Work
Fault localization remains challenging in large-scale systems, and LLM-agent approaches add repository exploration capabilities but still face instability, inefficiency, and context-management trade-offs. OrcaLoca responds with dynamic action scheduling, decomposed-action scoring, and context pruning.
- Traditional fault localization uses spectrum-based, mutation-based, and learning-based techniques, but large systems overwhelm standalone learning models.
- LLM-based fault-localization methods enhance exploration with agents and tools such as repository retrieval, code analysis, and log analysis.
- OrcaLoca introduces dynamic action scheduling and decomposed-action scoring to balance context granularity with search-space coverage.
- ASE agents address project-level tasks requiring repository interaction beyond function-level coding, including code search, patch creation, and bug localization.
- Existing agent systems may rely entirely on LLM action management or include all search results as context, causing unstable, ineffective, inefficient, or misleading exploration.
3. Methodology
ORCALOCA combines a CodeGraph-based search environment with LLM reasoning, priority scheduling, action decomposition, and distance-aware context pruning. Its workflow iteratively selects, executes, refines, and prunes search actions until convergence or queue exhaustion.
- 3.1. Search System Setup and Agent Workflow: The CodeGraph represents containment and reference relationships among functions, classes, methods, and files, with hierarchical UIDs encoding entity locations.Containment covers methods within classes and classes within files; reference edges include function calls.
- 3.1. Search System Setup and Agent Workflow: The agent generates observations, potential bug locations, and search actions within a constrained reason-and-act workflow.Its state includes prior observations, potential bug locations, and retrieved search results, while actions query search APIs.
- 3.1. Search System Setup and Agent Workflow: The exploration loop executes prioritized actions, decomposes class or file results, prunes accumulated context, updates agent state, and concludes with predicted bug locations.Termination occurs when the action queue is empty or the convergence condition is met.
- 3.2. Priority-Based Scheduling for LLM-Guided Actions: A dynamic Action Scheduler Queue prioritizes one LLM-generated action at a time using contextual relevance, urgency counters, and stronger structural relationships.Repeated actions increase their counters, while related actions can receive elevated priority; redundant actions are skipped.
- 3.3. Action Decomposition with Relevance Scoring: Action decomposition breaks coarse class or file searches into finer-grained actions scored for relevance, reducing noisy traversal while addressing repository ambiguities.The approach targets skeletal searches that expose signatures but can require traversing every method.
- 3.4. Distance-Aware Searched Context Pruning: The Context Manager prunes search results by retaining relevant entries linked to valid UIDs and aligning them with potential bug locations in the CodeGraph.It is applied at every exploration step and excludes disambiguation and skeleton messages from the retained context.
4. Evaluation
ORCALOCA is evaluated on SWE-bench Lite using localization and resolution metrics, compared with open- and closed-source approaches, and tested through ablations. It achieves leading localization results and improves resolved-rate performance when integrated with Agentless-1.5 patch-generation components.
- Datasets: SWE-bench Lite contains 300 filtered instances, while SWE-bench Common contains 93 instances used for ablation studies.The evaluation uses SWE-bench Lite for the main experiments and SWE-bench Common for ablations.
- Baselines: ORCALOCA is compared with 17 SWE-bench Lite approaches spanning closed-source and open-source solutions.The comparison includes systems such as OpenHands, AutoCodeRover, Agentless, RepoGraph, HyperAgent, and SWE-Agent.
- Metrics: Function Match Rate and File Match Rate measure localization accuracy by comparing agent outputs with the golden patch.The function metric treats functions and methods under the general term function, while file matching maps localized graph nodes to containing files.
- Leaderboard performance: 65.33% Function Match Rate and 83.33% File Match Rate establish ORCALOCA as the open-source SOTA on SWE-bench Lite.These correspond to 196 of 300 and 250 of 300 instances, respectively.
- Resolved-rate performance: 41.00% of SWE-bench Lite issues are resolved, while Agentless-1.5 integration adds 6.67 percentage points in Function Match Rate and 6.33 percentage points in final Resolved Rate.The resolved count is 123 of 300 issues; the integration uses Agentless-1.5 Repair, Patch Validation, and Patch Selection components.
- Complementarity: ORCALOCA uniquely localizes 6 issues and resolves 8 issues compared with Agentless, AutoCodeRover, and OpenHands.The authors present these unique cases as evidence that ORCALOCA complements other open-source agents.
5. Discussion
The discussion describes practical extensions, overhead considerations, and scope boundaries for OrcaLoca. These include reproducer-guided initialization, configurable retrieval and batching, dynamic graph costs, model generalization, and language support.
- Reproducer-derived calling stacks initialize the agent with relevant context, avoiding cold starts based only on problem statements and search APIs.
- Top-k retrieval lets users choose final bug locations from Context Manager search results and adjust the precision-recall tradeoff.
- Batch execution reduces reasoning length and token consumption, but experiments use batch size 1 because larger batches may slightly reduce accuracy.
- Dynamic code search and per-repository CodeGraph reconstruction introduce overhead, motivating future caching to reduce redundant graph construction.
- Experiments primarily use Claude, while the framework is described as model-agnostic and future work targets locally deployed Qwen and LLaMA models.
- The current implementation focuses on Python repositories, although the framework is considered language-agnostic in principle and other languages require language-specific parsers.
6. Conclusion
OrcaLoca combines prioritized action scheduling, relevance-scored action decomposition, and distance-aware context pruning for software issue localization. It achieves a 65.33% function match rate on SWE-bench Lite and a 41.00% final resolution rate after patch-generation integration.
- 65.33% function match rate establishes OrcaLoca as a new open-source SOTA on SWE-bench Lite.
- OrcaLoca integrates priority-based scheduling, relevance-scored action decomposition, and distance-aware context pruning to streamline localization.
- 41.00% final resolution rate represents a 6.33 percentage points improvement over the original framework after integrating its patch-generation component.
A.1. Graph Construction Process
The framework builds a CodeGraph that represents containment and reference relationships among code entities, then exposes search APIs for navigating files, classes, methods, callables, and source snippets.
- The CodeGraph combines AST-derived containment structure with directory hierarchy and reference edges between code entities.
- Containment edges encode relationships such as methods within classes, functions within files, files within directories, and directories within the repository root.
- Reference edges capture execution dependencies including function calls, variable references, and module imports; function-call edges use vcaller → vcallee ∈e2.
- The system searches code through APIs for classes, methods, callables, file contents, file skeletons, and source-code snippets, with optional file or directory paths.
- Class and file searches return skeletons when content exceeds 100 or 200 lines, while method searches provide detailed method snippets after class inspection.
C. Reproducer Agent
The reproducer agent supplements static repository exploration by attempting to reproduce issues, capture execution traces, and prioritize suspicious code. Because reproduction succeeds for only 38.0% of issues, it remains complementary rather than core.
- 38.0% of issues are successfully reproduced in the experiment, so the reproducer agent serves as a complementary analysis step.
- C.1-C.3: The agent identifies suspicious functions and files, generates and executes reproduction snippets with retries, then filters and reranks execution-trace information.
- C.1: It extracts initial keywords by separately processing tracebacks, code snippets, and natural-language descriptions with tailored prompts.
- C.2: Reproduction runs in a Dockerized conda environment, records traces with VizTracer, and uses an LLM judge to determine success before forwarding logs for analysis.
- C.3: Trace filtering prioritizes entries near suspicious keywords, their descendants, and the trace root, retaining the top K = 25 candidates.
- C.3: A weighted combination of keyword priority and LLM relevance scoring retains candidates above an absolute threshold and within the top 5.
E. Convergence Configuration
OrcaLoca uses embedding-based similarity and a sliding-window rule to detect convergence during lengthy action sequences, allowing execution to stop once observations remain effectively unchanged.
- Early Stop Convergence Mode: Once convergence is triggered, the agent terminates execution and reaches a conclusion.This mode targets lengthy action sequences requiring multiple execution steps.
- Early Stop Convergence Mode: A BERT embedding model compares consecutive observations using cosine similarity to assess convergence.The observations are O_t and O_t+1.
- Early Stop Convergence Mode: Similarity above 0.97 marks two consecutive observations as equivalent.
- Early Stop Convergence Mode: The system requires this similarity condition for K = 15 consecutive steps before triggering convergence.A sliding window stabilizes the termination decision.
F. Cost Breakdown Analysis
The cost analysis treats token usage as the primary efficiency metric, identifies editing as the largest OrcaLoca cost component, and reports lower localization cost after batched execution.
- Cost Metric: Token cost is the primary metric because LLM inference dominates the system’s time and monetary expenses.With API-based model providers, inference time is approximately proportional to token usage.
- Cost Breakdown: Over half of OrcaLoca’s 1.77 average token cost comes from editing, including 0.90 attributed to the editing phase.The editing cost primarily reflects the adopted Agentless-1.5 editing component.
- Batched Optimization: OrcaLoca-batch groups top-priority scheduler actions during localization to reduce token usage.The comparison is summarized in Table 5.
- Batched Optimization: 34%: estimated per-instance localization cost fell from 0.87 to 0.58 across ten sampled issues without negatively impacting localization correctness.Weighted averages across cost bins were used because of budget constraints.
- Future Efficiency: Future work targets further efficiency improvements, including possible kv-cache integration during inference.
G. Other Competing Methods
The competing-method landscape includes coding agents, repository tools, multi-agent systems, and simpler issue-resolution pipelines spanning different levels of autonomy and specialization.
- Coding Agents: Blackbox AI Agent is developing a coding agent intended to transform software development.
- Coding Agents: OpenHands + CodeAct v2.1 provides flexible agents that write code, use command lines, and browse the web like human developers.
- Repository and Multi-Agent Tools: Moatless Tools emphasizes supplying relevant context and handling responses rather than relying solely on agent reasoning for large codebases.
- Issue Resolution: AutoCodeRover-v2.0 combines LLMs with code search to autonomously solve GitHub issues and produce patches.
- Issue Resolution: Agentless-1.5 uses a simpler three-phase process of localization, repair, and patch validation without LLM-directed future actions or complex tools.
- Repository and Multi-Agent Tools: RepoGraph manages repository-level structure as a plug-in module for modern AI software-engineering systems.
- Repository and Multi-Agent Tools: HyperAgent is a generalist multi-agent system addressing software-engineering tasks across multiple programming languages by emulating human workflows.
- Coding Agents: SWE-agent combines language-model agents with a custom agent-computer interface for repository navigation, code editing, and program execution.