PaperlayerView exact revision on arXiv

Source-linked AI summary

XAgent: eXecution-guided Agentic AI for Effective Localization and Resolution of GitHub Issues

Hieu Huynh, Patanamon Thongtanunam, Michael Fu, Bach Le, Kla Tantithamthavorn

arXiv:2609.09769v1cs.SE

TL;DR

Existing issue-resolution agents rely on static issue descriptions for localization and validation, with dynamic execution behavior and additional program context not fully incorporated. XAgent compares failing and passing executions to localize buggy functions and uses program context to augment validation tests. On SWE-bench-lite, it reports a 62.0% resolve rate and 72.8% function localization accuracy, outperforming state-of-the-art approaches.

  • Problem

    Existing agents rely on static issue descriptions for localization and validation, while dynamic execution behavior and additional program context are not fully incorporated.

    Source: §1 · pp. 2–3

  • Method

    XAgent compares failing and passing executions to localize buggy functions and augments validation tests with relevant program context.

    Source: §1 · p. 3

  • Results

    62.0% resolve rate and 72.8% function localization accuracy, with 37% lower cost, are reported for XAgent on SWE-bench-lite versus state-of-the-art approaches.

    Source: §9 · p. 24

  • Takeaways & Limitations

    XAgent’s 7 additional fixes show it can address issues left unresolved by top existing baselines.

    Source: §1 · p. 4

    XAgent depends on agent-generated reproduction-script quality; it creates buggy/non-buggy script pairs for 93% of evaluated issues, with the remaining 7% falling back to LLM localization.

    Source: §8 · p. 24

Abstract

from arXiv · show

Agentic AI has enabled capabilities in leveraging Large Language Models (LLMs) to autonomously resolve repository-level GitHub issues. However, due to the reliance on limited static description of issues, existing agentic approaches suffer from incorrect localization and incomplete validation. Solely relying on this information can bias LLM reasoning toward the narrow scope of the issue description, leading to incomplete patches that fail to address the underlying issue. In this paper, we present XAgent, an execution-guided agentic framework that analyzes dynamic behavior and additional program context to localize and validate issues. The experimental results on the SWE-bench-lite dataset demonstrate that XAgent outperforms other existing approaches, achieving a resolve rate of 62.0% and a function localization accuracy of 72.8%, while maintaining cost efficiency. Our analysis further shows that XAgent successfully resolves 7 additional issues that the top existing baselines fail to address. This work highlights a shift from static, description-oriented patch generation toward dynamic execution-guided issue resolution, opening new opportunities for LLM-based coding agents to achieve more robust and generalizable software maintenance.

1 Introduction

XAgent addresses the limitations of static, issue-description-driven localization and validation with execution-guided analysis and broader, context-aware tests. On SWE-Bench-Lite, it reports a 62.0% Resolve Rate and 72.8% function-level localization accuracy while reducing cost by 37%.

1 Introduction

  • Noisy or incomplete issue descriptions can misdirect keyword-based localization and narrow validation, yielding patches that pass generated tests but miss the underlying issue.The paper identifies these as limitations of existing approaches and links them to overfitting patches (Smith et al. 2015; Le et al. 2018).Source: §1 · pp. 2–3
  • XAgent localizes bugs by comparing failing and passing executions, identifying suspicious functions present only in failing runs and using tree edit distance to analyze execution divergence.This execution-guided method is inspired by differential analysis (Zeller 2009).Source: §1 · p. 3
  • XAgent’s context-aware test augmentor uses relevant files and functions to generate edge-case and related-component tests beyond the issue description.The broader validation scope aims to ensure patches address the underlying issue rather than only the description’s narrow scope.Source: §1 · p. 3
  • XAgent resolves 7 issues that SWE-Agent (Yang et al. 2024a), ExpeRepair (Mu et al. 2025), and Refact Agent (Refact.ai 2025) failed to fix.The ablation study reports that execution-guided localization and the context-aware test augmentor together contribute up to 9% relative improvement in overall performance.Source: §1 · pp. 3–4
  • XAgent achieves a 62.0% Resolve Rate and 72.8% function-level localization accuracy on SWE-Bench-Lite while reducing cost by 37%.The reported cost is approximately $1.56 per issue.Source: §1 · pp. 3–4

2 Background & Related Work

Repository-level issue resolution uses LLMs and agentic tools to reproduce, localize, patch, and validate bugs, but reliance on issue descriptions and reproduction tests can produce mislocalization and incomplete fixes.

Background

  • Repository-level issue resolution involves understanding, reproducing, localizing, fixing, and testing bugs, with agentic LLMs planning and adapting through tool use and environmental feedback.Existing approaches organize this workflow into reproduction, localization, and patch generation and validation.Source: §2 · p. 4

Reproduction

  • Reproduction approaches generate tests from issue descriptions and repository code context, then execute them to verify the reported behavior.The reproduction test is one of the limited inputs that can bias localization toward the failure scenario.Source: §2 · pp. 4–5

Localization

  • Localization methods use keyword search, hierarchical narrowing, fault localization, code graphs, error traces, or reproduction-test execution information to identify candidate files and functions.Examples include AGENTLESS (Xia et al. 2025), AutoCodeRover (Zhang et al. 2024), LocAgent (Chen et al. 2025), OrcaLoca (Yu et al. 2025), and PatchPilot (Li et al. 2025).Source: §2 · pp. 4–5

Limitations

  • Limited issue descriptions and reproduction tests can bias localization toward the failure scenario, while description-scoped validation can pass patches that miss the underlying issue.Existing localization and validation methods rely heavily on those narrow inputs, without accounting for broader program context.Source: §2 · p. 5

Patch Generation & Validation

  • Patch generation iteratively proposes fixes for localized functions, then checks reproduction and regression tests; agentic systems select actions and adapt plans using validation feedback.SWEagent (Yang et al. 2024a) and OpenHands (Wang et al. 2024b) interact with coding environments, while EXPEREPAIR (Mu et al. 2025) adds dual memory and generates tests with varied hyperparameters.Source: §2 · p. 5

3 Motivating Examples

The examples motivate two ideas: contrasting buggy and passing execution traces to localize defects, and using broader program context to validate fixes beyond the reported symptom.

3.1 Example 1

  • In Matplotlib, clf() removed a slider without releasing its mouse event, freezing interaction; fixing delaxes to release active mouse events restored responsiveness.Source: §3.1 · p. 6
  • Comparing traces of 756 buggy-run and 758 passing-run functions isolated 15 buggy-only functions, including delaxes, the actual Matplotlib defect location.The buggy-only functions included clf, delaxes, and clear; the fix changed delaxes to release active mouse events before clearing the figure.Source: §3.1 · pp. 6–7

3.2 Example 2

  • Seaborn’s pairplot regression mishandled unselected hue categories: lookup failure should map them to transparent (0, 0, 0, 0), fixed in HueMapping.lookup_single.Source: §3.2 · p. 7
  • Existing approaches (Refact.ai 2025; Yang et al. 2024a; Mu et al. 2025; Yang et al. 2025) tested pairplot narrowly and missed scatterplot validation, motivating context-aware tests of related functions that share plotting logic.Source: §3.2 · pp. 7–8

4 XAgent: eXecution-guided Agentic AI

XAgent combines execution-trace-based localization with program-context-aware validation in a workflow of reproduction, localization, and patch generation. It uses related files and functions to broaden validation beyond the issue description and iterates patches against validation and regression tests.

4.1 Overview

  • XAgent’s three phases reproduce buggy and correct behavior, localize suspicious functions through execution, then generate and validate patches.The framework pairs dynamic behavior analysis for localization with program context for more reliable validation tests.Source: §4.1 · pp. 8–9

4.2 Reproduction

  • The reproduction agent creates similar buggy and reference scripts, checks their outputs against the reported behavior, and regenerates them if the expected behaviors are not confirmed.Keeping input differences minimal reduces unrelated trace differences that could obscure root-cause analysis.Source: §4.2 · p. 10

4.3 Localization

  • XAgent compares buggy and non-buggy call trees with APTED, then ranks uniquely buggy and issue- or reproduction-highlighted functions by UniXcoder similarity to the issue description.It retains the top K candidates for subsequent patching.Source: §4.3 · p. 11

4.4 Patch Generation & Validation

  • The validation augmentor extracts relevant files and functions from the reproduction trajectory and generates extra tests, while the patch agent iterates through ranked functions until tests pass or a limit is reached.The patch agent checks five validation tests and existing regression tests before and after candidate patch generation; reusing trajectory context avoids redundant exploration and reduces token and computational costs.Source: §4.4 · p. 12

5 Experimental Setup

The experiments evaluate XAgent’s effectiveness and efficiency, component contributions, and performance across issue-complexity levels using SWE-bench-Lite and comparisons with leaderboard methods.

5.1 Research Questions

  • The evaluation asks how accurate and cost-efficient XAgent is, how its components contribute, and how resolve rate changes with issue complexity.These are the three research questions: effectiveness and efficiency, ablation, and resolve rate versus issue complexity.Source: §5.1 · pp. 12–13

5.2 Benchmark

  • SWE-bench-Lite provides 300 real GitHub issues from 12 Python repositories, testing repository-level changes without prior knowledge of issue locations.Unlike isolated-function benchmarks, its tasks require reasoning across files; the public leaderboard also enables comparison with prior methods.Source: §5.2 · p. 13

5.3 Experimental Setup

  • The setup uses Claude Sonnet 4, limits suspicious functions to K = 20, and caps reproduction and patch agents at 100 and 200 actions.K = 20 covers 80% of cases; validation-test temperatures are 0.4, 0.6, and 0.8, while reproduction uses 0 and patch generation uses 0.5.Source: §5.3 · p. 13

5.4 Baselines

  • Nine top-performing SWE-bench-Lite leaderboard methods are compared using their official public submissions.The selected methods are the highest-ranked leaderboard entries.Source: §5.4 · p. 14

6 Experimental Results

Across SWE-bench-lite experiments, XAgent achieves strong localization and issue-resolution results with lower token usage and cost. Ablations attribute improvements to execution-guided localization, validation-test augmentation, and semantic-similarity ranking, while results across complexity levels show continued effectiveness on harder issues.

6.1 RQ1: How accurate and cost-efficient is XAgent in bug localization and issue resolution?

  • 86.6% file-level localization F1, 72.8% function-level localization F1, and a 62% issue resolution rate give XAgent the best overall performance across both tasks.XAgent also uses nearly 40% fewer tokens than the prior state of the art, reducing total cost by approximately $300 across all instances.Source: Table 1 · p. 15, §6.1 · pp. 15–16
  • XAgent achieves 85.1% file-level Precision and 86.6% file-level F1, plus 72.6% function-level Precision and 72.8% function-level F1.Refact Agent has the highest recall but substantially lower precision; XAgent perfectly localizes 181 instances and uniquely identifies correct locations in 11, with differential analysis accounting for 7 of those 11.Source: Figure 4 · p. 15, §6.1 · p. 16

6.2 RQ2: How does each key component of XAgent contribute to its overall performance?

  • The execution-guided bug localizer raises function-level F1 by 4.5% to 73.5%, improves %Resolved by 1.3% (four issues), and reduces token usage by 13% versus the vanilla agent.Adding validation-test augmentation raises %Resolved from 58.0% to 62.0%, resolving 12 additional issues, while decreasing function-level F1 by 0.7%.Source: Table 2 · p. 18, §6.2 · p. 19
  • The embedding similarity ranker achieves the highest Top-10 and Top-20 localization accuracy among the three tested ranking strategies.Commit-history ranking can miss older or long-stable code, while proximity ranking can misrank faults deeper in the execution call chain.Source: Table 3 · p. 19, §6.2 · p. 19

6.3 RQ3: How does the resolve rate change across different levels of issue complexity?

  • For one-hunk issues, XAgent resolves 70% versus SWE-Agent’s 64%; for three-hunk issues, it leads with 44%, while ExpeRepair and Refact Agent each reach 35%.For two-hunk issues, Refact Agent leads at 54% and XAgent follows at 51%; all methods perform best on single-hunk cases.Source: Figure 5 · p. 20, §6.3 · pp. 20–21
  • For issues modifying one function, XAgent achieves the highest %Resolved rate at 67%; for two to five functions, it ties Refact Agent at 41%.Across ranges of changed lines, resolution rates decline as line changes increase, but XAgent consistently records the highest performance.Source: Figure 5 · p. 20, §6.3 · p. 21

7 Discussion

The discussion finds that execution-guided localization sharply narrows the function search space, while XAgent produces functionally correct patches without relying on exact-match memorization and adds little runtime overhead.

7.1 How much search space can be reduced by our execution-guided bug localizer?

  • The execution-guided localizer reduces the average search space from 20,000 repository functions or 234 buggy-executed functions to 52 uniquely buggy-executed functions, with a median of 11.Searching the whole repository can span up to 33,618 functions; 234 functions remain substantial context for an LLM to analyze effectively.Source: Table 4 · p. 22, §7.1 · p. 22

7.2 How similar are XAgent’s patches to human-written patches?

  • XAgent’s exact-match rate is 8.3%, and for complex bugs it generates functionally correct alternatives to human-written patches, indicating it does not rely on patch memorization.Most exact matches were simple 1–2-line edits, while complex bugs required multi-line or multi-function changes with multiple valid fixes.Source: Figure 6 · p. 22, §7.2 · pp. 22–23

7.3 How long does XAgent take to resolve an issue?

  • XAgent takes approximately 8.8 minutes per issue, with reproduction and patch generation accounting for 80.1% of total time.Execution-guided localization takes 49 seconds (9.3%), while test augmentation takes 56 seconds (10.6%), indicating both components add limited runtime overhead.Source: §7.3 · p. 23

8 Limitations

XAgent’s reported effectiveness depends on reproduction-script quality and available code information, while the authors also examine cross-dataset performance and validation reproducibility. The limitations include under-specified issues, flaky tests, and potential randomness in higher-temperature outputs.

8 Limitations

  • XAgent and TRAE both achieved 82% %Resolve on the 93-instance overlap between SWE-Bench-lite and SWE-Bench-verified, indicating comparable performance across the datasets.The authors use SWE-Bench-lite for fair comparison because most baselines use it, despite its potential under-specified issues and flaky tests; TRAE is an ensemble that may cost more.Source: §8 · pp. 23–24
  • XAgent generated buggy and non-buggy reproduction-script pairs for 93% of evaluated issues; the remaining 7% fell back to LLM localization.Its effectiveness depends on script quality, and it can use only information already available in code to generate scripts.Source: §8 · p. 24
  • Spectrum-based fault localization can be more thorough, but often requires comprehensive test suites that are rarely available for newly reported bugs (Jones et al. 2002; Abreu et al. 2007; Zhang et al. 2024).Accordingly, XAgent relies on existing code information when generating reproduction scripts.Source: §8 · p. 24
  • Across three runs at temperature 0.8, generated validations had 91.2% average pairwise cosine similarity (SD = 0.038), indicating reasonably consistent outputs.Reproduction and patch generation used temperatures 0 and 0.5; validation generation used 0.8 for diversity, and the complete implementation and experimental setup are publicly available.Source: §8 · p. 24

9 Conclusion

XAgent combines dynamic program execution with context-aware test generation to localize and validate GitHub issue fixes. On SWE-bench-lite, it outperforms the named SOTA approaches while maintaining lower cost, though TRAE’s generation cost was unavailable for comparison.

9 Conclusion

  • 62.0% resolve rate and 72.8% function localization accuracy on SWE-bench-lite outperform SOTA approaches including ExpeRepair, Refact Agent, and SWE-Agent, with 37% lower cost.The paper notes that TRAE did not provide its actual generation cost, so cost could not be compared against it.Source: §9 · p. 24
  • XAgent compares failing and successful executions to localize buggy functions and augments tests with program context to improve validation coverage and reduce overfitting to issue descriptions.The framework integrates dynamic program execution with context-aware test generation for patch validation.Source: §9 · p. 24
  • TRAE did not provide its actual generation cost, preventing a cost comparison.Source: §9 · p. 24

10 Data Availability

The source code and experimental details are publicly available in an anonymous GitHub repository; the authors also report no funding or relevant competing interests, with ethics approval not applicable.

10 Data Availability

  • The paper states that its source code and experimental details are publicly available at an anonymous GitHub repository, identified parenthetically as xag 2026-01-28.Source: §10 · p. 25

Declarations

  • The authors report no funding or relevant competing interests, and ethics approval is not applicable.Source: Declarations · p. 25

Related papers

Loading 2609.09769v1…