PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories

Oorja Majgaonkar, Zhiwei Fei, Xiang Li, Federica Sarro, He Ye

arXiv:2511.00197v1cs.SEcs.AI

TL;DR

Code agents can resolve complex software issues, but success-rate evaluations leave their problem-solving behaviour opaque. This study compares execution trajectories from OpenHands, SWE-agent, and Prometheus on SWE-Bench. It finds that success involves context gathering and approximate fault localisation, while failures often have longer, more variable trajectories.

  • Problem

    Success-rate evaluations treat code agents largely as black boxes, leaving limited understanding of how they approach tasks and why they fail.

    Source: §1 · p. 1

  • Method

    The authors comparatively analyse successful and failed SWE-Bench trajectories from OpenHands, SWE-agent, and Prometheus.

    Source: §1 · p. 1

  • Results

    Across the studied agents, success involves context gathering and approximate rather than exact fault localisation, while longer, more variable trajectories often signal failure.

    Source: §4.4 · pp. 7–8, §6 · p. 8

  • Takeaways & Limitations

    Trajectory analysis offers a foundation for interpreting and improving code agents beyond binary success metrics.

    Source: §1 · p. 1

    RQ1 case studies cover only a small, uniformly sampled subset of trajectories, and extracting trajectory events may be subjective.

    Source: §5 · p. 8

Abstract

from arXiv · show

The increasing deployment of Large Language Model (LLM) agents for complex software engineering tasks has created a need to understand their problem-solving behaviours beyond simple success metrics. While these agents demonstrate impressive capabilities in automated issue resolution, their decision-making processes remain largely opaque. This paper presents an empirical study of agent trajectories, namely the execution traces capturing the steps agents take when attempting to resolve software issues. We analyse trajectories from three state-of-the-art code agents (OpenHands, SWE-agent, and Prometheus) on the SWE-Bench benchmark, examining both successful and failed attempts. Our investigation reveals several key insights into agent behaviour. First, we identify how distinct problem-solving strategies, such as defensive programming and context gathering, enable success in different scenarios. Second, we find that failed trajectories are consistently longer and exhibit higher variance than successful ones, with failure patterns differing significantly between agents. Third, our fault localisation analysis shows that while most trajectories correctly identify problematic files (72-81\% even in failures), success depends more on achieving approximate rather than exact code modifications. These and other findings unveiled by our study, provide a foundation for understanding agent behaviour through trajectory analysis, contributing to the development of more robust and interpretable autonomous software engineering systems.

1 Introduction

Code agents can resolve repository-level issues, but success-rate evaluations leave their decision-making opaque. This study compares three agents’ execution trajectories to identify strategies, failure patterns, and fault-localisation behaviour.

1 Introduction

  • Existing evaluations focus mainly on success rates, leaving unclear whether failures arise from insufficient context, flawed reasoning, poor fault localisation, or other factors.Prior work has used trajectories to improve performance or study high-level patterns, but not to explain in detail how agents approach tasks and fail.Source: §1 · p. 1
  • The first comparative SWE-Bench trajectory analysis examines successful and failed attempts by OpenHands, SWE-agent, and Prometheus.It moves beyond binary outcomes to study the pathways agents follow.Source: §1 · p. 1
  • The study investigates agent-specific problem-solving strategies, structural differences between successful and failed trajectories, and fault localisation at multiple granularities.The questions cover uniquely solved issues, trajectory length and variance, and how precisely agents identify problematic code.Source: §1 · p. 1
  • The authors argue that robustness and interpretability, rather than raw leaderboard scores, are central challenges for next-generation autonomous software engineering systems.They present trajectory-level understanding as a foundation for interpreting and improving code agents.Source: §1 · p. 1
  • The study assembles and normalises traces from three agents on SWE-Bench Lite and Verified, then analyses strategies, failure modes, trajectory lengths, and localisation accuracy.Its contributions include a cross-agent trajectory study and eight takeaways on agent behaviour.Source: §1 · p. 2

2 Related Work

Code agents combine language models with scaffolding and tools to handle complex, multi-step software tasks, but their diverse architectures and reasoning strategies remain difficult to understand. Prior trajectory studies have supported performance improvement and high-level pattern analysis, while fine-grained comparison of behaviour and partial progress in failures remains limited.

2.1 Code Agents

  • Code agents combine one or more core LLMs with scaffolding and tooling to reason, act, observe outcomes, and plan subsequent steps.The paper names SWE-agent, AutoCodeRover, OpenHands, RepairAgent, Prometheus, and Agentless as examples.Source: §2.1 · p. 2

2.2 Agent Trajectory Studies

  • Trajectory research has used traces for contrastive learning, fine-tuning, and iterative optimisation, while studies of agent behaviour have examined execution errors, patch properties, coordination, visual inspection, and high-level patterns.The authors identify a gap in fine-grained, bottom-up analysis, especially insight into partial progress in failing trajectories.Source: §2.2 · p. 2

3 Empirical Study Design

The empirical study combines case studies of uniquely solved issues with quantitative comparisons of trajectory length and fault-localisation ability. It examines three agents on SWE-Bench Lite and Verified using a shared trajectory framework and patch comparisons against benchmark gold patches.

3 Empirical Study Design

  • The study proceeds from case studies of uniquely solved issues to comparisons of trajectory lengths and fault-localisation ability.This design combines qualitative and quantitative analyses.Source: §3 · p. 2

3.1 Data Collection

  • OpenHands, SWE-agent, and Prometheus were selected for being open source, having available trajectories, and being recently released and maintained; both SWE-Bench Lite and Verified were used.OpenHands and SWE-agent are described as established baselines, while Prometheus is specialised in codebase-level understanding.Source: §3.1 · p. 2

3.2 Research Questions

  • The research questions ask which bugs agents uniquely resolve, how successful and failed trajectory lengths differ, and how precisely patches locate faults.The localisation question uses patch matches to explore repository understanding, problem-solving progress, and fault localisation.Source: §3.2 · pp. 2–3

3.3 Methodology for RQ1

  • For RQ1, the authors inspect three uniformly sampled trajectories from each agent’s uniquely solved issue set, segmenting traces into five problem-solving phases.The phases are diagnosis, problem reproduction, hypothesis and plan, implementation, and verification; repeated attempts are treated as loops.Source: §3.3 · p. 3

3.4 Methodology for RQ2

  • For RQ2, trajectories are counted as individual thought, action, or result steps, rather than iterations, to preserve event-level granularity.Agent-specific parsers convert disparate logs into a unified format.Source: §3.4 · p. 3

3.5 Methodology for RQ3

  • For RQ3, candidate patches are compared with gold patches for matching files, functions, and hunks, with a lenient hunk match allowing line differences within five lines.The study tallies both marginal metric frequencies and joint combinations; gold patches guide comparison but do not determine success, which depends on passing tests.Source: §3.5 · p. 3

4 Empirical Study Results

Across the three agents, the study compares uniquely repaired issues, successful and failed trajectory lengths, and fault localisation at file, function, and hunk levels. Successful patches usually locate the right file but need not match the gold function or hunk exactly; failures are longer and more variable, with patterns differing by agent and benchmark.

4.1 RQ1: Uniquely Repaired Issues

  • The study presents randomly selected case studies from each agent’s uniquely solved issue set to examine differences in their trajectories.Figure 1 illustrates uniquely resolved issues across SWE-Bench Lite and Verified.Source: §4.1 · p. 3, Figure 1 · p. 4

4.1.1 Prometheus uniqueness.

  • The case studies show that Prometheus uniquely resolved 7 Lite issues and 8 Verified issues, with repository-aware context gathering helping it identify an enum-representation fix in Django.Its Django patch followed existing repository patterns, unlike the more complex paths taken by SWE-agent and OpenHands.Source: §4.1.1 · p. 4

4.1.2 OpenHands uniqueness.

  • OpenHands uniquely resolved 11 Lite issues and 17 Verified issues; in the Pylint case, it found the right functionality, while Prometheus failed after extended context gathering and SWE-agent regressed before debugging.The finding cautions that limiting recursion alone may not prevent unproductive reasoning loops.Source: §4.1.2 · pp. 4–5

4.1.3 SWE-agent uniqueness.

  • SWE-agent uniquely resolved 46 Lite issues and 71 Verified issues, and its SymPy case shows a defensive check achieving the fix despite less mathematical insight than the gold patch.Prometheus and OpenHands pursued symbolic-value hypotheses and did not identify the root cause.Source: §4.1.3 · p. 5

4.2 RQ2: Length of Successful and Failing Trajectories

  • Failed trajectories are longer on average and more widely distributed than successful ones, with the size and variability of the gap differing across agents and benchmarks.Prometheus failures show especially high variability; SWE-agent failures are comparatively less divergent.Source: §4.2 · p. 5, Table 4 · p. 6

4.3 RQ3: Fault Localisation in Successful and Failing Trajectories

  • Over 90% of successful patches target the gold file, but only about 27% target the gold function, indicating that successful fixes can use approximate edits within the correct file.Only a handful match the exact gold hunks, while around 9–11% are within five lines.Source: §4.3 · pp. 6–7, Table 5 · p. 6

4.4 Discussion

  • In Verified, 35%–40% of failed trajectories miss the correct file, compared with 20%–25% in Lite, and wrong-file failures take similar time to failures exploring correct paths.The discussion also identifies close hunk localisation and earlier abandonment of incorrect paths as possible improvement targets, rather than exact line matching.Source: §4.3 · p. 7, §4.4 · pp. 7–8, Figure 4 · p. 7

5 Threats to Validity

The case studies cover only a small subset of trajectories, limiting potential generalisability; sampling and trajectory-step extraction also introduce constraints.

5 Threats to Validity

  • The RQ1 case studies cover only a small subset of trajectories, which may limit generalisability.Uniform sampling reduces bias, but the sample may miss behavioural diversity.Source: §5 · p. 8
  • Extracting thought, action, and result steps may be subjective.Cross-agent comparisons also depend on consistent parsing; the study focuses on repository exploration and code modification events.Source: §5 · p. 8

6 Conclusion

Comparing trajectories from OpenHands, SWE-agent, and Prometheus reveals behavioural insights beyond binary outcomes, including strategies and failure patterns. Success involves approximate fault localisation, while perfect localisation is neither necessary nor sufficient.

6 Conclusion

  • Trajectory analysis of OpenHands, SWE-agent, and Prometheus reveals behaviours that outcome-only evaluation obscures.The study moves beyond binary success metrics to examine how agents gather context, recognise architecture patterns, and abandon unproductive searches.Source: §6 · p. 8
  • Longer, more variable trajectories often signal failure, and failure modes differ across agents.The study identifies distinct problem-solving styles across the three agents.Source: §6 · p. 8
  • Success depends on approximate rather than exact fault localisation; perfect localisation is neither necessary nor sufficient.The study links success with context gathering, recognising architecture patterns, and abandoning unproductive searches.Source: §6 · p. 8
  • The findings support moving beyond leaderboard-based evaluation toward frameworks for more interpretable and robust code agents.Source: §6 · p. 8
Loading 2511.00197v1…