Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study

1 Introduction

• The study traces complete agent decision pipelines because leaderboard scores reveal little about reasoning, failures, or comparison with human behavior.Earlier work documents APR agents' multi-step use of planning, self-reflection, and external tools.Source: §1 · p. 2
• Even when passing the test suite, agent patches often diverge from developer patches, typically becoming more verbose and structurally complex.This motivates tracing agent trajectories to understand how patches are generated and where decisions break down.Source: §1 · p. 2
• Agents struggle with logic-intensive issues and overfit to fail-to-pass tests, while additional test cases provide stronger constraints against spurious fixes.The paper emphasizes patch validation through testing as a result.Source: §1 · p. 2
• Agents often fail to generate tests that reproduce reported issues and misidentify suitable regression tests.Enhanced localization and scalable inference methods have potential to improve test-generation effectiveness.Source: §1 · p. 3
• Most agents rely on simple bash scripts with limited testing and debugging support.The authors identify richer tooling as an opportunity for APR systems.Source: §1 · p. 3
• The authors recommend earlier, higher-quality test generation and validation, richer integrated tools, complementary agent architectures, and benchmarks prioritizing semantic repair and test fidelity.The paper presents these directions as guidance from empirical analysis, rather than proposing new tools or automation.Source: §1 · p. 3

2.1 Trace Collection

• Each repair trace includes the generated patch, a resolution verdict based on fix and regression tests, and the ordered action trajectory.The traces support analysis of outcomes such as resolution and overfitting, as well as process-level tool use.Source: §2.1 · p. 3
• Researchers annotated trajectory steps, cross-checked them against agent implementations and documentation, and normalized agent and developer patches for later comparison.Step-level annotations summarize intended operations and rationale; lengthy traces become easier to interpret and verify manually.Source: §2.1 · pp. 3–4

2.2 Constructing a Taxonomy

• The taxonomy was manually derived from recurring trajectory patterns, independently annotated, adjudicated, and cross-validated against agent source code and documentation.Different actions with the same function, such as retrieving external information, could be grouped into one node.Source: §2.2 · p. 4
• Five agents share a high-level flow, but differ in localization, browser use, patch ranking, and when they generate tests.MASAI generates tests early; SWE-Agent and OpenHands defer test generation until after patch synthesis.Source: §2.2 · p. 5
• OpenHands, SWE-Agent, and AutoCodeRover iteratively refine a single patch, whereas MASAI and Agentless sample multiple candidates; only MASAI refines across them.Refinement can occur during file or class selection, context generation, and test generation.Source: §2.2 · p. 5
• Graph analysis identifies localization, patch generation, and reproduction testing as core workflow components across the agents.Frequently visited nodes are defined by high in-degree and used as proxies for core workflow functions.Source: §2.2 · p. 5

3.1 Benchmark Selection

• SWE-bench contains 2,294 real-world tasks from 12 Python repositories, with patches evaluated on bug-fixing and regression tests.SWE-bench Verified is a curated 500-task subset; annotators removed 68.3% of tasks for issues including invalid test harnesses, vague descriptions, or environment dependencies.Source: §3.1 · p. 5
• The study uses task difficulty to analyze agent performance and uses Verified-50 for controlled, computationally intensive experiments and manual analysis.Verified-50 is described as an unbiased random subset validated as representative of the full Verified benchmark.Source: §3.1 · pp. 5–6

3.2 Agent Selection

• Five agents were selected for leaderboard performance, open-source code or documentation, and inspectable trajectories; comparisons hold the underlying model constant where possible.Public trajectory availability constrained configurations, including ACR+GPT-4o and MASAI+Claude-3.5-Sonnet; the study contrasts architecture rather than base-model capabilities.Source: §3.2 · pp. 6–7

4.1 Motivation

• The analysis asks whether agent-generated fixes align with developer-written patches in semantic intent, complexity, and minimalism, extending evaluation beyond test-passing correctness.The comparison motivates examining how agents’ approaches differ from human strategies.Source: §4.1 · p. 7

4.2 Approach

• Patch comparison combines manual review of Verified-50 tasks, clone classification by an LLM judge, character-level Levenshtein distance, and edit-pattern and hunk statistics.A manual audit of 50 patch pairs found strong agreement with the clone judge (Cohen’s κ=0.81).Source: §4.2 · p. 8
• Manual analysis found that Agentless often produces shorter fixes but can miss bug locations in complex changes, while SWE-Agent’s larger structural edits can address extensive changes but are often unnecessarily complex.For simple fixes, SWE-Agent’s verbose patches are harder to interpret, evaluate, and maintain if merged.Source: §4.3 · p. 8

4.3 Results

• Workflow agents produce more developer-aligned patches (35–41% Type-1/2 clones; 22–23 changed lines), while open-process solutions reach 40× larger and 43–68% of successful patches are non-clones.These findings are corroborated by edit-distance and manual analyses.Source: Figure 2 · p. 9, §4.3 · p. 10
• Clone similarity decreases as task difficulty increases, with non-clones rising sharply; on easy tasks, workflow agents show higher Type-1/2 clone shares than open-process agents.The caption identifies the figure panels as clone distributions for resolved patches and clone breakdowns for easy tasks.Source: §4.3 · p. 9, Figure 2 · p. 9
• Open-process agents make broader, multi-file changes, while workflow agents produce concise, localized edits; OpenHands has the largest diffs, averaging 474.2 lines with Claude-3.5 and 331.2 with GPT-4o.Developer patches fall between these styles, averaging 2–3 hunks across 1–2 files.Source: §4.3 · pp. 9–10, Table 2 · p. 10

5 Agent Effectiveness in Patch Generation (RQ2)

• The analysis is motivated by identifying where agents excel and struggle to inform opportunities for improving their design and effectiveness.Source: §5.1 · p. 10

5.1 Motivation

• The section examines performance by issue difficulty, overlap in solved issues, and how fail-to-pass test availability relates to patch success and overfitting.The authors use agent trajectories from Claude-3.5-Sonnet and GPT-4o and manually analyze patches that pass generated reproducers but fail validation.Source: §5.2 · p. 11

5.3 Results

• In the moderate tier, agents solve different issues: only 27 of 261 are solved by all five, and unique solutions vary by agent.Manual review associates agent specializations with different bug types, including boundary-condition logic for SWE-Agent and serialization/ORM bugs for Agentless.Source: §5.3 · p. 11, Figure 3 · p. 12
• More fail-to-pass tests are associated with lower resolution across agents and backbones, while reproducer-passing patches can still fail ground-truth tests.The study reports overfitting rates of 17–26% for SWE-agent and 4–5% for Agentless.Source: §5.3 · p. 12, Figure 4 · p. 13
• Agent success falls from 66–73% on easy tasks to 4.8–19% on hard tasks, and no agents succeed on very hard issues.Only 27 of 261 moderate issues are solved by all agents; overfitting is higher in open-process agents (17–26%) than workflow agents (4–5%).Source: §5.3 · p. 12, Figure 3 · p. 12, Figure 4 · p. 13

6.1.1 Motivation

• The analysis extracts executed Python files as candidate reproducers and uses trajectory markers or logs to assess whether bugs are triggered and resolved.Workflow agents’ explicit reproduction steps allow direct parsing, while open-process agents’ executed Python files count even if they fail to trigger the bug.Source: §6.1.2 · p. 13

6.1.2 Approach

• Workflow agents generate reproducers for all tasks, while most open-process agents do so for fewer than 80%; resolution rates decline as difficulty rises.For Agentless and SWE-Agent, trigger rates are 41–57%, and Agentless has lower overfitting (4–5%) than SWE-Agent (18–30%).Source: §6.1.3 · pp. 13–14, Figure 5 · p. 14

6.1 Reproduction Testing

• The study investigates bug-localization context and inference-time sampling and temperature as ways to improve test generation after observing low triggering rates.It defines success as generating at least one test that reproduces the bug, with successful tests reused for patch validation.Source: §6.1.1 · p. 13, §6.1.4 · p. 14

6.1.4 Improving Reproducer Generation

• Low temperature (t=0.2) yields 46.0% reproduction success versus 42–43% at higher temperatures across both sample sizes.Higher temperatures increase average unique error types from 2.08 to 2.76, but success rates decline; 11 of 50 issues fail across all configurations.Source: §6.1.4 · p. 15, Table 3 · p. 15
• Bug localization increases reproduction success in 54% of Verified-50 issues, raising successful reproductions from 614 to 697 of 1,500 samples.It converts seven previously unreproduced issues to reproduced ones; each test requires API calls and execution, so better odds of early success can improve resource efficiency.Source: Table 3 · p. 15, §6.1.4 · p. 16

6.2 Regression Test Selection

• The section examines how often agents run regression tests and how accurately they select tests relevant to the issue across agent, backbone, and task-difficulty combinations.Relevant regression tests help preserve existing functionality while a patch fixes the reported bug.Source: §6.2.1 · p. 16

6.2.2 Approach

• The analysis identifies gold tests as pass-to-pass tests in SWE-bench and labels other project tests non-gold, comparing agent-run filenames against files containing gold tests.For workflow agents, trajectories are parsed for explicitly formatted tests; for open-process agents, execution of a pre-existing repository file counts as a regression test.Source: §6.2.2 · p. 16

6.2.3 Results

• Tasks with gold regression tests show higher resolution rates than tasks with only non-gold tests, but regression tests alone do not establish that a patch passes or resolves the issue.The resolution trend also persists as tasks become harder, even when regression tests are run.Source: §6.2.3 · p. 16
• Workflow agents run regression tests 90%+ of the time but often miss relevant tests; MASAI runs gold tests in 4% of cases versus 45% for Agentless.ACR runs gold tests in 100% of cases; MASAI has no dedicated regression-testing step and couples these tests with reproducer execution.Source: §6.2.3 · pp. 16–17

7 Agent Tooling (RQ4)

• The comparison groups tool APIs by category across OpenHands, SWE-Agent, and MASAI, excluding Agentless and ACR because their workflows preclude LLM action selection.It considers tools used in SWE-bench trajectories and excludes OpenHands web browsing.Source: §7.1 · p. 17

7.2 Results

• MASAI uses many specialized tools, while OpenHands has fewer and uses bash more frequently, regardless of LLM backbone.Across agents, running Python files is the most frequent bash use, followed by changing directories, listing contents, and searching; MASAI uses bash almost exclusively to run Python files, at least 10 percentage points more than any other agent.Source: §7.2 · p. 17
• Tool usage varies across agents, and no pair of sequential actions is common to all five agents in the study’s trajectory taxonomy.Action frequencies are largely robust to LLM choice, although GPT-backed configurations tend to run Python commands more and cd and ls commands less than Claude-Sonnet-backed configurations.Source: §7.2 · p. 18
• Despite diverse implementations, APR agents’ specialized action spaces reduce to similar low-level tools for viewing or editing files and executing bash commands.Their tools do not encapsulate higher-level procedures such as localization or program analysis, unlike sophisticated tools used in human workflows.Source: §7.2 · pp. 17–18

• The study compares five workflow-based and open-process APR agents across repair phases, complementing prior trajectory analyses that omitted workflow-based baselines or lacked APR-specific depth [7] [27] [28] [30] [58].It uses a trajectory-based taxonomy to contrast behavior in patching, testing, and tool use.Source: §8 · p. 18
• Unlike prior work on patch correctness or edit-site localization, the study examines how agent patches relate to human fixes through clone detection and construct-level analysis [10] [40] [49].It also differs from studies comparing human and human–AI testing practices [16] [32].Source: §8 · p. 18
• The study examines reproduction-test strategies within end-to-end repair and quantifies how localization information affects test generation, complementing earlier work on prompting and benchmark prompts [19] [33].The passage identifies issue reproduction as a prerequisite for repair and a signal for validating fixes [33].Source: §8 · p. 18

• The study evaluates fixed-snapshot, human-authored repositories, so its results do not cover evolving codebases with increasingly mixed human- and AI-authored code.Current benchmarks do not capture this distribution shift.Source: §9 · p. 19
• The analysis covers five high-performing, architecturally diverse agents and representative high-impact workflows, so results may not generalize to all systems or decision pathways.Some analyses use selected agent subsets, and extending the study to more agents is left for future work.Source: §9 · p. 19
• Benchmark contamination remains possible, although agents trigger bugs only 50% of the time and comparative findings may be robust if memorization benefits agents similarly.The authors also point to architecture-dependent differences in resolution and overfitting rates as evidence that design and intermediate reasoning signals matter.Source: §9 · p. 20

10 Conclusions

• Workflow agents produce concise, developer-aligned fixes on simpler tasks, while open-process agents handle complex refactorings but can generate patches up to 40× larger.No single agent dominates.Source: §10 · p. 20
• On harder issues, agents increasingly diverge from developer style and plateau beyond ∼40 tasks, while architectural scaffolding shapes their repair strategies.Source: §10 · p. 20
• Agents often fail to generate reproduction tests, triggering bugs only half the time, and struggle to select relevant regression tests; workflow agents also overfit to self-generated reproducers.Selective inference scaling, localization context, and higher-quality test generation can mitigate validation problems and reduce spurious patches.Source: §10 · p. 20
• Agents mostly rely on primitive commands rather than debuggers or analyzers, while specialized APIs can shift behavior, including MASAI’s 2× higher Python execution frequency.Source: §10 · p. 20
• The authors recommend hybrid architectures, shift-left testing, localization with selective scaling, decoupled test selection, richer tooling, and ensembles of complementary agents.Agentless shows a 12× improvement with decoupled, LLM-based test selection and early filtering.Source: §10 · p. 20
• Benchmarks should assess reproduction-test quality, regression-test selection, and semantic correctness beyond test passing, report stratified results, and model distribution shift in evolving codebases.The authors attribute observed behavior primarily to scaffolding rather than LLM capability and argue that the findings likely transfer to newer systems built on related scaffolding.Source: §10 · p. 20