AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systems

1 INTRODUCTION

• Agent harnesses combine backbone LLMs with complex infrastructure for context management, guardrails, orchestration, and tool integration.Harness improvements can enhance agent effectiveness even when the underlying backbone LLM remains unchanged.Source: §1 · pp. 1–2
• Harness bugs arise from agent-specific structures and behaviors, including workflow orchestration and interactions with external resources, and are harder for software agents to repair than general bugs.Prior work reports repair success rates of 4.67% for harness bugs versus 40.67% for general software bugs.Source: §1 · p. 2
• Existing agent-failure benchmarks largely focus on backbone-LLM failures, while executable harness-bug collections are small, fixed, and labor-intensive to construct.AgentIssue-Bench contains 43 bugs; reproducing a small number of harness bugs required 150 manual hours.Source: §1 · p. 2
• General-software reproduction techniques face agent-specific challenges because harness failures can depend on external resources and particular execution states or interaction contexts.These dependencies include LLM providers, tools, network services, and dynamically changing environments.Source: §1 · p. 2
• AgentBug-Smith automates three reproduction phases: identifying harness-bug issues, constructing agent execution environments, and generating failure-reproducing tests.Applying it to open-source agents produced Live-Harness-Bench, which contains 200 reproducible bugs; distilled repair skills improved harness-bug repair success by 6.32%.Source: §1 · pp. 2–3
• 10.67%-27.56% percentage-point improvements in harness-bug reproduction success over general-software techniques hold across all studied backbone LLMs.The benchmark evaluation also reports 95% accuracy in identifying agent repositories and 92% accuracy in identifying harness-bug issues.Source: §1 · p. 2

2 RELATED WORK

• Prior agent-failure studies mainly examine runtime failures from underlying LLMs, while harness-bug studies often lack executable environments or artifacts.AgentIssue-Bench is described as the only executable harness-bug benchmark, with 43 bugs and 150 hours of manual construction.Source: §2 · p. 3
• Automated bug-benchmark construction often synthesizes artificial bugs, whereas this work targets real-world bugs for a more realistic bug distribution.Prior real-world benchmark construction approaches primarily target general software rather than agentic systems.Source: §2 · p. 3
• Agentic harnesses pose reproduction challenges because they extensively interact with external resources, distinguishing them from conventional software systems.Source: §2 · p. 3

3 AGENTBUG-SMITH

• AgentBug-Smith uses a three-stage multi-agent pipeline to identify harness-bug issues, construct agent execution environments, and generate tests that trigger the reported failures.Source: §3 · p. 3

3.1 HARNESS BUG IDENTIFICATION

• Repository discovery combines curated agent lists with near-real-time GitHub Archive events, then filters candidates for adoption, tests, and agent structure before mining harness-bug issue/PR pairs.Issues must link to a merged PR with functional code changes; LLM inspection retains issues whose patches occur in harness components.Source: §3.1 · pp. 4–5

3.2 AGENT EXECUTION ENVIRONMENT CONSTRUCTION

• Environment construction synthesizes Dockerfiles for agent-specific dependencies and isolates external model calls by routing them to local test harnesses through container-level mock-service variables.It begins with rule-guided configuration inference and refines the Dockerfile using build errors and a lightweight sample of existing tests.Source: §3.2 · p. 5

3.3 FAILURE-REPRODUCING HARNESS TEST GENERATION

• Failure-reproducing tests must fail on the buggy commit with the issue-reported failure and pass on the patched commit, including for issues without in-patch tests.Generated tests must exercise the actual implementation; when test-only refinement is insufficient, the workflow can jointly refine the test and environment Dockerfile.Source: §3.3 · p. 6

4.1 EFFECTIVENESS OF HARNESS BUG IDENTIFICATION

• AgentBug-Smith identifies agent repositories with 95% accuracy and harness-bug-related issues with 92% accuracy, and is more stable across repeated runs than vanilla LLM invocation.The identification evaluation sampled 132 of 225 issues and used two annotators; Cohen’s kappa was 0.88.Source: §4.1 · pp. 6–7, Table 1 · p. 7

4.2 REPRODUCTION SUCCESS RATE

• AGENTBUG-SMITH reproduces up to 28.00% of harness bugs, outperforming existing baselines by 10.67–27.56 percentage points; improvements are consistent across all backbone LLMs, while the best baseline achieves only 13.78%.It reproduced 75 bugs, including 45 unique bugs, and manual inspection found 74 (98.67%) valid. The reported cost was $0.56–$2.44 per issue.Source: §4.2 · p. 7

4.3 ABLATION ANALYSIS

• 10.67%–35.11% percentage-point gains in environment-construction success over both baselines suggest agent-specific dependencies and execution characteristics must be accounted for.Table 3 compares environment construction across the studied techniques and backbone models.Source: §4.3 · p. 8, Table 3 · p. 8
• AgentBug-Smith generates failure-reproducing tests for issues without in-patch tests, while both baselines reproduce none of those issues.Table 3 reports the overall number of reproduced issues and separates those with and without in-patch tests.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 LIVE-HARNESS-BENCH: STATISTICS AND DOWNSTREAM APPLICATION

• Live-Harness-Bench covers diverse harness components, including tool registries and action interfaces (31.5%) and context and memory management (30.5%).Its reproduced bugs involve large repositories and non-trivial developer-submitted patches; the benchmark spans time and includes bugs reported as recently as August 2026.Source: §4.4 · pp. 8–9, Figure 5 · p. 8, Table 4 · p. 8
• Live-Harness-Bench can be extended with newly reported bugs and serves as a foundation for accumulating real-world harness failures and fixes.The authors note that periodic extension can help mitigate concerns about contamination from model training data.Source: §4.4 · pp. 9–10
• State-of-the-art software agents achieve at most 9.00% resolution on Live-Harness-Bench, compared with a reported 40.67% on SWE-Bench Verified.The evaluation distinguishes plausible patches that pass failure-reproducing tests from correctly resolved patches judged semantically equivalent to developer patches.Source: §4.4 · p. 9, Table 5 · p. 9
• A distilled skill improves mini-SWE-agent’s correct resolution rate by 6.32% and achieves 31.81% function-level localization accuracy.The test set contains 79 bugs, separated from 121 training bugs so bugs from the same agent repository do not appear in both sets.Source: §4.4 · p. 9, Table 6 · p. 10

5 CONCLUSION

• AgentBug-Smith builds Live-Harness-Bench, a live and extensible benchmark that currently contains 200 reproducible harness bugs and can grow as new bugs emerge.The conclusion also identifies evaluation and reusable repair knowledge as downstream uses of the benchmark.Source: §5 · p. 10

A REPOSITORY SELECTION RULES

• Repository discovery seeds from curated agent-system lists and expands them with a keyword family that can later grow by mining verified repositories.The curated-list and keyword streams are merged and deduplicated.Source: A REPOSITORY SELECTION RULES · p. 14
• Hourly GHArchive ingestion treats recent pushes, pull requests, issues, stars, forks, and releases as visibility signals, not necessarily evidence of code change.A star and a push both indicate that a repository is currently visible.Source: A REPOSITORY SELECTION RULES · p. 14
• Repository candidates must have at least 50 stars, a parseable README, in-repository tests, and a path matching the detector.Source: A REPOSITORY SELECTION RULES · p. 14

C ISSUE SELECTION RULES

• The repository selection document gives the issue judge a checklist and screening criteria, including repository metadata and test-path requirements.The listed repository gate requires at least 50 stars, a parseable README, in-repository tests, and a path matching the detector.Source: B AGENTIC SYSTEM SPECIFICATION · p. 14

B AGENTIC SYSTEM SPECIFICATION

• An Agentic System is a codebase whose operational logic is driven by an LLM and exhibits some autonomous planning, memory management, environmental perception, or tool use.The listed system components include an LLM “brain,” perception, action or tooling, orchestration, and provider integration.Source: B AGENTIC SYSTEM SPECIFICATION · p. 14
• Agentic systems differ from conventional software through nondeterminism, external dependencies, cross-component failures, prompt and context concerns, and workflow-oriented behavior.Source: B AGENTIC SYSTEM SPECIFICATION · p. 15

C ISSUE SELECTION RULES

• Issue selection records closure, a nonempty description, a linked merged PR targeting main or master, and a patch of at least 20 added-plus-deleted lines; in-patch tests are recorded but not required.Source: C ISSUE SELECTION RULES · p. 15

D HARNESS BUG SPECIFICATION

• A Harness Bug is a user-reported problem in an Agentic System concerning agent-specific machinery such as provider integration, tools, memory, LLM operation, workflows, or utilities.Source: D HARNESS BUG SPECIFICATION · p. 16
• The taxonomy has six categories and twenty sub-categories, with utility issues included so conventional-software failures can be excluded.Its categories include provider incompatibility, tool issues, memory issues, LLM operation issues, workflow issues, and utilities.Source: D HARNESS BUG SPECIFICATION · p. 16

E. Workflow issues E.1 Scheduling / loops hangs, infinite loops, skipped steps

• Workflow screening accepts an issue only when it is closed with a developer patch, at least one harness-bug checklist item applies, and it is not utility-only.The checklist covers provider/model references, prompts, memory, tools, workflow anomalies, and patch changes to harness components.Source: E. Workflow issues E.1 Scheduling /… · pp. 16–17

E ONE-TO-ONE ISSUE–PR–COMMIT MATCHING

• For each pull request, the matching procedure collapses its commits into a squash commit and keeps only one-to-one issue–PR–commit relationships.The resulting instance links an issue, pull request, and squash commit; the PR base is the buggy snapshot and the squash commit is the patched snapshot.Source: E ONE-TO-ONE ISSUE–PR–COMMIT MATCHING · p. 17

• After 25 test-only attempts across five epochs remained failing on both snapshots, a full-generation co-fix changed the container install path and rewrote the test around failed requests.post behavior.Source: F EXAMPLE OF JOINT OPTIMIZATION · p. 17
• Re-verification failed on the buggy snapshot and passed after the patch, producing the instance’s first failure-on-buggy, pass-on-patched outcome.Source: F EXAMPLE OF JOINT OPTIMIZATION · p. 17
• The trace illustrates how one diagnosis can reconcile the container source path and test oracle, but it does not establish that either edit is independently necessary.The authors note that the two crossed combinations were not evaluated.Source: F EXAMPLE OF JOINT OPTIMIZATION · p. 17

G FAILURE-MODE ANALYSIS

• Failure distributions vary across backbone models, suggesting that some failure modes are model-dependent and may need stage-specific improvements.The suggested targets include dependency resolution, environment construction, and execution-feedback collection.Source: G FAILURE-MODE ANALYSIS · p. 17
• Table 7 presents failure modes of unsuccessful reproduction runs.Source: Table 7 · p. 18
• Future work could prioritize dependency installation, Docker image construction, and preserving diagnostic logs when container disk exhaustion occurs.Source: G FAILURE-MODE ANALYSIS · p. 18

H STABILITY METRIC

• The stability check independently classifies each retained issue five times and reports set agreement and normalised entropy.Both scores equal 1 when every run returns the same label for every issue.Source: H STABILITY METRIC · p. 18

I SKILL DISTILLATION PROCESS

• The distilled skill is a human-readable SKILL.md with lightweight pattern scanners, supplied to the agent as prompt text rather than fine-tuned weights.Source: I SKILL DISTILLATION PROCESS · p. 18
• When a training failure matches a recognizable bug shape, the pipeline proposes a generic scanner specifying a pattern to detect and a minimal edit to apply.Source: I SKILL DISTILLATION PROCESS · p. 18
• Skill updates are constrained to prevent repository-specific memorization, and proposed updates referencing the current repository are rejected.The skill contains no repository name, file path, issue ID, or gold-patch text.Source: I SKILL DISTILLATION PROCESS · p. 18
• After one pass through the training issues, the skill is frozen before evaluation on the held-out test set.Source: I SKILL DISTILLATION PROCESS · p. 19

J DETAILED ANALYSIS OF FIXING EFFECTIVENESS

• 24 unique issues were resolved collectively; mini-SWE-agent and OpenHands participated in 18 and 17 total resolutions, respectively.mini-SWE-agent had the strongest independent problem-solving capacity, with 7 total resolutions.Source: J DETAILED ANALYSIS OF FIXING… · p. 19
• Tool-related bugs have the highest fixing rate across all studied agents.Source: Figure 7 · p. 19, J DETAILED ANALYSIS OF FIXING… · p. 19

K CASE: HARNESS-SDK #362

• C5 was not specialized to this repository, and this case was one of five new held-out fail-to-pass successes.Source: K CASE: HARNESS-SDK #362 · p. 19
• The issue reports that system_prompt is not passed into structured_output, leaving a fail-to-pass spy to observe system_prompt=None.Source: K CASE: HARNESS-SDK #362 · p. 19
• The relevant call is in structured_output_async(); function-level localization initially misses it, and typical attempts edit unrelated code or produce no patch.Source: K CASE: HARNESS-SDK #362 · p. 19
• The generated call-site edit matches the gold patch, turns the original test green, and flips function-level localization from miss to hit.The passage reports the same sequence for the other four new successes.Source: K CASE: HARNESS-SDK #362 · p. 20