PaperlayerView exact revision on arXiv

Source-linked AI summary

Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild

Yifan Xiong, Jingyi Ge, Zhenpeng Chen, Yiling Lou

arXiv:2610.04921v1cs.SEcs.AI

TL;DR

Real-world harness test adequacy remains poorly characterized despite harness implementation bugs and reliability concerns. The paper studies this gap and proposes HarnessTester, which generates LDH tests with explicit agent-harness contract support. HarnessTester reports larger adequacy gains than general-purpose baselines and detects real-world harness bugs, including previously unknown bugs.

  • Problem

    Real-world agent harness test adequacy remains largely unexplored despite harness implementation bugs and reliability concerns.

    Source: §1 · pp. 1–2

  • Method

    HarnessTester generates tests targeting uncovered LDH code and uses explicit agent-harness contracts to construct contract-faithful test contexts.

    Source: §1 · p. 3, §6 · p. 11

  • Results

    75.95%/84.76% larger mean LDH line/branch coverage gains and 69.89% larger mean mutation-score gain than the strongest per-agent, per-metric baselines were achieved by HarnessTester.

    Source: §1 · p. 3

  • Takeaways & Limitations

    HarnessTester detected 122 real-world harness bugs, including 88 previously unknown bugs and 69 confirmed by developers.

    Source: §1 · p. 3

    The findings' generality is scoped to the studied agents, and other test-generation techniques or models may yield different results.

    Source: §9 · p. 18

Abstract

from arXiv · show

LLM-based agentic systems are emerging as a new software paradigm. Modern agents are typically composed of backbone LLMs and a surrounding harness that serves as the operational software infrastructure for agent execution. As agent harnesses grow increasingly complex, agents suffer from diverse harness implementation bugs, raising substantial reliability concerns. In this work, we conduct the first empirical study to systematically investigate the test adequacy of harness in real-world agentic systems. Our analysis reveals that agent harness remains substantially undertested. In particular, LLM-dependent harness (LDH) code, despite its critical role in processing LLM outputs and governing agent behavior, receives limited testing attention, with less than half of its lines and branches covered by existing tests. Motivated by these findings, we further propose HarnessTester, the first harness-oriented test generation technique that incorporates explicit agent-harness contract support to construct contract-faithful test setups and extensively exercise LDH code. Our evaluation shows that HarnessTester substantially outperforms state-of-the-art general-purpose test generation techniques in achieving 75.95%/84.76% larger line/branch coverage gains and 69.89% larger mutation-score gains. Furthermore, HarnessTester detects 122 real-world harness bugs in widely-used agentic systems (e.g., OpenClaw), among which, 88 bugs are previously-unknown bugs and 69 bugs have been confirmed by agent developers. These results highlight the practical effectiveness of HarnessTester in improving test adequacy and assuring the reliability of real-world agentic systems.

1 Introduction

The introduction argues that complex agent harnesses face reliability concerns, while the adequacy of their tests remains largely unexplored. It focuses on LLM-dependent harness code and introduces HarnessTester, which uses agent-harness contracts to generate tests; evaluations report higher adequacy and real-world bug detection.

1 Introduction

  • Harness test adequacy has remained largely unexplored because agent benchmarks typically assess end-to-end task success without examining internal harness exercise.The study focuses on LDH, whose processing of nondeterministic LLM outputs governs subsequent agent behavior.Source: §1 · p. 2
  • Existing tests cover less than half of LDH lines and branches, with an average mutation score of 33.60%.The study identifies contract-faithful setups across expected data structures, runtime states, and component interaction protocols as a challenge to high adequacy.Source: §1 · p. 2
  • 75.95%/84.76% larger mean LDH line/branch coverage gains and 69.89% larger mean mutation-score gain than the strongest per-agent, per-metric baselines were achieved by HarnessTester.The ablation study also shows a substantial contribution from agent-harness contract support.Source: §1 · p. 3
  • HarnessTester found 122 harness bugs in studied systems, including 88 previously unknown bugs and 69 confirmed by developers.It also detected 43 of 250 historical harness bugs, compared with 2 for the best baseline.Source: §1 · p. 3
  • HarnessTester combines harness-oriented test generation with explicit contract support to construct contract-faithful execution contexts and exercise diverse LDH code.The paper presents this as the first harness-oriented test generation technique.Source: §1 · p. 3

2 Background and Related Work

Prior agent evaluations emphasize end-to-end task success, while studies of developer testing mainly characterize testing patterns. This work instead examines white-box harness test adequacy and adds contract-aware test generation.

2 Background and Related Work

  • Agent benchmarks primarily measure end-to-end task success as black-box evaluations, whereas this work analyzes white-box harness test adequacy.The harness provides operational functions including context and memory management, workflow orchestration, and tool interfaces.Source: §2 · p. 3
  • Prior studies of agent-repository testing mainly characterize patterns such as unit- or system-level testing, unlike this work's focus on harness adequacy and LDH.This work also proposes HarnessTester and evaluates its test-adequacy and bug-detection capabilities.Source: §2 · pp. 3–4
  • General-purpose LLM-based test generators are not designed around the contract-faithful setups that the authors identify as necessary for high-adequacy LDH tests.HarnessTester adds agent-harness contract support and is reported to improve coverage and bug detection relative to general-purpose generators [3].Source: §2 · p. 4

3 Empirical Study Setting

The study examines test adequacy in real-world agent harnesses, with a specific focus on code dependent on LLM outputs. It analyzes that code through program dependencies and evaluates systems using coverage and mutation metrics.

3 Empirical Study Setting

  • The empirical study investigates test adequacy in real-world agent harnesses.The section introduces the study setting and its research questions.Source: §3 · p. 4

3.1 LLM-Dependent Harness Code

  • LLM-dependent harness (LDH) code comprises harness statements whose computation depends on LLM outputs, identified through data or control dependencies.The extraction analysis models LLM invocations as sources and propagates dependencies through assignments, accesses, parameter and return flows, and other taint-preserving transformations.Source: §3.1 · p. 4

3.2 Studied Agents

  • The study selects 10 open-source agents using literature-backed candidates, quality and test-suite criteria, and diversity across application domains and programming languages.Included systems have at least 10,000 GitHub stars, a recent official release, and a reproducible automated test suite.Source: §3.2 · p. 5

3.3 Test Adequacy Metrics

  • Adequacy is measured using line and branch coverage at project and LDH levels, plus mutation scores over LDH.Mutation score is the proportion of generated mutants killed by tests.Source: §3.3 · p. 5

4 RQ1: Overall Test Adequacy

Existing agent tests leave substantial harness code uncovered and detect relatively few injected faults in LDH. The analysis finds that adequate LDH testing requires diverse tests built on contract-faithful contexts across harness components.

4 RQ1: Overall Test Adequacy

  • LDH is less than 45% of executable lines across studied agents, but its branch share is 1.10–1.75 times its line share.The analysis attributes the concentration of decisions to logic that parses and validates responses, dispatches actions, and updates agent state.Source: §4 · p. 6
  • LDH line coverage is lower than project-level line coverage in eight of ten agents, while LDH branch coverage is comparable or higher in six and lower in four.Across the entire codebase, more than half of executable lines and branches remain uncovered on average.Source: §4 · pp. 6–7
  • 33.60% average LDH mutation score means about two-thirds of injected faults go undetected; 38.36% are unexercised and 25.92% are exercised but survive.Mutation scores range from 3.58% for RD-Agent to 70.94% for Kimi Code.Source: §4 · p. 7
  • Existing tests cover less than half of LDH lines and branches, with only 33.60% average mutation scores; average agent-code line and branch coverage is below 50% and 40%.These results reveal a gap between LDH's role in processing LLM outputs and the adequacy of its testing.Source: Table 2 · p. 6, §4 · p. 7
  • Kimi Code combines relatively high project and LDH coverage with reusable helpers, scripted interactions, and runtime fakes that support contract-faithful, varied test contexts.Its LDH line coverage reaches 92.83%, and its mutation score is 70.94%.Source: §4 · p. 7
  • High LDH adequacy requires diverse test setups that satisfy component-specific data structures, runtime states, and interaction protocols.Developers must identify the boundaries traversed by model-derived values and assemble mocks that satisfy the relevant contracts while supporting behavioral variation.Source: §4 · p. 8

5 RQ2: Semantic Test Adequacy

Coverage gaps vary substantially by LDH predicate semantics: explicitly defined, bounded, and controllable behaviors are better covered than dynamic contexts and open-ended values.

Semantic taxonomy

  • The authors sampled 700 LDH branch predicates and built a three-dimension taxonomy through open coding, with independent annotation agreement κ=0.96.The dimensions are component intention, judgment pattern, and LLM value form.Source: §5 · pp. 8–9

Semantic patterns

  • Provider I/O normalization and tool action handling comprise 29.14% and 23.57% of predicates, while presence/shape checks account for 47.14%.Predicates also process indirect context state and heterogeneous values accumulated during agent execution, not only immediate model outputs.Source: §5 · p. 9

Overall coverage

  • 25.57% of the sampled predicates are fully covered, 24.57% partially covered, and 49.86% entirely uncovered.The study distinguishes decisions never reached from decisions reached without exercising every outcome.Source: §5 · p. 10

Component intention

  • Tool action handling leads full coverage at 38.18%, versus 14.29% for orchestration, 18.39% for feedback, and 17.65% for error recovery.The authors suggest explicit tool schemas help, while other categories depend on evolving contexts and cross-component behavior.Source: §5 · p. 10

Judgment patterns

  • Policy guards reach 50.00% full coverage, while presence/shape checks, size/threshold checks, and value/pattern matching reach 22.12%, 18.69%, and 30.47%.The authors note that less directly controllable decisions can require reasoning backward through upstream inputs and states.Source: §5 · p. 10

LLM value forms

  • Tool-action payload predicates are 43.42% fully covered, versus 17.71% for generated content, 20.57% for feedback signals, and 21.71% for agent context.Generated content and runtime-evolving context or feedback are harder to enumerate and reproduce along the execution flow.Source: §5 · pp. 10–11

6 HarnessTester: Harness-Oriented Test Generation

HarnessTester uses coverage-driven test generation that explicitly accounts for agent-harness contracts, helping construct valid test contexts and exercise uncovered LLM-dependent harness code.

Coverage-driven generation

  • HarnessTester iteratively targets LDH code uncovered by existing tests and augments general LLM-based generation with agent-harness contract support.Contracts specify requirements such as expected data structures, execution preconditions, and interaction protocols among harness components.Source: §6 · p. 11

Contract guidance

  • Prompt guidance steers tests to preserve runtime representations, required histories, cross-component calling conventions, and contract-faithful test doubles.It also encourages distinct behavioral partitions rather than more variants of already exercised behavior.Source: §6 · pp. 11–12

Context retrieval

  • An on-demand retriever supplies repository-specific definitions and context needed to construct or control valid test setups.It resolves requested symbols with AST-based matching and bounded extraction, exposing fields and types for contract-faithful mocks.Source: §6 · p. 12

7 RQ3: Test Adequacy Improvement

HarnessTester outperforms general-purpose test-generation baselines across LDH and project-wide coverage, semantic coverage gains, and mutation detection; its contract support contributes substantially to these gains.

Evaluation setup

  • The evaluation compares HarnessTester with four general-purpose LLM test-generation techniques and a contract-agnostic variant across project-wide and scoped LDH-wide settings.All techniques use GPT-5-mini and receive a 120-minute timed budget per project.Source: §7 · pp. 12–13

LDH-wide coverage

  • 82.27% and 94.36% larger LDH line- and branch-coverage gains than the strongest scoped baselines, respectively, at 120 minutes.HarnessTester has the highest final LDH-wide gains among studied baselines; branch gains lead from 30 minutes and line gains from 75 minutes.Source: §7 · pp. 13–14, Figure 3 · p. 13

Contract-support ablation

  • 96.51% larger mean line-coverage gains and 96.24% larger branch-coverage gains than the contract-agnostic variant.On OpenHands, removing contract guidance or the context retriever also reduces LDH line/branch gains by 43.26%/45.98% and 37.38%/41.00%, respectively.Source: Figure 3 · p. 13, §7 · p. 14

Semantic coverage

  • Across semantic categories, 12.12–25.49% of predicates move from uncovered to fully covered, and another 2.86–10.87% move from partially to fully covered.For Agent Orchestration Control, 22.45% move from uncovered to fully covered and 5.10% from partially to fully covered.Source: §7 · p. 14, Figure 5 · p. 14

Project-wide coverage

  • 13.90 and 17.30 percentage-point macro-averaged project-wide line- and branch-coverage gains exceed baseline gains of 10.36 and 11.58 points.HarnessTester has the highest project-level gains on eight of ten agents for line coverage and nine of ten for branch coverage.Source: Figure 4 · p. 13, §7 · p. 14

Mutation detection

  • HarnessTester achieves the highest mutation-score gain on every studied agent, including 43.01 points on SWE-agent versus 22.15 for CoverUp (Scoped).On OpenClaw, a 0.39-point gain corresponds to 185 additional mutant kills, compared with 147 for Qodo Cover (Scoped) and 141 for the contract-agnostic variant.Source: §7 · p. 15, Figure 6 · p. 15

8 RQ4: Real-World Harness Bug Detection

HarnessTester is evaluated on historical and newly discovered real-world harness bugs, with tests assessed for whether they expose bugs under contract-faithful setups. It detects substantially more historical bugs than the best baseline and uncovers developer-confirmed previously unknown bugs.

8.1 RQ 4.a: Detecting Historical Harness Bugs

  • H-Bench contains 250 historical bugs across 10 systems, selected from fixes involving LDH code and not exposable by tests in the buggy revision.The benchmark includes 140 bugs from seven Python agents and 110 from three TypeScript agents.Source: §8.1 · pp. 15–16
  • Historical-bug evaluation hides patches and bug descriptions from generators, and counts a detection only when a test fails on the buggy version but passes on the fixed version.Manual review checks whether plausible detections match the developer-fixed bug or are false positives.Source: §8.1 · p. 16
  • HarnessTester detects 43 historical bugs (17.20%), versus 2 (0.80%) for the best baseline.HarnessTester cost $77.58 across 250 cases, or $1.80 per revealed bug, compared with $10.84 per bug for the baseline.Source: §8.1 · p. 16, Table 4 · p. 16
  • Removing contract support reduced historical bugs detected from 43 (17.20%) to 15 (6.00%) and increased false positives from 14.00% to 25.00%.Reviewed false positives included fixtures violating production contracts or assuming unsupported behavior.Source: §8.1 · p. 16

8.2 RQ 4.b: Detecting Previously-Unknown Harness Bugs

  • The previously unknown-bug evaluation targets 100 LDH-containing functions per codebase and manually inspects failures before reporting new bugs to developers.OpenHands was represented by its Agent Canvas and SDK components following a refactor.Source: §8.2 · p. 16
  • HarnessTester identifies 122 true harness bugs, including 88 previously unknown bugs, 69 of which developers confirmed.Of the 88 previously unknown bugs, 64 were fixed or assigned an open fix, and one received a CVE number.Source: §8.2 · pp. 17–18, Table 5 · p. 17
  • A Browser Use test exposed nested-list sensitive data left unredacted in saved history; neither CoverUp nor the contract-agnostic variant found this bug.The test used schema-valid objects and expected sensitive-data mappings, then inspected the saved file; upstream fixed the bug with recursive redaction.Source: §8.2 · pp. 17–18

9 Threats to Validity

The authors scope external validity to the studied agents and acknowledge that other test-generation techniques or models may produce different results.

External Validity

  • Findings generalize only to the studied agents; other test-generation techniques or models may yield different results.The study selected diverse, actively developed systems across application domains, languages, and architectures, and releases its benchmark and implementation for further evaluation.Source: §9 · p. 18

Internal Validity

  • RQ2 branch-pattern labeling may reflect human judgment, though two annotators achieved κ=0.96 and resolved disagreements with a third reviewer..Source: §9 · p. 18
  • Metric calculations and implementation may contain errors; the authors checked intermediate results, inspected representative cases, and used established tools where applicable.Source: §9 · p. 18

10 Conclusion

The paper studies test adequacy in real-world agent harnesses, finds substantial gaps in testing LLM-dependent code, and proposes HarnessTester to generate contract-faithful tests. Its evaluation reports stronger coverage and mutation-score gains than general-purpose test generation, plus detection of 122 real-world harness bugs.

10 Conclusion

  • The first real-world study of agent-harness test adequacy finds substantial gaps in testing critical LLM-dependent harness code.The gaps are especially pronounced when tests require dynamic execution contexts, implicit input spaces, and cross-component interactions.Source: §10 · p. 18
  • HarnessTester incorporates agent-harness contract support to construct contract-faithful tests for LLM-dependent harness code.Source: §10 · p. 18
  • HarnessTester outperforms general-purpose test generation in coverage and mutation-score gains, with contract support contributing significantly to the improvements.Source: §10 · p. 18
  • HarnessTester detects 122 real-world harness bugs, including 88 previously unknown bugs, 69 confirmed by developers.Source: §10 · p. 18
Loading 2610.04921v1…