PaperlayerView exact revision on arXiv

Source-linked AI summary

From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws

Mengzhuo Chen, Junjie Wang, Zhe Liu, Yawen Wang, Haiming Zheng, Qing Wang

arXiv:2606.06324v2cs.SEcs.MA

TL;DR

Existing approaches often improve observed outcomes without diagnosing which trajectory evidence and harness mechanisms caused failures. HarnessFix structures traces and harness artifacts for diagnosis, maps flaw records to scoped repairs, and improves performance across four benchmarks by 6.3%–18.4% over initial harnesses while significantly outperforming human-designed and self-evolution baselines.

  • Problem

    Existing self-improvement and harness-evolution methods often do not diagnose responsible trajectory evidence or the concrete harness mechanism causing failure, limiting repair scope.

  • Method

    HarnessFix compiles traces and harness artifacts into HTIR, attributes failures to responsible steps and artifacts, then generates constrained patches through scoped repair operators and flaw-specific specifications.

  • Results

    6.3%–18.4%: HarnessFix improves performance over initial harnesses across four benchmarks and significantly outperforms human-designed and self-evolution baselines.

  • Takeaways & Limitations

    Failed trajectories can serve as structured evidence for diagnosing and repairing the harness mechanisms behind agent failures.

Abstract

from arXiv · show

LLM agents increasingly rely on agent harness: the runtime infrastructure around the base model that defines execution environments, tool interfaces, context, lifecycle orchestration, observability, verification, and governance. Existing self-improving agents and automatic harness evolution methods mainly improve agents through runtime supervision, prompt optimization, workflow search, or harness modification based on final outcomes. However, they often fail to diagnose where the responsible evidence lies in failed trajectories and which harness implementation mechanism causes the unreliable behavior, resulting in broad, indirect, or poorly scoped changes. This paper proposes HarnessFix, a trace-grounded and diagnosis-driven framework for repairing agent harnesses. HarnessFix compiles raw execution traces and harness artifacts into a Harness-aware Trace Intermediate Representation (HTIR), which normalizes fragmented trajectory evidence, captures step-level data-flow and control-flow relations, and aligns runtime steps with the harness artifacts that shape their behavior. It then attributes failures to responsible steps and harness artifacts, and consolidates recurring diagnoses into repair-oriented flaw records. Finally, HarnessFix maps these records to scoped repair operators, generates patches under flaw-specific repair specifications, and accepts them through regression-aware validation. We evaluate HarnessFix on four popular benchmarks, and results show that it improves the performance over the initial harnesses by 6.3% to 18.4%, significantly outperforming human-designed and self-evolution baselines. HarnessFix highlights the value of treating failed trajectories not only as feedback signals, but also as structured evidence for diagnosing and repairing the harness mechanisms behind agent failures.

I. INTRODUCTION

LLM-agent reliability depends on the harness surrounding the base model, yet failures are difficult to localize because runtime evidence and implementation mechanisms are fragmented. HarnessFix addresses this gap by grounding diagnosis and scoped repair in structured trajectory evidence, improving benchmark performance over initial harnesses.

  • Motivation: Agent reliability depends on runtime infrastructure governing tools, context, execution flow, state, and other harness responsibilities.The harness complements the base model by shaping how agents interact with external environments.
  • Motivation: Failures emerge in interleaved execution trajectories and cannot usually be mapped directly to specific harness source code, prompts, or other artifacts.The paper identifies fragmented evidence and implicit runtime dependencies as central diagnostic challenges.
  • Approach: HarnessFix builds HTIR to normalize heterogeneous trajectories, reconstruct step-level data-flow and control-flow links, and align runtime evidence with harness artifacts.This representation supports failure attribution, root-cause explanation, and recurring flaw records.
  • Approach: HarnessFix maps diagnosed flaw records to scoped repair operators and flaw-specific specifications for constrained patch generation and regression-aware validation.Operator selection uses implicated layers, root causes, diagnostic evidence, and the targeted TraceStep.
  • Empirical foundation: The empirical study finds harness flaws across all seven ETCLOVG responsibilities, motivating multi-layer repair operators rather than prompt-only changes.The study analyzes 30 repositories and approximately 57,780 development records.
  • Evaluation: 6.3%–18.4%: HarnessFix improves performance over initial harnesses across GAIA, SWE-Bench Verified, AppWorld, and Terminal-Bench 2.0 Verified.It also significantly outperforms human-designed and self-evolution baselines.

II. BACKGROUND AND MOTIVATION

The paper frames the agent harness as editable runtime infrastructure around the base model and studies how harness flaws arise in real-world systems. Harness-related development is common, and flaws span the full harness taxonomy.

  • A. Background: An agent harness is the runtime infrastructure surrounding the base model, providing tools, context, orchestration logic, and related mechanisms.In this paper, the harness is distinct from the model parameters.
  • A. Background: Harness repair modifies prompt templates, tool specifications, orchestration code, configurations, adapters, logging hooks, and verification scripts.Repair targets editable implementation artifacts rather than base-model parameters.
  • A. Background: The ETCLOVG taxonomy organizes harness responsibilities into seven layers used to connect runtime evidence with mechanisms needing repair.The taxonomy supplies the paper’s vocabulary for analyzing harness flaws.
  • B. Data Collection: 45.3%: harness-related records account for approximately this share of 57,780 development records, and all 30 repositories contain such changes.The records include issues, pull requests, commits, and release notes from popular open-source agents.

2) Distribution of Harness Flaws:

HARNESSFIX represents runtime behavior and aligns it with harness artifacts to support scoped repair. Its repair operators are informed by observed flaw distributions across harness responsibilities.

  • Flaw distribution: 29 out of 30 analyzed agents exhibited flaws related to Execution, Tooling, and Observability.The dominant flaw layer varies by repository focus: Lifecycle is more common in long-horizon systems, while Verification is more common in benchmark-facing harnesses.
  • Scoped repair: Repair operators organize common harness repair actions by layer and constrain edits to reduce unstable changes or broad regressions.HARNESSFIX uses these operators to guide scoped harness repair from trace-grounded diagnoses.
  • Runtime and harness modeling: HARNESSFIX models trajectories as TraceSteps and links them to harness implementations through implementation anchors.This alignment supports failure attribution and localization of editable artifacts.

2) Data-flow Alignment: Linking Evidence Propagation to Harness Logic:

HARNESSFIX reconstructs how information propagates between runtime steps and how harness logic transitions execution. These links expose evidence loss and incorrect controller decisions behind failures.

  • Data-flow alignment: Data-flow links explain how information entered, disappeared from, or was transformed in later model-facing requests.They inspect earlier steps and request-construction logic, including memory assembly, tool descriptions, and prompt components.
  • Data-flow alignment: Each data-flow link records source and target steps, precise source and target spans, and a reuse relation such as copied, summarized, or semantically reused.These records provide concrete evidence attribution across trajectory steps.
  • Control-flow alignment: Control-flow links record harness transitions such as continue, retry, validate, finalize, or terminate, together with triggering conditions when available.They complement data-flow links by explaining step-transition logic.
  • Control-flow alignment: A success status with no observable external effect incorrectly enabled finalization in the AppWorld completion-guard example.The completion guard treated status=success as evidence of progress, allowing complete_task() despite the absent state change.

4) Implementation Anchors: Localizing Runtime Evidence in Harness Artifacts:

Implementation anchors connect runtime evidence and cross-step relations to the concrete harness artifacts that realize or govern them. This makes diagnosis actionable at the artifact level.

  • Implementation anchors: Implementation anchors localize editable harness artifacts associated with TraceSteps, data-flow links, and control-flow links.They connect runtime behavior to the implementation mechanisms responsible for that behavior.
  • Implementation anchors: HARNESSFIX uses runtime signals such as tool names, prompt templates, adapter invocations, validator events, and controller actions to locate artifacts.When behavior spans components, it follows established data-flow and control-flow links.
  • Implementation anchors: The S3–S4 data-flow link is grounded in the prompt-rendering artifact that exposes S3 API documentation and constructs S4 input.The example identifies prompts/react_code_agent/instructions.txt and the simplified agent’s initialize() routine.

5) Harness-layer Responsibility Mapping:

HTIR maps anchored runtime evidence to harness responsibility layers, enabling recurring flaw consolidation and repair planning. Attribution proceeds from failure symptoms through linked evidence to responsible steps and layers.

  • Harness-layer mapping: Responsibility mapping uses TraceSteps, associated links, implementation anchors, and ETCLOVG criteria rather than static file-level classification.A TraceStep may map to multiple layers when its anchored runtime evidence involves multiple responsibilities.
  • Harness-layer mapping: In the AppWorld example, S6 maps to Lifecycle and Verification because it finalizes after status=success despite an omitted required field and absent state effects.The S3–S4 data-flow evidence connects the premature finalization to the upstream request-construction error.
  • Failure attribution: Failure attribution identifies responsible TraceSteps, determines the root cause, and links the failure to specific harness layers.The diagnosis process begins with symptom localization and backtracks through data-flow and control-flow links to rank candidate steps.
  • Diagnosis records: Each diagnosis record contains an attribution result, failure signal, and harness diagnosis describing implicated ETCLOVG layers and the harness flaw.These records provide the structured basis for later consolidation and repair.
  • Flaw consolidation: HARNESSFIX merges diagnosis records with overlapping layers and matching root-cause evidence into recurring flaw records before modifying the harness.Each flaw record summarizes the recurring flaw and the supporting diagnosis records used to establish it.

C. Scoped Repair

HARNESSFIX constrains harness modifications by mapping diagnosed flaw records to scoped repair operators and flaw-specific repair specifications. These specifications bind edits to implicated artifacts and required post-repair behavior.

  • HARNESSFIX maps each flaw record to applicable repair operators using implicated harness layers, root causes, and diagnostic evidence.
  • It designates a primary operator targeting the implicated TraceStep, while auxiliary operators expose evidence, enforce constraints, or modify adjacent mechanisms.
  • Repair specifications convert operator intent into flaw-specific implementation contracts for the target harness.
  • The specification binds target scope to the flaw record, implicated layers, selected operators, and representative diagnoses.
  • Edit constraints identify concrete editable artifacts and forbidden artifacts in the target implementation.
  • Required behavior states the harness behavior that must hold after applying the candidate change, such as evidence requirements before completion.

D. Patch Validation and Harness Memory

HARNESSFIX validates candidate patches for specification compliance, implementation correctness, target-flaw improvement, and regression control. It records accepted and rejected outcomes as harness repair memory for future repairs.

  • Patch Validation: HARNESSFIX performs pre-validation checks for repair-specification conformity, syntax, and static correctness.
  • Patch Validation: A patch is accepted only when it mitigates the target flaw and remains within the regression limit on previously solved validation tasks.
  • Harness Memory: Repair memory stores flaw diagnoses, repair specifications, diff summaries, validation results, regressions, and applicability conditions.
  • Harness Memory: Rejected records identify pre-validation failure, insufficient target improvement, or excessive regression to discourage repeating ineffective changes.
  • Evaluation Questions: The evaluation asks whether HARNESSFIX improves task performance and token efficiency, diagnosis, component choices, and cross-model transfer.
  • Evaluation Setup: Experiments cover four representative agent benchmarks using initial harnesses and training, validation, and held-out test splits.

C. Baselines

HARNESSFIX is evaluated against human-designed harnesses and automated self-evolution or repair methods under a common experimental setup. Across benchmarks and models, it improves over initial harnesses and competing baselines.

  • Baselines: RQ1 compares HARNESSFIX with human-designed harnesses and self-evolution or repair baselines.
  • Baselines: The human-designed comparison includes representative harnesses selected separately for GAIA, SWE-Bench Verified, AppWorld, and Terminal-Bench 2.0 Verified.
  • Baselines: Automated baselines include GEPA, SCOPE, ReCreate, and Meta-Harness, with token consumption recorded for these approaches.
  • Experimental Setup: Experiments use GPT-5 mini by default, compare four additional LLMs, and average task-performance values over three independent runs.
  • Evaluation Design: The diagnosis evaluation uses human-annotated failed test trajectories as gold labels for responsible steps, causes, anchors, layers, and operators.
  • Evaluation Design: Ablations remove prompt-only restrictions, trace-grounded diagnosis, scoped operators, or regression-aware acceptance to test HARNESSFIX’s design choices.
  • Results: HARNESSFIX consistently improves over H0 across four benchmarks and five LLMs, with an average gain of 11.1%.GAIA gains range from 14.5% to 18.4%, while AppWorld gains range from 5.9% to 9.3%.
  • Results: HARNESSFIX outperforms human-designed harnesses by 6.3% on average and automated self-evolution baselines by 6.9% on average.

B. Failure Diagnosis (RQ2)

Full HTIR provides accurate diagnosis across step, implementation-anchor, harness-layer, and repair-operator dimensions. Ablations show that removing core diagnosis or scoped-repair choices reduces performance, emphasizing targeted runtime changes.

  • Failure Diagnosis: Full HTIR achieves 85.0% step accuracy, 81.3% implementation-anchor accuracy, 86.2% harness-layer macro-F1, and 82.5% repair-operator accuracy.
  • Failure Diagnosis: HTIR’s fine-grained step localization exceeds existing techniques that typically achieve around 45–52% step accuracy.
  • Ablation Study: All ablations reduce performance relative to full HARNESSFIX, indicating that each evaluated design choice contributes to the final result.
  • Ablation Study: The largest degradation occurs with prompt-only repair or without scoped repair operators, highlighting the importance of targeted runtime changes.

D. Cross-model Transfer (RQ4)

HARNESSFIX’s repairs transfer across models and address diverse, benchmark-specific harness flaws through diagnosis-driven, scoped edits. The framework links recurring failures across harness layers to repair specifications and validation rather than relying on exhaustive search.

  • Cross-model transfer: 18.4% improvement for the repair-source model and 5.5%–9.5% gains for four target models show consistent cross-model transfer on GAIA.The target-model gains are smaller than the source-model gain but remain consistently positive.
  • Repair scope: HARNESSFIX covers diverse harness flaws with scoped edits, whereas most baselines modify only one or two harness layers.Meta-Harness can modify all seven layers but uses untargeted optimization and incurs higher token costs.
  • Benchmark-specific flaws: GAIA repairs target Tool Interface and Observability issues such as missing API configuration, unsupported formats, fragile media conversion, and insufficient trace evidence.
  • Benchmark-specific flaws: AppWorld failures combine Tool Interface, Lifecycle, Verification, and Governance concerns around API validation, persistent state effects, and premature completion.The harness must track artifact and state effects while preventing acceptance before required side effects occur.
  • Cross-layer implications: Different benchmarks require different runtime mechanisms, and lifecycle repairs can alter verification evidence and finalization conditions across layers.HARNESSFIX aggregates implicated layers around recurring flaws to guide repair specifications and validation.
Loading 2606.06324v2…