PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Harness: Harnesses That Improve Themselves

Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, Shuyue Hu

arXiv:2606.09498v3cs.CL

TL;DR

Agent performance depends on both base models and the harnesses mediating their environmental interaction, yet manually tailoring harnesses does not scale across diverse, rapidly changing models. Self-Harness lets an agent mine its execution failures, propose targeted harness edits, and validate them by regression testing; across nine model–benchmark combinations, every final harness improves held-in and held-out performance, with relative gains up to 132%.

  • Problem

    Different models have distinct behaviors and failure modes, while manually redesigning model-specific harnesses becomes increasingly costly as models rapidly evolve.

  • Method

    Self-Harness iteratively mines clustered execution traces, proposes diverse minimal edits tied to failure mechanisms, and accepts candidates only after regression testing.

  • Results

    Every final harness improves held-in and held-out Pass across all nine model–benchmark combinations, with relative improvements up to 132%.

  • Takeaways & Limitations

    Self-Harness can turn model-specific weaknesses into auditable harness changes that address benchmark-specific bottlenecks in artifact handling, patch verification, and application-state retrieval.

  • Takeaways & Limitations

    The study evaluates bounded edits under fixed benchmarks, and accepted edits may reflect benchmark-specific failure patterns while relying on verifier outcomes and trace quality.

Abstract

from arXiv · show

The performance of LLM-based agents is jointly shaped by their base models and the harnesses that mediate their interaction with the environment. Because different models exhibit distinct behaviors, effective harness design is inherently model-specific. Yet agent harnesses are still largely engineered by human experts, a paradigm that scales poorly as modern LLMs become increasingly diverse and rapidly evolving. In this paper, we introduce Self-Harness, a new paradigm in which an LLM-based agent improves its own operating harness, without relying on human engineers or stronger external agents. We operationalize Self-Harness as an iterative loop with three stages: Weakness Mining, which identifies model-specific failure patterns from execution traces; Harness Proposal, which generates diverse yet minimal harness modifications tied to these failures; and Proposal Validation, which accepts candidate edits only after regression testing. We instantiate Self-Harness across Terminal-Bench-2.0, SWE-bench Verified, and AppWorld using a minimal initial harness and three base models from diverse families: MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5. Across all nine model--benchmark combinations, every final harness improves both held-in and held-out pass rates, with overall relative gains of up to 132%. Qualitative analyses further show that the retained mechanisms address benchmark-specific bottlenecks in artifact handling and runtime control, software-patch verification, and application-state retrieval. These results suggest a path toward LLM-based agents that are not merely shaped by their harnesses, but can also participate in reshaping them.

1. Introduction

Self-Harness addresses the difficulty of manually tailoring agent harnesses to rapidly evolving, behaviorally diverse models by having an agent improve its own harness. It mines execution failures, proposes targeted edits, validates them by regression testing, and improves held-in and held-out performance across evaluated model–benchmark combinations.

  • Different models exhibit distinct behavioral patterns and tool-use habits, making manually tuning a model-specific harness increasingly costly and untenable.
  • Self-Harness enables an LLM-based agent to improve its own operating harness without human engineering or stronger external-agent guidance.
  • Self-Harness iteratively mines clustered execution traces, generates diverse minimal edits tied to failure mechanisms, and validates candidates through regression testing.
  • Every final harness improves held-in and held-out pass rates across nine model–benchmark combinations, with relative improvements up to 132%.The reported overall Pass gains reach up to 40.6 percentage points, and held-out improvements remain substantial on Terminal-Bench-2.0.
  • Qualitative analyses find model-specific interventions addressing artifact creation, structured outputs, tool-use loops, dependency checks, environment preservation, and implementation speed.The retained mechanisms can also include subagent-based decomposition and middleware creation.

2. Background and Related Work

Prior work treats harnesses as a broad control layer around the model and explores self-improving inputs, workflows, and externally optimized agent designs. Self-Harness is positioned within these lines of work while focusing on an agent improving its own harness.

  • Agent harnesses include prompts, tools, memory, verification rules, permissions, adapters, and runtime mechanisms that mediate between models and environments.
  • Harness-layer failures can involve unverified artifacts, unproductive retries, lost sources of truth, or missing recovery actions.
  • Self-improving-agent research adapts inputs, memories, contexts, or workflows over time through methods such as Reflexion, agentic context engineering, and STOP.
  • Automated agent-design systems and Meta-Harness optimize agent structures or harness code externally using designs, scores, and execution traces.
  • Self-Harness is closest to self-improvement and automated harness-optimization research while studying harness improvement internalized within the target agent.

3. Self-Harness: An Iterative Loop for Model-Specific Harness Improvement

Self-Harness iteratively improves a fixed model’s surrounding execution harness using verifier-grounded failure evidence. It proposes bounded, model-specific edits and retains only candidates that pass held-in and held-out regression checks.

  • Loop overview: A fixed model iteratively improves its surrounding harness through an explicit evidence-driven loop rather than external optimization.The evaluation system mines recurring failures, while the same model proposes harness changes under the current harness.
  • Weakness Mining: Failures are converted into structured, verifier-grounded patterns by analyzing terminal causes, agent behavior, and reusable mechanisms.Deterministic clustering groups cases only when these signature components agree, avoiding aggregation based solely on superficial outcomes.
  • Weakness Mining: The evidence bundle summarizes dominant recurring weaknesses without prescribing an edit, separating verifier outcomes from agent-level mechanisms.Clusters are ordered by support and estimated actionability so the proposer sees potentially valuable interventions first.
  • Harness Proposal: The proposer generates diverse candidate bundles, but each edit must target an addressable mechanism, modify only the necessary harness surface, and preserve unrelated behavior.Candidates include audit records describing their targeted failure, edited surface, expected effect, and regression risks.
  • Proposal Validation: Candidates are tested on held-in and held-out splits, and an edit is accepted only if it improves at least one split without degrading the other.Held-in evaluation tests the motivating evidence, while held-out evaluation checks for regressions on unseen behaviors.

4. Experiments

Experiments evaluate Self-Harness across three benchmarks and three fixed model backends, varying only the harness. Final harnesses improve held-in and held-out performance while retaining model-specific workflow interventions.

  • Experimental Setup: Self-Harness is evaluated on Terminal-Bench-2.0, SWE-bench Verified, and AppWorld across MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5.The evaluation uses minimal benchmark-specific harnesses and fixed model, tool, budget, environment, and evaluator settings.
  • Main Results: Every final harness improves Pass on both held-in and held-out splits across all nine model–benchmark combinations.The largest relative gain is 132% for Qwen3.5-35B-A3B on AppWorld; the largest absolute gain is 40.6 percentage points for GLM-5 on AppWorld, from 44.4% to 85.0%.
  • Main Results: Harness-level edits produce measurable gains while the model backend, tool set, budget, benchmark environment, and evaluator remain fixed.No promoted harness degrades either held-in or held-out performance, supporting regression testing against split-specific overfitting.
  • Harness Evolution: Terminal-Bench-2.0 edits primarily target artifact handling, failure recovery, and runtime control, with mechanisms differing by model.MiniMax emphasizes early artifact creation and bounded execution; Qwen emphasizes dependency checks, loop breaking, retry discipline, and recovery; GLM-5 emphasizes persistent environment changes and implementation-oriented exploration.
  • Harness Evolution: The retained edits alter concrete execution workflows, including earlier artifact creation, recovery from failed actions, persistent environment changes, and explicit verification before completion.Trace examples associate these workflow changes with the quantitative gains reported in Table 1.

5. Conclusion

Self-Harness frames harness improvement as evidence-driven, bounded state transitions that are evaluated under fixed protocols. Across three benchmarks and three model backends, it improves pass rates while retaining auditable edits, but remains limited to bounded, benchmark-dependent settings.

  • Conclusion: Self-Harness evaluates a fixed model under its current harness, uses execution-trace evidence to propose bounded edits, and promotes only non-regressive candidates.The model, evaluator, and benchmark protocol remain fixed while harnesses evolve through tested transitions.
  • Conclusion: Across nine model–benchmark combinations, Self-Harness improves Pass (%) while preserving held-in and held-out performance under the acceptance rule.The experiments use Terminal-Bench-2.0, SWE-bench Verified, and AppWorld with three model backends.
  • Conclusion: Retained edits are small, auditable changes targeting artifact handling and runtime control, software-patch verification, and application-state retrieval.These bottlenecks are addressed through configurable harness surfaces constrained by execution evidence and regression testing.
  • Limitations: Self-Harness studies bounded harness edits under fixed benchmarks rather than open-ended self-improvement, and accepted edits may reflect benchmark-specific failure patterns.The protocol also depends on verifier outcomes and trace records.
  • Limitations: Higher-stakes harness changes require stronger acceptance gates than pass-rate non-regression alone.
  • Conclusion: The paper points toward harnesses evolving through recorded, testable, and reversible changes grounded in behavioral evidence.The proposed engineering style prioritizes evidence over a proposer’s rationale for a plausible edit.

A.1. Model Inference Services

The experiments use hosted inference services for MiniMax M2.5 and GLM-5, while Qwen3.5-35B-A3B runs locally on four NVIDIA H200 GPUs.

  • A.1. Model Inference Services: MiniMax M2.5 and GLM-5 use hosted inference services through MiniMax’s hosted API and OpenRouter, respectively.
  • A.1. Model Inference Services: Qwen3.5-35B-A3B is deployed locally on four NVIDIA H200 GPUs using an internal image derived from a public SGLang Docker image.

A.2. Benchmark Configurations

The benchmark configurations keep evaluation settings fixed within comparisons while specifying task splits, environments, evaluators, and runtime constraints for each benchmark.

  • A.2. Benchmark Configurations: Each candidate evaluation uses two repeated attempts, with task sets, model backends, benchmark environments, evaluators, and non-editable runtime settings fixed within comparisons.
  • A.2. Benchmark Configurations: Terminal-Bench-2.0 uses Harbor3, a 2 MB/s outbound bandwidth cap, and mirrored stable external resources to reduce incidental network-latency failures.Unreliably accessible external-resource tasks are excluded from the described configuration.
  • A.2. Benchmark Configurations: SWE-bench Verified uses a fixed 100-case subset split into 67 held-in and 33 held-out cases, scored by the official evaluator in isolated Docker instances.
  • A.2. Benchmark Configurations: AppWorld uses a fixed 180-example set with 90 held-in and 90 held-out examples, executing each task in a fresh isolated context with the official evaluator.

B.1. Terminal-Bench-2.0

The Terminal-Bench-2.0 case studies illustrate how edited harnesses address concrete execution failures. They show artifact restoration and validation for Qwen3.5, plus bounded operations, archive checks, and sanity-check repair for GLM-5.

  • B.1. Terminal-Bench-2.0: Qwen3.5’s edited harness restores the missing extractor and validates its output on the extract-elf task.
  • B.1. Terminal-Bench-2.0: The case studies contrast failed initial-harness traces with edited-harness traces that produce, restore, or verify required task artifacts and outputs.The MiniMax count-dataset-tokens example is described in the supplied Figure 7 caption.
  • B.1. Terminal-Bench-2.0: GLM-5’s successful build-pov-ray trace replaces long monolithic downloads with bounded staged operations and checks archive evidence before further work.The edited trace also repairs a failed sanity check before finalizing.

B.2. SWE-bench Verified

Across SWE-bench Verified, Self-Harness improves each model’s overall pass rate through model-specific verification mechanisms. The retained harnesses address incomplete patch checking, test execution, and dependency recovery.

  • MiniMax M2.5: 46.0% to 52.5% overall Pass for MiniMax M2.5, using separate empty-diff detection and targeted local testing.The promoted harness uses diff inspection to check whether a patch exists and targeted testing to verify the relevant failing test.
  • Qwen3.5-35B-A3B: 19.5% to 41.5% overall Pass for Qwen3.5-35B-A3B, with patch verification and direct pre-submission checks.The retained harness inspects the current diff, runs targeted tests, and places the SQLite casting guard at the inner window-compatible expression.
  • GLM-5: 52.0% to 55.5% overall Pass for GLM-5, after adding dependency-aware recovery to the FAIL-TO-PASS verification cycle.The harness parses import failures, installs missing packages, reruns the exact test, and continues debugging when necessary.
  • GLM-5: The GLM-5 Astropy trace improves from incomplete syntax-only checking to executable verification, passing all four FAIL-TO-PASS and 68 PASS-TO-PASS tests.The initial harness leaves four FAIL-TO-PASS and five PASS-TO-PASS tests failing; the promoted harness repairs dependencies and completes verification.

B.3. AppWorld

Across AppWorld, Self-Harness improves pass rates by retaining mechanisms for exhaustive state retrieval, temporal reasoning, and correct action-versus-information completion semantics. The mechanisms differ by model but address recurring application-state bottlenecks.

  • MiniMax M2.5: 48.6% to 58.9% overall Pass for MiniMax M2.5, using state auditing with pagination and temporal-boundary guidance.In the Venmo case, the promoted harness identifies four participants who already paid and requests payment from only the remaining three.
  • Qwen3.5-35B-A3B: 22.5% to 52.2% overall Pass for Qwen3.5-35B-A3B, with completion-contract and pagination guards.The Spotify trace shows complete library retrieval, play-count comparison, and return of the expected song instead of a null result.
  • GLM-5: 44.4% to 85.0% overall Pass for GLM-5, after adding action-versus-information completion rules and exhaustive pagination before state mutation.In the Venmo case, the promoted harness exhausts all pages, mutates all 12 target records, and returns a null answer for the action-only task.
  • Cross-model pattern: AppWorld mechanisms converge on complete state retrieval and correct completion semantics through model-specific harness designs.The retained mechanisms include state auditing, pagination, temporal boundaries, completion contracts, target-object checks, and pre-mutation retrieval requirements.
  • Terminal-Bench-2.0: 46.1% to 57.0% overall Pass for GLM-5 on Terminal-Bench-2.0, with retained edits for persistent environment configuration and the exploration-to-implementation transition.The figure marks accepted candidates in green, rejected candidates with gray crosses, and explored branch endpoints with red crosses.
  • SWE-bench Verified: SWE-bench Verified figures show overall Pass increases of 46.0% to 52.5% for MiniMax M2.5 and 19.5% to 41.5% for Qwen3.5-35B-A3B.The retained edits separate empty-diff detection from targeted testing for MiniMax and add patch verification with pre-submission checks for Qwen.
Loading 2606.09498v3…