Source-linked AI summary
Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification
Jinghan Xu, Yikai Zhang, Aili Chen, Weiyuan Li, Jiaqing Liang, Deqing Yang
TL;DR
Existing harness-evolution methods often verify every candidate on fixed task sets, making it difficult to focus evaluation on affected behaviors under limited budgets. HarnessLens explores tasks and configurable components, derives modifications from trajectories, and selectively verifies them with attributable evidence. Across three harnesses and four benchmarks, it reports 7.6–13.6% average performance improvements with substantially less evaluation budget.
Problem
Existing propose-and-verify methods use the same evaluation tasks for different modifications, wasting rollouts on unrelated behaviors and potentially obscuring intended behavioral changes or regressions.
Method
HarnessLens explores tasks and configurable components, diagnoses trajectories, proposes modifications, and verifies candidates on behavior-relevant tasks using attributable behavioral evidence.
Results
7.6–13.6% average performance improvements are reported across three harnesses and four benchmarks with a substantially smaller joint rollout-and-analysis budget.
Takeaways & Limitations
Reliable harness evolution depends on directing verification toward the behaviors each modification is intended to change, not only on increasing verification quantity.
Takeaways & Limitations
The evaluation covers one model family, three harnesses, and four public benchmarks, and does not normalize token usage, latency, or monetary cost across roles and benchmarks.
Abstract
from arXiv · showhide
Agent harnesses shape how language-model agents use instructions, tools, and runtime components, but adapting these harnesses requires costly verification. Existing propose-and-verify methods typically score every candidate on a fixed task set, wasting rollouts on unrelated behaviors and allowing aggregate scores to obscure specific regressions. We introduce HarnessLens, a budget-aware framework for automated harness evolution. HarnessLens jointly explores the task space and user-configurable components, derives candidate modifications from execution trajectories, and selectively verifies each candidate on behavior-relevant tasks using an attributable-evidence gate. Across three agent harnesses and four benchmarks, HarnessLens improves average held-out performance by 7.6-13.6% while consuming substantially less evaluation budget than competing baselines. These results demonstrate that behavior-aware verification with explicit attribution enables more reliable and sample-efficient harness evolution under constrained interaction budgets. Our code is available at https://github.com/jhxu5214/HarnessLens.
1 Introduction
HarnessLens addresses the limitations of fixed-task verification by evolving agent harnesses with behavior-aware, evidence-based task selection under a constrained budget. Across three harnesses and four benchmarks, it improves performance while using fewer rollouts than fixed-set methods.
- Motivation: Agent harnesses shape how language-model agents perceive tasks, use tools, and act through configurable instructions, skills, memory, permissions, and roles.Their configuration substantially affects agent performance, but building effective harnesses requires substantial manual effort.
- Verification gap: Fixed evaluation tasks can waste rollouts on behaviors unrelated to a modification and obscure whether its intended behavior emerged or caused regressions.Aggregate results may therefore dilute candidate-specific signals when selected tasks do not cover affected behaviors.
- Approach: Behavior-aware verification selects tasks and rollout allocations for each modification using supporting trajectories, affected components, task patterns, and regression risks.Candidate trajectories are compared to attribute behavioral changes and preserve existing capabilities.
- Results: 7.6–13.6% average performance improvements are achieved across three harnesses and four benchmarks with a substantially smaller joint rollout-and-analysis budget.Reported success-rate improvements reach up to 13.6% on OpenCode, 7.6% on Codex, and 9.2% on Pi.
- Approach: HarnessLens autonomously discovers editable components, diagnoses interaction trajectories, and iteratively proposes, verifies, and reviews harness modifications.Its stages include context exploration, trajectory diagnosis, and harness evolution.
2 Related Work
Prior harness-evolution methods update prompts, workflows, programs, and other configurable components, but verification commonly uses shared task collections or globally selected subsets. HarnessLens instead adapts tasks and rollout allocation to each modification’s intended behavior and accumulated evidence.
- Self-improving agents and harness evolution: Self-improving-agent methods revise prompts, workflow graphs, harness specifications, domain scaffolds, harness programs, tools, skills, and memory.Harness evolution extends earlier work on relatively lightweight structures to richer harness components.
- Harness verification and rollout allocation: Most existing verification methods evaluate candidates on shared, predefined task collections, including fixed, held-out, or current evaluation batches.These approaches apply the same evaluation structure across different candidate modifications.
- Harness verification and rollout allocation: Other methods narrow verification to predefined flaw- or failure-related tasks or reduce cost with a globally preselected coreset.These strategies constrain verification using task sets selected before candidate-specific evidence is available.
- Harness verification and rollout allocation: HarnessLens adapts verification tasks and rollout allocation to each modification’s intended behavior, affected components, regression risks, and accumulated evidence.This design targets irrelevant rollouts while preserving modification-specific signals.
3 Problem Formulation
The paper formulates harness evolution as budget-constrained optimization over editable component configurations, with behavior inferred from execution trajectories and performance evaluated on held-out tasks.
- 3.1 Definitions: A harness framework exposes user-configurable components whose configurations determine a concrete harness.Components may include instructions, skills, prompt templates, tools, integrations, agent roles, and runtime extensions.
- 3.1 Definitions: The admissible harness space contains configurations that share a fixed framework and execution mechanisms but may differ in component content and instances.
- 3.1 Definitions: An agent is a base language model operating within a concrete harness.
- 3.1 Definitions: A harness modification changes one or more user-configurable component configurations while leaving the framework unchanged.Evolution therefore operates over component configurations rather than the framework itself.
- 3.2 Optimization Objective: A trajectory records alternating observations and actions generated while an agent executes a task under a model and harness.
- 3.2 Optimization Objective: Behavior is a recurring model-visible action pattern identified from trajectories, so equal task rewards can still correspond to different behaviors.
- 3.2 Optimization Objective: Training tasks are available during evolution, whereas disjoint held-out tasks measure final generalization performance.
- 3.2 Optimization Objective: The optimization seeks a harness maximizing expected task success under a fixed interaction budget.The budget counts LLM sessions and task trials consumed along the evolution path, including exploration, diagnosis, and evolution.
4 Method
HarnessLens explores task and harness spaces, diagnoses trajectory evidence, and iteratively proposes and verifies component modifications with behavior-aware, regression-sensitive decisions.
- Harness Evolution: HarnessLens evolves configurable state within a fixed framework using a deterministic controller that tracks budget, isolates executions, reuses rollouts, and enforces update criteria.
- Method: The method has three stages: Context Exploration, Trajectory Diagnosis, and Harness Evolution.These stages characterize editable context, extract evidence, and construct, verify, and update candidate harnesses.
- Context Exploration: Context Exploration organizes training tasks by goals and identifies editable components, their update mechanisms, affected behaviors, and reliable scope.
- Trajectory Diagnosis: Trajectory Diagnosis extracts reusable experiences and deficiencies from initial and verification rollouts, linking each item to supporting trajectories.
- Trajectory Diagnosis: Experience Analysis combines trajectory evidence, task groups, and component descriptions to propose modifications targeting specific behaviors.
- Candidate Proposal: Each candidate applies a selected modification to a copy of the current confirmed harness, and accepted modifications accumulate across iterations.A runtime check prevents candidates with unsuccessfully applied component changes from entering verification.
- Behavior-Aware Verification: Behavior-aware verification selects supporting tasks, related tasks, and regression-revealing tasks, with broader modifications receiving wider task-group coverage.The candidate and current harness are evaluated under matched trial conditions, after which trajectories are diagnosed for targeted improvement and regressions.
- Harness Review and Update: Candidates require attributable behavioral evidence and confirmation beyond primary-metric improvement before acceptance.Confirmation uses a new batch containing at most two further-confirmation tasks and unused tasks prioritized from uncovered groups; evolution stops when evidence or budget is insufficient.
5 Experiments
HARNESSLENS is evaluated across three harnesses and four benchmarks against fixed-set evolution baselines under constrained budgets. It achieves strong held-out performance while avoiding regressions and using fewer evaluation resources.
- Main results: HARNESSLENS attains the best or tied-best pass rate in eight of twelve harness–benchmark pairs.It uses two-thirds of HarnessFix’s budget and one twenty-fourth of Self-Harness’s budget.
- Stability: HARNESSLENS never falls below the initial harness H0, with its worst outcome being an exact tie.When no attributable improvement without regression is found, the run returns H0 unchanged.
- Stability: Self-Harness and Meta-Harness fall below H0 in half of their 24 harness–benchmark pairs, losing about ten points at worst.HARNESSLENS instead turns uninformative iterations into ties rather than regressions.
- Budget and acceptance: More verification rollouts do not imply better harnesses because fixed-task aggregate pass rates can obscure modification-specific gains and regressions.The baselines use 4,800, 660, and 300 TRAIN rollouts at their configured maxima, while HARNESSLENS is capped at 200 total units.
6 Analysis
The analysis isolates behavior-aware task selection and attributable-evidence gating as complementary mechanisms. Their value depends partly on task-space structure: recurring related tasks provide stronger support than highly diverse objectives.
- 6.1 Ablation: Selection and Gating: Removing either behavior-aware selection or attributable-evidence gating substantially weakens performance.The ablation uses B = 200 while changing only verification-stage configuration.
- 6.1 Ablation: Selection and Gating: Fixed, random, and RHO-based batches often include unrelated tasks, producing evidence that cannot be attributed to the targeted behavior.These variants therefore often fail to produce candidates that pass verification.
- 6.1 Ablation: Selection and Gating: Metric-Only Gate accepts aggregate gains unsupported by attributable evidence, yielding no clear held-out improvement on Banking or BIRD.Behavior-aware selection improves evidence relevance, while attributable gating prevents noisy or unsupported gains from being retained.
- 6.2 How Does Task Diversity Affect Evolution?: Recurring actionable weaknesses allow related tasks to reinforce a modification across examples, whereas Retail’s strong initial harness leaves less room for improvement.Retail trajectories mainly serve as regression checks, making retention of the original harness often desirable.
- 6.2 How Does Task Diversity Affect Evolution?: In BIRD, related extreme-value-query tasks expose a shared SQL decision, enabling attributable recovery and an accepted edit.The BIRD TEST result is cumulative across accepted edits.
- 6.2 How Does Task Diversity Affect Evolution?: In Terminal-Bench 2.0, diverse goals and execution paths provide insufficient support for a broad rule, which causes a regression and is rejected.Diversity within both TRAIN and TEST limits modifications supported by TRAIN and broadly applicable to TEST.
7 Conclusion
HARNESSLENS evolves agent harnesses by directing verification toward behaviors targeted by each modification and requiring attributable evidence before acceptance. Across three harnesses and four benchmarks, it improves or preserves initial performance under limited rollouts.
- Conclusion: HARNESSLENS selects modification-relevant tasks and accepts changes based on attributable behavioral evidence.This is the framework’s behavior-aware verification strategy.
- Conclusion: 7.6–13.6% average performance improvements are achieved across three harnesses and four benchmarks while limited rollouts are used more effectively.The framework improves or preserves initial performance and avoids unsupported regressions.
Limitations
The evaluation covers one model family, three harnesses, and four public benchmarks, so broader validation remains open. Its budget also does not normalize token usage, latency, or monetary cost.
- Scope: The effectiveness of behavior-aware verification under broader model, harness-architecture, and open-ended deployment settings has not been fully validated.The reported evaluation spans one model family, three harnesses, and four public benchmarks.
- Cost accounting: The budget counts LLM sessions and task trials as auditable interaction units without normalizing token usage, latency, or monetary cost.Budget comparisons therefore use the paper’s interaction-unit accounting.
A Implementation and Verification Details
HarnessLens uses controller-managed, budget-aware evolution with isolated candidates, behavior-targeted verification, and attributable evidence before updates. Its protocol combines exploration, trajectory diagnosis, paired trials, confirmation, and strict runtime and budget checks.
- Budget control: The controller starts an iteration only when both verification rounds and a three-unit retry buffer are affordable.Invalid structured outputs may be retried once during exploration and twice during diagnosis or review, with every retry charged.
- Candidate isolation: Each candidate snapshots the current configurable state while keeping the framework, model, provider, permissions, tools, and evaluator fixed.Candidate patches must bind to existing tool and parameter names and cannot introduce new tools.
- Behavior-aware verification: Verification batches contain at least five distinct TRAIN tasks, including a conversion task and one linked to the supporting trajectory.Remaining tasks cover related groups, affected tools, and preservation risks; confirmation uses mostly fresh tasks and prioritizes uncovered groups.
- Verification protocol: Paired trials align current and candidate harnesses within each round, while fresh confirmation seeds support an additional check.Pairing reduces avoidable variation without assuming provider-side determinism.
- Acceptance gate: Updates require attributable positive evidence, no attributable regression, and improvement under the primary metric on confirmation.Preservation alone is insufficient, and comparison is independently checked before diagnosis receives the candidate diff.
B Experimental Protocol
The experiments compare harness-evolution methods under shared tasks, runtime conditions, and initial harnesses, then evaluate final systems on a blind TEST split. TRAIN decisions use trial-level pass rate, whereas reported TEST performance uses held-out pass@1.
- Systems: The evaluated agents and model-based roles use deepseek-v4-flash-preview across OpenCode, Codex CLI, and Pi Coding Agent.The evaluated harness versions are OpenCode 1.17.13, Codex CLI 0.144.4, and Pi Coding Agent 0.80.10.
- Runtime controls: Evaluator-owned overrides fix the model, provider, tools, inference limits, and permissions.Candidates therefore cannot enlarge their own budgets, enable external retrieval, or alter task tools.
- Metrics: TRAIN uses K = 2 trials per task, while held-out TEST pass@1 uses one fresh trial per task under the final harness.Pass@1 is reserved for the single-trial TEST metric.
- Blind evaluation: TEST tasks, trajectories, feedback, and rollout services remain unavailable to all evolution, diagnosis, verification, and review roles.Initial and evolved harnesses use the same ordered TEST split, runtime limits, and pairing offsets.
- Shared protocol: All methods use the same initial harness, 30-task TRAIN split, benchmark runtime, verifier, and component-change mechanisms.Configured protocols can consume less than their maxima because of early stopping, retries, or method-specific validation.
- Baselines: Self-Harness uses 4,800 TRAIN rollouts, Meta-Harness 660, and HarnessFix 300 under their configured protocols.These totals derive from their respective iteration, candidate, task, trial, and repair schedules.
C.1 OpenCode–BIRD Evolution Trace
In the OpenCode–BIRD trace, HarnessLens accumulated four modifications supported by attributable evidence and rejected unsupported changes. The final harness improved held-out TEST pass@1 by 8.33 percentage points.
- Accepted modifications: Four later edits were accumulated after both verification rounds supported targeted improvements without attributable regressions.The edits added deterministic tie handling, conditional aggregation, output-column restriction, and native precision guidance.
- Rejected modifications: The final skill candidate was rejected because it was not invoked during verification and produced no attributable change.The initial instruction edit was likewise rejected after producing no attributable improvement.
- Run scope: The four-run analysis used B = 200 for each benchmark and recorded 21 candidate iterations, with 19 completing paired verification comparisons.Three runs accepted at least one modification, while one returned H0.
- Run scope: The Terminal-Bench run accepted no modification and therefore returned H0.Table 10 reports H0 and Final as held-out TEST pass@1 percentages.
C.2.2 Re-scoring the Recorded Decisions
Re-scoring recorded OpenCode decisions shows that attributable-evidence gating selects different modifications from a metric-only rule. The analysis compares decisions on stored verification evidence, while TEST effects require a separate ablation.
- Decision differences: The metric-only rule and attributable-evidence gate differed on 7 of 19 paired iterations.The metric-only rule would accept 10 modifications, whereas the attributable-evidence gate accepted 5.
- Decision differences: A candidate gained 50.0 percentage points on its verification batch but was not accepted because no task recovery was attributable to the modified component.This is the largest disagreement between the two rules.
- Decision differences: One iteration advanced despite a 10.0-point verification-batch decrease because it showed attributable targeted recovery without regression.Acceptance still required improvement on the confirmation batch.
- Scope: The re-scoring does not estimate counterfactual metric-only TEST performance because those harnesses were not constructed.Table 2’s separate Metric-Only Gate ablation supplies the corresponding TEST result.
- Budget accounting: Across the four runs, initial and verification rollouts, comparison, and post-rollout diagnosis consumed 82.2% of charged interaction units.These categories represented 79.0% to 83.8% within individual runs.