Source-linked AI summary
HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution
Xiaotian Luo, Dizhan Xue, Fengxingyu Wang, Chuanrui Hu, Yafeng Deng
TL;DR
HarnessBank addresses the difficulty of reliably evolving agent harnesses when greedy search and noisy feedback can limit exploration and verification. It combines a semantic gene bank with gated screening, and reports consistent gains across seven benchmarks, with evolved harnesses adapting to model-specific failure patterns.
Problem
Agent performance depends on its harness, but existing self-evolution approaches often use greedy search and noisy selection, leaving concerns about collapse, overfitting, and unverifiable gains.
Source: §1 · pp. 1–2Method
HarnessBank pairs a task agent with an evolver, preserves semantically distinct harnesses for reinvention and recombination, and screens offspring before full evaluation.
Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5Results
Across seven agent benchmarks, HarnessBank consistently improves performance by 5.1% to 15.4%.
Source: §5 · p. 7Takeaways & Limitations
Cross-model results indicate that evolved harnesses adapt to model-specific failure patterns rather than forming a universally optimal harness.
Source: §1 · p. 2, §5 · p. 7The test set is withheld from evolution decisions and is used only to assess the train-selected harness.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Large Language Models (LLMs) have enabled capable agents across diverse applications. Beyond the foundation model, the performance of an agent is governed by the surrounding agent harness, including prompts, tools, control loops, etc. Automatically evolving this harness offers a promising pathway to agent improvement, yet existing approaches typically rely on greedy candidate selection and noisy self-generated feedback, rendering their gains susceptible to search collapse, task-specific overfitting, and poor verifiability. To tackle these challenges, we introduce HarnessBank, a trustworthy agent-harness self-evolution framework that pairs a task agent with a separate evolver agent for iterative failure diagnosis, harness generation, and evolution verification. HarnessBank maintains a Harness Gene Bank composed of high-performing harnesses of different semantic coordinates. Those harnesses are reinvented, recombined, screened, and selected during the self-evolution procedure. Moreover, we propose a Gated Harness Screening mechanism to efficiently filter high-quality harnesses and reduce the cost of evaluating numerous offspring harnesses. Across seven agent benchmarks, HarnessBank produces consistent performance improvements from 5.1% to 15.4%. Cross-model experiments further verify that the improvements come from the model-specific self-evolving process, instead of a universally optimal harness. Our code will be publicly available upon acceptance.
1 Introduction
HarnessBank targets the limits of hand-engineered harnesses and greedy, noisy self-evolution by combining semantic diversity with gated verification. It reports consistent gains across seven benchmarks while keeping the task model frozen.
1 Introduction
- Hand-engineered harnesses are difficult to scale, while greedy evolution can collapse toward narrow edits and produce gains that are noisy or unverifiable.HarnessBank frames these risks as search collapse, task-specific overfitting, and poor verification.Source: §1 · pp. 1–2
- HarnessBank pairs a task agent with a separate evolver that diagnoses failures and generates harness offspring without changing model parameters.Its Harness Gene Bank preserves high-performing harnesses across semantic coordinates for reinvention and recombination.Source: §1 · p. 2
- Gated Harness Screening checks candidate validity, mechanism activation, statistical significance, and gain before full training-set evaluation.Only candidates passing the sequential checks proceed to compete for admission to the bank.Source: §1 · p. 2
- Across seven agent benchmarks, HarnessBank improves frozen task agents by 5.1% to 15.4%.The reported contribution also says cross-model experiments find the evolved harnesses model-specific rather than universally optimal.Source: §1 · p. 2
2 Related Work
Related work moves from optimizing individual harness components toward full-harness self-evolution. HarnessBank addresses limitations in existing evolution methods by preserving semantically diverse harnesses and verifying candidate improvements.
2.1 LLM Agents and Agent Harnesses
- An agent harness is executable scaffolding around a backbone LLM, spanning prompts, knowledge, tools, control logic, recovery, validation, and runtime configuration.The section describes this unified harness as important when model weights are frozen, closed, or costly to update.Source: §2.1 · pp. 2–3
2.2 Agent-Harness Self-Evolution
- Partial self-evolution optimizes restricted components such as prompts, workflows, or memory, establishing trajectory- or score-guided optimization but limiting the search space.The cited examples include prompt optimization (Zhou et al. 2023; Yang et al. 2024; Khattab et al. 2024; Agrawal et al. 2026), workflow optimization (Lu et al. 2026), and memory updates (Zhang et al. 2025c; Liu et al. 2026; Yu et al. 2026; Zhang et al. 2025a).Source: §2.2 · p. 3
- Full-harness methods diagnose trajectories, propose modifications, and retain beneficial changes, but often use greedy search and noisy selection criteria.Examples include Self-Harness (Zhang et al. 2026a), HarnessFix (Chen et al. 2026a), and AHE (Lin et al. 2026); HarnessBank responds with a semantic gene bank and gated screening.Source: §2.2 · p. 3
3 Method
HarnessBank evolves a frozen model’s harness through a task agent, an evolver, a semantic gene bank, and gated screening. It searches over valid harness modifications, retains distinct high-performing mechanisms, and screens offspring before full evaluation.
3.1 Problem Formulation
- HarnessBank keeps the model and immutable harness kernel fixed while optimizing the mutable surface, and selects a train-set utility-maximizing harness for held-out test assessment.The test set is withheld from evolution decisions.Source: §3.1 · p. 3
3 Method
- The task agent executes tasks, the evolver diagnoses failures and proposes patches, a deterministic evaluator scores candidates, and the gene bank preserves high-performing harnesses.The procedure selects a parent, diagnoses training-task outcomes, generates offspring, screens them, then evaluates passing candidates for competitive bank admission.Source: §3.2 · p. 3, Figure 2 · p. 4
3.2 Self-Evolution Based on Harness Gene Bank
- The Harness Gene Bank preserves verified harnesses by modified component and targeted failure pathology, allowing distinct mechanisms to remain available for reinvention or recombination.Harnesses targeting the same pathology compete within a cell, while different pathologies remain available; offspring are generated from training-task diagnosis and the bank.Source: §3.2 · pp. 3–4
3.3 Gated Harness Screening
- Gated Harness Screening evaluates offspring on a sampled training subset, checking evaluation validity, patch activation, and paired significance before passing candidates to full training-set evaluation.The paired significance threshold uses 1.96, corresponding to the conventional two-sided 5% significance level.Source: §3.3 · pp. 4–5
4 Experiments
HarnessBank is evaluated with a frozen Qwen3.6-27B task agent across seven domains, against vanilla and prior harness-evolution methods. Its test gains are generally credited, while the results also show that harness improvements depend on matching patches to model-specific failure patterns.
4.1 Experimental Setup
- Experiments use disjoint train/test splits across seven domains, with the harness evolving while the task-agent backbone remains frozen.Qwen3.6-27B is the default task agent, Claude Opus 4.8 is the evolver, and the primary metric is per-task Pass@1 averaged over K=3 attempts.Source: §4.1 · p. 5
4.3 Comparison with Existing Methods
- HarnessBank is credited on four of five sealed tests, versus one for DGM and none for GEPA; SWE-bench is the small-sample exception.GEPA’s training gains can wash out on test, while DGM’s ungated selection includes an uncredited spike and a regression below vanilla.Source: §4.3 · p. 5
4.4 Significance and Generalization
- All six test gains clear z ≥1.96, and each train-selected winner is compared once on held-out tasks; the two-sided gate also culls a −6.4% TB2 regression.Test lift retains 65–148% of train lift, but the paper cautions that these ratios are noisy; pass@3 also rises on every credited domain (+5.5 to +15.4%).Source: §4.4 · pp. 5–6
4.2 Main results
- +9% to +15.4% test gains over vanilla clear the paired-2σ bar on the reported test sets; SWE-bench’s +5.1% at n=26 remains preliminary.The main-results account identifies harness-level recovery of empty thinking-runaway turns and, on some domains, a verify-finalize self-check; BrowseComp+ is credited at +13.9%.Source: §4.2 · p. 5, Table 1 · p. 6
4.5 Diagnosis and Cost
- Two recurring frozen-model failure patterns—empty thinking-runaway turns and premature or unverified finalization—are addressed by distinct runtime mechanisms.On five domains, one pathology dominates 49–88% of vanilla failures; an LLM-assigned diagnosis can be wrong, but the gate rejects the proposed patch.Source: §4.5 · p. 6
4.6 Cross-Model Dissociation
- Cross-model results favor patches matched to each model’s pathology, rather than a universally optimal harness.On Omni-MATH, the 27B-evolved stack transfers nearly loss-free to the other Qwen model, while the opposite adjustment helps Gemini and can harm the 397B model; AppWorld also shows distinct model-specific patches.Source: §4.6 · pp. 6–7, Table 2 · p. 7
4.7 Ablation Study
- On TB2, removing the paired-2σ gate lets neutral candidates appear to make progress in 62–76% of post-convergence rounds, so stopping occurs only at the cap.The paper also reports that per-cell elites preserve mechanisms for recombination and that accepted edits span four levers rather than prompts alone.Source: Table 3 · p. 7, §4.7 · p. 7
- On GDPval, a variant ranked below the train winner scores higher on test (+11.5% vs. +9.2%), showing train selection can miss the best held-out candidate.The paper argues that semantic rather than task-indexed archive coordinates avoid preserving harnesses keyed to the tasks used for selection.Source: §4.7 · p. 7
5 Conclusion
HarnessBank combines a semantic Harness Gene Bank with Gated Harness Screening for agent-harness evolution without updating the underlying model. Across seven benchmarks, it improves performance, while cross-model results indicate that evolved harnesses address model-specific failure patterns rather than provide a universally optimal harness.
5 Conclusion
- HarnessBank achieves consistent gains of 5.1% to 15.4% across seven agent benchmarks.Source: §5 · p. 7
- Its semantic Harness Gene Bank preserves and recombines high-performing harnesses across distinct failure pathologies, while Gated Harness Screening filters functional and statistically reliable improvements.Source: §5 · p. 7
- Cross-model results show evolved harnesses adapt to model-specific failure patterns rather than constitute a universally optimal harness.Source: §5 · p. 7