Source-linked AI summary
The Last Harness You'll Ever Build
Haebin Seong, Li Yin, Haoran Zhang, Zhan Shi
TL;DR
Domain-specific agents require intensive human-built harnesses, so this paper proposes nested loops that automate both harness optimization and the evolution process itself. The framework separates task-level improvement from meta-level blueprint optimization to learn how harnesses can be adapted across tasks.
Problem
Effective agent harnesses require intensive, specialized human engineering for each task domain, including scaffolding, evaluation, feedback, and context management.
Method
A Harness Evolution Loop iteratively improves a worker harness through execution, adversarial evaluation, and evolution, while a Meta-Evolution Loop optimizes that blueprint across diverse tasks.
Results
The paper presents a two-level formulation in which meta-evolution learns a blueprint that enables rapid adaptation of worker harnesses to unseen tasks.
Takeaways & Limitations
The framework automates manual harness engineering and extends automation to the design of the harness-evolution process itself.
Takeaways & Limitations
Empirical results on diverse, difficult workflows are planned as future work, so the supplied passages do not report validation of the framework’s effectiveness.
Abstract
from arXiv · showhide
AI agents are increasingly deployed on complex, domain-specific workflows -- navigating enterprise web applications that require dozens of clicks and form fills, orchestrating multi-step research pipelines that span search, extraction, and synthesis, automating code review across unfamiliar repositories, and handling customer escalations that demand nuanced domain knowledge. \textbf{Each new task domain requires painstaking, expert-driven harness engineering}: designing the prompts, tools, orchestration logic, and evaluation criteria that make a foundation model effective. We present a two-level framework that automates this process. At the first level, the \textbf{Harness Evolution Loop} optimizes a worker agent's harness $\mathcal{H}$ for a single task: a Worker Agent $W_{\mathcal{H}}$ executes the task, an Evaluator Agent $V$ adversarially diagnoses failures and scores performance, and an Evolution Agent $E$ modifies the harness based on the full history of prior attempts. At the second level, the \textbf{Meta-Evolution Loop} optimizes the evolution blueprint $Λ= (W_{\mathcal{H}}, \mathcal{H}^{(0)}, V, E)$ itself across diverse tasks, \textbf{learning a blueprint $Λ^{(\text{best})}$ that enables rapid harness convergence on any new task -- so that adapting an agent to a novel domain requires no human harness engineering at all.} We formalize the correspondence to meta-learning and present both algorithms. The framework \textbf{shifts manual harness engineering into automated harness engineering}, and takes one step further -- \textbf{automating the design of the automation itself}.
1 INTRODUCTION
The paper introduces a two-level framework that automates harness engineering for complex agent tasks. It first evolves a worker’s harness through execution, adversarial evaluation, and history-based modification, then evolves that process itself across diverse tasks to enable rapid adaptation without human harness engineering.
- Motivation: Harness engineering can substantially amplify agent performance, but designing task-specific scaffolding remains highly intensive and specialized human work.The scaffolding includes execution environments, feedback loops, evaluation criteria, and context management.
- Harness Evolution Loop: The Harness Evolution Loop optimizes a worker agent’s harness for a single task through iterative execution, adversarial evaluation, and harness modification.The Worker executes the task, the Evaluator verifies outcomes and diagnoses failures, and the Evolution Agent responds to those failure patterns.
- Harness Evolution Loop: The Worker, Evaluator, and Evolution Agent together form an evolution blueprint Λ = (W_H, H^(0), V, E) that returns the best-performing harness after K iterations.The Evolution Agent can modify prompts, tools, orchestration logic, observations, and model configuration using the full history of prior attempts.
- Meta-Evolution Loop: The Meta-Evolution Loop optimizes the evolution blueprint Λ across diverse tasks, learning Λ^(best) for rapid harness convergence on new tasks.This turns both harness engineering and the design of the harness-engineering process into automated optimization problems.
2 THE HARNESS EVOLUTION LOOP
The Harness Evolution Loop treats an agent as a model plus a configurable harness, then iteratively evaluates and evolves that harness for a task. Its worker, evaluator, and evolution agents use execution traces, diagnostics, scores, and history to retain improvements, while the Meta-Evolution Loop optimizes the broader evolution blueprint across training tasks.
- 2.5 Harness Evolution Loop: Across iterations, the loop evaluates each harness, retains the highest-scoring version, records improved or regressed outcomes, and evolves the next harness from the full history.The Evolution Agent classifies recurring failure patterns and uses prior reports, scores, and attempted variants to avoid repeating unsuccessful strategies.
- 2.6 Meta-Evolution Loop: The Meta-Evolution Loop runs harness evolution across meta-training tasks, aggregates task scores, and evolves the blueprint from the best-performing blueprint and accumulated meta-history.The blueprint includes the worker agent, initial harness, evaluator, and evolution agent, with an inner-loop budget K.
- 2.1 Agent Harnesses: An agent is defined as a model plus a harness comprising the code, configuration, and execution logic that makes the model’s intelligence useful.Harness components include prompts, tools and skills, infrastructure, orchestration logic, middleware, and model configurations.
- 2.2 Task Definition: A task consists of instructions and verifiable success criteria, which guide execution and evaluator judgment.Formally, t = (I, S), with S = {s1, s2, . . . , sm}.
- 2.3 Worker Agent: The Worker Agent executes a task through its harness and produces a trace containing environment observations, action logs, and timing information.The worker interface is WH.execute(t), where the worker receives the task instructions and interacts with the target environment through tools.
- 2.4 Evaluator Agent: The Evaluator Agent adversarially reviews the worker trace, verifies perceived state, checks every success criterion, and produces a diagnostic report with a numerical score.It also audits execution performance by separating LLM inference latency from environment-interaction latency.
3 META-EVOLUTION: LEARNING TO EVOLVE HARNESSES
Meta-evolution treats the harness evolution blueprint Λ itself as an optimizable harness, learning Λ(best) across diverse tasks to make inner-loop adaptation faster and more reliable on unseen tasks. The framework corresponds directly to meta-learning, with performance judged by adaptation speed, final performance, and robustness.
- Blueprint definition: Λ packages the worker agent, initial worker harness, evaluator, and evolution agent that together define how the harness evolution loop operates.In the current system, human engineers design Λ and keep it fixed during evolution.
- Blueprint optimization: Meta-evolution optimizes Λ as a higher-level harness containing prompts, tools, observations, and orchestration logic.Its modifiable components include evaluator and evolution prompts, worker telemetry, inter-agent information flow, and scoring-function design.
- Two-level optimization: The inner loop evolves a worker harness for one task, while the outer loop modifies Λ across training tasks to improve adaptation speed.The inner loop runs for K iterations and measures the worker’s convergence trajectory; the outer loop evaluates how effectively Λ drives that process.
- Optimization objective: Λ(best) is selected to maximize final task performance across meta-training tasks, based solely on each task’s final best score.The harness evolution procedure returns the best-performing harness, its score, and the full evolution history.
- Generalization: On held-out tasks, generalization measures convergence speed, final performance, and robustness across different task adaptations.A successful Λ(best) should produce effective worker harnesses with fewer iterations and less compute than a manually designed evolution loop.
4 CONCLUSION
The paper presents a two-level framework that automatically evolves task-specific agent harnesses and the blueprint for evolving them, replacing manual harness engineering with automated adaptation. It also outlines planned empirical validation on brittle, specialized workflows and a product based on the learned blueprint.
- Harness Evolution Loop: The Harness Evolution Loop iteratively improves an agent harness through task execution, adversarial evaluation, code modification, and convergence-history tracking.The harness includes prompts, tools, orchestration logic, and infrastructure surrounding a foundation model.
- Meta-Evolution Loop: The Meta-Evolution Loop optimizes the evolution blueprint across diverse training tasks so it can rapidly adapt harnesses to unseen tasks.The inner loop adapts a harness to one task, while the outer loop optimizes the adaptation procedure.
- Overall contribution: Together, the loops transform manual harness engineering into automated harness engineering and automate the design of the automation itself.The outer loop learns how to evolve harnesses rather than evolving any single harness.
- Future work: The authors plan empirical evaluation on brittle, specialized workflows and intend to release a product built on the learned evolution blueprint.Planned targets include customized customer workflows and domain-specific enterprise processes.