Source-linked AI summary
Rethinking the Evaluation of Harness Evolution for Agents
Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, Teng Xiao
TL;DR
Existing harness-evolution evaluations may conflate reusable harness improvements with extra test-time search and task-specific overfitting. This paper compares harness evolution with test-time discovery under unified budgets and finds inconsistent gains with poor generalization beyond search tasks.
Problem
Existing evaluations leave unresolved whether harness evolution improves harness design beyond repeated task-level search and whether gains transfer to held-out tasks.
Method
The paper compares parallel sampling, sequential refinement, harness evolution, and harness scaling under a unified compute-budget framework across matched feedback settings and disjoint tasks.
Results
Harness evolution does not consistently outperform test-time discovery and shows no advantage on disjoint evaluation tasks.
Takeaways & Limitations
Current evidence for automatic harness evolution should be interpreted cautiously, motivating fairer evaluation protocols and benchmarks for harness design.
Takeaways & Limitations
Most harness edits memorize task-specific fixes, while persistent prompt text can cause context bloat and leave deep reasoning failures unaffected.
Abstract
from arXiv · showhide
We revisit the evaluation of automatic harness evolution for LLM agents. Existing harness evolution methods use unit test cases to search for harness configurations and then report final performance on the same public benchmark. This protocol raises two fundamental concerns. First, harness evolution is itself an iterative search procedure that repeatedly evaluates and revises candidate harnesses using task feedback. As in agentic test-time scaling, it should therefore be compared with simple task-level search baselines under matched feedback and inference budgets to determine whether its gains arise from improved harness design or from additional search alone. Second, because the search and the final evaluation share the same benchmark, the reported gains risk overfitting to that specific task set. To address these concerns, we conduct an extensive evaluation comparing harness evolution with simple test-time scaling and discovery baselines under comparable feedback and inference budgets, and also evaluate evolved harnesses on held-out tasks to assess whether the discovered improvements generalize. Experiments on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6 show that automatic harness evolution does not consistently outperform simple test-time scaling methods and exhibits limited generalization. Our results raise important questions about the effectiveness of automatic harness evolution and highlight the need for fairer evaluation protocols and benchmarks for automatic harness design. Our code is available at https://github.com/rethinking-harness-evolution.
1 Introduction
The introduction argues that harness evolution is important but its evaluation is confounded by benchmark reuse and unequal search budgets. It motivates comparisons against task-level test-time scaling and assessments of whether improvements generalize beyond the search tasks.
- Harnesses and agent performance: Harnesses shape how LLM agents observe tasks and act through prompts, tools, memory, verification routines, and control logic.Prior work indicates harness engineering can substantially affect performance even when the underlying model is fixed, but harness development remains largely manual.
- Evaluation concerns: Existing harness-evolution studies often use verifier feedback on benchmark tasks and evaluate the resulting harness on that same public benchmark.This protocol can make reported improvements specific to the benchmark used during search.
- Evaluation concerns: Test-time scaling baselines allocate additional computation directly to evaluation tasks through parallel sampling, sequential refinement, or task-level revision.Under comparable feedback and inference budgets, these baselines test whether harness evolution adds benefits beyond repeated task-level search.
- Evaluation concerns: The unresolved question is whether harness evolution produces generalizable harness-design improvements or primarily benefits from repeated sampling.This motivates evaluating both matched task-level search and generalization beyond the tasks used for harness search.
2 Related Work
Related work spans test-time scaling and automatic harness evolution as ways to improve agent outputs without changing model weights. Test-time scaling adds inference computation, while harness evolution searches over external scaffolds using execution feedback and, increasingly, complete harnesses.
- Test-time Scaling: Test-time scaling improves language-model outputs by allocating additional inference computation without modifying model weights.Its main categories are parallel sampling and sequential refinement.
- Test-time Scaling: Parallel sampling generates independent candidate answers and aggregates them through voting or verifier reranking.
- Automatic Harness Evolution: Early automatic harness evolution optimizes prompts, demonstrations, language-model programs, or reusable experience from execution feedback without retraining the underlying model.These approaches externalize agent improvements outside model weights but typically optimize one component at a time.
- Automatic Harness Evolution: Recent methods instead treat the full harness as the search object, evolving harness code or prompts using source code, scores, and execution traces.Meta-Harness searches harness code, while Agentic Harness Engineering evolves prompts too.
3 Rethinking the Evaluation of Harness Evolution
This section formalizes harness evaluation under a unified compute-budget view, distinguishing reusable harness improvement from additional test-time discovery. It defines four methods—parallel sampling, sequential refinement, harness evolution, and harness scaling—and specifies their feedback, updates, and final outputs.
- Unified evaluation framework: Under a fixed agent policy, task distribution, and compute budget, each method is defined by what it updates and which feedback it observes.Every method consumes budget K and returns a final trajectory for each task.
- Test-time scaling baselines: Parallel sampling uses K independent trajectories with a fixed harness, selecting by model self-judgment without tests or verifiable outcomes when tests are available.With unit tests, any trajectory accepted by the outcome is returned.
- Test-time scaling baselines: Sequential refinement allocates budget to depth, conditioning each trajectory on the previous attempt while keeping the harness fixed and optionally exposing test outcomes.Without tests, the final trajectory y_K is returned; with tests, any accepted trajectory is returned.
- Harness evolution: Harness evolution iteratively updates a harness across a task batch, using accumulated rollout evidence and a meta agent to optimize expected task-distribution success.The selected harness may be based on evolved state without tests or aggregate outcomes on the batch or held-out set with tests.
- Harness scaling: Harness scaling adapts the harness for one evaluation instance by repeatedly revising it after each trajectory, making it a harness-level analogue of test-time scaling.Updates can condition on the task, prior harness, trajectory, and—when available—the prior outcome.
4 Experiments
Across three settings on Terminal-Bench 2.1, automatic harness evolution does not consistently outperform simpler test-time scaling or discovery methods. Its gains are limited when evaluated on disjoint held-out tasks, suggesting weak generalization.
- Experimental design: Experiments compare harness evolution with simple test-time discovery under comparable feedback and inference budgets, covering settings without unit tests, with unit tests, and with disjoint search and evaluation tasks.Both methods use the same initial harness, one rollout per task for AHE, and a compute budget of K = 5.
- Without unit test cases: Without unit test cases, Parallel Sampling improves the average score from 68.2 to 72.3 and gains on all three models, while harness evolution does not consistently outperform scaling methods.Sequential Refinement provides only a marginal average improvement of 1.1 points and slightly degrades GPT-5.4 relative to direct sampling.
- With unit test cases: With unit test cases, all methods improve over direct sampling, but neither Harness Evolution nor Harness Scaling outperforms simpler baselines on pass@1 or pass@5.Unit tests provide feedback for refinement and an oracle for selecting the final trajectory, making pass@5 meaningful.
- With unit test cases: On pass@1, Parallel Sampling achieves the best average of 86.0, while Sequential Refinement leads pass@5 with an average of 91.8, surpassing Harness Evolution (86.2) and Harness Scaling (89.3).The results suggest harness-evolution gains largely reflect making multiple attempts rather than reliably solving previously unsolved tasks through improved harness design.
- Generalization: On disjoint held-out tasks, Harness Evolution improves Claude Opus 4.6 by 1.2 points, yields no improvement on GPT-5.4, and produces an average gain of just 0.6 points.The evaluation uses 45 training tasks, 10 validation tasks, and 34 held-out test tasks, indicating limited generalization and overfitting to the search set.
5 Discussion
The discussion shows that harness evolution makes rational edits across prompt, middleware, and tool layers, but a stable core of hard tasks remains unaffected. Its gains are limited because many edits memorize task-specific fixes rather than distilling general strategies, making simple test-time computation scaling often more effective.
- Harness Evolution: Harness Evolution modifies prompts, middleware, and tools to target recurring failure classes and improve delivery, verification, and command execution.Examples include behavioral rules, turn-budget reminders, tool-output truncation, finalization gates, corrected tool guidance, and recovery hints.
- Harness Scaling: Harness Scaling primarily carries failure-specific knowledge into later prompts or memory while streamlining inefficient workflows.Recorded information includes bugs, implementation templates, verification checks, file paths, and command sequences; other edits replace repeated polling and exploratory commands.
- Limitations: Most edits memorize fixes rather than distilling strategies, saving time on solvable tasks but rarely converting failures into successes.The passage attributes this limitation to information that a competent agent could rediscover through exploration within one rollout, alongside a stable core of hard failures.
- Limitations: Automatic harness evolution is often less effective than simple methods that scale computation at test time on Terminal-Bench.The discussion notes that this may reflect limited model capability to revise the entire harness, while also offering additional explanations for the result.
- Future Work: Future evaluations should target tasks with substantial improvement headroom and strong dependence on specialized tools, skills, or workflows.Under these conditions, a better harness can expand the set of tasks agents can solve.
6 Conclusions
The paper argues that current evaluations of automatic harness design conflate genuine harness improvements with test-time discovery and risk overfitting by searching and evaluating on the same public benchmark. It therefore compares harness evolution with test-time discovery baselines under a unified budget.
- Current methods search harness configurations using unit-test feedback and report final performance on the same public benchmark.
- This evaluation conflates genuine advances in harness design with simple test-time discovery baselines.
- Searching and evaluating on the same benchmark introduces a risk of overfitting.
- The study compares automatic harness evolution with test-time discovery baselines under a unified budget.
A Experimental Details · A.1 Initial Harness
All four methods begin from the same minimal code-agent harness, providing only a single bash tool and no additional agent infrastructure.
- A.1 Initial Harness: All four methods initialize the code agent from an identical harness.
- A.1 Initial Harness: The initial harness is minimal by design.
- A.1 Initial Harness: Each agent is granted access to a single bash tool.
- A.1 Initial Harness: No skills are included in the initial harness.
- A.1 Initial Harness: The initial harness contains no middleware.
- A.1 Initial Harness: The agents begin without persistent memory.
A.2 Summarization Map · A.3 Agentic Harness Engineering (AHE) · A.4 Configuration
The paper summarizes rollout trajectories with the Agent Debugger, separates reusable harness evolution from benchmark-specific retrieval in AHE, and fixes a reference configuration across experiments using isolated E2B sandboxes.
- A.2 Summarization Map: The Agent Debugger distills actionable evidence from raw rollout trajectories collected for each task.It is a summarization agent introduced in the AHE framework.
- A.2 Summarization Map: It represents grouped trajectories sharing a query as a navigable file environment explored through shell and scripting tools.Each trajectory message resides in its own file.
- A.3 Agentic Harness Engineering (AHE): AHE ordinarily combines an explore agent that retrieves benchmark-tuned harnesses with an evolution loop that refines harnesses from benchmark feedback.The two components jointly define the standard AHE procedure described here.
- A.3 Agentic Harness Engineering (AHE): The explore agent conflates reusable harness evolution with retrieval because it imports harnesses already fitted to the evaluation benchmark.This retrieval occurs instead of discovering improvements from feedback.
- A.4 Configuration: The full experimental configuration is listed in Table 4 and remains fixed across every experiment.The values come from a snapshot of the reference run.
- A.4 Configuration: Each rollout executes in its own freshly provisioned E2B remote sandbox.This is part of the fixed configuration used throughout the experiments.
A.5 Evaluation Metrics
The evaluation uses pass@1 as its primary metric, averaging binary rollout rewards across tasks and attempts. It also reports pass@k when unit tests are available to quantify the benefit and variability of repeated sampling.
- pass@1: pass@1 is the primary metric, defined as the average binary reward across all rollouts.For task set D with k rollouts per task, each rollout receives reward r_i,j ∈ {0, 1}.
- Scoring convention: Infrastructure exceptions, including sandbox crashes and API timeouts, are scored as failures rather than excluded.This convention applies to both pass@1 and pass@k.
- pass@k: pass@k measures the fraction of tasks solved by at least one of k rollouts when unit test cases are available.It is used to assess how much repeated sampling helps.
- pass@k: pass@5 upper bounds pass@1, while their gap reflects agent variance across independent attempts.A task counts as solved if any sampled rollout succeeds.
B Case Study
The case study shows that Harness Scaling modifies the harness in response to recurring weaknesses observed in failed rollouts. These changes address setup, execution, cleanup, dependency, data-format, state-preservation, embedding, and agent-behavior issues.
- Harness Modifications: Harness Scaling repairs fragile setup and build procedures, wasteful polling loops, and overly short shell timeouts.These modifications target recurring weaknesses observed in failed rollouts.
- Harness Modifications: The meta agent modifies unsafe cleanup behavior, unpinned dependency versions, and incorrect assumptions about data formats.The changes are intended to address recurring failure modes in harness execution.
- Harness Modifications: Other modifications preserve state during inspection, correct embedding prompt types, and address attempts by the agent to weaken the harness.Figure 3 presents representative examples of these harness modifications.