PaperlayerView exact revision on arXiv

Source-linked AI summary

VeRO: A Harness for Agents to Optimize Agents

Varun Ursekar, Apaar Shanker, Veronica Chatrath, Yuan Xue, Samuel Marc Denton

arXiv:2602.22480v4cs.AIcs.CLcs.LG

TL;DR

Coding agents can optimize agent harnesses, but the field lacks systematic benchmarks and controlled evaluation for this stochastic, code-editing task. VeRO supplies versioning, budget-controlled evaluation, and structured traces, while VeRO-Bench enables cross-task comparisons showing gains on tool-use tasks but limited progress on reasoning tasks. The work positions agent optimization as tractable but far from solved, while noting important evaluation limitations.

  • Problem

    Agent harness optimization remains underexplored, with no standardized benchmark for coding agents that modify and evaluate the broader agent program.

  • Method

    VeRO is an outer harness that controls versioned target-agent snapshots, budgeted evaluation, structured traces, and optimizer observations, supporting benchmarked code optimization.

  • Results

    Tool-use tasks show consistent gains, whereas reasoning-heavy tasks show little to no improvement; VERO Default reaches 0.61 average performance versus 0.54 for Resources Only and GEPA.

  • Takeaways & Limitations

    Agent optimization is tractable but far from solved, with current optimizers favoring prompt edits over structural changes and requiring broader benchmark progress.

  • Takeaways & Limitations

    The evaluation uses call-based rather than token- or cost-based budgets and does not compare allocation strategies, include human baselines, or control API instability and reward hacking.

Abstract

from arXiv · show

An important emerging application of coding agents is agent harness optimization: the iterative improvement of a target agent by editing and evaluating its code. Despite its relevance, the community lacks a systematic understanding of coding agent performance on this task. Harness optimization differs from conventional software engineering: agent harnesses interleave deterministic code with stochastic LLM completions, requiring structured capture of both intermediate execution traces and downstream outcomes. To address these challenges, we introduce (1) VeRO (Versioning, Rewards, and Observations), an outer harness that provides versioned snapshots, budget-controlled evaluation, and structured execution traces of target harnesses, and (2) VeRO-Bench, a benchmark suite of target agents and tasks with reference evaluation procedures. Using VeRO, we conduct an empirical study comparing optimizers across tasks and analyzing which modifications reliably improve target agent harnesses. We release VeRO to support research on agent optimization as a core capability for coding agents. Code is available at https://github.com/scaleapi/vero.

1. Introduction

Agent harness optimization extends beyond prompt editing to modifying an agent’s broader program structure, but remains underexplored and lacks standardized coding-agent evaluation. VeRO addresses this gap with controlled infrastructure for versioning, evaluation, and execution traces.

  • Agent harness optimization modifies broader program structure beyond prompts and tool descriptions, yet remains underexplored.
  • Manual agent optimization is labor-intensive and does not scale as demand for specialized agents accelerates.
  • Existing coding-agent benchmarks evaluate software-engineering tasks, while no standardized benchmark frames agent optimization as a coding task.
  • The work contributes an outer harness and empirical study infrastructure for understanding coding agents as agent optimizers.
  • VERO provides versioned snapshots, budget-enforced evaluation, and structured execution traces for target agents.

2. VERO-BENCH Optimization Benchmark. A stan-

VERO-BENCH standardizes evaluation across target agents and tasks, while VeRO experiments reveal that optimizer instructions and modification choices affect performance and generalization.

  • VERO’s architecture combines an optimization trajectory with system components enforcing versioning, reproducible execution, and controlled feedback.
  • VERO-BENCH spans math reasoning, tool use, multi-step question answering, and long-horizon coding with reference evaluation procedures.
  • VeRO experiments show meaningful optimization of both minimal and sophisticated target agents.
  • Optimizer instructions strongly affect performance variance and cross-task generalization.
  • Current optimizers default to prompt modifications and produce limited diversity in their changes.

2. Related Work

Prior work studies LLM optimization, automated agent design, and coding-agent evaluation, but generally treats agents or workflows as fixed or does not benchmark open-ended agent optimization as coding.

  • LLM optimization research includes black-box search, evolutionary algorithm discovery, and systems that optimize an agent’s own code.
  • Prompt frameworks tune instructions and examples while holding the underlying agent workflow fixed.
  • Other frameworks represent workflows as computational graphs or code functions and use evaluation feedback to update agents.
  • Coding-agent benchmarks have expanded from function synthesis to repository-scale tasks, but existing benchmarks treat agents as static artifacts.
  • Meta-Harness independently studies the same problem using versioned candidate histories and budget-bounded evaluation, with different constraints from VERO.
  • VERO distinguishes itself by providing optimization guardrails for coding agents.

3. Methodology

The methodology formalizes agent optimization as improving stochastic Python agents under held-out evaluation and budget constraints, then implements the process through VeRO’s controlled interfaces, tracing, and versioned execution.

  • 3.1. Formal Problem Statement: The target agent maps task instances to final responses and execution traces, while both the agent and evaluator may be stochastic.
  • 3.1. Formal Problem Statement: The objective maximizes expected held-out performance under a maximum number of evaluation calls, with noise controlled through fixed seeds and averaging.
  • 3.1. Formal Problem Statement: The restricted Python search space limits models, APIs, file access, and other resources to support fair comparison and production constraints.
  • 3.1. Formal Problem Statement: The optimizer iteratively modifies the target agent, using evaluation traces and an observation interface that controls exposed history and feedback.
  • 3.2. Harness Requirements: VeRO’s protocol requires controlled conditions, informative feedback, post-hoc interpretability, versioning, budget enforcement, permission control, reproducibility, structured traces, and a standardized observation interface.
  • 3.3. VERO: VeRO implements these requirements through isolated Git worktrees, controlled datasets and filesystems, an experiment database, and a gated evaluator.

4. Experimental Setup

The experiments evaluate optimizer configurations across diverse agentic tasks, robustness settings, agent complexities, and long-horizon coding environments. Protocols control optimizer budgets, repetitions, models, traces, and evaluation interfaces while measuring improvement, stability, and efficiency.

  • 4.1. Benchmark Study: 120 experiments compare 8 optimizer configurations across 5 tasks, spanning math reasoning, tool use, multi-step reasoning, factual QA, and science QA.The benchmark uses three averaged iterations per configuration and examines whether gains reflect harness improvements rather than prompt-only changes.
  • 4.1. Benchmark Study: The benchmark protocol uses budget B = 8, with Claude Sonnet 4.5 as the default optimizer model and variants differing in tools, delegation, model choice, and access to VERO resources.VERO-AGENT includes Default, Orchestrator, and Resources-Only variants; Claude Code includes VERO Tools and Pure variants.
  • 4.2. Robustness Study: Robustness is tested by re-evaluating optimized commits with Claude Sonnet-4.5, GPT-4.1, Gemini 2.5 Flash, and Qwen3 target models.Commits are selected from Orchestrator variants on GAIA, GPQA, SimpleQA, and TAU-Bench Retail before substituting target models.
  • 4.3. Case Study: GAIA with Realistic Agents: The GAIA case study contrasts Pawn, a minimal agent with 4 tools and a 25-line prompt, with Knight, a sophisticated agent with 6 tools and a 140-line prompt.The pairing tests whether optimization can expand a minimal agent’s capabilities and refine an already-sophisticated agent.
  • 4.5. Optimizer Instructions: The GAIA instruction-template study runs each configuration 4 times with B = 5 and evaluates held-out GAIA, FACTS Search, and SimpleQA sets using lift, variance, and runtime.These measures capture improvement, optimization stability, and inference-time efficiency tradeoffs.
  • 4.4. Long-Horizon Coding Agent: The long-horizon case study uses Terminus-KIRA on 89 TerminalBench-2 tasks and exposes traces and datasets through either VERO tools or filesystem JSON artifacts.Three runs use Claude Code with sample budgets of B = 89 or B = 178; the full evaluation costs approximately $180.

5. Results and Discussion

VeRO-enabled optimization produces meaningful but task-, agent-, and instruction-dependent gains, with broader harness changes outperforming prompt-only variants. Improvements can trade off peak performance, transferability, diversity, and runtime efficiency.

  • 5.1. Benchmark Study Results: 0.61 average performance for VERO Default exceeded 0.54 for both Resources Only and GEPA, with the full harness yielding 8% average gains versus 2% from added tools alone.The reported +7% gap is consistent with broader harness changes beyond the components mutated by GEPA and Resources Only.
  • 5.1. Benchmark Study Results: Gains were consistent on tool-use tasks but limited on GPQA and MATH, and varying budgets did not explain the flat reasoning-task performance.The paper interprets this as greater optimization impact on tasks requiring more complex agentic behavior.
  • 5.3. Case Study: Optimization on Realistic Agents: Pawn achieved larger maximum lifts than Knight: +11.5% versus +6.9% on GAIA, +10.5% versus +5.6% on FACTS, and +13.3% versus +4.5% on SimpleQA.The authors relate this asymmetry to greater optimization headroom in the less sophisticated agent.
  • 5.3. Case Study: Optimization on Realistic Agents: Instruction templates trade variance for upside: Cookbook+Reasoning best served Pawn, while Minimal best served Knight; lower-variance templates produced more consistent but lower peaks.For Knight, Minimal reduces prompt size by 65% and removes cookbook access.
  • 5.3. Case Study: Optimization on Realistic Agents: Optimization gains did not uniformly transfer: Pawn gained +5.75% on GAIA but regressed -17.8% on SimpleQA after adding a complex verification tool.The tool helped multi-hop reasoning but added overhead for simple factual queries.
  • 5.3. Case Study: Optimization on Realistic Agents: Evidence-Based agents ran about 2× faster than Tool-Centric agents while maintaining competitive accuracy, reaching 26.2s versus 56.6s per Knight sample and 12.6s versus 32.8s for Pawn.The efficiency gain came from lightweight modifications and discouraging complex tool additions.
  • 5.4. Case Study: Long-Horizon Coding: Tools runs improved TerminalBench-2 pass rate by +3.4pp at B = 89 and +6.8pp at B = 178, while Filesystem-B=178 reduced crashes from 46.1% to 30.3% without increasing pass rate.Different runs found different fixes; Tools-B=178 instead improved pass rate through token compression.
  • 5.5. Interpretability: Prompt changes appeared in over 50% of phases after the first, while Orchestrator produced greater change diversity and entropy generally declined across phases.The decline suggests agents often return to prompt changes after more ambitious modifications fail to yield gains.

6. Limitations

The work identifies evaluation and methodological limitations that constrain its conclusions and point to future improvements.

  • Evaluation budgets cover calls rather than tokens or API cost, introducing variance in measured optimization performance.The study also does not compare alternative budget allocations, such as distributing evaluations across independent sessions.
  • The evaluation lacks human baselines, does not control public API instability, and remains vulnerable to reward hacking through leaked ground truth.

7. Conclusion

VeRO frames agent optimization as iterative code-based improvement and exposes both progress and unresolved gaps in the field.

  • VeRO enables coding agents to iteratively improve target agents through code modifications.
  • Current optimizers favor prompt edits over structural changes, while tool-use gains do not extend to reasoning tasks.
  • Agent optimization is tractable but far from solved, motivating VeRO’s release for benchmarking progress and developing improved models.

Impact Statement

The paper presents VeRO and a standardized benchmark as infrastructure for autonomous agent optimization, while emphasizing both its potential benefits and safety requirements.

  • VeRO and its standardized benchmark aim to lower the barrier to creating robust, self-improving systems.
  • Automating agent-as-code optimization could accelerate AI tools for scientific research, education, and accessibility while reducing reliance on manual engineering.
  • Self-evolving code requires safety guardrails against unintended behavior and circumvention of original safety constraints.
  • Transparency and interpretability become critical as agents build and refine other agents.
  • The authors encourage using VeRO to develop verification methods for autonomous code evolution alongside performance improvements.

A.1. VERO vs. VERO-AGENT

VeRO is a coding-agent-agnostic outer harness, while VERO-AGENT is one available implementation built around VeRO’s tools and abstractions.

  • VeRO exposes tools and hooks that can integrate with any coding agent able to consume them.
  • VERO-AGENT is a Python class that combines VeRO tools with LLM completion and tool-execution logic.
  • Different VERO-AGENT variants are configurations of the same class, differing in selected tools, tool settings, and completion settings.
  • ExperimentRunner, ExperimentViewer, and DatasetViewer let optimizer agents select, run, and inspect target-agent evaluations.
  • Table 5 catalogs VERO’s toolsets and links each toolset to a core VeRO abstraction.

A.2. Benchmark Study: Supplementary Materials

This section provides supplementary details and analyses for the benchmark study.

  • The section supplements the benchmark study with additional details and analyses.

A.2.1. DATASET SPLITS AND RATIONALE

Across benchmark tasks, optimizer benefits depend strongly on task demands, optimization budget, and modification strategy. Tool-use tasks show meaningful gains, whereas reasoning-heavy tasks remain near a capability ceiling; broader changes can produce larger peaks but also greater variance and regressions.

  • Cross-task results: 8–9% lift appears on tool-use tasks GAIA, TAU-Bench Retail, and SimpleQA, while GPQA and MATH show almost no gains across configurations.Table 2 summarizes average best-commit scores across the five benchmark tasks, with Figure 5 visualizing average lift by task and configuration.
  • Budget effects: GAIA reaches 0.195 at budget B = 8 from 0.119 at B = 2, then declines to 0.170 at B = 32 as longer trajectories become less reliable.The inverted-U pattern is associated with entropy collapse, prompt-centric drift, diminishing reliability, and later reverts.
  • Budget effects: GPQA remains within [0.600, 0.630] and MATH within [0.890, 0.897] across budgets, consistent with target-model capability ceilings rather than insufficient evaluation count.MATH’s 0.87 baseline is described as near-ceiling for the target model.
  • Modification outcomes: Individual modifications can trade gains across tasks: polished_band improves FACTS, GAIA, and SimpleQA simultaneously, while black_lake improves GAIA but sharply degrades SimpleQA.polished_band gains +10.45pp, +11.49pp, and +13.33pp respectively; black_lake gains +5.75pp on GAIA and loses 17.78pp on SimpleQA.
  • Optimizer strategies: The Evidence-Based strategy minimizes runtime and variance but limits breakthrough changes, whereas Cookbook+Reasoning produces diverse, high-variance modifications with the highest peaks for Pawn.Tool-Centric modifications produce moderate performance and variance; Minimal exploration can yield either breakthroughs or failures.
  • Optimization dynamics: Optimization often finds the main solution shape early, with diminishing returns after iterations 2–3 and later changes sometimes overfitting broader distributions or adding costly overhead.Early iterations commonly produce the largest GAIA improvements; later iterations can improve FACTS while regressing GAIA, and one peak SimpleQA result required 4× baseline runtime.
Loading 2602.22480v4…