PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Verification in Agent Harnesses

Zhengxin Zhang, Erkang Zhu, Yin Lin, Claire Cardie, Bolin Ding, Jingren Zhou

arXiv:2610.34577v1cs.AI

TL;DR

The paper addresses limited comparative guidance on how agent harnesses should organize verification and whether its gains justify added computation. It evaluates lifecycle-based designs and finds that independent review offers an accuracy–cost tradeoff across coding benchmarks, with gains on VisualBench as well.

  • Problem

    Evidence remains limited on how to design and scale harness verification, when designs improve performance, and whether gains justify additional computation.

    Source: §1 · p. 1

  • Method

    The paper compares 17 verification designs across four lifecycle layers, varying intervention timing, checker, and how feedback leads to revision.

    Source: §1 · pp. 1–2

  • Results

    Independent review improves average accuracy by 5.9–9.5 percentage points on TB and 3.9–6.6 on SWE at 1.13–1.25× and 1.00–1.04× baseline cost, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results provide empirical guidance for verification design and support independent review as a practical axis of test-time scaling.

    Source: §6 · p. 9

Abstract

from arXiv · show

Agent harnesses enable large language models to act through tools, observe the results, and iteratively work toward a task goal. Verification checks and revises agents' work to improve output quality. Yet there is limited guidance on how harnesses should organize this effort. We study verification as an axis of test-time scaling by examining when to intervene, who checks the work, and how checking leads to revision. Our framework spans four layers of the agent lifecycle: initial instructions, mid-turn reminders, completion gates, and independent review. We compare 17 designs across seven frontier coding models on Terminal-Bench (TB) and SWE-bench Verified (SWE), and introduce VisualBench for expert evaluation of visual and interactive artifacts. Initial instructions and execution-time reminders yield modest, model-dependent average gains on the coding benchmarks, without substantially increasing average output token use. Completion gates can yield larger gains but may substantially increase cost. Independent review by a critic offers a more favorable accuracy-cost tradeoff, improving average accuracy by 5.9-9.5 percentage points on TB and 3.9-6.6 on SWE at 1.13-1.25 and 1.00-1.04 times baseline cost, respectively. On VisualBench, independent review achieves net preferences of 13.2-23.8 points over the baseline across its four designs, exceeding the evaluated L0-L2 designs. Additional review rounds or independent critics further improve average TB accuracy by up to 3.5 percentage points. Together, these results provide empirical guidance for verification design in agent harnesses and support independent review as a practical axis of test-time scaling.

1 INTRODUCTION

Evidence is limited on how agent harnesses should organize verification and whether its gains justify added computation. The paper compares lifecycle-based verification designs and finds that independent review offers a favorable accuracy–cost tradeoff across coding benchmarks, with gains also on VisualBench.

1 INTRODUCTION

  • Systematic evidence remains limited on how to design and scale harness verification, when designs improve performance, and whether gains justify additional computation.Source: §1 · p. 1
  • The framework organizes verification around when to intervene, who checks the work, and how findings lead to revision across four agent-lifecycle layers.The layers are initial instructions, execution-time reminders, completion gates, and independent review.Source: §1 · pp. 1–2
  • Independent review improves average accuracy by 5.9–9.5 percentage points on TB and 3.9–6.6 on SWE at 1.13–1.25× and 1.00–1.04× baseline cost, respectively.The costs include critic review and agent revision.Source: §1 · p. 2
  • On VisualBench, four independent-review designs outperform evaluated L0–L2 designs, with net preferences of +13.2–+23.8 points over baseline.Source: §1 · p. 2
  • The study compares 17 designs across seven frontier coding models on TB and SWE, and introduces VisualBench for expert evaluation of visual and interactive artifacts.Source: §1 · p. 2
  • Initial instructions and execution-time reminders yield modest, model-dependent coding gains, while completion gates can yield larger gains but may substantially increase cost.Additional review rounds or independent critics further improve average TB accuracy by up to 3.5 percentage points.Source: §1 · p. 2

2 RELATED WORK

Prior research supplies assessment, self-correction, tool-assisted checking, and agent-workflow mechanisms. The paper addresses limited comparative guidance on how harnesses should organize checking and revision across models, benchmarks, and verification effort.

2 RELATED WORK

  • Prior verification work includes learned solution assessment, critique-driven self-correction, and tool-assisted checks that use execution evidence to guide repair.Source: §2 · p. 2
  • Agent-harness research studies tool interaction, execution platforms, structured repair, and how computation and information are used across attempts.Source: §2 · p. 2
  • Existing mechanisms and workflows offer limited comparative guidance on organizing harness checking and revision, motivating a systematic study across models, benchmarks, and verification effort.Source: §2 · pp. 2–3

3 VERIFICATION DESIGNS

The verification designs follow the agent lifecycle, varying when checks occur, who performs them, and how feedback leads to revision. They range from prompt guidance to a separate critic reviewing the deliverable.

3 VERIFICATION DESIGNS

  • L0 adds verification guidance to the initial prompt, varying whether the same instruction appears in the system prompt or initial user request.For visual or interactive tasks, the instruction also asks the agent to render the artifact and exercise core interactions.Source: §3 · p. 3
  • L1 reminders arrive after tool results and before the next model call, with periodic or after-edit triggers and system- or user-message variants.Only 6.2–13.6% of detected editing turns were followed by a recognized checking command before another edit or the run’s end.Source: §3 · pp. 3–4
  • L2 completion gates reopen work for checking and revision, varying the requested action, message role, automated marker, and intervention timing.Source: §3 · p. 4
  • L3 assigns review to a separate critic in a fresh context, which checks the deliverable and returns findings for agent revision.The study varies critic context and whether review is task-first or also guided by the agent’s claims and assumptions.Source: §3 · p. 4

4 EXPERIMENTAL SETUP

The experiments evaluate coding agents on two automatically graded benchmarks and visual artifacts through blinded expert comparisons. They use fresh sandboxes and report accuracy and cost for coding tasks.

4 EXPERIMENTAL SETUP

  • Seven coding models are evaluated on selected Terminal-Bench 2.1 and SWE-bench Verified tasks, with three trials per model–task pair.Both coding benchmarks use automatic graders; selection spans task difficulties using baseline success rates or human time-to-fix tiers.Source: §4 · p. 4
  • VisualBench contains 18 visual and interactive tasks, with blinded expert comparisons of functionality and visual quality against the baseline.Raters compare wins, losses, ties, and cases where both artifacts fail; two trials are configured per task, model, and condition.Source: §4 · p. 4
  • Trials run in fresh E2B sandboxes orchestrated with Harbor, and independent-review critics access the same workspace to examine the deliverable and run checks.The critic cannot modify the deliverable; the agent can revise it using the returned feedback.Source: §4 · pp. 4–5
  • For TB and SWE, the study reports task-solving accuracy and cost averaged across trials for each model–task pair.Cost includes the coding agent, critic, and subsequent revisions, priced separately by model.Source: §4 · p. 5

5 RESULTS

Verification designs differ in their accuracy-cost trade-offs: independent review improves coding and visual results at modest cost, while completion-time self-checking often costs more than its gains warrant. Initial instructions and execution-time reminders show model-dependent coding gains, and further review rounds or critics can improve accuracy.

5.1 L0: INITIAL INSTRUCTIONS

  • System-role L0 instructions improve accuracy by +3.3 points on TB and +3.9 on SWE, versus +1.2 and −1.1 for user-role instructions.System/user instructions cost 1.08/1.03× baseline on TB and 0.94/0.93× on SWE; output tokens and turns remain close to or below baseline.Source: §5.1 · p. 5

L0 Takeaways

  • L0 benefits vary across models, so the authors recommend concise system-prompt guidance that leaves room for other harness instructions.In one case, guidance helped correct an edge-orientation error; in another, testing led an agent to delete a repository and fail.Source: §5.1 · p. 6, L0 Takeaways · p. 6

5.2 L1: EXECUTION-TIME REMINDERS

  • L1 reminders yield model- and benchmark-dependent gains; after-edit user reminders average +5.4 points on TB but −0.3 on SWE.Across L1 designs, output tokens fall 14–25% on TB and 7–10% on SWE; costs range from 0.90–1.12× and 0.85–0.89× baseline, respectively.Source: §5.2 · p. 6, Figure 2 · p. 7

L1 Takeaways

  • After-edit reminders can prompt task-relevant checks, but execution after a reminder does not guarantee sufficient verification coverage.Some agents checked configuration, responses, and logs, while others missed invalid substitutions or runtime criteria.Source: §5.2 · p. 7, L1 Takeaways · p. 7

5.3 L2: COMPLETION GATES

  • L2 gains can be large, but rejection designs cost 1.48–1.76× baseline, while early user rechecking costs 1.13/1.20× on TB/SWE.Early rejection gains reach +6.0/+6.2 points on TB/SWE; early rechecking yields +4.8/+4.1 points and +2.7 net preference on VisualBench.Source: §5.3 · p. 7, Figure 3 · p. 8
  • Completion gates create repair opportunities but do not ensure a valid final artifact: agents can correct defects or leave harmful test changes in place.The examples include a repaired regex that passes and deployment tests committed to served branches, which cause failure.Source: §5.3 · pp. 7–8

5.4 L3: INDEPENDENT REVIEW

  • L3 gains are +5.9–9.5 points on TB and +3.9–6.6 on SWE at 1.13–1.25× and 1.00–1.04× baseline cost, respectively.All four independent-review designs achieve positive VisualBench net preference (+13.2 to +23.8 points), exceeding evaluated L0–L2 designs; trajectory-based designs cost 29–43% more on TB and SWE than changes-based designs.Source: §5.4 · p. 8
  • Four rounds reach 66.1% TB accuracy and four critics reach 65.8%; four critics improve accuracy by 3.1 points over one critic at 1.33× total cost.Four critics approach four-round accuracy at 19% lower total cost, including revisions.Source: Figure 4 · p. 9, §5.4 · p. 9

6 CONCLUSION

The paper compares verification designs across lifecycle stages and reports that independent review offers a favorable accuracy–cost tradeoff on coding benchmarks and stronger VisualBench preferences than evaluated L0–L2 designs.

6 CONCLUSION

  • Independent review achieves a more favorable accuracy–cost tradeoff on TB and SWE than completion gates, while its four VisualBench designs exceed evaluated L0–L2 designs.On VisualBench, independent review earns net preferences of 13.2–23.8 points over baseline.Source: §6 · p. 9
  • Initial instructions and execution-time reminders yield modest, model-dependent coding gains without substantially increasing average output tokens.Source: §6 · p. 9
  • Additional review rounds or critics further improve average TB accuracy by up to 3.5 percentage points.Source: §6 · p. 9

ETHICS STATEMENT

VisualBench’s human evaluation used internal company data-team raters under an approved, blinded comparison protocol.

ETHICS STATEMENT

  • The VisualBench human evaluation received ethics approval, and raters provided informed consent.Source: ETHICS STATEMENT · p. 10
  • Raters were internal company data-team members compensated under their existing company contracts.Source: ETHICS STATEMENT · p. 10
  • The interface explained the study and criteria while hiding verification conditions and randomizing artifact placement.Appendix C documents the protocol, tasks, instructions, and interface.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

The statement assigns research ideas, system design, and manuscript writing to the authors, with generative AI assisting implementation and polishing.

AI USE STATEMENT

  • The authors developed the research ideas and system design, led manuscript writing, and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

B Additional coding results

The supplied passage lists appendix sections on prompts and execution rules and on VisualBench tasks and evaluation.

B Additional coding results

  • The appendix listing gives “Prompts and execution rules” page 62 and “VisualBench tasks and evaluation” page 29, preserving the respective page references for both entries.Source: B Additional coding results · p. 13

A.1 TASK SELECTION FOR TERMINAL-BENCH AND SWE-BENCH

This appendix describes task sampling and supporting coding results, including review-scaling accuracy–cost comparisons and the work that follows execution reminders and completion gates.

A.1 TASK SELECTION FOR TERMINAL-BENCH AND SWE-BENCH

  • Task selection balanced easy, intermediate, and hard difficulties to observe both improvements and regressions.The task selection procedure is described for Terminal-Bench and SWE-bench.Source: §A.1 · p. 13
  • The 31-task Terminal-Bench study set balanced tasks usually solved, solved inconsistently, or usually failed in a baseline pilot.The pilot covered the 89-task version 2.1 registry and used four coding models to guide selection.Source: §A.1 · p. 13
  • SWE-bench Verified began with 30 tasks evenly sampled across three human time-to-fix tiers, then adjusted to include improvement headroom and regression detection.The tiers were under 15 minutes, 15 minutes to one hour, and over one hour.Source: §A.1 · p. 13

A.2 API COST PRICING

  • Table 4 lists the model rates used for cost calculations, with Alibaba Cloud Model Studio pricing as the reference.The rates are expressed in USD per million tokens.Source: Table 4 · p. 13, §A.2 · p. 13

B.1 ABSOLUTE ACCURACY BY VERIFICATION DESIGN

  • Accuracy figures compare baseline and verification-design values, while per-model figures repeat that comparison across coding models.Figure 5 presents absolute accuracy on TB and SWE; Figures 6–12 show the same comparison for each model.Source: §B.1 · pp. 13, 15, Figure 5 · p. 14, Figure 6 · p. 15

B.4 REVIEW SCALING AND TOTAL COST

  • 65.8% accuracy at $0.62 per trial with four critics compares with 66.1% at $0.77 for four rounds.One critic and one round achieve 62.7% at $0.47; two rounds reach 65.6% at $0.61, while two critics reach 64.4% at $0.60.Source: Figure 23 · p. 27, §B.4 · p. 27

B.6 L1: CHECKING DURING EXECUTION

  • After-edit user reminders reach 80–100% of runs, and recognized checks follow 58–72% of reminder events.Table 8 reports reminder coverage and checking or execution within the next two agent turns.Source: §B.6 · p. 28, Table 8 · p. 28

B.7 L2: WORK AFTER A COMPLETION GATE

  • After L2 user rejection, models take 12–19 additional steps at the median, with file-changing actions in 86–96% of triggered runs.Table 9 reports subsequent steps and file-changing actions after completion-gate interventions.Source: §B.7 · p. 28, Table 9 · p. 28

C.1 COMPLETE TASK CATALOG

The catalog presents VisualBench task briefs, deliverables, and human-rating instructions for interactive visual, graphics, and application-building tasks, with detailed requirements for their behavior, appearance, and artifacts.

C.1.1 BLENDER DESK SCENE; C.1.2 INTERACTIVE CLOTH

  • The Blender and cloth tasks require, respectively, a composed, lit desk-scene render with a turntable and stable, interactive cloth simulation.The cloth brief additionally calls for live energy and constraint-error readouts, adjustable simulation parameters, and views of both the structure and shaded fabric.Source: §C.1.1 · p. 31, §C.1.2 · p. 32

C.1.3 FLOWLINE KANBAN

  • Flowline is a populated team kanban board with card editing, filtering, bulk actions, undo and redo, WIP enforcement, keyboard use, and persistent state.In Progress is initially at its limit of three; bulk actions should be undoable as one step, and board data can be exported and imported as JSON.Source: §C.1.3 · p. 33

C.1.4 SALES DATA EXPLORER

  • The sales explorer must reveal online and retail trends across six regions while transparently repairing messy data, accounting for exclusions, and avoiding misleading displays.A missing value in one series should create a gap rather than a zero; excluded rows must be counted and individually explained.Source: §C.1.4 · p. 34

C.1.5 BREAKOUT; C.1.6 2D PLATFORMER

  • Breakout calls for varied bricks, paddle-controlled play, power-ups, progression, and persistent scoring, while the platformer requires responsive jumping, hazards, coins, and two-level progression.The platformer brief specifies variable jump height, coyote time, a following camera, respawning, and a win screen; Breakout includes multiple levels, pause, and restart.Source: §C.1.5 · p. 35, §C.1.6 · p. 35

C.1.7 TOWER DEFENSE

  • The tower-defense task combines path-based waves and constrained tower placement with distinct upgradeable towers, varied enemies, resource and life tracking, and win-or-lose outcomes.Players can call waves early, pause or speed up play, and restart after either outcome.Source: §C.1.7 · p. 36

C.1.8 VOXEL SANDBOX

  • The voxel sandbox specifies an effectively endless, streamed world and first-person building, mining, inventory, crafting, day-night lighting, and persistence across reloads.The brief calls for offline operation using the locally supplied three.min.js, with smooth performance while moving through the world.Source: §C.1.8 · p. 37

C.1.9 GENERATIVE ART SERIES

  • The generative-art task asks for a reproducible series whose pieces share a visual style but differ in composition, with seed controls and print-resolution export.The brief also emphasizes deliberate composition, expressive marks and palettes, and progressive rendering that remains responsive.Source: §C.1.9 · p. 38

C.1.10 VECTOR BOOLEAN EDITOR

  • The vector-editor brief centers on computing editable path results for union, intersection, difference, and exclusion without geometry libraries or SVG-based workarounds.The resulting path must remain available for further editing and boolean operations.Source: §C.1.10 · p. 39
  • The vector editor must combine closed Bézier paths, four boolean operations, and editable results that remain usable in subsequent operations.The brief also calls for robust handling of touching or overlapping edges, holes, disjoint pieces, and self-intersections, plus overlays, undo/redo, and SVG export.Source: §C.1.10 · p. 39

C.1.11 CITY IN A STORM / C.1.12 STORMY OCEAN

  • The storm-scene briefs require a raymarched city over reflective water and a separate open-ocean simulation, both with dynamic weather, lightning, mouse-controlled views, and smooth full-screen rendering.The city brief emphasizes towers, unevenly lit windows, and broken reflections; the ocean brief specifies multi-scale waves, evolving foam, wave light scattering, and rain disturbing the surface.Source: §C.1.11 · p. 40, §C.1.12 · p. 41

C.1.13 GRIDWORK SPREADSHEET

  • Gridwork must be a 40-by-20-or-larger spreadsheet with A1 navigation, formulas and functions, dependency updates, actionable error handling, range tools, sorting, undo/redo, and persistence.Formula changes should update downstream cells without recomputing unrelated cells; cycles and other errors must be reported without hanging or taking down the sheet.Source: §C.1.13 · p. 42

C.1.14 ANIMATED CONCERT POSTER

  • The animated synthwave poster must use exact event text, integrated typographic hierarchy, layered lighting and depth, and multiple distinct seamless animations.It must be a standalone SVG with generic fonts, no external references, and a viewBox that stays crisp when enlarged.Source: §C.1.14 · p. 43

C.1.15 3D LAMP CONFIGURATOR

  • The lamp configurator must provide adjustable components, shade shapes, finishes, and lighting, with animated transitions, current configuration and price, saved and shareable settings, and interactive viewing.The brief also requests reset, damped orbit controls, zoom, resize handling, and a higher-resolution PNG download.Source: §C.1.15 · p. 44

C.1.16 GPU PARTICLE FIELD

  • The GPU particle field must sustain a million particles interactively, using an evolving curl-noise flow with momentum and trails, mouse attraction or repulsion, depth cues, and visible count and frame rate.Particles should recycle when they drift out, and resizing should not require rebuilding the world.Source: §C.1.16 · p. 45

C.1.17 PRODUCT HERO SCENE

  • The product hero task calls for a believable wireless headphone assembled from primitives, photographed-style studio lighting and materials, and interactive, smoothly animated exploration.Requested interactions include damped orbit and zoom, idle animation, transitioning color variants, an exploded view with labels, and clickable detail hotspots.Source: §C.1.17 · p. 46

C.1.18 FLEET TERMINAL DASHBOARD

  • The fleet dashboard must be a live multi-pane terminal interface that surfaces service health and trends, scrolls logs, updates plausible values, responds visibly to status changes, and adapts to narrow terminals.The deliverables include a 30-second-or-longer asciinema recording showing live updates and a status transition, plus an exported frame.Source: §C.1.18 · p. 47

C.2 HUMAN EVALUATION PROTOCOL

The human evaluation compared verification designs with matched baselines on the same tasks and coding models, using blinded expert judgments of task-specific functionality and visual quality. The protocol specified the task environment, rater briefing, interaction checks, and rules for choosing a preference, tie, or both-fail outcome.

C.2 HUMAN EVALUATION PROTOCOL

  • Seven designs were compared with the baseline for the same task and coding model: three representative L0–L2 interventions and four L3 review variants.The L3 variants used changes-based or trajectory-based review, each with or without required visual inspection; review depth and width were studied separately on Terminal-Bench.Source: §C.2 · p. 48
  • Tasks provided a description and supporting files without a reference implementation, and ran in an environment with Python, Chromium, and ffmpeg, plus other tools as needed.Browser artifacts ran offline, with bundled libraries only where the task permitted them.Source: §C.2 · p. 48
  • Raters from an internal company data team received a study briefing and worked example; the evaluation had ethics approval, and raters consented and were compensated under existing company contracts.Source: §C.2 · p. 48
  • Conditions were hidden and artifact sides randomized; raters saw the task brief, judging criteria, and task-specific interaction steps, and could inspect artifacts side by side or individually.They interacted with live artifacts when available and inspected recordings or renders when appropriate.Source: §C.2 · p. 48
  • Raters judged artifacts against the task’s functional and visual requirements, following its interaction checklist and using full-size views or focusing keyboard-controlled artifacts when needed.They chose the better artifact, a tie when both worked and quality was indistinguishable, or both-fail only when neither made a meaningful attempt; poor performance alone did not warrant both-fail.Source: §C.2 · p. 48

C.3 EXPERT EVALUATION INTERFACE

The evaluation interface presented two Blender desk-scene deliverables alongside task instructions and judging controls. Raters could inspect both artifacts and record a preference, tie, or failure of both.

C.3 EXPERT EVALUATION INTERFACE

  • The Blender desk-scene interface displayed rendered images and turntable videos for both artifacts, with expandable task guidance and controls for preference, tie, or both-fail judgments.Raters could inspect the artifacts side by side or at full size, and optionally explain their decision in a comment.Source: §C.3 · p. 49, Figure 24 · p. 49

C.4 DETAILED VISUALBENCH RESULTS

VisualBench results vary across L0–L2 designs, while all four independent-review designs receive positive net preferences. The accompanying trajectories and execution rules illustrate how checking, review, and revision are implemented.

C.4 DETAILED VISUALBENCH RESULTS

  • All four L3 review designs receive positive net preferences (+13.2 to +23.8 points); changes-based review scores higher than trajectory-based review in both comparisons.Required visual inspection raises changes-based review from +21.2 to +23.8 points and trajectory-based review from +13.2 to +21.6 points.Source: Table 12 · p. 50, §C.4 · p. 50
  • Evaluated L0 and L1 designs have negative net preferences, while early user rechecking is +2.7 points, with substantial variation across models.The L0 system instruction is −3.2 points; the L1 after-edit user reminder is −3.2 points.Source: §C.4 · p. 50, Table 12 · p. 50

D.1 OVERVIEW OF THE PAIRED TRAJECTORIES

  • Ten paired trajectories span all four layers and show both repairs after checking and cases where defects or requirements remain unresolved.Examples include corrected reversed edges and a confirmed cancellation-cleanup defect without a completed repair.Source: §D.1 · p. 51, Table 13 · p. 51, Figure 25 · p. 52

E VERIFICATION INSTRUCTIONS AND EXECUTION RULES

  • Initial instructions and execution-time reminders ask agents to run changes at their real interface, inspect actual output, and fix detected problems.The reminder is delivered after tool results and before the next model call.Source: §E.1 · p. 62, §E.2 · p. 62

E.3 L2: COMPLETION GATES

  • Completion gates reopen work after a turn without a tool call; designs vary the requested check, message role, and trigger timing.Some variants trigger at the next turn boundary after 80% of the task time limit if the agent has not stopped.Source: §E.3 · p. 62

E.4 L3: INDEPENDENT REVIEW

  • Independent critics inspect the live sandbox and verify requirements through observed probes; their verdicts require demonstrated evidence rather than a clean read.The critic is instructed not to mutate the deliverable and to report defects for the agent to address.Source: §E.4 · pp. 63–64

E.5 REVIEW DEPTH AND WIDTH

  • Review depth permits up to n critic rounds subject to remaining budget, while review width uses n separate critics on the same delivery.Width reviews run sequentially, and the agent’s subsequent revision is not reviewed again in the width sweep.Source: §E.5 · p. 65

E.5.1 PROMPTS FOR ADDITIONAL REVIEW ROUNDS

  • Later review rounds rebuild the requirement checklist, re-probe failed items after rework, and spot-check previously met items for regressions.Prior feedback serves as a coverage map, not as authority for the new verdict.Source: §E.5.1 · p. 65
Loading 2610.34577v1…