Source-linked AI summary
VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
Caiqi Zhang, Rujun Han, Zifeng Wang, Zoey CuiZhu, Nigel Collier, Tomas Pfister, Chen-Yu Lee
TL;DR
Long-horizon artifacts are hard to verify, and common rollout-based methods rely on what the model already believes rather than environmental evidence. VeriHarness equips the generator’s model with evidence tools and complementary claim-checking tasks; across five benchmarks and two models, it leads evaluated baselines on selection scores, while evidence-backed revision improves performance over a single rollout.
Problem
Long-horizon artifacts can require tracing claims, recomputing quantities, and interpreting requirements across files, while common verification methods rely on rollouts rather than evidence from the environment.
Source: §1 · pp. 1–2Method
VeriHarness gives the generator’s model a workspace, evidence tools, and reusable skills to resolve disputed claims, challenge consensus and omissions, and guide artifact selection and revision.
Source: §1 · p. 2Results
Across five benchmarks, VeriHarness achieves the highest selection scores among evaluated baselines with both models; revision gains 6.2 points with Gemini 3.5 Flash and 6.4 with Claude Opus 4.8 over a single rollout.
Source: §1 · p. 3Takeaways & Limitations
Verification can be built around the existing model using its rollouts, environmental evidence, and accumulated experience; verification records could also provide claim-level signals for generator supervision.
Source: §8 · p. 12Verification is a multi-turn investigation that adds wall-clock time, a trade-off the authors target for professional deliverables where quality matters more than response time.
Source: Limitations · p. 12
Abstract
from arXiv · showhide
As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification capability can be strengthened with a fixed base model, without access to reference answers or grading rubrics at test time. Repeated sampling yields multiple rollouts that can contain complementary correct claims, but we need a reliable verification mechanism to determine which claims to trust. We first find that disagreement often exposes correct alternatives, while consensus can conceal errors. These observations motivate VeriHarness, which turns the underlying LLM a generator uses into an agentic verifier by giving it a workspace, evidence tools, and reusable verification skills. A disagreement resolver checks competing claims against environmental evidence, while a consensus challenger tests shared claims and searches for omitted requirements. Their findings guide the selection and revision of the final artifact. Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision further improves average performance, bringing gains over a single rollout to 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. We further show that verification skills can self-improve from failure feedback, demonstrating VeriHarness as a novel and critical approach for scaling long-horizon agentic verification. We release the full pool of approximately 26,000 rollouts across all five benchmarks and both models, produced at a cost of over $100,000, to support future research on agentic verification.
1. Introduction
Long-horizon artifacts are difficult to verify without reference answers or a stronger judge. VeriHarness uses the generator’s model, rollout comparisons, and environmental evidence to check claims and guide artifact selection and revision.
1. Introduction
- Roughly a third of agreed values are judged incorrect, while disputed claims often contain a correct alternative that the most frequent answer misses.These observations motivate checking alternatives against environmental evidence rather than relying on voting or selection alone.Source: §1 · p. 2, Figure 2 · p. 2
- VeriHarness combines a workspace, evidence tools, and reusable skills in a plug-and-play harness, with a resolver for disputed claims and a challenger for consensus claims and omissions.Findings guide selection and revision of the final artifact; skills can accumulate from development-task failure feedback.Source: §1 · pp. 2–3
- Across five benchmarks, VeriHarness has the highest selection score among evaluated baselines with both models; evidence-backed revision gains 6.2 points with Gemini 3.5 Flash and 6.4 with Claude Opus 4.8 over a single rollout.The harness also achieves higher held-out performance with evolved skills than with the human-authored library.Source: §1 · p. 3
- The paper identifies skill evolution from failure feedback as a contribution, including development from either an empty or human-authored library.On held-out tasks, skills evolved from an empty library outperform the human-authored library; evolving from the human-authored library adds 6.8 points on APEX-Agents and 3.7 on SpreadsheetBench 2 over that library.Source: §1 · pp. 2–3
2. Problem Formulation
The section formalizes verification of artifacts from repeated rollouts in a shared task environment, without access to reference answers or grading rubrics. It defines claim-level evidence checks and distinguishes unanimous claims from disputed ones.
2. Problem Formulation
- A task pairs a description with an environment, and each rollout contains a delivered artifact, final workspace state, and action trace.Repeated sampling forms a pool of rollouts whose artifacts make task-relevant claims.Source: §2 · p. 3
- The verifier selects or revises a rollout and records checks, findings, and unresolved issues, aiming to maximize artifact quality without access to the grader, references, or rubrics.All compared methods receive the same rollout pool.Source: §2 · p. 3
- Evidence checks test claims against the environment, and the verifier actively chooses checks to distinguish candidates, test constraints, or identify omitted requirements.Evidence may support, contradict, or correct a candidate claim; the worked example uses version history and source metadata to check revenue and currency.Source: §2 · p. 3, Figure 3 · p. 4
- A claim is consensus when its answer entropy is zero and disputed when entropy is positive; zero entropy measures unanimity, not correctness.The candidate set treats equivalent expressions as one value and omissions as a distinct value.Source: §2 · pp. 3–4
- On ten-rollout APEX-Agents pools, 34% of consensus values are incorrect; 74% of disputed claims contain a correct candidate, while the most frequent value is correct in only 47%.These findings motivate evidence checks that distinguish alternatives and actively test shared claims.Source: §2 · p. 4
3. VeriHarness
VeriHarness equips the generator’s same model with a workspace, evidence tools, a verification protocol, and reusable skills. Separate investigators check disagreements and challenge consensus; adjudication uses their evidence to select and revise the delivered artifact.
3.1. A General-Purpose Verification Harness
- The harness pairs the generator’s frozen model with a workspace, evidence tools, a verification protocol, and reusable skills for artifact-independent checking.The protocol assigns distinct checking tasks to separate contexts; only the skills contain artifact-specific knowledge.Source: §3 · p. 4, §3.1 · p. 5
3.2. Resolving Disagreement through Evidence
- For disputed claims, the resolver traces competing values to sources or computations, runs discriminating checks, and records which claims the evidence settles.It stops when further checks are unlikely to change the outcome; if evidence contradicts every candidate without establishing a new value, the claim remains open.Source: §3.2 · p. 5
3.3. Challenging Consensus
- The consensus challenger tests shared values, readings, and omissions against the environment, guided by skills that expose ways an apparently settled artifact could fail.It checks overlooked requirements against the task description and records its findings for adjudication.Source: §3.3 · pp. 5–6
3.4. Adjudication and Evidence-Backed Revision
- A fresh context reviews both evidence records alongside the task and rollouts, then outputs a base rollout, an evidence-supported revision plan, and unresolved claims.It selects the base with the fewest evidence-supported defects on substantive requirements, weighted by importance; if none is usable, it sets the base to empty.Source: §3.4 · p. 6
- The full harness applies the revision plan and delivers the resulting artifact with a verification record linking claims, checks, evidence, and verdicts.Unresolved claims remain marked as unsettled in the delivered record.Source: §3.4 · p. 6
4. Experiments
Across five workspace benchmarks and two frontier models, VeriHarness leads the evaluated selection baselines, and evidence-backed revision further improves results. The evaluation also tests transfer to existing agent harnesses and compares benchmark-specific scores and revisions.
4.1. Setup
- The five-benchmark evaluation uses Gemini 3.5 Flash and Claude Opus 4.8 as both generator and verifier, with the same ten-rollout pool per task and graders unavailable to the verifier.Baselines include voting, candidate judging, an agentic verifier without VeriHarness protocol or skills, and revision methods.Source: §4.1 · pp. 6–7
4.2. Results
- Most selection gains transfer to Gemini CLI, Claude Code, and Codex, though CLI variants fall to single-rollout performance on JobBench and, for Claude Code, SpreadsheetBench 2.With Flash, both CLIs match the paper’s implementation within two points on every benchmark; Opus transfer varies across benchmarks.Source: §4.2 · p. 7
- Evidence-backed revision improves all ten model–benchmark settings and raises the average gain over a single rollout to 6.2 points with Flash and 6.4 points with Opus.Aggregating the rollout pool without evidence checks stays below the harness on every benchmark.Source: §4.2 · pp. 7–8
- VeriHarness improves the single-rollout average by 4.4 points with Flash and 4.1 points with Opus, exceeding every selection baseline in all ten model–benchmark settings.The paper reports these main results in Table 2; gains exceed the cross-seed spread in every cell.Source: §4.2 · p. 7, Table 2 · p. 8
5. VeriHarness Can Self-Evolve
VeriHarness evolves reusable verification skills from development-task failures while keeping the model, protocol, and general tools fixed. On held-out tasks, learned libraries outperform the human-authored library, and human initialization improves final scores further.
5. VeriHarness Can Self-Evolve
- Each round proposes candidate libraries from development failures and grader feedback, retaining a candidate when its development score matches or exceeds the current library’s score.Held-out scores are recorded but never used for library selection; held-out verification has no reference answers or grading rubrics.Source: §5 · p. 9
- Starting from an empty library, evolution improves held-out scores by 11.0 points on APEX-Agents and 5.7 points on SpreadsheetBench 2, and exceeds the human-authored library on both.The model and protocol remain fixed in these comparisons.Source: §5 · p. 9
- Evolving from the human-authored library adds 6.8 points on APEX-Agents and 3.7 points on SpreadsheetBench 2, yielding the highest final score on both.The evolved library from human initialization reaches the empty-initialized library’s final trajectory score earlier on both benchmarks, with a larger reduction in rounds on APEX-Agents.Source: §5 · p. 9
- Learned skills encode concrete checks, such as matching requested items to evidence or aligning a spreadsheet growth-rate formula with the period in its label.The experiments use Claude Opus 4.8 on held-out subsets of APEX-Agents and SpreadsheetBench 2.Source: §5 · pp. 9–10
6. Analysis
The harness gains most when rollouts disagree, while its disagreement resolver and consensus challenger contribute complementary improvements. Consensus challenges are generally safe but can miss errors in later steps, and evolved skills become more specific while addressing that blind spot.
6. Analysis
- With Claude Opus 4.8, the resolver alone raised the five-benchmark average from 49.6 to 54.3, the challenger alone to 52.6, and both together to 56.1.Merging their investigations into one context cost 0.8 points; the same component ordering held with Gemini 3.5 Flash.Source: §6 · p. 10
- Selection gains were 0.6 points above the pool mean on 154 consensus pools, versus 6.1 points on 262 disputed pools; disputed-pool gains rose from 4.5 at low entropy to 7.7 at medium entropy.The pools are Claude Opus 4.8 APEX-Agents tasks; Figure 5 plots selection score minus pool mean by claim entropy.Source: §6 · p. 10, Figure 5 · p. 10
- The challenger never refuted a claim that every rollout got right, but about 70% of shared errors it missed lay in a later step than the correct intermediate result it checked.The analysis says directing it to check beyond where rollouts stopped requires prior knowledge of artifact failure modes, which the skill library supplies.Source: §6 · p. 10
- Of 95 checks in four evolved libraries, 5 restated human checks, 48 made them concrete, and 42 were new.About half of evolved checks turn a general human check into a procedure for a particular artifact type.Source: §6 · p. 10
- Evolved libraries learned to avoid treating a recomputation that matches the pool as confirmation, addressing a consensus blind spot.The proposer used development failures, verifier evidence records, and grader verdicts to identify shared values the verifier had wrongly confirmed.Source: §6 · pp. 10–11
7. Related Work
Prior work studies judging and test-time scaling through rollout assessment, decomposition, tool use, harness evolution, and skill libraries. VeriHarness is distinguished by using rollouts to choose evidence checks and by evolving verifier skills with held-out evaluation.
Verifying agent outputs
- Unlike methods that train verifiers or check a single output against given criteria, VeriHarness is training-free, uses the generator’s model, and checks rollout-derived targets against environmental evidence before returning a better rollout.Related work describes prior approaches including prompted judges [Zheng et al., 2023], labeled-data verifiers [Cobbe et al., 2021; Lightman et al., 2024; Zhang et al., 2025], and tool-using verifiers [Yuan et al., 2026; Zeng et al., 2026b; Zhang et al., 2026d; Zhuge et al., 2025].Source: §7 · p. 11
Test-time scaling for agents
- Where rollout voting, tournaments, and aggregation operate on outputs, VeriHarness brings evidence acquisition to tasks without executable tests and uses rollouts to decide what to check.Prior test-time scaling approaches include voting [Chen et al., 2023; Wang et al., 2023], tournaments [Kim et al., 2026], and aggregation [Lee et al., 2026b].Source: §7 · p. 11
Harnesses and self-evolution
- Unlike harness evolution from execution traces or skill-library growth, VeriHarness evolves verifier skills, admits candidates on development data, and evaluates on held-out tasks without deployment ground truth.The passage contrasts this with harness methods that can overfit when development and evaluation sets coincide (Wang et al., 2026).Source: §7 · p. 11
8. Conclusion
VeriHarness uses the generator’s model and environmental evidence to resolve disagreement and challenge consensus. Across five workspace benchmarks and two models it leads evaluated baselines, and its development-feedback skills outperform human-authored skills on held-out tasks.
8. Conclusion
- VeriHarness uses the generator’s own model to resolve disagreement and challenge consensus through environmental evidence.The harness’s findings guide selection and revision of the final artifact.Source: §8 · p. 11
- Across five workspace benchmarks and two models, VeriHarness achieves the highest selection scores among evaluated baselines, while revision improves average performance over one rollout by more than six points with each model.The conclusion also reports that skills evolved from development feedback outperform the human-authored library on held-out tasks, and human initialization further improves final performance.Source: §8 · pp. 11–12
- The verification record links each claim to its check, evidence, and verdict, offering a claim-level signal that could supervise the generator.The conclusion identifies converting these records into training data for long-horizon agents as a natural next step.Source: §8 · p. 12
Limitations
VeriHarness is bounded by its ten-rollout evaluation budget, same-model verifier setting, and added latency. Its claim-level checks also cannot resolve shared blind spots when required claims go unexamined or available evidence is inconclusive.
Limitations
- Reported results use ten rollouts per task, giving all Table 2 baselines the same generation budget; verification costs are reported separately.The ten-rollout budget is described as the standard test-time-scaling budget.Source: Limitations · p. 12
- The harness does not assign scalar scores to individual rollouts; converting claim-level judgments into calibrated rollout scores, such as reward signals, remains future work.Artifacts are judged claim by claim against evidence because their claims may be independently correct or incorrect.Source: Limitations · p. 12
- The study restricts verification to the generator’s model; a stronger verifier could raise scores, but would confound harness gains with model capability differences.Separating those contributions is outside the paper’s scope.Source: Limitations · p. 12
- Verification adds wall-clock time through multi-turn investigation; the authors target professional deliverables where quality matters more than response time.They compare this trade-off with deep research systems that run for many minutes to return a better answer.Source: Limitations · p. 12
A. Claim-Level Analysis Protocol
- The claim-level analysis is an offline computation from ten-rollout Claude Opus 4.8 pools on 434 APEX-Agents tasks, using benchmark rubrics unavailable to the verifier.It covers 1,692 claims; the rubric access is specific to this analysis, not the verifier’s test-time setting.Source: A. Claim-Level Analysis Protocol · p. 19
B. Why Same-Model Verification Can Improve Outputs
- Shared blind spots can persist when required claims go unexamined or available evidence cannot settle them, so same-model verification depends on finding informative checks.The paper describes different information, narrower objectives, and accumulated checking experience as ways a fixed model can detect generation errors.Source: B. Why Same-Model Verification Can… · p. 19
C. Verification Procedure and Delivery Details
- The procedure opens two independent verifier contexts for disagreement resolution and consensus challenging, then uses their evidence records for adjudication and delivery.The program supplies the contexts and information access; the model chooses claims, checks, and stopping points.Source: C. Verification Procedure and Delivery… · p. 20
D. Experimental Details
Experiments compare verification methods on five workspace benchmarks using two models and shared ten-rollout pools. The released pool supports reproducing pool-dependent results, including expected selection-oracle scores across pool sizes.
D. Experimental Details
- The five benchmarks span investment-banking, legal, consulting, office, coding, spreadsheet, and other workplace tasks, with WorkBuddy security excluded because API calls consistently triggered safety alerts.The experiments use Workspace-Bench Lite’s 100-task subset and WorkBuddy code, office, and web domains.Source: D. Experimental Details · p. 20
- Each benchmark-model pair uses ten frozen rollouts from the benchmark’s agent scaffold, so every method evaluates the same candidates.The pools include each rollout’s artifact, final workspace state, and action trace.Source: D. Experimental Details · pp. 20–21
- The primary comparison selects an existing rollout, while full VeriHarness can apply the adjudicator’s revision plan in one editing pass without another investigation.Methods that generate a new artifact directly from the pool are outside the selection comparison.Source: D. Experimental Details · p. 21
- Benchmark-native scores are reported as three-seed means, gains over the single-rollout score, and cross-seed standard deviations; judge-scored comparisons use same-round differences to control judge drift.Metrics include task success, mean rubric score, task-weighted domain means, and headline accuracy, depending on benchmark.Source: D. Experimental Details · p. 21
E. Released Rollout Pool
- The released pool covers both models across all five benchmarks, with ten rollouts per task and each rollout’s trajectory, delivered files, and grader score.It excludes benchmark content but retains official task identifiers so inputs can be obtained from the original benchmarks.Source: E. Released Rollout Pool · p. 22
- Figure 6 plots the expected score of the best rollout among N draws from each ten-rollout pool, with N=1 representing a single rollout and N=10 the selection oracle.The curves use released-pool grader scores and each benchmark’s native scale; the passage gives no curve values.Source: Figure 6 · p. 22, E. Released Rollout Pool · p. 22
F. Ablation of Harness Components
The section examines how harness components, runtime choices, and verification skills affect performance, then details self-evolution experiments and their scope. It also analyzes shared errors that survive the consensus challenger.
F. Ablation of Harness Components
- Removing skills lowers scores on every benchmark, while combining the resolver and challenger in one context costs less but also lowers scores on every benchmark.The ablations use the same model, tools, adjudication, and delivery; revision is applied because consensus errors can be realized only through revision.Source: F. Ablation of Harness Components · p. 22, Table 4 · p. 23
G. Why a Dedicated Runtime
- The protocol and skills recover most of their gain inside existing agent CLIs, but weaker Opus results on JobBench and SpreadsheetBench 2 suggest host-runtime effects.The dedicated runtime explicitly executes separate investigations, passes evidence records to adjudication, and configures workspace tools consistently.Source: G. Why a Dedicated Runtime · p. 23
H. Tools and Skills
- Figure 7 reports per-task invocation rates for each of eighteen skills, pooled across both models and split by benchmark.An invocation includes loading a skill into context, reading its file, or running one of its scripts.Source: Figure 7 · p. 24
- The eighteen skills cover evidence reading, resolving disagreements, challenging shared errors, and revising deliverables; benchmark usage follows deliverable type, with heavier spreadsheet-skill use on SpreadsheetBench 2.Document-format evidence skills are used at similar rates across benchmarks, while type-specific skills track the requested deliverables.Source: H. Tools and Skills · p. 24
I. Self-Evolution Details
- Candidate libraries replace the current library only when they produce a nonnegative paired gain on development tasks, so a retained update can still lower held-out scores.The final library follows the pre-fixed schedule rather than selection by held-out score.Source: §I · p. 26
- 49.6 is first reached at round 8 by C and round 1 by D on APEX-Agents; on SpreadsheetBench 2, C reaches 38.5 at round 8 and D reaches 39.4 at round 7.Round counts do not establish a compute or elapsed-time advantage; both trajectories include held-out regressions, and the figures do not establish statistical significance.Source: §I · p. 26
- Learned libraries help on unseen tasks within the evaluated splits, but generalization beyond those task distributions and improvement without development grader feedback remain unestablished.Artifact scores combine the learned skills' effects on checking, adjudication, and delivery.Source: §I · p. 27
J. Why Shared Errors Survive
- On APEX-Agents with Claude Opus 4.8, the analysis matches challenger-upheld shared claims to rubric items and classifies the surviving errors by claim-to-item relation.Table 7 reports the share of unanimous errors that the challenger upheld.Source: J. Why Shared Errors Survive · p. 27, Table 7 · p. 27
K. Compute Cost
VeriHarness is on the cost–gain frontier for both models, with or without revision, and selection costs differ by model. The comparison uses a common accounting for compute and gain.
K. Compute Cost
- VeriHarness lies on the cost–gain frontier for both models, with or without revision; selection costs $1.44 per task with Gemini 3.5 Flash and $3.92 with Claude Opus 4.8.The plot uses cached-input pricing; several baseline and CLI markers are estimates rather than metered runs.Source: K. Compute Cost · p. 27, Figure 8 · p. 28