PaperlayerView exact revision on arXiv

Source-linked AI summary

Evidence-First Agentic Verification

Reza Sanayei, Razvan-Gabriel Dumitru, Sunjie Hou, MohammadHossein Rezaei, Haris Riaz, Robert Vacareanu, Minglai Yang, Ying Liu, Mihai Surdeanu, Daniel Yue Zhang

arXiv:2610.17724v1cs.AIcs.LG

TL;DR

Rubric-based evaluation of long-horizon agent tasks must assess evidence spread across state and trace, where judges may miss relevant evidence or accept unsupported claims. Sauron independently verifies claims from both sources before using them in verdicts, and leads the reported comparisons across PA100, PAFault, and TRAIL. Its controlled fault-injection results do not establish robustness to policies optimized to exploit Sauron.

  • Problem

    In long-horizon agent tasks, evidence for rubric criteria can be spread across state and trace, while judges may overlook evidence or accept unsupported claims.

    Source: §1 · pp. 1–2

  • Method

    Sauron investigates state and trace, independently validates claims against their sources, and uses bounded queries and joint investigation of related criteria.

    Source: §1 · p. 2

  • Results

    Across PA100, PAFault, and TRAIL, Sauron leads the reported verifier comparisons; on PA100 it has the highest F1 across eight backbones while costing less than Gandalf.

    Source: §7 · p. 12

  • Takeaways & Limitations

    Sauron’s evidence-first verification is a supported approach for rubric evaluation across the tested benchmarks, including detection and localization of real agent errors on TRAIL.

    Source: §7 · p. 12

    Controlled fault injections do not establish robustness to policies optimized to exploit Sauron.

    Source: A Limitations · p. 16

Abstract

from arXiv · show

Long-horizon, semi-verifiable agentic tasks often rely on rubric-based evaluation, but the evidence needed to assess these tasks can be spread across the environment state and the execution trace. A judge may overlook this evidence or accept unsupported claims, including an agent's own reports of success. When rubric verdicts serve as reinforcement-learning rewards, such errors can reinforce undesirable behavior. We introduce Sauron, an evidence-first agentic verifier that investigates both state and trace and relies on a claim only after independently validating it against its source. Bounded queries and joint investigation of related criteria reduce context use and redundant work, and each verdict carries an auditable receipt of its supporting evidence. Across 2,586 human-labeled criteria from 100 personal-assistant episodes, Sauron achieves the highest F1 of five verifiers on all eight backbones tested while costing less than Gandalf, the strongest baseline. With GPT-5.6 Sol, it reaches 0.96 F1 versus Gandalf's 0.92 at 23% lower cost, with a wider margin on process criteria. On a paired fault-injection suite covering 13 families, several of which simulate reward hacking, Sauron detects 97% of injected failures while flagging only 2% of clean controls, outperforming baselines even when they are given the full trace. On TRAIL, a benchmark for detecting and localizing real agent errors, Sauron leads both Gandalf and TRAIL's own judge on all three quality metrics.

1 Introduction

Rubric evaluation of long-horizon agent tasks must draw on evidence scattered across responses, environment state, and execution traces, yet judges can miss evidence or accept unsupported claims. Sauron independently verifies claims from state and trace before using them in verdicts, with bounded, joint investigation to reduce cost.

1 Introduction

  • Rubric criteria can require judgment as well as deterministic checks, while relevant evidence may be spread across an agent’s response, environment state, and execution trace.Rubric verdicts used as reinforcement-learning rewards can reinforce undesirable behavior when judges accept false reports of success.Source: §1 · pp. 1–2
  • Sauron independently checks claims against state and trace, and accepts a proposed verdict only when verified claims support it.This lets it assess outcomes from final state, actions from trace, and consistency across both sources.Source: §1 · pp. 2–3
  • Bounded queries and joint investigation of criteria likely to share evidence reduce context use and redundant work in long-horizon verification.The evaluation covers three benchmarks, with mechanism ablations and manual analysis of residual disagreements with human judgments.Source: §1 · pp. 2–3
  • Sauron outperforms four baselines in F1 on all eight PA100 backbones while costing less than Gandalf; with GPT-5.6 Sol, it reaches 0.96 versus 0.92 F1 at 23% lower cost.Its lead widens on process criteria; it also detects 97% of injected PAFault failures while flagging 2% of clean controls, and leads on all three TRAIL quality metrics.Source: §1 · p. 2

2 Related Work

Prior work studies rubric-based judgments and rewards, LLM and agentic judges, evidence-based process evaluation, and verification over long trajectories. Sauron combines these directions by validating claims against state and trace before they support rubric verdicts.

Rubric-based evaluation and rewards

  • Rubric judgments are used for semi-verifiable work and as RL rewards, where flawed rewards can be exploited; Sauron grounds verdicts in evidence checked against state and trace.Prior work also finds models can learn to hide reward-hacking evidence from chain-of-thought monitors.Source: §2 · p. 3

LLM and agentic judges

  • LLM judges can be sensitive to response position and length or favor their own generations, while agentic judges gather evidence through tools or environment inspection.Unlike these judges, Sauron separates evidence gathering from claim verification against original sources.Source: §2 · p. 3

Evidence and process evaluation

  • Sauron combines claim-level factuality checks, process supervision, and state-based evaluation across final state, recorded actions, and their consistency.Source: §2 · p. 3

Verification over long trajectories

  • Long-trajectory verification faces challenges as inputs grow and relevant evidence moves toward the middle; Sauron adapts bounded queries over state and trace.TRAIL benchmarks error categorization and localization in long trajectories.Source: §2 · p. 3

3 Sauron Architecture

Sauron grades rubric criteria by investigating final state and execution trace, independently verifying claims against their sources, and requiring verified evidence to support each verdict. Bounded queries and shared investigation reduce redundant work while each criterion retains its own verdict and receipt.

3 Sauron Architecture

  • Sauron separates adaptive evidence gathering from verification: claims are checked against their sources, and proposed verdicts require verified support.Source: §3 · p. 4

3.1 Setting and task interface

  • Each task supplies a user request, final response, environment state and artifacts, execution trace, and rubric criteria covering outcomes, process, or consistency.Criteria receive MET when verified evidence supports them and UNMET otherwise, with a receipt recording the evidence used.Source: §3.1 · p. 4

3.2 System overview

  • A smaller model groups criteria likely to share evidence and prepares an advisory gist before parallel investigators query state and trace.The gist guides investigation but cannot serve as evidence; a separate finalizer accepts a proposed verdict only when verified claims support it.Source: §3.2 · p. 4

3.3 Bounded, adaptive interrogation

  • Broad queries locate relevant entries or events, then targeted queries retrieve specific values or short trace windows under query and session budgets.In the example, investigators reuse the Chase lookup and compare the saved Amazon total with the retrieved account data.Source: §3.3 · p. 5, Figure 2 · p. 5

3.4 Evidence verification and receipts

  • An evidence atom pairs a claim with its evidence type and source locator; exact claims are checked mechanically, while interpretive claims go to an isolated faithfulness checker.Claims whose sources cannot be retrieved are rejected with a reason, and the evidence store records accepted and rejected atoms for an audit trail.Source: §3.4 · pp. 5–6, Figure 3 · p. 6

3.5 Shared evidence, independent verdicts

  • Related criteria can reuse retrieved content and verified atoms, avoiding redundant queries and verification, while each criterion receives an independent verdict and receipt.Source: §3.5 · p. 6

4 Experimental Setup

The experiments compare verifiers on fixed trajectories across PA100, PAFault, and TRAIL, with distinct criteria and metrics for outcomes, injected faults, and localized trace errors. TRAIL evaluation uses recorded traces only, so all verifiers operate without environment state.

4 Experimental Setup

  • All verifiers grade the same fixed trajectories and use the same backbone within each comparison; baselines use released defaults unless specified otherwise.Source: §4 · p. 7

4.1 PA100

  • PA100 samples 100 personal-assistant episodes with 2,586 human-labeled criteria, about 30% UNMET, and evaluates eight backbones using F1 as the primary metric.The median run has 53 trace events and 20 tool calls; 411 process and 2,175 outcome criteria are reported separately.Source: §4.1 · p. 7

4.2 PAFault

  • PAFault pairs fault-injected variants with clean controls from non-overlapping episodes, using detection rate minus false-alarm rate as its primary metric.Its 13 fault families include simulated reward hacking, trace–state mismatches, and environment failures; trace-blind defaults and full-trace variants are evaluated.Source: §4.2 · p. 7

4.3 TRAIL

  • TRAIL evaluates 148 recorded traces containing 841 annotated errors across 1,987 spans and 21 categories, with every verifier operating without environment state.Its metrics measure category detection, recovered error spans, and recovered span–category pairs; Sauron reports spans cited by verified evidence atoms.Source: §4.3 · pp. 7–8

5 Results

Sauron leads the quality–cost comparison on PA100 and also performs strongly on process criteria. It detects injected failures while limiting false alarms, and leads detection and localization of real errors on TRAIL.

5.1 Leading the quality–cost frontier

  • Across eight backbones, Sauron has the highest F1 at lower cost than Gandalf; with GPT-5.6 Sol, it leads by 0.04 F1 and 0.05 MCC at 23% lower cost.It reads fewer than half as many input tokens as Gandalf with GPT-5.6 Sol; on a 10-episode slice, it finishes in 17 minutes versus Gandalf’s 71.Source: §5.1 · p. 8, Table 1 · p. 9

5.2 The lead widens on process criteria

  • Sauron outperforms Gandalf on both outcome and process criteria, with a 0.07 F1 gain on PAProc; its process-criteria lead is larger across all eight backbones.Gandalf reaches 0.90 F1 on PAProc without trace access, while some process criteria leave evidence in the final state or can be inferred from the agent’s response.Source: Table 1 · p. 9, §5.2 · p. 9

5.3 Detecting injected failures

  • On state mismatch, Gandalftrace and Autorubrictrace score −0.27 and −0.94 versus Sauron’s 0.90; on silent write failures, they score −0.10 and −0.20 versus 1.00.Negative scores mean the verifier flags clean controls more often than faulty variants; Sauron catches these inconsistencies by checking actions against the final state.Source: §5.3 · p. 9
  • ∆fault = 0.95, with 97% of injected failures detected and 2% of clean controls flagged.The PAFault suite pairs injected faults with clean controls across 13 families.Source: §5.3 · p. 9, Table 2 · p. 10
  • Full-trace baselines still trail Sauron: Gandalftrace reaches ∆fault = 0.67 and Autorubrictrace 0.71, both at least 0.24 below Sauron.Gandalftrace costs 4.7× as much per variant as Sauron and takes twice as long; Autorubrictrace costs 1.2× as much.Source: §5.3 · p. 9, Table 2 · p. 10

5.4 Localizing real agent failures on TRAIL

  • On TRAIL, Sauron leads Gandalf and JudgeTRAIL on all three metrics across splits and backbones; on GAIA it gains 0.07 wF1, 0.18 location accuracy, and 0.19 joint accuracy.With GPT-5.6 Terra, Sauron outperforms both Sol-backed baselines on every metric at roughly half their cost.Source: Table 3 · p. 10, §5.4 · p. 10

6 Ablations and Error Analysis

Ablations show that trace access, claim verification, and joint investigation contribute to Sauron’s accuracy or efficiency; most reviewed disagreements arose from stricter criterion readings.

6.1 What each mechanism buys

  • On PAAbl (860 criteria, GPT-5.6 Sol), trace-blind Sauron loses 0.15 F1, the largest ablation drop.Table 4 reports mechanism ablations relative to full Sauron.Source: §6.1 · pp. 10–11, Table 4 · p. 11
  • Without bounded queries, F1 rises by 0.01, but cost triples and investigation turns more than double; without the gist, F1 is unchanged as cost rises 15% and turns 16%.The gist contributes efficiency rather than accuracy.Source: §6.1 · p. 11, Table 4 · p. 11
  • Disabling claim verification reduces F1 by 0.03 while lowering cost by 40%; the gain comes from checking interpretive claims.Disabling only the faithfulness checker produces the same drop.Source: §6.1 · p. 11, Table 4 · p. 11
  • Disabling clustering reduces F1 by 0.01, increases cost by 63%, and requires roughly 2.5 times as many investigation turns.Clustering cuts redundant work without hurting quality.Source: §6.1 · p. 11, Table 4 · p. 11

6.2 Where judgments diverge

  • Of 65 disagreements with human judgments, nearly three-quarters (48) reflected stricter readings by Sauron, and all but three concerned criterion interpretation rather than evidence.The 48 stricter readings included 34 literal-wording or subclause judgments and 14 negative-criterion judgments.Source: §6.2 · p. 11

7 Conclusion

Sauron verifies rubric verdicts against trace and state, using bounded queries and joint investigation to reduce verification cost. It outperforms baselines across PA100, PAFault, and TRAIL.

7 Conclusion

  • Sauron grounds verdicts in verified evidence from execution traces and final states, retrieving relevant portions through bounded queries and jointly investigating related criteria.The conclusion presents these mechanisms as ways to reduce verification cost.Source: §7 · p. 12
  • On PA100, Sauron achieves the highest F1 of five verifiers on all eight backbones while costing less than Gandalf; on PAFault, it detects 97% of injected failures and flags 2% of clean controls.Its lead widens on process criteria, which depend on trace evidence.Source: §7 · p. 12
  • On TRAIL, Sauron outperforms both baselines in detecting and localizing real agent errors; the authors suggest evidence-first verification may supply more reliable rewards for future agent training.Evaluating Sauron as an RL reward provider is left for future work.Source: §7 · p. 12

A Limitations

This section describes Sauron’s evaluation limitations and its policy for handling unresolved criteria and linking verdicts to verified evidence.

A Limitations

  • Evaluations cover English tasks; PA100 and PAFault use Opus 4.8 in personal assistance, while TRAIL broadens model and domain coverage without systematically testing model-family bias.The evaluation does not establish whether verifiers favor agents from their own model family.Source: A Limitations · p. 16
  • The faithfulness checker can still err, evidence may be inaccessible, and agent-authored content may mislead checks; controlled injections do not establish robustness to policies optimized to exploit Sauron.The authors propose evaluating Sauron as an RL reward provider by measuring effects on task performance and reward hacking.Source: A Limitations · p. 16

B.1 Input representation and source access

  • Recorded messages, tool calls, and results become separate linked events, while final state and artifacts are exposed through read-only structured snapshots or workspace files.Artifact readers expose document text, tabular data, and stored spreadsheet formulas.Source: §B.1 · p. 16

B.2 Planning and session initialization

  • GPT-5.6 Luna clusters criteria by shared evidence, with a default maximum of 12; deterministic repair ensures each criterion appears once, and a shared task gist offers advisory inspection guidance.Investigators receive assigned criteria, the gist, the final response excerpt, and run metadata; invalid plans ultimately fall back to singleton clusters.Source: §B.2 · pp. 16–18

SHARED INSTRUCTIONS

  • Investigators treat state as the source of truth, inspect state or trace as needed, and finalize only with verified evidence supporting the criterion.Response-text criteria may be decided from the final response alone; process-dependent criteria require trace inspection, and the agent’s own claim is never sufficient.Source: SHARED INSTRUCTIONS · pp. 18–19

B.3 Investigation and bounded queries

  • Broad and targeted queries inspect trace, state, and artifacts, but absence claims must match the checked source scope and truncated results cannot establish absence from unseen content.Query responses are size-limited, and session budgets constrain turns, tokens, and tool calls; bounded queries save less when tasks require broad trace coverage.Source: A Limitations · p. 16, §B.3 · pp. 20–23

B.4 Evidence verification and receipts

  • Evidence proposals identify source locators, undergo retrieval and mechanical or faithfulness checks, and support verdicts only when verified claims are relevant and sufficiently cover the criterion.Finalization requires verified evidence for MET or UNMET verdicts, except unresolved criteria at budget exhaustion, which receive an evidence-missing marker.Source: §B.4 · pp. 23–25
  • When the budget runs out, unresolved criteria default to UNMET with an evidence-missing marker; budget exhaustion alone does not establish a violation.UNMET can also rest on verified evidence against the criterion.Source: §B.4 · p. 25
  • Each receipt links a criterion’s verdict and rationale to cited verified atoms, including their claims and source locators.Receipts are saved with the evidence store in each criterion’s output record.Source: §B.4 · p. 25

C Experimental Configurations

The evaluation uses released verifier implementations with their standard instructions and distinct evidence-access configurations. Sauron’s investigator and verification calls use the evaluated backbone, while clustering and gist generation use a fixed lightweight model.

C Experimental Configurations

  • Baseline configurations use released implementations and built-in instructions, with no added domain-specific guidance.Trace-enabled Gandalf and Autorubric variants are described separately for PAFault.Source: C Experimental Configurations · pp. 25–26

C.1 Sauron model configuration

  • Sauron uses the evaluated backbone for investigation and model-based verification, while GPT-5.6 Luna handles clustering and gist generation across configurations.The lightweight-model cost is included in Sauron’s totals.Source: §C.1 · p. 26

C.2 Baseline configurations

  • Gandalf can inspect final state and workspace but receives no execution trace in the primary comparison; AAAJ can inspect the trace and workspace through its fixed retrieval workflow.Gandalf uses batch-size target 16 and a 300-second timeout; AAAJ uses its released gray_box configuration and default voting workflow.Source: §C.2 · pp. 26–27
  • Archipelago judges from the final response and selected changed files, while Autorubric receives the request and final response without tools or environment access.Archipelago’s primary configuration does not see the execution trace; Autorubric uses its default single-judge prompt without few-shot examples.Source: §C.2 · p. 27

D Datasets and Evaluation Protocols

The protocols define the personal-assistant, fault-injection, and TRAIL evaluation sets, then specify how criteria, evidence, and verifier outputs are labeled and scored. TRAIL evaluation uses recorded traces and maps verified evidence to annotated error locations.

D Datasets and Evaluation Protocols

  • The PA100 sample contains 100 stratified episodes from a 700-episode corpus, with trajectory and rubric sizes used to cover varied workloads.The corpus includes requests, trajectories, responses, environment states, workspace artifacts, and criterion-level rubric judgments.Source: D Datasets and Evaluation Protocols · p. 27
  • Prompts, evidence types, budgets, clustering, and decision policy were fixed before evaluation; two annotators achieved 95% pre-adjudication agreement across 2,586 criteria.A separate 50-episode development set did not overlap with PA100; disagreements were adjudicated by a senior annotator.Source: D Datasets and Evaluation Protocols · p. 27
  • Criteria are classified from criterion text alone as outcome or process; manual review of 200 criteria agreed with 99% of the classifier’s labels.PA100 contains 2,175 outcome and 411 process criteria; process criteria require evidence about tool use, order, transient actions, value production, or cross-source consistency.Source: D Datasets and Evaluation Protocols · p. 27, Figure 10 · p. 28
  • F1 and MCC pool criterion-level counts with UNMET as positive, and abstentions or missing verdicts count as errors.Criteria contribute equally regardless of rubric weight, with outcome and process subsets scored separately.Source: D Datasets and Evaluation Protocols · p. 28
  • PAFault contains 637 paired variants across 13 fault families; its families cover selected taxonomy modes, with persistent-memory and peer or subagent failures outside scope.Twelve families map to ten interaction-centric taxonomy modes, while reordering is added as a separate test; reward-hacking cases are controlled simulations, not policy-optimization discoveries.Source: D Datasets and Evaluation Protocols · p. 28, Table 10 · p. 29
  • PAFault pairs compare injected variants with controls, and the family score subtracts the false-alarm rate from the detection rate before an unweighted 13-family mean.Some criteria asserting a violation expect MET on faulty variants; blameless-failure pairs expect MET on both sides.Source: D Datasets and Evaluation Protocols · pp. 29, 31, Table 11 · p. 30

D.3 TRAIL

  • TRAIL evaluation uses 148 annotated traces from GAIA and SWE-bench Lite without access to their original execution environments.Each trace receives criteria for 21 error categories, and reference labels mark a category UNMET when its annotations contain at least one error.Source: §D.3 · p. 31
  • TRAIL scores category detection with weighted F1 and error localization with location and joint accuracy; Sauron’s locations come only from verified atoms.Verified event identifiers map to original span identifiers; location normalization is shared across verifiers and does not use reference annotations.Source: §D.3 · pp. 31–32

E Additional Results

The additional results report PA100 cross-backbone comparisons, PAFault performance by family and resource use, and runtime comparisons on a shared episode slice. On PAFault, Sauron scores highly on 12 fault families, with reordering as its weakest family.

PA100 Results across backbones

  • Table 12 reports quality metrics, error counts, cost, and token usage for five verifiers across eight PA100 backbones.Its F1 and cost values correspond to Figure 4, while Table 13 separates outcome and process criteria.Source: E Additional Results · p. 32

Runtime and concurrency

  • Table 14 compares elapsed grading time for Sauron and Gandalf on the same 10 PA100 episodes at four concurrency levels.Each verifier’s within-task parallelism and other settings remain fixed; speedup is Gandalf’s elapsed time divided by Sauron’s.Source: E Additional Results · p. 32

Resource use

  • 30 seconds per variant for Sauron versus 66 for Gandalftrace; Sauron reads 7.6× fewer input tokens, 57.8 million versus 440.4 million.Default Gandalf averages 140 seconds per variant; the resource totals cover all 1,274 variants.Source: E Additional Results · p. 32

PAFault Results by fault family

  • 0.90–1.00 ∆fault on 12 of 13 families; Sauron’s lowest score is 0.69 on reordering, where Autorubrictrace scores 1.00.Figure 11 reports GPT-5.6 Sol results across the five verifiers and two trace-enabled variants.Source: E Additional Results · p. 32, Figure 11 · p. 33

F Ablation Details

The ablation variants isolate evidence access, evidence verification, and joint investigation while holding other settings fixed where specified. The evaluation combines PA100 criteria with PAFault variants to include both outcome and process verification.

F Ablation Details

  • PAAbl combines 506 PA100 criteria and 354 PAFault variants into 860 decisions from disjoint source episodes.Fault and control variants increase coverage of process-verification cases, which comprise about 16% of PA100.Source: F Ablation Details · p. 33
  • State-only removes trace queries, while trace-in-context supplies the full trace directly and removes trace-query tools; both retain state and artifact queries.No-gist removes gist generation and its advisory context.Source: F Ablation Details · p. 33
  • Verification ablations separately remove the faithfulness checker, mechanical checks, or both, while retaining source-locator validation and verdict finalization.Without the faithfulness checker, retrieved sources are accepted without checking whether they support claims; without mechanical checks, claims go to the faithfulness checker.Source: F Ablation Details · p. 33
  • Unclustered investigation gives each criterion its own session and evidence store, preventing reuse of retrieved content and verified atoms across criteria.The gist and budget-setting rule remain unchanged.Source: F Ablation Details · p. 34

G Error Analysis

A manual review of all 65 PA100 disagreements grouped them by disagreement type after examining criterion text, reference labels, rationales, evidence atoms, and budget flags.

G Error Analysis

  • 34 missed awards arose when Sauron read positive criteria strictly, rejecting dates without years, rounded amounts, or information in a neighboring section.For example, it marked a billing date unmet when the text omitted the year, although annotators considered the criterion satisfied.Source: G Error Analysis · p. 34, Table 16 · p. 37
  • 14 false awards arose when Sauron read negative criteria more broadly than annotators, counting actions outside the criterion’s intended scope as violations.One example treated a write to a dated memory file as a violation despite an exception for updates to MEMORY.md.Source: G Error Analysis · p. 34, Table 16 · p. 37
  • 6 false awards came from permissive readings of positive criteria, including accepting partial compliance or separately listed amounts that summed to a required total.Sauron accepted $1,470.00 and $570.00 as meeting a $2,040.00 debt-payment requirement; annotators marked it unmet.Source: G Error Analysis · p. 34, Table 16 · p. 37
  • One missed award occurred when Sauron accepted an Airtable milestone labeled “LLC filed” as support, while annotators judged the filing claim unsupported.The annotators marked the violation criterion met; Sauron returned unmet.Source: G Error Analysis · p. 34, Table 16 · p. 37
  • Four missed awards and one false award involved misinterpreting a criterion or treating a conditional requirement as failed before its trigger occurred.In one case, Sauron marked a recheck unmet even though the user had not asked the follow-up question that would trigger it.Source: G Error Analysis · p. 34, Table 16 · p. 37
  • Three missed awards reflected polarity differences: both reviewers agreed no violation occurred, but Sauron encoded absence as UNMET and annotators as MET.Two further missed awards followed investigation-budget exhaustion, when Sauron returned UNMET for criteria labeled MET by annotators.Source: G Error Analysis · p. 34, Table 16 · p. 37

Related papers

Loading 2610.17724v1…