PaperlayerView exact revision on arXiv

Source-linked AI summary

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

Jianing Yin, Ziyi Chen, Hao Wang, Yiyou Sun, Zijian Jin, Dawn Song, Daoan Zhang

arXiv:2610.28679v1cs.SEcs.AI

TL;DR

Incorrect verifiers can reward violations or discard valid solutions, yet evaluating them requires testing both valid alternatives and invalid cases. The paper builds and audits VerifierBench, then finds that generated verifiers reject invalid solutions more reliably than they accept valid ones, with many analyzed false rejections coming from verifiers that reject every case for a task.

  • Problem

    Incorrect verifiers can repeatedly reward violations or discard valid solutions, so their reliability matters when agents generate training data or rewards.

    Source: §1 · p. 1

  • Method

    VerifierBench tests generated verifiers on 222 tasks and 4,993 hidden cases, including valid alternatives and attacks, after V4V screens source verifiers and test suites.

    Source: §1 · p. 1, §2.3 · p. 2

  • Results

    Generated verifiers reject invalid solutions more reliably than they accept valid ones; TPR ranges from 21.1% to 70.2%, versus TNR of 79.2% to 96.3%.

    Source: §1 · p. 2, §3.2 · p. 4

  • Takeaways & Limitations

    Generated verifiers need tests that challenge each acceptance condition against task requirements, including permitted variations beyond a chosen solution.

    Source: §7 · p. 8

    Scores measure agreement with retained reference verifiers on finite, constructed test suites, which does not certify every construction label as semantically correct.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites. Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones. This apparent strength can mask overly restrictive verification: most false rejections in the analyzed records come from verifiers that accept no tested solution to a task. Case studies reveal unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions. These errors survive self-testing because agents test solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes. These findings suggest testing both whether a verifier accepts a correct solution and whether it accepts other solutions permitted by the task. The project page and dataset are publicly available.

1 Introduction

The paper studies executable verifiers because incorrect ones can reward violations or discard valid solutions in data-oriented learning. It introduces VerifierBench and audits its source verifiers and test suites, then finds that generated verifiers often reject valid solutions and can miss that failure in self-testing.

1 Introduction

  • Incorrect verifiers can repeatedly reward violations or discard valid solutions, making their reliability critical to data-oriented recursive self-improvement.The paper evaluates generated verifiers alongside 17 further model–harness configurations and analyzes restrictive checks and self-testing.Source: §1 · pp. 1–2
  • A verifier must accept every valid solution and reject invalid ones, distinguishing task requirements from choices that may vary across valid solutions.A verifier can reject a permitted read-only query when its tool allowlist omits that query, while self-tests fail to establish that a complete valid solution passes.Source: §1 · p. 1
  • VerifierBench tests 222 tasks from six agent benchmarks with 4,993 hidden cases covering valid alternatives, ordinary errors, and attacks.V4V applies six checks to source verifiers and test cases; retained source verifiers serve as references.Source: §1 · pp. 1–2
  • Generated verifiers accept valid solutions less reliably than they reject invalid ones, with false rejections concentrated in all-reject verifiers.Across 19 models in a common harness, TPR ranges from 21.1% to 70.2%, versus TNR of 79.2% to 96.3%; all-reject verifiers account for 87.9–96.0% of false rejections in analyzed records.Source: §1 · p. 2

2 VerifierBench

VerifierBench evaluates executable verifiers on task-specific cases spanning tool calls, environment states, and code changes. It screens source verifiers and test cases, then scores generated verifiers against retained reference decisions using normalized task-averaged balanced accuracy.

2.1 Task Formulation

  • Agents receive the task instruction and environment, generate an executable verifier, and may self-test it before it is fixed and evaluated on every test case.The reference verifier and hidden cases are withheld during generation; tool-call, state, and code verifiers receive different candidate inputs, and state/code tasks can use run_public.Source: §2.1 · p. 2

2.2 Task Sources and Test Cases

  • The benchmark covers tool-call, state, and code tasks, with gold solutions, valid variants, ordinary errors, and attacks labeled from task requirements.Valid variants change presentation, implementation, or execution path while satisfying every requirement; attacks exploit checking omissions, fabricate success signals, or interfere with verification.Source: §2.2 · p. 2

2.3 Screening with Verifier4Verifier

  • All 222 retained tasks pass V4V’s six checks, and their source verifiers become the reference verifiers.V4V checks replay, repeatability, provenance, input-label consistency, trivial baselines, and mutation testing; disagreements are resolved against task requirements, since agreement alone does not establish semantic correctness.Source: §2.3 · pp. 2–3

2.4 Metrics

  • Normalized task-averaged balanced accuracy equally weights tasks and the two classes within each task, while attack false-positive rate measures attacks receiving PASS.TPR averages the per-task fractions of positive cases receiving PASS; TNR averages the per-task fractions of negative cases receiving FAIL. Constant valid decisions score 0 and perfect agreement scores 100.Source: §2.4 · p. 4

3 Experiments

The experiments evaluate 19 models in a shared harness, then examine alternative harnesses, reasoning effort, and where false rejections concentrate. Performance is limited, with valid solutions accepted less reliably than invalid solutions are rejected.

3.1 Experimental Setup

  • The study scores all 36 configurations on the same 222 tasks; cross-harness comparisons describe configurations, not model capability alone.The main release supplies scores; archived records support the concentration analysis, while the separate Astra trace batch supports effort comparisons and trace analysis.Source: §3.1 · p. 4

3.2 Main Results

  • BAnorm ranges from 10.6 to 62.1 across the 19 Pi-harness models; GPT-6-astra ranks first, followed by Claude Opus 4.8 (54.0) and Kimi K3 (53.4).No model approaches perfect agreement.Source: §3.2 · p. 4, Table 2 · p. 6
  • TPR is lower than TNR for every model: TPR ranges from 21.1% to 70.2%, versus TNR from 79.2% to 96.3%.The gap is widest on state and code tasks, where average TPR is 27.7% and 40.2%, compared with 79.2% on tool-call tasks.Source: §3.2 · p. 4, Figure 3 · p. 5

3.3 Effects of Harness and Reasoning Effort

  • Native harness effects vary: Claude Code raises Claude Opus 4.8 by 12.2 points to 66.2, while Codex lowers all four GPT models by 4.1–26.6 points.Across the alternative harnesses, no harness improves every model it runs.Source: §3.3 · pp. 4–5
  • Harness effects are not a fixed acceptance–rejection trade-off: Codex raises Qwen3.7 Plus TNR from 82.6% to 92.1% but lowers TPR from 57.6% to 34.0%.Claude Code raises both rates for Claude Opus 4.8.Source: §3.3 · pp. 4, 6
  • HIGH reasoning effort barely changes aggregate BAnorm, but its task-type effects differ: code improves with both harnesses while state worsens.With Pi, HIGH raises tool-call TPR but also introduces 128 false acceptances while correcting three.Source: §3.3 · p. 6
  • With HIGH effort, runs exhausting the 20-execution budget rise from 4 to 77 with Pi and from 21 to 98 with Claude Code among 127 state and code tasks.On 32 τ 2-bench tasks, runs exhausting the budget before first writing the verifier rise from 0 to 30 with Pi and from 7 to 20 with Claude Code.Source: §3.3 · p. 6

3.4 Where False Rejections Concentrate

  • All-reject verifiers contribute 87.9–96.0% of positive FAIL decisions in archived records; the Astra trace batch reproduces the concentration at 93.3–95.5%.In each Astra condition, 68–75 of the 222 verifiers are all-reject.Source: §3.4 · p. 6

4 Why Generated Verifiers Reject Valid Solutions

The section analyzes unjustified checks behind valid-solution rejections and asks why self-tests did not expose them; the selected trace review identifies mechanisms, not prevalence.

4.1 Unjustified Checks

  • The targeted review finds unjustified checks in 16 verifiers from 12 tasks; Table 4's 119 positive FAIL decisions cover 90 cases, and the review does not estimate prevalence.A check is unjustified when task instructions, tool definitions, and environment rules do not require it, yet it rejects permitted solutions.Source: §4.1 · p. 7, Table 4 · p. 7, §4 · p. 7
  • Checks excluded permitted read-only or user-side calls, legal Matplotlib source changes, and a retail item omitted from a keyword list.The banking tool restrictions rejected positive cases before environment execution; the cited examples include nine positives for the omitted dispute-history query and 27 for other authorized calls.Source: §4.1 · p. 7
  • Other checks accepted only some permitted representations, including ISO-only currencies, numeric-only device identifiers, or incomplete TeX-spacing decoding.These representation restrictions are described as remaining cases in Appendix I.2.Source: §4.1 · p. 7

4.2 Self-Tests That Confirm Without Challenging

  • Self-tests used only allowed patch locations or accepted representations; a Qwen verifier's tests always supplied an optional argument it mishandled.The positive self-tests modified other Matplotlib locations, while the Qwen verifier treated an omitted optional argument as empty instead of using its documented default.Source: §4.2 · pp. 7–8
  • Confirmatory self-tests use chosen solutions that the unjustified check already allows, without asking whether its other conditions follow from the task.Negative tests may also violate a genuine requirement while tripping the unjustified check, so their rejection does not justify that check.Source: §4.2 · p. 7

4.3 Self-Tests That Never Confirm Acceptance

  • Eight banking and retail verifiers never achieved a complete positive PASS during generation and later rejected all 72 positive judgments without environment execution.Their intended-positive self-tests did not establish acceptance of even the agents' own intended solutions.Source: §4.3 · p. 8
  • Generation-time exploration and self-testing share an execution budget, often exhausted in HIGH-effort state and code runs, which can leave checks unresolved.The authors say these limits do not directly cause hidden rejections; local execution also failed in two traces where unshare was unavailable.Source: §4.3 · p. 8

5 Implications

The implications section recommends testing each verifier condition against task requirements and reserving execution budget for completed testing.

5 Implications

  • Test each acceptance condition against the instruction, tool definitions, or environment rules, using permitted variants and cases that violate a requirement.Vary one permitted property while keeping the task satisfied; examples include changing a legal file, omitting a defaulted argument, or making an authorized user-side call.Source: §5 · p. 8
  • Examples already allowed by a check cannot challenge that check, and permitting legal source changes need not weaken rejection of tested violations.The Django configurations illustrate that legal source changes can be permitted while tested violations are still rejected.Source: §5 · p. 8
  • Reserve execution budget for final-verifier testing and distinguish execution failures from failed requirements; blocked tests leave checks unresolved.The paper does not evaluate the effectiveness of these directions.Source: §5 · p. 8

6 Related Work

The related work situates verifier generation among executable specifications, test generation, and automated training-data construction.

6 Related Work

  • Prior executable-specification work evaluates generated specifications on valid and faulty behavior, while CheckerBench studies agent-generated static-analysis checkers.The paper focuses instead on task-completion verifiers for tool-call, state, and code tasks, generated from task-solving instructions and environments.Source: §6 · p. 8
  • Test-oracle and test-generation work spans generated assertions, unit-test generation, and issue-reproducing tests that assess candidate patches.VerifierBench uses multiple valid variants and violations per task and examines whether self-tests challenge unjustified checks.Source: §6 · p. 8
  • Unlike automated training-data construction that jointly generates tasks and verification materials, this study holds tasks fixed to isolate verifier generation.The related-work discussion notes that verifier errors could affect which solutions enter training data.Source: §6 · p. 8

7 Conclusion

The conclusion reports that generated verifiers more reliably reject invalid than accept valid solutions, and links false rejections to unjustified checks and self-testing gaps.

7 Conclusion

  • Across all 36 configurations, generated verifiers accept valid solutions less reliably than they reject invalid ones, with most analyzed false rejections from all-reject verifiers.The conclusion connects code and trace inspection to unjustified checks and self-tests that either confirm only allowed solutions or never confirm acceptance.Source: §7 · p. 8
  • Verifier generation needs tests that challenge each acceptance condition against task requirements; in data-oriented RSI, unjustified checks could exclude legitimate solution strategies.The stated implication is scoped to settings where generated verifiers select training data.Source: §7 · p. 8

Limitations

The reported scores measure agreement with retained reference verifiers on finite, constructed test suites, and the analysis has boundaries in its records and configuration comparisons. The task-source material describes the contract and screening behind the retained benchmark.

Limitations

  • The analysis uses archived per-task records for nine models and one Astra trace batch; selected failure cases identify mechanisms, not their prevalence.Cross-harness differences do not isolate a harness component, and rankings are point estimates without repeated generations or significance tests.Source: Limitations · p. 9

A Task Contract

  • Table 5 summarizes what the model receives, must produce, and how the resulting verifier is scored.The verifier-generation task contract is the shared scope of the table.Source: A Task Contract · p. 11, Table 5 · p. 11

B Task Sources and Reference Validation

  • The retained set contains 222 tasks from seven source subsets across six benchmarks; 45 tasks were excluded for reported source defects.Terminal-Bench contributes no retained tasks, and AgentDojo contributes separate user-task and injection subsets.Source: B Task Sources and Reference Validation · p. 11

B.1 Excluded Tasks

Screening removed tasks with confirmed source defects and recorded six checks for every retained task. The appendix also reports mutation detection, targeted rechecks, test-case construction, and scoring conventions.

B.1 Excluded Tasks

  • 45 tasks were excluded for source defects, including five BFCL tasks, all 37 Terminal-Bench tasks, and three SWE-bench tasks.Terminal-Bench exclusions include 35 cases where skipped tests were counted as passes; the others involved browser failures or inconsistent decisions.Source: §B.1 · p. 11

B.2 Checks on Retained Tasks

  • All 222 retained tasks pass all six V4V checks, yielding 1,332 recorded task–check outcomes.The frozen screening records contain one row per retained task.Source: §B.2 · p. 12

B.3 Mutation Construction and Detection

  • All 222 retained tasks pass C6: their corresponding test suites detect every mutation in the task’s operator set across 1,410 configurations.A detection requires the original and mutated verifiers to return different valid PASS/FAIL decisions on the same test case; crashes or invalid outputs do not count.Source: §B.3 · p. 12

B.4 Targeted Rechecks

  • Targeted rechecks agree with labels for 320 cases across 16 τ 2 banking tasks and 38 cases across three retained SWE tasks.Each SWE task was executed three times from the correct repository base.Source: §B.4 · p. 14

C Test Cases and Construction Categories

  • The final test suites contain 1,566 positive cases and 3,427 negative cases; labels distinguish valid variants, ordinary errors, and attacks.Valid variants combine cosmetic changes and alternative solutions, while attacks combine hacks and process violations.Source: C Test Cases and Construction Categories · p. 14

D Metrics and Scoring Protocol

  • Invalid or missing outputs remain in the scoring denominators, and an invalid or missing output does not count as an accepted attack.Low attack false-positive rates therefore need interpretation alongside valid-output coverage and the primary accuracy metric.Source: D Metrics and Scoring Protocol · pp. 15–16

E Generation Configurations

The generation configurations vary models, harnesses, and reasoning settings, so cross-configuration comparisons do not isolate one component. Reported comparisons show mixed changes in acceptance and rejection across settings and task types.

E Generation Configurations

  • Generation budgets are 5,400 seconds per task and 32,768 output tokens per reply, with tools for inspection, public execution, and reset where supported.The budget details count execution requests separately from total tool calls or conversation turns.Source: E Generation Configurations · pp. 16, 18
  • For Qwen3.7 Plus, Codex raises TNR from 82.6% with the Pi harness to 92.1%, but lowers TPR from 57.6% to 34.0%; BAnorm falls from 40.2 to 26.1.For Claude Opus 4.8, Claude Code raises TPR from 67.7% to 75.8% and TNR from 86.3% to 90.4%, increasing BAnorm by 12.2 points.Source: E Generation Configurations · p. 16, Table 11 · p. 17
  • The 17 alternative configurations vary harness families and pairings; cross-harness comparisons describe configurations rather than isolated model capability.Harnesses differ in prompts, tools, and reasoning settings.Source: Table 12 · p. 17
  • The reasoning-effort ablation tests GPT-6-astra under LOW and HIGH effort with Pi and Claude Code, using all 222 tasks in each condition.Each task has a 5,400-second limit and at most 32,768 output tokens per response; these conditions are separate from the main-table configurations.Source: E Generation Configurations · p. 18
  • HIGH effort improves code-task scores with both harnesses but reduces state-task scores; in Pi tool-call tasks it accepts 63 additional positive cases while introducing 128 false acceptances and correcting three.The positive-acceptance increase therefore does not uniformly improve discrimination between positive and negative cases.Source: E Generation Configurations · p. 18

F Results by Task Type

Model performance and positive-solution acceptance vary across task types, while grouped test-case results show that explicit rejection accounts for most unaccepted positive cases. Low attack acceptance alone does not establish reliable verification.

F Results by Task Type

  • GPT-5.6-sol leads tool-call verification at 85.2, Claude Opus 4.8 leads state-based verification at 34.6, and Kimi K3 leads code-based verification at 76.9.Rankings also reverse across task groups; the breakdown covers 95 tool-call, 80 state, and 47 code tasks, with equal task weighting within each type.Source: F Results by Task Type · p. 18
  • Mean TPR is 79.2% on tool-call tasks, 27.7% on state tasks, and 40.2% on code tasks, while mean TNR is 85.8%, 85.2%, and 90.6%, respectively.All models have lower TPR than TNR on state and code tasks, but eight have higher TPR than TNR on tool-call tasks; low positive acceptance includes missing or invalid decisions.Source: F Results by Task Type · pp. 18, 21
  • On state tasks, GPT-5.6-sol has FPRh = 0.0% but accepts only 23.0% of positives; on code tasks, 17 models have FPRh = 0.0% while BAnorm ranges from −25.9 to 76.9.Attack acceptance must be interpreted alongside acceptance of valid solutions and successful rejection of negatives.Source: F Results by Task Type · p. 21

G Grouped Test-Case Results

  • For every model, explicit FAIL accounts for most unaccepted positive test cases, with rejection shares from 57.1% to 100%.The grouped diagnostics distinguish reference agreement from missing or invalid outputs; their test-case weighting differs from the task-averaged leaderboard.Source: G Grouped Test-Case Results · p. 21

H Task-Level Error Analysis

Task-level analyses find that all-reject verifiers account for most false rejections in the analyzed records. Selected cases illustrate distinct failure mechanisms, while a purposive review of verifier checks is not an estimate of their prevalence.

H Task-Level Error Analysis

  • The archived task-level analysis covers nine Pi-harness models; Gemini 3.5 Flash is excluded because its archived judgments differ from final aggregates, while all 19 remain in the leaderboard.Aggregate counts agree with the final release, but the archive predates final rejudging and does not establish identity of every task-level decision.Source: H Task-Level Error Analysis · p. 21
  • In the Astra trace batch, all-reject verifiers account for 93.3–95.5% of false rejections across four conditions.Equal-task-weight shares are 93.9%, 95.1%, 93.4%, and 94.4% for Pi/LOW, Pi/HIGH, Claude Code/LOW, and Claude Code/HIGH, respectively.Source: H Task-Level Error Analysis · p. 23
  • State-task positive acceptance varies across source benchmarks, yet removing any single source leaves overall TPR below TNR in the archived nine-model records.Four models show the opposite relation within tool-call tasks, so global asymmetry and source-dependent differences coexist.Source: H Task-Level Error Analysis · p. 23
  • All-reject verifiers contribute 87.9–96.0% of positive FAIL decisions across nine Pi-harness configurations.The equal-task-weight share is also 87.9–96.0%; for GPT-6-astra it is 91.1%, versus 92.7% with test-case weighting.Source: H Task-Level Error Analysis · p. 21, Figure 5 · p. 24

I.1 Comparing Models on the Same Test Cases

  • Selected same-test-case comparisons separate input-parsing failures, omitted required checks, early rejection before state replay, and timeouts.The examples include rejection of all six positives on ACE preference 17, all nine positives on retail/34, and incomplete execution on Django-11848; they illustrate mechanisms, not task-type rankings.Source: §I.1 · p. 25

I.2 Attributing Rejections to Unjustified Checks

  • The review attributes selected rejections to unjustified checks by inspecting requirements, generated checks, positive cases, judgments, and generation traces.It is purposive rather than random, and source inspection identifies sufficient rejection conditions without proving each is the unique cause.Source: §I.2 · p. 25

I.3 Source-File Restrictions and Self-Test Coverage

The case studies show that generated verifiers reject valid code changes, authorized tool calls, and permitted input variations when their checks impose unsupported restrictions or self-tests share those assumptions. Selected cases illustrate these mechanisms but do not estimate their frequency or isolate harness effects.

I.3 Source-File Restrictions and Self-Test Coverage

  • Four Django positives and four distinct Matplotlib positives are rejected by unjustified file restrictions, while generation-time tests modify only permitted locations.The Django self-test receives the intended PASS and FAIL outcomes but never challenges the restriction; both Matplotlib verifiers reject four positive cases each.Source: §I.3 · pp. 25–26

I.4 Tool-Call Restrictions

  • Five banking verifiers reject 27 authorized user-side calls, while two more verifiers reject 18 permitted read-only calls as unsupported operations.The banking self-tests use assistant calls; the read-only-call self-tests omit the same allowed read.Source: §I.4 · p. 26

I.5 Incomplete Category Vocabulary

  • Three retail verifiers produce 27 positive FAIL decisions by omitting Headphones from the office-item vocabulary, with 18 distinct cases across related tasks.For retail/34, the hidden positives include three office items, but one verifier recognizes only two; the tasks share an order background.Source: §I.5 · p. 26
  • Other Table 4 cases involve unjustified representation restrictions, while a compatibility-dependent Django case is excluded because available materials do not fully specify compatibility requirements.The additional cases concern ISO-only currency, numeric-only device identifiers, and incomplete TeX-spacing decoding; the Django case contributes 16 positive rejections on four distinct cases.Source: §I.5 · p. 27

I.6 Incorrect Handling of a Specified Default

  • A Qwen3.7 Plus verifier rejects a valid DNA insertion call that omits optional mutation_type, because it compares the missing value with an empty string instead of the specified default.The fully specified solution passes, and the visible positive self-tests all supply mutation_type, leaving the permitted omission untested.Source: §I.6 · p. 27, §I.9 · p. 30

I.7 Self-Tests That Share a Parsing Assumption

  • A parser rejects the reference ACEBench solution because it uses Python keyword arguments, while the passing self-test uses the JSON-like representation accepted by the parser.The raw-call field has no concrete grammar in the generation prompt, so this case indicates input incompatibility and inadequate self-test coverage rather than disregard of an explicit syntax requirement.Source: §I.7 · p. 27

I.8 Compatibility with Code Forms of Positive Test Cases

  • An AppWorld verifier rejects a positive solution before execution because its AST check forbids imports used by that solution.Its direct-API positive self-test avoids imports and leaves the restriction untested; the archived prompt does not fully specify the solution-code entry convention, so this case is not counted among unjustified checks.Source: §I.8 · p. 28

I.9 Comparative Generation Trajectories

  • The comparative cases connect task requirements, generated checks, self-tests, and hidden judgments, but do not estimate mechanism frequency or isolate harness effects.The DNA comparison shows successful self-tests with explicit defaults alongside hidden valid omissions that receive FAIL; the paired harness comparisons hold model, task, and test cases constant.Source: §I.9 · pp. 28–30

Related papers

Loading 2610.28679v1…