LLM-as-a-Verifier: A General-Purpose Verification Framework

Source-linked AI summary

LLM-as-a-Verifier: A General-Purpose Verification Framework

arXiv:2607.05391v2

TL;DR

Standard LM judges provide coarse, discrete scores that limit fine-grained verification of candidate solutions. LLM-as-a-Verifier computes continuous rewards from scoring-token logits and achieves state-of-the-art performance across coding, robotics, and medical benchmarks.

Problem
Standard LM judges provide coarse scores that often tie distinct trajectories, motivating a generalizable framework for fine-grained verification signals.
Method
LLM-as-a-Verifier computes expected scores over scoring-token logit distributions and scales verification through granularity, repeated evaluation, and criteria decomposition.
Results
LLM-as-a-Verifier achieves state-of-the-art performance across coding, robotics, and medical benchmarks, including 86.5% on Terminal-Bench V2.
Takeaways & Limitations
The fine-grained verifier signal supports trajectory ranking, agent and robot progress monitoring, and denser rewards for reinforcement learning.
Takeaways & Limitations
The framework assumes access to scoring-token logits, excluding frontier models available only through restricted APIs.

Abstract

from arXiv · show

1. Introduction

The paper introduces LLM-as-a-Verifier, a training-free probabilistic framework that uses scoring-token logit distributions to provide fine-grained verification feedback. It characterizes verification scaling through score granularity, repeated evaluation, and criteria decomposition, enabling trajectory evaluation, efficient candidate ranking, and denser rewards for reinforcement learning.

Framework: LLM-as-a-Verifier estimates candidate quality from the expectation over scoring-token logits, providing dense, fine-grained feedback without additional training.This probabilistic formulation addresses limitations of discrete LM judges that can produce biased or noisy scores.
Verification Scaling: Verification accuracy improves by scaling score-token granularity, repeated evaluations, and decomposition of evaluation criteria.Repeated evaluations reduce variance, while criteria decomposition reduces prompt bias; these benefits are compared with a discrete LM judge under controlled budgets.
Agentic Evaluation: The verifier evaluates entire agent interaction trajectories and, with a cost-efficient ranking algorithm, outperforms frontier models across coding, robotics, and medical benchmarks.The approach also correlates verifier scores with chronological step order, supporting task-progress monitoring and extensions for Claude Code and Codex.
Reinforcement Learning: On LIBERO, LLM-as-a-Verifier provides ≈1.8× higher sample efficiency than sparse-reward baselines, while on MATH it provides ≈1.1× higher sample efficiency.The LIBERO result also reaches a higher final success rate when fine-tuning a π0 policy with DSRL-SAC; the MATH result uses GRPO to fine-tune Qwen3-8B.

2. Preliminaries

The paper models agent–environment interaction as a finite-horizon MDP and specializes this framework to LLM-based agents. It also formalizes conventional language-space reward models as discrete token-generated scores.

2. Preliminaries: Agent–environment interaction is modeled as a finite-horizon Markov Decision Process (MDP).The MDP includes contexts, states, actions, transition dynamics, rewards, and horizon H.
2. Preliminaries: Each episode samples a task prompt, starts from an initial state, and proceeds through H state–action transitions.The agent observes each state, selects an action, and transitions according to P conditioned on the prompt, state, and action.
2. Preliminaries: For LLM-based agents, states are prior interaction histories, actions are token sequences, and a language model π_θ samples actions autoregressively.The policy maps contexts and states to a distribution over actions.
2. Preliminaries: Conventional reward models prompt LLMs to generate discrete language-space scores, formalized as R_LM(x, τ) ∈ {1, . . . , G}.The score is the generated token, and reward models assign scalar scores to actions or trajectories.

3. Proposed Approach: LLM-as-a-Verifier

LLM-as-a-Verifier is a probabilistic framework that provides fine-grained verification by scaling score granularity, repeated evaluation, and criteria decomposition. It also introduces a budget-efficient pivot tournament for selecting among many candidate trajectories.

Motivation: Oracle Pass@K reaches 98.9% on Terminal-Bench V2 when sampled trajectories are pooled and an oracle verifier selects the optimal trajectory.This result motivates verification methods that can identify correct solutions among repeated samples.
Motivation: Standard language-model judges produce coarse evaluations because discrete score outputs often tie complex solutions, yielding a 27% tie rate on Terminal-Bench.Reward models [15] are presented as an alternative, but their dependence on training data can limit cross-domain generalization.
Fine-Grained Reward Estimation: The framework estimates fine-grained rewards from score-token distributions, repeated evaluations, and decomposed criteria instead of collapsing each evaluation into one discrete score.Continuous rewards are normalized and converted into pairwise preferences using the Bradley–Terry model.
Probabilistic Pivot Tournament: PPT reduces pairwise verification cost from O(N^2) to O(Nk) by comparing candidates against k ≪ N pivots selected from empirical ring-pass leaders.The random Hamiltonian ring pass places every candidate once in each prompt position, canceling positional bias in expectation; pivot comparisons then aggregate normalized win rates.
Probabilistic Pivot Tournament: PPT outperforms prior ranking approaches such as V1 [5] while requiring fewer comparisons, and its performance improves consistently as the number of pivots increases.The benchmark uses 20 trajectories per task curated with the Terminus-2 harness and evaluates the budget–accuracy trade-off in Table 9.

4. Verification Scaling

Verification scales along three complementary axes—score granularity, repeated evaluations, and evaluation criteria—that target distinct reward-estimation errors. Finer scoring improves calibration, repeated evaluations reduce variance, and criterion ensembles improve long-horizon judgment accuracy.

Scaling Axes: Scaling verification combines score granularity, repeated evaluations, and evaluation criteria, with each axis addressing a distinct source of reward-estimation error.Granularity improves score separation, repeated evaluation averages verifier-pass biases, and criteria decompose correctness into separately verifiable factors.
Score Granularity: 0.775 to 0.799: increasing scoring tokens from G=1 to G=20 raises SNR on Terminal-Bench, producing better-calibrated separation between correct and incorrect trajectories.SNR measures signal strength relative to inconsistency in pairwise score gaps, and pairwise verification accuracy increases monotonically with SNR.
Score Granularity: 77/100: G=20 ranks the correct query-optimize trajectory higher, versus 69/100 using expectations over the same 1–5 scale, while discrete judging ties 88/100 times.The case study uses 100 repeated evaluations on Terminal-Bench V2; probabilistic scoring eliminates the coarse judge’s ties and further sharpens discrimination.
Repeated Evaluation: Averaging K independent evaluations reduces estimator variance as O(1/K) without changing bias, complementing granularity by removing noise that finer scoring cannot eliminate.A single-pass verifier at K=1 already matches a heavily ensembled judge at K=16, indicating that fine-grained probabilistic scoring provides a stronger signal.
Evaluation Criteria: 78.3%: an ensemble of Specification, Output, and Errors criteria outperforms individual criteria, which achieve 75.2%–76.4% accuracy on long-horizon tasks.These criteria assess task requirements, final-output formatting, and failure signals; the final reward averages expected scores across them.

5. Experiments

Across four benchmarks spanning coding, robotics, and medical domains, LLM-as-a-Verifier selects among candidate trajectories via pairwise probabilistic pivot tournaments and consistently improves over Pass@1 while recovering substantial oracle headroom. The method also achieves state-of-the-art performance on each benchmark and generalizes across candidate pools and harnesses.

Evaluation protocol: Across coding, robotics, and medical benchmarks, a generation policy produces N candidate trajectories, the verifier scores every pair with a probabilistic pivot tournament, and the highest normalized-score trajectory is submitted.The evaluation covers Terminal-Bench V2 [17], SWE-Bench Verified [18], RoboRewardBench, and MedAgentBench.
Overall results: LLM-as-a-Verifier consistently improves over Pass@1, recovers a large portion of oracle Pass@N headroom, and achieves state-of-the-art performance on all four benchmarks.Table 3 reports per-benchmark baseline, Pass@1, oracle Pass@N, and verifier accuracy under a fixed agent harness.
Terminal-Bench V2: Terminal-Bench V2 accuracy rises from 83.1% to 86.5%, surpassing several competing agent systems and setting a new state of the art on Terminal-Bench V2.1.The gains also generalize beyond the primary harness to Terminus-2 and Terminus-Kira.
SWE-Bench Verified: On SWE-Bench Verified, the verifier selects the strongest trajectory from a heterogeneous pool of N=3 candidates produced by different model families.This setting contrasts with the homogeneous proposal pool used on Terminal-Bench and tests selection across diverse candidates.
RoboRewardBench: On RoboRewardBench, LLM-as-a-Verifier evaluates multi-frame rollout videos by integrating visual context to judge which trajectory makes more physical progress, outperforming trained robotics reward models.The benchmark uses paired rollouts following the same natural-language instruction but differing in progress.
MedAgentBench: MedAgentBench stress-tests general-purpose verification on medical EHR tasks involving information retrieval, guideline lookup, and multi-step tool use, where verification errors have safety consequences.The evaluation samples N=5 trajectories per task using the AgentBench harness and Claude Opus 4.8.

6. Fine-grained Verifier Signals as a Proxy for Task Progress

LLM-as-a-Verifier’s fine-grained scores provide a scalar progress proxy, quantified by Value-Order Correlation (VOC), and distinguish successful from stalled or failing trajectories. On robotics, it substantially outperforms competing progress estimators, while TurboAgent extends the verifier to existing coding-agent clients without modifying their harnesses or backend models.

Signal definition: VOC is the Spearman rank correlation between chronological step index and verifier-predicted value for the corresponding trajectory prefixes, following Ma et al. [22].A progress-tracking verifier should assign monotonically higher scores to later steps.
VOC on code generation: A 0.08 Spearman gap separates successful and failed Terminal-Bench V2 trajectories, with successful rollouts showing near-monotonic verifier progress.Failed rollouts exhibit weaker or inconsistent correlation, making VOC useful for detecting limited progress.
Coding Agent Extension: TurboAgent is an inference-time proxy for Claude Code and OpenAI-API-compatible clients that requires no changes to the agent harness or backend model.Its transparent proxy design also supports integration with existing benchmarks.

7. Dense Reward for Reinforcement Learning

LLM-as-a-Verifier provides dense rewards for both off-policy and on-policy reinforcement learning, improving sample efficiency without reward-model training or environment-specific shaping. In off-policy RL, verifier progress rewards improve LIBERO performance and reduce the environment steps needed to reach matched success rates.

Dense rewards: Verifier progress rewards are a drop-in dense signal for off-policy and on-policy RL, improving sample efficiency without reward-model training or environment-specific shaping.The approach targets the credit-assignment problem by supplying fine-grained scores as intermediate rewards.
Off-policy RL: In off-policy DSRL-SAC, verifier scores are computed over rendered rollout frames, used to relabel stored transitions, and incorporated into replay-buffer returns without changing the SAC objective.The coefficient λ trades off environment and verifier rewards, adding dense signals offline at no additional algorithmic cost.
On-policy RL: In on-policy GRPO, probabilistic pivot-tournament preferences assign normalized reasoning-trace scores, preventing zero advantages when all sampled answers are initially incorrect.This supplies a learning signal when group-relative advantages would otherwise collapse to zero.
Empirical findings: 1.8× higher sample efficiency lets DSRL-SAC on LIBERO reach success rates from 0.2 to 0.6 with verifier shaping, compared with sparse rewards.The verifier progress reward also reaches a higher final success rate: 0.76 vs. 0.69.

8. Discussion

The paper frames verification as an underexplored axis of scaling and proposes LLM-as-a-Verifier, a general-purpose framework for fine-grained feedback on agentic tasks. Unlike standard LM judges that output a single discrete score, it computes a continuous reward from scoring-token logit expectations.

LLM-as-a-Verifier treats verification as an underexplored axis of scaling for agentic tasks.
The framework delivers fine-grained feedback rather than a single discrete score.
It computes a continuous reward by taking the expectation over scoring-token logit distributions.

9. Related Work · Appendix

Related work spans test-time scaling, LLM-based judging, and verifiable rewards. These approaches respectively improve inference through additional computation, provide scalable output evaluation, and generate feedback for selecting or optimizing candidate behavior.

9. Related Work: Test-time scaling improves model performance by allocating additional inference compute to deliberation, search, or candidate generation.
9. Related Work: Single-response scaling elicits intermediate reasoning [25] [26], decomposes problems into simpler subproblems [27], or marginalizes over sampled reasoning paths [28].
9. Related Work: Other scaling methods search over intermediate thoughts [29], actions [32], or latent world states using test-time feedback.
9. Related Work: LLM-as-a-judge methods offer a scalable alternative to human evaluation by prompting large models to score or compare generated outputs.
9. Related Work: Judge-based evaluation extracts richer scoring signals through probability- and form-filling-based evaluators [43] [44] and uses LLM preferences for benchmark-style assessment [45] [46] [47].
9. Related Work: Verifiable reward models convert candidate solutions, actions, or trajectories into scalar feedback for selection, monitoring, or policy optimization.

A. Limitations and Future Work … B.3. LLM-as-a-Verifier as a Process and Outcome Reward Model

LLM-as-a-Verifier generalizes across agent harnesses and supports both process and outcome verification, while its current dependence on accessible scoring-token logits and incomplete scaling axes motivate future work. Efficient tournament selection further reduces verification cost while preserving accuracy gains.

A. Limitations and Future Work: The framework’s current requirement for scoring-token logits excludes frontier models exposed only through restricted APIs.A two-stage workaround routes a closed model’s reasoning through an open verifier with accessible logits and recovers most of the gain.
A. Limitations and Future Work: The proposed scaling axes are not exhaustive, motivating future work on learning criteria decomposition.The supplied discussion identifies criteria decomposition as a direction for extending the framework’s scaling approach.
B. Additional Results and Analyses: The additional results evaluate the verifier beyond the main Capy scaffold using Terminus-Kira with Claude Opus 4.6 and Terminus-2 with GPT-5.3-Codex, sampling N=5 trajectories with G=20 and K=8.Both harnesses use the same Gemini 2.5 Flash verifier and the three-criterion decomposition from Section 4.3.
B.1. Agent Harness Generalization on Terminal-Bench V2: LLM-as-a-Verifier transfers across Terminus-Kira and Terminus-2, achieving 79.4% and 71.2% accuracy and gaining approximately 5 and 2.7 points over the strongest relevant baselines, respectively.The gains persist across different harness setups, observation formats, and tuned models, indicating reasoning about terminal state and task progress rather than scaffold-specific syntax.
B.2. Probabilistic Pivot Tournament: Budget–Accuracy Trade-off: 66.17% selection accuracy with k=3 and 4,723 queried pairs already surpasses the best V1 result under a comparable verification budget, while PPT improves further as the pivot count increases.PPT uses a ring pass to select pivots and compares remaining candidates only against them, reducing verification from quadratic to approximately O(Nk).
B.3. LLM-as-a-Verifier as a Process and Outcome Reward Model: 9.5%, 18.5%, and 21.3% absolute pass@1 improvements over the base model occur on SWE-Bench Lite, AIME, and HMMT, respectively, while LLM-as-a-Verifier outperforms pointwise and pairwise baselines with a smaller budget than V1 [5].As a PRM, pass@1 also rises monotonically with sampled actions per step, from 48.7% to 55.7% on TauBench and from 49.8% to 54.3% on Terminal-Bench as k increases from 1 to 9, at 3× less compute than V1 [5].

B.4. Case Study: query-optimize

The query-optimize case study shows that verification must compare outputs on the canonical, unmodified database: indexing a copy makes the check unsound, whereas waiting for the original query enables a valid diff. Across 16 traces, Gemini 2.5 Flash reliably identifies this methodological failure.

Task specification: The task requires optimizing an OEWN SQLite query while preserving its output and saving a single semicolon-terminated query in /app/sol.sql.The trajectory pair uses the OpenHands harness, with Claude Opus 4.5 proposing and Gemini 2.5 Flash verifying.
Ground-truth breakdown: The failing trajectory receives reward 0 because it compares the optimized query on the canonical database with the original query on an indexed copy, changing physical access paths.Both trajectories save optimized SQL and pass an internal diff check, but only one matches the hidden grader’s result.
Ground-truth breakdown: The correct trajectory waits 5m03s for the original query on the canonical database, then runs a direct diff with exit code 0.This preserves the task’s implicit requirement that outputs be compared against the original query on the unindexed database.
Gemini 2.5 Flash reasoning trace: Across 16 reasoning traces with thinking enabled, Gemini 2.5 Flash reliably identifies that modifying a copied database violates the task’s verification constraint.The verifier notes that the original query was interrupted twice before the agent indexed a database copy to obtain a reference output.

B.5. Scaling Repeated Evaluation and Visual Context on RoboRewardBench · B.6. Recovering Continuous Rewards for Logit-Restricted Frontier Models · B.7. LLM-as-a-Verifier as a Dense Reward for RL

Repeated evaluation improves RoboRewardBench accuracy and transfers across visual robotic inputs, while a two-stage open-verifier workaround recovers continuous rewards for logit-restricted frontier models. The appendix also specifies verifier-based dense rewards for both off-policy and on-policy RL under otherwise matched experimental conditions.

B.5. Scaling Repeated Evaluation and Visual Context on RoboRewardBench: The RoboRewardBench results show that repeated-evaluation gains transfer from text to multi-frame robotic manipulation despite the changed input modality.LLM-as-a-Verifier beats LLM-as-a-Judge, TOPReward, RoboReward-8B, and Robometer-4B at every evaluation budget.
B.5. Scaling Repeated Evaluation and Visual Context on RoboRewardBench: 87.4% trajectory-preference accuracy at K=8, up from 81.5% at K=1, with LLM-as-a-Verifier outperforming all baselines across budgets.Gains saturate at larger K as the noise floor is reached.
B.6. Recovering Continuous Rewards for Logit-Restricted Frontier Models: Because frontier APIs such as GPT-5.5 and Claude Opus 4.7 may expose completions without token logprobs, direct verifier substitution cannot compute Eq. 3.1.The workaround decouples reasoning from scoring to recover most of the calibrated reward signal.
B.6. Recovering Continuous Rewards for Logit-Restricted Frontier Models: The two-stage pipeline has GPT-5.5 generate free-form reasoning and Gemini 2.5 Flash score that reasoning with logprobs at score-token positions to produce continuous rewards.The evaluation uses the task, both trajectories, and GPT-5.5’s reasoning as Gemini’s input.
B.6. Recovering Continuous Rewards for Logit-Restricted Frontier Models: +2.1 points at K=16, with zero ties, shows the continuous workaround retains an advantage even at the largest tested budget.Table 12 compares accuracy and tie rate for discrete GPT-5.5 scoring with the GPT-5.5-to-Gemini 2.5 Flash continuous pipeline.
B.7. LLM-as-a-Verifier as a Dense Reward for RL: In on-policy RL, GRPO adds group-standardized Gemini 2.5 Flash preferences to correctness and format rewards with weight β=0.1.The setup uses Qwen3-8B on Hendrycks MATH with group size M=16.