Source-linked AI summary
No Free Checker: A Survey of Verifiers for Robot Policies
Yang Wan, Xihang Yue, Zhirui Liu, Ziyuan Chu, Shuxun Wang, Yuhan Chen, Xiaonan Jiang, Xukun Zhu, Yubo Dong, Linchao Zhu
TL;DR
Robot-policy verifiers must balance costly, delayed judgments against credible evidence of task performance. This survey analyzes roughly 150 verifiers and finds that availability and credibility trade off, with no free checker.
Problem
Robot learning lacks cheap, well-defined verification criteria because success assessment, policy evaluation, early intervention, and continuous performance are difficult in the physical world.
Method
The paper surveys roughly 150 verifiers, groups them by judgment source, and compares their availability and credibility.
Results
Across verifier families, availability and credibility move in opposite directions as signals become cheaper, earlier, and denser but less directly tied to task performance.
Takeaways & Limitations
There is no free checker: obtaining both high availability and high credibility for robot-policy verification is costly.
Takeaways & Limitations
Agreement rates and downstream policy gains were measured on non-adversarial candidates, so they can overstate verifier value under optimization.
Abstract
from arXiv · showhide
A verifier for robot policies reads a candidate behavior and returns a score for how well it did, used both to evaluate vision-language-action policies and to train them. Verifiers range from success detectors and reward models to runtime monitors, safety filters, and temporal-logic specifications. We survey roughly 150 verifiers and compare them along two properties. Availability is how much a verdict costs, how early in a rollout the verdict arrives, and how often a verdict can be asked for. Availability rises as verdicts get cheaper, earlier, and denser. Credibility is how much a high score tells us about the task. Credibility falls as the judgment becomes gameable and self-serving. We group the verifiers by who supplies the judgment: human verifiers, rule-based and formal verifiers, learned and pretrained verifiers, and model-intrinsic verifiers. Across the four families, we find that credibility falls as availability rises. Regardless of who supplies the judgment, there is no free checker. We then examine what validates a verifier itself, and how much a high score tells us. Three measures appear in the literature: agreement with human labels, the performance of the policy it trains, and behavior under reward hacking. We close with nine metrics that make a verifier claim checkable, and coordinates for the verifiers still to be built.
1 Zhejiang University, 2 City University of Hong Kong
The paper concerns robot policy evaluation, including reward models, vision-language-action models, runtime monitoring, and safety filters. It also addresses reward hacking, world models, and data curation for imitation learning.
- The paper covers robot policy evaluation, reward models, and vision-language-action models.
- It includes runtime monitoring and safety filters as evaluation-related topics.
- It also considers reward hacking, world models, and data curation for imitation learning.
1 Introduction
The introduction frames robot-policy verification as necessary but difficult in the physical world, where verifiers trade off availability against credibility. It surveys roughly 150 works across four judge-source families and identifies three ways to assess whether a verifier is trustworthy.
- Motivation: Robot verification supports training and evaluation, but physical tasks lack the cheap, well-defined criteria available in mathematics and code.Success is costly to infer from imperfect sensory observations, and policy evaluation may require time-consuming real-robot rollouts.
- Survey scope: The survey covers roughly 150 works and compares verifiers by availability—the cost, timing, and density of verdicts—and credibility—their connection to actual task performance.Availability rises when verdicts are cheaper, earlier, and denser, while credibility is weakened by gameability and self-serving judgments.
- Verifier families: Existing methods are grouped into four families: human, rule-based and formal, learned and pretrained, and model-intrinsic verifiers.These families differ in who supplies the judgment, from direct human assessment and predefined criteria to neural models and signals computed by the policy or predictive model.
- Availability–credibility trade-off: Across the four families, availability and credibility exhibit a trade-off: direct judgments and explicit criteria are more task-relevant, whereas learned and model-intrinsic signals are cheaper and denser.Human judgments remain costly and explicit criteria depend on accurate state estimates and physical assumptions; learned judgments depend on model accuracy, calibration, and generalization.
- Verifier validation: The paper evaluates verifier credibility using agreement with a fixed reference, the policy produced by verifier-based training, and behavior when searching for inputs where the verifier is wrong.It closes by identifying reporting metrics that make verifier claims checkable by others.
2 Human Verifiers
Human verifiers assess robot behavior through trajectory comparisons, time-indexed scores, and interventions during execution, while also validating generated rollouts. Their judgments commonly become supervision for cheaper learned verifiers, trading human cost for possible distributional and gaming risks.
- Trajectory comparisons: Preference comparisons produce relative judgments by asking which of two trajectory segments shows better task performance.RoboArena aggregates double-blind human comparisons of fixed policies into a policy ranking.
- Time-indexed scoring: Reward sketching produces scalar supervision by tracing perceived progress from 0 to 1 across each time step of a recorded trajectory.A reward model trained on a small set of curves can score every time step in a much larger corpus.
- Four human-verifier roles: Human verification comprises trajectory comparisons, time-indexed scoring, execution-time intervention, and validation of generated rollouts.The first three roles judge real-robot behavior; the fourth judges simulated or world-model rollouts for outcome preservation and video fidelity.
- Execution-time intervention: During execution, human-gated supervisors take over and provide corrective actions, preventing failures while supplying training examples where the policy needs help.Robot-gated methods automate help requests when states appear unfamiliar or likely to cause failure, subject to intervention budgets.
- Scaling human judgments: Across all four roles, human verdicts supervise cheaper verifiers, including reward models, corrective-demonstration buffers, and rollout-rating models.The substitution cost depends on how far the cheaper verifier’s training data is from the data to which it is later applied.
3 Rule-Based and Formal Verifiers
Rule-based and formal verifiers apply prewritten or generated criteria without human judgment at inference time, reading full trajectories, terminal states, or physical constraints. Their outputs range from formal margins and geometric scores to binary goal tests and safety guarantees.
- 3 Rule-Based and Formal Verifiers: Rule-based and formal verifiers return verdicts without asking a person at inference time, reading full trajectories, terminal states, or physical constraints.The section groups these procedures by what their criteria read.
- 3.1 Full-Trajectory Verification: Full-trajectory verification includes temporal-logic scoring, geometry-based trajectory scoring, and model-written verification code.These methods process complete state sequences or rollouts.
- 3.1.1 Temporal-Logic Scoring: Temporal-logic scoring assigns a satisfaction margin: positive values indicate satisfaction, negative values violation, and magnitude measures distance from the boundary.The same margin applies to recorded executions, simulated rollouts, and predicted future trajectories.
- 3.1.2 Geometry-Based Scoring: Geometry-based scoring evaluates trajectory shape without a hand-written task specification, using recorded demonstrations when available.RINSE combines spectral arc length and trajectory envelope distance as reference-free measures.
- 3.1.2 Geometry-Based Scoring: Geometry scores measure execution quality rather than task success, so smooth failures and jerky successes remain possible.A separate check is needed to determine whether the task was completed.
- 3.1.3 Model-Written Rules: Model-written verification logic serves both as a training objective and as a runtime check.Generated code can produce reward functions for policy optimization or spatio-temporal constraint verifiers for monitoring.
- 3.2 Terminal-State Verification: A goal predicate tests only the final state and returns a binary result rather than a margin.It is cheap and supports benchmark scoring, training-data generation, and reinforcement-learning rewards, but perception-estimated poses confine it to simulation.
- 3.3 Physical-Constraint Verification: Physical-constraint verification can use barrier functions to prove forward invariance, ensuring every trajectory under the closed-loop dynamics remains within an allowed region.Related results extend guarantees to an 𝜖-optimal approximate value function and evaluate learned approximations with validity and coverage metrics.
4 Learned and Pretrained Verifiers
Learned and pretrained verifiers serve three roles in robot learning: evaluating behavior, selecting actions or plans at inference time, and feeding feedback into policy training or data selection. They range from process-level scores localized within rollouts to trajectory-level outcome judgments, and their outputs can guide both decision-making and policy improvement.
- Overview: Learned and pretrained verifiers evaluate executions or policies, select candidate actions, skills, or plans, and provide feedback for policy training or data curation.These roles correspond to evaluation, inference-time selection, and training or data-selection feedback.
- Behavior and policy evaluation: Process-level verifiers assign localized scores or judgments to states, transitions, clips, trajectory prefixes, or subtasks, whereas trajectory-level verifiers return one holistic verdict for a completed rollout.Policy evaluations and rankings can aggregate process- or trajectory-level outcomes across repeated rollouts.
- Behavior and policy evaluation: Pretrained and learned models provide dense or sparse feedback, from frame-level goal similarity rewards to terminal success classifications and holistic video-language judgments.Examples include frame-wise similarity rewards, end-of-episode sparse rewards, and binary or ordinal trajectory-level outcomes.
- Inference-time selection: At inference time, verifiers score candidate actions, skills, or plans and execute the highest-scoring option, while world-model methods use predicted consequences or monitoring to guide selection.Some systems add test-time computation through direct candidate scoring, uncertainty estimation, or predictive lookahead.
- Policy training and data curation: Verifier feedback changes the policy through reward or preference optimization, self-improvement loops, rollout filtering, and curation or reweighting of demonstrations.The feedback may enter the training objective directly or determine which generated experiences and preexisting demonstrations are used for training.
5 Model-Intrinsic Verifiers
Model-intrinsic verifiers reuse quantities computed by the robot’s policy or learned world model to detect failures, select actions, judge outcomes, or choose training environments. This makes them the survey’s cheapest verifiers, but their scores can reward failures the model does not recognize.
- 5 Model-Intrinsic Verifiers: Model-intrinsic scores come from the robot’s existing policy or learned world model, supporting execution monitoring, candidate selection, outcome verification, and environment selection.Policy-based self-verification reads policy signals, while prediction-based outcome verification reads predicted consequences from a learned world model.
- Policy-based self-verification: Policy-based self-verification judges whether behavior looks reliable or familiar to the policy, not whether the task succeeded.A familiar-looking failure may therefore still receive a high score.
- Prediction-based outcome verification: World-model verifiers use trajectory likelihood, latent discrepancy, model uncertainty, or reachability value to evaluate predicted outcomes and future safety.Reachability value can reject an unsafe action before execution by assessing whether failure remains avoidable under predicted dynamics.
- Section-wide tradeoff: Because model-intrinsic verifiers reuse models already running on the robot, they are the survey’s cheapest verifiers, while also inheriting those models’ blind spots.A model can therefore assign a high score to a failure it does not recognize.
6 Validating the Verifier
The section evaluates verifier error through agreement with references, downstream policy improvement, and behavior under reward hacking. It argues that training changes the inputs a verifier encounters, so claims require broader, comparable reporting through nine metrics.
- Validation criteria: A verifier’s error propagates across demonstration selection, policy or action ranking, reward supply, and runtime execution decisions.Because one verifier supports all four uses, one error can affect training data, reported rankings, rewards, and runtime control.
- Validation criteria: The literature evaluates verifier error through reference agreement, the policy produced by verifier-based training, and performance on high-scoring but unsuccessful candidates.These are the section’s three metrics for measuring a verifier itself.
- Agreement with references: Agreement rates omit disagreement attribution, confidence intervals, and rollout type, so they do not fully characterize verifier reliability.Agreement depends on whether the verifier or reference is wrong, the evidence supporting the rate, and the kind of input evaluated.
- Downstream policy improvement: A verifier can agree with people on ordinary rollouts yet fail on training-induced rollouts, because optimization drives policies toward regions where the verifier is wrong.The resulting policy gain is expensive to measure, rare in the literature, and can diverge from agreement, especially when the verifier supplies training rewards.
- Reward hacking: Reward hacking makes reported reward gains overestimates and invalidates release-time calibration as optimization reaches inputs that exploit verifier errors.A verifier used as a reward or runtime gate must remain reliable in regions reachable by search, not merely on unsought inputs.
- Reporting standards: The section proposes nine metrics whose reporting makes verifier claims comparable across papers and exposes the conditions omitted by a single agreement rate.The metrics extend evaluation from one fixed rollout set, reference, and agreement number to the conditions under which the verifier is tested and applied.
7 Conclusion
The survey organizes roughly 150 robot-policy verifiers into four judgment-source families and finds a consistent availability–credibility trade-off: cheaper, earlier, more frequent verdicts provide weaker evidence of task performance. Establishing verifier credibility is separate from building one and can rely on human-label comparisons or benchmarked error rates.
- Verifier families: Roughly 150 verifiers are grouped into human, rule-based and formal, learned and pretrained, and model-intrinsic families.The families are compared by availability and credibility.
- Availability–credibility trade-off: As judgment moves closer to the evaluated model, verdicts become cheaper, earlier, and more frequent, while evidence of actual task performance becomes harder to assemble.Availability and credibility move in opposite directions across the four families.
- Availability–credibility trade-off: Human verifiers provide the most direct reference to task intent but are costly, whereas rule-based and formal verifiers are inexpensive and repeatable when required state information is available.Rule-based and formal verifiers reach the survey’s strongest guarantees where their predicates apply.
- Credibility validation: Credibility assessment is separate from verifier construction and uses either comparisons ending in human labels or benchmarks reporting the verifier’s own error rate.The error of human labels is rarely reported alongside the claim they support.
Use of Large Language Models
Parts of the survey text were drafted with a large language model, while the authors checked every claim, citation, and count against the cited papers.
- Use of Large Language Models: The authors used a large language model to draft parts of the survey and verified every claim, citation, and count against the cited papers.This verification was performed by the authors.