PaperlayerView exact revision on arXiv

Source-linked AI summary

VeriFine: Scaling Verification for Self-Improvement in Embodied Reasoning

Zewei Zhou, Rachel Luo, Yulong Cao, Chaowei Xiao, Chensheng Peng, Boyi Li, Thomas Tian, Zheng Lian, Yan Wang, Jiaqi Ma, Boris Ivanovic, Marco Pavone, Wenhao Ding

arXiv:2610.08761v1cs.AIcs.RO

TL;DR

Static judges and training data can become inadequate as self-improving policies expose new failure patterns, especially in embodied reasoning. VeriFine couples failure-driven policy and curriculum updates with selective human–agent judge calibration when verification becomes a bottleneck. Experiments report improvements in both policy and judge capability across driving and robot navigation with reinforcement and supervised fine-tuning.

  • Problem

    Static judges can become bottlenecks as policies improve, and static training data can offer limited learning signal once scenarios are solved.

    Source: §1 · p. 2

  • Method

    VeriFine couples a reference-free judge-driven policy and curriculum loop with selective human guidance and coactive judge calibration when verification becomes a bottleneck.

    Source: §1 · p. 2

  • Results

    Experiments demonstrate improvements in both policy and judge capability across driving and robot navigation using reinforcement and supervised fine-tuning.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The results show that scaling verification supports continuous self-improvement as policy failure patterns evolve.

    Source: §5 · p. 10

    Policy evaluation retains a fixed anchor set, and the framework still relies on human-verified reasoning for policy evaluation.

    Source: §A.5 · p. 27

Abstract

from arXiv · show

Self-improving policies continually expose new failure patterns, changing what their judges must be able to verify. However, current fixed judges constrain both optimization feedback and the discovery of useful training examples, limiting further self-improvement. This challenge is even more acute in embodied reasoning, where reliable evaluation must account for spatial grounding, causal reasoning, and safety-aware decision-making. We introduce VeriFine, an agent harness framework that scales verification through the co-evolution of the policy, training curriculum, and judge. The Policy Improvement Loop uses a rubric judge to diagnose recurring failures, construct an adaptive curriculum, and optimize the policy. When progress plateaus and verification becomes a bottleneck, the Judge Improvement Loop selectively queries human guidance on informative failure cases and refines the judge through coactive calibration, in which humans and agents resolve disagreements and converge toward the objective rubric of physical reasoning. The revised judge then guides the next stage of data selection and policy optimization. Experiments on driving and robot navigation tasks demonstrate continuous self-improvement in both policy and judge capability across reinforcement and supervised fine-tuning. These results show how scaling verification supports continuous self-improvement as policy failure patterns evolve.

1. Introduction

Static judges and static training data can become inadequate as policies improve, motivating verification that co-evolves with the policy and curriculum. VeriFine uses coupled policy- and judge-improvement loops, and experiments report improvements in policy and judge capability across driving and robot navigation.

1. Introduction

  • Static judges can become a bottleneck as policies improve, while solved scenarios leave static training data with limited learning signal.The paper asks how the judge, curriculum, and policy can co-evolve to keep pushing the performance frontier.Source: §1 · p. 2
  • Reliable embodied-reasoning verification is challenging because it must account for spatial grounding, temporal understanding, causal reasoning, and safety-aware decisions.Prior evaluators often use constrained question-answering formats and costly human references, while free-form embodied reasoning remains underexplored.Source: §1 · p. 2
  • VeriFine couples a failure-driven Policy Improvement Loop with a Judge Improvement Loop that selectively queries humans and calibrates the judge through human–agent disagreement resolution.The policy loop selects high-value data and optimizes without human intervention; judge refinement is activated when self-improvement plateaus near the judge’s verification boundary.Source: §1 · p. 2
  • VeriFine demonstrates self-improvement in policy reasoning and judge capability across driving and robot navigation, supporting its applicability to reinforcement and supervised fine-tuning.Driving uses reinforcement fine-tuning; robot navigation uses supervised fine-tuning.Source: §1 · pp. 2–3

2. Related Work

Prior work studies verification, agent self-improvement, and embodied-reasoning evaluation, but the cited approaches leave open how to scale verification while connecting adaptive data selection, policy optimization, judge evolution, and selective human calibration.

2.1. Scaling Verification.

  • Existing verification work covers inference-time scaling and training feedback, but does not close the loop among adaptive data selection, policy optimization, judge evolution, and selective human calibration.Verification can support candidate selection, policy objectives, and filtering or prioritizing training data.Source: §2.1 · p. 3

2.2. Self-improvement and Autoresearch.

  • Agentic self-improvement makes verifier adaptation relevant as policies shift, and selectively acquired human guidance can recalibrate judges near their effective limits.The paper connects emerging policy failures to human–agent calibration and subsequent data selection and policy optimization.Source: §2.2 · p. 3

2.3. Embodied Reasoning.

  • Embodied-reasoning evaluators often emphasize constrained question-answering, costly human references, trajectory quality, task progress, or action alignment rather than free-form reasoning grounded in video.Direct video-grounded verification of free-form embodied reasoning without costly human-annotation references remains underexplored.Source: §2.3 · p. 3

3. VeriFine

VeriFine treats scaling verification as co-evolution among a policy, its training curriculum, and a reference-free judge. Its loops use judge feedback to select and optimize training data, then selectively acquire human guidance to refine the judge when verification becomes a bottleneck.

3. VeriFine

  • The coupled framework links policy optimization, curriculum construction, and judge evolution through repeated improvement loops.When verification becomes a bottleneck, informative policy failures are used to refine the judge so it can guide subsequent learning.Source: §3 · p. 4, §3.1 · p. 4, §3.4 · p. 6

3.1. Problem Formulation and Framework Overview

  • The framework maintains supervision state across rounds, while experiments restart each policy from the same base initialization and use a fixed per-policy training budget.System-level improvement therefore accumulates through verification and curriculum changes across rounds.Source: §3.1 · pp. 4–5

3.2. Reference-Free Rubric Judge as the Verification Substrate

  • VeriFine’s reference-free rubric judge evaluates policy outputs from physical context and returns an overall score, structured diagnostic scores, and an optional explanation.A teacher–student design distills a frontier vision-language model’s evaluation capability into a compact student for efficiency; the rubric includes action and component criteria.Source: §3.2 · p. 5

3.3.1. Failure Discovery and Adaptive Curriculum

  • The Policy Improvement Loop aggregates evaluations to find recurring failures, distinguishing systematic errors from isolated ones, then turns patterns into curriculum selection criteria.Selection prioritizes contexts where the policy needs improvement while filtering unreliable supervised targets; the curriculum is revisited as policy weaknesses and the judge change.Source: §3.3.1 · p. 5

3.3.2. Judge-Guided Policy Optimization

  • Driving policy updates use reinforcement fine-tuning with the judge’s overall score as reward, while navigation uses supervised fine-tuning on judge-selected high-value, high-quality demonstrations.For supervised fine-tuning, judge scores determine training-set membership rather than directly entering the optimization objective; candidate policies are monitored consistently.Source: §3.3.2 · pp. 5–6

3.4.1. Boundary Detection and Human Query Selection

  • The Judge Improvement Loop monitors policy progress on a human-annotated anchor set and triggers when progress plateaus or judge-reward change diverges from measured performance change.The query set is built from newly updated policy outputs and includes uncertain, inconsistent, and high-impact cases, retaining both positive and negative examples.Source: §3.4.1 · p. 6

3.4.2. Coactive Calibration

  • Coactive calibration adds selected evaluation cases to a cumulative judge set, then iteratively revises the rubric with human input on disputed evaluations.Humans can confirm or correct rubric dimensions and clarify criteria; humans and agents revisit disagreements until alignment gains plateau.Source: §3.4.2 · pp. 6–7

4. Experiments

VeriFine is evaluated through driving and robot-navigation self-improvement, with results covering policy performance, judge quality, test-time selection, and framework generalization. Across these settings, policy and judge capabilities improve as verification and training curricula evolve.

4.1. Main Experimental Setup

  • Driving experiments used three self-improvement rounds, a 2-million-clip dataset, matched policy-training budgets, and fixed reference-based policy evaluation.The setup also used separate policy and judge test sets, with test labels excluded from rubric refinement.Source: §4.1 · pp. 7–8

4.2. Self-Improvement Results with Scaling Verification

  • Driving reasoning rose from 60.56 to 71.61 (18.2% relative improvement), while navigation reasoning rose from 68.09 to 77.59 (14%).The navigation results also report final judge–human correlation approaching 0.85.Source: §4.2 · p. 9, §4.4 · p. 10, Figure 4 · p. 10
  • With the reward judge fixed, judge-guided selection raised reasoning scores from 64.03 to 67.70 over random selection; later rounds reached 70.59 and 71.61.The final policy also achieved the lowest ADE, and the final reference-free evaluator showed a 22.2% relative gain across rounds.Source: §4.2 · p. 9
  • Teacher judge correlation rose from 0.55 to 0.85; the final student reached 0.82 correlation and 0.17 MAE, comparable to the reference-based judge.On later challenging scenarios, student correlation increased from 0.07 in R1 to 0.74 in R2 after refinement.Source: §4.2 · p. 9

4.3. Test-Time Scaling with Evolved Judge (Q3)

  • At six candidates, the final judge selected outputs scoring 78.6 on human-rated reasoning, outperforming R2 and LingoJudge and performing comparably to VeriFine-Judge-RB.Candidate generation was held fixed, so the reported gains reflect selection rather than a stronger generating policy.Source: §4.3 · pp. 9–10

4.4. Framework Generalization to Robot Navigation

  • For navigation, VeriFine used supervised fine-tuning, with the judge shaping the training curriculum rather than supplying an RL reward.The reported navigation results show policy and judge improvement as the optimization focus shifts toward exploration efficiency.Source: §4.4 · p. 10

5. Conclusions

VeriFine couples policy, curriculum, and judge evolution to support continuous self-improvement as policy failure patterns change.

5. Conclusions

  • VeriFine coordinates the evolution of policy, training curriculum, and judge to scale verification for continuous self-improvement.Its Policy Improvement Loop adapts curriculum to recurring failures, while the Judge Improvement Loop uses selective human guidance for coactive calibration.Source: §5 · p. 10
  • A reference-free judge provides structured feedback connecting policy optimization with judge refinement.Source: §5 · p. 10
  • Experiments on driving and robot navigation report improvements in both policy and judge across reinforcement and supervised fine-tuning.Source: §5 · p. 10

A.1. Additional Experiment Results

The appendix reports additional analyses of VeriFine’s improvement loops and judge behavior.

A.1. Additional Experiment Results

  • Additional analyses examine reward supervision and curriculum construction.Source: §A.1 · p. 16
  • The analyses compare teacher backbones and student architectures.Source: §A.1 · p. 16
  • Judge performance is analyzed across evolving failure distributions.Source: §A.1 · p. 16

A.1.1. Additional Results on Policy Improvement Loop

These additional results separate reward supervision from curriculum selection and examine their roles in VeriFine’s coordinated policy–curriculum–judge improvement under matched training budgets.

A.1.1. Additional Results on Policy Improvement Loop

  • With random data selection, RFT using LingoJudge and VeriFine-Judge-RB improved reasoning scores by 4.8% and 5.7%, respectively, over the base policy.These results characterize reward-based optimization alone.Source: §A.1.1 · p. 16
  • The curriculum comparison replaces random selection with judge-guided selection while holding each reward judge fixed.Source: §A.1.1 · p. 16
  • With the reference-free judge, R1 matched the strongest reference-based configuration; subsequent rounds refined verification and curriculum as new failure patterns emerged.The results support coordinated verification and curriculum refinement under a fixed per-policy training budget.Source: §A.1.1 · pp. 16–17
  • Judge-guided selection improved RB reasoning scores over random selection by 14.6%, 8.1%, and 4.5% in R1, R2, and R3, respectively.The RF evaluator showed the same ordering; comparisons held reward judge, initialization, and training budget fixed within each round.Source: Table 2 · p. 17, §A.1.1 · p. 17

A.1.2. Additional Results on Judge Improvement Loop

Rubric-based judging and coactive calibration strengthen judge–human alignment, while round-specific evaluation sets expose and address failures as the policy distribution shifts.

Teacher Judge Model Backbone

  • With the final rubric held constant across backbones, rubric-based judging consistently outperforms judging without an explicit rubric, while coactive calibration substantially improves Opus 5 over its pre-calibration counterpart.Source: §A.1.2 · p. 17
  • Figure 5 compares calibration curves and Pearson correlation/MAE across teacher-judge backbone variants, with rubric variants sharing the same final rubric except without coactive calibration.The figure includes no straightforward-instruction variant without detailed rubrics.Source: Figure 5 · p. 18

Evolving Judge Test Sets

  • In policy-rollout cases with obstacles encroaching into the ego lane, which form subset R2 and expose substantial distribution shift, correlation improves from 0.07 for the R1 student to 0.74 for the R2 student.Separately, the R1 student judge achieves a Pearson correlation of 0.72 on subset R1.Source: §A.1.2 · p. 18
  • Round 3 adds no evaluation subset and raises pooled student correlation from 0.71 to 0.82, while the final teacher reaches 0.85.The round balances performance across accumulated scenarios and targets known failure cases.Source: §A.1.2 · p. 18

A.2. VeriFine-Judge: Reference-free Rubric Judge

VeriFine uses a reference-free rubric judge to score embodied reasoning from physical context, then calibrates a teacher with human input and distills its structured evaluations into a compact student. Driving and navigation use task-specific rubric aggregation, and the student supports lower-cost verification.

A.2.1. Teacher Judge Model

  • A frontier VLM teacher evaluates candidate reasoning from visual context, ego state, task instructions, and a structured rubric; its structured evaluations are distilled into a compact student judge.The teacher returns rubric scores, a coherence diagnostic, an overall score, and a concise explanation.Source: §A.2 · p. 18, §A.2.1 · pp. 18–19
  • Figure 6 tracks calibration and judge–human alignment across evolving test subsets, improvement rounds, and reference-based baselines.Its calibration plots show mean judge score within each human-score bin; alignment is measured with Pearson correlation and MAE.Source: Figure 6 · p. 19
  • Driving judgment scores instruction match, safety, visual grounding, component completeness, and action–component causation, with reasoning coherence used as a diagnostic.An action-match gate removes action credit for instruction-inconsistent decisions, while a separate component term preserves credit for grounded scene understanding.Source: §A.2.1 · pp. 19–20
  • Navigation judgment evaluates safe movement and progress toward the destination, retaining grounding and causation without a separate completeness score.Unsafe movements receive zero overall credit; for manipulation instructions, the rubric evaluates navigation to the target rather than manipulation completion.Source: §A.2.1 · pp. 20–21
  • Driving and navigation aggregate rubric dimensions differently: driving preserves grounded scene-understanding credit despite unsafe actions, whereas navigation makes safety a prerequisite and emphasizes goal consistency.The distinction reflects driving’s critical scene elements and navigation’s need for productive exploration before target localization.Source: §A.2.1 · p. 21
  • Human experts revise rubric criteria through evidence-based discussion of remaining disagreements, accepting revisions only when current-batch alignment improves without substantial regression on earlier cases.Clarifications may correct a dimension, identify missing physical evidence, or specify an underspecified criterion.Source: §A.2.1 · p. 21

A.2.2. Student Judge Model

  • Frontier-VLM querying is costly and slow during high-frequency optimization, so the student predicts all rubric dimensions from the same physical context in one forward pass.Its training uses independent teacher-labeled samples and equal-weight binary cross-entropy over rubric dimensions; the overall score is not the direct training signal.Source: §A.2.2 · pp. 21–22
  • The driving student processes samples approximately 300 times faster than the teacher under the reported hardware settings, supporting online training and evaluation at low cost.The selected classifier head with rubric subscores provides interpretability rather than slower autoregressive generation.Source: §A.2.2 · p. 22

A.3. Evaluation and Testing Details

The section frames reliable evaluation as necessary for scaling verification and outlines evaluation sets and protocols for tracking both policy and judge through self-improvement.

A.3. Evaluation and Testing Details

  • The section introduces policy and judge evaluation sets and protocols used across self-improvement iterations.Reliable evaluation is presented as foundational to scaling verification.Source: §A.3 · p. 22

A.3.1. Driving Reasoning Evaluation and Testing

The evaluation design pairs a fixed policy set with a cumulative judge set that incorporates newly exposed failures, alongside independent tests and task-specific data resources. Metrics, reference-based comparisons, and loop protocols provide evaluation across iterations; the discussion identifies fixed policy evaluation and reasoning-annotated data as boundaries.

A.3.1. Driving Reasoning Evaluation and Testing

  • The fixed policy set tracks progress and supports verification-boundary detection, while the cumulative judge set supports human calibration and judge refinement using expert-verified samples.The judge set retains high- and low-quality outputs to identify false positives, refine rubric boundaries, and detect reward hacking.Source: §A.3.1 · pp. 22, 24

Data Curation

  • Robot-navigation curation converts VLNVerse trajectories into 19k destination-instruction samples linking visual evidence to next-step decisions without intermediate route commands.Claude Opus segments trajectories into movement intervals and generates landmark-grounded descriptions.Source: Data Curation · p. 24

Policy Evaluation Set of Robot Navigation Reasoning

  • The robot policy test set contains 220 samples from 46 episodes across 28 scenes; judge sets expand from 175/211 to 212/281 evaluation/test candidates after 107 contrast candidates target exploration failures.The policy evaluation and test sets contain 182 and 220 samples, respectively.Source: Policy Evaluation Set of Robot… · pp. 24–25

A.3.3. VeriFine-Judge-RB: Reference-Based Rubric Judge

  • VeriFine-Judge-RB is a fixed reference-based DeBERTa-v3-base cross-encoder initialized from Lingo-Judge and fine-tuned on approximately 500K reference–candidate pairs.Claude Opus, GPT, and Gemini provide rubric-based annotations whose scores are averaged as training supervision.Source: §A.3.3 · p. 25

A.4.1. Evaluation Metrics

  • Reasoning scores are reported as means scaled by 100; judge alignment uses Pearson correlation for relative quality and MAE for absolute calibration, while trajectory accuracy reports displacement error in meters.Higher correlation and lower MAE indicate better human alignment; lower trajectory error indicates better accuracy.Source: §A.4.1 · p. 25

A.4.2. Implementation Details

  • Driving uses GRPO reinforcement fine-tuning with a fixed 25K-sample curriculum selected from 400K candidates, while robot navigation uses supervised fine-tuning of Qwen3-VL 2B.The student judge is trained on 35K teacher-labeled examples; the reference-based model uses approximately 500K pairs.Source: §A.4.2 · p. 26

A.4.3. Loop Execution Protocol

  • Boundary checks occur every 150 training steps; curriculum rules select driving and navigation examples from observed failures, and rubric revisions are assessed on current and retained calibration cases.Human review is requested when average correlation improvement remains below 0.2 for more than ten iterations; independent test labels are frozen and excluded from refinement and model selection.Source: §A.4.3 · p. 26

A.5. Limitations and Future Work

  • The framework retains a fixed policy evaluation set and still relies on reasoning-annotated data, including human-verified reference reasoning for policy evaluation.Future work could add an adaptive policy evaluation set, evolve a reasoning annotator, and distill human guidance to reduce routine involvement.Source: §A.5 · p. 27

Related papers

Loading 2610.08761v1…