PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation

Siyuan Ma, Yu Qi, Cheng Ye, Ruotian Peng, Xinyi Xu, Zitiantao Lin, Zedong Cai, Yijian Huang, Weiyang Liu, Siyuan Wang, Yandong Wen

arXiv:2610.27782v1cs.CVcs.RO

TL;DR

Embodied video judges need to assess physical validity and instruction following, but their agreement with human judgments has not been systematically evaluated. RoboJudge-Bench evaluates that agreement on embodied videos, and RoboJudge-9B, trained with human-verified supervision, achieves the highest overall agreement among the evaluated model judges.

  • Problem

    Agreement between MLLM judges and human judgments on embodied videos has not been systematically evaluated, despite the need to assess both physical validity and instruction consistency.

    Source: §1 · p. 1

  • Method

    RoboJudge-Bench evaluates MLLM judges on 800 human-annotated videos across Physical Adherence and Instruction Alignment, and RoboJudge-9B is fine-tuned on 12K videos with human-verified scores and model-assisted rationales using supervised learning followed by reinforcement learning.

    Source: §1 · p. 2

  • Results

    Pearson r of 0.719 gives RoboJudge-9B the highest overall agreement among evaluated model judges on RoboJudge-Bench, and training also yields gains on VideoPhy2 and Physion-Eval.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Human-calibrated PA and IA supervision is valuable for embodied video evaluation, and RoboJudge-9B can also be used to benchmark video generators.

    Source: §1 · p. 2, §6 · p. 10

Abstract

from arXiv · show

Video generation models are promising foundations for embodied world models, but only if their generated rollouts are physically valid and faithful to task instructions. Multimodal large language models (MLLMs) are increasingly used to judge these properties, yet their agreement with human judgments on embodied videos has not been systematically evaluated. To address this gap, we introduce RoboJudge-Bench, a benchmark for evaluating MLLMs as judges of embodied video generation. The benchmark measures two fundamental dimensions, Physical Adherence and Instruction Alignment, across 800 human-annotated videos from 8 source corpora and 10 video generators, spanning grippers, human hands, and dexterous hands. Experiments reveal uneven alignment between existing judges and human ratings: general-purpose MLLMs show lower agreement on physical adherence, while specialized video judges designed for general-domain physical reasoning do not consistently transfer to embodied scenarios. Motivated by these findings, we construct RoboJudge-Train, comprising 12K videos annotated with 25K human-verified scores and model-assisted rationales for Physical Adherence and Instruction Alignment. We fine-tune a 9B MLLM on this dataset with supervised learning followed by reinforcement learning, yielding RoboJudge-9B. Among the model judges evaluated on RoboJudge-Bench, RoboJudge-9B achieves the highest overall agreement, reaching a Pearson correlation of 0.719 with a statistically significant 0.089 gain over Gemini-3.7-Flash, the best-performing proprietary baseline in our evaluation. Training on RoboJudge-Train also yields gains on external benchmarks, including VideoPhy2 and Physion-Eval. We further use RoboJudge-9B as a downstream evaluator to benchmark video generation models according to Physical Adherence and Instruction Alignment.

1 INTRODUCTION

Embodied video rollouts must be physically valid and instruction-consistent, but existing evaluation approaches do not reliably distinguish physical failures from task-level failures. RoboJudge-Bench assesses judges against human ratings, and RoboJudge-9B achieves the strongest overall agreement among the evaluated model judges.

1 INTRODUCTION

  • Embodied rollouts must predict physical evolution under actions or instructions, so evaluation needs to assess physical validity as well as visual plausibility.Open-ended task execution also makes reference-based similarity intrinsically unreliable, and similarity metrics cannot disentangle physical from task-level failures.Source: §1 · pp. 1–2
  • RoboJudge-Bench evaluates MLLM judges on 800 human-annotated rollouts from 8 corpora and 10 generators across Physical Adherence and Instruction Alignment.The benchmark spans grippers, dexterous hands, and human hands in simulated and real environments.Source: §1 · pp. 2–3
  • Existing judges show gaps with human judgments: GPT-5.5 reaches r = 0.49 on Physical Adherence, and specialized video judges reach at most r = 0.42 on Physical Adherence and 0.59 on Instruction Alignment.Failure analysis identifies overlooked interaction failures and physical hallucinations.Source: §1 · pp. 2–3
  • Pearson r of 0.719 gives RoboJudge-9B the highest overall agreement among evaluated model judges, compared with 0.401 for its base model.The model is trained on RoboJudge-Train and also shows gains on VideoPhy2 and Physion-Eval.Source: §1 · pp. 2–3
  • RoboJudge-Train contains 12K videos with 25K human-verified scores and model-assisted rationales, and supports fine-tuning a 9B MLLM with supervised learning followed by reinforcement learning.RoboJudge-9B is also used to benchmark video generators using the same PA and IA criteria.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work evaluates video quality, physical reasoning, generated-world-model behavior, and multimodal judge reliability, but these lines do not fully address judge reliability across diverse embodied videos. RoboJudge-Bench focuses on agreement with human assessments along two embodied-specific dimensions.

Judge Model Benchmarks

  • Embodied MLLM benchmarks also diagnose atomic perception, reasoning, and planning capabilities.This provides a related benchmark context distinct from evaluating judges of generated embodied videos.Source: §2 · p. 3

Benchmarks for Video Generative Models

  • Video-generation benchmarks increasingly assess physical and embodied behavior, but primarily evaluate generated videos or generative models rather than the reliability of their judges.The literature includes general visual-quality and physical-commonsense benchmarks as well as embodied and world-model benchmarks.Source: §2 · p. 3

Video Quality Evaluation Metrics

  • Pixel and perceptual metrics, dataset-level FVD, learned representations, and optical-flow measures capture similarity or motion signals but do not explicitly judge physical validity or task completion.Recent MLLM evaluators add language-conditioned video judgments, though they are generally designed for broader video-quality or physical-reasoning settings.Source: §2 · pp. 3–4

Judge Model Benchmarks

  • Existing video-judge benchmarks mostly cover general-domain videos, while Physion-Eval and RoboGaze remain limited in embodiment, source, and generator diversity.The cited benchmarks cover video quality, physical reasoning, preference, feedback, and artifact judgments.Source: §2 · p. 4
  • RoboJudge-Bench evaluates judge reliability, rather than generator quality alone, using 800 videos from 8 corpora and 10 generators with paired human references for Physical Adherence and Instruction Alignment.RoboJudge-Train adds human-verified dimensional scores and model-assisted rationales for training RoboJudge-9B.Source: §2 · p. 4

3 ROBOJUDGE-BENCH

RoboJudge-Bench evaluates MLLMs as judges of generated embodied-manipulation videos by comparing their scores with human annotations. Its two complementary dimensions separate physical plausibility from instruction completion, and its data curation combines diverse generated videos with adjudicated human references.

3.1 OVERVIEW

  • RoboJudge-Bench pairs an instruction and generated video with human ratings on two axes, and measures judge performance by agreement with those ratings.Its 800 videos span diverse embodiments and real-world and simulated settings, with 10 generators producing a broad range of quality and human scores.Source: §3.1 · pp. 4–5

3.2 EVALUATION DIMENSIONS

  • Physical Adherence assesses agent, scene, and interaction consistency independently of the instruction, whereas Instruction Alignment checks agent match, object correctness, and goal completion against the instruction and initial frame.The axes distinguish physical plausibility from task completion, which can diverge across current generators.Source: §3.2 · p. 5

3.3 DATA CURATION

  • The benchmark draws source clips from seven real-world corpora and RoboTwin, refines underspecified instructions with human verification, and selects 800 generated videos from 10 of 14 generators.Test items receive independent labels from two annotators followed by expert adjudication; training uses human-verified model-assisted labels.Source: §3.3 · p. 5

4 ROBOJUDGE-9B

RoboJudge-Train supplies human-reviewed score and rationale supervision from a leakage-disjoint video pool. RoboJudge-9B is initialized from Qwen3.5-9B and trained through supervised fine-tuning followed by reinforcement learning that directly rewards exact-score agreement.

4.1 TRAINING DATA CURATION

  • RoboJudge-Train contains 12,351 videos: 11,520 generated videos from 14 generators and 831 verified successful real demonstrations, yielding 24,702 axis-level PA and IA samples.Its source clips are disjoint from those used to construct RoboJudge-Bench.Source: §4.1 · pp. 5–6
  • Three closed-source MLLMs draft PA and IA scores that annotators review and revise, while annotator evidence notes are rewritten into structured reasoning traces conditioned on human-verified scores.These annotations and reasoning traces supervise RoboJudge-9B.Source: §4.1 · p. 6

4.2 MODEL TRAINING

  • RoboJudge-9B is initialized from Qwen3.5-9B, first receives full-parameter supervised fine-tuning, then undergoes GRPO reinforcement learning optimized for exact-score agreement.The RL reward is 1 for an exact match to the reference score and 0 otherwise; training runs for 200 RL steps.Source: §4.2 · p. 6

5 EXPERIMENTS AND RESULTS

The evaluation finds that existing judges align unevenly with human ratings, especially on Physical Adherence, while RoboJudge-9B performs best overall and transfers to external benchmarks. Ablations and downstream rankings show the value of evaluating both physical adherence and instruction alignment.

5.1 EVALUATED MODELS AND BASELINES

  • The evaluation compares 29 model-judge baselines and 8 non-MLLM metrics against human annotations using correlation, exact-match accuracy, and mean squared error.Overall correlation and MSE pool PA and IA predictions, while Overall Acc averages the two axis-specific accuracies.Source: §5.1 · p. 6, §5.2 · p. 6, Table 2 · p. 8

5.3 RESULTS AND ANALYSIS

  • RoboJudge-9B achieves the strongest model-judge agreement, with overall r = 0.719, PA r = 0.645, and IA r = 0.775.Paired cluster-bootstrap analysis confirms robust gains in IA and overall agreement.Source: §5.3 · pp. 6–7, Table 2 · p. 8
  • Visual similarity metrics remain weaker than RoboJudge-9B for human-rated PA, with MUSIQ and pixel consistency especially weak.MUSIQ and pixel consistency reach r = 0.078 and 0.155; DINO and V-JEPA Paired Similarity reach r = 0.392 and 0.370.Source: §5.3 · p. 7, Figure 5 · p. 7
  • Before task-specific training, judges correlate better with human IA than PA ratings, while embodied and specialized judges transfer poorly to embodied evaluation.The best pre-training IA correlation is 0.733 versus 0.611 for PA; embodied models reach at most 0.07 on PA, and VideoPhy2, VideoScore2, and VideoCon at most 0.20.Source: §5.3 · p. 8
  • RoboJudge-9B is more balanced on PA and IA than several baselines, suggesting better calibration to human scores.Qualitative failures include missed entity or interaction inconsistencies and incorrect inferences of task completion from final states.Source: Figure 4 · p. 7, §5.3 · p. 8

5.4 ABLATION STUDY

  • Reinforcement learning improves on SFT, raising overall correlation from 0.700 to 0.719 and increasing exact-match accuracy.SFT itself substantially improves the base Qwen3.5-9B on both PA and IA; the final model uses SFT followed by RL.Source: §5.4 · p. 9, Table 3 · p. 9

5.5 GENERALIZATION STUDY

  • Training on RoboJudge-Train improves performance across evaluated OOD metrics on VideoPhy2 and Physion-Eval.SFT+RL reaches Pearson correlations of 0.346 on VideoPhy2 PC and 0.572 on SA; SFT reaches Youden’s J of 0.105 on Physion-Eval versus 0.056 for the base model.Source: §5.5 · p. 9, Table 4 · p. 9

5.6 DOWNSTREAM APPLICATION: VIDEO GENERATION LEADERBOARD

  • RoboJudge-9B ranks generators across PA and IA, exposing cases where physically plausible motion still fails the instructed task.HappyHorse and Wan2.6 Flash score highly on both; LingBot Video MoE Base, LongCat, and SVD-XT have relatively high PA but much lower IA.Source: Figure 6 · p. 7, §5.6 · p. 9

6 CONCLUSION

The paper concludes that RoboJudge-9B achieves strong human agreement on embodied video judgments and transfers to external physical-reasoning benchmarks, with potential downstream uses.

6 CONCLUSION

  • RoboJudge-9B achieves the strongest overall agreement among evaluated model judges on RoboJudge-Bench and transfers to external physical-reasoning benchmarks.The authors highlight human-calibrated PA and IA supervision and suggest potential for online rollout evaluation and agentic embodied video generation.Source: §6 · p. 10

A ANNOTATION PROTOCOL AND INTERFACE

  • Annotators score each generated video on physical adherence and instruction alignment, complete diagnostic checklists, and provide evidence for their decisions.Source: Figure 7 · p. 17

A.1 ANNOTATOR TRAINING AND CALIBRATION

Annotators were calibrated before production, with agreement improving across practice rounds and production labels receiving independent review and expert adjudication. The appendix also details the model-assisted training annotations and protocols used to evaluate general, embodied, and specialized judges.

A.1 ANNOTATOR TRAINING AND CALIBRATION

  • Before production, annotators completed four 100-video alignment rounds using the main scoring checklist, evidence comments, and feedback on disagreements.PA judgments excluded task instructions; IA judgments used the instruction and initial frame.Source: §A.1 · p. 18
  • Exact agreement rose from 32.8% to 52.9%, agreement within one point from 76.2% to 92.1%, and mean absolute difference fell from 0.98 to 0.58.The metrics aggregate pairwise comparisons of annotators’ primary PA and IA scores across four practice rounds.Source: §A.1 · p. 18, Table 5 · p. 18

A.2 MODEL-ASSISTED TRAINING ANNOTATION

  • Three MLLMs proposed training labels, which human annotators corrected and supported with evidence notes before GPT-5.5 rewrote the notes as structured reasoning traces.The resulting supervision reflected human revisions rather than raw model outputs.Source: §A.2 · p. 18

A.3 ANNOTATION INTERFACE

  • The annotation interface collects PA and IA scores, six checklist labels, and free-text evidence while hiding model identity to reduce generator-specific bias.It presents the generated video and reference context.Source: §A.3 · p. 18

B EVALUATION PROTOCOL

  • The evaluation protocol separates unified inputs for general and embodied VLMs, original interfaces for specialized judges, and non-MLLM metric implementations.General and embodied VLMs receive the same prompt and JSON fields, with up to 32 uniformly sampled frames at 4 fps.Source: B EVALUATION PROTOCOL · p. 18, §B.1 · pp. 18–19

B.1.1 PHYSICAL ADHERENCE

  • Physical Adherence is judged from the video alone using agent integrity, scene and object consistency, and interaction realism on a 1–5 scale.The criteria include structural stability, temporal consistency, and physically plausible contact and motion.Source: §B.1.1 · p. 19

B.1.2 INSTRUCTION ALIGNMENT

  • Instruction Alignment judges task execution using the instruction and initial scene, assessing agent match, target-object correctness, and goal completion on a 1–5 scale.The first frame is the scene on which the video was conditioned.Source: §B.1.2 · pp. 19–20

B.2 SPECIALIZED VLM JUDGES

  • Specialized VLM judges use their original scoring interfaces when available, with native scores mapped to PA and IA as specified for each model.WBench supplies only a visual-quality score mapped to PA, so its IA and overall results are not reported.Source: §B.2 · p. 20

B.3 NON-VLM METRICS

The section describes non-MLLM metrics that measure complementary aspects of video quality and consistency, then explains how their scores are compared with human PA and IA ratings. These metrics provide item-level signals, but some measure motion or visual stability rather than physical plausibility or task completion.

Metric evaluation protocol

  • Each metric yields one scalar per video, aligned by canonical item id with human PA and IA ratings on the RoboJudge-Bench test set.Raw distance- or error-based metrics have their direction reversed for correlations so that larger values correspond to higher quality or stronger consistency.Source: §B.3 · pp. 21, 24

Pixel Frame Consistency

  • Pixel frame consistency uses adjacent-frame mean absolute pixel differences; lower values indicate stronger local consistency, but legitimate camera or object motion may be penalized.The metric direction is reversed for correlation analysis.Source: §B.3 · pp. 21–22

CLIP Frame Consistency

  • CLIP frame consistency compares each frame with its predecessor and the first frame, using complete-frame features rather than a segmented background.Higher scores indicate stronger global visual consistency over time.Source: §B.3 · p. 22

DINO Frame Consistency

  • DINO frame consistency compares each frame with both its predecessor and the initial frame, capturing short-term continuity and longer-term identity or appearance drift.The metric follows VBench’s subject-consistency formulation.Source: §B.3 · p. 22

Optical-Flow Motion Magnitude

  • Optical-flow motion magnitude measures apparent motion, which may be physically plausible or erroneous and has no intrinsic higher-is-better interpretation.The metric therefore retains its original direction when correlated with human ratings.Source: §B.3 · pp. 22–23

V-JEPA Similarity

  • V-JEPA paired similarity compares each generated rollout with its corresponding real trajectory from the same source dataset, task, and episode; manifold similarity instead measures proximity to real-video features without a paired reference.The reported main comparison uses paired cosine similarity because cosine similarity and feature MSE provide closely related signals; manifold distance is reversed for correlation analysis.Source: §B.3 · pp. 23–24

C DATA STATISTICS

The appendix describes prompt construction and variation for embodied-video generation, along with the training and test set sizes. Rewriting templates constrain prompts around the instructed object and task completion, while prefix variants add agent and scene context without rewriting the instruction.

D VIDEO PROMPT ENGINEERING

  • Generation prompts combine an initial frame and instruction with a rewrite template, and the rewriting model receives the initial frame as image input.The prompt tells the model to prioritize the image if it conflicts with the instruction and to output only the rewritten prompt.Source: D VIDEO PROMPT ENGINEERING · p. 24

D.1.1 ROBOT REWRITING TEMPLATES

  • Five robot rewriting templates keep prompts within 100 words, preserve the instructed object name, and describe actions through task completion.The templates vary narrative, reach-grasp-execute-release sequencing, imperative phrasing, action-first phrasing, and approach-contact-completion decomposition.Source: §D.1 · p. 24, §D.1.1 · p. 25

D.1.2 HUMAN REWRITING TEMPLATES

  • Five human-hand rewriting templates likewise preserve the object name and completion state within 100 words, varying narrative, imperative, action-first, and decomposed task descriptions.The decompositions include reach, grasp, execute, and release/complete, or approach, contact, and completion.Source: §D.1 · p. 24, §D.1.2 · p. 26

C DATA STATISTICS

  • The training set contains 11,520 generated videos from 14 generators and 831 verified successful real demonstrations, while the test set contains 800 videos from 10 generators.The real demonstrations are assigned PA=5 and IA=5.Source: Table 6 · p. 25

D.2 PROMPT PREFIXING

  • Prefix-based prompts prepend fixed context selected by manipulation type to the original instruction, without passing it through the rewriting model.The variants preserve task semantics while varying the acting-entity and visual-context description.Source: §D.2 · p. 26, §D.2.2 · p. 27

D.2.1 ROBOT PREFIXES

  • Robot prefixes describe a fixed or controlled workspace and a stable, physically consistent robotic arm before the task instruction.Three variants provide alternative contextual descriptions.Source: §D.2.1 · p. 26

D.2.2 HUMAN PREFIXES

  • Human prefixes frame the action as a real hand in an egocentric, first-person, or head-mounted view before the original instruction.Figure 8 describes RoboJudge as balanced around human scores while baselines tend to over- or under-score.Source: §D.2.2 · pp. 26–27, Figure 8 · p. 27

E ADDITIONAL ANALYSIS

The additional analyses examine RoboJudge-9B’s directional scoring bias, the robustness of its agreement gains, video-generation scores, and judge agreement patterns. Together, they describe how judge scores compare with human ratings and how RoboJudge-9B’s results are assessed.

E ADDITIONAL ANALYSIS

  • RoboJudge-9B’s PA and IA scores are more balanced than several baselines, whose PA scores show substantial over- or under-scoring.Qwen3.5-9B and Cosmos-R2-2B over-score 73.3% and 85.8% of PA examples, while Gemini-3.7 under-scores 65.5%.Source: E ADDITIONAL ANALYSIS · p. 27

E.1 STATISTICAL CONFIDENCE

  • Bootstrap intervals confirm robust gains in IA and overall agreement, while RoboJudge-9B has the highest observed PA correlation and improves on the strongest baseline in all three point estimates.The paired cluster bootstrap jointly resamples generated videos derived from source clips, stratified by source corpus.Source: §E.1 · p. 27

E.2 VIDEO GENERATION LEADERBOARD

  • Table 8 reports RoboJudge-9B evaluation scores for video-generation models ranked in Figure 6, with PA, IA, and their average reported as Overall.The supplied passage does not provide the table’s score values.Source: §E.2 · p. 27, Table 8 · p. 28

E.3 JUDGE AGREEMENT LANDSCAPE

  • Figure 9 plots each judge’s mean Pearson correlation with human PA and IA ratings against its mean squared error.It complements the agreement results in Table 2.Source: §E.3 · p. 28, Figure 9 · p. 28

F ADDITIONAL EXAMPLES OF ROBOJUDGE-BENCH

The appendix presents additional benchmark examples across embodiments and generation models, with PA/IA scores and rationales. It also reports the training computation and settings for RoboJudge-9B.

F ADDITIONAL EXAMPLES OF ROBOJUDGE-BENCH

  • Additional examples span diverse embodiments, source datasets, and video generators, with representative frames, PA/IA scores, and fine-grained rationales.The appendix also includes SFT loss and GRPO reward training curves.Source: F ADDITIONAL EXAMPLES OF ROBOJUDGE-BENCH · p. 28, Figure 10 · pp. 29–30, Figure 11 · p. 30
  • The dexterous-hand corn handover example receives PA 4 and IA 5, with slight ambiguity in transfer contact noted for physical adherence.Its rationale also describes coherent hands and stable objects, and matches the instructed hand roles, target corn, and transfer goal.Source: F ADDITIONAL EXAMPLES OF ROBOJUDGE-BENCH · p. 29
  • The RoboTwin alarm-clock example receives PA 4 and IA 4, with limited button feedback and an unclear final release noted in the rationales.The rationales describe a coherent gripper and correct target button, while noting the release is not clearly maintained.Source: F ADDITIONAL EXAMPLES OF ROBOJUDGE-BENCH · p. 29

G.1 COMPUTATION

  • Training used four NVIDIA H20 GPUs; one SFT epoch took about two hours, and GRPO took about 12 hours to reach 200 steps.SFT used full-parameter fine-tuning with batch size 64 and learning rate 1 × 10−5; GRPO used learning rate 1 × 10−6 and 512 sampled completions per update.Source: §G.1 · p. 30

G.3 FRAME-RATE ABLATION

Table 9 describes a frame-rate ablation on ROBOJUDGE-BENCH using exact-match accuracy and identifies 4 FPS as the default evaluation setting.

G.3 FRAME-RATE ABLATION

  • Table 9 reports a frame-rate ablation on ROBOJUDGE-BENCH, using exact-match accuracy (Acc) and reusing main-table results for the default 4 FPS setting.The 4 FPS entries are the corresponding canonical evaluations.Source: §G.3 · p. 30, Table 9 · p. 31

Related papers

Loading 2610.27782v1…