PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboWorM: Evaluating Movement Realism, Consistency and Acceleration Affects in Embodied Video World Models

Doanh Le Thien, Viet-Thanh Nguyen, Pham Tri Quang, Nhi Pham, Christopher Wewer, Minh-Loi Nguyen, Hoang H. Tran, Tho Le, Tung M. Luu, Dung D. Le, Vu N. Duong, Jonas Fischer, Ngan Le, An Thai Le, Daniel Sonntag, Gim Hee Lee, James Zou, Mathias Niepert, Jan Peters, Duy M. H. Nguyen, Ngo Anh Vien

arXiv:2610.33807v1cs.RO

TL;DR

Existing VWM benchmarks often assess visual quality or use VLM judges that lack strict 3D spatial and kinematic reasoning. RoboWorM uses deployment priors to benchmark structural fidelity and efficiency, introducing KIVA and RIGIS and auditing acceleration across models and tasks. Its metrics outperform VLM judges in human alignment, while the audit reveals that acceleration can compromise physical constraints despite superficial visual quality.

  • Problem

    Many VWM benchmarks treat models as open-world video generators or rely on VLMs that lack strict 3D spatial and kinematic reasoning, leaving deployment-specific structural fidelity unmeasured.

    Source: §1 · p. 2

  • Method

    RoboWorM curates a diverse robot-video benchmark and uses deployment priors to define deterministic, VLM-free metrics for intra-view kinematics and multi-view geometry.

    Source: §1 · pp. 2–3

  • Results

    KIVA improves human-aligned hallucination detection by 25.8% (58.4% to 73.5%) over frontier VLMs, while acceleration methods can compromise physical constraints despite superficial visual quality.

    Source: §1 · p. 3, §4.2 · p. 8

  • Takeaways & Limitations

    RoboWorM provides a complementary deployment-aware lens for identifying kinematic, multi-view, and acceleration-related failures that generic metrics may miss.

    Source: §5 · p. 10

    KIVA and RIGIS do not verify contact-level validity, are limited to rigid-body motion, and do not identify which internal computations cause physical-fidelity loss under acceleration.

    Source: §5 · p. 10

Abstract

from arXiv · show

Video world models (VWMs) are rapidly emerging as scalable neural simulators for robotic manipulation. However, most benchmarks only evaluate them as open-world video generators, using Vision-Language Models (VLMs) that lack strict 3D spatial and kinematic reasoning. To bridge this gap, we introduce RoboWorM, a comprehensive, deployment-aware benchmark that leverages known hardware priors such as robot kinematic trees and fixed camera configurations, to systematically examine structural and spatiotemporal fidelity. RoboWorM curates 1,605 episodes across single-arm, bimanual, and humanoid morphologies, spanning both Markovian and long-horizon non-Markovian tasks under single- and multi-view setups. To explicitly measure physical consistency, we propose two deterministic, VLM-free metrics: KIVA for intra-view kinematic stability, and RIGIS for multi-view geometric consistency. By exploiting intrinsic robotic invariants, KIVA improves hallucination detection alignment with human annotations by 25.8% relative to the strongest frontier VLMs, while RIGIS achieves a 130% higher Spearman correlation with human preferences. Furthermore, since real-world deployment requires low inference latency, we utilize RoboWorM to conduct the first large-scale audit of the quality-efficiency trade-off. Evaluating 9 state-of-the-art acceleration techniques across 3 leading world models (over 29,000 generated rollouts), we demonstrate how these acceleration methods can retain superficial visual quality while silently compromising deployment-critical physical constraints. Using our metrics, we achieve 2.3$\times$ maximum inference speed-up with minimal quality loss.

1 INTRODUCTION

Existing VWM benchmarks emphasize open-world visual quality or VLM judgments, which do not strictly assess robotic kinematics and 3D geometry. RoboWorM uses deployment priors to evaluate structural fidelity and efficiency across a diverse benchmark, and its metrics reveal physical failures and acceleration trade-offs.

1 INTRODUCTION

  • Existing evaluations emphasize perceptual quality, text alignment, or VLM judgments, which struggle with strict 3D reasoning and can be optimistic about physical plausibility.The introduction frames a robot video as a physical system in motion, with mechanical and cross-camera constraints.Source: §1 · p. 2
  • RoboWorM curates 1,605 episodes across single-arm, bimanual, and humanoid morphologies, including short- and long-horizon tasks and up to four synchronized camera views.The benchmark is designed to evaluate structural fidelity and computational efficiency using known deployment priors.Source: §1 · p. 2
  • KIVA and RIGIS are deterministic, physics-grounded metrics that use embodiment and geometric priors to assess kinematic stability and multi-view consistency.KIVA audits intra-view physical hallucinations; RIGIS checks whether synchronized views form a unified 3D scene with stable camera parameters.Source: §1 · pp. 2–3
  • Across 9 acceleration strategies and 3 world models, more than 29,000 rollouts expose how computational shortcuts can compromise physical constraints despite preserving superficial visual appearance.The audited methods yield up to a 2.3× maximum speed-up.Source: §1 · p. 3
  • The evaluation finds that visually convincing robotic videos can contain kinematic and multi-view inconsistencies missed by standard metrics and VLM judges.The authors report that such failures compound over longer rollouts.Source: §1 · p. 3

2 RELATED WORK

Prior VWM benchmarks often rely on external VLMs or off-the-shelf methods without using deployment-specific priors, and many focus on narrow, short-horizon settings. RoboWorM is presented as a deterministic, prior-grounded benchmark spanning complex embodiments, multi-view and long-horizon tasks, and acceleration audits.

2 RELATED WORK

  • RoboWorM uses dataset-native hardware priors for deterministic evaluation across multi-view setups, complex morphologies, and long-horizon tasks, while also auditing cross-view geometry and inference acceleration.Table 1 describes prior benchmarks as relying on external VLMs or off-the-shelf models that do not exploit camera calibration, proprioception, or actions.Source: Table 1 · p. 3
  • Prior robotics suites largely use subjective VLM judges or unadapted off-the-shelf algorithms, while simulation-based inverse-kinematics pipelines can accumulate real-to-simulation conversion errors.RoboWorM instead uses explicit priors such as kinematic URDFs, task instructions, and source videos for deterministic intra-view and inter-view measurements.Source: §2 · pp. 3–4
  • The benchmark composition includes Markovian and non-Markovian tasks, varied skills, and three robot morphologies, alongside an audit of 9 acceleration methods over 29,000+ rollouts from three world models.The supplied Figure 2 caption describes a balanced mix of task dynamics and an even distribution across morphologies.Source: Figure 2 · p. 4
  • Earlier dataset evaluations often skew toward single-arm, single-view tabletop tasks with short Markovian dynamics, leaving complex embodiments and deterministic spatial reasoning under-explored.RoboWorM addresses this gap with multi-view, long-horizon tasks across complex embodiments.Source: §2 · p. 4
  • Acceleration methods are commonly assessed with perceptual metrics, while their effects on generated robotic physical fidelity remain untested in prior work described here.RoboWorM audits how these computational shortcuts affect deployment-critical kinematic linkage and geometric stability.Source: §2 · p. 4

3 RO B OWO RM

RoboWorM curates varied robot rollouts, evaluates multiple world-model families and acceleration methods, and defines KIVA and RIGIS to measure intra-view kinematics and multi-view spatial consistency. Its metrics use hardware and camera priors to assess physical structure in generated videos.

3.1 DATASET CURATION AND DEPLOYMENT TASK FORMULATION

  • The dataset combines five public corpora into six configurations spanning five robot platforms and three morphology classes, with both Markovian and non-Markovian rollouts in each configuration.Free-form tasks receive automated LLM regime labels; multi-step instructions are manually labeled.Source: §3.1 · p. 5

3.2 VIDEO WORLD MODEL EVALUATION SUITE AND ROLLOUT GENERATION

  • The evaluation covers DreamGen, DreamDojo, and Ctrl-World across 12 matched checkpoints, nine fine-tuned on target domains and three using public weights.The three model families use scalable general-purpose DiT, action-conditioned DiT, and spatio-temporal U-Net architectures, respectively.Source: §3.2 · p. 5

3.3 SYSTEMATIC INFERENCE ACCELERATION AUDIT

  • The acceleration audit compares nine algorithms with a dense baseline across cache strategies, sparse attention, and token pruning.Token pruning requires DiT architectures and is incompatible with U-Nets; the audit generated approximately 29,000 samples.Source: §3.3 · p. 5

3.4.1 KIVA: KINEMATIC INVARIANT VIOLATION ASSESSMENT

  • KIVA trains a video-conditioned diagnostic model on forward-kinematics targets, averages five DDIM samples, and measures rigidity and jerk on reconstructed metric trajectories.Its dimensionless violation ratios are normalized using real-video reference percentiles; raw errors have no absolute meaning, and the decision boundary is calibrated per embodiment against human labels.Source: §3.4.1 · p. 6

3.4.2 RIGIS: ROBOTIC INTER-VIEW GEOMETRY & INTRINSIC-EXTRINSIC STABILITY

  • RIGIS uses known static camera configurations and workspace limits to assess whether reprojected views form a unified 3D scene without drift from the robotic setup.It evaluates geometric consistency across views and time and camera-pose stability; the final score is the unweighted mean of spatial fidelity and rig stability, with higher scores indicating more coherent multi-view 3D representations.Source: §3.4.2 · pp. 6–7

4 EXPERIMENTS

RoboWorM’s metrics align more closely with human judgments than VLM judges, while revealing model-specific long-horizon behavior and distinct acceleration trade-offs. The experiments evaluate these findings across human alignment, model configurations, and inference speed.

4.1 EVALUATION PROTOCOL

  • The evaluation uses human-graded subsets of 291 single-view videos and 300 multi-view videos, and measures speedup as dense-baseline generation time divided by accelerated generation time per segment.The segment timing excludes model loading and VAE decoding overhead.Source: §4.1 · p. 7

4.2 HUMAN ALIGNMENT AND CROSS-METRIC ANALYSIS

  • KIVA improves human-aligned hallucination detection by 25.8% (58.4% to 73.5%) over frontier VLMs, and RIGIS achieves a 130% higher Spearman correlation with human preferences (0.326 to 0.749).Even the best-performing judge, Gemma 4 31B + CoT, remains below KIVA.Source: §4.2 · p. 8
  • VLM judges show score standard deviations from 0.076 for embodied physics to 0.591 for multi-view consistency, while the algorithmic metrics achieve better latency.The passage also reports massive computational overheads for chain-of-thought reasoning.Source: §4.2 · p. 8
  • Motion Smoothness and Motion Amplitude Score capture surface transitions and are more sensitive to hallucinations, whereas KIVA evaluates physical plausibility through embodied kinematic constraints.Conventional feature-matching and SLAM pipelines fail under large viewpoints, while RIGIS evaluates holistic multi-view consistency.Source: §4.2 · p. 8

4.3 LONG-HORIZON DRIFT IN VIDEO WORLD MODELS

  • Longer rollouts show stronger degradation, but non-Markovian regime and rollout depth are confounded in the deepest single-arm DreamGen and DreamDojo configurations.Their supports are strictly disjoint, so these results do not isolate rollout depth as the root cause.Source: §4.3 · p. 8
  • Ctrl-World consistently outperforms DreamGen in RIGIS across multi-view comparisons, while Ctrl-World uses explicit spatiotemporal processing and DreamGen uses self-attention on tiled frames.The passage attributes the performance gap to this architectural difference.Source: §4.3 · p. 9

4.4 ACCELERATION TRADE-OFFS

  • Caching achieves the largest speedups with near-zero degradation, while token pruning degrades most; Figure 6 reports caching’s 1.7× mean speedup and minimal quality loss (−0.004 RIGIS).Bars show speedup, the middle axis shows physical audit, and dots indicate NR.Source: Figure 6 · p. 9, Figure 5 · p. 9

5 CONCLUSION AND DISCUSSION

The authors argue that deployment-aware auditing complements visual assessment by tracking physical consistency and acceleration trade-offs. They identify limits in contact-level validity, motion types, and diagnosing internal causes of fidelity loss.

5 CONCLUSION AND DISCUSSION

  • Tracking kinematic violations, multi-view consistency, and inference-acceleration trade-offs helps reveal failure modes missed by generic metrics.The authors frame this as a complementary lens to standard visual assessment.Source: §5 · p. 10
  • KIVA and RIGIS do not verify contact-level validity, cover only rigid-body motion, and do not identify which computation layers cause physical-fidelity loss.Contact-level verification would require integrating full physics simulators; deformable materials and force-driven dynamics are also outside the current scope.Source: §5 · p. 10

AI USE STATEMENT

The authors state that LLMs were used only to improve manuscript grammar, clarity, and readability, and that the research and conclusions are their own. The included source-corpus description explains how benchmark settings are sourced, labeled, and sampled.

AI USE STATEMENT

  • LLMs were used solely to improve manuscript grammar, clarity, and readability; the authors state that they are responsible for the research and conclusions.The authors say they reviewed and take full responsibility for the manuscript’s content.Source: AI USE STATEMENT · p. 10

A.1 SOURCE CORPORA & CURATION PROTOCOL

  • The twelve evaluation settings draw on five public corpora, with the two bimanual camera tracks sharing episodes and the other listed single- versus multi-view tracks using distinct corpora.RoboCOIN’s two camera rigs count as one corpus, not two datasets.Source: §A.1 · p. 16
  • Free-form instructions are classified into Markovian or non-Markovian regimes using a fixed LLM rubric, while task-level instructions are labeled manually.Episodes without a label are dropped rather than guessed.Source: §A.1 · p. 16
  • Episode caps vary by setting: DROID and GR1 are sampled to 130 episodes per regime, RoboChallenge to 60 per task, and bimanual and humanoid multi-view settings are ingested exhaustively.DROID’s pool is also restricted to the shards downloaded for the study.Source: §A.1 · p. 16

A.2 DATA PREPARATION

Data preparation matches each generator’s training configuration, while benchmark taxonomy groups source instructions into manipulation primitives. The preparation and taxonomy descriptions also identify constraints on comparisons and coverage.

A.2 DATA PREPARATION

  • Input preparation matches each generator’s training configuration to avoid attributing input-distribution shifts to acceleration methods.The generator families differ in spatial, temporal, and view-composition conventions.Source: §A.2 · p. 16
  • Frames are bilinearly resized to checkpoint resolution without preserving source aspect ratios, so absolute pixel-space metrics are not comparable across settings with different source geometry.Comparisons are made within cells, where every method receives pixel-identical inputs.Source: §A.2 · p. 16
  • Multi-view generation uses a composed canvas because none of the three generators natively emits multiple independent synchronized streams.Inactive DreamGen quadrants are filled with black frames and excluded from metrics; Ctrl-World’s bimanual configuration drops one external view.Source: §A.2 · p. 17

A.3 TASK TAXONOMY & INSTRUCTION PROMPTS

  • The taxonomy groups 509 benchmark instructions by mapping each clause’s leading verb to a canonical primitive, with 45 distinct verbs mapped to 14 primitives.The lexicon reports unmatched verbs rather than silently assigning them, and none of the 509 instructions is unmapped.Source: §A.3 · p. 18
  • Pick & place accounts for 60% of instructions, and six primitives have fewer than 10; the taxonomy is treated as evidence of breadth, not a balanced factor.The authors report no per-primitive metrics for the long tail.Source: §A.3 · p. 18
  • Compositional instructions concentrate in the non-Markovian regime, where correct continuation can depend on phase state unavailable in the current frame.Humanoid single-view has the greatest scene diversity, while other corpora are recorded in one fixed tabletop scene.Source: §A.3 · p. 19

A.4 DISTRIBUTION OF THE GENERATED ACCELERATION DATA

The generated acceleration data spans 12 valid model configurations and nine methods, with rollout counts varying by supported model–camera combinations. The benchmark contains 29,343 generated rollouts across dense and accelerated settings.

A.4 DISTRIBUTION OF THE GENERATED ACCELERATION DATA

  • 3,166 dense rollouts cover 12 valid model configurations: 1,605 from DreamGen, 805 from Ctrl-World, and 756 from DreamDojo.The per-model counts reflect supported camera tracks rather than sampling imbalance; Figure A2 depicts the configuration coverage.Source: §A.4 · p. 20, Figure A2 · p. 21
  • Six acceleration methods cover all 12 configurations with 3,166 rollouts each; SiTo and ITM exclude DreamDojo, while FasterCache excludes Ctrl-World.SiTo and ITM each contain 2,410 rollouts, and FasterCache contains 2,361; Figure A2 marks unsupported backbone–method combinations.Source: §A.4 · p. 20, Figure A2 · p. 21
  • 29,343 generated rollouts comprise 3,166 dense, 8,693 caching-based, 12,664 sparse-attention, and 4,820 token-reduction rollouts.These totals describe dataset coverage, not relative performance across acceleration families.Source: §A.4 · p. 20

B METRICS FORMALISM & IMPLEMENTATION

The appendix formalizes RoboWorM’s KIVA and RIGIS metrics and specifies the auxiliary metrics, generator protocols, and baseline training and sampling setups used in evaluation. It also describes finetuning as the reference for quality-retention evaluation and notes that unseeded Ctrl-World comparisons may reflect sampling variance.

B.1 DETAILED FORMULATION OF KIVA

  • KIVA averages diffusion-based 3D keypoint predictions across noise seeds, then measures hardware-link rigidity violations and anomalous jerk against real-video references, with embodiment-specific thresholds calibrated to human labels.The thresholds are 1.70 and 2.10 for single-arm, 1.00 and 1.50 for bimanual, and 0.32 and 0.80 for humanoid; calibration rollouts are excluded from reported agreement.Source: §B.1 · pp. 22–23
  • KIVA flags segments above embodiment-calibrated thresholds and reports the rollout’s flagged-segment percentage, averaged across rollouts for each benchmark cell.The same violation rate and thresholds are also reported for held-out real recordings as a reference level.Source: §B.1 · p. 23

B.2.3 RIGIS SCORE

  • RIGIS combines geometric consistency and camera-pose stability into a rollout score, then normalizes morphology-level benchmark scores by the mean real-video baseline.The raw rollout score is S = 1/2(Sgeo + Spose); higher normalized scores indicate superior multi-view consistency.Source: §B.2 · p. 23, §B.2.3 · p. 25

B.2.1 GEOMETRIC CONSISTENCY

  • RIGIS geometric consistency evaluates whether fused, unprojected scene points reproject consistently across camera views, assigning zero geometry score when mutual frustum overlap falls below 2%.The method discards pixels with relative neighbor-depth discontinuity above 0.1 and scores mutually visible regions.Source: §B.2.1 · p. 24

B.2.2 CAMERA POSE STABILITY

  • Camera-pose stability measures temporal variation in estimated intrinsic and extrinsic parameters, reflecting the fixed-camera assumption of physical robot setups.If fewer than two non-disjoint frames remain, Spose is set to 0.Source: §B.2.2 · p. 25

B.3.1 VISUAL INTEGRITY

  • Visual Integrity reports the 10th-percentile CLIP-IQA+ frame score, while Text Alignment averages CLIPScore against only the task clause and applies only to text-conditioned models.The percentile is intended to expose brief visual degradation that a temporal mean could dilute; image-only models receive no Text Alignment score.Source: §B.3.1 · p. 26, §B.3.2 · p. 26

B.3.3 FRÉCHET VIDEO DISTANCE

  • FVD compares generated and real-video feature distributions only under matched reference and sampling protocols, while PSNR retention uses the corresponding dense rollout as its baseline.SSIM and LPIPS are retained for per-cell analysis; PSNR degradation across settings is reported in dB.Source: §B.3.3 · p. 26, §B.3.4 · p. 27

C.1 BASE WORLD MODEL FINETUNING PROTOCOLS

  • The benchmark fixes base world models and evaluates accelerations within each model, using dense checkpoints as references; the three families use distinct training recipes and are not compared across families.Ctrl-World, DreamGen, and DreamDojo are fully finetuned; their input representations differ, and reported Ctrl-World multi-view layouts are selected by embodiment.Source: C WORLD MODEL TRAINING & ACCELERATION… · p. 28, §C.1 · p. 28
  • Ctrl-World uses ACCELERATE data parallelism with micro-batch size 1, while DreamGen and DreamDojo use FSDP across all ranks.DreamGen also uses degree-2 context parallelism during training, disabled for accelerated inference because the evaluated caching and sparse-attention methods do not support it.Source: §C.1 · p. 28
  • All rollouts and timing measurements run on a single A100-80GB, and sampling settings—including denoising steps, guidance scale, and seeds—are reported in Table C1.Table C1 defines dense baselines for 14 benchmark settings and lists backbone size, adaptation, denoising steps, guidance scale, and released rollout count.Source: §C.1 · p. 28, Table C1 · p. 29
  • Finetuning mitigates the unfine-tuned model’s progressive loss of task-relevant structure, including object position and cross-view consistency, and defines the dense reference for quality-retention evaluation.Figure C1 compares ground truth, unfine-tuned, and finetuned rollouts at matched time points across camera views; reported PSNR values cover the full episode.Source: Figure C1 · p. 29
  • Seed handling differs by model family: Ctrl-World is unseeded, DreamGen uses seed 0 with segment-index offsets for autoregressive multi-view rollouts, and DreamDojo uses per-chunk seeds.These configurations are specified for the 14 dense benchmark settings in Table C1.Source: Table C1 · p. 29
  • The dense baselines are the comparison reference for acceleration methods evaluated in their corresponding benchmark settings.Table C1 describes 14 dense benchmark settings and specifies the baseline configuration for each corresponding acceleration-method cell.Source: Table C1 · p. 29
  • Because Ctrl-World is unseeded, differences between its methods may reflect sampling variance; results are single runs without seed-wise error bars.The authors interpret within-family comparisons accordingly.Source: §C.1 · p. 29

C.2 DIFFUSION ACCELERATION METHOD CONFIGURATIONS

The acceleration audit pairs dense and accelerated runs within matching configurations and measures speedup from matched generation units. Its supplementary results show that physical retention varies by method, model, and embodiment, while timing coverage and cross-cell comparability remain limited.

C.2 DIFFUSION ACCELERATION METHOD CONFIGURATIONS

  • Speedup uses paired dense and accelerated timings from the same configuration, measured at matched generation units rather than episode totals.The protocol separates first-unit warm-up from steady-state speedup; per-episode totals can distort comparisons across clips of different lengths.Source: §C.2 · pp. 30–31
  • Timing ratios from tuning trials are within-cell comparisons, not a globally calibrated cross-cell scale, because evaluated episode counts vary across backbones.The reported aggregate retention Q compares metrics with the dense rollout of the same backbone; values above 1 are valid, and unavailable metrics may be omitted explicitly.Source: §C.2 · p. 31
  • Reported timing coverage is incomplete: 6 of 56 method–cell pairs with quality results lack recorded generation times.A shared timing pass covers 117 of 120 possible pairs but had completed for only DreamDojo humanoid single-view at the time of writing; there, DiCache reaches 1.56×, WorldCache 1.08×, and FasterCache 0.91×.Source: §C.2 · p. 31
  • Across multi-chunk clips, severity increases with chunk index in 130/193 cases (mean ρ = +0.20, sign-test p < 10^-3), but the trend is not universal.Rollout depth is not matched between Markovian and non-Markovian regimes; for single-arm single-view, severity instead decreases with depth (ρ = −0.10).Source: §C.2 · p. 32
  • On humanoid multi-view Ctrl-World, sparse attention reduces RIGIS from 0.879 to 0.680–0.686, while caching retains 0.875–0.878 and token pruning 0.822–0.835.On single-view DreamDojo, accelerated variants improve KIVA relative to dense (13.0%), with FasterCache reaching Q = 1.13.Source: Figure C2 · p. 33
  • On single-arm multi-view, ITM lowers RIGIS to 0.810 on DreamGen and 0.721 on Ctrl-World, versus 0.838–0.915 for remaining methods.Single-view dense KIVA is already 59.4% versus 10.1% for real video, so the accelerated 55–80% range is not acceleration-induced degradation; missing entries are unmeasured.Source: Figure C3 · p. 34

D HUMAN ASSESSMENT & VLM EVALUATION PROTOCOLS

The appendix documents human and VLM evaluation protocols for embodied physics and multi-view consistency. It describes prepared evaluation sets, distinct rubrics and annotation workflows, human-score adjudication, and matched VLM evaluation procedures.

D HUMAN ASSESSMENT & VLM EVALUATION PROTOCOLS

  • The appendix presents protocols, dataset statistics, implementation details, and supplementary KIVA and RIGIS alignment and correlation results.It also documents the VLM-as-a-judge infrastructure, including configurations, multi-turn querying, and prompt engineering.Source: D HUMAN ASSESSMENT & VLM EVALUATION… · p. 35

D.1.1 PREPARED EVALUATION SETS

  • The two prepared evaluation corpora are balanced across model quality tiers, robot morphologies, and task dynamics, with distributions summarized in Table D1.RoboWorM’s data derive from five public corpora and span six configurations, five robot platforms, and three morphology classes.Source: §D.1.1 · p. 35, Table D1 · p. 35

D.1.2 EVALUATION RUBRIC & SCORING CRITERIA

  • Human scoring separates segment-level embodied physics from video-level multi-view consistency, whose criteria assess robot artifacts versus cross-view agreement and camera stability.The multi-view rubric counts synchronized anomalies as consistent even if physically implausible, and excludes task completion and action quality.Source: §D.1.2 · pp. 35–36

D.1.3 ANNOTATION INTERFACE & WORKFLOW

  • Multi-view raters score synchronized views side-by-side once per video, while embodied-physics raters score single-camera clips in 3-second intervals.Two independent scorers rate each instance; disagreements go to an expert checker for the final label.Source: §D.1.3 · p. 37, Figure D1 · p. 38

D.1.4 HUMAN ASSESSMENT RESULTS

  • Human assessment results report final-score distributions and inter-rater agreement measures for video-level multi-view consistency and segment-level embodied physics.The agreement analysis includes exact agreement rates and ordinal correlation metrics.Source: §D.1.4 · p. 37, Figure D2 · p. 39

D.2 VLM JUDGE EVALUATION SETUP

  • The VLM evaluation investigates whether automated judges capture fine-grained physical and spatial-temporal properties that remain underexplored in prior VLM evaluation.The study evaluates closed- and open-source models on the same split as humans, with three independent turns per model-and-prompt variant.Source: §D.2 · p. 37, §D.2.1 · p. 37

D.2.1 VLM JUDGE CONFIGURATION AND WORKFLOW

  • VLM configurations include Gemini 3.6 Flash, Qwen 3.6-Plus, Gemma 4 31B, and Qwen 3.6-35B, evaluated through an OpenRouter-compatible API.The appendix documents model configurations and prompt variants in Table D2.Source: §D.2.1 · p. 37, Table D2 · p. 37

D.2.2 PROMPT DESIGN

  • VLM prompts mirror the human rubrics and constrain outputs to structured scores and rationales, with separate instructions for embodied physics and multi-view consistency.The multi-view prompt judges synchronization and camera stability, not task success or the physical realism of anomalies that match across views.Source: §D.2.2 · pp. 39–40

D.2.3 VLM RESULTS SUMMARY

VLM judges show variable agreement, score stability, and latency across embodied-physics and multi-view evaluations. Chain-of-thought prompting can worsen these measures, while KIVA and RIGIS process inputs faster than the compared baselines.

D.2.3 VLM RESULTS SUMMARY

  • Table D3 reports judge self-consistency, internal score standard deviation, and processing time per video second across evaluation turns.The table’s agreement rate is the percentage of videos receiving identical scores across turns.Source: §D.2.3 · p. 41, Table D3 · p. 42
  • For embodied physics, Qwen 3.6 Plus reaches 90.3% agreement and 0.076 score standard deviation at 4.396 seconds, while Qwen 3.6 35B with chain-of-thought falls to 38.3% agreement, 0.344 standard deviation, and 15.438 seconds.KIVA processes inputs in 0.289 seconds.Source: §D.2.3 · p. 41, Table D3 · p. 42
  • For multi-view consistency, vanilla Qwen 3.6 Plus reaches 77.0% agreement and 0.191 score standard deviation, while chain-of-thought lowers its agreement to 33.7% and raises standard deviation to 0.591.RIGIS processes inputs in 0.356 seconds, faster than the baseline alternatives.Source: §D.2.3 · pp. 41–42, Table D3 · p. 42

D.3 SUPPLEMENTARY RESULTS FOR KIVA

KIVA’s scores generally order human-labeled quality levels, but separation varies by morphology. Calibrating its decision bands changes overlap with human labels, and comparisons with VLM judges show stronger aggregate alignment on several measures.

D.3 SUPPLEMENTARY RESULTS FOR KIVA

  • KIVA’s median scores fall from Bad to Good across all three morphologies, with Spearman ρ of 0.78 for Bimanual, 0.60 for Humanoid, and 0.39 for Single-Arm.Single-Arm distributions overlap heavily despite the ordered medians.Source: §D.3 · p. 42
  • After calibration on nine labeled videos per morphology, macro overlap rises from 60.8% to 67.5% and micro overlap from 69.5% to 73.5%.Calibration changes only the cuts and leaves Spearman’s ρ unchanged; Bimanual overlap moves from 85.5% to 84.2%.Source: §D.3 · p. 42
  • Across morphologies, KIVA reaches macro ρ = 0.591 and macro overlap 67.5%, versus 0.525 and 61.2% for the best VLM; pooled overlap is 73.5% versus 58.4%.KIVA has the highest Humanoid Spearman (0.603 versus 0.506), while matching the best VLM on overlap (59.1% versus 58.8%).Source: §D.3 · p. 43, Table D4 · p. 43

D.4 SUPPLEMENTARY RESULTS FOR RIGIS

RIGIS aligns with human multi-view consistency ratings across robot morphologies, while the extension protocol specifies how to add world models, embodiments, datasets, and acceleration methods. Extending to new embodiments requires registering and calibrating physical invariants rather than relying on inherited settings.

D.4 SUPPLEMENTARY RESULTS FOR RIGIS

  • RIGIS scores rise stepwise from Bad to Good human ratings across morphologies, with minimal overlap between adjacent tiers.Single-Arm distributions are most compact; Bimanual and Humanoid rating tiers also remain well separated.Source: §D.4 · pp. 43–44, Figure D4 · p. 44
  • RIGIS achieves mean Spearman ρ of 0.749, a 138% relative improvement over the best VLM baseline, and mean Rank Accuracy of 87.4% versus the highest VLM result of 42.0%.RIGIS maintains at least 0.706 Spearman ρ and 84.7% Rank Accuracy across embodiments, while VLM alignment varies substantially.Source: Table D5 · p. 44, §D.4 · p. 45
  • Qualitative multi-view examples compare synchronized trajectories with VLM scores and human consensus to expose evaluation discrepancies.The trajectories represent geometry and rig stability; VLM scores are averaged across methods, turns, and prompting strategies.Source: Figure D5 · p. 45, §D.4 · p. 45

E USING AND EXTENDING ROBOWORM

  • RoboWorM extensions add a world model as a table row, an embodiment or dataset as a column, or an acceleration method as a speed-quality intervention.Model and accelerator additions reuse released episodes and checkpoints; embodiment additions require training.Source: E USING AND EXTENDING ROBOWORM · p. 46, Figure E1 · p. 46

E.1 EVALUATING A NEW WORLD MODEL

  • New world models need only emit frame-aligned RGB rollouts from benchmark conditioning, but covered settings must be evaluated completely across regimes, cameras, and episodes.Fine-tune on the released corpus and report dense quality first; standardize clips before scoring, and do not rank single-view FVD across configurations.Source: §E.1 · pp. 46–47

E.2 ADDING A NEW EMBODIMENT OR DATASET

  • A new embodiment or dataset requires registered physical invariants, including link connectivity, morphology motion bounds, and camera extrinsics.The audits rely on this closed-world metadata rather than inferring it during evaluation.Source: §E.2 · pp. 47–48
  • New embodiment evaluations require morphology-specific KIVA readers, real-segment threshold calibration, and co-located real-video anchors for RIGIS.The protocol also calls for real-video pools for FVD, validated by split-half exchangeability checks.Source: §E.2 · pp. 48–49
  • Fitting thresholds to human labels reduces minority-class emissions from 18.6% to 3.0% and yields a worse detector despite superficially improved scores.Calibration thresholds are derived from the 95th percentile of real data, not human-fitted values.Source: §E.2 · pp. 48–49

E.3 EVALUATING A NEW ACCELERATION METHOD

RoboWorM’s acceleration protocol evaluates interventions against matched dense rollouts, tunes settings per cell, and reports paired quality changes alongside speed. Its standardized claim gate and reporting rules support comparable results, while its additional use cases describe how ranked models may support downstream robotics workflows.

E.3 EVALUATING A NEW ACCELERATION METHOD

  • Speedups require a dense baseline rerun in the same cell, on identical episodes and configuration; timing runs use one idle A100-SXM4-80GB GPU sequentially.Report generation times from authoritative logs, sample both task regimes, and separate warm-up from steady-state ratios.Source: §E.3 · pp. 49–50
  • Methods are tuned at the most aggressive setting whose quality remains indistinguishable from the cell’s dense rollout.Sweep the primary quality-speed knob per cell, compare PSNR, SSIM, and LPIPS against dense rollouts, and publish full curves including degraded settings.Source: §E.3 · p. 49
  • Per-cell tuning matters: accelerator-cell pairs have median absolute disagreement of 0.102 in aggregate retention, with nearly half on opposite sides of Q = 0.95.Optimal settings depend on backbone runtime composition and sequence length, so global defaults can misclassify quality retention.Source: §E.3 · pp. 49–50
  • Acceleration quality claims use paired deltas between accelerated and dense clips from the same episode, view, and matched frame grid.Report speedup with ΔKIVA, ΔRIGIS for multi-view settings, and paired deltas for reference-free columns.Source: §E.3 · p. 50
  • Audit improvements under acceleration do not establish superior rollouts, especially when dense baselines already have high hallucination rates.The cited example contrasts 59.4% KIVA on a dense baseline with a 10.1% real-video baseline.Source: §E.3 · p. 50

E.4 STANDARDISED EVALUATION AND REPORTING PROTOCOL

  • A reported difference passes the claim gate only if its confidence interval excludes zero, Wilcoxon signed-rank p < 0.05, and n ≥ 8.The protocol rejected 226 of 1,836 acceleration pairs, including a cell where frame-rate and length mismatches manufactured spurious continuity gains.Source: §E.4 · pp. 50–51
  • Absolute configuration medians describe each configuration against its own dense baseline, while paired deltas are the exclusive surface for ranking acceleration methods.Cells with n < 8 are reported as untested, not as showing no effect.Source: §E.4 · pp. 50–51

E.5 ADDITIONAL USE CASES OF ROBOWORM

  • RoboWorM positions high-ranking models as potential interactive simulators, rollout synthesizers for sparse demonstrations, and sources of trajectories for policy post-training.The described applications include non-Markovian task execution and few-shot policy adaptation.Source: §E.5 · pp. 51–52

Related papers

Loading 2610.33807v1…