Source-linked AI summary
TriWorldBench: A Tri-View Consistency Perspective on Embodied World Models
Xuanyi Liu, Haofeng Wang, Ruiqi Li, Danni Yu, Rui Wan, Ruixu Zhang, Siyu Tao, Xue Yang, Shaofeng Zhang, Zicheng Zhang, Jiaqi Zhang, Siwei Ma
TL;DR
Action-conditioned world models can inform robot learning and planning, but independent camera-video evaluation cannot establish whether views describe the same manipulation event. TRIWORLDBENCH evaluates synchronized head and wrist videos with camera-aware, task-grounded measures. It makes cross-view consistency measurable while retaining task and visual quality as distinct dimensions.
Problem
Individually plausible head and wrist videos may disagree about whether an object is held, released, or transferred, and single-view quality cannot establish their shared state.
Source: §1 · p. 1Method
Results
The framework makes cross-view consistency measurable without reducing world-model quality to consistency alone.
Source: §5 · p. 3Takeaways & Limitations
The framework supports targeted diagnosis of failures in triple-view predictions for robot learning and planning.
Source: §5 · p. 3
Abstract
from arXiv · showhide
Embodied world models predict the outcomes of robot actions to support learning and planning. For robots equipped with head and wrist cameras, this requires complementary views: the head view captures the overall task, while wrist views reveal local gripper-object interactions. However, evaluating these views independently cannot determine whether they describe the same action and object state. We introduce TRIWORLDBENCH, a benchmark for evaluating embodied world models through synchronized head, left-wrist, and right-wrist videos. It contains 500 episodes across 50 bimanual manipulation tasks and uses 19 metrics to assess tri-view consistency, task alignment, physical and 3D coherence, motion quality, temporal consistency, and visual quality. By combining cross-view checks with measurements tailored to each camera, the benchmark evaluates whether plausible individual videos also form a consistent prediction of the intended task. We summarize overall performance with TWB-Score and retain per-view results to identify where predictions fail. This extends world-model evaluation beyond single-view visual quality. Code, data, and metric definitions are available at https://github.com/TriWorldBench/TriWorldBench.
1. Introduction
Action-conditioned world models can provide training trajectories and planning inputs, but plausible-looking videos may misrepresent manipulation events. TRIWORLDBENCH evaluates synchronized camera views together to test their agreement with each other and the intended task.
1. Introduction
- A plausible-looking grasp that violates object contact can mislead synthetic supervision or guide a planner toward an infeasible action.Source: §1 · p. 1
- Head views provide workspace context while wrist views reveal local contact, and individually plausible streams may still disagree about the shared object state.Different appearances can reflect camera baselines, wrist motion, or occlusion; the question is compatibility with the same event, not visual similarity.Source: §1 · p. 1, Figure 1 · p. 1
- TRIWORLDBENCH treats a synchronized triplet as one prediction and separately evaluates view plausibility, cross-view agreement, and task match.Its 19 metrics use camera-specific evidence and reference action phases; Table 1 groups them across six dimensions.Source: §1 · p. 1, Table 1 · p. 2
2. Related Work
Prior work develops action-conditioned robot videos, multi-view prediction methods, and benchmarks for visual, physical, and task-relevant quality. TRIWORLDBENCH adapts these evaluation concerns to complementary robot cameras and explicitly measures agreement between synchronized views.
Embodied world models
- Action-conditioned robot-video models and post-training rewards make task behavior and cross-camera agreement relevant to evaluating world models used as data and decision inputs.Examples include EnerVerse-AC, Genie Envisioner, DreamDojo, Motus, and PAVXploreRL [2] [6] [12] [16] [25].Source: §2 · p. 2
Multi-view prediction and consistency
Video and world-model benchmarks
- Existing benchmarks assess perceptual and temporal quality, dynamics, instruction following, physical plausibility, or executable behavior; TRIWORLDBENCH adapts such dimensions to camera roles and synchronized-view agreement.The cited benchmarks include VBench [10], WorldScore [5], WorldModelBench [15], EWMBench [27], WorldSimBench [21], WorldArena [22], RBench [4], and RoboWM-Bench [11].Source: §2 · p. 2
3. Benchmark Overview
The benchmark pairs synchronized head and wrist predictions with task instructions and action sequences across bimanual manipulation episodes. Reference-derived phases provide shared temporal context for evaluating views and expected motion.
Task and data
- The test set contains 500 episodes across 50 bimanual RoboTwin 2.0 tasks, with synchronized initial frames, an instruction, and an action sequence for three predicted views.Tasks include pick-and-place, stacking, pouring, handover, switch actuation, and tool use.Source: §3 · p. 2
- Clean and domain-randomized backgrounds add appearance variation, and a separate 100-episode validation split includes reference videos and state annotations.Source: §3 · p. 2
Reference-derived action states
- Fixed rules over reference gripper opening and end-effector displacement assign coarse phases, acting-arm labels, and expected moving or static patterns for each view.These heuristic labels select phase-level keyframes and active head-to-wrist pairs; they are not verified task-success labels, and candidate failures cannot change the reference-derived phase boundaries.Source: §3 · pp. 2–3
4. Evaluation Protocol
The protocol combines reference comparisons and judgments of generated videos across 19 metrics, adapting evidence selection to camera roles and expected motion. Dimension, view, and joint scores preserve diagnostic detail alongside the aggregate TWB-Score.
4. Evaluation Protocol
- The 19 metrics separate within-view quality, cross-view agreement, and task alignment, combining reference-based comparisons with judgments of generated videos.Source: §4 · p. 3
- Tri-view consistency uses reference-based PSNR and SSIM as indirect signals, VLM judgments of head and active-wrist compatibility, and reference-phase VQA questions.The protocol distinguishes compatibility despite occlusion, explicit verification in both views, and calibration against labeled examples; reference similarity is not proof of agreement because valid executions may differ.Source: §4 · p. 3
- Metrics select cameras according to their evidence: task context and trajectories use the head view, while interaction quality and perspective are assessed separately in all three views.JEPA Similarity compares triplets with camera order preserved, and per-view scores retain differences in camera reliability.Source: §4 · p. 3
- State-conditioned measures judge motion against expected phase behavior, combining moving-or-static alignment, optical-flow magnitude, and time-warped head-view trajectory accuracy.Phase-conditioned penalties adjust subject, background, and photometric consistency when motion is missing or unexpected, preserving legitimate stillness.Source: §4 · p. 3, Figure 2 · p. 3
- Penalty-adjusted visual-quality scores discount attractive predictions when head-view trajectory accuracy or cross-view consistency is low, while raw visual quality remains separate.Metrics are averaged within dimensions and combined into TWB-Score; dimension-level, per-view, head-to-wrist, and joint scores remain available for diagnosis.Source: §4 · p. 3
5. Conclusion
TRIWORLDBENCH measures whether triple-view predictions are individually plausible, mutually consistent, and aligned with the task, while retaining broader world-model quality dimensions.
5. Conclusion
- The framework distinguishes whether each view is plausible, whether views agree, and whether the prediction matches the task.Cross-view consistency is measurable without reducing world-model quality to consistency alone.Source: §5 · p. 3
- Camera-aware metrics and reference-derived phases make these questions explicit across 19 metrics, with TWB-Score providing a compact summary.Source: §5 · p. 3
- The framework supports targeted diagnosis of failures in triple-view predictions for robot learning and planning.Source: §5 · p. 3