Source-linked AI summary
GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation
GigaWorld Team, Angyuan Ma, Boyuan Wang, Bohan Li, Chaojun Ni, Guo Li, Guan Huang, Guosheng Zhao, Hao Li, Hengtao Li, Jingyu Liu, Jiwen Lu, Qiuping Deng, Tingdong Yu, Xuancheng Xu, Xinyu Zhou, Xiuwei Xu, Xinze Chen, Xiaofeng Wang, Xiaoyu Tian, Yang Wang, Yifan Chang, Yukun Zhou, Yun Ye, Zhenyu Wu, Zhanqian Wu, Zheng Zhu
TL;DR
Robot-policy evaluation is a costly real-world bottleneck, and the properties that make world models reliable surrogate evaluators remain unclear. The paper addresses this gap with WMBench and a systematic study, finding that evaluator alignment depends on long-horizon action faithfulness, balanced pretraining, and design choices, then instantiating these findings in GigaWorld-1.
Problem
Robot policies require slow, costly real-world rollouts, while the properties that make world models reliable policy evaluators remain poorly understood.
Method
The paper introduces WMBench and systematically studies world models, action representations, rollout evaluation, data, pretraining, and architecture using paired real-world and simulated policy rollouts.
Results
Evaluator quality is associated with long-horizon action-faithful consistency, balanced world knowledge and robot controllability, and architectural choices; GigaWorld-1 improves evaluator-alignment metrics by 14.9% over strong baselines.
Takeaways & Limitations
Reliable world-model evaluation requires preserving broad world knowledge, remaining controllable under action input, and sustaining long-horizon consistency.
Takeaways & Limitations
WMBench does not yet cover the full space of mobile manipulation, dexterous in-hand manipulation, or safety-critical autonomy, and the study focuses primarily on video-centric world models.
Abstract
from arXiv · showhide
Evaluating embodied robot foundation models remains a critical bottleneck; unlike large language models efficiently assessed via digital benchmarks, robotic policies require slow, costly real-world rollouts limited by hardware and human supervision, which has driven interest in world models as surrogate policy evaluators, yet the key properties that make a world model reliable for policy assessment remain poorly understood. This work presents a systematic study of world models for robotic policy evaluation and introduces WMBench, a benchmark constructed from real-robot teleoperation data and matched policy rollouts covering diverse manipulation tasks to enable controlled comparisons across model families, action encodings, rollout horizons, and evaluation metrics. Using WMBench, we analyze 7 video world models, 4 action representation schemes, and over 324,000 simulated policy rollouts paired with real robot executions, further enriching our analysis with large-scale community submissions from the CVPR 2026 GigaBrain Challenge, curated synthetic trajectories, and a training videos spanning more than 12,000 hours. Our experiments deliver three core insights: evaluator quality is dominated by long-horizon, action-faithful rollout consistency rather than short-term visual realism; pretraining gains stem not only from data scale but from balancing general world knowledge with robot-specific controllability; and architectural choices including action encoding, memory design, and evaluator-focused post-training strongly determine alignment with real-world robot behavior. Drawing on these results, we derive a practical design roadmap and realize it in \textit{GigaWorld-1}, a world model specially optimized for policy evaluation, and we fully release our code, models, datasets, and toolkits to advance scalable evaluation research for embodied foundation models.
1. Introduction
The paper frames robot-policy evaluation as a costly real-world bottleneck and asks which world-model properties make surrogate evaluation reliable. It introduces WMBench for systematic study and derives design principles instantiated in GigaWorld-1.
- Motivation: Real-robot policy evaluation is slow, costly, hardware-intensive, and dependent on continuous human monitoring.These constraints make evaluation a primary bottleneck for progress in robot policy models.
- Research questions: The paper studies how to evaluate world-model evaluators beyond generic video quality, including effects of pretraining, data, architecture, and algorithmic choices.The stated goal is to move from proof-of-concept demonstrations toward principled design rules.
- Benchmark: WMBench pairs teleoperated expert data with robot policy rollouts to test whether generated videos preserve real-world policy outcomes.The benchmark supports controlled comparisons across world models, action representations, and robotic manipulation tasks.
- Study scale: The study covers 7 world models, 4 action representations, 8 manipulation tasks, and over 324,000 evaluation rollouts.These experiments are used to distill empirical conclusions about evaluator reliability.
- Outcome: GigaWorld-1 operationalizes the resulting design map and improves the core evaluator-alignment metric by 14.9% over strong baselines.The authors also release code, model weights, datasets, and auxiliary toolkits.
2. Related Work
World models can serve as scalable, visually rich policy-evaluation environments, but their reliability as evaluators is distinct from their usefulness as data engines, policies, or interaction environments. The paper positions this role between expensive real-world testing and imperfect classical simulation.
- World-model roles: World models support embodied AI through data generation, policy control, and learned interaction environments, but these paradigms do not directly establish reliable policy evaluation.Policy evaluation requires judging whether a policy succeeds under realistic visual and physical dynamics.
- Evaluation spectrum: Real-world testing is trustworthy but expensive, slow, and difficult to scale under broad distribution shifts.Physical evaluation directly measures execution in target environments but limits comprehensive testing.
- Evaluation target: A policy evaluator must judge decision-relevant behavior, not merely generate visually plausible observations.The relevant target is agreement with real-world policy outcomes.
- Evaluation spectrum: Classical simulators provide cheap, repeatable, and safe testing, yet often struggle with the sim-to-real gap in visually complex, contact-rich, or deformable scenarios.These limitations motivate alternatives that retain scalability while improving realism.
- World-model evaluators: World-model evaluators aim to combine simulation-like scalability with greater visual and physical realism without manual digital-twin construction.They operate directly in visually open environments using large-scale spatiotemporal priors.
3. Preliminaries
A robot policy maps observations, optional state, and task instructions to actions, while a world-model evaluator predicts future observations conditioned on those actions. Its success is measured by whether simulated policy outcomes agree with real-world outcomes across policies and conditions.
- Policy interaction: A robot policy π receives observation o_t, optional state s_t, and instruction l, then outputs action a_t = π(o_t, s_t, l).In real-world evaluation, the resulting trajectory supports estimates of task success and failure modes.
- World-model interaction: A world model M_θ replaces the physical environment and predicts future observations conditioned on the initial context and policy actions.Iterating these predictions over horizon H yields a world-model trajectory.
- Evaluator target: Evaluator quality concerns preservation of policy ranking, success prediction, and risk profiles observed in real-world execution.Visual plausibility alone is insufficient when it does not preserve decision-relevant properties.
- Evaluator target: The primary target is agreement between real-world success rates S_real(π) and world-model-inferred success rates S_wm(π).A key alignment measure is ranking correlation evaluated across policies, checkpoints, tasks, or rollout conditions.
4. WMBench: A Benchmark for World Models as Policy Evaluators
WMBench evaluates world models through paired real-world and learned-environment rollouts, strict episode-disjoint training splits, closed-loop policy interaction, and outcome-plus-diagnostic metrics. Its annotations distinguish correct outcomes from visual and physical rollout fidelity.
- Benchmark design: WMBench measures evaluator reliability by connecting visual generation with physical policy outcomes through paired real-world and world-model rollouts.The benchmark defines a hierarchical metric system for generation quality and evaluator reliability.
- Data: The benchmark contains 2,989 paired trajectories across eight tasks from teleoperated real-world data and policy rollouts, with a near-1:1 source ratio.Episode-disjoint splitting prevents overlap between training and test trajectories.
- Large-scale annotations: 324,000 rollout segments from over 100 challenge teams are chained into long-horizon policy–world-model interaction episodes and human-annotated.Complete episodes contain approximately 20 to 30 rollout segments.
- Outcome metric: WMES uses four ordinal levels that separately encode whether the final task outcome is correct and whether visual execution has high or low fidelity.Scores 3 and 2 have correct outcomes, whereas scores 1 and 0 have incorrect outcomes, with fidelity distinguishing each pair.
- Evaluation protocol: The protocol collects real closed-loop rollouts, trains on a strict holdout split, runs closed-loop policy interaction in the world model, and evaluates metrics plus outcomes.The final assessment combines automatic visual and motion metrics with WMES from human annotation or a VLM.
- Metric system: Diagnostic metrics cover frame and representation fidelity, geometry, semantics, interaction, and long-horizon behavior rather than relying on visual quality alone.Geometry Accuracy compares generated and reference depth after median-based scale alignment.
5. What Matters in Building World Models For Evaluating Robot Policies?
Evaluator quality depends on whether world models preserve policy-relevant state and action consequences across long-horizon rollouts, not merely on snapshot or visual realism. Correlation analysis, rollout studies, VLM evaluation, pretraining comparisons, and architecture experiments identify the metrics and design choices most aligned with WMES.
- Metric alignment: Visual Fidelity (ρ= 0.78), Geometry (ρ= 0.71), and Semantics (ρ= 0.59) are the strongest group-level predictors of WMES.At the individual-metric level, Subject Consistency (ρ= 0.88), Perspectivity (ρ= 0.86), and Instruction Following (ρ= 0.84) are strongest.
- Metric alignment: Background Consistency (ρ= −0.45), Photometric Consistency (ρ= −0.42), and Interaction Quality (ρ= −0.11) negatively correlate with WMES and can mislead evaluator ranking.Static videos can score highly on appearance stability while ignoring actions, and current VLMs do not robustly assess physical realism for fine-grained ranking.
- Rollout protocol: Long-horizon rollout quality, rather than single-step generation quality, is necessary because autoregressive state errors compound and can change policy-level conclusions.Generic backbones may begin with plausible segments but later exhibit viewpoint drift, object-identity collapse, texture accumulation, and severe degradation.
- Pretraining and data: Transferable physical priors can matter more than parameter scale: Cosmos-Predict2.54 reaches AVG 0.6123, while Wan 2.2 5B reaches AVG 0.5948 versus LTX 2.3 at AVG 0.5775.The comparison favors compatibility between pretrained priors and robot-conditioned rollout over raw model size alone.
- Architecture and control: Channel-concatenated control maps achieve the best paired scores, while persistent hierarchical memory improves long-horizon rollout quality across all evaluated intervals.The memory design uses a persistent first-frame anchor plus short-, mid-, and long-range temporal memories to reduce drift and preserve scene identity.
6. Final Design Map and GigaWorld-1
GigaWorld-1 turns the paper’s data-to-model-to-evaluation design map into an evaluator-oriented world model, combining balanced multi-source data, explicit control, hierarchical memory, and long-horizon rollout mechanisms. It improves evaluator-alignment and rollout stability while better matching real-world policy outcomes, including under distribution shifts.
- Design map: Evaluator quality depends on balancing generic world knowledge with robot-specific controllability, while targeting agreement with real-world policy success under in-distribution and OOD conditions.The design map identifies data balance as a data-level factor and real-world policy agreement as the decisive evaluation target.
- Data and architecture: GigaWorld-1 combines real robot, policy rollout, egocentric, simulation-derived, and filtered challenge data with explicit control, hierarchical memory, relative temporal encoding, and progressive training.Its corpus spans multiple embodiment and video sources, while the architecture preserves pretrained priors and stabilizes iterative rollouts.
- Rollout stability: GigaWorld-1’s anchored hierarchical memory preserves appearance statistics and multi-scale temporal context, while relative temporal positions reduce repetitive motion and instability.SLERP complements memory by smoothing semantic transitions between task phases when abrupt prompt changes would otherwise cause over-conditioning.
- Efficiency: 35.93×: combining SageAttention, six-step DMD2, and Ulysses sequence parallelism accelerates inference in the reported benchmark.Attention-kernel optimization alone provides 1.25×–1.31× acceleration.
- Benchmark comparison: 14.9%: GigaWorld-1 improves the average evaluator-relevant score over Wan 2.2 5B under the controlled comparison.The normalized average covers Aesthetic Quality, Image Quality, JEPA Similarity, Semantic Alignment, Subject Consistency, and Trajectory Accuracy.
- Evaluator-relevant metrics: GigaWorld-1 achieves the best JEPA Similarity (0.9337), Semantic Alignment (0.8926), and Trajectory Accuracy (0.3561), while matching the best Subject Consistency score (0.8883).These metrics are reported as the strongest component results in the comparison.
- Long-horizon rollout: 40 seconds: GigaWorld-1 maintains the best PSNR, FID, and FVD throughout autoregressive long-horizon generation.Generic baselines instead exhibit viewpoint drift, object-identity collapse, and accumulated texture artifacts.
- Policy calibration: GigaWorld-1 better preserves action outcomes and policy difficulty under OOD shifts, following the real-world success-rate diagonal more closely than challenge baselines.Its success-rate deviations are smaller overall, whereas other challenge models tend to over-confidently predict success.
7. Discussion and Conclusion
The discussion frames reliable policy evaluation as preserving action-conditioned behavior over long horizons, not merely generating realistic frames. GigaWorld-1 supports practical policy iteration, while the benchmark and conclusions remain bounded by task, model-family, and annotation limitations.
- Core findings: Long-horizon action-faithful consistency is more important for policy evaluation than short-term visual realism.Plausible local motion can still drift from the action-conditioned evolution needed for reliable policy comparison.
- Robustness: World-model evaluators should be tested under distribution shifts spanning appearance, object content, background, surface, and action-outcome changes.The OOD cases include both successful and failed executions.
- Practical implications: Reliable evaluation requires broad world knowledge, action controllability, and sustained rollout consistency within a practical repeated-use pipeline.These properties connect simulator capability with evaluator usefulness for iterative policy improvement.
- Limitations: WMBench does not yet cover the full space of mobile manipulation, dexterous in-hand manipulation, or safety-critical autonomy.The authors also note that conclusions focused on video-centric models may not transfer to structured state-space or hybrid 3D approaches.
- Evaluator alignment: GigaWorld-1 better calibrates task difficulty than challenge baselines, with generated success rates closer to real-robot success rates.Figure 16 compares closed-loop generated and real-robot success rates against a perfect-agreement diagonal.