PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey on World Models for Embodied AI

Xinqing Li, Xin He, Le Zhang, Min Wu, Xiaoli Li, Yun Liu

arXiv:2510.16732v3cs.CV

TL;DR

Embodied AI needs models that anticipate how actions change future states, but existing world-model research spans heterogeneous functions, temporal paradigms, and spatial representations. This survey formalizes the problem, proposes a three-axis taxonomy, and systematizes datasets, metrics, and model comparisons across domains. It concludes that progress depends on unified evaluation, physically grounded objectives, efficient control-oriented models, and improved long-horizon consistency.

  • Problem

    Embodied AI requires agents to perceive, act, and anticipate future world states, while world-model research spans heterogeneous functions, temporal paradigms, and spatial representations.

  • Method

    The survey formalizes world-model learning and organizes approaches by decision coupling, temporal modeling, and spatial representation while reviewing datasets, metrics, and benchmarks.

  • Results

    Across robotics, autonomous driving, and general-purpose video, the survey maps model trade-offs and reports progress in pixel fidelity, temporal coherence, scene understanding, and control performance.

  • Takeaways & Limitations

    Future progress should combine general-purpose video and 3D models with action- and reward-conditioned control and causally and physically grounded objectives.

Abstract

from arXiv · show

Embodied AI requires agents that perceive, act, and anticipate how actions reshape future world states. World models serve as internal simulators that capture environment dynamics, enabling forward and counterfactual rollouts to support perception, prediction, and decision making. This survey presents a unified framework for world models in embodied AI. Specifically, we formalize the problem setting and learning objectives, and propose a three-axis taxonomy encompassing: (1) Functionality, Decision-Coupled vs. General-Purpose; (2) Temporal Modeling, Sequential Simulation and Inference vs. Global Difference Prediction; (3) Spatial Representation, Global Latent Vector, Token Feature Sequence, Spatial Latent Grid, and Decomposed Rendering Representation. We systematize data resources and metrics across robotics, autonomous driving, and general video settings, covering pixel prediction quality, state-level understanding, and task performance. Furthermore, we offer a quantitative comparison of state-of-the-art models and distill key open challenges, including the scarcity of unified datasets and the need for evaluation metrics that assess physical consistency over pixel fidelity, the trade-off between model performance and the computational efficiency required for real-time control, and the core modeling difficulty of achieving long-horizon temporal consistency while mitigating error accumulation. Finally, we maintain a curated bibliography at https://github.com/Li-Zn-H/AwesomeWorldModels.

I. INTRODUCTION

This survey frames world models as internal simulators for embodied agents and organizes the field around unified foundations, taxonomy, resources, and evaluation. It formalizes latent-state modeling while emphasizing temporal coherence, spatial fidelity, and the efficiency–control trade-off.

  • World models capture environment dynamics to support forward and counterfactual rollouts for perception, prediction, and decision making.
  • The survey addresses temporal evolution and spatial scene encoding because long-horizon rollouts accumulate errors and coarse layouts limit geometry-aware planning.
  • It introduces a three-axis framework covering functionality, temporal modeling, and spatial representation, and applies it across robotics, autonomous driving, and general-purpose video.
  • The framework integrates standardized datasets and evaluation metrics to support quantitative comparisons and an actionable map of future research.
  • In the formalization, a learned latent state is inferred from observations and prior latent dynamics, then used to reconstruct observations under partial observability.
  • World-model training uses an ELBO whose Markov-factorized form combines reconstruction with KL regularization aligning filtered posteriors and dynamics priors.

III. TAXONOMY

The taxonomy organizes world models by decision coupling, temporal prediction, and spatial representation. It links each choice to characteristic trade-offs in control alignment, generalization, coherence, fidelity, and efficiency.

  • Decision coupling: Decision-Coupled models align dynamics with specific control objectives, while General-Purpose models prioritize broad visual and physical coverage across tasks and domains.
  • Decision coupling: Decision-Coupled models improve sample efficiency and closed-loop performance but generalize less broadly, whereas General-Purpose models may misalign with downstream control.
  • Temporal reasoning: Sequential Simulation and Inference unfolds future states autoregressively for fine-grained closed-loop planning, but incurs rollout-length cost and long-horizon error accumulation.
  • Spatial representation: Spatial representations span compact global vectors, attention-based token sequences, spatial latent grids, and explicit or decomposed geometric representations.
  • The taxonomy tables classify representative robotics and autonomous-driving approaches and provide a roadmap for subsequent analysis.

A. Decision-Coupled World Models

Decision-coupled world models are learned simulators tied to control objectives, evolving from RSSM-based latent dynamics toward token, state-space, and inverse-dynamics formulations for planning and transfer.

  • Global Latent Vector: Global latent-vector models prioritize efficient real-time and long-horizon prediction, but discard fine-grained spatial and temporal detail.RNNs are the primary temporal backbone in this representation family.
  • Foundations: Early decision-coupled models encode observations into latent states and use recurrent dynamics for policy optimization and long-horizon imagination.PlaNet introduced RSSM by combining deterministic memory with stochastic components; Dreamer variants extended this formulation.
  • Transfer and Deployment: Decision-coupled world models increasingly address transfer across modalities, tasks, embodiments, and simulation-to-real settings.Methods use embodiment-agnostic action representations, residual latent-dynamics corrections, selective fine-tuning, and limited real-world data.
  • Temporal and Inverse Dynamics: Transformer and state-space variants improve long-horizon modeling, while object-conditioned inverse dynamics predicts actions needed to reach specified target states.Mamba-based models combine linear-time complexity with long-horizon capacity; inverse-dynamics models connect target states to required actions.
  • Token Feature Sequence: Token feature sequences model dependencies among discrete tokens to support causal inference, multimodal integration, and LLM reuse.Applications include cross-modal driving prediction, unified perception-planning-action modeling, and imagined policy rollouts.

2) Global Difference Prediction:

Global difference prediction models forecast future representations in parallel, using tokenized, grid-based, diffusion, and 3D-aware mechanisms for controllable embodied-world generation.

  • Token Feature Sequence: Token feature sequences predict future tokens in parallel, reducing error accumulation while supporting multimodal diversity.These representations are especially useful for global prediction because compact global latent vectors discard fine-grained spatiotemporal detail.
  • Token Feature Sequence: Tokenized representations support controllable scene synthesis, trajectory-conditioned generation, and latent future representations for control.Examples include object-level scene tokens, 3D trajectory-conditioned views, and diffusion policies aligned with latent futures.
  • Spatial Latent Grid: Spatial grid models forecast BEV or voxel maps in parallel while preserving locality and uncertainty for fast control.They produce planner-ready maps from ego-stabilized views.
  • Diffusion and 3D Generation: Diffusion-based models combine video generation with differentiable physics, 4D reconstruction, instruction decomposition, and action-tree or depth guidance.These designs target photorealistic, physically consistent, compositional, and temporally consistent futures.
  • Planning: World-model predictions can directly support planning through future-motion cues, multimodal policy learning, and image-based trajectory selection.Applications span label-free cross-robot manipulation, accelerated policy learning, and safer driving trajectories.

B. General-Purpose World Models

General-purpose world models pretrain task-agnostic dynamics to capture environmental physics and generate future scenes, emphasizing transferability across tasks and settings.

  • Scope: General-purpose models prioritize task-agnostic dynamics that capture environmental physics and generate future scenes for transferability.This objective distinguishes them from models tightly coupled to specific control tasks.
  • Video and Interaction Pretraining: Unlabeled interaction-video pretraining supports action-free forecasting, discrete latent actions, spatiotemporal tokens, and later adaptation to control.These models learn forecasting or interactive-environment capabilities before downstream control adaptation.
  • Diffusion World Models: Diffusion-based general-purpose models generate controllable futures conditioned on actions, text, instructions, or camera motions.Examples span driving simulation, goal-directed control, and dynamic 3D exploratory worlds.
  • Geometric Consistency: Explicit 3D priors and geometry-aware memory improve geometric consistency and long-horizon generation in diffusion-based models.These methods combine geometric foundation features or visual-geometric prediction targets with temporal-consistency mechanisms.
  • Sequential Simulation: Sequential general-purpose models provide action-conditioned rollouts for policy evaluation and training, including reinforcement learning with verifiable rewards.WorldGym and WorldEval use rollout critics, while RLVR-World aligns pretraining and task objectives through explicit metrics.
  • Long-Horizon Modeling: Long-range consistency is also addressed with state-space memory, local attention, object-slot dynamics, and sparsity-regularized causal interactions.These mechanisms target sustained memory, short-term coherence, and causal reasoning over environments.
  • Spatial Latent Grid: Spatial latent grids preserve locality and support efficient rollouts, multimodal fusion, and planner-ready maps.Structured-grid and physics-informed methods encode geometry and dynamics for controllable rollouts and downstream driving tasks.
  • Decomposed Rendering: Decomposed rendering models update explicit 3D primitives or combine voxel generation, video synthesis, and dynamic Gaussian representations for view-consistent trajectories.GaussianWorld separates ego-motion, object dynamics, and newly observed regions when updating scene primitives.

2) Global Difference Prediction:

Global difference prediction extends general-purpose modeling through masked or generative latent prediction, unified multimodal scene representations, and explicit 3D rendering structures.

  • Token Feature Sequence: Token feature sequences support global prediction through masked and generative modeling with parallel long-horizon rollouts and multimodal conditioning.The paradigm centers on tokenized feature sequences for general-purpose world models.
  • Masked Latent Prediction: JEPA-based models predict latent features of occluded spatiotemporal regions without pixel reconstruction or contrastive learning.V-JEPA 2 scales this pretraining to Internet videos and incorporates limited robot interaction data for robotic planning.
  • Diffusion Modeling: Diffusion models represent video as unified spacetime patches or combine language, image, video, audio, and action sequences in multimodal architectures.These designs position global prediction as a basis for general-purpose understanding, generation, simulation, and action prediction.
  • Spatial Latent Grid: Spatial-grid models forecast voxel grids in parallel and fuse multiview features into unified maps for general-purpose world modeling.Related systems jointly model future appearance and depth, connect scene understanding with prediction, and forecast synchronized multimodal BEV representations.
  • 4D Scene Generation: Tokenized 4D and occupancy representations enable trajectory-conditioned diffusion and controllable high-fidelity scene dynamics.Additional consistency mechanisms decouple ego-motion from scene evolution and incorporate physics-informed constraints.
  • Decomposed Rendering: Decomposed rendering combines explicit 3D structure with video-generative priors for global prediction.This representation family targets structured scene synthesis rather than only compact latent forecasting.
  • Trade-offs: Across the taxonomy, decision coupling, temporal modeling, and spatial representation create trade-offs between rollout efficiency, fidelity, geometric coherence, and deployment cost.Grid-based and decomposed-rendering models provide strong geometric and multiview coherence but remain difficult to deploy under strict latency and memory budgets.
  • Evaluation Scope: The survey organizes resources and evaluation around diverse embodied-AI tasks, including simulation, interactive benchmarks, offline datasets, and real-world robot platforms.Its broader assessment covers robotics, autonomous driving, and general video settings.

A. Data Resources

The survey organizes embodied world-model data into simulation platforms, interactive benchmarks, offline datasets, and real-world robot platforms, spanning robotics, driving, and general video.

  • Simulation Platforms: Simulation platforms provide controllable, scalable environments for training and evaluating world models across robotics, autonomous driving, and indoor navigation.Examples include MuJoCo, NVIDIA Isaac, CARLA, and Habitat.
  • Interactive Benchmarks: Interactive benchmarks standardize task suites and closed-loop protocols for reproducible evaluation, including control, games, manipulation, lifelong learning, and autonomous driving.Examples range from DeepMind Control and Atari100k to Meta-World, RLBench, LIBERO, and nuPlan.
  • Offline Datasets: Offline datasets provide precollected trajectories for reproducible evaluation and data-efficient pretraining without interactive rollouts.The surveyed resources include real-world robot demonstrations, cross-embodiment trajectories, multimodal driving data, and large-scale video corpora.
  • Offline Datasets: Driving resources capture multimodal sensing and 3D structure through cameras, radar, LiDAR, GPS/IMU, HD maps, and voxel occupancy labels.nuScenes, Waymo, and Occ3D support multimodal fusion, traffic-dynamics modeling, long-horizon prediction, and scene understanding beyond bounding boxes.
  • Offline Datasets: General video resources support fine-grained action understanding and self-supervised pretraining through large, curated collections with motion-rich sampling.VideoMix22M scales to 22 million samples, while VisionMix163M expands image and video sources and emphasizes temporally consistent visual features.

4) Real-world Robot Platforms:

Real-world robot platforms provide physical embodiments for closed-loop evaluation, high-fidelity data collection, and sim-to-real validation under real-world constraints.

  • Real-world Robot Platforms: Real-world robot platforms enable closed-loop evaluation and data collection under physical constraints, supporting sim-to-real validation.The survey highlights these platforms as embodied testbeds rather than purely virtual environments.
  • Real-world Robot Platforms: Franka Emika is a 7-DoF collaborative arm with joint torque sensing and 1 kHz torque control for precise, contact-rich manipulation.ROS integration broadens its use as a research platform.
  • Real-world Robot Platforms: Unitree Go1 is a cost-effective quadruped with panoramic depth sensing, 1.5 TFLOPS onboard computing, and a maximum speed of 4.7 m/s.These capabilities support locomotion and embodied-AI research.
  • Real-world Robot Platforms: Unitree G1 is a compact humanoid offering up to 43-DoF, 120 N·m knee torque, integrated 3D LiDAR and depth cameras, and a practical low-cost testbed.Its multimodal sensing, onboard compute, ROS support, and swappable batteries support world-model training and evaluation.

B. Metrics

The survey organizes world-model metrics into pixel prediction quality, state-level understanding, and task performance, progressing from sensory fidelity toward goal attainment.

  • Metrics: Metrics span three abstraction levels: pixel prediction quality, state-level understanding, and task performance.This progression moves from low-level signal fidelity to high-level goal attainment.
  • Pixel Prediction Quality: Pixel-level metrics evaluate image fidelity, temporal consistency, perceptual similarity, and generated-video quality.The surveyed measures include FID, FVD, SSIM, PSNR, LPIPS, and VBench.
  • Pixel Prediction Quality: Lower FID indicates closer alignment between real and generated image distributions, while FVD additionally evaluates video appearance and dynamics.FID penalizes mean and covariance mismatch; FVD uses motion-aware features and penalizes artifacts such as flickering.
  • Pixel Prediction Quality: SSIM compares luminance, contrast, and structure, with values closer to 1 indicating higher similarity.The score is averaged over sliding windows.
  • Pixel Prediction Quality: PSNR measures pixel-wise distortion, whereas LPIPS compares pretrained-network features and generally treats lower values as greater perceptual similarity.LPIPS is designed to correlate with human judgments and remain robust to distortions.

2) State-level Understanding:

State-level metrics assess whether world models capture semantic structure, geometry, localization, and object evolution beyond visual appearance.

  • State-level Understanding: State-level understanding evaluates objects, layouts, semantics, geometry, occupancy, and trajectories rather than appearance alone.The metrics include segmentation, detection, occupancy, geometry, and trajectory accuracy.
  • State-level Understanding: Higher mIoU reflects more precise semantic scene understanding through class-averaged segmentation overlap.IoU measures overlap with ground truth while penalizing segmentation errors.
  • State-level Understanding: Higher mAP indicates better instance recognition, localization, and confidence calibration across classes and IoU thresholds.mAP averages per-class Average Precision after confidence-ranked matching to ground-truth instances.
  • State-level Understanding: Displacement errors measure spatial accuracy for keypoints, object centers, and trajectory waypoints, with lower ADE and FDE indicating more accurate localization.ADE averages displacement over steps, whereas FDE measures displacement at the final step.
  • State-level Understanding: Chamfer Distance measures geometric similarity between predicted and ground-truth sets and complements IoU for surfaces, occupancy, BEV, and 3D structures.Its differentiability also permits use as a training loss.

3) Task Performance:

Task performance evaluates whether world models support effective decision-making under safety and efficiency constraints. The survey organizes comparisons by task objectives and uses standardized benchmarks alongside metrics for success, efficiency, reward, collision safety, and video quality.

  • Task-level metrics assess goal achievement under safety and efficiency constraints in embodied settings.
  • Success Rate measures the fraction of evaluation episodes satisfying a predefined success condition, including collision-free route completion in autonomous driving.
  • Sample Efficiency quantifies the samples required to reach target performance through fixed-budget benchmarks, data-performance curves, or demonstrations.
  • Reward reports cumulative or average return in reinforcement learning, based on discounted timestep rewards.
  • Collision safety is measured by the proportion of episodes with at least one collision or exposure-normalized rates such as collisions per kilometer.
  • For nuScenes video generation, DrivePhysica achieves the best visual fidelity while MiLA achieves the strongest temporal coherence.FID evaluates appearance fidelity, whereas FVD evaluates temporal consistency.

B. Scene Understanding

The survey evaluates world models across scene forecasting and control tasks using task-specific protocols and metrics. Results expose both performance trade-offs and persistent comparison challenges caused by heterogeneous implementations, while broad transferability remains unresolved.

  • 4D occupancy forecasting predicts 3 s of future scene dynamics from 2 s of past 3D occupancy using mIoU and per-horizon IoU.
  • DMC evaluates control-relevant dynamics with 64×64×3 observations using Episode Return over 1,000 steps.The theoretical maximum is 1,000 when rt ∈[0, 1].
  • A broadly transferable model across tasks, modalities, and datasets remains an open challenge.
  • RLBench measures manipulation performance with Success Rate, but differing episode budgets, resolutions, and modalities complicate like-for-like comparison.
  • UniAD+DriveWorld achieves the lowest L2, while SSR attains the best collision rate with competitive L2 without extra supervision.
  • Camera-based planning methods surpass privileged-occupancy models, reflecting the growing maturity of end-to-end planning.

A. Data & Evaluation

The survey identifies fragmented data and evaluation practices as major barriers, then outlines directions for unified benchmarks, physically grounded metrics, efficient architectures, and more stable long-horizon modeling. Its three-axis framework organizes these trade-offs across embodied world-model design.

  • Data & Evaluation: Unified large-scale datasets remain lacking across navigation, manipulation, and autonomous driving, with multi-agent data especially underrepresented.This fragmentation constrains generalization as model scale grows faster than task-relevant data availability.
  • Data & Evaluation: FID and FVD emphasize pixel fidelity while overlooking physical consistency, dynamics, and causality, leaving a gap between video-simulator and embodied-controller evaluation.Existing structured benchmarks remain task-specific and lack cross-domain standards.
  • Data & Evaluation: Future benchmarks should be unified, multimodal, and cross-domain, coupling scenes, actions, rewards, multi-agent interaction, and safety-critical events.Evaluation should also assess causal reasoning, physical consistency, and long-horizon error accumulation.
  • Computational Efficiency: Real-time deployment is constrained by the inference cost and memory footprint of Transformers and diffusion decoders on onboard robotic platforms.Practical systems therefore often use compact recurrent models and global latent vectors, trading capacity for predictable behavior.
  • Computational Efficiency: Quantization, pruning, low-rank and sparse computation, SSMs, and adaptive resource use are proposed to improve fidelity and efficiency jointly.High-capacity video-diffusion components may serve offline generation, while lightweight cores handle online control and safety monitoring.
  • Modeling Strategy: World models face a trade-off between compact autoregressive simulation, which accumulates errors, and global prediction, which improves coherence but increases computation and weakens closed-loop interactivity.Hybrid local-autoregressive and global or masked-update designs are proposed to improve long-horizon stability.
Loading 2510.16732v3…