A Comprehensive Survey on World Models for Embodied AI

• World models capture environment dynamics to support forward and counterfactual rollouts for perception, prediction, and decision making.Source: §I · p. 1
• The survey addresses temporal evolution and spatial scene encoding because long-horizon rollouts accumulate errors and coarse layouts limit geometry-aware planning.Source: §I · p. 1
• It introduces a three-axis framework covering functionality, temporal modeling, and spatial representation, and applies it across robotics, autonomous driving, and general-purpose video.Source: §I · p. 2
• The framework integrates standardized datasets and evaluation metrics to support quantitative comparisons and an actionable map of future research.Source: §I · p. 2
• In the formalization, a learned latent state is inferred from observations and prior latent dynamics, then used to reconstruct observations under partial observability.Source: B. Mathematical Formulation of World… · p. 2
• World-model training uses an ELBO whose Markov-factorized form combines reconstruction with KL regularization aligning filtered posteriors and dynamics priors.Source: B. Mathematical Formulation of World… · p. 2

Decision coupling

• Decision-Coupled models align dynamics with specific control objectives, while General-Purpose models prioritize broad visual and physical coverage across tasks and domains.Source: §III · p. 3
• Decision-Coupled models improve sample efficiency and closed-loop performance but generalize less broadly, whereas General-Purpose models may misalign with downstream control.Source: §III · p. 3

Temporal reasoning

• Sequential Simulation and Inference unfolds future states autoregressively for fine-grained closed-loop planning, but incurs rollout-length cost and long-horizon error accumulation.Source: §III · p. 3

Spatial representation

• Spatial representations span compact global vectors, attention-based token sequences, spatial latent grids, and explicit or decomposed geometric representations.Source: §III · p. 3

• The taxonomy tables classify representative robotics and autonomous-driving approaches and provide a roadmap for subsequent analysis.Source: §III · p. 3, Table I · p. 4

Global Latent Vector

• Global latent-vector models prioritize efficient real-time and long-horizon prediction, but discard fine-grained spatial and temporal detail.RNNs are the primary temporal backbone in this representation family.Source: 1) Sequential Simulation and Inference · p. 4

Foundations

• Early decision-coupled models encode observations into latent states and use recurrent dynamics for policy optimization and long-horizon imagination.PlaNet introduced RSSM by combining deterministic memory with stochastic components; Dreamer variants extended this formulation.Source: 1) Sequential Simulation and Inference · p. 4

Transfer and Deployment

• Decision-coupled world models increasingly address transfer across modalities, tasks, embodiments, and simulation-to-real settings.Methods use embodiment-agnostic action representations, residual latent-dynamics corrections, selective fine-tuning, and limited real-world data.Source: 1) Sequential Simulation and Inference · p. 5

Temporal and Inverse Dynamics

• Transformer and state-space variants improve long-horizon modeling, while object-conditioned inverse dynamics predicts actions needed to reach specified target states.Mamba-based models combine linear-time complexity with long-horizon capacity; inverse-dynamics models connect target states to required actions.Source: 1) Sequential Simulation and Inference · p. 6

Token Feature Sequence

• Token feature sequences model dependencies among discrete tokens to support causal inference, multimodal integration, and LLM reuse.Applications include cross-modal driving prediction, unified perception-planning-action modeling, and imagined policy rollouts.Source: 1) Sequential Simulation and Inference · p. 6

Token Feature Sequence

• Token feature sequences predict future tokens in parallel, reducing error accumulation while supporting multimodal diversity.These representations are especially useful for global prediction because compact global latent vectors discard fine-grained spatiotemporal detail.Source: 2) Global Difference Prediction · p. 8
• Tokenized representations support controllable scene synthesis, trajectory-conditioned generation, and latent future representations for control.Examples include object-level scene tokens, 3D trajectory-conditioned views, and diffusion policies aligned with latent futures.Source: 2) Global Difference Prediction · p. 8

Spatial Latent Grid

• Spatial grid models forecast BEV or voxel maps in parallel while preserving locality and uncertainty for fast control.They produce planner-ready maps from ego-stabilized views.Source: 2) Global Difference Prediction · p. 8

Diffusion and 3D Generation

• Diffusion-based models combine video generation with differentiable physics, 4D reconstruction, instruction decomposition, and action-tree or depth guidance.These designs target photorealistic, physically consistent, compositional, and temporally consistent futures.Source: 2) Global Difference Prediction · p. 8

Planning

• World-model predictions can directly support planning through future-motion cues, multimodal policy learning, and image-based trajectory selection.Applications span label-free cross-robot manipulation, accelerated policy learning, and safer driving trajectories.Source: 2) Global Difference Prediction · p. 8

Scope

• General-purpose models prioritize task-agnostic dynamics that capture environmental physics and generate future scenes for transferability.This objective distinguishes them from models tightly coupled to specific control tasks.Source: 1) Sequential Simulation and Inference · p. 8

Video and Interaction Pretraining

• Unlabeled interaction-video pretraining supports action-free forecasting, discrete latent actions, spatiotemporal tokens, and later adaptation to control.These models learn forecasting or interactive-environment capabilities before downstream control adaptation.Source: 1) Sequential Simulation and Inference · p. 8

Diffusion World Models

• Diffusion-based general-purpose models generate controllable futures conditioned on actions, text, instructions, or camera motions.Examples span driving simulation, goal-directed control, and dynamic 3D exploratory worlds.Source: 1) Sequential Simulation and Inference · pp. 8–9

Geometric Consistency

• Explicit 3D priors and geometry-aware memory improve geometric consistency and long-horizon generation in diffusion-based models.These methods combine geometric foundation features or visual-geometric prediction targets with temporal-consistency mechanisms.Source: 1) Sequential Simulation and Inference · p. 9

Sequential Simulation

• Sequential general-purpose models provide action-conditioned rollouts for policy evaluation and training, including reinforcement learning with verifiable rewards.WorldGym and WorldEval use rollout critics, while RLVR-World aligns pretraining and task objectives through explicit metrics.Source: 1) Sequential Simulation and Inference · p. 9

Long-Horizon Modeling

• Long-range consistency is also addressed with state-space memory, local attention, object-slot dynamics, and sparsity-regularized causal interactions.These mechanisms target sustained memory, short-term coherence, and causal reasoning over environments.Source: 1) Sequential Simulation and Inference · p. 9

Spatial Latent Grid

• Spatial latent grids preserve locality and support efficient rollouts, multimodal fusion, and planner-ready maps.Structured-grid and physics-informed methods encode geometry and dynamics for controllable rollouts and downstream driving tasks.Source: 1) Sequential Simulation and Inference · p. 9

Decomposed Rendering

• Decomposed rendering models update explicit 3D primitives or combine voxel generation, video synthesis, and dynamic Gaussian representations for view-consistent trajectories.GaussianWorld separates ego-motion, object dynamics, and newly observed regions when updating scene primitives.Source: 1) Sequential Simulation and Inference · p. 9

Token Feature Sequence

• Token feature sequences support global prediction through masked and generative modeling with parallel long-horizon rollouts and multimodal conditioning.The paradigm centers on tokenized feature sequences for general-purpose world models.Source: 2) Global Difference Prediction · pp. 9–10

Masked Latent Prediction

• JEPA-based models predict latent features of occluded spatiotemporal regions without pixel reconstruction or contrastive learning.V-JEPA 2 scales this pretraining to Internet videos and incorporates limited robot interaction data for robotic planning.Source: 2) Global Difference Prediction · p. 10

Diffusion Modeling

• Diffusion models represent video as unified spacetime patches or combine language, image, video, audio, and action sequences in multimodal architectures.These designs position global prediction as a basis for general-purpose understanding, generation, simulation, and action prediction.Source: 2) Global Difference Prediction · p. 10

Spatial Latent Grid

• Spatial-grid models forecast voxel grids in parallel and fuse multiview features into unified maps for general-purpose world modeling.Related systems jointly model future appearance and depth, connect scene understanding with prediction, and forecast synchronized multimodal BEV representations.Source: 2) Global Difference Prediction · p. 10

4D Scene Generation

• Tokenized 4D and occupancy representations enable trajectory-conditioned diffusion and controllable high-fidelity scene dynamics.Additional consistency mechanisms decouple ego-motion from scene evolution and incorporate physics-informed constraints.Source: 2) Global Difference Prediction · p. 10

Decomposed Rendering

• Decomposed rendering combines explicit 3D structure with video-generative priors for global prediction.This representation family targets structured scene synthesis rather than only compact latent forecasting.Source: 2) Global Difference Prediction · p. 10

Trade-offs

• Across the taxonomy, decision coupling, temporal modeling, and spatial representation create trade-offs between rollout efficiency, fidelity, geometric coherence, and deployment cost.Grid-based and decomposed-rendering models provide strong geometric and multiview coherence but remain difficult to deploy under strict latency and memory budgets.Source: 2) Global Difference Prediction · pp. 10–11

Evaluation Scope

• The survey organizes resources and evaluation around diverse embodied-AI tasks, including simulation, interactive benchmarks, offline datasets, and real-world robot platforms.Its broader assessment covers robotics, autonomous driving, and general video settings.Source: §IV · p. 11

Simulation Platforms

• Simulation platforms provide controllable, scalable environments for training and evaluating world models across robotics, autonomous driving, and indoor navigation.Examples include MuJoCo, NVIDIA Isaac, CARLA, and Habitat.Source: A. Data Resources · p. 11, 1) Simulation Platforms · p. 11

Interactive Benchmarks

• Interactive benchmarks standardize task suites and closed-loop protocols for reproducible evaluation, including control, games, manipulation, lifelong learning, and autonomous driving.Examples range from DeepMind Control and Atari100k to Meta-World, RLBench, LIBERO, and nuPlan.Source: 2) Interactive Benchmarks · p. 11

Offline Datasets

• Offline datasets provide precollected trajectories for reproducible evaluation and data-efficient pretraining without interactive rollouts.The surveyed resources include real-world robot demonstrations, cross-embodiment trajectories, multimodal driving data, and large-scale video corpora.Source: 3) Offline Datasets · pp. 11–12
• Driving resources capture multimodal sensing and 3D structure through cameras, radar, LiDAR, GPS/IMU, HD maps, and voxel occupancy labels.nuScenes, Waymo, and Occ3D support multimodal fusion, traffic-dynamics modeling, long-horizon prediction, and scene understanding beyond bounding boxes.Source: 3) Offline Datasets · p. 11
• General video resources support fine-grained action understanding and self-supervised pretraining through large, curated collections with motion-rich sampling.VideoMix22M scales to 22 million samples, while VisionMix163M expands image and video sources and emphasizes temporally consistent visual features.Source: 3) Offline Datasets · p. 12

Real-world Robot Platforms

• Real-world robot platforms enable closed-loop evaluation and data collection under physical constraints, supporting sim-to-real validation.The survey highlights these platforms as embodied testbeds rather than purely virtual environments.Source: 4) Real-world Robot Platforms · p. 12
• Franka Emika is a 7-DoF collaborative arm with joint torque sensing and 1 kHz torque control for precise, contact-rich manipulation.ROS integration broadens its use as a research platform.Source: 4) Real-world Robot Platforms · p. 12
• Unitree Go1 is a cost-effective quadruped with panoramic depth sensing, 1.5 TFLOPS onboard computing, and a maximum speed of 4.7 m/s.These capabilities support locomotion and embodied-AI research.Source: 4) Real-world Robot Platforms · p. 12
• Unitree G1 is a compact humanoid offering up to 43-DoF, 120 N·m knee torque, integrated 3D LiDAR and depth cameras, and a practical low-cost testbed.Its multimodal sensing, onboard compute, ROS support, and swappable batteries support world-model training and evaluation.Source: 4) Real-world Robot Platforms · p. 12

Metrics

• Metrics span three abstraction levels: pixel prediction quality, state-level understanding, and task performance.This progression moves from low-level signal fidelity to high-level goal attainment.Source: B. Metrics · p. 12

Pixel Prediction Quality

• Pixel-level metrics evaluate image fidelity, temporal consistency, perceptual similarity, and generated-video quality.The surveyed measures include FID, FVD, SSIM, PSNR, LPIPS, and VBench.Source: 1) Pixel Generation Quality · pp. 12–13
• Lower FID indicates closer alignment between real and generated image distributions, while FVD additionally evaluates video appearance and dynamics.FID penalizes mean and covariance mismatch; FVD uses motion-aware features and penalizes artifacts such as flickering.Source: 1) Pixel Generation Quality · p. 13
• SSIM compares luminance, contrast, and structure, with values closer to 1 indicating higher similarity.The score is averaged over sliding windows.Source: 1) Pixel Generation Quality · p. 13
• PSNR measures pixel-wise distortion, whereas LPIPS compares pretrained-network features and generally treats lower values as greater perceptual similarity.LPIPS is designed to correlate with human judgments and remain robust to distortions.Source: 1) Pixel Generation Quality · p. 13

State-level Understanding

• State-level understanding evaluates objects, layouts, semantics, geometry, occupancy, and trajectories rather than appearance alone.The metrics include segmentation, detection, occupancy, geometry, and trajectory accuracy.Source: 2) State-level Understanding · p. 13
• Higher mIoU reflects more precise semantic scene understanding through class-averaged segmentation overlap.IoU measures overlap with ground truth while penalizing segmentation errors.Source: 2) State-level Understanding · p. 13
• Higher mAP indicates better instance recognition, localization, and confidence calibration across classes and IoU thresholds.mAP averages per-class Average Precision after confidence-ranked matching to ground-truth instances.Source: 2) State-level Understanding · p. 13
• Displacement errors measure spatial accuracy for keypoints, object centers, and trajectory waypoints, with lower ADE and FDE indicating more accurate localization.ADE averages displacement over steps, whereas FDE measures displacement at the final step.Source: 2) State-level Understanding · pp. 13–14
• Chamfer Distance measures geometric similarity between predicted and ground-truth sets and complements IoU for surfaces, occupancy, BEV, and 3D structures.Its differentiability also permits use as a training loss.Source: 2) State-level Understanding · p. 14

• Task-level metrics assess goal achievement under safety and efficiency constraints in embodied settings.Source: 3) Task Performance · p. 14
• Success Rate measures the fraction of evaluation episodes satisfying a predefined success condition, including collision-free route completion in autonomous driving.Source: 3) Task Performance · p. 14
• Sample Efficiency quantifies the samples required to reach target performance through fixed-budget benchmarks, data-performance curves, or demonstrations.Source: 3) Task Performance · p. 14
• Reward reports cumulative or average return in reinforcement learning, based on discounted timestep rewards.Source: 3) Task Performance · p. 14
• Collision safety is measured by the proportion of episodes with at least one collision or exposure-normalized rates such as collisions per kilometer.Source: 3) Task Performance · p. 14
• For nuScenes video generation, DrivePhysica achieves the best visual fidelity while MiLA achieves the strongest temporal coherence.FID evaluates appearance fidelity, whereas FVD evaluates temporal consistency.Source: A. Pixel Generation · p. 14

• 4D occupancy forecasting predicts 3 s of future scene dynamics from 2 s of past 3D occupancy using mIoU and per-horizon IoU.Source: B. Scene Understanding · p. 14
• DMC evaluates control-relevant dynamics with 64×64×3 observations using Episode Return over 1,000 steps.The theoretical maximum is 1,000 when rt ∈[0, 1].Source: C. Control Tasks · p. 14
• A broadly transferable model across tasks, modalities, and datasets remains an open challenge.Source: C. Control Tasks · p. 15
• RLBench measures manipulation performance with Success Rate, but differing episode budgets, resolutions, and modalities complicate like-for-like comparison.Source: C. Control Tasks · p. 15
• UniAD+DriveWorld achieves the lowest L2, while SSR attains the best collision rate with competitive L2 without extra supervision.Source: C. Control Tasks · p. 15
• Camera-based planning methods surpass privileged-occupancy models, reflecting the growing maturity of end-to-end planning.Source: C. Control Tasks · p. 15

Data & Evaluation

• Unified large-scale datasets remain lacking across navigation, manipulation, and autonomous driving, with multi-agent data especially underrepresented.This fragmentation constrains generalization as model scale grows faster than task-relevant data availability.Source: A. Data & Evaluation · p. 16
• FID and FVD emphasize pixel fidelity while overlooking physical consistency, dynamics, and causality, leaving a gap between video-simulator and embodied-controller evaluation.Existing structured benchmarks remain task-specific and lack cross-domain standards.Source: A. Data & Evaluation · p. 16
• Future benchmarks should be unified, multimodal, and cross-domain, coupling scenes, actions, rewards, multi-agent interaction, and safety-critical events.Evaluation should also assess causal reasoning, physical consistency, and long-horizon error accumulation.Source: A. Data & Evaluation · p. 16

Computational Efficiency

• Real-time deployment is constrained by the inference cost and memory footprint of Transformers and diffusion decoders on onboard robotic platforms.Practical systems therefore often use compact recurrent models and global latent vectors, trading capacity for predictable behavior.Source: B. Computational Efficiency · p. 16
• Quantization, pruning, low-rank and sparse computation, SSMs, and adaptive resource use are proposed to improve fidelity and efficiency jointly.High-capacity video-diffusion components may serve offline generation, while lightweight cores handle online control and safety monitoring.Source: B. Computational Efficiency · p. 16

Modeling Strategy

• World models face a trade-off between compact autoregressive simulation, which accumulates errors, and global prediction, which improves coherence but increases computation and weakens closed-loop interactivity.Hybrid local-autoregressive and global or masked-update designs are proposed to improve long-horizon stability.Source: C. Modeling Strategy · pp. 16–17