PaperlayerView exact revision on arXiv

Source-linked AI summary

3D and 4D World Modeling: A Survey

Lingdong Kong, Yu Yang, Jianbiao Mei, Youquan Liu, Ao Liang, Dekai Zhu, Dongyue Lu, Wei Yin, Xiaotao Hu, Mingkai Jia, Junyuan Deng, Kaiwen Zhang, Yang Wu, Tianyi Yan, Shenyuan Gao, Song Wang, Linfeng Li, Liang Pan, Yong Liu, Jianke Zhu, Wei Tsang Ooi, Steven C. H. Hoi, Ziwei Liu

arXiv:2509.07996v4cs.CVcs.RO

TL;DR

World modeling lacks consistent definitions and comprehensive coverage of native 3D/4D modalities, despite their importance for geometry-grounded dynamic environments. This survey establishes definitions, organizes VideoGen, OccGen, and LiDARGen methods, and synthesizes datasets, metrics, applications, challenges, and directions. It also identifies practical trade-offs among generative paradigms and reports improving benchmark performance in video-based models.

  • Problem

    Existing world-modeling literature emphasizes 2D data, uses inconsistent definitions, and lacks a unified framework for native 3D/4D approaches.

  • Method

    The survey defines 3D/4D world modeling, introduces a hierarchical VideoGen–OccGen–LiDARGen taxonomy, and synthesizes datasets, evaluation protocols, applications, challenges, and directions.

  • Results

    Recent video-based world models reach FID around 4–5 and FVD near 60, while downstream evaluations report detection up to 43.7 NDS and segmentation up to 70.8% road mIoU.

  • Takeaways & Limitations

    Native 3D/4D representations provide geometry and physical grounding for modeling dynamic environments beyond observation-space generation.

  • Takeaways & Limitations

    Autoregressive models support flexible sequence modeling but generate samples slowly because they produce them sequentially.

Abstract

from arXiv · show

World modeling has become a cornerstone in AI research, enabling agents to understand, represent, and predict the dynamic environments they inhabit. While prior work largely emphasizes generative methods for 2D image and video data, they overlook the rapidly growing body of work that leverages native 3D and 4D representations such as RGB-D imagery, occupancy grids, and LiDAR point clouds for large-scale scene modeling. At the same time, the absence of a standardized definition and taxonomy for "world models" has led to fragmented and sometimes inconsistent claims in the literature. This survey addresses these gaps by presenting the first comprehensive review explicitly dedicated to 3D and 4D world modeling and generation. We establish precise definitions, introduce a structured taxonomy spanning video-based (VideoGen), occupancy-based (OccGen), and LiDAR-based (LiDARGen) approaches, and systematically summarize datasets and evaluation metrics tailored to 3D/4D settings. We further discuss practical applications, identify open challenges, and highlight promising research directions, aiming to provide a coherent and foundational reference for advancing the field. A systematic summary of existing literature is available at https://github.com/worldbench/awesome-3d-4d-world-models

1 Introduction

This survey addresses fragmented definitions and limited 3D/4D coverage in world modeling by organizing native geometric and spatiotemporal approaches into a unified framework. It defines key concepts, proposes a modality- and function-based taxonomy, and summarizes datasets, evaluations, applications, challenges, and research directions.

  • Motivation: Native 3D/4D representations include RGB-D imagery, occupancy grids, LiDAR point clouds, and sequential forms that capture temporal dynamics.These modalities provide explicit geometry and physical grounding for embodied and safety-critical systems.
  • Motivation: World modeling literature uses inconsistent definitions and remains fragmented, while existing surveys largely emphasize 2D or vision-only modalities.This leaves native 3D and 4D challenges and opportunities underexplored.
  • Taxonomy: Its taxonomy separates geometry, action, and semantics as conditions from four functional roles: Data Engines, Action Interpreters, Neural Simulators, and Scene Reconstructors.This decoupling supports comparison across fidelity, consistency, controllability, and scalability.
  • Contributions: The survey establishes definitions, categorizes VideoGen, OccGen, and LiDARGen methods, and covers datasets and evaluation protocols tailored to 3D/4D scenarios.It also reviews applications, challenges, and future research directions.
  • Scope: The survey focuses on native 3D/4D world modeling while distinguishing complementary video, panorama, mesh, and object-centric asset-generation research.It also notes practical systems that combine assets, 3D priors, and occupancy- or LiDAR-based dynamics.

2 Preliminaries

The preliminaries define 3D/4D world modeling through scene representations, conditioning signals, and complementary generative and predictive paradigms. They distinguish generation, forecasting, simulation, and reconstruction by their inputs, outputs, temporal roles, and interaction settings.

  • 3D/4D representations: The survey organizes core scene representations by how they encode spatial geometry, temporal dynamics, and semantic context.These representations serve as inputs, outputs, or intermediate states for generation and prediction.
  • 3D/4D representations: Video-based modeling emphasizes geometric coherence and temporal consistency for physically plausible simulation and accurate forecasting.Video streams are represented across time, height, width, and channels.
  • 3D/4D representations: Occupancy grids represent occupied voxels and extend to 4D by recording scene evolution over time, enforcing spatial constraints for physics-consistent generation.The survey distinguishes occupancy as a generation or forecasting target from occupancy used as a geometric conditioning signal.
  • World modeling definitions: The survey separates 3D/4D generation from world modeling: generation synthesizes signals, whereas world modeling predicts scene evolution from observations and actions.World models may predict in a learned representation space rather than reconstructing pixels or voxels.
  • Conditions: Geometric, action-based, and semantic conditions constrain scene structure, agent behavior, and high-level intent, independently or in combination.These conditions shape realism, controllability, and diversity in generated or forecasted scenes.
  • Functional types: World models span generative synthesis, predictive forecasting, closed-loop neural simulation, and scene reconstruction as four complementary functional types.Generative models create scenes; action interpreters forecast future states; neural simulators iterate policy-conditioned interactions; scene reconstructors recover scenes from observations.
  • Generative paradigms: VAEs provide stable, interpretable latent spaces but may produce blurrier samples than other generative paradigms.Their role includes compressing representations into latent codes for downstream generation.

3 Methods: A Hierarchical Taxonomy

The survey organizes 3D/4D world modeling by representation modality into VideoGen, OccGen, and LiDARGen, while VideoGen methods are grouped by primary function.

  • The taxonomy categorizes approaches by representation modality as VideoGen, OccGen, and LiDARGen.
  • VideoGen models generate multi-view or egocentric sequences for training-data synthesis, future-outcome prediction, and interactive simulation.
  • VideoGen methods are grouped into data engines, action interpreters, and neural simulators according to their primary function.

Data Engines,

Data engines generate controllable driving scenes for perception, planning, and simulation, spanning augmentation, safety-critical scenario mining, and scene editing.

  • Generative 3D data engines create diverse and controllable driving scenes for perception, planning, and simulation.
  • Perception-oriented synthesis addresses real-world data scarcity and long-tail challenges through BEV-guided, spatially consistent scene generation.
  • Planning-oriented data mining synthesizes rare or failure-like safety-critical scenarios to improve sample efficiency and planning performance.
  • Scene editing methods modify weather, lighting, objects, trajectories, and geometry, increasingly using occupancy, world volumes, and explicit 3D conditions.

C L 12Hz DiT VG, 3SR

The surveyed VideoGen literature includes increasingly controllable, geometry-aware, and temporally stable models for video generation, reconstruction, and planning-related tasks.

  • Representative models target trajectory alignment, temporal stability, pose controllability, depth-free guidance, and long-horizon generation.

C L 2Hz SD,DiT,AR VG

Neural simulators support closed-loop planning and interaction through generation-driven or reconstruction-centric scene modeling, but both families face distinct generalization and consistency limits.

  • Closed-loop simulators create virtual worlds for planning, decision-making, and interaction through generation-driven and reconstruction-centric approaches.
  • Generation-driven simulation uses conditional generative frameworks to create interactive, high-fidelity environments.
  • Reconstruction-centric simulation converts driving logs into interactive neural environments using techniques such as NeRF and 3D Gaussian Splatting.
  • Long rollouts still suffer from flicker, geometric drift, and appearance or identity changes caused by accumulated errors.
  • Reconstruction-centric simulators retain metric geometry but struggle with wide viewpoint extrapolation, whereas generation-driven simulators are flexible but must suppress drift.

Scene Representors,

OccGen models use occupancy grids as geometry-consistent intermediates for structured 3D scene representation, perception robustness, forecasting, and simulation. They are organized into scene representors, occupancy forecasters, and autoregressive simulators.

  • Scene Representors: Occupancy-based generation treats occupancy grids as geometry-consistent intermediates for structured 3D scene representation and downstream tasks.The paradigm supports perception robustness and 3D scene generation.
  • Taxonomy: OccGen methods are categorized by functionality into scene representors, occupancy forecasters, and autoregressive simulators.Figure 5 presents this three-way categorization.
  • Occupancy Forecasters: Occupancy forecasting predicts future occupancy from past observations and ego actions for self-supervised pretraining and behavior-aware, controllable prediction.Representative methods include joint ego-motion and environment modeling and trajectory-conditioned 4D occupancy generation.
  • Autoregressive Simulators: Autoregressive OccGen simulators generate large-scale, temporally coherent 4D occupancy for perception, planning, decision-making, and interactive simulation.Research emphasizes scalable unbounded environments and long-horizon controllable dynamics.
  • Summary & Insights: Dense 3D/4D voxels impose cubic memory and compute costs, while class imbalance, long-tail geometry, and autoregressive error accumulation limit detail, resolution, and horizon.Suggested directions include sparse or latent representations, rare-class handling, and tighter coupling with downstream planning.

Data Engines,

LiDARGen data engines generate controllable point clouds to address limited training data, improve perception, complete scenes, model rare conditions, and support multimodal synthesis. The literature progresses from GAN and range-image methods toward diffusion, latent diffusion, and autoregressive approaches.

  • Data Engines: LiDAR-based data engines mitigate scarce, costly, and difficult-to-annotate training data by generating diverse and controllable point clouds.They support perception robustness, geometrically accurate scene completion, and rare or cross-modal scenario synthesis.
  • Perception Data Augmentation: Early LiDAR generators model uncertainty and spatial structure, while LiDARGen introduced Langevin dynamics and helped motivate subsequent diffusion-based methods.DUSty disentangles depth from measurement uncertainty, and LiDARGen outperformed GANs and VAEs according to the survey passage.
  • Perception Data Augmentation: Latent diffusion methods compress raw LiDAR data with pretrained VAEs before diffusion, improving generation speed while preserving quality.LiDARGRIT discretizes latent space with VQ-VAE and uses an autoregressive transformer, adding raydrop estimation to improve raydrop-noise modeling.
  • Scene Completion: Scene-completion methods reconstruct dense, coherent geometry from sparse or occluded scans using representations such as discrete voxels and DDPM denoising.UltraLiDAR uses VQ-VAE for controllable sparse-to-dense completion, while LiDiff and DiffSSC reposition duplicated points through denoising.
  • Rare Condition Modeling: Controllable LiDAR generation covers adverse weather, object-level and scene-level fidelity, and aligned image–point-cloud synthesis.Text2LiDAR uses textual conditioning, WeatherGen targets rainy, snowy, and foggy conditions, and X-Drive jointly generates aligned LiDAR and camera images.

T R S 1 Enc-Dec LG, SEG

The supplied table fragments identify LiDARGen entries associated with LiDAR generation, detection, and scene completion, including WeatherGen and SuperPC. They provide model and task labels but no comparative performance outcomes.

  • Data Engines: WeatherGen is listed with LiDAR generation, detection, and scene-completion tasks.The table fragment associates WeatherGen with CVPR’25 and an encoder-decoder architecture.
  • Data Engines: SuperPC is listed as an encoder-decoder model for LiDAR generation and detection.The supplied fragment identifies SuperPC as a CVPR’25 entry.
  • Model Listings: The table fragments also list OpenDWM, HERMES, X-Drive, and LidarDM among LiDARGen-related entries.The fragments do not provide enough complete field values to infer a unified task or performance comparison for all entries.

H N/A LDM L4G, DET

LiDARGen action forecasters and autoregressive simulators extend point-cloud modeling into future-state prediction and temporally coherent simulation. They use tokenization, diffusion, multimodal generation, mesh-based worlds, and sequential LiDAR synthesis, while retaining sensor-specific challenges.

  • Action Forecasters: Copilot4D predicts the next 1–3 seconds of LiDAR frames from 1–3 seconds of history and future ego actions.It tokenizes point clouds with VQ-VAE and uses a discrete diffusion formulation for parallel denoising and decoding.
  • Action Forecasters: ViDAR predicts future LiDAR frames from historical camera frames and supports pre-training for perception, prediction, and planning.This connects camera-based history with LiDAR-based future-state prediction.
  • Multi-Modal Action Forecasters: BEVWorld and DriveX extend action forecasting to multimodal outputs involving images, point clouds, and semantic maps.DriveX separates spatial world-representation learning from latent future decoding.
  • Autoregressive Simulators: LiDAR autoregressive simulators generate temporally coherent sequences for realistic and interactive simulation supporting perception, planning, and decision-making.The approaches include sequential autoregressive point-cloud generation and mesh-based scene-scale simulation.
  • Summary & Insights: LiDARGen remains constrained by sparse, anisotropically sampled scans, ray-drop and intensity modeling, scanline preservation, and limited temporal coherence.These challenges arise when methods use BEV or voxel abstractions and extend generation to 4D sequences.

4 Datasets & Evaluations

The survey consolidates holistic evaluation protocols and quantitative findings across video, occupancy, and LiDAR world models. Results favor geometry-, temporal-, and behavior-aware representations, while cross-paper comparisons remain indicative because evaluation settings differ.

  • Holistic Benchmarking: WorldLens evaluates driving world models across five aspects and 24 standardized dimensions, combining pixel quality, 4D geometry, closed-loop behavior, and human perceptual alignment.Its evaluation pairs the 26,808-entry WorldLens-26K preference dataset with the explainable WorldLens-Agent evaluator.
  • Holistic Benchmarking: No evaluated approach dominates all axes: texture-rich generators violate geometry, geometry-aware models lack behavioral fidelity, and the strongest systems score only 2–3 out of 10 on human realism.This exposes a persistent gap between visual realism and realistic behavior.
  • Generation Fidelity: DiST-4D reaches FVD 22.67 in multi-view generation, while UniScene and DriveScape also achieve FVD below 80 through stronger geometric and spatio-temporal alignment.Earlier multi-view models generally reduce FID but remain above FVD 120, indicating temporal instability.
  • Downstream Evaluations: Explicit geometry, temporal consistency, and motion dynamics improve downstream detection, segmentation, and planning, whereas photorealistic generation alone remains insufficient.Synthetic scenes still trail real nuScenes data, although systems such as DriveArena and DreamForge support preliminary closed-loop driving.
  • Occupancy Reconstruction Quality: X-Scene achieves 92.40% mIoU and 85.60% IoU for occupancy reconstruction, with triplane factorization providing the largest reported gains.The survey attributes these gains to geometric consistency and finer spatial detail rather than simply increasing latent dimensionality.
  • 4D Occupancy Forecasting Quality: I2World reaches 39.73% mIoU for 4D occupancy forecasting, while T3Former reaches 76.40% IoU, showing complementary strengths in temporal balance and spatial coherence.The comparisons emphasize structured priors, triplane factorization, and temporal modeling for longer-horizon forecasting.
  • LiDAR Generation Quality: LiDAR generation requires balancing semantic controllability with distributional realism, as Text2LiDAR’s textual conditioning increases FRD despite its strong prompts.Modality choices also trade smoother temporal continuity against fidelity to native point-cloud patterns.
  • Open Challenges: Long-tail imbalance causes rare and small objects to exhibit unrealistic shapes, distorted geometry, or omission.Pedestrians, cyclists, and traffic signs remain especially difficult to generate convincingly.

4.4.3 Qualitative Analyses of LiDARGen Models

Qualitative LiDAR analyses compare global scan structure, point sparsity, and object completeness across voxel-, range-, and volumetric-generation paradigms. No representation is uniformly superior across these attributes.

  • Global Patterns: OpenDWM emphasizes coherent geometry but produces overly regularized patterns, whereas LiDARCrafter preserves sharper native scanline rings with some occlusion artifacts.The comparison concerns global patterns in representative LiDAR generation paradigms.
  • Point Cloud Sparsity: LiDARCrafter maintains more uniform angular density, while OpenDWM becomes overly sparse at long ranges and UniScene can artificially fill regions.These differences reflect the trade-off between realistic density distributions and globally complete coverage.
  • Object Completeness: LiDARCrafter provides better surface completion than OpenDWM, while UniScene reconstructs volumetrically plausible objects but lacks sharp real-scan boundaries.OpenDWM frequently underrepresents vehicle contours, producing fragmented or partial shapes.
  • Takeaways: LiDAR generation must jointly preserve coherent scene geometry, sensor-specific density patterns, and complete dynamic-agent surfaces.The survey identifies global patterns, point sparsity, and object completeness as the three attributes requiring balance.

5 Applications

3D and 4D world models support applications spanning autonomous driving, robotics, games and XR, and digital twins. Their central practical value is controllable, spatially grounded simulation and interaction across dynamic environments.

  • Autonomous Driving: Autonomous-driving world models support simulation, evaluation, and scenario synthesis in controllable, interactive, safety-critical environments.They enable traffic simulation, closed-loop driving evaluation, and targeted generation of rare or safety-critical scenes.
  • Autonomous Driving: Closed-loop driving setups couple generative models with agents to test perception, planning, and control under distribution shifts, rare events, and recovery scenarios.The environments adapt to agent actions by jointly modeling ego behavior and surrounding traffic dynamics.
  • Robotics: Robotic world models provide spatial-temporal grounding and scalable synthetic environments for navigation, manipulation, and robust policy learning.Applications include forecasting dynamic layouts, anticipating contact dynamics, and generating varied scenes for sim-to-real transfer.
  • Video Games & XR: Gaming and XR systems use world models for procedural content generation, immersive rendering, and environments that adapt geometry, layout, and agent behavior to user actions.Maintaining geometric, temporal, semantic, and physical coherence supports believable evolution and interactive comfort.
  • Digital Twins: Digital twins enable city-scale reconstruction, event replay, forecasting, and interactive editing for planning, analysis, and simulation.They integrate multimodal sensing and support traffic monitoring, infrastructure planning, disaster response, and what-if intervention studies.

6 Challenges & Future Directions

The survey identifies standardized evaluation, long-horizon fidelity, continuous 4D dynamics, physical realism, controllability, generalization, efficiency, and cross-modal coherence as central challenges for 3D/4D world models.

  • 6.1 Standardized Benchmarking & Evaluations: Standardized benchmarks should jointly evaluate physical plausibility, temporal consistency, and controllability across diverse realistic settings.Different datasets and ad hoc metrics currently make meaningful comparison difficult; emerging protocols combine fidelity, consistency, and controllability while emphasizing closed-loop behavior.
  • 6.2 High-Fidelity & Long-Horizon Generation: Long-horizon rollouts accumulate errors because closed-loop inference conditions on model outputs that differ from the training distribution.Scheduled sampling, rollout augmentation, diffusion-forcing, memory mechanisms, and self-consistency objectives are proposed, but robust extended stability remains open.
  • 6.3 Continuous 4D Dynamics: Pseudo-4D frame stacking can cause flicker, geometric drift, and physically implausible motion because it does not guarantee continuous dynamical evolution.Continuous-time dynamics, scene-flow-aware representations, and explicit physical or kinematic constraints are suggested for coherent 4D trajectories.
  • 6.4 Physical Fidelity, Controllability & Generalizability: Current driving world models remain limited by physical realism, fine-grained controllability, and generalization to new environments and rare objects.Reported issues include implausible collisions, coarse environmental editing, and overfitting that limits real-world applicability.
  • 6.5 Efficiency and Real-Time Responsiveness: Heavy architectures and multi-step sampling undermine large-scale generation and simulation through latency and memory overhead.Sparse computation, inference acceleration, interactive-rate simulation, and dynamically queried occupancy models are presented as routes toward scalable responsiveness.
  • 6.6 Cross-Modal Coherence: Cross-modal misalignment can make generated imagery conflict with underlying 3D structure, undermining downstream perception and planning.Integrated architectures with shared latent representations aim to keep visual, geometric, and semantic outputs consistent.

7 Conclusion

The survey concludes that 3D/4D world modeling benefits from geometry-grounded representations, organized taxonomies, and standardized evaluations across modalities and tasks. It also identifies scaling, multimodal alignment, and evaluation for controllability, safety, and generalization as remaining priorities.

  • 7 Conclusion: The survey provides a systematic review of 3D/4D world modeling, with definitions, a VideoGen–OccGen–LiDARGen taxonomy, and summaries of datasets, evaluations, and applications.Its scope spans generation, forecasting, planning, reconstruction, and downstream tasks across 2D, 3D, and 4D settings.
  • 7 Conclusion: Native 3D/4D representations support geometry-grounded modeling aimed at plausibility, controllability, and physical consistency.The survey frames these methods as data engines, action interpreters, neural simulators, and scene reconstructors.
  • 7 Conclusion: Future progress requires scaling to real-world complexity, aligning multimodal signals, and standardizing evaluation for controllability, safety, and generalization.The conclusion also points to unified generative-predictive paradigms, language and reasoning integration, simulation ecosystems, open benchmarks, reproducible codebases, and large-scale datasets.
  • 7 Conclusion: Evaluation spans perceptual, geometric, temporal, semantic, planning, and closed-loop metrics across VideoGen, OccGen, and LiDARGen models.Examples include keypoint alignment, depth consistency, temporal transformation error, Chamfer-based stability, collision rate, and navigation success rate.
Loading 2509.07996v4…