3D and 4D World Modeling: A Survey

Why do native 3D and 4D matter?

• Native 3D/4D signals encode metric geometry, visibility, and motion, supporting consistency, kinematics, occlusion reasoning, and adherence to maps and topology [20] [173].RGB-D, occupancy, and LiDAR provide useful inductive biases for geometry-, causality-, and controllability-aware modeling in safety-critical settings.Source: §1 · pp. 2–3

Position in the broader landscape

• Video, panorama, and mesh world models and object-centric 3D asset generators complement native 3D/4D models, which provide geometry-grounded dynamics and interaction [118] [355] [381].The survey notes that practical systems can combine these capabilities, such as driving mesh or panorama worlds with occupancy- or LiDAR-based dynamics.Source: §1 · p. 3

From conditions to functions

• The survey separates geometry, action, and semantics conditions from four functions: Data Engines, Action Interpreters, Neural Simulators, and Scene Reconstructors.This decoupling enables comparisons across heterogeneous methods on common axes.Source: §1 · p. 3

Contributions

• The survey establishes precise definitions and a hierarchical taxonomy of native 3D/4D methods across VideoGen, OccGen, and LiDARGen.It compares methods along shared dimensions including fidelity, consistency, controllability, and scalability.Source: §1 · p. 3
• It provides extensive coverage of datasets and evaluation protocols tailored to 3D/4D benchmarking.The survey also benchmarks recent methods in this area.Source: §1 · p. 3

Scope

• Its focus on RGB-D, occupancy grids, LiDAR, and spatiotemporal forms addresses gaps in surveys centered on 2D generative models or limited world-modeling contexts [179] [302] [416] [71] [310] [331] [388] [389].The survey aims to serve as a foundational reference for robust and generalizable 3D/4D generative models.Source: §1 · pp. 2–3

Organization

• The survey covers applications in autonomous driving, robotics, and simulation, then discusses open challenges and future research directions.Its later sections also provide preliminaries, a modality-based taxonomy, and a conclusion.Source: §1 · pp. 3–4

2.1 3D and 4D Representations

• Videos emphasize geometric coherence and temporal consistency; occupancy grids encode static or evolving voxel geometry, while timestamped LiDAR scans capture motion and remain robust to appearance conditions.NeRFs encode continuous volumetric fields and Gaussian splatting uses explicit primitives; temporal extensions support 4D reconstruction and simulation.Source: §2.1 · p. 4

2.2 Definition of World Modeling in 3D and 4D

• Generation synthesizes signals without necessarily modeling dynamics, whereas world modeling predicts scene evolution and simulation closes the loop by feeding action-driven states back into later steps.World models are action-conditioned predictive models; JEPA can predict in learned representation space without pixel- or voxel-level reconstruction, a complementary direction to the survey’s focus on explicit video, occupancy, and LiDAR representations.Source: §2.2 · pp. 4–5
• Geometric, action-based, and semantic conditions specify spatial layout, agent behavior, and high-level intent, and can be combined to shape scene realism, controllability, and diversity.Examples include camera pose, depth maps, occupancy volumes, trajectories, control commands, navigation goals, text prompts, and scene graphs.Source: §2.2 · p. 5

2.2.1 Model Definitions

• Generative models create scenes from noise or partial inputs under conditions, while predictive models use historical observations and often actions to forecast scene representations over future steps.Together, these paradigms capture the ability to imagine controllable worlds and anticipate their evolution.Source: §2.2.1 · pp. 5–6

2.2.2 Model Categorizations

• The first two functional types generate diverse scenes from geometric and semantic cues, optionally with actions, or forecast future states from historical observations under specified actions.These types support data augmentation and scenario creation, as well as trajectory planning, behavior prediction, and policy evaluation.Source: §2.2.2 · p. 6
• Neural simulators iteratively generate states in closed-loop agent-environment interaction, while scene reconstructors complete coherent scenes from partial, sparse, or corrupted observations.Their stated applications include interactive simulation and high-fidelity mapping, digital-twin restoration, and post-event analysis.Source: §2.2.2 · pp. 6–7

2.3 Generative Models

• VAEs offer stable training but blurrier samples, GANs can yield high fidelity but face instability and mode collapse, diffusion offers stability and quality but slow iterative sampling, and autoregressive generation is sequentially slow.Autoregressive models nevertheless support flexible sequence modeling and structured 3D scene generation and forecasting.Source: §2.3 · pp. 7–8
• Native 3D/4D adaptations use VAEs as latent tokenizers, GANs for efficient synthesis, diffusion for high-fidelity controllable generation, and autoregressive tokenizations for long-horizon forecasting and closed-loop rollouts.Spatial and temporal autoregressive tokens also integrate naturally with language interfaces for instruction-conditioned generation.Source: §2.3 · p. 8

3 Methods: A Hierarchical Taxonomy

• The taxonomy categorizes 3D and 4D world-modeling approaches by representation modality: VideoGen, OccGen, and LiDARGen.The section discusses video, occupancy, and LiDAR generation models, respectively.Source: §3 · p. 8

3.1 World Modeling from Video Generation

• Video-generation models use multi-view or egocentric sequences to model real-world scenarios, synthesize training data, predict futures, and create interactive simulations.The passage groups existing video-based methods by their primary function but does not name the categories here.Source: §3.1 · p. 8, §3 · p. 8

3.1.1 Data Engines

• For perception augmentation, generative scenes address data scarcity and long-tail challenges through BEV, 3D geometry, semantic, camera-pose, and temporal conditioning.Methods progress from BEVGen [273] and BEVControl [353] to MagicDrive [80], SyntheOcc [164], and PerLDiff [377]; Panacea, DrivingDiffusion, SubjectDrive, MyGo [365], and NoiseController [59] target multi-view consistency and diversity.Source: §3.1.1 · pp. 8–9
• Planning-oriented data mining synthesizes rare and safety-critical scenarios using failure cases, query-driven trajectories, fine-grained tokens, physics-aware refinement, and controllable world editing.Delphi [205] targets failure scenarios; DriveDreamer-2 converts queries into trajectories and HDMaps; Nexus [412], Challenger [342], and OmniSCS [61] generate challenging or hazardous cases.Source: §3.1.1 · p. 9
• Scene editing and style transfer evolve from appearance changes toward precise, temporally consistent, and grounded manipulation across sensors and representations.Examples include world-volume-aware synthesis in WoVoGen, occupancy-based editing in SyntheOcc [164], simulator-conditioned diffusion in SimGen [411], and geometry-guided editing, object insertion, and editable Gaussian Splatting rollouts in GeoDrive [37], DriveWeaver [133], and RealityBridge [323].Source: §3.1.1 · pp. 9–10

3.1.2 Action Interpreters

• Action-guided video generation maps controls to plausible rollouts, progressing toward longer-horizon, multimodal, and multi-agent predictions.GAIA-1 [104] and GAIA-2 [256] combine video, text, and action inputs; later methods address error accumulation, trajectory alignment, pose control, and jointly modeled ego- and surrounding-agent motion [415].Source: §3.1.2 · p. 10
• Forecasting-driven planners evaluate candidate futures before acting, using predicted states, video rollouts, rewards, or jointly learned world evolution and control.Drive-WM [304] scores candidate maneuver rollouts, while later approaches include integrated perception-prediction-planning, latent-action forecasting, and zero-shot driving; the stated goal is improved generalization and safety.Source: §3.1.2 · p. 10

C L 12Hz DiT VG, 3SR

• Challenger [342] and LongDWM [299] are listed as arXiv’25 works, alongside descriptors including 12Hz DiT VG.The subsection heading also includes 3SR.Source: C L 12Hz DiT VG, 3SR · p. 11

T L 10Hz DiT VG

• In the DiT VG grouping, Cosmos-Drive [252] and OmniNWM [158] are listed as arXiv’25 works, with Cosmos-Drive marked P and OmniNWM marked N.Source: T L 10Hz DiT VG · p. 11

T C P L 12Hz DiT VG, E2E

• WorldSplat [417], DriveDreamer-2, and SubjectDrive are listed with 12Hz DiT VG, E2E grouping descriptors.The works are dated arXiv’25, AAAI’25, and AAAI’25, respectively.Source: T C P L 12Hz DiT VG, E2E · p. 11
• DriveScape, DriveDreamer4D, and DrivingSphere [343] are each listed with CVPR’25; the cited passages do not establish a shared 12Hz DiT VG, E2E grouping.Source: T C P L 12Hz DiT VG, E2E · p. 11

P L 12Hz DiT VG

• The cited entries list UniScene at CVPR’25 with P L 12Hz SD VG, GEM [97] at CVPR’25 with L 10Hz SD VG, and MaskGWM [229] at CVPR’25 with T T L 10Hz DiT VG.Source: P L 12Hz DiT VG · p. 11
• The supplied grouping lists DriveArena [356], MagicDrive-V2, and InfiniCube, each marked ICCV’25; their accompanying entries specify 12Hz for the first two and 10Hz for InfiniCube.The accompanying labels include SD, DiT VG, SD VG, and 3SR.Source: P L 12Hz DiT VG · p. 11

T L 5Hz DiT VG, E2E

• The cited entries identify three works: AutoScape [40] with ICCV’25, Glad [329] with ICLR’25, and DualDiff [162] with ICRA’25; no PosePilot entry appears in the cited passages.Source: T L 5Hz DiT VG, E2E · p. 11

C L 2Hz SD,DiT,AR VG

• The supplied entries list six works: OmniDrive [213], OmniDreams [232], Xiaomi-WM [402], GenieDrive [363], GaussianDWM [56], and UniFuture [174], each with a venue/year and letter labels.Source: C L 2Hz SD,DiT,AR VG · p. 11

3.1.3 Neural Simulators

• Generation-driven simulators are evolving from environment rendering toward interactive, feedback-adapting closed-loop systems.Examples include DriveArena [356], DreamForge [211], DrivingSphere [343], Nexus [412], ReSim [352], and real-time systems CausalDrive [346] and Point-as-Skeleton [316].Source: §3.1.3 · p. 12
• Reconstruction-centric simulators build interactive neural environments from driving logs, while generation-enhanced methods improve novel-view handling and reconstruction robustness.Examples include StreetGaussian [348], HUGSIM [400], and UniSim; methods such as ReconDreamer [228] and Stage-1 [295] address dynamic-scene ghosting and controllable 4D synthesis, respectively.Source: §3.1.3 · p. 12
• Long-rollout consistency and object-level control remain difficult, while reconstruction-centric simulators struggle with viewpoint extrapolation and generation-driven simulators must suppress drift.The survey points to hybrid generation–reconstruction, explicit memory and history re-anchoring, and physics- or geometry-grounded conditioning as promising directions.Source: §3.1.3 · pp. 12–13

3.2 World Modeling from Occupancy Generation

• Occupancy-generation models generate, forecast, or simulate 3D/4D occupancy to provide geometry-consistent scaffolds for perception, action-contingent prediction, and large-scale simulation.The passage says these methods are grouped into three categories but does not supply their names.Source: §3.2 · p. 13

3.2.1 Scene Representors

• Occupancy-based scene representors use geometry-consistent grids to improve perception robustness and guide spatially and temporally coherent 3D/4D generation.Methods range from sparse-to-dense semantic reconstruction and coarse-to-fine occupancy denoising to occupancy-guided video, image, and LiDAR synthesis.Source: §3.2.1 · p. 13

3.2.2 Occupancy Forecasters

• Occupancy forecasters predict future scenes from past observations and ego actions, serving both as self-supervised pretraining objectives and as behavior-aware, controllable predictors.Approaches include differentiable point-cloud reconstruction from 4D occupancy, multimodal pretraining, trajectory-conditioned forecasting, and integration with downstream planning.Source: §3.2.2 · pp. 14–15

3.2.3 Autoregressive Simulators

• Autoregressive occupancy simulators generate scalable environments and long-horizon dynamics for interactive, controllable simulation, but dense voxels impose cubic costs and long rollouts accumulate error.Work spans coarse-to-fine open-world generation and dynamic simulation, including OccSora’s 16-second sequences and OccSim’s multi-kilometer horizons; sparse or latent representations and rare-class handling are identified as promising directions.Source: §3.2.3 · pp. 15–16

3.3 World Modeling from LiDAR Generation

• LiDAR-based generation models point clouds to provide geometry-aware, appearance-invariant representations for robust scene understanding and high-fidelity geometric simulation.The section groups these methods into data engines, action forecasters, and autoregressive simulators.Source: §3.3 · p. 16

3.3.1 Data Engines

• LiDAR data engines augment perception data with realistic scans, using uncertainty-aware GANs, diffusion, flow matching, and latent or autoregressive generation.LiDARGen [421] introduced Langevin dynamics for point-cloud generation, while R2DM [225] uses positional encoding and R2Flow [227] accelerates generation with flow matching; latent diffusion compresses data before generation and reconstructs it at original resolution.Source: §3.3.1 · pp. 16–17
• LiDAR completion methods densify sparse or occluded scans, enable scene editing and novel-scene synthesis, and increasingly target unified or faster completion.UltraLiDAR [333] uses discrete voxel representations, while PointDiffusion [2] applies single-step latent diffusion and PatchScene [337] uses patch-based voxel diffusion with spatio-temporal fusion.Source: §3.3.1 · p. 17
• Controllable LiDAR generation supports adverse-weather and object-level scenarios, while multimodal methods generate aligned LiDAR and image data.WeatherGen [320] generates rainy, snowy, and foggy scans validated on detection tasks; OLiDM [345] uses a two-stage object-to-scene pipeline, and X-Drive [332] uses dual branches for aligned point clouds and camera images.Source: §3.3.1 · p. 17
• The entries list Distill-DPO [387], DriveX [264], and LiSTAR [189] under Data Engines.Source: §3.3.1 · p. 18
• The subsection also lists LiDARDraft [308] and two Enc-Dec entries labeled LG, SC and L4G, OP, E2E.Source: §3.3.1 · p. 18

• OLiDM [345] is listed with AAAI’25.Source: T R S 1 Enc-Dec LG, SEG · p. 18
• WeatherGen [320] is listed with CVPR’25.Source: T R S 1 Enc-Dec LG, SEG · p. 18
• SuperPC [65] is listed with CVPR’25.Source: T R S 1 Enc-Dec LG, SEG · p. 18

R P 1 Enc-Dec LG, SC

• OpenDWM is listed with CVPR’25 and the accompanying fields W, 25, A, 3, N, and W.Source: R P 1 Enc-Dec LG, SC · p. 18
• VQ-VAE appears with the fields 3, H, N/A, LG, and L4G.Source: R P 1 Enc-Dec LG, SC · p. 18
• HERMES [408] is listed with ICCV’25 and the accompanying fields W, 26, O, and N.Source: R P 1 Enc-Dec LG, SC · p. 18

T T N/A AR, LLM L4G, E2E

• X-Drive [332] is listed with ICLR’25 and the tag N.Source: T T N/A AR, LLM L4G, E2E · p. 18
• LidarDM [422] is listed with ICRA’25 and the tags 3 W.Source: T T N/A AR, LLM L4G, E2E · p. 18

H N/A LDM L4G, DET

• LiDAR-EDIT [101] is listed with ICRA’25 and the tag N.Source: H N/A LDM L4G, DET · p. 18
• The subsection also contains the method fragment “P 1 Enc-Dec LG, SC, DET.”Source: H N/A LDM L4G, DET · p. 18
• R2Flow [227] is listed with ICRA’25 and the tags 3 N.Source: H N/A LDM L4G, DET · p. 18
• DiffSSC [32] is listed with IROS’25 and the tags S 3.Source: H N/A LDM L4G, DET · p. 18

P S 1 Enc-Dec LG, SC

• X-Drive’s cross-modality epipolar condition module improves point-cloud/image consistency, and X-Drive supports controllable 3D generation from heterogeneous inputs [332].Source: P S 1 Enc-Dec LG, SC · p. 18

3.3.2 Action Forecasters

• Action forecasters predict future LiDAR from past observations and future states, with methods extending generation to multiple modalities and outputs.Copilot4D tokenizes point clouds with a VQ-VAE and uses discrete diffusion for parallel denoising; it predicts 1–3 seconds of future LiDAR from 1–3 seconds of past frames and future ego actions. ViDAR maps historical camera frames to future LiDAR, while BEVWorld, DriveX, and HERMES extend forecasting to other modalities or textual descriptions.Source: §3.3.2 · pp. 18–19

3.3.3 Autoregressive Simulators

• Autoregressive simulators generate controllable, temporally coherent LiDAR sequences, including instruction-driven, ray-centric, sequential, and uncertainty-aware 4D approaches.HoloDrive jointly generates camera images and LiDAR with a depth branch for alignment; La La LiDAR’s two-stage layout framework is extended by LiDARCrafter [322] [194] [173]. Gen-4D-LiDAR [172], LiSTAR [189], DriveLiDAR4D [30], and U4D [340] provide the other listed 4D approaches. LidarDM [422] ray-projects moving objects into generated mesh worlds to synthesize long sequential scans; LaGen [405] targets scalable rollouts, while GEM [318] uses state-space sequence modeling for efficient future-scene generation.Source: §3.3.3 · p. 19
• LiDAR generation still faces sensor-specific fidelity and structural challenges, comparatively underexplored temporal coherence, and brittle cross-modal alignment.Open directions include sensor-faithful representations, weather- and condition-controllable generation, and long-horizon 4D synthesis.Source: §3.3.3 · p. 19

4.1 Datasets

• Real-world, web-scale, and simulated datasets complement one another, trading calibration for scale or providing controllable labels and counterfactual scenarios.Driving datasets support video, occupancy, and LiDAR modeling with modality-specific annotations and conditions; scene-reconstruction collections such as RealEstate10K, ACID, and DL3DV add wide-baseline motion, scene geometry, and view extrapolation.Source: §4.1 · pp. 20–21

4.2.1 Generation Quality

• Evaluation criteria span generation fidelity, consistency, controllability, and human preference, with distinct metrics for perceptual realism, geometry, temporal coherence, and condition adherence.Generation-quality measures include feature-space Fréchet distances and geometric distribution metrics; consistency measures assess multi-view alignment, adjacent-frame coherence, and subject identity, while control metrics depend on the conditioning signal.Source: §4.2 · p. 21, §4.2.1 · p. 22

4.3.1 Benchmarking Video Generation Models

• MaskGWM and GeoDrive reach roughly 4–5 FID and near 60 FVD, while DiST-4D records 22.67 FVD; geometry-aware temporal modeling improves video fidelity.In multi-view generation, DiST-4D, UniScene, and DriveScape achieve FVD below 80; methods combining spatial structure and temporal reasoning are identified as important for scalable video generation.Source: Table 6 · p. 22, §4.3.1 · pp. 25–26

4.2.2 Forecasting Quality

• Forecasting evaluation separates spatial accuracy from temporal plausibility, using image or geometry errors and checks on object dimensions and static-scene consistency.Frames and videos use FID, FVD, and frame-level L1/L2; occupancy uses horizon-specific IoU, point clouds use Chamfer distance and depth-wise errors, and 4D measures test object-size priors and scene rigidity under ego-motion.Source: §4.2.2 · p. 23

4.2.3 Planning-Centric Quality

• Planning evaluation distinguishes non-interactive open-loop scoring from interactive closed-loop behavior, assessing trajectory errors, safety, rule compliance, progress, and comfort.Closed-loop benchmarks include CARLA and nuPlan checks, while World-in-World scores task success, DriveE2E tests real intersections in simulation, and nuPlan-R and ReactSim-Bench target reactive behavior.Source: §4.2.3 · p. 24

4.2.4 Reconstruction-Centric Quality

• Reconstruction and downstream evaluations test novel-view quality and practical task support, using photometric or geometric measures alongside detection, segmentation, occupancy, tracking, and reasoning metrics.When ground-truth views exist, PSNR, SSIM, and LPIPS assess rendering; without them, Novel Trajectory Agent IoU tests plausible foreground behavior. Downstream language-grounded evaluation includes exact-match Top-1 accuracy.Source: §4.2.4 · p. 24, §4.2.5 · p. 24

4.2.6 Toward Physics- and Causal-Consistency Evaluation

• Perceptual scores can miss physical and causal errors, motivating collision, kinematic, temporal, counterfactual, and task-grounded checks and standardized physics-aware benchmarks.The survey notes that visually realistic scenes may violate dynamics or respond inconsistently to actions; early work includes admissibility certification, causal-consistency probes, and VLM-guided occupancy auditing.Source: §4.2.6 · p. 25

4.3.1 Benchmarking Video Generation Models

• Generated scenes improve downstream perception and planning, but remain behind real data; UniMLVG and CogDriving achieve 70.8% road and 32.1% vehicle mIoU.DrivePhysica and Glad reach 35.5 mAP and 43.7 NDS, while DriveArena and DreamForge show reduced planning errors and collisions, including 0.81 PDMS; photorealism alone is insufficient for task utility.Source: Table 7 · p. 23, Table 8 · p. 23, §4.3.1 · p. 26

4.3.2 Benchmarking Occupancy Generation Models

• X-Scene [358] leads occupancy reconstruction at 92.40% mIoU and 85.60% IoU, while I2World [177] offers the best balance across forecasting horizons at 39.73% mIoU.Triplane factorization supports spatial fidelity, with T3Former [334] reaching 76.40% IoU in forecasting; naive generative approaches deteriorate at longer horizons, while explicit geometric priors and effective compression support fidelity.Source: §4.3.2 · pp. 26–27

4.3.3 Benchmarking LiDAR Generation Models

• WeatherGen [320] performs best across LiDAR fidelity metrics, while temporal generation results expose horizon and representation trade-offs.The evaluations use FRD, FPD, JSD, and MMD on SemanticKITTI [17], and TTCE and CTC on nuScenes [28]. Autoregressive methods maintain short-horizon consistency but accumulate errors at longer horizons; vector quantization can improve temporal stability, while BEV and range-based generation trade continuity against raw-point-cloud fidelity.Source: Table 12 · p. 27, §4.3.3 · pp. 27–28, Table 13 · p. 27

4.3.2 Benchmarking Occupancy Generation Models

• Occ-LLM [339] reports 0.28 meter in L2 error, while occupancy world models and structured refinements improve planning accuracy and safety over trajectory-based methods.Drive-OccWorld [359] reaches 0.85 m in L2 error and 0.29% collisions; UniAD+DriveWorld [219] reaches 0.69 meter in L2 error and 0.19% collisions.Source: §4.3.2 · p. 27

4.4.1 Qualitative Analyses of VideoGen Models

• OpenDWM produces the most realistic, consistent, and controllable video results, but video models still struggle with physical plausibility, precise geometric control, and long-tail objects.The analyses identify five needs: realism, consistency, controllability, physical plausibility, and generalization; diverse training datasets support robustness, while rare classes remain difficult to generate convincingly.Source: §4.4.1 · pp. 28–29, Figure 7 · p. 28, Figure 8 · p. 29

4.4.2 Qualitative Analyses of OccGen Models

• Occupancy generation preserves scene semantics and cross-view alignment, but fine geometry and rare layouts or categories remain challenging.The analyses highlight geometric consistency, fine-grained fidelity for small or dynamic objects, and generalization across rare layouts and long-tail classes as key needs.Source: §4.4.2 · pp. 29–30, Figure 9 · p. 30

4.4.3 Qualitative Analyses of LiDARGen Models

• LiDAR generation methods trade off coherent scene geometry, sensor-specific scan structure, realistic point density, and complete object contours.OpenDWM can produce overly regularized or sparse patterns and fragmented objects; LiDARCrafter [173] better preserves scanlines and surface completion but may create occlusion-boundary artifacts or noisy detail; UniScene can oversmooth or artificially fill scans.Source: Figure 10 · p. 31, §4.4.3 · pp. 31–32

5.1 Autonomous Driving

• World models enable controllable traffic simulation, closed-loop driving evaluation, and synthesis of rare or safety-critical scenarios.Volumetric representations support coherent traffic dynamics, while interactive evaluation tests long-horizon driving stacks under distribution shifts and rare events; scenario generation can target occlusions, conflicts, and adverse weather. [294] [374] [173] [361] [194] [413]Source: §5.1 · p. 32

5.2 Robotics

• Robotics uses world models for navigation, object-centric manipulation, and scalable scene generation for simulation.Predictive spatial-temporal cues support long-horizon exploration and obstacle avoidance; geometry and physical-transition models aid grasping and rearrangement, while varied synthetic scenes support robust policy learning and sim-to-real transfer. [135] [274] [404] [16] [207] [407] [67] [274] [404] [407]Source: §5.2 · pp. 32–33

5.3 Video Games & XR

• In games and XR, world models generate virtual worlds, render evolving scenes, and adapt environments to player actions.Coherent procedural content can use maps, player states, or prompts; immersive rendering requires viewpoint-consistent geometry and efficient high-frame-rate pipelines, while adaptation can change terrain, layouts, or agents while preserving style, physics, and narrative coherence. [106] [163] [106] [368] [84] [173]Source: §5.3 · p. 33

5.4 Digital Twins

• Digital twins use world models for city-scale reconstruction, event replay and forecasting, and interactive scene editing.Multimodal sensing captures infrastructure and activity, while streaming and 4D compression support metropolitan-scale deployment; replay and what-if simulation aid incident analysis and intervention assessment, and editing supports planning and visualization. [20] [178] [330] [54] [58] [54] [84]Source: §5.4 · pp. 33–34

5.5 Other Emerging Applications

• Emerging applications span scientific and environmental modeling, healthcare and biomechanics, industrial simulation, and disaster response.Examples include climate and hazard forecasting, anatomy and tissue modeling for clinical planning, rehabilitation motion prediction, virtual prototyping, and simulations of emergencies and hazardous conditions.Source: §5.5 · p. 34

6.1 Standardized Benchmarking & Evaluations

• Standardized benchmarks are needed because inconsistent datasets and metrics hinder comparison; emerging frameworks broaden evaluation beyond appearance toward fidelity, consistency, controllability, physics, and behavior.4DWorldBench [204] combines fidelity, consistency, and controllability, while WorldLens [150] evaluates multiple aspects and finds no approach dominates across pixel quality, 4D geometry, closed-loop behavior, and human realism. Future protocols should cover closed-loop and real-world scenarios, varying traffic, weather, and urban layouts; examples include World-in-World [375], DriveE2E [367], nuPlan-R [240], ReactSim-Bench [386], and DrivingGen [410], with simulator verdicts validated before use [234].Source: §6.1 · pp. 34–35

6.2 High-Fidelity & Long-Horizon Generation

• Long-horizon generation remains unstable because small prediction errors compound as autoregressive inputs drift from training distributions, degrading behavior and scene consistency.Mitigations include scheduled sampling, rollout or noise augmentation, diffusion-forcing and chain-of-forward training [379], memory or history re-anchoring [94], and self-consistency objectives; future methods should explicitly penalize long-term divergence. PerpetualWonder [373], AutoScape [40], and OccSim [191] demonstrate efforts across video and geometric representations.Source: §6.2 · p. 35

6.3 From Pseudo-4D to Continuous 4D Dynamics

• Stacking discrete 3D frames creates pseudo-4D that can flicker, drift geometrically, or move implausibly; continuous-time dynamics and physical constraints could enable coherent trajectories.A seriality gap in video diffusion, where frames are produced in parallel without sequential dynamics, contributes to the problem [36]. Early directions include ray-centric 4D LiDAR sequences [189], trajectory fields for any-view, any-time geometry [134], and occupancy-guided video generation [363].Source: §6.3 · p. 35

6.4 Physical Fidelity, Controllability & Generalizability

• Driving world models remain limited in physical realism, fine-grained controllability, and generalization, with implausible events, restricted editing, and failures in novel environments and rare-object scenarios.Physics-aware generation [414] and causality-aware models [47] [346] target realism; editable driving worlds [61], semantic-occupancy scenes [12], and point-conditioned insertion [133] broaden control. Long-tail scenario generation [187] may improve coverage, and overcoming the limitations jointly is necessary for faithful, controllable, generalizable models.Source: §6.4 · pp. 35–36

6.5 Computational Efficiency & Real-Time Performance

• Heavy architectures and multi-step sampling impose latency and memory costs, while sparse computation and inference acceleration offer paths to scalable, responsive generation.INSPATIO-WORLD [126] simulates spatiotemporal 4D at interactive rates, alongside real-time closed-loop simulators [232] and streaming interactive models with long-horizon memory [307]; sparse, dynamically queried 4D occupancy models [51] reduce compute without sacrificing coverage.Source: §6.5 · p. 36

6.6 Cross-Modal Generation Coherence

• Cross-modal misalignment can make generated imagery conflict with 3D structure, undermining perception and planning; shared latent representations and strict consistency are key research directions.Joint reconstruction-and-generation models [402] and Gaussian-centric driving models [56] couple scene understanding with multimodal generation, while fine-grained spatial alignment and temporal synchronization remain crucial.Source: §6.6 · p. 36

7 Conclusion

• The survey presents a systematic review that clarifies definitions, organizes VideoGen, OccGen, and LiDARGen in a hierarchical taxonomy, and summarizes datasets, evaluations, and applications.It aims to provide a coherent foundation and roadmap for robust, interpretable, and generalizable world models for embodied AI.Source: §7 · p. 36
• Native 3D and 4D representations shift modeling beyond visual realism toward plausibility, controllability, and physical consistency, supporting data engines, action interpreters, neural simulators, and scene reconstructors.The survey frames these capabilities as roles for geometry-grounded world models.Source: §7 · p. 36
• The field still faces challenges in scaling to real-world complexity, aligning multimodal signals, and standardizing evaluation of controllability, safety, and generalization.The survey emphasizes open benchmarks, reproducible codebases, and large-scale 3D/4D datasets as community needs for comparability.Source: §7 · p. 36
• Promising directions include unifying generative and predictive paradigms, integrating language and reasoning, and advancing simulation and digital twin ecosystems.The survey presents these as avenues for advancing world models.Source: §7 · p. 36
• Table 14 summarizes evaluation metrics for generation, forecasting, planning, reconstruction, and downstream tasks across VideoGen, OccGen, and LiDARGen in 2D, 3D, and 4D.Its scope includes metrics for both quality assessment and downstream evaluation.Source: Table 14 · p. 37

Downstream Evaluation - Segmentation

• Segmentation uses average Intersection-over-Union across semantic classes and class-wise BEV IoU for freespace, lane, and dynamic agents against HD maps [243].These metrics measure segmentation accuracy and scene consistency, respectively.Source: §7 · p. 41

Downstream Evaluation - Tracking

• Tracking evaluation includes a composite score for false positives, false negatives, and identity switches, plus mean IoU of matched predictions and ground truth across frames.The composite score reflects overall tracking reliability; mean IoU quantifies spatial localization precision.Source: §7 · p. 41
• 3D MOTA averages tracking accuracy across recall thresholds, while 3D MOTP averages localization precision over thresholds using IoU or center distance [313].Averaging over recall thresholds reduces sensitivity to threshold choices and reflects localization robustness.Source: §7 · p. 41

Downstream Evaluation - Occupancy Prediction

• Occupancy evaluation uses IoU between predicted and labeled voxel occupancies, while voxelized panoptic quality combines semantic segmentation accuracy with instance detection recall [269].Occupancy IoU can be computed at class or scene granularity.Source: §7 · p. 41

Downstream Evaluation - VQA

• Autonomous-driving VQA uses top-1 exact-match accuracy across question categories to measure answer reasoning reliability.The metric evaluates whether predicted answers exactly match the ground truth.Source: §7 · p. 41