PaperlayerView exact revision on arXiv

Source-linked AI summary

GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generation

Zhenya Yang, Zhe Liu, Yuxiang Lu, Liping Hou, Chenxuan Miao, Siyi Peng, Bailan Feng, Xiang Bai, Hengshuang Zhao

arXiv:2512.12751v1cs.CV

TL;DR

Video-only driving world models can lack physical constraints and inherit dataset bias, risking incorrect future predictions. GenieDrive predicts 4D occupancy before generating occupancy-guided multi-view video. It reports improved forecasting mIoU, fast inference with a compact model, and reduced video-generation FVD.

  • Problem

    Video-based driving world models map actions directly to videos without physical modeling or constraints, making predictions vulnerable to driving-video dataset bias.

    Source: §1 · p. 2

  • Method

    GenieDrive predicts future 4D occupancy from historical occupancy and driving controls, then uses occupancy guidance and Normalized Multi-View Attention to generate multi-view videos.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Results

    GenieDrive reports a 7.20% forecasting mIoU improvement, 41FPS with 3.4M parameters, and a 20.7% reduction in FVD for video generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that GenieDrive provides physically plausible feedback from driving actions and performs strongly in occupancy forecasting and driving-video synthesis.

    Source: §5 · p. 9

Abstract

from arXiv · show

Physics-aware driving world model is essential for drive planning, out-of-distribution data synthesis, and closed-loop evaluation. However, existing methods often rely on a single diffusion model to directly map driving actions to videos, which makes learning difficult and leads to physically inconsistent outputs. To overcome these challenges, we propose GenieDrive, a novel framework designed for physics-aware driving video generation. Our approach starts by generating 4D occupancy, which serves as a physics-informed foundation for subsequent video generation. 4D occupancy contains rich physical information, including high-resolution 3D structures and dynamics. To facilitate effective compression of such high-resolution occupancy, we propose a VAE that encodes occupancy into a latent tri-plane representation, reducing the latent size to only 58% of that used in previous methods. We further introduce Mutual Control Attention (MCA) to accurately model the influence of control on occupancy evolution, and we jointly train the VAE and the subsequent prediction module in an end-to-end manner to maximize forecasting accuracy. Together, these designs yield a 7.2% improvement in forecasting mIoU at an inference speed of 41 FPS, while using only 3.47 M parameters. Additionally, a Normalized Multi-View Attention is introduced in the video generation model to generate multi-view driving videos with guidance from our 4D occupancy, significantly improving video quality with a 20.7% reduction in FVD. Experiments demonstrate that GenieDrive enables highly controllable, multi-view consistent, and physics-aware driving video generation.

1. Introduction

GenieDrive addresses physically inconsistent action-conditioned video generation by predicting 4D occupancy before generating occupancy-guided multi-view video. The introduction situates this approach among video-based and occupancy-based driving world models.

1. Introduction

  • Video-based driving models learn from video denoising without explicit physical constraints, risking dataset-driven errors such as straight-ahead outputs after turn commands.Source: §1 · p. 2
  • GenieDrive uses 4D occupancy as an intermediate physical constraint, with a tri-plane VAE and Mutual Control Attention for occupancy forecasting and Normalized Multi-View Attention for video generation.Source: §1 · p. 2, Figure 2 · p. 3
  • GenieDrive reports a 7.20% forecasting mIoU improvement, 41FPS with 3.4M parameters, and a 20.7% reduction in FVD for video generation.Source: §1 · p. 2

2.1. Video-Based Driving World Model

  • Prior video-based driving models range from short, single-view generation to long, high-resolution, and multi-view generation, but can require substantial compute and yield physically inconsistent predictions.Source: §2.1 · p. 2

2.2. Occupancy-Based Driving World Model

  • Occupancy forecasting methods use diffusion with continuous VAE/DiT representations or autoregressive discrete VQ-VAE tokens; GenieDrive combines VAE feature preservation with autoregressive modules.Source: §2.2 · pp. 2–3

2.3. Occupancy-Guided Video Generation

  • Unlike BEV-input translators and six-frame occupancy prediction, GenieDrive predicts occupancy from historical observations and driving actions, supporting sequences up to 241 video frames.Source: §2.3 · p. 3

3. Methods

GenieDrive separates occupancy forecasting from video synthesis: a compact controlled occupancy model predicts future 4D structure, then occupancy-derived guidance supports physics-aware multi-view video generation.

3. Methods

  • GenieDrive first predicts future occupancy, then uses it as physical guidance for generating driving videos.The two-stage process comprises a lightweight occupancy world model and a video generator.Source: §3 · p. 3, §3.2 · p. 4

3.1. Lightweight Occupancy World Model

  • A VAE compresses occupancy into three axis-projected latent planes to reduce redundancy while preserving important features.The planes encode the occupancy feature along the X, Y, and Z axes.Source: §3.1 · p. 3
  • Future occupancy is predicted autoregressively from historical occupancy and driving controls, with MCA modeling their interaction and intermediate transformation supervision.The predicted latent is passed through subsequent spatial-temporal transformer blocks and supervised against future occupancy.Source: §3.1 · p. 4
  • The tri-plane VAE and occupancy predictor are trained end-to-end to reduce misalignment between reconstruction-trained representations and forecasting.The regularization loss remains in the end-to-end objective.Source: §3.1 · p. 4

3.2. Physics-Aware Driving Video Generation

  • Rendered occupancy semantic maps condition a pretrained flow-based video model, while normalized MVA links same-height features across views and stabilizes fine-tuning.The normalization rescales MVA output to the distribution of the pretrained representation.Source: §3.2 · p. 5

4. Experiments

GenieDrive evaluates control-responsive occupancy forecasting and occupancy-guided multi-view video generation. Results cover physical plausibility, forecasting accuracy and efficiency, longer horizons, and video quality.

4.1. Evaluation on Physics-Aware Video Generation

  • Only GenieDrive produces physically plausible videos for all three tested controls—turn left, go straight, and turn right—while supporting multi-view generation.Vista and Epona support single-view generation; Figure 4 illustrates the occupancy-guided generation stages.Source: §4.1 · p. 6, Figure 4 · p. 6

4.2. Evaluation on 4D Occupancy Forecasting

  • 7.2% higher forecasting mIoU and 4% higher IoU than I2-World, with 41 FPS and 3.47M parameters.The comparison uses six future occupancy frames from four past frames.Source: §4.2 · p. 6, Table 1 · p. 7
  • Without additional training, GenieDrive maintains strong occupancy forecasting at 6 s while comparison methods degrade sharply.The authors report its 6 s performance surpasses most methods’ 4 s performance.Source: Table 2 · p. 7, §4.2 · p. 7
  • End-to-end training improves GenieDrive forecasting, but collapses DOME and degrades I2-World; continuous representation also fares better than discrete representation after joint training.The authors attribute DOME’s collapse primarily to diffusion loss favoring a simpler latent space.Source: §4.2 · p. 7, Table 3 · p. 7

4.3. Evaluation on Multi-view Video Generation

  • Across video lengths and metrics, GenieDrive is reported to outperform occupancy-based methods, with mIoU and mAP surpassing MagicDrive-V2.Its L model rolls out to 241 frames (about 20 s); Figure 5 also shows occupancy-space object insertion or removal and the video ablations.Source: Table 4 · p. 8, §4.3 · pp. 8–9, Figure 5 · p. 8

4.2. Evaluation on 4D Occupancy Forecasting

  • Removing Mutual Control Attention causes a significant forecasting drop, particularly at 3 s, while the module models control’s influence on future occupancy.Figure 5 reports grid artifacts and blurry details without normalization, and multi-view inconsistency without Multi-View Attention.Source: §4.2 · p. 8, Figure 5 · p. 8, §4.3 · pp. 8–9

5. Conclusion

GenieDrive uses 4D occupancy as an intermediate representation for controllable, physics-aware driving video generation. Experiments report plausible action feedback and strong occupancy forecasting and video synthesis.

5. Conclusion

  • GenieDrive introduces 4D occupancy as an intermediate representation to produce highly controllable, physics-aware outputs.The conclusion identifies closed-loop evaluation as a potential application for its visually and physically realistic videos.Source: §5 · p. 9

6. Implementation Details of Tri-Plane VAE

The tri-plane VAE downsamples occupancy into a feature volume, decomposes it into three planes, and concatenates them into a unified representation. The resulting latent is smaller than previous methods’ representation while retaining strong reconstruction performance.

6. Implementation Details of Tri-Plane VAE

  • The VAE applies a 3D convolution filter to downsample occupancy O into a feature volume S.The feature volume has dimensions h×w×d×C.Source: §6 · p. 12
  • The feature volume is decomposed into Zxy, Zyz, and Zxz planes, then concatenated as Z ∈ R50×58×64 for Occ3D-NuScenes resolution.The cited occupancy resolution is 200×200×16; the concatenated representation facilitates processing by the subsequent attention module.Source: §6 · p. 12
  • The tri-plane latent uses 58% of the size of previous methods’ representation and achieves 86.15 mIoU and 75.53 IoU reconstruction performance.The paper attributes efficiency and performance to a more 3D-aware latent than a 2D BEV feature.Source: §6 · p. 12

7. Details of 4D Occupancy Forecasting

The occupancy forecasting encoder uses Mutual Control Attention to exchange information between occupancy and control latents. It then fuses a transformed control latent with the occupancy tri-plane before spatial-temporal processing and decoding.

7. Details of 4D Occupancy Forecasting

  • Mutual Control Attention iteratively injects information between the occupancy tri-plane latent and the control latent.A shallow transformation head transforms the control latent for intermediate supervision.Source: §7 · p. 12
  • Cross-attention fuses the tri-plane with the transformed control latent, allowing different tri-plane regions to be influenced to varying degrees.The transformed control latent is produced by the transformation head.Source: §7 · p. 12
  • Spatial self-attention and temporal processing feed the output tri-plane to the VAE decoder to predict the next occupancy timestep.Temporal processing concatenates the previous k tri-planes with the current one and projects the channels back to C.Source: §7 · p. 12

8. Qualitative Comparison on Forecasting

GenieDrive’s qualitative forecasts preserve driving dynamics and scene details that comparison methods often miss. The comparisons also describe an end-to-end training trade-off between forecasting and reconstruction.

8. Qualitative Comparison on Forecasting

  • GenieDrive consistently predicts pedestrians and preserves vehicle dynamics and truck geometry across forecast timesteps, unlike the compared methods.The comparisons describe missed or inaccurate occupancy, deformed or vanishing trucks, and unreasonable trajectories in other methods.Source: §8 · pp. 12–13
  • As end-to-end training proceeds, forecasting performance improves while reconstruction performance decreases.The caption for Table 6 states that forecasting peaks as training epochs increase, while reconstruction falls.Source: Table 6 · p. 13
  • The truck forecast follows a reasonable right turn because prior observations do not sufficiently indicate a straight trajectory.Source: §8 · p. 13

9. End-to-End Training of Occupancy World Model

GenieDrive first trains its prediction module with the tri-plane VAE frozen, then jointly trains both components. End-to-end training improves forecasting details, but forecasting eventually degrades from overfitting and reconstruction performance declines.

9. End-to-End Training of Occupancy World Model

  • End-to-end training improves GenieDrive’s forecasts of pedestrians, roadway features, and vehicle dynamics compared with the variant without it.The qualitative comparison also describes better preservation of roadway consistency and fewer vanishing vehicles.Source: §9 · p. 13
  • Forecasting accuracy peaks at epoch 16 and then degrades mainly due to overfitting, while reconstruction performance decreases during end-to-end training.Source: §9 · p. 13
  • DOME’s forecasting breaks down after end-to-end training, while I2-World gains no accuracy and its prediction accuracy declines.The comparisons do not show improved forecasting for either method.Source: §9 · p. 13

10. Efficiency of Video Generation

GenieDrive uses pretrained video-generation models and a two-stage framework to reduce training resources while supporting multi-view generation. It also reports inference time and VRAM use below the cited MagicDrive-V2 figure for VRAM.

10. Efficiency of Video Generation

  • One week of training on 8 GPUs replaces the 192–1080 hours and 32–128 GPUs reported for Vista, Epona, and MagicDrive-V2.GenieDrive’s week comprises 3 days for the first-stage occupancy world model and 4 days to fine-tune the video model.Source: §10 · p. 13
  • GenieDrive averages 4.36 seconds per frame and uses 11.72 GB of VRAM, compared with MagicDrive-V2’s 39.76 GB VRAM use.The passage says GenieDrive can perform inference on a single GPU.Source: §10 · p. 13
  • Vista and Epona generate single-view videos, whereas MagicDrive-V2 and GenieDrive support six-view generation.Source: §10 · p. 14

11. More Video Generation Results

GenieDrive generates long multi-view videos, edits scenes through occupancy changes, and transfers simulated occupancy into realistic-looking driving videos. The examples describe maintained quality and consistency across these uses.

11.1. Long Video Generation

  • 241-frame (approximately 20-second) sequences retain high generation quality and multi-view coherence after two rollouts in daytime and nighttime examples.GenieDrive-L produces 81-frame videos before rollout.Source: §11.1 · p. 14, Figure 11 · p. 18

11.2. Driving Scenario Editing

  • Editing occupancy enables progressive car removal and truck insertion, with the resulting videos described as natural and spatially and temporally consistent.Source: §11.2 · p. 14, Figure 12 · p. 19

11.3. Sim-to-Real Driving Scenario Generation

  • CARLA simulated occupancy is rendered as realistic multi-view video while preserving driving behavior and scene details such as vegetation and surrounding vehicles.The examples include left turns and overtaking.Source: §11.3 · p. 14, Figure 13 · p. 20
  • The qualitative occupancy comparison highlights physically reasonable scene forecasts and preservation of details such as pedestrians.Source: Figure 8 · p. 15
  • End-to-end training preserves forecast details in GenieDrive, whereas DOME breaks down and I2-World gains no accuracy after such training.The visual comparison also notes loss of scene details for I2-World.Source: Figure 9 · p. 16, Figure 10 · p. 17
Loading 2512.12751v1…