Source-linked AI summary
Epona: Autoregressive Diffusion World Model for Autonomous Driving
Kaiwen Zhang, Zhenyu Tang, Xiaotao Hu, Xingang Pan, Xiaoyang Guo, Yuan Liu, Jingwei Huang, Li Yuan, Qian Zhang, Xiao-Xiao Long, Xun Cao, Wei Yin
TL;DR
Existing diffusion world models provide visual fidelity but struggle with flexible-length prediction and trajectory planning, while autoregressive alternatives can degrade visual quality. Epona addresses these gaps with decoupled spatiotemporal modeling, modular trajectory and video prediction, and chain-of-forward training; it achieves a 7.4% FVD improvement over Vista, extends generation beyond 2 minutes, and outperforms strong end-to-end planners on NAVSIM.
Problem
Diffusion world models struggle with flexible-length long-horizon prediction and trajectory planning because they model fixed-length videos jointly rather than localized per-timestep distributions.
Method
Epona uses causal temporal modeling with separate diffusion transformers for future-frame rendering and trajectory generation, trained with a chain-of-forward strategy.
Results
7.4% FVD improvement over Vista on NuScenes accompanies generation extending from 15 seconds to over 2 minutes, while Epona outperforms strong end-to-end planners on NAVSIM.
Takeaways & Limitations
Epona jointly predicts high-fidelity future trajectories and driving scenes and can serve as a real-time motion planner.
Abstract
from arXiv · showhide
Diffusion models have demonstrated exceptional visual quality in video generation, making them promising for autonomous driving world modeling. However, existing video diffusion-based world models struggle with flexible-length, long-horizon predictions and integrating trajectory planning. This is because conventional video diffusion models rely on global joint distribution modeling of fixed-length frame sequences rather than sequentially constructing localized distributions at each timestep. In this work, we propose Epona, an autoregressive diffusion world model that enables localized spatiotemporal distribution modeling through two key innovations: 1) Decoupled spatiotemporal factorization that separates temporal dynamics modeling from fine-grained future world generation, and 2) Modular trajectory and video prediction that seamlessly integrate motion planning with visual modeling in an end-to-end framework. Our architecture enables high-resolution, long-duration generation while introducing a novel chain-of-forward training strategy to address error accumulation in autoregressive loops. Experimental results demonstrate state-of-the-art performance with 7.4\% FVD improvement and minutes longer prediction duration compared to prior works. The learned world model further serves as a real-time motion planner, outperforming strong end-to-end planners on NAVSIM benchmarks. Code will be publicly available at \href{https://github.com/Kevin-thu/Epona/}{https://github.com/Kevin-thu/Epona/}.
1. Introduction
Epona addresses diffusion world models’ limitations in variable-length prediction and trajectory planning with autoregressive diffusion, decoupled spatiotemporal modeling, and modular multimodal generation. It achieves long-duration video prediction, trajectory planning, and strong benchmark performance.
- Diffusion world models struggle with variable-length long-range prediction and trajectory planning because they model fixed-length videos jointly rather than per-timestep local distributions.
- Epona uses causal temporal modeling in compressed latent space while twin diffusion transformers separately render future frames and generate trajectories.
- Epona supports up to 2 minutes of generation and trajectory planning at rates of up to 20 Hz by allowing planning while video prediction is deactivated.
- 7.4% lower FVD than Vista on NuScenes accompanies generation extending from 15 seconds to over 2 minutes, or 600 frames.
- Epona outperforms strong end-to-end planners on NAVSIM without perception inputs and learns traffic knowledge such as stopping at red lights from self-supervised future prediction.
- The model processes historical frames with a multimodal spatiotemporal transformer, then predicts the next frame and an N-frame future pose trajectory through specialized DiTs.
2. Related Work
Prior driving world models face a trade-off: diffusion methods offer visual fidelity but limited temporal flexibility, while autoregressive token-based methods support flexible generation at the cost of quality and planning precision.
- Vision-centric driving world models gained prominence, but earlier methods often lacked planning modules or produced only low-resolution, short-term videos.
- Long Video Generation: Long-term prediction is crucial because it tests whether a world model captures consistent environment dynamics and real-world temporal progression.
- Long Video Generation: Noise rescheduling, overlapped generation, and hierarchical generation extend video length but often produce inconsistencies and abrupt visual changes.
- Long Video Generation: Autoregressive methods naturally support variable-length generation but suffer quality degradation from teacher-forcing domain shift and accumulated sampling errors.
3. Method
Epona reformulates autonomous-driving world modeling as sequential prediction in continuous representations, separating temporal dynamics from fine-grained frame and trajectory generation. Its modular architecture combines historical-context encoding with diffusion-based trajectory and next-frame prediction, while chain-of-forward training addresses autoregressive drift.
- Reformulation of World Model Designs: Unlike fixed-length video diffusion, Epona predicts fine-grained future frames autoregressively in continuous space rather than quantized image tokens.This formulation separates causal temporal modeling from whole-frame visual prediction and supports flexible-length generation.
- Epona: Autoregressive Diffusion World Model: Epona factorizes trajectory planning and visual generation through specialized diffusion transformers conditioned on historical context.TrajDiT models future trajectories, while VisDiT models the conditional distribution of the next camera frame; the next frame can use predicted or externally provided actions.
- Epona: Autoregressive Diffusion World Model: Epona uses a multimodal spatiotemporal transformer to encode historical observations and actions into a compact representation for future prediction.The transformer interleaves multimodal spatial attention with causal temporal attention and uses the last-frame latent embedding for subsequent prediction.
- Chain-of-Forward Training: Chain-of-forward training periodically rolls out self-predicted frames to reduce the teacher-forcing gap and error accumulation during autoregressive inference.The method estimates denoised latents from predicted velocity in one step instead of sampling next-frame latents from pure noise.
- Epona: Autoregressive Diffusion World Model: A temporal-aware DCAE decoder is introduced because framewise decoding with an image autoencoder lacks temporal interactions and can cause flickering.The decoder is designed to enhance interframe consistency while retaining pretrained parameters and minimizing architectural changes.
4. Experiment
Epona is evaluated for video generation, trajectory planning, and architectural components, including long-horizon quality, controllability, and autoregressive robustness. Results show strong visual and planning performance, while ablations identify benefits from shared latent prediction, temporal decoding, and longer conditioning context.
- Implementation Details: Epona is trained from scratch on NuPlan and 700 NuScenes scenes using 512×1024 images and rectified flow objectives for video and trajectory prediction.The model contains 2.5 B parameters across multimodal, video diffusion, and trajectory planning components.
- Video Generation: Predefined pose trajectories control generated future frames, allowing motion paths to be specified for autonomous-driving video synthesis.The qualitative evaluation highlights controllability under extreme driving scenarios.
- Trajectory Planning: Epona achieves the lowest collision rate for a 1-second NuScenes horizon and state-of-the-art overall PDMS on NAVSIM for 4-second trajectory prediction.The NAVSIM evaluation uses 2 seconds of past observations and predicts 4-second future trajectories.
- Ablation Study: Jointly predicting the next scene through a shared latent improves NAVSIM planning performance compared with trajectory-only training.A temporal-aware DCAE decoder reduces flickering and improves smoothness, while increasing conditioned frames improves FVD up to the 10-frame setting.
- Ablation Study: Chain-of-Forward training increasingly improves visual quality and FID as autoregressive sequences become longer, mitigating inference-error accumulation.Without this strategy, visual quality deteriorates rapidly after 10–20 seconds; with it, minute-long videos retain high visual quality.
5. Conclusion
The conclusion presents Epona as an autoregressive diffusion world model that jointly predicts high-fidelity trajectories and driving scenes from historical context. Its decoupled spatiotemporal modeling and asynchronous multimodal generation support high-quality, long-term prediction and real-time trajectory planning.
- 5. Conclusion: Epona jointly predicts high-fidelity future trajectories and driving scenes from historical driving context.The model is designed for autonomous driving world modeling.
- 5. Conclusion: Decoupled spatiotemporal modeling and asynchronous multimodal generation support high-quality, long-term prediction.The conclusion identifies these strategies as central to Epona’s prediction capabilities.
- 5. Conclusion: Epona can serve as a real-time motion planner through trajectory planning.The conclusion frames this capability as part of the model’s potential for next-generation autonomous driving systems.
A. Detailed Architecture of Dual-Single-Stream DiT
The dual- and single-stream DiT blocks form the trajectory and visual diffusion modules. They differ in how condition information and noise are combined before attention processing.
- A. Detailed Architecture of Dual-Single-Stream DiT: TrajDiT and VisDiT use dual-stream and single-stream DiT blocks to construct the trajectory and visual generation modules.The architecture is inspired by recent image and video generation models.
- A. Detailed Architecture of Dual-Single-Stream DiT: Dual-stream DiT processes condition information and noise separately, allowing interaction only within the attention mechanism.This preserves separate streams until attention-based interaction.
- A. Detailed Architecture of Dual-Single-Stream DiT: Single-stream DiT concatenates condition information and noise from the beginning for unified processing.The two stream designs therefore encode conditioning differently before subsequent computation.
B. More Discussions with Related Works
Epona differs from prior multimodal driving world models by generating continuous multi-step trajectories directly through diffusion within the video-generation process. Its visual performance is competitive, but encoder choices create a documented quality–efficiency trade-off.
- B. More Discussions with Related Works: Epona predicts an entire N-step future trajectory in one shot, unlike prior transformer approaches that interleave single-step image and action generation.This design is presented as beneficial for real-time motion planning.
- B. More Discussions with Related Works: Epona generates high-resolution continuous trajectories directly, avoiding tokenized discrete action representations used by existing methods.The paper links continuous prediction to more precise planning and control.
- B. More Discussions with Related Works: Epona records an FVD of 82.83, compared with 94.84 for MagicDriveDiT and 70.06 for InfinityDrive.The reported scores indicate competitive visual-generation performance among concurrent methods.
- B. More Discussions with Related Works: MagicDriveDiT achieves slightly better visual quality, while Epona’s DCAE offers better compression and training efficiency but may introduce additional visual artifacts.The authors identify improving the DCAE component as future work.
- B. More Discussions with Related Works: Unlike scene-synthesis video diffusion methods, Epona combines frame-wise latent autoregression with diffusion to address temporal dynamics and coherence.The comparison distinguishes Epona from token-wise text autoregression paired with image diffusion in Transfusion and JanusFlow.
C. More Long-term Video Generation Results
Epona generates minute-long driving videos while preserving high visual quality and detailed vehicles and buildings. The figure also presents the architecture of its dual-stream and single-stream DiT blocks.
- Epona continuously generates new frames without context drift during minute-long video generation.
- The dual-stream and single-stream DiT blocks use nearly identical architectures for TrajDiT and VisDiT, with action control applied only to VisDiT.
- 140 seconds: Epona generates extended videos while maintaining high visual quality and detailed vehicles and buildings.