Source-linked AI summary
DreamerAD: Efficient Reinforcement Learning via Latent World Model for Autonomous Driving
Pengxuan Yang, Yupeng Zheng, Deheng Qian, Zebin Xing, Qichao Zhang, Linbo Wang, Yichen Zhang, Shaoyu Guo, Zhongpu Xia, Qiang Chen, Junyu Han, Lingyun Xu, Yifeng Pan, Dongbin Zhao
TL;DR
Autonomous-driving RL is costly and unsafe on real-world data, while pixel-level diffusion world models are too slow for high-frequency interaction. DreamerAD performs RL in an interpretable latent world model using efficient sampling, dense latent rewards, and constrained exploration, achieving state-of-the-art results on NavSim.
Problem
Real-world RL training incurs prohibitive costs and safety risks, while pixel-level diffusion world models suffer from 100-step sampling latency and limited driving-relevant understanding.
Method
DreamerAD performs latent-space RL with shortcut-forced single-step world-model rollouts, autoregressive dense rewards, and Gaussian-weighted sampling from physically plausible trajectory vocabularies.
Results
87.7 EPDMS on NavSim v2 establishes DreamerAD as a new state of the art for closed-loop planning.
Takeaways & Limitations
Latent imagination training can support efficient, visually interpretable reinforcement learning for safe autonomous-driving behavior without real-world trial and error.
Takeaways & Limitations
Existing world models remain vulnerable to hallucinations under out-of-distribution actions, motivating constrained exploration in DreamerAD.
Abstract
from arXiv · showhide
We introduce DreamerAD, the first latent world model framework that enables efficient reinforcement learning for autonomous driving by compressing diffusion sampling from 100 steps to 1 - achieving 80x speedup while maintaining visual interpretability. Training RL policies on real-world driving data incurs prohibitive costs and safety risks. While existing pixel-level diffusion world models enable safe imagination-based training, they suffer from multi-step diffusion inference latency (2s/frame) that prevents high-frequency RL interaction. Our approach leverages denoised latent features from video generation models through three key mechanisms: (1) shortcut forcing that reduces sampling complexity via recursive multi-resolution step compression, (2) an autoregressive dense reward model operating directly on latent representations for fine-grained credit assignment, and (3) Gaussian vocabulary sampling for GRPO that constrains exploration to physically plausible trajectories. DreamerAD achieves 87.7 EPDMS on NavSim v2, establishing state-of-the-art performance and demonstrating that latent-space RL is effective for autonomous driving.
1 Introduction
DreamerAD addresses the cost, safety, latency, and representation limits of autonomous-driving RL by training within an interpretable latent world model. It compresses diffusion-based imagination and achieves 87.7 EPDMS on the NavSim v2 closed-loop benchmark.
- RL is motivated by long-tail driving problems, but real-world trial-and-error is costly and unsafe, while conventional simulators introduce sim-to-real discrepancies.
- Existing pixel-level diffusion world models require 100 sampling steps, causing latency that is incompatible with high-frequency RL interaction.They also prioritize visual fidelity over the spatial and dynamic understanding needed for driving safety.
- DreamerAD performs reinforcement learning entirely within the latent imagination space of a video-generation model.The framework leverages denoised latent features that retain spatial and semantic structure.
- Shortcut forcing compresses world-model sampling from 100 steps to one step while preserving lossless decoding into high-fidelity RGB frames.This supports low-latency latent-space RL with visual interpretability.
- 87.7 EPDMS establishes DreamerAD as state of the art on the NavSim v2 closed-loop benchmark.
2 Related Works
Prior autonomous-driving world models use video generation and other representations, but high-frequency RL remains constrained by diffusion latency and hallucinations under out-of-distribution actions. DreamerAD targets these limitations through reduced sampling complexity and exploration-constrained reinforcement learning.
- Recent driving world models use autoregressive scene generation or condition diffusion models on BEV maps and 3D bounding boxes.Other approaches use scene-generation systems such as GAIA-1, DriveDreamer, and MagicDrive.
- Existing world models struggle with high-frequency RL because multi-step diffusion inference is slow and models can hallucinate under out-of-distribution actions.
- DreamerAD addresses these gaps by reducing sampling complexity and constraining exploration to physically plausible trajectories.
- World-model imagination has been used for RL, including trajectory-conditioned video synthesis, latent rollouts, 3D Gaussian Splatting, and occupancy-based representations.These methods face limitations involving efficiency, annotation dependency, or underuse of latent world-model features.
3 Method
DreamerAD performs autonomous-driving reinforcement learning in a latent imagination space, combining efficient world-model rollouts, dense temporal rewards, and constrained trajectory exploration. Its shortcut-forcing world model reduces inference to 1–4 steps while preserving prediction fidelity under one-step inference.
- Method: DreamerAD couples latent world-model imagination with autoregressive dense rewards and Gaussian-weighted vocabulary sampling for reinforcement-learning trajectory optimization.The pipeline evaluates sampled trajectories through latent rollouts, predicts step-wise rewards across driving metrics, and optimizes the policy with GRPO.
- World Model: Latent visual and action embeddings condition the generator to predict both the next-frame latent and future trajectory.Historical observations are compressed by a visual autoencoder, actions by an action encoder, and their unified latent representation drives flow-matching prediction.
- Shortcut Forcing World Model: 1–4 sampling steps replace Epona’s 100-step inference, achieving up to 80× faster world-model inference while preserving prediction fidelity.SF-WM uses recursive shortcut forcing over multi-resolution step sizes and can generate predictions with one to four sampling steps.
- Reinforcement Learning with Vocabulary Sampling: The exploration vocabulary filters 8192 trajectories near human-driving endpoints, then uniformly samples 256 candidates across lateral deviations.Candidates are constrained by endpoint position and heading thresholds before representative trajectories are selected.
- Autoregressive Dense Reward Modeling: The autoregressive reward model evaluates predicted latent states across eight horizons and eight reward dimensions, capturing both trajectory quality and temporal reward evolution.Rewards are generated at horizons from 0 to 4.0 seconds in 0.5-second increments, producing eight time-step scores.
4 Experiment
DreamerAD is evaluated on NavSim v1/v2 through closed-loop benchmarks, component ablations, inference-step tests, reward-data scaling, and qualitative driving comparisons. The method achieves strong safety-oriented performance while preserving efficient latent imagination training.
- 4.3 Main Results: 87.7 EPDMS on NAVSIM v2 exceeds all existing methods and the Epona baseline by 2.6 points.The method also improves NC, TTC, DAC, LK, HC, and EC relative to Epona, while EP decreases by 0.8 points as a safety-first trade-off.
- 4.3 Main Results: 88.7 on NAVSIM v1 leads world-model-based methods and exceeds Epona by 2.5 points overall.DAC improves by 2.1 and TTC by 0.5, although AutoVLA and RecogDrive report higher overall scores under different training setups.
- 4.4 Ablation Studies: Shortcut Forcing improves driving performance under extreme step compression, while the autoregressive dense reward model supplies temporal-grained reward signals.These component comparisons are reported in the ablation study alongside alternative reinforcement-learning sampling methods.
- 4.4 Ablation Studies: 87.7 EPDMS with one-step inference and 0.03s latency remains competitive with 16-step and 4-step settings.The result supports compressing sampling without compromising downstream policy planning.
- 4.4 Ablation Studies: Reward-model results differ little between using 100% and 20% of training data, indicating robust reward learning from imagined future states.The authors report that the model learns essential differences between good and bad driving behaviors with limited data.
- 4.3 Main Results: After RL training, the model decelerates and stops behind obstacles instead of reproducing the high-speed collisions observed with SFT.The qualitative comparison includes stationary vehicles and curb collisions in BEV planning maps.
5 Conclusion
DreamerAD combines efficient latent-world-model inference, dense reward modeling, and constrained exploration for autonomous-driving reinforcement learning. It reaches 87.7 EPDMS on NavSim v2 while supporting safe driving behavior without real-world trial-and-error.
- 5 Conclusion: DreamerAD combines shortcut forcing, autoregressive dense reward modeling, and Gaussian vocabulary sampling for latent-space autonomous-driving RL.These mechanisms target faster inference, fine-grained credit assignment, and physically plausible exploration, respectively.
- 5 Conclusion: 87.7 EPDMS on NavSim v2 establishes a new state-of-the-art for closed-loop planning.The conclusion presents latent imagination-based RL as effective for learning safe driving behaviors without real-world trial-and-error.
- 5 Conclusion: Figure 5 contrasts SFT and RL BEV trajectories, marking SFT collisions in red and RL safe passage in green.The visual uses the front-view camera image as the current-timestep reference and compares the two planning outputs in adjacent columns.