Source-linked AI summary
Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)
Zhenjie Yang, Xiaosong Jia, Qifeng Li, Xue Yang, Maoqing Yao, Junchi Yan
TL;DR
End-to-end autonomous driving needs alternatives to imitation learning, but reinforcement learning with raw sensors remains difficult because prior model-based methods rely on privileged information. Raw2Drive uses a dual-stream model-based framework in which a privileged world model guides raw-sensor world-model and policy training, achieving state-of-the-art results on CARLA v2 and Bench2Drive. Its scope remains tied to training with ground-truth bounding boxes and HD-Map and to simulator-based policy learning.
Problem
Raw-sensor model-based reinforcement learning for end-to-end autonomous driving remains underexplored because raw observations are high-dimensional, redundant, and noisy, whereas prior successful MBRL uses privileged information.
Method
Raw2Drive trains a privileged world model and planner, then uses a Guidance Mechanism to align a raw-sensor world model and guide its end-to-end policy.
Results
Raw2Drive is reported as the first raw-image-to-planning MBRL framework and achieves state-of-the-art performance on CARLA v2 and Bench2Drive, surpassing imitation-learning methods by a large margin.
Takeaways & Limitations
The work provides a model-based reinforcement-learning route for end-to-end autonomous driving beyond imitation learning and privileged-input-only approaches.
Takeaways & Limitations
The setting uses ground-truth bounding boxes and HD-Map as privileged input, while real-world reinforcement learning remains a technical issue and evaluation is conducted in CARLA.
Abstract
from arXiv · showhide
Reinforcement Learning (RL) can mitigate the causal confusion and distribution shift inherent to imitation learning (IL). However, applying RL to end-to-end autonomous driving (E2E-AD) remains an open problem for its training difficulty, and IL is still the mainstream paradigm in both academia and industry. Recently Model-based Reinforcement Learning (MBRL) have demonstrated promising results in neural planning; however, these methods typically require privileged information as input rather than raw sensor data. We fill this gap by designing Raw2Drive, a dual-stream MBRL approach. Initially, we efficiently train an auxiliary privileged world model paired with a neural planner that uses privileged information as input. Subsequently, we introduce a raw sensor world model trained via our proposed Guidance Mechanism, which ensures consistency between the raw sensor world model and the privileged world model during rollouts. Finally, the raw sensor world model combines the prior knowledge embedded in the heads of the privileged world model to effectively guide the training of the raw sensor policy. Raw2Drive is so far the only RL based end-to-end method on CARLA Leaderboard 2.0, and Bench2Drive and it achieves state-of-the-art performance.
1 Introduction
Raw2Drive addresses the difficulty of applying reinforcement learning to raw-sensor end-to-end autonomous driving through a dual-stream model-based framework. It uses privileged information to guide raw-sensor world-model and policy learning, achieving strong benchmark performance with stated training costs.
- Motivation: Imitation learning is widely used for end-to-end driving but suffers from causal confusion and poor generalization to unseen situations.These limitations become problematic in complex, dynamic scenarios requiring decisions about environmental interactions.
- Motivation: RL has been less explored for autonomous driving because prior approaches faced efficiency and convergence problems, while model-based methods relied on privileged information.MaRLn required approximately 50M steps and 57 days of training, and Think2Drive used privileged environment states rather than raw sensors.
- Raw2Drive: Raw2Drive introduces a dual-stream MBRL framework that first trains a privileged world model and planner, then trains a raw-sensor world model and end-to-end planner.The raw-sensor stream uses alignment with privileged-model frame-wise features rather than reconstructing computationally expensive multi-view videos.
- Raw2Drive: The Guidance Mechanism enforces consistency between the two world models during rollouts and transfers privileged-model head knowledge to supervise the raw-sensor policy.This combines rollout guidance for future-state consistency with head guidance for accurate and stable policy supervision.
- Results: Raw2Drive is presented as the first raw-image-to-planning MBRL framework for end-to-end autonomous driving and achieves state-of-the-art performance on CARLA v2 and Bench2Drive.The authors report that it surpasses imitation-learning methods by a large margin.
- Results: The final planner uses 64 H800 GPU days, reducible to 40 GPU days when Think2Drive is reused, compared with about 30 GPU days for UniAD.The cited comparison states that UniAD solves only 3∼4 corner cases in CARLA v2.
2 Problem Formulation & Related Works
Raw2Drive separates training-time privileged observations from inference-time raw sensor observations. Its pipeline trains a privileged world model and policy first, then uses that model to guide the raw sensor stream through rollout and head guidance.
- Problem Formulation: The problem formulation contains privileged observations with ground-truth bounding boxes and HD-Map, alongside raw sensor observations from cameras, LiDAR, and IMU.Privileged observations are available only during training, while raw sensor observations define the inference setting.
- Training Pipeline: During training, privileged input trains a privileged world model and paired policy, which then guides training of the raw sensor stream.The pipeline uses the privileged model as the source of guidance rather than requiring privileged input at deployment.
- Guidance Mechanism: The guidance mechanism has Rollout Guidance for future-modeling consistency and Head Guidance for accurate, stable supervision of the raw sensor policy.These are the two guidance components identified in the pipeline description.
3 Method
Raw2Drive uses a dual-stream model-based reinforcement-learning framework that first trains a privileged stream, then guides raw-sensor world-model and policy learning through aligned rollouts and privileged supervision.
- Dual-stream framework: Raw2Drive contains two world models and two corresponding policies, with privileged observations available only during training and raw sensors used during inference.The privileged stream uses BEV semantic masks, while the raw stream processes multi-view images and IMU.
- Privileged stream: The privileged world model supports reinforcement-learning rollouts for its policy and guides training of the end-to-end raw-sensor stream.Its encoder, RSSM, and heads provide the auxiliary structure for both privileged policy training and raw-stream guidance.
- Raw sensor stream: The raw-sensor world model uses a decoder head supervised by BEV semantic masks rather than reconstructing multi-view videos.Its architecture differs from the privileged model in the encoder and heads, and scalar reward and continuation heads are omitted.
- Policy training: The raw-sensor policy is trained by reinforcement learning while privileged-model heads provide reward and continuation signals.Both streams execute the same action during rollouts, and the privileged model supplies the supervision signals used for raw-policy learning.
- Guidance mechanism: Rollout Guidance aligns encoded, deterministic, and stochastic states between streams to maintain spatial, temporal, dynamic, and static consistency.The guidance addresses instability in raw encoded states and discrepancies that can accumulate during stochastic rollouts.
4 Experiments
Experiments evaluate Raw2Drive in CARLA and Bench2Drive using raw sensors at inference, while ablations examine guidance, heads, parameter sharing, fine-tuning, and latency. The reported results emphasize strong short-route benchmark performance and efficient inference, alongside limitations of long-route scoring.
- Evaluation setup: Evaluation uses CARLA 0.9.15.1 closed-loop driving, with privileged observations prohibited during evaluation.The study reports official CARLA metrics and Bench2Drive’s additional Success Rate.
- Evaluation setup: Long-route CARLA Leaderboard 2.0 evaluation is reported as a poor reflection of real driving performance because of cumulative penalty scoring.The authors therefore focus mainly on short-route closed-loop Bench2Drive results.
- Main results: Raw2Drive achieves state-of-the-art performance on the Bench2Drive multi-ability and raw-sensor end-to-end closed-loop benchmarks.The reported comparison is based on the short-route evaluation that the authors prioritize.
- Guidance ablations: Removing rollout-guidance components limits the model to simple behaviors, while spatial and temporal alignment are both essential and complementary for stable rollouts.The ablations connect state misalignment with discrepancies between the two world-model predictions.
- Ablations: Head guidance improves policy training, but additional heads increase training complexity and degrade performance, so Raw2Drive retains only decoder-head guidance.The selected configuration follows the reported ablation conclusion.
- Efficiency: World-model and policy modules each run under 2ms, while the vision encoder is the main raw-sensor inference bottleneck.The latency analysis identifies perception rather than the world model or policy as the dominant cost.
5 Conclusion
Raw2Drive introduces a dual-stream end-to-end model-based reinforcement learning approach for autonomous driving, addressing limitations of imitation learning and privileged-information dependence. It combines privileged and raw-sensor world models to support reinforcement-learning-based policy learning.
- Raw2Drive is presented as the first end-to-end model-based reinforcement learning method for autonomous driving.
- The method uses separate world models for privileged and raw sensor information, connected through a guidance mechanism.
- Imitation learning remains limited by causal confusion, poor generalization, and difficulty completing long-term decisions in complex traffic scenarios.
B Details of Training Pipeline
Raw2Drive trains its models in two stages: first with privileged observations, then with raw sensor observations using rollout guidance and head guidance. The second stage aligns the raw sensor world model with the privileged model while adapting the policy.
- Stage I: Privileged World Model and Policy Training: Stage I alternates updates to a privileged world model and behavior policy using privileged observations and simulated trajectories.
- Stage II: Raw Sensor World Model and Policy Training: Stage II trains the raw sensor world model with an additional rollout-guidance loss and fine-tunes the privileged policy through the raw sensor world model.
- Stage II: Raw Sensor World Model and Policy Training: Rollout guidance aligns latent-state predictions between the raw sensor and privileged world models during raw-world-model training.
- Stage II: Raw Sensor World Model and Policy Training: Head guidance supplies predicted rewards and continuous flags used to construct trajectories for training the raw sensor policy.
C Details of the Experiment
The experiment represents privileged observations with BEV semantic masks and ego-vehicle vectors, while raw sensor observations are encoded from surrounding RGB images into grid-shaped BEV features.
- Privileged inputs combine BEV semantic segmentation masks with ego-vehicle information represented as a vector.The BEV representation contains 43 channels covering static and dynamic object categories.
- Raw sensor inputs use BEVFormer to encode surrounding RGB images into grid-shaped BEV features.
C.2 Action Space in Raw2Drive
Raw2Drive simplifies control by decomposing the continuous action space into a finite set of discrete actions.
- 39 discrete actions represent specific combinations of throttle, steering, braking, and reverse values.The actions are described as rational and legitimate, with the full list provided in Table 12.
C.3 Model Configuration
The model configuration combines weighted world-model objectives with behavior-policy optimization, while rollout guidance supports consistency between the dual-stream models.
- The implementation adopts Think2Drive’s reward design and reward-shaping approach.
- Both world models use prediction, dynamics, and representation losses with respective weights βpred, βdyn, and βrep.
- The privileged behavior-policy critic is trained with maximum likelihood using return targets and value estimates.
- Actor optimization uses entropy regularization, exponential moving-average smoothing, and stop-gradient operations for gradient stability.
- Rollout guidance enforces consistency between the dual-stream world models and is critical for convergence of the raw sensor world model.
G Inference Latency
The supplied material identifies a latency comparison between privileged and raw sensor streams and describes BEV reconstruction by the raw sensor decoder.
- Table 13 compares inference latency between the privileged and raw sensor streams.
- The raw sensor decoder evaluates reconstruction quality within the dual-stream architecture.
- The raw sensor world model replaces expensive multi-view video reconstruction with more efficient BEV reconstruction.
H.2 Perception Confusion
Perception confusion arises when visually similar adjacent frames correspond to different rewards or insufficient continuation signals, complicating learning in corner cases.
- Adjacent frames can look highly similar while identical actions produce opposite rewards in the ParkingExit corner case.
- Near task completion, subtle visual changes combined with a binary continuation flag provide limited learning information.
- Figure 8 visualizes ground-truth and raw sensor decoder outputs, marking a camera blind spot with a red circle.