Source-linked AI summary
DiDrive: A Risk-Aware Hierarchical Diffusion Framework for Safe Offline Reinforcement Learning in Autonomous Driving
Qisong Guo, Jingtang Chen, Zhilin Chen, Pei Xu, Mingjian Fu, Wenxi Liu, Yuanlong Yu
TL;DR
Offline reinforcement learning for autonomous driving must handle distribution shift, heavy-tailed risk signals, OOD action generation, and redundant high-dimensional states. DiDrive combines the 3DICE policy optimization paradigm with the RHDif representational architecture to address these challenges. In CARLA, it achieves an 85% success rate and 4295.68 average reward in high-density scenarios with 60 background vehicles, while maintaining 81% success in zero-shot cross-town tests.
Problem
Offline-RL driving policies remain vulnerable to distribution shift, heavy-tailed risk signals, OOD actions, and redundant high-dimensional traffic representations.
Method
DiDrive combines 3DICE support-aware diffusion policy optimization with RHDif risk-aware hierarchical state representation.
Results
85% success rate and 4295.68 average reward were achieved in CARLA scenarios with 60 background vehicles, while zero-shot cross-town success reached 81%.
Takeaways & Limitations
DiDrive provides a risk-aware policy redistribution framework within the support of real driving data for safe offline autonomous-driving decision-making.
Takeaways & Limitations
Multi-step diffusion denoising introduces inference latency, and the simulator-based evaluation leaves the simulation-to-reality gap unexplored.
Abstract
from arXiv · showhide
While diffusion models effectively capture multimodal behavioral priors for autonomous driving, offline reinforcement learning (RL) policies remain susceptible to distribution shift, heavy-tailed risk signals, out-of-distribution (OOD) action generation, and high-dimensional state redundancy. To address these challenges, we propose DiDrive, a distribution-guided offline diffusion framework featuring two synergistic components: the Risk-Aware Hierarchical Diffusion (RHDif) architecture and the 3DICE policy optimization paradigm. In the state space, RHDif utilizes a low-level risk-gated encoder and a high-level contextual modulator to filter environmental redundancy and focus on safety-critical threats. In the action space, 3DICE mitigates OOD overestimation and gradient oscillation through in-sample calibrated guidance, spatiotemporal optimization, and ensemble-based candidate ranking. Evaluations on the CARLA benchmark demonstrate DiDrive's superiority over baselines like IQL, CQL, and Diffusion-QL, particularly in complex, high-density traffic scenarios with 60 vehicles, where it achieves an 85% success rate and a 4295.68 average reward, providing a robust pathway for safe autonomous driving decision-making.
1. Introduction
DiDrive addresses offline-RL challenges in autonomous driving by combining support-aware diffusion policy optimization with risk-aware hierarchical state representation. Its components target distribution shift, heavy-tailed risk signals, OOD actions, and redundant environmental information.
- Offline reinforcement learning avoids the safety hazards and costs of online trial-and-error by learning policies from static datasets.
- Distribution shift, heavy-tailed collision penalties, gradient oscillation, and redundant high-dimensional traffic inputs limit reliable offline-RL driving policies.
- DiDrive integrates support-aware policy optimization with risk-aware state representation to keep diffusion decisions within real-data support and isolate critical safety cues.
- 3DICE uses in-sample guidance and progressive parameter integration to shift behavior policies toward high-quality, in-support actions while mitigating value overestimation and heavy-tailed-reward instability.
- RHDif combines RGSE bottom-up risk perception with CMCM top-down contextual filtering to amplify threats and suppress irrelevant background interference.
- DiDrive significantly outperforms state-of-the-art baselines across long-tailed, high-density, and safety-critical CARLA scenarios.
2. Related Work
Related work progresses from conservative offline-RL methods to expressive generative policies and distribution correction, but autonomous driving still exposes gaps in multimodal modeling, OOD action handling, risk representation, and training stability.
- Offline reinforcement learning learns policies from static datasets without additional environment interaction, making it valuable when data collection is costly or safety requirements are stringent.
- Conservative methods such as BCQ, TD3-BC, CQL, and IQL reduce extrapolation or OOD overestimation but often use restrictive unimodal Gaussian policy parameterizations.
- DICE-family distribution-correction methods estimate stationary distribution ratios and reduce reliance on explicit value estimates for unseen actions.
- Driving datasets with heavy-tailed rewards, including severe collision penalties, can induce numerical instability and gradient oscillation that distort policy guidance.
- Diffusion models capture multimodal driving behavior and support behavior planning, prediction, simulation, and end-to-end planning.
- Existing diffusion architectures may generate unsupported actions, amplify OOD value errors, and process traffic inputs without sufficiently differentiated spatiotemporal risk attention.
3. Methodology
DiDrive combines risk-aware hierarchical representation learning with distribution-corrected diffusion policy optimization for offline autonomous driving. RHDif prioritizes safety-critical state information, while 3DICE keeps action guidance within supported data regions and stabilizes candidate selection.
- Framework overview: DiDrive couples RHDif behavior-prior modeling with 3DICE distribution correction across perception and decision-making.RHDif extracts risk-aware multimodal state representations, while 3DICE shifts behavior toward high-quality in-support actions.
- Problem formulation: The driving MDP uses continuous multimodal states and continuous longitudinal and lateral actions learned solely from a static offline dataset.The reward jointly encourages efficient driving, lane keeping, smooth control, and safety, including severe collision and off-road penalties.
- 3DICE policy optimization: 3DICE structures policy optimization into representation, guidance, optimization, and selection stages to reduce OOD value overestimation.Its in-sample guidance adjusts diffusion denoising using historical experience weights, and candidate ranking occurs within sampled behavior-consistent actions.
- Problem formulation: Offline autonomous-driving policies face distribution shift, multimodal behavior modeling limits, heavy-tailed rewards, and redundant high-dimensional state information.These issues include OOD action generation, value overestimation, unstable gradients, and delayed identification of hazardous targets.
- Risk-Aware Hierarchical Diffusion: RHDif hierarchically enhances local risk features and suppresses irrelevant global context through RGSE and CMCM.RGSE uses local risk perception, while CMCM applies contextual filtering to reduce background interference and preserve safety-relevant responses.
4.1. Experimental Setup
Experiments use CARLA Town03 for training and in-domain evaluation, with unseen Town05 testing cross-town generalization under standardized closed-loop conditions.
- Environment and generalization: Town03 provides training and in-domain evaluation, while unseen Town05 tests transfer across different road layouts and interaction patterns.Town03 includes a multi-lane roundabout, tunnels, and irregular junctions; Town05 contains gridlike multi-lane networks and highway intersections.
- Dataset and observations: The offline dataset contains more than 7,000 trajectories and 1.1 million timesteps, combining expert and random-exploration policies at an 8:2 ratio.Inputs are 307-dimensional, covering ego state, lane information, LiDAR, surrounding vehicles, and navigation waypoints.
- Training protocol: DiDrive uses a two-stage procedure that first pretrains the RHDif behavior prior and then fine-tunes 3DICE for distribution correction.The reported diffusion configuration uses 15 denoising iterations and 32 inference samples.
- Evaluation protocol: Each configuration is evaluated in 100 online closed-loop episodes using Success Rate, Average Reward, and Average Distance.Success Rate counts episodes without collision or road departure within 1000 timesteps.
- Baseline comparison: IQL, CQL, ODICE, TD3-BC, QGPO, and Diffusion-QL share the same dataset, representations, reward function, and evaluation settings for fair comparison.Diffusion-based baselines also use the same denoising-step and candidate-sampling parameters as DiDrive.
4.2. Comparative Experiments
DiDrive converges more smoothly than the baselines and achieves the strongest closed-loop performance across traffic densities, including unseen Town05 environments.
- Training analysis: QGPO stagnates in total reward and keeps lane deviation above 1.0 m for most of training, while ODICE exhibits strongly oscillating learning behavior.ODICE degradation and increasing deviation later in training indicate instability associated with high-dimensional features and heavy-tailed rewards.
- Training analysis: DiDrive rapidly reaches the highest reward range, above 5200, while maintaining lane deviation near 0.1 m throughout training.The reported trend links this behavior to hierarchical prior features and PPI-based smooth optimization.
- Town03 closed-loop testing: In Town03’s 60-vehicle complex scenario, DiDrive achieves an 85.00% success rate, 4295.68 average reward, and 254 m average driving distance.These results exceed CQL’s 3819.05 average reward despite CQL’s 74.00% success rate.
- Interpretation: The reported advantage combines safety, traffic efficiency, and trajectory quality through in-sample guidance, stabilized optimization, support-set selection, and risk-focused state processing.3DICE addresses action-space issues while RHDif emphasizes safety-relevant interactions in high-dimensional inputs.
- Town05 generalization: On unseen Town05, DiDrive maintains high success rates and average rewards across simple, medium, and complex traffic densities.The experiments are designed to test transfer to new road layouts and interaction patterns rather than overfitting to Town03.
- Town05 generalization: Town05 baseline performance degrades as traffic density rises, with TD3-BC’s success rate falling to 35.00% in the complex scenario.CQL remains safe but becomes overly timid and struggles to preserve driving efficiency on unfamiliar road topologies.
4.3. Ablation Study
Ablation results show that DiDrive’s stable performance depends on the coordination of 3DICE with RHDif’s complementary local risk amplification and contextual filtering. Removing these components causes training instability, reward degradation, weaker risk alignment, or poorer closed-loop safety.
- Training dynamics: DiDrive’s full model converges rapidly to around 5200 reward while maintaining approximately 0.10 m lane deviation.The full model shows the most stable training trend in the ablation curves.
- Training dynamics: Removing 3DICE causes the largest degradation, with severe sawtooth-like reward behavior during training.The comparison identifies policy optimization as central to stable system optimization.
- Representation ablations: CMCM removal produces lateral oscillations and reward fluctuations, whereas RGSE removal leaves smoother lane keeping but reduces sensitivity to sudden dangerous interactions.CMCM filters background noise and pseudo-risks, while RGSE enhances local risk responses.
- Representation ablations: Without RGSE and CMCM, total reward stagnates at around 3000 despite baseline lateral stability on straightforward road segments.The standard diffusion backbone is insufficient for fine-grained multi-vehicle interactions and complex, long-tailed traffic.
- Overall finding: The ablations confirm that representation filtering and policy optimization are interdependent for balancing driving efficiency and safe control.Performance improvements rely on coordination rather than a single component.
- Closed-loop testing: 85% success and 4295.68 average reward are achieved by the full model in complex traffic, exceeding the 44% success and 3313.04 reward obtained without 3DICE.The full model also reaches 86% success and 299 m driving distance in the medium scenario.
- Risk-aware module analysis: CMCM improves Risk Attention Alignment by filtering background interference and preserving semantically meaningful cues for genuine threats.The module calibrates amplified local risk features against the diagnostic reference risk distribution.
- Risk-aware module analysis: Removing RGSE increases collision rate from 13.0% to 18.0% and decreases success rate from 85.0% to 78.0%.The ablation is also associated with a higher TTC < 2.5 s Rate, increasing from 30.0% to 33.0%.
5. Conclusion
DiDrive combines 3DICE policy optimization with RHDif representation learning to address distribution shift, traffic redundancy, and heavy-tailed risk signals in offline autonomous-driving RL. In CARLA, it achieves the strongest reported complex-traffic performance and maintains zero-shot cross-town generalization, while real-time deployment and simulation-to-reality transfer remain open challenges.
- Contribution: DiDrive performs risk-aware policy redistribution within real-driving-data support, reducing reliance on extrapolated value estimation for OOD actions.The framework synergizes 3DICE optimization with RHDif representations.
- Results: 85% success rate and 4295.68 average reward are achieved in CARLA’s complex scenarios with 60 background vehicles.DiDrive outperforms Diffusion-QL at 77% success and CQL at 74% success in the cited comparison.
- Generalization: 81% success rate is maintained in zero-shot cross-town generalization tests.The reported result is presented as evidence of generalized safety-critical feature extraction rather than map-specific overfitting.
- Limitations: Multi-step diffusion denoising introduces inference latency, and simulator-only evaluation leaves the simulation-to-reality gap unexplored.The paper proposes accelerated sampling and future validation on real-world datasets.