Source-linked AI summary
RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning
Hao Gao, Shaoyu Chen, Bo Jiang, Bencheng Liao, Yiang Shi, Xiaoyang Guo, Yuechuan Pu, Haoran Yin, Xiangyu Li, Xinbang Zhang, Ying Zhang, Wenyu Liu, Qian Zhang, Xinggang Wang
TL;DR
End-to-end AD methods based on IL face causal confusion and an open-loop gap. RAD addresses these issues with closed-loop RL in a photorealistic 3DGS digital replica, safety-oriented objectives, and IL regularization; it achieves stronger closed-loop performance, including a 3× lower collision rate than IL-based methods.
Problem
IL-based end-to-end AD can capture correlations rather than causal relationships and struggles with open-loop-to-closed-loop gaps and safety-critical events.
Method
RAD trains an end-to-end AD policy with closed-loop RL in a photorealistic 3DGS digital replica, using safety rewards, auxiliary objectives, and IL regularization.
Results
RAD achieves stronger closed-loop performance than IL-based methods, including a 3× lower collision rate.
Takeaways & Limitations
Combining RL and IL enables RAD to address causation and the open-loop gap while maintaining human driving alignment.
Takeaways & Limitations
3DGS rendering remains limited for non-rigid pedestrians, unobserved views, and low-light scenarios.
Abstract
from arXiv · showhide
Existing end-to-end autonomous driving (AD) algorithms typically follow the Imitation Learning (IL) paradigm, which faces challenges such as causal confusion and an open-loop gap. In this work, we propose RAD, a 3DGS-based closed-loop Reinforcement Learning (RL) framework for end-to-end Autonomous Driving. By leveraging 3DGS techniques, we construct a photorealistic digital replica of the real physical world, enabling the AD policy to extensively explore the state space and learn to handle out-of-distribution scenarios through large-scale trial and error. To enhance safety, we design specialized rewards to guide the policy in effectively responding to safety-critical events and understanding real-world causal relationships. To better align with human driving behavior, we incorporate IL into RL training as a regularization term. We introduce a closed-loop evaluation benchmark consisting of diverse, previously unseen 3DGS environments. Compared to IL-based methods, RAD achieves stronger performance in most closed-loop metrics, particularly exhibiting a 3x lower collision rate. Abundant closed-loop results are presented in the supplementary material. Code is available at https://github.com/hustvl/RAD for facilitating future research.
1 Introduction
Existing end-to-end AD methods largely use IL but face causal confusion, open-loop deployment gaps, and weak robustness to safety-critical or unseen situations. RAD addresses these issues with 3DGS-based closed-loop RL, safety-focused rewards, and IL regularization, achieving stronger closed-loop performance.
- Motivation: Most end-to-end AD algorithms use IL, which directly maps sensory inputs to driving actions but faces real-world deployment challenges.IL replaces a modularized pipeline with a holistic one, offering system simplicity and generalization ability.
- Motivation: IL can learn correlations instead of causal relationships, encouraging shortcut learning and limited sensitivity to safety-critical events such as collisions.The training data predominantly contains common driving behaviors and does not adequately cover long-tailed distributions.
- Motivation: Open-loop IL training differs from closed-loop deployment, where accumulated trajectory errors produce compounding errors and out-of-distribution scenarios.These unseen situations raise concerns about the robustness of IL-trained policies.
- RAD Framework: RAD establishes a 3DGS-based closed-loop RL paradigm using a photorealistic digital replica for extensive state-space exploration and large-scale trial and error.Specialized safety-related rewards guide responses to safety-critical events and real-world causations.
- RAD Framework: RAD combines RL and IL, using IL regularization to maintain similarity to human driving behavior while RL addresses causation and the open-loop gap.The joint optimization is designed to balance robustness with human alignment.
- RAD Framework: RAD introduces dense collision- and deviation-related auxiliary objectives and a streamlined, decoupled action space to reduce RL exploration costs.These objectives help constrain the full action distribution and alleviate sparse rewards and slow convergence.
- Evaluation: 3× lower collision rate is achieved by RAD than IL-based approaches across a benchmark of diverse, unseen 3DGS environments.RAD outperforms IL-based approaches across most closed-loop metrics.
2 Related Work
Prior work applies 3DGS mainly to scene reconstruction and closed-loop evaluation, while end-to-end AD largely relies on IL and prior RL methods use less realistic settings. RAD jointly combines IL and RL for end-to-end learning from real-world sensor inputs in a photorealistic 3DGS environment.
- Dynamic Scene Reconstruction: Prior dynamic-scene methods use implicit neural representations or 3DGS, but slow rendering limits implicit approaches and reconstructed scenes are mainly used for evaluation.RAD extends 3DGS use toward training.
- End-to-End Imitation Learning: End-to-end IL work improves planning through integrated perception, vectorized representations, single-trajectory methods, multimodal planning, and diffusion policies.These approaches are characterized as recent advances in learning-based planning driven by large-scale data.
- Reinforcement Learning: Prior autonomous-driving RL studies often use non-photorealistic simulators or require perfect perception inputs, whereas RAD trains end-to-end in a photorealistic 3DGS environment.RAD is presented as the first such system in this setting.
- Combining Imitation and Reinforcement Learning: Existing combined IL and RL methods use two-stage or joint optimization, but remain limited to non-photorealistic simulators or structured BEV and virtual sensor inputs.RAD instead jointly integrates IL and RL in a photorealistic digital-twin environment using real-world sensor inputs.
3 RAD
RAD maps multi-view observations to probabilistic lateral and longitudinal actions, then trains the policy through perception, planning, and reinforced post-training. Its 3DGS interaction loop combines parallel rollouts, human demonstrations, structured rewards, and auxiliary objectives.
- End-to-End Driving Policy: RAD transforms multi-view image sequences into scene tokens and samples lateral and longitudinal actions from their probabilistic distributions.The scene representation includes map, agent, and image tokens; navigation information and ego state also condition the action distributions.
- Training Paradigm: RAD trains in three stages: perception pre-training, planning pre-training, and reinforced post-training.Perception supervision teaches map and agent tokens, demonstrations initialize action distributions, and RL with IL fine-tunes the policy.
- End-to-End Driving Policy: The policy uses a decoupled discrete action space over a 0.5-second horizon, reducing dimensionality for more efficient optimization.Lateral and longitudinal actions are separated under constant linear and angular velocity assumptions.
- Training Paradigm: N parallel workers roll out policies in independently reconstructed 3DGS environments, store state-action-reward trajectories, and alternate PPO updates with supervised IL updates.Updated policies are synchronized across workers, while the BEV encoder, map head, and agent head remain frozen during post-training.
- Interaction Mechanism: The 3DGS interaction loop updates the ego pose with a kinematic bicycle model and feeds the resulting state back into the next inference step.The policy predicts actions over a 0.5-second horizon, from which linear velocity and steering angle are derived.
- Reward Modeling: RAD’s reward design penalizes dynamic and static collisions plus positional and heading deviations from the expert trajectory.These four reward sources are intended to penalize unsafe actions and encourage expert-trajectory alignment.
4 Experiments
RAD is evaluated on diverse unseen 3DGS environments through closed-loop metrics, ablations, comparisons with IL-based methods, and consistency checks against real-world behavior. The experiments show that combining IL and RL, using comprehensive rewards and auxiliary objectives, improves safety while preserving behavioral fidelity.
- Experimental Settings: 3968 clips are used for RL training and 337 clips for closed-loop evaluation after reconstructing 4305 real-world scenes into 3DGS environments.The scenes cover diverse road types, traffic densities, and agent behaviors.
- Training Strategy Comparison: IL+RL achieves CR 0.089 and ADD 0.257, balancing the lower deviation of pure IL with the improved safety of pure RL.Pure IL records ADD 0.238 and CR 0.229, while pure RL records ADD 0.345 and CR 0.143.
- Reward Source Analysis: The full reward setup achieves the lowest CR 0.089 while maintaining ADD 0.257, whereas omitting dynamic collision rewards produces CR 0.238.The results support including all reward terms, especially the dynamic collision component.
- Auxiliary Objective Analysis: Removing auxiliary objectives increases collision ratio, and excluding the PPO objective yields CR 0.187 despite retaining the auxiliary objectives.The reported results indicate that auxiliary objectives work most effectively when combined with PPO.
- Closed-loop Comparison: RAD achieves better performance than IL-based methods on most closed-loop metrics, with a 3× lower collision rate.The comparison uses the same amount of human driving demonstrations for all methods.
- Consistency and Qualitative Analysis: Qualitative and quantitative comparisons indicate high behavioral consistency between 3DGS and real-world environments, while RAD avoids dynamic obstacles more effectively than the IL-only policy.The IL-only policy fails to yield to pedestrians in the illustrated scenario, whereas RAD successfully yields.
5 Conclusion
RAD is presented as a 3DGS-based closed-loop RL framework that combines RL and IL with targeted rewards and auxiliary objectives. It achieves lower collision rates than state-of-the-art IL methods, while 3DGS remains limited in several rendering conditions.
- 5 Conclusion: RAD combines RL and IL so RL addresses causation and the open-loop gap while IL supports human alignment.The framework also uses targeted rewards and auxiliary objectives.
- 5 Conclusion: RAD achieves 3× lower collision rates than state-of-the-art IL methods, with strong performance in unprotected left-turns and dense traffic.These are the challenging scenarios specifically highlighted in the conclusion.
- Limitations and Future Work: 3DGS rendering still needs improvement for non-rigid pedestrians, unobserved views, and low-light scenarios.Future work is described as addressing these issues and scaling up RL.
A.1 3DGS Reconstruction and Rendering Optimization
The reconstruction pipeline extends StreetGaussian to improve geometric accuracy and rendering realism for off-trajectory views. Mesh constraints, separate sky modeling, and foreground-object optimization support consistent novel-view rendering in dynamic scenes.
- A.1 3DGS Reconstruction and Rendering Optimization: The method extends StreetGaussian to improve rendering realism and geometric accuracy, particularly for high-fidelity off-trajectory views.These improvements target 3DGS environments used for closed-loop training.
- A.1 3DGS Reconstruction and Rendering Optimization: Mesh modeling constrains road-surface geometry, while separate sky modeling reduces foreground confusion under complex lighting.Gaussian spheres are constrained to the road mesh surface from any viewpoint.
- A.1 3DGS Reconstruction and Rendering Optimization: Foreground-object poses are optimized with depth and normal consistency supervision to improve dynamic-scene geometry and surface-detail fidelity.The optimization improves rendering quality in novel viewpoints while preserving observed motion trajectories and surface details.
- A.1 3DGS Reconstruction and Rendering Optimization: These reconstruction improvements provide the realism and geometric accuracy needed for large-scale trial-and-error learning in closed-loop driving.The stated role is to make the 3DGS environment more effective for closed-loop training.
A.2 Action Space Details
RAD defines a short-horizon discrete action space for stable control and efficient learning. The ego vehicle’s lateral and longitudinal movements are represented separately over 0.5 seconds.
- A.2 Action Space Details: The action space models ego-vehicle movement as discrete lateral and longitudinal displacements over a 0.5-second horizon.This design is intended to support stable control and efficient learning.
- Lateral Displacement: Lateral displacement ax is discretized into Nx symmetric options around zero, including an option to maintain the current trajectory.The options represent leftward and rightward movement as well as no lateral change.
- Lateral Displacement: The maximum lateral displacement is 0.75 m, with intermediate values sampled uniformly.
- Longitudinal Displacement: Longitudinal displacement ay is discretized into Ny forward-displacement options, including an option to maintain the current position.The longitudinal component uses the same 0.5-second horizon as the lateral component.
- Longitudinal Displacement: The maximum longitudinal displacement is 15m, with intermediate values sampled uniformly.
A.3 Policy Optimization Details
RAD’s policy optimization uses PPO-style clipped objectives with generalized advantage estimates for lateral and longitudinal actions, supporting stable updates through dimension-specific importance-ratio clipping.
- GAE computes advantage estimates from temporal-difference errors for lateral and longitudinal dimensions, with γ discounting future rewards and λ controlling the bias–variance trade-off.
- The full PPO objectives use separate importance-sampling ratios for lateral and longitudinal action dimensions.
- Dimension-specific constants ϵx and ϵy define clipping ranges that ensure stable policy updates for lateral and longitudinal controls.
A.4 Auxiliary Objectives Details
RAD adds auxiliary objectives that shape action probabilities toward collision avoidance and alignment with expert trajectories. These objectives use directional corrections for dynamic and static obstacles, positional deviation, and heading deviation.
- Auxiliary objectives penalize undesirable behavior using reward sources for dynamic collisions, static collisions, positional deviations, and heading deviations.
- The action distribution is decomposed into four parts to evaluate the selected actions from the old AD policy.
- Dynamic Collision Auxiliary Objective: For dynamic collisions, the objective promotes deceleration when a collision is ahead and acceleration when it is behind.
- Static Collision Auxiliary Objective: For static obstacles, the objective promotes steering away from the obstacle, using opposite steering directions for obstacles on the left and right.
- Positional Deviation Auxiliary Objective: For positional deviation, the objective promotes rightward correction after leftward deviation and leftward correction after rightward deviation.
- Heading Deviation Auxiliary Objective: For heading deviation, the objective promotes clockwise correction after counterclockwise deviation and counterclockwise correction after clockwise deviation.
A.5 Implementation Details
RAD uses planning pre-training followed by alternating reinforcement and imitation learning, and evaluates driving with collision, deviation, smoothness, and qualitative closed-loop measures. Additional comparisons cover diverse scenarios including detours, dense traffic, congestion, and U-turns.
- RAD’s implementation includes planning pre-training and reinforced post-training, with detailed hyperparameters reported for both stages.
- Planning Pre-Training: Planning pre-training discretizes steering-speed actions into predefined anchors and matches ground-truth positions using normalized nearest-neighbor matching.
- Planning Pre-Training: The imitation-learning objective applies dual focal loss to discrete steering and longitudinal action classifications.
- Reinforced Post-Training: Reinforced post-training alternates four rounds of RL with one round of IL, and the reported 4:1 RL-to-IL ratio yields the best results.
- Metric Details: Evaluation uses nine key metrics, including dynamic, static, and total collision ratios, positional and heading deviation ratios, average deviation distance, and motion jerk.
- Metric Details: Average deviation distance measures mean closest distance to the expert trajectory during safe, non-deviating time steps, while jerk metrics measure acceleration changes and driving smoothness.
- A.7 More Qualitative Results: Qualitative comparisons report smoother trajectories, improved collision avoidance, and better adaptability across detours, dense traffic, congestion, and U-turns.