DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving

Motivation

• Diffusion models risk mode collapse, while DiffusionDrive improves multimodality but can generate colliding trajectories because imitation learning leaves negative modes unconstrained.Unimodal planners also fail to provide alternatives, whereas anchor-based generation partitions behavior into distinct driving intentions.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Reinforcement learning constrains all modes with goal-alignment rewards and unsafe-action penalties while exploring policies that may exceed expert behavior in quality and efficiency.This exploration-constraint view raises both the model’s lower bound and upper bound.Source: §1 · p. 2

Method

• Intra-Anchor GRPO estimates advantages only among trajectories sharing an anchor, while Inter-Anchor Truncated GRPO supplies a global perspective without comparing incompatible intentions.Comparing turning and straight-driving trajectories directly could instead promote mode collapse.Source: §1 · p. 2

Results

• 91.2 PDMS on NAVSIM v1 and 85.5 EPDMS on NAVSIM v2 establish state-of-the-art closed-loop performance with a ResNet-34 backbone.The method also achieves the best trade-off between trajectory diversity and consistent quality among diffusion-based generative models.Source: §1 · pp. 2–3

Contributions

• DiffusionDriveV2 combines RL with scale-adaptive multiplicative noise to improve output quality while preserving the Gaussian Mixture Model’s multimodal generation.The work presents this as the first successful migration of GRPO to a truncated diffusion model.Source: §1 · pp. 2–3

End-to-End Autonomous Driving

• End-to-end systems replace modular pipelines with policies learned directly from raw sensor inputs, reducing component-level separation in driving-system design.UniAD integrates perception tasks differentiably, while VAD uses vectorized scene representations.Source: §2 · p. 3
• Diffusion-based planners can model multimodal action distributions, but vanilla diffusion often collapses to one mode and imitation learning can still produce low-quality trajectories.DiffusionDrive counters collapse with anchor-based truncated denoising, while retaining the broader quality-versus-diversity dilemma.Source: §2 · p. 3

Reinforcement Learning for Autonomous Driving

• Recent GRPO applications in autonomous driving and diffusion trajectory modeling remain limited to vanilla diffusion, where mode collapse persists.DiffusionDriveV2 instead adapts GRPO to anchored truncated diffusion.Source: §2 · p. 3

End-to-End Autonomous Driving

• A trajectory is represented as future ego-vehicle waypoints, with each waypoint giving the vehicle’s location over the planning horizon.The trajectory is written as a sequence of coordinate pairs.Source: §3 · p. 3

Truncated Diffusion Model

• Vanilla diffusion iteratively refines Gaussian noise but can collapse to one behavior, limiting alternatives such as following versus overtaking or straight driving versus turning.This motivates structured multimodal trajectory modeling.Source: §3 · p. 3
• DiffusionDrive uses K-Means-derived anchor trajectories to represent distinct driving intents and predicts scene-conditioned offsets from each anchor within a Gaussian Mixture Model.The mixture weight gives the probability of selecting an anchor’s driving intent under scene context.Source: §3 · pp. 3–4
• Truncated diffusion shortens the standard noise schedule to T_trunc steps, with T_trunc much smaller than the full schedule T.The model trains on noisy trajectories and predicts denoised outputs and probability scores.Source: §3 · p. 4

Limitations of DiffusionDrive

• Because each scene provides one ground-truth trajectory, the nearest anchor is assigned as the positive mode and all other anchors are negative modes.Only the positive mode receives direct imitation-learning supervision, leaving many generated trajectories potentially low-quality or colliding.Source: §3 · p. 4

4.1. Truncated Diffusion Generator

• DiffusionDrive is initialized from pretrained imitation-learning weights and refines anchor-sampled noisy trajectories over N_infer denoising steps.The truncated diffusion decoder conditions generation on perception features.Source: §4.1 · p. 4

4.2. Reinforcement Learning for Diffusion Generator

• Trajectory-level reinforcement learning treats conditional denoising as an MDP, applying constraints across modes while exploring superior driving policies.The denoising process begins from an anchor-conditioned Gaussian policy whose likelihood supports REINFORCE updates.Source: §4.2 · p. 5

4.3. Scale-Adaptive Multiplicative Exploration Noise

• Scale-adaptive multiplicative noise preserves smooth exploratory paths by applying longitudinal and lateral Gaussian scaling instead of pointwise additive perturbations.Training uses stochastic DDPM-equivalent sampling with η = 1, while validation uses deterministic DDIM sampling with η = 0.Source: §4.3 · p. 5, Figure 3 · p. 5

4.4. Intra-Anchor GRPO for Trajectory Generation

• Intra-Anchor GRPO samples G variations per anchor and estimates advantages within each group, preventing comparisons between trajectories representing different driving intentions.The update is discounted across denoising steps and retains an imitation-learning loss to protect general driving capabilities.Source: §4.4 · pp. 5–6

4.5. Inter-Anchor Truncated GRPO for Trajectory Generation

• Inter-Anchor Truncated GRPO truncates negative advantages to zero and assigns -1 to collisions, rewarding relative improvements while penalizing absolute failures.This supplies a clearer cross-mode learning signal than purely local comparisons, whose advantages can be misleading across modes.Source: §4.5 · p. 6

4.6. Mode Selector

• The two-stage mode selector scores trajectories with attention over BEV, agent, and map features, then applies coarse top-k selection followed by fine-grained ranking.It is trained with BCE and an additional margin-rank loss that compares relative trajectory quality rather than absolute scores.Source: §4.6 · p. 6

5.1. Benchmark

• The benchmarks use planning-centric NAVSIM data with eight cameras, merged point clouds from five LiDARs, and 1,192 training plus 136 evaluation scenes.Source: §5.1 · p. 6

5.2. Implementation Details

• For fair comparison, the model matches DiffusionDrive’s decoder and uses a ResNet-34 backbone, training for 10 epochs and generating predictions with two denoising steps.Source: §5.2 · pp. 6–7

5.3. Main Results

• 91.2 PDMS on NAVSIM v1 establishes state-of-the-art performance, surpassing DiffusionDrive by 3.1 PDMS and DIVER by 2.9 PDMS.The model also outperforms methods using a larger V2-99 backbone despite using ResNet-34.Source: §5.3 · p. 7, Table 1 · p. 7
• DiffusionDriveV2 achieves a new state-of-the-art EPDMS on the more challenging NAVSIM v2 benchmark using aligned ResNet-34 backbones.Source: §5.3 · p. 7, Table 2 · p. 8
• Raw-output analysis compares diversity and Top-K PDMS before selection modules to assess whether generators consistently produce high-quality trajectories.The evaluation generates 20 trajectories per model and reports average waypoint diversity alongside Top-K PDMS.Source: §5.3 · pp. 7–8, Table 3 · p. 8
• DiffusionDriveV2 resolves the diversity–quality dilemma by constraining all modes while exploring better policies, improving both Top-10 and Top-1 PDMS.Source: §5.3 · p. 8

5.4. Ablation Studies

• Scale-adaptive multiplicative noise outperforms additive noise by addressing scale inconsistency between proximal and distal trajectory segments.Source: §5.4 · p. 8, Table 4 · p. 8
• Intra-Anchor GRPO is critical for Gaussian Mixture Models that partition action space by driving intention, while Inter-Anchor Truncated GRPO supplies globally comparable cross-mode information.Removing the inter-anchor component produces misleading advantage signals and inferior performance.Source: §5.4 · p. 8, Table 5 · p. 8, Table 6 · p. 8

• The framework achieves a better trade-off between consistent planning quality and mode diversity, with comprehensive experiments supporting state-of-the-art closed-loop performance.Source: §6 · pp. 8–9

• NAVSIM v1 uses PDMS as a closed-loop planning metric, while NAVSIM v2 extends it with EPDMS and additional compliance, lane-keeping, collision, and comfort measures.Source: §7 · p. 12
• The reinforcement-learning stage uses AdamW, a cosine schedule with warmup, 10 epochs, batch size 512, multiplicative-noise exploration, and denoising discount factor γ = 0.8.Source: §7 · p. 12
• A second 20-epoch stage trains a mode selector with trajectory-noise augmentation and 1% sampling from the fixed GTRS trajectory vocabulary.Source: §7 · p. 12
• The supplementary material includes a hyperparameter table for DiffusionDriveV2.Source: Table 7 · p. 12

• The two-stage coarse-to-fine selector improves performance by 0.2 PDMS over a single-stage selector.The coarse stage selects top-k trajectories before fine-grained comparison.Source: §8 · p. 12, Table 8 · p. 13
• An auxiliary ranking loss adds a further 0.2 PDMS by improving fine-grained discrimination among continuous metrics.Source: §8 · p. 12, Table 8 · p. 13
• DiffusionDriveV2’s superiority persists with matched selectors, although equipping DiffusionDrive with the same selector improves it by 1 PDMS.Source: §8 · p. 13, Table 9 · p. 13

• The qualitative study generates 20 trajectories per method and highlights the Top-1 and Top-10 candidates for comparison.Source: §9 · p. 13

Going straight scenarios

• In straight-driving scenarios, all DiffusionDriveV2 candidates are high quality, whereas Vanilla Diffusion collapses and DiffusionDrive produces colliding or off-road trajectories.Source: §9 · p. 13, Figure 4 · p. 14

Turning scenarios

• In turning scenarios, DiffusionDriveV2 preserves diverse behaviors while maintaining high quality across candidates, unlike DiffusionDrive’s delayed-turning and colliding trajectories.The Top-1 trajectory can follow conservatively, while Top-10 can execute aggressive overtaking.Source: §9 · p. 13, Figure 5 · p. 15

Multi-modal scenarios

• In multi-modal scenarios, DiffusionDriveV2 resolves the trade-off between diverse options and consistently high-quality trajectories.DiffusionDrive provides alternatives such as turning and going straight but does not guarantee consistent quality, while Vanilla Diffusion collapses.Source: §9 · p. 13, Figure 6 · p. 17