PaperlayerView exact revision on arXiv

Source-linked AI summary

DIVER: Reinforced Diffusion Breaks Imitation Bottlenecks in End-to-End Autonomous Driving

Ziying Song, Lin Liu, Hongyu Pan, Bencheng Liao, Mingzhe Guo, Lei Yang, Yongchang Zhang, Shaoqing Xu, Caiyan Jia, Yadan Luo

arXiv:2507.04049v5cs.CVcs.RO

TL;DR

Single-expert imitation can constrain diverse trajectory generation, and diffusion methods trained with such supervision can still exhibit mode collapse. DIVER combines conditioned diffusion with reinforcement learning to generate diverse, feasible trajectories; experiments report enhanced trajectory diversity and planning robustness across closed- and open-loop benchmarks.

  • Problem

    Single-expert imitation constrains diverse trajectory generation, and diffusion planners trained with that supervision can still produce trajectories clustered around a ground truth.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    DIVER combines diffusion-based trajectory generation conditioned on map elements and surrounding agents with reinforcement-learning reward guidance for safety and diversity.

    Source: §1 · p. 2

  • Results

    DIVER significantly enhances trajectory diversity and planning robustness across closed-loop Bench2Drive and NAVSIM and open-loop nuScenes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DIVER addresses the mode-collapse issue associated with imitation learning in multi-mode trajectory prediction.

    Source: §1 · p. 2

    Ensuring the correctness of trajectory selection remains a challenge for DIVER.

    Source: §6 · p. 15

Abstract

from arXiv · show

Most end-to-end autonomous driving methods rely on imitation learning from single expert demonstrations, often leading to conservative and homogeneous behaviors that limit generalization in complex real-world scenarios. In this work, we propose DIVER, an end-to-end driving framework that integrates reinforcement learning with diffusion-based generation to produce diverse and feasible trajectories. At the core of DIVER lies a reinforced diffusion-based generation mechanism. First, the model conditions on map elements and surrounding agents to generate multiple reference trajectories from a single ground-truth trajectory, alleviating the limitations of imitation learning that arise from relying solely on single expert demonstrations. Second, reinforcement learning is employed to guide the diffusion process, where reward-based supervision enforces safety and diversity constraints on the generated trajectories, thereby enhancing their practicality and generalization capability. Furthermore, to address the limitations of L2-based open-loop metrics in capturing trajectory diversity, we propose a novel Diversity metric to evaluate the diversity of multi-mode predictions.Extensive experiments on the closed-loop NAVSIM and Bench2Drive benchmarks, as well as the open-loop nuScenes dataset, demonstrate that DIVER significantly improves trajectory diversity, effectively addressing the mode collapse problem inherent in imitation learning.

1 INTRODUCTION

Imitation-based planning, including diffusion methods trained on a single expert trajectory, can leave multi-mode predictions clustered around one ground truth. DIVER combines diffusion generation with reinforcement learning and introduces a Diversity Metric; experiments report improved diversity and planning robustness across benchmarks.

1 INTRODUCTION

  • Imitation-based single-mode planning misses action diversity, while multi-mode methods can still cluster around one ground-truth trajectory.The authors attribute this diversity bottleneck to training on a single expert trajectory.Source: Figure 1 · p. 1, §1 · p. 2
  • Diffusion-based approaches remain constrained when their optimization follows imitation learning on a single expert trajectory.The introduction says this can leave generated modes converging toward one ground truth.Source: §1 · p. 2
  • DIVER's Policy-Aware Diffusion Generator conditions on map elements and surrounding agents, while reinforcement-learning rewards guide generation toward safety and diversity.The framework also introduces a Diversity Metric to evaluate multi-mode trajectory predictions.Source: §1 · p. 2
  • DIVER significantly improves trajectory diversity, safety, and feasibility over state-of-the-art methods across Bench2Drive, NAVSIM, and nuScenes.The introduction also reports improved planning robustness and says DIVER addresses imitation learning's mode-collapse issue.Source: §1 · p. 2

2 RELATED WORK

The related work describes end-to-end driving, diffusion-based trajectory generation, and reinforcement learning as intersecting directions. It positions DIVER as combining diffusion-based generation with reinforcement learning for task-aware trajectory planning.

2.1 End-to-end Autonomous Driving

  • Existing multi-mode planning methods do not fundamentally resolve imitation learning's mode collapse, while DIVER combines diffusion generation with reinforcement learning.The section also reviews selection, filtering, distillation, and trajectory-momentum approaches.Source: §2.1 · p. 2

2.2 Diffusion Models for Autonomous Driving

  • Diffusion models have been applied to trajectory distributions, scenario synthesis, BEV representation, and motion prediction, but diffusion research in end-to-end driving remains limited.Examples include DiffusionDrive, DiffScene, DiffBEV, MotionDiffuser, VDT-Auto, and DiFSD.Source: §2.2 · p. 3

2.3 Reinforcement Learning for Autonomous Driving

  • Prior autonomous-driving reinforcement-learning work spans urban driving, expert supervision for imitation agents, photorealistic simulation, and planning with reasoning.AlphaDrive is described as integrating GRPO-based reinforcement learning with planning and reasoning.Source: §2.3 · p. 3

3 PRELIMINARY

The preliminary section describes multi-mode trajectory prediction and explains why single-ground-truth imitation objectives can collapse predicted modes. It motivates a hybrid imitation- and reinforcement-learning framework with diversity-aware rewards.

3.1 End-to-End Autonomous Driving

  • Multi-mode planning predicts several future trajectories, from which a final trajectory is selected using scoring functions; their diversity can represent possible driving strategies.The passage gives six waypoints over a three-second horizon at 2 Hz as a common dataset setup.Source: §3.1 · p. 3

3.2 Diffusion Models for Driving Policy Diversity

  • Diffusion planners trained with single-ground-truth imitation supervision can still collapse their outputs around that trajectory rather than represent genuine multimodality.The analysis describes the resulting distribution as having near-zero diversity and motivates DIVER's hybrid IL-RL training with diversity-aware rewards.Source: §3.2 · pp. 3–4
  • The theoretical setup treats imitation learning as maximizing the likelihood of the observed trajectory, reducing the per-input data distribution to a single trajectory.The passages state that this concentrates probability mass on the unique trajectory; diffusion denoising likewise reconstructs the single ground truth from noisy perturbations.Source: §3.2 · p. 4

4.1 Overall Architecture

DIVER combines imitation learning and reinforcement learning in an end-to-end perception and motion-planning framework. Its planner uses conditioned diffusion to generate trajectory candidates and reinforcement-learning rewards to align them with safety and diversity objectives.

4.1 Overall Architecture

  • DIVER combines imitation learning and reinforcement learning, with a perception module feeding a motion planner that outputs a distribution over future trajectories.The framework addresses single-demonstration imitation limits by using multiple reference ground truths alongside trajectory anchors in its Policy-Aware Diffusion Generator.Source: §4.1 · p. 4
  • The Policy-Aware Diffusion Generator conditions trajectory generation on map elements and surrounding agents, while reinforcement-learning rewards guide outputs toward safety and diversity.The framework is described as aiming for physically feasible and behaviorally safe trajectories in complex scenarios.Source: §4.1 · p. 4

4.2 Policy-Aware Diffusion Generator (PADG)

PADG uses conditional diffusion and scene context to generate diverse, feasible trajectories. Forward noise, conditional feature fusion, and reference-GT guidance support distinct, goal-directed modes.

4.2.2 Conditional Diffusion Decoder

  • PADG iteratively denoises trajectory noise using scene semantics and a dual-branch architecture that models predicted and expert trajectories.Trajectory-aware pooling samples multiscale image features along the noisy trajectory; the decoder then attends to agent, map, and navigation-level information.Source: §4.2 · p. 4, §4.2.2 · pp. 5–6

4.2 Policy-Aware Diffusion Generator (PADG)

  • PADG is DIVER’s core conditional denoising component for generating diverse, feasible trajectories from noise using scene semantics and predicted and expert trajectory branches.The model conditions generation on map, agent, and anchor-trajectory features.Source: §4.2 · p. 4, §4.2.2 · p. 5

4.2.1 Diverse Multi-Mode Trajectory Forward Diffusion

  • Forward diffusion adds Gaussian noise to predicted modes and the GT trajectory, enabling denoising across noise levels while preserving trajectory structure.Noise on predicted modes encourages broader latent-space exploration; noise on the GT supports supervised denoising grounded in realistic motion.Source: §4.2.1 · pp. 4–5

4.2.3 Reference GTs-Guided Multi-Mode Trajectories.

  • Reference GTs guide each predicted mode toward a distinct feasible motion intent, while Hungarian matching discourages collapse toward a mean or dominant mode.At inference, candidate trajectories are scored for comfort and scene compliance, and the highest-scoring trajectory is selected.Source: §4.2.3 · pp. 6–7

4.3 Reinforcement Learning for Trajectory Planning

DIVER treats diffusion as a stochastic policy and uses GRPO to guide trajectory generation with rewards for diversity, safety, consistency, and lane keeping. The method combines trajectory-level reinforcement learning with imitation learning and adds online PDMS-based guidance on NAVSIM.

4.3.1 Overview of Group Relative Policy Optimization

  • GRPO is framed for multi-agent or multi-mode scenarios, updating policies relative to a shared group baseline using normalized group rewards for the advantage.The objective uses a clipped policy ratio with hyperparameters.Source: §4.3.1 · p. 7

4.3.2 GRPO for Trajectory Planning

  • GRPO updates the diffusion policy using trajectory-level rewards, enabling optimization of non-differentiable objectives and exploration beyond expert demonstrations.The framework targets mode collapse and missing safety awareness associated with single-GT supervision and absent task-specific guidance.Source: §4.3.2 · p. 7
  • On NAVSIM, online PDMS-based reinforcement guidance uses driving-quality metrics including collisions, drivable-area compliance, time-to-collision, comfort, and ego progress.Unlike Hydra-MDP's static metric distillation, DIVER integrates PDMS-based rewards during training.Source: §4.3.2 · p. 7
  • Pairwise trajectory dissimilarity rewards broader behavior coverage, while a distance-based safety reward penalizes paths too close to obstacles.The safety cost uses a map encoding inverse proximity to obstacles and a safety-margin threshold.Source: §4.3.2 · p. 7
  • Trajectory-consistency and lane-keeping rewards address temporal shifts and frequent or unnecessary lane changes across planning cycles.The lane-keeping rule uses a lane-centerline distance threshold and consecutive compliant waypoints.Source: §4.3.2 · p. 8
  • The total reward combines diversity, safety, trajectory consistency, and lane keeping, while the training objective balances trajectory matching with reinforcement learning.The authors describe this reward design as scalable and extensible to different objectives and human values.Source: §4.3.2 · p. 8

5 EXPERIMENTS

The experiments span closed-loop Bench2Drive and NAVSIM evaluations and open-loop datasets, with metrics covering driving performance, collisions, and multi-mode diversity. The proposed Diversity Metric is designed to compare trajectory variety across scenes and scenarios.

5.1 Dataset

  • Bench2Drive evaluation uses 950 of 1,000 clips for training and 50 for open-loop validation, with closed-loop testing on 220 routes across 44 interactive scenarios.NAVSIM is a real-world planning-oriented dataset emphasizing dynamic changes in driving intentions and excluding stationary or constant-speed cruising cases.Source: §5.1 · p. 8
  • Open-loop experiments use nuScenes, Turning-nuScenes, Adv-nuSc, and nuScenes-C, including adversarial-agent and weather-corruption tests.Turning-nuScenes focuses on turning scenarios; nuScenes-C evaluates Rain, Snow, and Fog corruptions.Source: §5.1 · pp. 8–9

5.2 Evaluation Metrics

  • Closed-loop metrics include route success, driving score, efficiency, comfort, multi-ability, and NAVSIM collision, compliance, comfort, and progress measures.Open-loop evaluation on nuScenes, Adv-nuSc, Turning-nuScenes, and nuScenes-C adopts collision rate.Source: §5.2 · p. 9
  • The Diversity Metric addresses L2 distance's inability to capture multi-mode variety and is designed to be scale-invariant, modality-invariant, and bounded in [0, 1].The passage describes L2 as measuring proximity to a single ground-truth trajectory.Source: §5.2 · p. 9
  • Diversity is computed over predicted modes and waypoints at future timestamps, then normalized by mean trajectory magnitude for comparability across scenes.The computation includes a small constant to avoid division by zero.Source: §5.2 · p. 9
  • Higher Diversity indicates more dispersed, well-separated predicted trajectories, while values near zero indicate similar modes; the metric applies only to multi-mode methods.The passage gives lane changes, braking, and yielding as examples of intent modes.Source: §5.2 · p. 9

5.3 Implementation Details

Across Bench2Drive, NAVSIM, and nuScenes, DIVER reports stronger closed- and open-loop planning results, including higher trajectory diversity. Its open-loop evaluation uses a Diversity Metric because L2 distance to one ground-truth trajectory does not capture the spread of multi-mode predictions.

5.3 Implementation Details

  • Experiments compare DIVER with E2E-AD baselines across Bench2Drive, nuScenes, and Adv-nuSc, using dataset-specific backbones, inputs, and planning-query configurations.Source: §5.3 · pp. 9–10

5.4.1 Close-Loop Results

  • Against VADmmt, DIVER raises Success Rate by 11.8%, Comfortness by 8.6%, Driving Score by 4.1%, and Efficiency by 4.0%; DiffAD has higher Success Rate (38.64% vs. 36.75%).Source: Table 1 · p. 10, §5.4.1 · p. 11
  • DIVER's Bench2Drive Driving Score is 68.90 versus DiffAD's 67.92, and its multi-ability mean is 42.09 versus 38.79.Source: Table 1 · p. 10, §5.4.1 · p. 11
  • With C&L inputs, DIVER reaches PDMS 88.3 on NAVSIM v1 while maintaining full comfort (100), with gains mainly driven by DAC and TTC.Source: Table 2 · p. 10, §5.4.1 · p. 11
  • On NAVSIM v2 navhard, DIVER attains EPDMS 26.8, outperforming LTF and DiffusionDrive and remaining close to GuideFlow under perturbed Stage 2 evaluation.Source: Table 4 · p. 10, §5.4.1 · p. 12

5.4.2 Open-Loop Results

5.4 Main Results

  • The authors avoid conventional open-loop L2 evaluation because distance to a single ground-truth trajectory misses the diversity and spread of multi-mode predictions.Source: §5.4 · pp. 10–11

5.5 Robustness Study

DIVER reports improved diversity and safety across long-horizon, turning-heavy, adversarial, and noisy-weather evaluations. Its results include lower collision rates than the cited baselines in the specified robustness settings.

Robustness Study

  • For 6-second nuScenes planning, DIVER's Diversity Metric averages 0.62, exceeding SparseDrive (0.43) and DiffusionDrive (0.49), while its 6-second collision rate is lowest at 1.91%.Source: §5.5 · p. 12
  • In turning-heavy scenarios, DIVER's average diversity is 0.31 versus 0.20 for DiffusionDrive and 0.19 for MomAD, and its average collision rate is 0.11 versus MomAD's 0.22.Source: §5.5 · p. 12
  • On nuScenes-C, DIVER matches the best baseline collision rate in clean scenes (0.05%) and has lower rates under snow, rain, and fog than SparseDrive and DiffusionDrive.Its average collision rates under snow/rain/fog are 0.09%/0.15%/0.21%.Source: §5.5 · p. 13
  • On Adv-nuSc, DIVER has the lowest average collision rate across horizons (0.752%), versus 3.95% for UniAD, 7.05% for VAD, 2.06% for SparseDrive, and 1.67% for DiffusionDrive.Source: §5.5 · p. 13

5.6 Ablation Study

The ablations assess PADG conditioning and reference guidance, loss and reward choices, and the TrajPooler module. The visualization discussion also reports broader trajectory spreads and describes remaining long-tail failure cases.

5.6 Ablation Study

  • On nuScenes, replacing the standard L1 objective with matching loss Lmatch yields consistently higher Div.(t), while GRPO gives stronger and more stable gains than PPO on NAVSIM v1.Source: §5.6 · p. 13
  • On Bench2Drive, combining map/agent conditioning and reference ground truths achieves the best overall results, while increasing reference ground truths improves Driving Score, Success Rate, and Merging until performance stabilizes around 6–7.Source: §5.6 · p. 13
  • TrajPooler improves NC by 1.2% and PDMS by 0.4% over full image features while focusing perception on trajectory-related regions and reducing computational cost.Source: §5.6 · p. 13
  • The reported Bench2Drive results show average diversity increasing from 0.21 to 0.35 (+66.7%) and success rate rising from 16.71% to 21.56% (+29.1%).Source: §5.6 · p. 14

5.7 Visualization

  • Figures 8 and 9 show DIVER producing a broader spread of trajectories than DiffusionDrive and SparseDrive in representative turning scenarios.Source: §5.7 · p. 14, Figure 9 · p. 14
  • Figure 10 illustrates suboptimal planning in long-tail conditions such as nighttime driving, where reduced visibility and rare interactions remain challenging.Source: §5.7 · p. 14, Figure 10 · p. 14

6 CONCLUSION

DIVER combines diffusion models with reinforcement learning to generate diverse trajectories and reports stronger diversity and safety across multiple benchmarks. The conclusion identifies correct trajectory selection as an unresolved challenge.

6 CONCLUSION

  • The conclusion presents DIVER as addressing planning-model collapse in imitation learning through diverse trajectory generation using diffusion and reinforcement learning.Source: §6 · p. 14
  • The paper reports superior diversity and safety performance across Bench2Drive, NAVSIM, and nuScenes relative to state-of-the-art approaches.Source: §6 · p. 14
  • Correct selection among generated trajectories remains a challenge; future work proposes jointly optimizing generation and decision-making with a Vision-Language-Action model.Source: §6 · pp. 14–15

Related papers

Loading 2507.04049v5…