Diffusion-Based Planning for Autonomous Driving with Flexible Guidance

Yinan Zheng; Ruiming Liang; Kexin Zheng; Jinliang Zheng; Liyuan Mao; Jianxiong Li; Weihao Gu; Rui Ai; Shengbo Eben Li; Xianyuan Zhan; Jingjing Liu

Source-linked AI summary

Diffusion-Based Planning for Autonomous Driving with Flexible Guidance

Yinan Zheng, Ruiming Liang, Kexin Zheng, Jinliang Zheng, Liyuan Mao, Jianxiong Li, Weihao Gu, Rui Ai, Shengbo Eben Li, Xianyuan Zhan, Jingjing Liu

arXiv:2501.15564v2Jan 26, 2025cs.ROcs.AIcs.LG

TL;DR

Autonomous-driving planners must handle multimodal behavior, safety, adaptability, and closed-loop quality without depending heavily on predefined rules. The paper introduces a transformer-based Diffusion Planner that jointly models prediction and planning and uses classifier guidance for behavioral alignment. It reports state-of-the-art closed-loop performance on nuPlan and a 200-hour delivery-vehicle dataset, with transferability across driving styles.

Problem
Existing learning-based planners have limited adaptability, struggle with multimodal driving behavior and safety, and often rely on rule-based refinement.
Method
Diffusion Planner uses a transformer architecture to jointly model prediction and planning through diffusion, with classifier guidance for safe or preferred driving styles.
Results
Diffusion Planner achieves state-of-the-art closed-loop performance on nuPlan and a 200-hour delivery-vehicle dataset, demonstrating adaptability across diverse driving styles.
Takeaways & Limitations
The approach provides a learning-based planner that generates high-quality trajectories without relying on rule-based refinement and can adapt behavior at runtime.
Takeaways & Limitations
Classifier guidance requires a pre-defined differentiable energy function to calculate guidance energy.

Abstract

from arXiv · show

Achieving human-like driving behaviors in complex open-world environments is a critical challenge in autonomous driving. Contemporary learning-based planning approaches such as imitation learning methods often struggle to balance competing objectives and lack of safety assurance,due to limited adaptability and inadequacy in learning complex multi-modal behaviors commonly exhibited in human planning, not to mention their strong reliance on the fallback strategy with predefined rules. We propose a novel transformer-based Diffusion Planner for closed-loop planning, which can effectively model multi-modal driving behavior and ensure trajectory quality without any rule-based refinement. Our model supports joint modeling of both prediction and planning tasks under the same architecture, enabling cooperative behaviors between vehicles. Moreover, by learning the gradient of the trajectory score function and employing a flexible classifier guidance mechanism, Diffusion Planner effectively achieves safe and adaptable planning behaviors. Evaluations on the large-scale real-world autonomous planning benchmark nuPlan and our newly collected 200-hour delivery-vehicle driving dataset demonstrate that Diffusion Planner achieves state-of-the-art closed-loop performance with robust transferability in diverse driving styles.

1 INTRODUCTION

The paper identifies limited adaptability, weak handling of multimodal driving behavior, safety challenges, and reliance on rule-based refinement in existing planners. It introduces Diffusion Planner to address these issues through diffusion modeling and flexible guidance.

Existing learning-based planners struggle to model the multimodal behaviors exhibited by human drivers.
Out-of-distribution scenarios can produce low-quality model outputs, often requiring rule-based trajectory refinement or filtering.
Auxiliary safety losses can create multiobjective conflicts and poor safety performance because they lack signals for recovery from mistakes.
Diffusion Planner learns trajectory score gradients to model multimodal driving distributions without rule-based refinement.
The proposed classifier guidance mechanism adapts planning behavior to different needs without additional training.
The paper reports state-of-the-art nuPlan performance, personalized runtime behavior, and a new 200-hour delivery-vehicle dataset.

2 RELATED WORK

Related work spans rule-based, imitation-learning, and diffusion-based approaches to autonomous planning. The paper positions Diffusion Planner against their limitations in adaptability, multimodal behavior modeling, and closed-loop trajectory quality.

Rule-based planners are controllable and interpretable but struggle with novel complex situations beyond predefined rules.
Behavior-cloning planners model human driving directly, but lack theoretical guarantees for multimodal behavior and can accumulate closed-loop errors.
Transformer-based behavior-cloning methods improve fitting efficiency but do not resolve the underlying multimodality limitation.
Prior diffusion work in prediction and simulation emphasizes open-loop performance or diversity rather than directly controlled planning quality and drivability.

3 PRELIMINARIES

The preliminaries describe diffusion as score-based generative modeling and classifier guidance as a way to steer sampling toward preferred outputs. These mechanisms motivate flexible trajectory generation for autonomous driving.

Closed-loop planning integrates real-time perception, prediction, and control, unlike open-loop planning or motion prediction under static conditions.
Diffusion models generate data by reversing a Markov chain that progressively adds Gaussian noise to the original data.
The noise schedule uses α_t and σ_t so the noised state approaches N(0, I) as t approaches 1.
A neural network learns the probability score ∇_x(t) log q_t(x(t)), enabling diffusion models to represent complex distributions.
Classifier guidance modifies the diffusion score using a classifier gradient to generate preferred data.
In autonomous driving, guidance offers flexibility by improving the model directly rather than relying on human-designed post-processing rules.

4 METHODOLOGY

Diffusion Planner reformulates autonomous planning as joint future-trajectory generation for the ego and neighboring vehicles, then uses a transformer architecture and classifier guidance to produce controllable driving behavior. The method integrates vehicle, historical, lane, navigation, and static-object information while supporting safety, comfort, speed, and drivable-area preferences.

4.1 TASK REDEFINITION: The task jointly generates ego planning and neighboring-vehicle predictions, enabling cooperative behavior modeling under shared scenario conditions.Conditions include current vehicle states, historical data, lane information, and navigation information.
4.1 TASK REDEFINITION: Diffusion modeling is adopted because behavior cloning struggles to represent the complex distributions arising from joint multi-vehicle trajectory generation.The planner predicts trajectories for the nearest M neighboring vehicles and the ego vehicle simultaneously.
4.2 DIFFUSION PLANNER: Diffusion Planner uses a DiT-based architecture that fuses noised future trajectories with conditional information through transformer attention mechanisms.Vehicle states are integrated with trajectory inputs, while contextual representations are aggregated and fused through self-attention and cross-attention.
4.2 DIFFUSION PLANNER: Historical vehicle status and lane information are encoded with separate MLP-Mixer networks, then aggregated with static-object and other representations by a transformer encoder.The resulting encoder representation is fused with the trajectory representation.
4.2 DIFFUSION PLANNER: Navigation route information is transformed into guidance representations and injected with the diffusion timestep condition to guide trajectory generation.Route lanes are represented using coordinates, and adaptive layer normalization applies the resulting guidance across tokens.
4.3 PLANNING BEHAVIOR ALIGNMENT VIA CLASSIFIER GUIDANCE: Classifier guidance modifies the trajectory score during inference to encode safety or preferred behaviors through differentiable energy functions.Supported energies cover target speed, comfort, collision avoidance, and staying within the drivable area; guidance combinations are training-free and flexible.

5 EXPERIMENTS

Experiments evaluate Diffusion Planner against rule-based, learning-based, and hybrid baselines on nuPlan and a delivery-vehicle dataset, alongside qualitative, guidance, ablation, and inference studies. The results indicate strong trajectory quality, multimodal behavior, flexible guidance, and transferability.

Evaluation Setup: Evaluations use nuPlan closed-loop benchmarks in both non-reactive and reactive modes, with scores averaged across scenarios from 0 to 100.The compared baselines include rule-based, learning-based, and hybrid methods, with an existing refinement module used for additional comparisons.
Main Results: Diffusion Planner achieves state-of-the-art performance across more nuPlan benchmarks than learning-based baselines, while its refined version surpasses hybrid and rule-based baselines.The refined model also exceeds reported human performance, according to the main results discussion.
Transferability: Diffusion Planner transfers strongly to delivery-vehicle driving, unlike benchmark-tailored PDM, GameFormer, and PLUTO methods whose performance drops on that task.The delivery-vehicle evaluation uses a newly collected dataset and tests adaptability across different driving behaviors.
Qualitative Results: Qualitatively, Diffusion Planner generates smooth ego trajectories and accurate neighboring-vehicle predictions, whereas several baselines produce low-quality samples or rely on refinement.The comparison includes a narrow-road turning scenario and evaluates future planning, neighboring-vehicle predictions, and ground-truth ego trajectories.
Multi-modal Planning Behavior: Multiple inferences from an intersection produce distinct left-turn, right-turn, and straight-ahead behaviors, while navigation information directs the model toward a left turn.This evaluates multimodal planning without low-temperature sampling from the same initial position.
Guidance and Design Studies: Inference-time classifier guidance adjusts target speed and comfort without additional training, including speed matching between 10m/s and 14m/s and reduced discomfort.Comfort guidance can be applied simultaneously with collision guidance; training and inference studies also examine normalization, augmentation, predicted-vehicle count, denoising steps, and sampling magnitude.

6 CONCLUSION

The paper presents Diffusion Planner as a transformer-based diffusion approach for high-quality closed-loop planning with flexible guidance. It reports state-of-the-art performance without rule-based refinement on nuPlan and a 200-hour delivery-vehicle dataset, with adaptability across driving styles.

6 CONCLUSION: Diffusion Planner jointly models multimodal motion prediction and planning through a diffusion objective in a transformer-based architecture.Classifier guidance aligns planning with safe or user-preferred driving styles.
6 CONCLUSION: Diffusion Planner achieves state-of-the-art closed-loop performance without rule-based refinement on nuPlan and a newly collected 200-hour delivery-vehicle driving dataset.The conclusion presents this result as evidence of adaptability across diverse driving styles.

A VISUALIZATION OF CLOSED-LOOP PLANNING RESULTS

The visualization shows closed-loop planning over time, combining ego-vehicle plans, neighboring-vehicle predictions, ground-truth ego trajectory, and ego driving history.

A VISUALIZATION OF CLOSED-LOOP PLANNING RESULTS: Each row depicts a scenario at 0, 5, 10, and 15 seconds, with future plans, neighboring-vehicle predictions, ground truth, and driving history shown together.The visualization supports comparison of planned and predicted trajectories as the closed-loop scenario unfolds.

B.1 COMPARED TO DIFFUSION-BASED PLANNING METHODS

The paper compares Diffusion Planner with other diffusion-based planning methods and discusses practical deployment considerations. The comparison emphasizes diffusion-based planning quality and computational cost.

B.1 COMPARED TO DIFFUSION-BASED PLANNING METHODS: Diffusion-es, STR-16M, and STR2-CPKS-800M use diffusion differently, including trajectory filtering, decoding, and PDM-like refinement.The compared methods also differ in their use of LLMs, parameter scale, and refinement modules.
B.1 COMPARED TO DIFFUSION-BASED PLANNING METHODS: Some diffusion-based planners have higher computational costs because they rely on LLMs or large numbers of model parameters, limiting real-world deployability.The paper contrasts this practicality concern with Diffusion Planner’s stated practical orientation.

C EXPERIMENTAL DETAILS

The experiments use nuPlan data, diffusion-specific training and sampling settings, and differentiable guidance energies for collision avoidance, speed, comfort, and drivable-area adherence.

Datasets: The nuPlan training set contains 1 million sampled scenarios with lane, navigation, and two-second neighboring-vehicle histories within 100 meters.Inputs are padded to a unified dimension, with attention masking removing irrelevant information.
Data augmentation: Data augmentation perturbs ego position, orientation, speed, and acceleration, then uses quintic interpolation to generate dynamically constrained trajectories.The augmented speed is constrained to remain positive, preventing reverse-motion learning.
Training: Diffusion training uses 8 NVIDIA A100 GPUs, batch size 2048, 500 epochs, AdamW, and a 5e−4 learning rate.A five-epoch warmup phase is also used.
Diffusion sampling: DPM-Solver++ with a variance-preserving noise schedule and low-temperature sampling is used for the diffusion reverse process.The authors report that temperature choice affects trajectory stability depending on whether refinement is applied.
Classifier guidance: Classifier guidance defines energies for collision avoidance, target-speed maintenance, comfort, and staying within the drivable area.The collision term produces gradients within a collision-sensitive distance, while the drivable-area term uses a differentiable signed-distance cost map.
Guidance design: The guidance design prioritizes smooth continuous gradients, sparse intervention, and indirect guidance for higher-order derivatives.The selected energy functions are intended mainly to validate diverse guidance types and may not be optimal.

D DETAILS ON DELIVERY VEHICLE EXPERIMENTS

The delivery-vehicle experiments use approximately 200 hours of real-world logistics data, converted into nuPlan-compatible format and evaluated with adapted, retrained baselines.

Dataset: The delivery-vehicle dataset contains approximately 200 hours of real-world autonomous logistics driving data.The vehicle follows designated routes while complying with traffic regulations, maintaining safety, and completing deliveries efficiently.
Data and evaluation setup: The data are transformed into nuPlan DB files, and the nuPlan training pipeline is used for both Diffusion Planner and the baselines.Crosswalk information is replaced with stop-line data for some baselines, and vehicle parameters are changed to match the delivery vehicle.
Evaluation: Delivery-vehicle evaluation modifies vehicle-related parameters and retrains and tests the models using the official training code.The same metrics and model families used on nuPlan are adopted.

E LIMITATIONS & DISCUSSIONS & FUTURE WORK

The discussion identifies limits in input representation, lateral flexibility, and sampling efficiency, while reporting broad closed-loop capability and suggesting end-to-end visual inputs as future work.

Scenario Inputs: The planner relies on vectorized maps and neighboring-vehicle detections, which lose information relative to end-to-end pipelines and require a data-processing module.The authors frame this choice as consistent with their focus on closed-loop planning.
Scenario Inputs: Future work could modify the encoder to accept images and enable an end-to-end training pipeline.The paper demonstrates closed-loop diffusion planning without rule-based refinement before proposing this extension.
Lateral Flexibility: Learning-based methods struggle with large lateral movements because the dataset contains few lane-change or avoidance examples.The planned-trajectory output can also diverge from downstream controller behavior, and augmentation only somewhat alleviates the issue.
Sample Efficiency: Diffusion planning requires multiple model inferences, reducing sample efficiency, although a high-order ODE solver enables 8-second planning at 10 Hz in 0.05 seconds.Consistency models and distillation-based sampling are suggested for further improvement.
Overall Discussion: The authors report extensive evidence for diffusion-based closed-loop planning and potential alignment with safe or human-preferred driving behaviors.They characterize the resulting planner as high-performance and adaptable within the demonstrated scope.