PaperlayerView exact revision on arXiv

Source-linked AI summary

Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuning

Zhiyu Huang, Xinshuo Weng, Maximilian Igl, Yuxiao Chen, Yulong Cao, Boris Ivanovic, Marco Pavone, Chen Lv

arXiv:2410.05582v1cs.RO

TL;DR

Autonomous driving planners must reason about interactions and uncertainties, but prediction and deterministic planning can leave the ego vehicle isolated from the social scene. Gen-Drive generates joint futures, evaluates them with a preference-trained reward model, and reports stronger performance than learning-based planning approaches, with further gains from RL fine-tuning.

  • Problem

    Separated prediction and deterministic planning can isolate the ego vehicle from social context, while planning with generative models also needs reliable scenario evaluation and probable futures from fewer samples.

    Source: §I · p. 1

  • Method

    Gen-Drive combines a diffusion model generating joint futures for all agents, a VLM-assisted pairwise-preference-trained scene evaluator, and RL fine-tuning of the generator.

    Source: §I · p. 2

  • Results

    Gen-Drive outperforms learning-based planning methods, and its planning performance improves further with RL fine-tuning.

    Source: B. Main Results · p. 6, §V · p. 6

  • Takeaways & Limitations

    Even with single-sample planning, the fine-tuned policy's overall planning score exceeds that of the untuned multiple-sample approach.

    Source: B. Main Results · p. 6

    The framework does not yet incorporate a perception module for sensor inputs; adding one is identified as future work.

    Source: §V · p. 6

Abstract

from arXiv · show

Autonomous driving necessitates the ability to reason about future interactions between traffic agents and to make informed evaluations for planning. This paper introduces the \textit{Gen-Drive} framework, which shifts from the traditional prediction and deterministic planning framework to a generation-then-evaluation planning paradigm. The framework employs a behavior diffusion model as a scene generator to produce diverse possible future scenarios, thereby enhancing the capability for joint interaction reasoning. To facilitate decision-making, we propose a scene evaluator (reward) model, trained with pairwise preference data collected through VLM assistance, thereby reducing human workload and enhancing scalability. Furthermore, we utilize an RL fine-tuning framework to improve the generation quality of the diffusion model, rendering it more effective for planning tasks. We conduct training and closed-loop planning tests on the nuPlan dataset, and the results demonstrate that employing such a generation-then-evaluation strategy outperforms other learning-based approaches. Additionally, the fine-tuned generative driving policy shows significant enhancements in planning performance. We further demonstrate that utilizing our learned reward model for evaluation or RL fine-tuning leads to better planning performance compared to relying on human-designed rewards. Project website: https://mczhi.github.io/GenDrive.

I. INTRODUCTION

Gen-Drive replaces separated prediction and deterministic planning with joint future-scene generation followed by learned evaluation. It trains the evaluator from VLM-assisted pairwise preferences and fine-tunes the diffusion generator with reinforcement learning.

I. INTRODUCTION

  • RL fine-tuning uses the learned reward model to improve diffusion generation for planning tasks.The paper frames fine-tuning as a response to the need for more probable planning scenarios with fewer samples.Source: §I · pp. 1–2
  • The reward model is trained on pairwise preferences for generated scenarios curated through a VLM-assisted pipeline.The pipeline is presented as a way to reduce human workload and improve scalability.Source: §I · pp. 1–2
  • The nuPlan closed-loop evaluation reports favorable performance for the diffusion driving policy, particularly after fine-tuning.The paper also reports significant planning-performance improvements after RL fine-tuning.Source: §I · p. 2
  • Gen-Drive generates diverse, scene-consistent futures for all agents and uses a learned reward model to evaluate their quality for planning.The framework's three contributions are a multi-agent trajectory diffusion model, a VLM-feedback preference-trained reward model, and an RL fine-tuning pipeline.Source: §I · p. 2

II. RELATED WORK

The related work describes limitations in prediction-planning and reward-evaluation approaches, motivating joint future-scene generation and preference-based reward modeling.

A. Prediction and Planning for Autonomous Driving

  • Integrated prediction-planning methods still isolate the ego vehicle from social interaction contexts and struggle with multimodal predictions and deployment complexity.The paper proposes generating diverse future scenarios for all agents, including the ego, to support scene consistency and capture interaction dynamics.Source: A. Prediction and Planning for… · p. 2

B. Reward Modeling for Autonomous Driving

  • Human-crafted metrics may diverge from human-like behavior, while inverse reinforcement learning depends on reward-structure assumptions that may not reflect preferences across conditions.Pairwise human preferences are described as a promising basis for learned reward models and fine-tuning generative models [17], [26].Source: B. Reward Modeling for Autonomous… · p. 2

C. Generative Models for Autonomous Driving

The paper positions diffusion as a way to generate interactive multi-agent futures for planning, then pairs a joint-action scene generator with a learned evaluator. The generator encodes scene context and iteratively denoises joint actions into future states.

C. Generative Models for Autonomous Driving

  • Gen-Drive uses diffusion to generate multi-agent future scenarios for planning, unlike prior diffusion planning work focused on ego-agent trajectory optimization.The paper situates diffusion among generative models used for traffic simulation and prediction, citing CTG++ [12], VBD [10], and Diffusion-ES.Source: C. Generative Models for Autonomous… · p. 2

III. METHOD

  • The framework generates joint behavior for the ego and other agents, then uses a scene evaluator trained on pairwise preferences to score scenarios and guide selection.The learned reward model also fine-tunes diffusion generation toward high-reward plans.Source: §III · pp. 2–3, Figure 2 · p. 3

A. Scene Generator

  • The scene generator encodes historical object trajectories and map polylines, then denoises joint object actions conditioned on the scene and ego route.It starts from Gaussian noise, reverses diffusion steps, and translates the resulting actions into states containing position, heading, and velocity.Source: A. Scene Generator · p. 3

B. Scene Evaluator

The scene evaluator processes generated futures with map context to produce both ego planning trajectories and scores for selecting among scenes.

B. Scene Evaluator

  • The evaluator encodes each generated future scene together with the vector map into a scene representation.It uses the ego planning encoding as a query over the scene elements to form a planning-centric feature.Source: B. Scene Evaluator · p. 3
  • Two MLP heads reconstruct ego planning trajectories and output scores for the generated scenes.The trajectory-reconstruction head is an auxiliary task intended to improve stability and effectiveness.Source: B. Scene Evaluator · p. 3
  • Multiple future scenarios can be generated in parallel from a batch of Gaussian noise before evaluator scoring.The evaluator handles S generated scenarios alongside the vector map.Source: B. Scene Evaluator · p. 3

C. Training Base Diffusion Model

The base diffusion model learns to recover clean future trajectories from noised scene-level inputs. The reward model is trained on pairwise preferences, with VLM assistance used to make preference-data collection more efficient and scalable.

C. Training Base Diffusion Model

  • The base diffusion model is trained to recover clean trajectories from noised joint trajectories under different noise levels and scene conditions.A shared noise level perturbs all object trajectories at each training step.Source: C. Training Base Diffusion Model · p. 4

D. Training Reward Model

  • VLM-assisted collection uses pairwise comparisons to reduce the workload of human annotators labeling planning preferences.Human-designed metrics may not reflect actual human values across diverse scenarios, while assigning reward values to scenes is difficult even for human evaluators.Source: D. Training Reward Model · p. 4
  • They extract 32 five-second anchor goals using K-means, then use guided diffusion to generate 32 diverse ego-planning trajectories with reactive behaviors from other scene objects before pairwise scenario sampling.Discrepancies between planned trajectories, collisions, and off-road behavior filter obvious failures; GPT-4o evaluates scenario pairs when those measures are insufficient to distinguish them.Source: D. Training Reward Model · p. 4
  • The reward model is trained on accepted and rejected scenes sampled from the same initial conditions, using the pairwise preference reward dataset.The model R predicts a score for a generated scene.Source: D. Training Reward Model · p. 4
  • The trained reward model produces reasonable scores for generated plans and scenes.Fig. 4 presents examples of scene-evaluation outputs.Source: D. Training Reward Model · p. 4

E. Fine-tuning Generation Model

The paper fine-tunes the diffusion model with reinforcement learning, treating denoising as a multi-step decision process and using the trained reward model to guide generation. Fine-tuning also includes a trajectory reconstruction term and updates only the denoiser.

E. Fine-tuning Generation Model

  • The diffusion denoising process is formulated as a multi-step MDP, with the denoiser acting as a policy conditioned on noise at each step.A reward is emitted at the end of the K-step trajectory, and DDPO is used to refine the generative policy.Source: E. Fine-tuning Generation Model · p. 4
  • A regression term is added to keep the denoised trajectory from deviating from the original trajectory.The loss is accumulated over the full diffusion trajectory; only the denoiser is learnable during fine-tuning, while the encoder remains fixed.Source: E. Fine-tuning Generation Model · p. 5
  • The fine-tuning algorithm samples generation sequences, scores their scenes with the reward model, and updates the generation model.The procedure starts from a pre-trained generation model and outputs a fine-tuned generation model.Source: E. Fine-tuning Generation Model · p. 5

F. Implementation Details

Implementation details specify the model’s scene and architecture dimensions, training schedule, and reinforcement-learning fine-tuning settings.

F. Implementation Details

  • The base diffusion model uses 6 encoding and 6 Transformer decoding layers with hidden dimension 256, and generates scenes for 50 nearby objects over 5 seconds.The evaluator has 3 query-centric attention layers and 3 cross-attention layers; diffusion uses 10 steps and a cosine noise schedule.Source: F. Implementation Details · p. 5
  • The base model is trained for 20 epochs, the reward model for 50 epochs, and diffusion fine-tuning runs for 1,000 steps.The reward dataset contains 5,000 scenarios with 50 pairwise comparisons each; training jobs use 8 NVIDIA A100 GPUs.Source: F. Implementation Details · p. 5

A. Experimental Setup

The experiments use OpenScene data derived from nuPlan and evaluate planning on the nuPlan closed-loop benchmark’s reduced Val14 set.

A. Experimental Setup

  • Base-model training uses 450,000 OpenScene segments, each with a 2-second history and 5-second future, using vectorized trajectories and maps.OpenScene is a compact nuPlan subset sampled at 2 Hz; 5,000 test-split scenarios are selected for reward-model training and generation fine-tuning.Source: A. Experimental Setup · p. 5
  • Closed-loop planning is tested on the nuPlan reduced Val14 benchmark with non-reactive agents at 1 Hz, using an iLQR trajectory controller.Evaluation metrics are overall planning score, collision score, and progress score, with higher scores indicating better performance.Source: A. Experimental Setup · p. 5

B. Main Results

Gen-Drive’s generative planner is evaluated against learning-based predictive planners and a rule-based planner, while additional results compare multiple-sample planning and RL fine-tuning. The reported results favor generation-and-evaluation over learning-based predictive planners, and show benefits from fine-tuning and learned reward evaluation.

B. Main Results

  • Table I presents closed-loop planning results on the nuPlan reduced Val14 benchmark.Figure 5 illustrates diverse plans and interaction outcomes, and compares base and fine-tuned policies.Source: Table I · p. 5, B. Main Results · p. 5, Figure 5 · p. 6
  • Multiple-sample planning evaluates 16 scenes in parallel with a learned reward model, improving plan diversity and overall planning score over single-sample inference.Average inference time is 282.5 ms for single-sample planning and 483.6 ms for multiple-sample planning on an RTX 4090 GPU; the paper reports both meet real-time requirements.Source: B. Main Results · pp. 5–6
  • Fine-tuning lets the single-sample policy exceed the overall planning score of the unfine-tuned multiple-sample approach; learned-reward fine-tuning also beats the PDM-based scorer.The fine-tuned policy is reported to yield and avoid off-road plans in the illustrated scenarios.Source: B. Main Results · p. 6, Figure 5 · p. 6
  • Gen-Drive outperforms learning-based predictive planners, while PDM-Closed achieves the highest scores but is optimized for nuPlan metrics that may lack human likeness and real-world adaptability.The comparison covers PlanTF, GameFormer, and DTPP.Source: B. Main Results · p. 6

C. Ablation Studies

The ablations identify 50 modeled objects and 1,000 RL fine-tuning steps as preferred settings. The conclusion reports stronger planning performance than other learning-based methods, with further enhancements through RL fine-tuning.

C. Ablation Studies

V. CONCLUSIONS

  • Gen-Drive with proper reward modeling outperforms other learning-based planning methods, and RL fine-tuning further enhances planning performance.Source: §V · p. 6
Loading 2410.05582v1…