PaperlayerView exact revision on arXiv

Source-linked AI summary

PlannerRFT: Reinforcing Diffusion Planners through Closed-Loop and Sample-Efficient Fine-Tuning

Hongchen Li, Tianyu Li, Jiazhi Yang, Haochen Tian, Caojun Wang, Lei Shi, Mingyang Shang, Zengrong Lin, Gaoqiang Wu, Zhihui Hao, Xianpeng Lang, Jia Hu, Hongyang Li

arXiv:2601.12901v1cs.RO

TL;DR

Diffusion planners struggle to provide multi-modal, scenario-adaptive exploration for reinforcement fine-tuning, limiting the usefulness of reward signals. PlannerRFT combines policy-guided denoising with dual-branch optimization and accelerated simulation, achieving state-of-the-art performance while producing safer, more efficient behaviors.

  • Problem

    Diffusion planners can suffer modality collapse and limited scenario-adaptive exploration, restricting effective reinforcement fine-tuning.

  • Method

    PlannerRFT uses policy-guided denoising, dual optimization of trajectory and exploration policies, closed-loop rollouts, and the GPU-parallel nuMax simulator.

  • Results

    PlannerRFT achieves state-of-the-art nuPlan performance, including +1.66 points on Val14 and +2.99 points on Test14-hard in reactive traffic settings.

  • Takeaways & Limitations

    The experiments associate PlannerRFT’s gains with effective exploration, improved safety and robustness, and distinct human-like driving behaviors.

  • Takeaways & Limitations

    PlannerRFT is verified with structured abstract inputs rather than sensory observations such as images, leaving visuomotor applicability underexplored.

Abstract

from arXiv · show

Diffusion-based planners have emerged as a promising approach for human-like trajectory generation in autonomous driving. Recent works incorporate reinforcement fine-tuning to enhance the robustness of diffusion planners through reward-oriented optimization in a generation-evaluation loop. However, they struggle to generate multi-modal, scenario-adaptive trajectories, hindering the exploitation efficiency of informative rewards during fine-tuning. To resolve this, we propose PlannerRFT, a sample-efficient reinforcement fine-tuning framework for diffusion-based planners. PlannerRFT adopts a dual-branch optimization that simultaneously refines the trajectory distribution and adaptively guides the denoising process toward more promising exploration, without altering the original inference pipeline. To support parallel learning at scale, we develop nuMax, an optimized simulator that achieves 10 times faster rollout compared to native nuPlan. Extensive experiments shows that PlannerRFT yields state-of-the-art performance with distinct behaviors emerging during the learning process.

1. Introduction

Diffusion planners can model human-like trajectories but suffer from limited exploration and objective misalignment during reinforcement fine-tuning. PlannerRFT addresses this with policy-guided, scenario-adaptive denoising, closed-loop optimization, and accelerated simulation.

  • Diffusion planners learn human-like driving trajectories from demonstrations but remain vulnerable to distributional shift and objective misalignment.
  • Vanilla diffusion planners collapse different noise inputs into nearly identical trajectories, limiting the optimization signal available to reinforcement fine-tuning.
  • PlannerRFT combines energy-based guidance with an exploration policy that adapts lateral and longitudinal guidance scales to scenario context.
  • The framework uses closed-loop interaction and PPO to guide exploration toward temporally consistent, safe, and efficient behaviors.
  • nuMax provides GPU-accelerated simulation for high-throughput parallel rollouts, supporting scalable reinforcement fine-tuning.
  • PlannerRFT achieves state-of-the-art nuPlan performance with improved safety, robustness, and distinct human-like driving behaviors.

2. Related Work

Diffusion planners support flexible trajectory generation and continuous-action reinforcement learning, but modality collapse restricts exploration. PlannerRFT targets this limitation with adaptive guidance rather than fixed or scenario-agnostic denoising strategies.

  • Diffusion models provide flexible guidance-based denoising and are suited to continuous-action reinforcement learning.
  • PlannerRFT uses policy-guided denoising to generate multi-modal, scenario-adaptive trajectories for more effective reinforcement optimization.
  • Rule-based guidance can impose fixed strength and competing gradient signals, causing performance variability across driving scenarios.
  • Modality collapse makes diffusion planners generate similar trajectories, restricting exploration and hindering policy adaptation during reinforcement fine-tuning.

3. Preliminary

The task is closed-loop motion planning for safe and feasible ego-vehicle trajectories. The planner uses a scene encoder and Diffusion Transformer decoder to iteratively denoise trajectory samples from structured scene inputs.

  • Motion planning generates safe and feasible ego-vehicle trajectories in dynamic driving environments.
  • This work fine-tunes an imitation-learning-pretrained diffusion planner to improve closed-loop safety, comfort, and efficiency.
  • The planner fuses surrounding agents, map features, and static obstacles into a scene representation, alongside an encoded navigation command.
  • A Diffusion Transformer decoder iteratively denoises noisy trajectory samples conditioned on the encoded environment and navigation inputs.

4. Method

PlannerRFT improves diffusion-planner exploration through guided denoising, closed-loop rollout, and dual optimization of trajectories and exploration. Its implementation combines context-conditioned guidance, survival rewards, GRPO, PPO, and accelerated simulation.

  • Overview: PlannerRFT freezes a global reference copy of the pretrained planner while adding an exploration policy and optimizing through closed-loop rollout and PPO.
  • Policy-guided Denoising: Energy-based guidance injects lateral and longitudinal residual offsets into denoising to produce diverse trajectories near a reference trajectory.
  • Policy-guided Denoising: Orthogonal lateral and longitudinal guidance gradients create multimodal trajectories through different combinations of their guidance scales.
  • nuMax Simulator: nuMax caches scenarios, calibrates tracking and scoring, and enables GPU-parallel rollouts with 10 times the speed of native nuPlan.
  • Policy-guided Denoising: The exploration policy predicts context-conditioned guidance distributions, allowing sampled scale pairs to represent distinct driving modalities.
  • Closed-Loop Rollout: Closed-loop rollout executes only the first action of a selected candidate trajectory before updating the simulator state and recording reward data.
  • Exploration Optimization: PPO optimizes exploration for temporally consistent, efficient, safe, and comfortable directions using long-term rewards propagated through GAE.
  • Trajectory Optimization: Survival rewards accumulate valid non-terminal rewards, encouraging trajectories that delay failure and improving exploration in hard scenarios.

5. Experiments

Experiments evaluate PlannerRFT across closed-loop benchmarks, qualitative scenarios, exploration policies, reward designs, data distributions, and guidance ranges. The results show improved performance and safer, more efficient, and more diverse behavior, while revealing important trade-offs in exploration and training-data composition.

  • Setup and Protocols: The evaluation uses nuPlan closed-loop benchmarks, including general Val14 and challenging Test14-hard settings, with a 5-step DDIM sampler for efficient stochastic exploration.The fine-tuning setup uses 144,494 scenarios and runs for 40M environment steps on eight NVIDIA H100 GPUs.
  • Main Results: PlannerRFT improves closed-loop performance across all four nuPlan benchmarks and achieves the best overall performance in three of four.Compared with the pretrained Diffusion Planner, gains are especially substantial in reactive settings: +1.66 points on Val14 and +2.99 points on Test14-hard.
  • Main Results: PlannerRFT’s qualitative behavior evolves from collision-prone lane changes to safe lane keeping and eventually decisive, efficient lane changes.After 10M fine-tuning steps, the planner avoids collision through lane keeping; after 25M steps, it achieves both safety and efficiency.
  • Ablation Study: Policy-guided denoising produces smooth lateral and longitudinal trajectory diversity while adapting exploration to driving context.Uniform exploration has the highest diversity but causes reward variance and instability, whereas fixed exploration restricts the search space and limits performance.
  • Ablation Study: A balanced Lt90 fine-tuning distribution performs best, while Fail training degrades broadly and All training provides weak optimization signals on hard scenarios.The IL-finetuned model performs worse on the same Lt90 data, supporting exploration-based learning rather than additional training iterations as the source of gains.
  • Ablation Study: Survival rewards improve hard-scenario optimization, and moderate guidance offsets provide a better exploration–exploitation trade-off than small or large offsets.Terminal rewards degrade on Test14-hard, short 2 s horizons underperform, and 4 s and 6 s horizons yield similar results.

6. Conclusion and Outlook

PlannerRFT improves closed-loop planning through effective, scenario-adaptive exploration and shows gains over IL fine-tuning without relying on additional training iterations. Its current scope is limited to planners using structured abstract inputs rather than sensory observations.

  • PlannerRFT improves closed-loop performance through effective exploration rather than additional training iterations.
  • Analyses of exploration policies highlight PlannerRFT’s scenario-adaptive advantage in sample efficiency.
  • PlannerRFT is verified on planners with structured abstract inputs, while applicability to visuomotor planners remains underexplored.

A. Discussions

The discussions attribute PlannerRFT’s effectiveness to guided exploration, closed-loop execution feedback, and trial-and-error adaptation, supported by distinct optimization branches and a scalable simulator.

  • What makes PlannerRFT effective?: Guided denoising reinforces lateral perturbations, enabling lane changes and obstacle avoidance in complex scenes.
  • What makes PlannerRFT effective?: Closed-loop rewards from vehicle-dynamics execution connect planning updates to collision and off-road outcomes, improving maneuver precision.
  • What makes PlannerRFT effective?: Trial-and-error closed-loop reinforcement learning adapts to dynamic traffic and improves interaction capability.
  • Why adopt a dual-branch optimization?: PPO updates the online exploration policy, while GRPO optimizes the high-dimensional trajectory generator through group-based updates.
  • Potential applications and future directions: nuMax supports the training pipeline as a fast online reinforcement-learning simulator for the nuPlan benchmark.

B. Implementation Details of PlannerRFT

PlannerRFT uses a two-module policy update around guided diffusion sampling: PPO trains exploration guidance, while GRPO fine-tunes the diffusion transformer toward higher-reward trajectories.

  • PlannerRFT performs reinforcement-learning sampling with policy-guided denoising using five DDIM steps for efficiency and exploration stochasticity.
  • The exploration policy and fine-tuned DiT are separate learnable modules with different optimization goals and losses.
  • PPO updates the exploration policy to maximize long-term cumulative reward in closed-loop planning.
  • GRPO updates the fine-tuned DiT to maximize reward within the prediction horizon at the current denoising timestep.
  • During inference, the unguided version uses the same five-step DDIM sampler without guided denoising or the reference planner.

C. Implementation Details of nuMax

nuMax is a JAX-based, GPU-parallel simulator designed for high-throughput nuPlan training, combining cached scenarios, closed-loop scoring, and a hybrid JAX–PyTorch distributed pipeline.

  • nuMax is a JAX-based GPU-parallel simulator built upon Waymax for large-scale closed-loop training on nuPlan.
  • Scenario Cache: Scenario caching reduces repeated database access by pre-caching temporal context and scene information for simulation.
  • Tracker and Scorer: The scorer combines terminal penalties with soft penalties to assess driving quality and safety during closed-loop rollouts.
  • Tracker and Scorer: Terminal violations immediately end simulation and set the reward to zero, while soft penalties cover non-terminal driving behavior.
  • Distributed RL Training Pipeline: nuMax retains PyTorch policy learning while executing JAX simulation on rank-0, distributing observations and replay samples across DDP ranks.
  • Limitations of nuMax: nuMax remains limited by XLA static-shape constraints and log-replay surrounding-vehicle simulation instead of IDM.

D. Additional Ablation Studies

PlannerRFT outperforms the pretrained planner on Test14-random and benefits from combining complementary guidance types. Larger GRPO groups improve performance, with group size 8 selected as a balance between performance and computational efficiency.

  • PlannerRFT achieves the best performance in Test14-random, improving over the pretrained planner by +1.62 (NR) and +3.17 (R).The result holds in both non-reactive and reactive settings.
  • Combining lateral and longitudinal guidance produces the best overall performance through complementary exploration.Lateral guidance helps Drivable and Comfort, while longitudinal guidance helps Collisions, TTC, Progress, and Speed.
  • The group-size ablation is reported on the Test14-random benchmark.
  • Larger GRPO group sizes outperform group size 4, with scores stabilizing around 72.21 R-score and 77.16 NR-score at sizes 8 or 12.The study selects group size 8 to balance performance and computational efficiency.

E. Additional Qualitative Results

Qualitative evaluations show PlannerRFT generating diverse, scenario-adaptive trajectories and safer maneuvers across challenging traffic situations. Compared with the IL-pretrained planner, it avoids or resolves collisions, off-road behavior, and blocked maneuvers through context-sensitive decisions.

  • Additional Visualization of Policy-Guided Denoising: Policy-guided denoising produces multi-modal, scenario-adaptive trajectories for sample-efficient reinforcement-learning training.The visualization compares Diffusion Planner, DiffusionDrive, and PlannerRFT over a 4-second planning horizon.
  • Qualitative Results on Safety-Critical Scenarios: PlannerRFT waits for pedestrians to finish crossing before completing an intersection right turn, whereas the IL-pretrained planner collides.
  • Qualitative Results on Safety-Critical Scenarios: PlannerRFT brakes at tsim = 1 s to avoid a stationary lead vehicle in reactive traffic, while the IL-pretrained planner collides.
  • Qualitative Results on Safety-Critical Scenarios: PlannerRFT changes lanes before reaching a stationary vehicle on an S-curve, avoiding the collision produced by lane keeping.
  • Qualitative Results on Safety-Critical Scenarios: PlannerRFT finely adjusts its trajectory between traffic cones, avoiding the collision caused by the IL-pretrained planner.
  • Qualitative Results on Safety-Critical Scenarios: PlannerRFT delays a lane change for a long vehicle and completes an unprotected right turn before oncoming traffic arrives.These maneuvers avoid collisions in blocked-right-turn and unprotected-right-turn scenarios.
  • Qualitative Results on Safety-Critical Scenarios: PlannerRFT proceeds straight after turning right instead of pulling over, avoiding the off-road behavior associated with causal confusion.The IL-pretrained behavior is linked to training scenarios where right turns are followed by passenger pickup.
Loading 2601.12901v1…