PaperlayerView exact revision on arXiv

Source-linked AI summary

$AutoDrive\text{-}P^3$: Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning

Yuqi Ye, Zijian Zhang, Junhong Lin, Shangkun Sun, Changhao Peng, Wei Gao

arXiv:2603.28116v1cs.ROcs.CV

TL;DR

VLM-based autonomous-driving systems often bypass staged reasoning or handle perception, prediction, and planning independently, weakening their integration. AutoDrive-P^3 addresses this with unified P^3-CoT reasoning and hierarchical P^3-GRPO supervision, achieving state-of-the-art performance across open-loop and closed-loop benchmarks while reducing nuScenes collision rate by 40%.

  • Problem

    Existing VLM driving systems either omit CoT reasoning between perception and planning or treat perception, prediction, and planning as separate tasks, limiting their integration.

  • Method

    AutoDrive-P^3 combines the P^3-CoT dataset with hierarchical, progressive P^3-GRPO supervision across perception, prediction, and planning, alongside detailed and fast thinking modes.

  • Results

    40% lower collision rate on nuScenes, with state-of-the-art performance on NAVSIMv1/v2 and nuScenes.

  • Takeaways & Limitations

    The framework establishes progressive connections among perception, prediction, and planning through unified reasoning chains and multi-task supervision.

Abstract

from arXiv · show

Vision-language models (VLMs) are increasingly being adopted for end-to-end autonomous driving systems due to their exceptional performance in handling long-tail scenarios. However, current VLM-based approaches suffer from two major limitations: 1) Some VLMs directly output planning results without chain-of-thought (CoT) reasoning, bypassing crucial perception and prediction stages which creates a significant domain gap and compromises decision-making capability; 2) Other VLMs can generate outputs for perception, prediction, and planning tasks but employ a fragmented decision-making approach where these modules operate separately, leading to a significant lack of synergy that undermines true planning performance. To address these limitations, we propose ${AutoDrive\text{-}P^3}$, a novel framework that seamlessly integrates $\textbf{P}$erception, $\textbf{P}$rediction, and $\textbf{P}$lanning through structured reasoning. We introduce the ${P^3\text{-}CoT}$ dataset to facilitate coherent reasoning and propose ${P^3\text{-}GRPO}$, a hierarchical reinforcement learning algorithm that provides progressive supervision across all three tasks. Specifically, ${AutoDrive\text{-}P^3}$ progressively generates CoT reasoning and answers for perception, prediction, and planning, where perception provides essential information for subsequent prediction and planning, while both perception and prediction collectively contribute to the final planning decisions, enabling safer and more interpretable autonomous driving. Additionally, to balance inference efficiency with performance, we introduce dual thinking modes: detailed thinking and fast thinking. Extensive experiments on both open-loop (nuScenes) and closed-loop (NAVSIMv1/v2) benchmarks demonstrate that our approach achieves state-of-the-art performance in planning tasks. Code is available at https://github.com/haha-yuki-haha/AutoDrive-P3.

1 INTRODUCTION

AutoDrive-P^3 addresses limitations in VLM-based autonomous driving by unifying perception, prediction, and planning through structured, staged reasoning and collaborative supervision. The framework combines P^3-CoT data with P^3-GRPO training and achieves state-of-the-art performance across open-loop and closed-loop benchmarks.

  • Traditional autonomous-driving pipelines decouple perception, prediction, and planning, causing error accumulation that degrades final trajectory quality.
  • Current VLM systems either directly output trajectories without CoT or treat perception, prediction, and planning separately, limiting reasoning and task synergy.
  • P^3-GRPO provides hierarchical and progressive supervision across all three modules, improving perception, prediction, reasoning coherence, and planning reliability.
  • AutoDrive-P^3 unifies perception, prediction, and planning in an end-to-end framework that explicitly captures their relationships.
  • The framework achieves state-of-the-art performance across multiple open-loop and closed-loop autonomous-driving benchmarks.

2 RELATED WORK

Related work traces autonomous driving from modular pipelines toward end-to-end and VLM-based systems, while GRPO-based methods increasingly target reasoning improvement.

  • End-to-end systems jointly integrate perception, prediction, and planning to improve planning performance compared with decoupled modular designs.
  • VLM-based driving approaches use large-scale pretraining, world knowledge, and reasoning capabilities to address compact models’ limited semantic understanding of complex environments.
  • GRPO has been applied to VLM reasoning and uses group-level optimization to improve reasoning capabilities through varied sampled response paths.

3 PRELIMINARIES

The preliminary formulation models autonomous driving as autoregressive trajectory prediction from ego state, sensor data, commands, and historical trajectory information. AutoDrive-P^3 uses structured perception, prediction, and planning reasoning, while GRPO optimizes sampled responses without a value function.

  • End-to-end driving maps ego state, sensor data, and commands to a trajectory represented by sequential ego-vehicle positions.
  • The trajectory distribution is autoregressively factorized, conditioning each position on current inputs and previously generated positions.
  • The structured outputs include object perception labels, predicted future actions, and planned trajectory points.
  • The model organizes reasoning into perception, prediction, and planning stages, producing structured answers and step-by-step rationales.
  • GRPO improves learning stability by removing the value function and optimizing a group-level, sample-wise objective with normalized response advantages.
  • The GRPO objective combines a clipped surrogate loss with a KL penalty and leverages diverse sampled responses to improve reasoning.

4 METHODOLOGY

AutoDrive-P^3 integrates perception, prediction, and planning through unified CoT sequences and hierarchical reinforcement learning. Its methodology combines P^3-CoT supervised cold-start training with P^3-GRPO rewards that jointly optimize structured outputs across all three stages.

  • P^3-CoT Dataset: P^3-CoT links perception, prediction, and planning in unified CoT sequences rather than fragmented question-and-answer pairs.The dataset is designed for VLM training and GRPO post-training, with key-object labels and detailed reasoning annotations.
  • P^3-CoT Dataset: The dataset construction samples existing data and annotations, filters key-object labels using rules and manual review, and uses an advanced VLM to construct connected CoT reasoning.The construction process focuses on the relationships among the three autonomous-driving stages.
  • Training Effects: P^3-CoT is reported to improve both holistic sequential reasoning and modular accuracy and reliability across perception, prediction, and planning.Cold-start SFT produces interpretable P^3-CoT outputs that form the basis for subsequent reinforcement learning.
  • Supervised Fine-Tuning: Supervised fine-tuning maps multimodal inputs to perception, prediction, and planning outputs, with each module containing a reasoning trace followed by an answer.The unified target is y = [yperception; yprediction; yplanning], where each module is organized as [ythinking; yanswer].
  • P^3-GRPO Algorithm: After cold-start SFT, P^3-GRPO jointly applies reinforcement learning to perception, prediction, and planning using coordinated rewards for structured outputs.The algorithm samples response groups, parses their three components, computes rewards, normalizes advantages, applies a KL penalty, and updates the policy.
  • P^3-GRPO Algorithm: The integrated reward is a weighted sum of format, perception, prediction, and planning terms, explicitly encoding that perception supports prediction and both support planning.Perception reward uses IoU, precision, and recall; prediction reward combines behavior-label correctness with IoU and detection quality; planning reward uses L2 distance.

5 EXPERIMENTS

Experiments evaluate AutoDrive-P^3 on open-loop nuScenes and closed-loop NAVSIM benchmarks, comparing planning performance, ablations, training settings, and inference modes. The framework achieves strong benchmark results, while P^3-GRPO improves staged reasoning and planning reliability.

  • 5.1 BENCHMARKS: nuScenes evaluates planning with L2 displacement error and collision rate, while NAVSIM uses closed-loop PDMS and EPDMS metrics.nuScenes contains 1,000 real-world driving sequences; NAVSIMv2 adds compliance, lane-keeping, comfort, and related components.
  • 5.3 COMPARISON WITH STATE-OF-THE-ART METHODS: 90.6 PDMS and 89.9 EPDMS are achieved on NAVSIMv1/v2 with vision-only input.The method also reports comparable L2 performance with a smaller model and less training data than cited methods, alongside the best collision rate.
  • 5.4 ABLATION STUDY: P^3-GRPO improves perception and prediction while significantly boosting planning relative to SFT and planning-only GRPO.The ablation compares SFT, SFT plus planning GRPO, and SFT plus P^3-GRPO against UniAD and OmniDrive.
  • 5.4 ABLATION STUDY: Increasing GRPO group size, adding historical trajectories, and using video sensors improve performance, with the full configuration optimal across metrics.The study attributes the gains respectively to more diverse reasoning samples, contextual understanding, and temporal dynamics.
  • Runtime and Dual thinking modes: Dual thinking modes balance efficiency and performance, and vLLM acceleration enables near-real-time inference at 1 Hz on an H100 GPU.The fast mode is designed for efficiency while retaining the P^3-CoT structure.
  • Qualitative Results: Qualitative visualizations show accurate pedestrian perception, safe passage prediction, and trajectories aligned with human driving behavior in complex interactions.The method avoids overly conservative decisions in the pedestrian scenario and identifies key objects and behaviors in vehicle interactions.

6 CONCLUSION AND FUTURE WORK

The paper concludes that AutoDrive-P^3 connects perception, prediction, and planning through unified reasoning and achieves state-of-the-art results on its evaluated benchmarks. It also identifies hallucinations and offline-simulator training as limitations for future work.

  • Conclusion: AutoDrive-P^3 combines the P^3-CoT dataset, supervised fine-tuning, and P^3-GRPO for progressive perception–prediction–planning reasoning.The framework introduces dual detailed and fast thinking modes to balance inference efficiency with performance.
  • Conclusion: Experiments on NAVSIMv1/v2 and nuScenes report state-of-the-art performance, including a 40% reduction in nuScenes collision rate.The paper presents dual thinking modes alongside these benchmark results.
  • Limitations and Future Work: The method remains limited by hallucinations during reasoning and reinforcement learning conducted in offline simulators without real-world interaction.Future work targets hallucination mitigation, lower inference time, and closed-loop deployment.

8 ETHICS STATEMENT

The authors state that the study follows the ICLR Code of Ethics and relevant dataset usage guidelines. They report no human subjects and describe efforts to avoid biased or discriminatory results.

  • Ethics Statement: The study involves no human subjects and uses nuScenes and NAVSIM under relevant usage guidelines and public licenses.The ethics statement also says the authors took care to avoid biased or discriminatory results.

9 REPRODUCIBILITY STATEMENT

The paper describes public code and datasets, detailed experimental settings, and author responsibilities intended to support reproducibility. It also documents dataset motivation, limitations of prior annotations, and P^3-CoT distributions.

  • Reproducibility Statement: Code and datasets are publicly available, with training steps, model configurations, and hardware details described for replication.The authors state that the experimental description is intended to facilitate verification and reproducibility.
  • Author Contributions: The paper identifies Yuqi Ye, Zijian Zhang, and Wei Gao as core authors, with Ye and Zhang as co-first authors and Gao as corresponding author.The contribution statement assigns framework, code, experiments, metrics, visualization, labeling, and checking responsibilities.
  • P^3-CoT Motivation: P^3-CoT is motivated by human-like driving that focuses on key objects and links perception, prediction, and planning stages.The paper argues that existing datasets provide fragmented or incomplete supervision for these staged connections and key-object impacts.
  • Dataset Distribution: Figures 7 and 8 report the category and action distributions in the P^3-CoT nuScenes dataset.The supplied figure captions identify the two distributions but do not state their numerical values.

C.2 DATA COLLECTION

P 3-CoT is constructed from sampled nuScenes and NAVSIM scenes with key-object annotations and connected perception, prediction, and planning reasoning.

  • The dataset samples key frames at 2 Hz, using 3 seconds of history as input and 3 seconds of future trajectory as ground truth.
  • Perception localizes key objects, prediction reasons about their future behavior, and planning uses both to produce final waypoints.
  • Key-object counts are summarized per frame, while prediction and planning action distributions are documented in dedicated tables and figures.
  • P 3-CoT contains 19,284 training frames and 6,019 test frames from nuScenes, plus 103,288 training frames and 12,146 test frames from NAVSIM.

P 3-CoT is composed of a training set of 19284 frames and a validation set of 5119 frames, attached with detailed annotations for perception, prediction and planning three stages with connection.

P^3-CoT combines perception, prediction, and planning annotations in a connected CoT format, with data organized to support staged interpretation and unified training.

  • The dataset annotates key objects and future actions for perception, prediction, and planning across autonomous-driving scenes.
  • P^3-CoT includes perception, prediction, and planning information while maintaining close connections among the three stages.
  • P^3-CoT uses a CoT format that preserves sequential reasoning from perception through prediction to planning.
  • The dataset comparison is structured around labeled frames, stage coverage, and data organization format.

D REWARD SETTING

The reward hierarchy evaluates perception, prediction, and planning progressively, linking accurate intermediate reasoning to final trajectory quality. Experiments indicate that balanced rewards and mutual reinforcement improve the overall planning objective.

  • Perception Reward: The perception reward combines matched-box IoU, precision, and recall to evaluate key-object detection and localization.
  • Prediction Reward: The prediction reward links behavioral-label accuracy with spatial localization, requiring predicted actions to correspond to accurately detected objects.
  • Planning Reward: The planning reward evaluates the ego vehicle’s trajectory using distance to ground-truth trajectories and incorporates PDMS for NAVSIM.
  • Reward Hierarchy: Planning is treated as the apex of the reward hierarchy, while perception and prediction provide necessary prerequisites for final driving decisions.
  • Reward Visualization: Training with P^3-CoT and P^3-GRPO produces consistent improvement in perception, prediction, planning, and all rewards.
  • Reward Weight Ablation: An unbalanced 1:1:1:7 configuration that overemphasizes planning hinders perception and prediction optimization and reduces overall planning performance.

E EXPERIMENTAL SETUP

Experiments evaluate P^3-GRPO on nuScenes and NAVSIM under consistent data settings, including an ablation of KL-divergence regularization.

  • Benchmark Setup: Experiments compare Cold-Start and P^3-GRPO on nuScenes and NAVSIM using consistent data settings.
  • KL-Divergence Ablation: The experimental setup includes an ablation that removes the KL-divergence term from the training objective.
  • KL-Divergence Ablation: The model without KL regularization suffers significant performance degradation as training progresses.

F QUALITATIVE COMPARISON OF TRAJECTORY PLANNING

The qualitative comparisons show AutoDrive-P^3 linking scene understanding, object-action prediction, and trajectory planning across nuScenes and NAVSIM examples. Its detailed CoT cases produce trajectories that better follow the stated ground-truth actions than competing methods in several challenging scenes.

  • Visualization setup: The visualization compares ground truth with detailed-CoT, fast-CoT, and competing trajectories for nuScenes and NAVSIM samples.For nuScenes, the competitors are OmniDrive and VAD; for NAVSIM, they are WoTE and DiffusionDrive.
  • nuScenes examples: In the night waiting-car scene, AutoDrive-P^3 identifies the closest vehicles, predicts their actions, and stops like the ground-truth trajectory.Comparison methods mistakenly move forward after ignoring the front car.
  • nuScenes examples: In the roadside-truck scene, AutoDrive-P^3 identifies key objects and provides good prediction answers despite its trajectory differing from ground truth.The passage notes that comparison methods do not work well in this sample.
  • Visualization setup: The examples include completed perception, prediction, and planning questions and answers for two nuScenes samples and four NAVSIM samples.The corresponding CoT and answers are displayed alongside each specific sample.
  • CoT examples: The detailed example detects a car and carries its scene interpretation into future-action prediction and ego-vehicle planning.The planning reasoning uses the vehicle speed, current action, and historical trajectory.
Loading 2603.28116v1…