PaperlayerView exact revision on arXiv

Source-linked AI summary

$AutoDrive\text{-}P^3$: Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning

Yuqi Ye, Zijian Zhang, Junhong Lin, Shangkun Sun, Changhao Peng, Wei Gao

arXiv:2603.28116v1cs.ROcs.CV

TL;DR

Autonomous-driving systems often separate perception, prediction, and planning, despite their interdependence. AutoDrive-P^3 links these tasks through structured reasoning and achieves state-of-the-art benchmark performance, including a reported 40% collision-rate reduction on nuScenes.

  • Problem

    Autonomous-driving systems often decouple perception, prediction, and planning, while planning-only optimization can neglect their interdependence, which matters for trajectory quality.

    Source: §1 · pp. 1–2

  • Method

    AutoDrive-P^3 uses unified perception-prediction-planning reasoning, a P^3-CoT dataset, and hierarchical P^3-GRPO supervision to jointly optimize the three tasks.

    Source: §1 · pp. 2–3, §4 · p. 4

  • Results

    State-of-the-art performance on NAVSIMv1/v2 and nuScenes, with collision rate reduced by 40% on nuScenes.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The paper presents detailed and fast thinking modes as a way to balance inference efficiency with performance.

    Source: §1 · p. 3, §6 · p. 10

    The authors report hallucinatory phenomena during reasoning as a limitation of AutoDrive-P^3.

    Source: §6 · p. 10

Abstract

from arXiv · show

Vision-language models (VLMs) are increasingly being adopted for end-to-end autonomous driving systems due to their exceptional performance in handling long-tail scenarios. However, current VLM-based approaches suffer from two major limitations: 1) Some VLMs directly output planning results without chain-of-thought (CoT) reasoning, bypassing crucial perception and prediction stages which creates a significant domain gap and compromises decision-making capability; 2) Other VLMs can generate outputs for perception, prediction, and planning tasks but employ a fragmented decision-making approach where these modules operate separately, leading to a significant lack of synergy that undermines true planning performance. To address these limitations, we propose ${AutoDrive\text{-}P^3}$, a novel framework that seamlessly integrates $\textbf{P}$erception, $\textbf{P}$rediction, and $\textbf{P}$lanning through structured reasoning. We introduce the ${P^3\text{-}CoT}$ dataset to facilitate coherent reasoning and propose ${P^3\text{-}GRPO}$, a hierarchical reinforcement learning algorithm that provides progressive supervision across all three tasks. Specifically, ${AutoDrive\text{-}P^3}$ progressively generates CoT reasoning and answers for perception, prediction, and planning, where perception provides essential information for subsequent prediction and planning, while both perception and prediction collectively contribute to the final planning decisions, enabling safer and more interpretable autonomous driving. Additionally, to balance inference efficiency with performance, we introduce dual thinking modes: detailed thinking and fast thinking. Extensive experiments on both open-loop (nuScenes) and closed-loop (NAVSIMv1/v2) benchmarks demonstrate that our approach achieves state-of-the-art performance in planning tasks. Code is available at https://github.com/haha-yuki-haha/AutoDrive-P3.

1 INTRODUCTION

AutoDrive-P^3 unifies perception, prediction, and planning through structured reasoning and progressive supervision. The paper reports state-of-the-art performance across open- and closed-loop driving benchmarks and introduces detailed and fast thinking modes.

1 INTRODUCTION

  • Existing VLM driving systems may omit CoT, separate perception, prediction, and planning, or apply GRPO only to planning metrics, limiting synergy, interpretability, and planning reliability.The introduction also notes that traditional decoupled pipelines accumulate errors, while small-scale end-to-end models lack world knowledge and perform poorly in long-tail scenarios.Source: §1 · pp. 1–2
  • The authors argue that accurate perception supports reliable prediction, and both are foundations for robust planning, motivating GRPO supervision across the full chain rather than planning alone.Treating perception and prediction as byproducts neglects their interdependence and limits coherent reasoning across the driving pipeline.Source: §1 · p. 2
  • The framework combines structured perception-prediction-planning reasoning with P^3-CoT supervised fine-tuning and hierarchical P^3-GRPO reinforcement fine-tuning across all three tasks.P^3-GRPO explicitly supervises perception, prediction, and planning, while detailed and fast thinking modes balance inference efficiency with performance.Source: §1 · pp. 2–3
  • AutoDrive-P^3 achieves state-of-the-art performance across open-loop and closed-loop autonomous driving benchmarks, with P^3-GRPO improving planning through progressive supervision.Evaluations cover open-loop nuScenes and closed-loop NAVSIMv1/v2; the reported mechanism systematically improves perception and prediction to support more reliable planning.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work has advanced end-to-end driving and VLM-based reasoning, but gaps in world knowledge, task integration, and reinforcement of perception and prediction leave planning capabilities constrained.

2.1 END-TO-END AUTONOMOUS DRIVING METHODS

2.2 GROUP RELATIVE POLICY OPTIMIZATION

3 PRELIMINARIES

The paper formulates end-to-end driving as autoregressive prediction of ego positions from ego state, sensor data, and commands. AutoDrive-P^3 organizes perception, prediction, and planning into a structured reasoning chain, while GRPO supplies group-based policy optimization.

VLM-based End-to-end Autonomous Driving Problem Formulation

  • Ego trajectories are modeled as sequential future-position predictions conditioned on ego state, sensor data, commands, and prior positions, capturing temporal dependencies.The trajectory distribution is autoregressively factorized, and the formulation integrates all inputs in one end-to-end framework.Source: §3 · pp. 3–4
  • AutoDrive-P^3 processes video and ego data through structured perception-prediction-planning reasoning, producing step-by-step rationale and structured outputs for all three tasks.The illustrated chain includes separate reasoning and answer stages for perception, prediction, and planning.Source: Figure 2 · p. 4, §3 · p. 4

Group Relative Policy Optimization (GRPO)

  • GRPO removes value-function dependence and optimizes a group-level, sample-wise objective to improve learning stability (Shao et al., 2024).It samples response groups from the behavior policy, uses normalized advantages, and combines a clipped surrogate loss with a KL penalty; diverse model-generated responses expose reasoning to varied paths and solutions.Source: §3 · p. 4

4 METHODOLOGY

AutoDrive-P^3 combines a unified perception–prediction–planning CoT dataset, supervised cold-start fine-tuning, and hierarchical GRPO to train VLMs for coordinated autonomous-driving reasoning.

4.1 P 3-CoT DATASET

  • P^3-CoT links perception, prediction, and planning labels in coherent, manually verified reasoning chains generated with Qwen2.5-VL-72B (Bai et al., 2025).The dataset contains 25,303 frames from 850 nuScenes scenes and 115,434 frames from 1,382 NAVSIM scenes.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2 SUPERVISED FINE-TUNING FOR COLD-START

  • Cold-start SFT trains the VLM on multimodal inputs to produce structured perception, prediction, and planning outputs, each pairing reasoning with an answer.The resulting interpretable P^3-CoT outputs establish a foundation for reinforcement learning.Source: §4.2 · pp. 5–6

4.3 P 3-GRPO ALGORITHM

  • P^3-GRPO jointly optimizes perception, prediction, and planning with a multicomponent reward, encoding how perception supports prediction and both support planning.Its reward combines format, perception, prediction, and planning terms; perception uses detection quality, prediction combines behavior-label correctness with IoU and detection quality, and planning measures trajectory quality by L2 distance.Source: §4.3 · p. 6

5 EXPERIMENTS

AutoDrive-P^3 is evaluated on nuScenes and NAVSIMv1/v2, with reported state-of-the-art vision-only NAVSIM scores and gains from joint P^3-GRPO. The experiments also assess training settings, inference modes, and qualitative driving behavior.

5.1 BENCHMARKS

  • The nuScenes evaluation uses L2 displacement error and collision rate under ST-P3 metric settings, while NAVSIM evaluates closed-loop driving with PDMS and EPDMS.PDMS includes no-collision, drivable-area compliance, ego progress, time-to-collision, and comfort; EPDMS adds driving-direction and traffic-light compliance, with NAVSIMv2's human penalty filter enabled to suppress penalties for human-agent violations.Source: §5.1 · pp. 7–8

5.2 IMPLEMENTATION DETAILS

  • Training uses Qwen2.5-VL-3B, AdamW, and 8 A100 GPUs, with 10 epochs on each benchmark and different camera inputs, resolutions, and batch sizes.nuScenes uses six front-view frames from 3 seconds at 448×252 with ego speed and batch size 8; NAVSIM uses four frames from 2 seconds across front, front-left, and front-right views at 672×168, ego velocities and accelerations, and batch size 32. P^3-GRPO uses eight samples per scenario, reward weights in a 1:2:2:5 ratio, and adds PDMS to NAVSIM's planning reward. Fast mode gives module answers without reasoning, while detailed mode provides full reasoning and answers.Source: §5.2 · pp. 8–9

5.3 COMPARISON WITH STATE-OF-THE-ART METHODS

  • 90.6 PDMS on NAVSIMv1 and 89.9 EPDMS on NAVSIMv2 are reported as state-of-the-art vision-only results; nuScenes results match SOTA L2 and improve collision rate by about 40%.The comparison also reports comparable L2 scores with a smaller Qwen2.5-3B model than OmniDrive’s LLaVA-7B and less training data than OpenDriveVLA (20k vs. 1000k samples).Source: Table 2 · p. 8, Table 3 · p. 8, §5.3 · p. 9

5.4 ABLATION STUDY

  • P^3-GRPO improves perception and prediction over the baselines and significantly boosts planning, while planning-only GRPO reduces 3s collision rate to baseline performance.Only SFT already achieves a lower 2s collision rate than the UniAD and OmniDrive baselines; the joint method yields holistic gains across the staged tasks.Source: §5.4 · p. 9, Table 4 · p. 9
  • Increasing GRPO group size from 4 to 8, using historical trajectories, and using video rather than images each improve performance; the full configuration is optimal across metrics.The reported explanations are more diverse reasoning samples, improved contextual understanding, and captured temporal dynamics, respectively.Source: §5.4 · p. 9, Table 5 · p. 9
  • 1 Hz near-real-time inference is achieved with vLLM 0.8.0 acceleration on an H100 GPU.The paper compares inference times for the detailed and fast thinking modes against other methods.Source: §5.4 · p. 9, Figure 5 · p. 10

5.5 VISUALIZATION

  • In pedestrian scenarios, the method perceives location and predicts safe passage to avoid overly conservative decisions; in complex vehicle interactions, it identifies key objects and behaviors and produces human-like trajectories.These qualitative examples are shown in Figure 6.Source: §5.5 · p. 10, Figure 6 · p. 10

6 CONCLUSION AND FUTURE WORK

AutoDrive-P^3 connects perception, prediction, and planning through unified reasoning and hierarchical supervision, achieving state-of-the-art results while retaining limitations in hallucination and real-world interaction.

6 CONCLUSION AND FUTURE WORK

  • 40% lower collision rate on nuScenes accompanies state-of-the-art performance on NAVSIMv1/v2 and nuScenes.The conclusion attributes the improvement to AutoDrive-P^3.Source: §6 · p. 10
  • AutoDrive-P^3 combines the P^3-CoT dataset, domain-adaptation supervised fine-tuning, and P^3-GRPO hierarchical multi-task supervision to connect perception, prediction, and planning.The dataset provides unified reasoning chains and labels; the framework also offers detailed and fast thinking modes to balance efficiency and performance.Source: §6 · p. 10
  • Reasoning hallucinations and offline-simulator-only reinforcement learning remain limitations; future work targets hallucination mitigation, faster inference, and closed-loop deployment.The conclusion notes that the reinforcement learning lacks interaction with real-world environments.Source: §6 · p. 10

8 ETHICS STATEMENT

The authors report adhering to the ICLR Code of Ethics, using no human subjects or personally identifiable information, and following dataset usage guidelines and public licenses. They also state that they sought to avoid biased or discriminatory results and maintain transparency and integrity.

8 ETHICS STATEMENT

  • The study involved no human subjects and used no personally identifiable information, according to the authors.The authors state that the experiments raise no privacy or security concerns.Source: §8 · p. 11
  • The nuScenes (Caesar et al., 2020) and NAVSIM (Dauner et al., 2024; Cao et al., 2025) datasets were used under relevant usage guidelines and public licenses.The authors state that this ensured no violation of privacy.Source: §8 · p. 11
  • The authors state that they sought to avoid biased or discriminatory results and are committed to transparency and integrity.They report that all authors read and adhered to the ICLR Code of Ethics.Source: §8 · p. 11

9 REPRODUCIBILITY STATEMENT … C.1 MOTIVATION

The supplementary sections document reproducibility and authorship practices, then motivate P^3-CoT as a key-object-focused dataset for connected perception, prediction, and planning reasoning. They identify shortcomings in prior datasets and annotation formats that the dataset is intended to address.

9 REPRODUCIBILITY STATEMENT

  • The authors report public code and datasets, detailed experimental setup and method descriptions, and measures intended to support replication and consistent evaluation.They say the paper describes training steps, model configurations, and hardware details.Source: §9 · p. 11

A A USE OF LARGE LANGUAGE MODELS

  • Gemini 2.5 Pro and DeepSeek-R1 assisted with manuscript language polishing and format checking, while human authors formulated and verified scientific content and retained final responsibility.The authors state this was in accordance with ICLR policy.Source: A A USE OF LARGE LANGUAGE MODELS · p. 15

B CONTRIBUTIONS

  • Yuqi Ye and Zijian Zhang are co-first authors; Ye led framework, code, experiments, and data processing, Zhang handled evaluation and data tasks, and Wei Gao is corresponding author.All authors participated in discussion and writing.Source: B CONTRIBUTIONS · p. 15

C P 3-CoT DATASET

  • The P^3-CoT dataset section introduces its motivation, data collection, composition and distribution, and comparisons with other datasets.The passage presents these as the topics covered in the section.Source: C P 3-CoT DATASET · p. 15

C.1 MOTIVATION

  • P^3-CoT is motivated as a VLM reasoning dataset focused on key objects and a unified perception-prediction-planning chain, rather than unnecessary object identification and localization.The authors argue that key-object focus supports efficient, safe driving reasoning and that the dataset should include detailed CoT annotations.Source: §C.1 · p. 15
  • Existing object-grounding and CoT datasets are described as insufficient: grounding too many objects can add noise, while prior CoT work overlooks staged reasoning and links between stages.The passage also notes that DriveLM recognizes key objects and connections among thinking steps.Source: §C.1 · pp. 15–16
  • DriveLM (Sima et al., 2024) lacks clear key-object definitions and impacts, complete frame and localization annotations, and flexible CoT-compatible formats, motivating three dataset challenges.Those challenges are comprehensive key-object annotations, unified perception-prediction-planning CoT, and a VLM-suitable format beyond question-answer pairs.Source: §C.1 · p. 16

C.2 DATA COLLECTION · C.3 DATA COMPOSITION

P^3-CoT is built around human-attention-inspired key objects and a unified perception–prediction–planning reasoning chain, with manually checked annotations and synthesized CoT data. Its nuScenes and NAVSIM splits support both supervised fine-tuning and reinforcement post-training.

C.2 DATA COLLECTION

  • The dataset defines key objects as those human drivers attend to prevent danger, and uses them as the focus of perception, prediction, and planning.Perception localizes key objects in RGB images; prediction reasons about their future behavior; planning uses objects and actions from both stages to produce waypoints.Source: §C.2 · p. 16
  • Frames are sampled at 2 Hz with 3 seconds of history and 3 seconds of future trajectory as ground truth, with object bounding boxes projected into the image view.Future waypoints are converted to ego-car coordinates for ground truth.Source: §C.2 · p. 16
  • Objects judged too distant are filtered out, remaining key-object annotations are manually checked, and annotators may add objects missing from the prior dataset.Source: §C.2 · p. 16
  • Future object behaviors are labeled from trajectories in general directions such as left or right, with video-based manual labels added when trajectories do not match an action.Source: §C.2 · p. 16
  • Qwen2.5-VL-72B synthesizes CoT from the three stage labels and 3 seconds of history, restricting each stage to results from the preceding stage.This restriction is intended to model connections among perception, prediction, and planning.Source: §C.2 · p. 16
  • P^3-CoT contains 19,284 nuScenes training frames from 700 scenes and 6,019 test frames from 150 scenes, plus 103,288 NAVSIM training frames from 1,192 scenes and 12,146 test frames from 136 scenes.The dataset is intended to support both supervised fine-tuning and reinforcement post-training.Source: §C.2 · p. 17
  • The unified staged reasoning is presented as reducing accumulated end-to-end driving error while adding explainability to autonomous-driving models.The dataset is designed to let VLMs benefit from both staged thought and connections among the stages.Source: §C.2 · p. 17

P 3-CoT is composed of a training set of 19284 frames and a validation set of 5119 frames, attached with detailed annotations for perception, prediction and planning three stages with connection. · C.4 COMPARISONS BETWEEN DATASETS · C.5 DATA QUALITY ASSESSMENT

P 3-CoT links perception, prediction, and planning in CoT-formatted data. Its dataset comparisons emphasize coverage and stage connections, while quality assessment checks reasoning structure, factual consistency, and plausibility.

P 3-CoT is composed of a training set of 19284 frames and a validation set of 5119 frames, attached with detailed annotations for perception, prediction and planning three stages with connection.

  • The perception annotations select sparse key objects suited to nuScenes road conditions, while category diversity and realistic action distributions characterize P 3-CoT.The authors describe the key-object selection pipeline as reasonable and as reducing unnecessary objects.Source: P 3-CoT is composed of a training set… · p. 17
  • For nuScenes, planning actions include more straight actions than prediction actions; NAVSIM prediction actions emphasize near-range and front-direction objects.NAVSIM uses the same command types as nuScenes but adapts prediction action types.Source: P 3-CoT is composed of a training set… · p. 17

C.4 COMPARISONS BETWEEN DATASETS

  • P 3-CoT connects perception, prediction, and planning in CoT format, supporting staged interpretation and unified training unlike other compared datasets.The comparison covers dataset scale and structure.Source: §C.4 · p. 17, Table 9 · p. 18
  • P 3-CoT annotates a substantial number of frames and covers perception, prediction, and planning, unlike datasets lacking that full stage coverage.Table 9 compares datasets by source, labeled frames, task coverage, and data organization format.Source: §C.4 · p. 17, Table 9 · p. 18

C.5 DATA QUALITY ASSESSMENT

  • Quality review checks that each CoT proceeds sequentially from perception through prediction to planning, includes all module outputs, and is correctly formatted.Modular checks assess factual consistency, language fluency, and logical plausibility against ground-truth labels.Source: §C.5 · p. 18
  • Ten dataset splits are reviewed by three annotators each, who randomly inspect 10% of their assigned split; splits must reach 90% accuracy or be regenerated and reassessed.The iterative review continues until all splits pass inspection by all annotators.Source: §C.5 · p. 18

D REWARD SETTING · E EXPERIMENTAL SETUP

The reward hierarchy scores perception, prediction, and planning with linked signals, and the experiments show mutual reward improvement while warning that unbalanced weights and removing KL regularization can impair training. The setup compares Cold-Start with P^3-GRPO on nuScenes and NAVSIM.

D REWARD SETTING

  • Perception reward is 1.0 when both box sets are empty, averages IoU with equal precision and recall weighting when both are nonempty, and is 0.0 otherwise.This design penalizes false positives and false negatives while rewarding precise localization.Source: D REWARD SETTING · pp. 18–19
  • Prediction reward combines IoU-weighted action-label correctness with detection quality based on average IoU and F1, tying behavioral accuracy to spatial alignment.A matched prediction receives label credit only when its action label matches ground truth; poor localization reduces reward even for correct labels.Source: D REWARD SETTING · p. 19
  • Planning reward exponentially transforms mean L2 trajectory error and adds PDMS for NAVSIM, making final trajectory quality the hierarchy’s ultimate objective.The paper frames planning as the validator of the reasoning chain: perception and prediction are prerequisites, while planning reward encourages coordinated optimization toward the final driving objective.Source: D REWARD SETTING · pp. 19–20
  • Training with P^3-CoT and P^3-GRPO consistently improves perception, prediction, planning, and all rewards, with enhanced upstream capabilities benefiting trajectory generation.The authors describe the modules as mutually reinforcing; Figure 9 visualizes the training and testing rewards.Source: Figure 9 · p. 20, D REWARD SETTING · p. 20
  • An unbalanced 1:1:1:7 reward weighting overemphasizes planning, hinders perception and prediction optimization, and yields inferior overall planning performance.The authors attribute the degradation to planning’s dependence on reliable inputs from preceding stages.Source: D REWARD SETTING · p. 20, Table 10 · p. 20

E EXPERIMENTAL SETUP

  • Experiments compare Cold-Start with P^3-GRPO on nuScenes and NAVSIM under consistent data settings, and Table 11 summarizes the configurations.The setup also ablates the KL divergence term.Source: E EXPERIMENTAL SETUP · p. 20, Table 11 · p. 21
  • Without KL regularization, performance degrades as training progresses and the model eventually collapses, so the authors recommend retaining the constraint.They attribute collapse to excessive deviation from the base policy and unstable optimization; Figure 10 presents the KL-divergence ablation.Source: E EXPERIMENTAL SETUP · p. 20, Figure 10 · p. 21

F QUALITATIVE COMPARISON OF TRAJECTORY PLANNING · G PROMPTS

Qualitative examples show AutoDrive-P^3 using scene understanding to make context-aware trajectory decisions, including both matching ground truth and deviating from it to avoid risk. The prompts specify structured, sequential reasoning for perception, prediction, and planning, with task-specific inputs and outputs.

F QUALITATIVE COMPARISON OF TRAJECTORY PLANNING

G PROMPTS

  • The training and inference prompt asks an expert assistant to analyze a 3-second video context and answer final-frame perception, prediction, and planning questions in separate think/answer tags.Source: G PROMPTS · p. 29
  • For P^3-CoT generation, Qwen2.5-VL-72B is prompted to reason step by step, conditioning prediction reasoning on perception and planning reasoning on both earlier tasks.Source: G PROMPTS · p. 29
  • The perception prompt requests coordinates and labels for key objects, while prediction and planning prompts use objects, video context, speed, historical trajectory, and reference answers.Source: G PROMPTS · p. 29
  • Prediction targets each object's next 3-second action; planning requests six ego-trajectory points at 0.5-second intervals, using signed x/y directions alongside context and motion history.Source: G PROMPTS · pp. 29–30

Related papers

Loading 2603.28116v1…