PaperlayerView exact revision on arXiv

Source-linked AI summary

ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving

Yongkang Li, Kaixin Xiong, Xiangyu Guo, Fang Li, Sixu Yan, Gangwei Xu, Lijun Zhou, Long Chen, Haiyang Sun, Bing Wang, Kun Ma, Guang Chen, Hangjun Ye, Wenyu Liu, Xinggang Wang

arXiv:2506.08052v2cs.CVcs.RO

TL;DR

End-to-end driving systems using VLMs face domain and language-action mismatches that can produce infeasible or invalid trajectories. ReCogDrive combines hierarchical cognitive data, a cognition-guided diffusion planner, and DiffGRPO reinforcement learning. It achieves state-of-the-art performance across NAVSIM and Bench2Drive evaluations while targeting safer and more comfortable driving.

  • Problem

    VLM-based trajectory planning lacks nuanced driving knowledge and mismatches discrete language outputs with continuous actions, causing infeasible or format-violating trajectories.

  • Method

    ReCogDrive instills driving cognition through a three-stage data pipeline, injects VLM priors into a diffusion planner, and applies DiffGRPO to optimize safety and comfort.

  • Results

    ReCogDrive achieves new state-of-the-art performance on NAVSIM and Bench2Drive in both open-loop and closed-loop settings.

  • Takeaways & Limitations

    The framework combines cognitive reasoning with continuous trajectory generation and reinforcement learning for safer, more comfortable driving behavior.

  • Takeaways & Limitations

    The model still has difficulty processing multiple cameras and video sequences and has relatively high inference latency.

Abstract

from arXiv · show

Recent studies have explored leveraging the world knowledge and cognitive capabilities of Vision-Language Models (VLMs) to address the long-tail problem in end-to-end autonomous driving. However, existing methods typically formulate trajectory planning as a language modeling task, where physical actions are output in the language space, potentially leading to issues such as format-violating outputs, infeasible actions, and slow inference speeds. In this paper, we propose ReCogDrive, a novel Reinforced Cognitive framework for end-to-end autonomous Driving, unifying driving understanding and planning by integrating an autoregressive model with a diffusion planner. First, to instill human driving cognition into the VLM, we introduce a hierarchical data pipeline that mimics the sequential cognitive process of human drivers through three stages: generation, refinement, and quality control. Building on this cognitive foundation, we then address the language-action mismatch by injecting the VLM's learned driving priors into a diffusion planner to efficiently generate continuous and stable trajectories. Furthermore, to enhance driving safety and reduce collisions, we introduce a Diffusion Group Relative Policy Optimization (DiffGRPO) stage, reinforcing the planner for enhanced safety and comfort. Extensive experiments on the NAVSIM and Bench2Drive benchmarks demonstrate that ReCogDrive achieves state-of-the-art performance. Additionally, qualitative results across diverse driving scenarios and DriveBench highlight the model's scene comprehension. All code, model weights, and datasets will be made publicly available to facilitate subsequent research.

1 INTRODUCTION

ReCogDrive combines VLM-based driving cognition with diffusion planning and reinforcement learning to address domain, modality, and imitation-learning limitations. It reports state-of-the-art results across open-loop and closed-loop evaluations.

  • End-to-end driving systems unify perception, prediction, and planning, but their black-box pipelines can struggle with rare corner cases.
  • VLM-based methods add world knowledge and reasoning, yet text-based trajectory generation can suffer domain gaps, modality mismatch, infeasible actions, and formatting errors.
  • ReCogDrive uses generation, refinement, and quality control to build a large-scale VQA dataset that instills human-like cognitive driving priors in a VLM.
  • A diffusion planner translates VLM latent cognitive representations into continuous and stable trajectories, while DiffGRPO optimizes safer and more comfortable behavior beyond imitation.
  • ReCogDrive establishes new state-of-the-art performance on NAVSIM and Bench2Drive in both open-loop and closed-loop evaluations.

2 RELATED WORK

Related work applies VLMs, diffusion models, and reinforcement learning to autonomous driving. These approaches respectively contribute world knowledge, trajectory-generation mechanisms, and experience-based policy optimization.

  • Vision-Language Models in Autonomous Driving: VLM-based autonomous-driving methods are categorized into dual-system approaches that guide end-to-end models and single-system approaches that directly predict trajectories.
  • Diffusion Models for Policy Learning: Diffusion models have been applied to autonomous-driving planning after demonstrating success in image generation, robotics, and traffic simulation.
  • Reinforcement Learning in Autonomous Driving: Recent autonomous-driving studies introduce reinforcement learning to improve model generalization and learn driving policies from experience.

3 PRELIMINARIES

The preliminaries define trajectory prediction and motivate cognition-guided diffusion planning. Existing systems can lack interpretability and rare-case generalization, while text-based planning can produce erroneous trajectories through language-action mismatch and decoding failures.

  • Problem Definition: The driving task predicts smooth, collision-free future trajectories from ego status, camera input, and navigation information.
  • Problem Definition: The trajectory Vtraj is a sequence of future waypoints and headings, while conventional end-to-end methods can impede interpretability and rare-corner-case generalization.
  • Hierarchical Data Pipeline: The hierarchical pipeline uses Generation, Refinement, and Quality Control to produce a high-quality dataset that mimics human-driver cognition.
  • VLM Trajectory Generation: VLM trajectory generation faces a mismatch between discrete language and continuous actions, with hallucination and decoding errors that can compromise safety.
  • Diffusion Policy: Diffusion policies reverse a Gaussian noising process, iteratively denoising noisy trajectory samples into smooth motion plans.

4 METHODOLOGY

ReCogDrive combines a cognition-guided VLM with a diffusion planner and trains the system through driving pre-training, imitation learning, and reinforcement learning. Its hierarchical data pipeline supplies driving priors, while DiffGRPO refines trajectory generation using safety and comfort feedback.

  • Scalable Hierarchical Data Pipeline: ReCogDrive uses a scalable hierarchical pipeline with Generation, Refinement, and Quality Control stages to construct cognitive driving data.The generation process follows foundational perception, dynamic understanding, planning and reasoning, and advanced reasoning levels; refinement normalizes and augments data, while quality control filters samples.
  • Cognition-Guided Diffusion Planner: The VLM encodes inputs into cognitive tokens that guide trajectory denoising in the diffusion planner.The architecture couples a VLM and diffusion planner, using cognitive tokens as guidance for trajectory generation.
  • Cognition-Guided Diffusion Planner: The cognition-guided diffusion planner addresses discrete-language and continuous-action mismatch by generating smooth and safety-aware trajectories.The planner uses VLM hidden states, ego status, and fused noisy actions, historical trajectories, and semantic priors as conditioning information.
  • Cognition-Guided Diffusion Planner: Historical trajectories and ego-vehicle status condition the planner to improve temporal consistency, stability, safety, and adaptability.Historical trajectories are concatenated with noisy trajectory embeddings, while vehicle speed and acceleration are injected through AdaLN modulation.
  • Diffusion Group Relative Policy Optimization: Imitation learning can produce averaged, incorrect, or unsafe paths when expert demonstrations are multi-modal.In rare intersection-turn scenarios, matching multiple expert trajectories may lead to an average trajectory rather than a valid maneuver.
  • Diffusion Group Relative Policy Optimization: DiffGRPO samples trajectory chains, evaluates them with NAVSIM feedback, and updates the diffusion planner using group-relative advantages.Rewards aggregate collision, drivable-area compliance, and comfort outcomes into the Predictive Driver Model Score; behavior cloning is retained to prevent collapse during exploration.

5 EXPERIMENTS

ReCogDrive is evaluated on NAVSIM, Bench2Drive, and Drive VQA benchmarks, with ablations examining its driving pre-training, diffusion planner, and reinforcement learning components. It achieves state-of-the-art planning and driving-understanding results, while qualitative and method comparisons support safer, faster trajectory generation.

  • Experimental Setup: ReCogDrive is trained with hierarchical driving VQA data using InternVL3 and evaluated primarily on NAVSIM and Bench2Drive.NAVSIM contains 1,192 training scenes and 136 evaluation scenes; Bench2Drive contains 220 CARLA routes with safety-critical scenarios.
  • NAVSIM Results: 90.8 PDMS on NAVSIM establishes a new state-of-the-art, exceeding DiffusionDrive by 2.7 PDMS and PARA-Drive by 6.8 PDMS using only camera input.It also improves over reproduced InternVL3 and QwenVL2.5 models trained directly on NAVSIM trajectories by 7.5 PDMS.
  • Bench2Drive Results: 45.45% scenario success and 71.36 Driving Score lead the Bench2Drive leaderboard, alongside 69.09% emergency braking and 71.34 traffic-sign compliance.The multi-ability mean is 42.03.
  • Ablation Study: DiffGRPO raises NAVSIM PDMS to 90.8 with a 4.3-point improvement after driving QA adaptation and diffusion planning are added.Driving QA adaptation adds 1.7 PDMS, while the diffusion planner adds 2.4 PDMS over the trajectory-only baseline.
  • Trajectory Generation: Diffusion-based trajectory generation is faster than autoregressive plain-text generation and further improves PDMS, avoiding format errors associated with textual trajectories.The comparison considers inference speed, PDM Score, and error rate.
  • Drive VQA and Qualitative Results: ReCogDrive achieves state-of-the-art DriveLM and DriveBench performance, surpassing GPT-4o on most key metrics and several open-source driving models.Qualitative comparisons show reinforcement learning producing safer and more reliable trajectories in challenging turning scenarios.

6 CONCLUSION

The conclusion presents ReCogDrive as a reinforced cognitive framework combining a VLM with a diffusion planner. Its hierarchical data pipeline, cognition-guided denoising, and DiffGRPO support state-of-the-art performance across multiple benchmarks.

  • Conclusion: ReCogDrive integrates a Vision-Language Model with a diffusion planner for end-to-end autonomous driving.The framework combines cognitive reasoning with continuous trajectory generation.
  • Conclusion: A scalable hierarchical data pipeline mimics human driving cognition to build a foundation model with strong driving priors.The conclusion identifies this foundation as the basis for subsequent cognition-guided planning.
  • Conclusion: Cognition-guided diffusion injects VLM-derived cognitive tokens into denoising to generate continuous, stable, and human-like trajectories.The denoising process is used to refine planning behavior.
  • Conclusion: DiffGRPO explicitly optimizes safety and comfort in closed-loop driving, and experiments across multiple benchmarks demonstrate state-of-the-art performance.The supplementary material includes benchmark results, ablations, data-pipeline details, and implementation information.

B QUESTIONS

The paper’s technical novelty is a reinforced cognitive framework that connects a VLM with a diffusion-based planner rather than generating trajectories purely in text. Its components target domain adaptation, continuous trajectory generation, and safety-oriented reinforcement learning.

  • Technical novelty: ReCogDrive integrates a VLM with a diffusion-based planner as a reinforced cognitive framework for autonomous driving.The framework injects VLM-derived driving priors into diffusion planning instead of producing trajectories solely in text space.
  • Diffusion planning: The diffusion planner addresses text-to-action limitations by generating continuous trajectories while avoiding slow inference, infeasible actions, and trajectory errors.It requires only a forward pass and avoids the format mismatch associated with autoregressive text decoding.
  • Reinforcement learning: DiffGRPO is introduced as a reinforcement-learning scheme for diffusion-based autonomous-driving planners, targeting safer and more comfortable driving.The paper describes it as enabling simulation-driven policy optimization for the planner.
  • Data pipeline: A scalable hierarchical pipeline uses generation, refinement, and quality control to produce driving QA data and adapt VLMs to diverse scenarios.The pipeline automatically generates, normalizes, scores, and filters annotations, avoiding costly manual annotation for domain transfer.
  • Planner benefits: Diffusion planning improves performance by +2.4 PDMS over plain-text VLM outputs and provides 7.8× faster inference.The planner also eliminates the 0.01% template mismatch errors observed in plain-text decoding.

C MORE RESULTS

Additional experiments report strong performance across extended NAVSIM metrics and DriveBench driving dimensions. Ablations show that discounting and diffusion-sampling choices affect stability and PDMS, while chain-of-thought guidance is unnecessary on the current NAVSIM scenarios.

  • Extended NAVSIM results: ReCogDrive achieves state-of-the-art scores in DDC, LK, EP, and TTC, with a 0.5 EPDMS improvement over ARTEMIS.These results come from NAVSIM evaluation with extended metrics.
  • DriveBench results: ReCogDrive-VLM achieves the best overall DriveBench performance across perception, prediction, planning, and behavior under clean, corrupted, and text-only settings.It outperforms both general-purpose VLMs and specialist driving models.
  • BC-loss ablation: Reducing the BC loss weight to λ = 0.001 raises ego progress to 87.3 and yields the highest PDMS despite slight NC, DAC, and TTC declines.The ablation shows a trade-off between more aggressive policy learning and selected safety-related metrics.
  • Sampling ablation: Setting the minimum diffusion sampling value to σexp_min = 0.02 achieves a PDMS of 90.8.Nonzero clipping encourages trajectory diversity, while overly large minimum noise can be detrimental.
  • VLM guidance ablation: Chain-of-thought guidance slightly decreases PDMS by 0.1, while trajectory-only output performs comparably to adding high-level commands.The authors therefore keep textual guidance optional for NAVSIM.
  • Discount-factor ablation: A discount factor of γ = 0.6 achieves the best PDMS of 90.8 by emphasizing later, more reliable denoising steps.Using γ = 1.0 gives equal weight to noisy early steps and destabilizes learning.

D.1 DATA COLLECTION

The data collection process combines 12 open-source driving QA datasets with a three-stage pipeline for scalable, diverse, and quality-controlled supervision. The resulting corpus covers perception, prediction, planning, and visual instruction tasks.

  • Data collection: Twelve open-source driving QA datasets provide over 3.1 million question-answer pairs across perception, prediction, and planning.The collection spans diverse real-world scenarios and includes datasets such as Talk2Car, SUTD, NuScenes-QA, and OmniDrive.
  • Dataset diversity: The source datasets cover varied driving content, including commands, weather and road conditions, risk locations, multi-frame reasoning, and scene descriptions.Examples include Talk2Car, SUTD, DRAMA, NuScenes-QA, and MAPLM.
  • Pipeline: The pipeline has three stages—Generation, Refinement, and Quality Control—to construct diverse, semantically consistent, high-quality annotations at scale.Generation covers perception, prediction, planning, and advanced reasoning tasks.
  • Refinement: Refinement standardizes heterogeneous annotation formats and augments linguistic diversity through paraphrasing and additional response generation.Bounding-box coordinates are converted to a standardized tag format and rescaled to [0,1000].
  • Quality control: Quality control discards QA pairs scoring below 60 and retains 2.3M high-quality pairs, including 775K structured NAVSIM pairs for VLM fine-tuning.The retained data is intended to enhance planning and reasoning capabilities.
  • Dataset composition: The final dataset contains 41% perception, 11% prediction, 24% planning, and 24% visual instruction samples.The visual instruction portion maintains instruction-following ability.

E EXPERIMENT DETAILS

Experiments evaluate planning with NAVSIM and safety-critical driving with Bench2Drive using metrics that capture safety, comfort, progress, route completion, and infractions. The model combines VLM and diffusion-planner components and is trained in multiple stages.

  • Evaluation metrics: NAVSIM uses PDMS to holistically assess safety, comfort, and progress through NC, DAC, TTC, C, and EP.PDMS is the official planning-oriented metric used for the NAVSIM benchmark.
  • Evaluation metrics: Bench2Drive reports Success Rate and Driving Score, with Driving Score combining route completion and multiplicative infraction penalties.The benchmark focuses on safety-critical scenarios.
  • Model architecture: ReCogDrive consists of a VLM and a diffusion-based trajectory planner using InternVL3 backbones and visual patch inputs.The VLM processes multi-view images, while the planner generates trajectories through diffusion.
  • Training configuration: Training fine-tunes the VLM on combined driving QA data, then trains the diffusion planner with behavior cloning before reinforcement-learning stages.The supplied configuration specifies three VLM epochs and 200 planner epochs for behavior cloning.
  • Hardware configuration: The implementation trains across 32 GPUs and performs inference on a single node with eight GPUs.Training uses four nodes, each equipped with eight NVIDIA H20 GPUs.

F LIMITATIONS

ReCogDrive achieves state-of-the-art performance on NAVSIM but remains constrained by multi-camera processing, video sequences, and inference latency. Future work targets more aligned 3D visual encoding, efficient architectures, and real-vehicle evaluation.

  • Limitations: ReCogDrive still faces difficulty processing multiple camera inputs, handling video frame sequences, and maintaining low inference latency.These limitations are stated despite state-of-the-art NAVSIM performance.
  • Future Work: Future work proposes a 3D vision encoder aligned with textual features and more efficient model architectures.
  • Future Work: The study plans to deploy and evaluate ReCogDrive on real vehicles.

I QUALITATIVE RESULTS

Qualitative evaluations show ReCogDrive producing safe, smooth, and stable trajectories across navigation tasks, diverse traffic conditions, and rare corner cases. Dialogue examples further show scene description, traffic-sign recognition, intent reasoning, and trajectory generation, while failure cases expose limits in aggression, perception, following-distance modeling, and evaluation.

  • Qualitative Results: ReCogDrive follows navigation instructions while maintaining safety and smoothness on representative Navtest scenarios.
  • Qualitative Results: On Bench2Drive, ReCogDrive demonstrates robustness across diverse traffic conditions and rare corner cases in closed-loop settings.
  • Qualitative Comparisons: Comparisons of Transfuser, imitation-learning, and reinforcement-learning variants show progressive gains in decision making, trajectory stability, and safety awareness.
  • Failure Cases: Failure cases include aggressive maneuvers, suboptimal multi-view perception during turns, unsafe following distances, and metric false negatives.The reported causes include ego-progress preferences, weak VLM perception, insufficient motion prediction, and visually close trajectories scored as failures.
  • Model–Human Dialogue: ReCogDrive provides detailed scene descriptions, recognizes traffic signs, reasons about driving intent, and generates safe turning trajectories through interactive dialogue.
  • Model–Human Dialogue: At red-light intersections, ReCogDrive identifies traffic lights and lead vehicles before predicting an accurate stop trajectory.
Loading 2506.08052v2…