Source-linked AI summary
AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoning
Bo Jiang, Shaoyu Chen, Qian Zhang, Wenyu Liu, Xinggang Wang
TL;DR
Autonomous-driving models and VLM planning systems remain limited by long-tailed scenarios, sparse reasoning data, and reliance on SFT. AlphaDrive combines four planning-specific GRPO rewards with two-stage SFT and RL training, improving planning performance and training efficiency while producing emergent multimodal planning capabilities.
Problem
Autonomous-driving models struggle with long-tailed scenarios, while VLM planning methods largely rely on SFT and general RL is poorly matched to driving’s unequal action importance and multiple valid solutions.
Method
AlphaDrive combines GRPO-based RL with four planning rewards and a two-stage reasoning strategy that uses SFT before RL.
Results
25.52% planning-accuracy improvement over the SFT-trained model, with AlphaDrive outperforming SFT by 35.31% using only 20% of the training data.
Takeaways & Limitations
AlphaDrive demonstrates that planning-specific RL and staged reasoning can improve autonomous-driving planning performance and training efficiency, with emergent multimodal planning after RL.
Takeaways & Limitations
AlphaDrive cannot yet output complex behaviors such as lane changes or nudges, and its pseudo-labeled reasoning data may contain inaccurate perception and missed key factors.
Abstract
from arXiv · showhide
OpenAI o1 and DeepSeek R1 achieve or even surpass human expert-level performance in complex domains like mathematics and science, with reinforcement learning (RL) and reasoning playing a crucial role. In autonomous driving, recent end-to-end models have greatly improved planning performance but still struggle with long-tailed problems due to limited common sense and reasoning abilities. Some studies integrate vision-language models (VLMs) into autonomous driving, but they typically rely on pre-trained models with simple supervised fine-tuning (SFT) on driving data, without further exploration of training strategies or optimizations specifically tailored for planning. In this paper, we propose AlphaDrive, a RL and reasoning framework for VLMs in autonomous driving. AlphaDrive introduces four GRPO-based RL rewards tailored for planning and employs a two-stage planning reasoning training strategy that combines SFT with RL. As a result, AlphaDrive significantly improves both planning performance and training efficiency compared to using only SFT or without reasoning. Moreover, we are also excited to discover that, following RL training, AlphaDrive exhibits some emergent multimodal planning capabilities, which is critical for improving driving safety and efficiency. To the best of our knowledge, AlphaDrive is the first to integrate GRPO-based RL with planning reasoning into autonomous driving. Code will be released to facilitate future research.
1. Introduction
Autonomous-driving models have improved planning but remain challenged by long-tail scenarios and limited reasoning, while existing VLM approaches mostly rely on SFT. AlphaDrive addresses these gaps with planning-specific RL rewards and two-stage reasoning training.
- End-to-end models improve planning from driving demonstrations but still struggle with complex, long-tailed scenarios because they lack common sense and reasoning.
- Existing VLM planning methods mainly use SFT and have not sufficiently explored training strategies or planning-specific optimization.
- Directly applying general RL to planning is suboptimal because driving actions differ in safety importance and cannot be weighted equally.
- Planning outputs may have multiple valid solutions, making exact ground-truth matching an imperfect evaluation target.
- AlphaDrive uses GRPO with four planning rewards covering accuracy, action importance, solution diversity, and output format.
- Its two-stage strategy distills planning reasoning with SFT before using RL to explore planning, while improving training stability and efficiency.
2. Related Work
Related work spans general VLM reasoning, reinforcement learning, autonomous-driving planning, and VLM-based driving systems. These studies motivate AlphaDrive’s combination of multimodal planning with GRPO and staged reasoning training.
- Vision Language Models: VLMs extend large-model capabilities to visual understanding and reasoning, while RL and reasoning techniques further enhance large-model performance.
- Reinforcement Learning and Reasoning: GRPO-based training has been associated with emergent reasoning behavior, and AlphaDrive similarly exhibits emergent multimodal planning after GRPO-based RL.
- Autonomous Driving Planning: End-to-end planning systems directly output trajectories or control signals from sensor data but remain vulnerable to long-tailed problems because they lack common-sense and reasoning capabilities.
- VLMs and Autonomous Driving: AlphaDrive’s framework combines GRPO-based RL, four planning rewards, and a two-stage SFT-to-RL training paradigm.
- VLMs and Autonomous Driving: VLM-based driving systems have been used for scene understanding, direct planning, or planning alongside end-to-end trajectory models.
3. AlphaDrive
AlphaDrive adapts GRPO-based reinforcement learning and reasoning to autonomous-driving planning through planning-specific rewards and a two-stage SFT-plus-RL training strategy. The framework addresses planning’s safety-weighted, multimodal actions and scarce reasoning data while improving reported performance, stability, and training efficiency.
- Overview: AlphaDrive combines four planning-specific GRPO rewards with a two-stage training strategy that integrates SFT and RL.The strategy uses a small SFT warm-up before RL on the full dataset to improve early training stability.
- Reinforcement learning algorithm: GRPO is selected because relative optimization across grouped outputs suits planning’s multiple feasible solutions and is reported to provide training stability and efficiency.The planning diversity reward encourages varied feasible solutions when outputs converge during later training.
- Planning rewards: The four rewards measure planning accuracy, action importance, output diversity, and output format.Accuracy evaluates lateral and longitudinal decisions separately; action weighting emphasizes safety-critical behaviors such as braking and steering.
- Planning reward modeling: Exact-match evaluation was replaced with extraction and F1-based scoring because early formatting imperfections made exact matching unstable.The reward design evaluates lateral and longitudinal planning components separately rather than treating planning as a single correct-answer decision.
- Planning reward modeling: The planning quality reward multiplies accuracy, action-weighted, and diversity rewards separately for speed and direction, then combines with format reward for GRPO updates.Algorithm 1 takes planning answers, ground-truth actions, action sets, and an answer format as inputs to produce planning rewards.
- Planning reasoning training: Because large-scale planning-reasoning data are unavailable and manual annotation is costly, a cloud model generates reasoning data from a small set of driving clips.The generated prompts include real driving actions, vehicle state, and navigation information to produce concise decision-making processes.
4. Experiments
Experiments evaluate AlphaDrive on MetaAD using high-level planning accuracy and planning-reasoning quality, then analyze rewards, reasoning strategies, data efficiency, and emergent multimodal planning. AlphaDrive achieves the strongest reported planning results, while combined SFT and RL improves reasoning and reduced-data performance.
- Experimental Settings: MetaAD contains 120k three-second driving clips, with 110k for training and 10k for validation; 30k training clips are sampled to generate planning reasoning.All reported results train on the training set and evaluate on the validation set.
- Experimental Settings: Evaluation measures lateral and longitudinal meta-action F1 scores, overall planning accuracy, and reasoning similarity using BLEU-4, CIDEr, and METEOR.The model receives a front-view image and planning prompt containing current speed and navigation information.
- Main Results: 25.5% planning-accuracy improvement over Qwen2VL-7B makes AlphaDrive the strongest model in Table 1, with improved steering, acceleration/deceleration, and planning-reasoning quality.The comparison identifies Qwen2VL-7B as the second-best performing model after AlphaDrive.
- Ablation Study: The Planning Format Reward provides a slight improvement, the Planning Accuracy Reward a significant improvement, Action-Weighted Reward improves acceleration/deceleration decisions, and Planning Diversity Reward yields the best planning performance.The ablation evaluates the four proposed GRPO planning rewards across planning metrics.
- Ablation Study: Combining SFT warm-up with RL achieves the best planning-reasoning capability, while reasoning particularly improves complex acceleration and deceleration actions.Using SFT before RL improves early training stability compared with relying solely on RL.
- Ablation Study: 70.83% planning accuracy with 50k samples demonstrates training efficiency, while RL reaches 46.08% with 20k samples, significantly exceeding the SFT-trained model.SFT is more affected than RL as training data size decreases.
- Emergence of Multimodal Planning Capability: After RL training, AlphaDrive generates multiple feasible solutions in complex scenarios, unlike the SFT-trained model’s single planning decision.A downstream action model can dynamically select an optimal solution from these alternatives.
5. Conclusions and Limitations
AlphaDrive is presented as a VLM framework combining GRPO-based reinforcement learning with planning reasoning to improve autonomous-driving planning performance and training efficiency. Its capabilities remain constrained by limited data annotation and imperfect pseudo-label quality.
- Conclusions: AlphaDrive combines planning-oriented GRPO reinforcement learning with a two-stage planning reasoning training paradigm.The framework is designed for high-level autonomous-driving planning.
- Conclusions: The framework significantly boosts planning performance and training efficiency compared with prior models using only supervised fine-tuning.
- Limitations: Limited rich data annotation prevents AlphaDrive from outputting more complex behaviors such as lane changes or nudges.
- Limitations: Planning reasoning data rely on large-model pseudo-labels generated from ground-truth actions, which may contain inaccurate perception and miss key factors.The authors call for systematic validation to improve data quality and verify AlphaDrive’s performance upper bound.