Source-linked AI summary
NaviDriveVLM: Decoupling High-Level Reasoning and Motion Planning for Autonomous Driving
Ximeng Tao, Pardis Taghavi, Dimitar Filev, Reza Langari, Gaurav Pandey
TL;DR
VLM-based autonomous driving must balance large models’ semantic reasoning against small models’ efficient adaptation for precise motion planning. NaviDriveVLM separates these roles with a frozen Navigator and trainable Driver, and experiments on nuScenes show stronger planning than single-VLM baselines while retaining interpretability and reducing adaptation cost.
Problem
VLM-based driving systems struggle to retain strong reasoning while achieving efficient adaptation and accurate motion planning.
Method
NaviDriveVLM uses a frozen large-scale Navigator for semantic guidance and a lightweight trainable Driver for future waypoint or action prediction.
Results
NaviDriveVLM improves end-to-end motion planning over single VLM baselines on nuScenes, with ablations showing that Navigator reasoning contributes to the gains.
Takeaways & Limitations
Separating semantic reasoning from motion planning provides an interpretable intermediate representation while preserving large-model reasoning and enabling efficient motion-prediction adaptation.
Abstract
from arXiv · showhide
Vision-language models (VLMs) have emerged as a promising direction for end-to-end autonomous driving (AD) by jointly modeling visual observations, driving context, and language-based reasoning. However, existing VLM-based systems face a trade-off between high-level reasoning and motion planning: large models offer strong semantic understanding but are costly to adapt for precise control, whereas small VLM models can be fine-tuned efficiently but often exhibit weaker reasoning. We propose NaviDriveVLM, a decoupled framework that separates reasoning from action generation using a large-scale Navigator and a lightweight trainable Driver. This design preserves reasoning ability, reduces training cost, and provides an explicit interpretable intermediate representation for downstream planning. Experiments on the nuScenes benchmark show that NaviDriveVLM outperforms large VLM baselines in end-to-end motion planning.
I. INTRODUCTION
VLM-based autonomous driving faces a trade-off: large models reason well but require costly adaptation for accurate actions, while smaller models adapt efficiently but often lose reasoning quality. NaviDriveVLM addresses this by separating reasoning from motion planning with a Navigator–Driver design.
- Large VLMs provide strong scene understanding and semantic reasoning but are not naturally optimized for precise motion prediction without costly task-specific adaptation.
- Lightweight VLMs can be fine-tuned efficiently for waypoint or action prediction, but often require additional supervision or distillation to recover stronger semantic guidance.
- NaviDriveVLM uses a frozen large-scale Navigator for semantic reasoning and a lightweight trainable Driver for future waypoint prediction.
- The Navigator produces scene descriptions, recommended actions, and reasoning that serve as an explicit interpretable intermediate representation for planning.
- On nuScenes, NaviDriveVLM outperforms single large-VLM baselines in end-to-end motion planning, while ablations show that Navigator reasoning improves planning performance.
II. RELATED WORK
Prior driving research increasingly uses language models for motion planning, behavior planning, instruction following, and interpretable decision making. However, reasoning commonly remains an auxiliary explanation or supervision signal rather than a separated intermediate representation for downstream control.
- Language-driven systems use language models to support motion planning, behavior planning, instruction following, and semantic understanding in autonomous driving.
- Interpretability-focused systems generate rationales or decompose reasoning across driving stages to expose intermediate decision logic.
- In most prior work, reasoning primarily functions as an auxiliary explanation signal rather than a separated intermediate representation for downstream control.
C. Decoupled and Hierarchical Driving Architectures
Recent autonomous-driving architectures separate high-level reasoning from motion planning through modular, distillation, or teacher–student designs. NaviDriveVLM instead uses a frozen large Navigator whose reasoning is explicitly passed to a lightweight Driver for waypoint or action prediction.
- C. Decoupled and Hierarchical Driving Architectures: Recent modular driving systems integrate reasoning and planning through teacher–student learning, VLM supervision, distillation, or dual-process designs.
- C. Decoupled and Hierarchical Driving Architectures: NaviDriveVLM preserves complex semantic guidance by decoupling a frozen large-scale Navigator from a lightweight trainable Driver.
- C. Decoupled and Hierarchical Driving Architectures: The Navigator processes surround images, ego state, past waypoints, and a high-level command to generate scene description, recommended action, and reasoning.
- C. Decoupled and Hierarchical Driving Architectures: Navigator reasoning tokens are concatenated with Driver inputs and passed to the Driver VLM for future waypoint or driving-action prediction.
B. Driver
The Driver is a lightweight VLM that uses Navigator reasoning alongside observations and task inputs to predict future waypoints. Its formulation treats waypoint prediction as probabilistic generation and trains on ground-truth waypoint sequences.
- B. Driver: The Driver uses Navigator reasoning as an auxiliary input alongside images, ego state, and task prompts to predict future waypoints.The reasoning output is incorporated to improve prediction accuracy.
- B. Driver: Waypoint prediction is modeled as the conditional distribution P(W | OR, I, Oego, QD, SD).W denotes the future waypoint sequence.
- B. Driver: Driver-VLM parameters are optimized by minimizing the negative log-likelihood of the ground-truth waypoint sequence during supervised fine-tuning.The target at frame t is wt, conditioned on previous ground-truth waypoints and the available inputs.
- B. Driver: Ground-truth waypoints serve as assistant prompts, while non-target tokens are masked under the autoregressive training objective.
A. Dataset
The nuScenes-Reason dataset is derived from 8-second clips sampled from nuScenes scenes, with historical context and a future prediction horizon. The section also contrasts large and small VLM behavior in qualitative driving examples.
- A. Dataset: nuScenes contains 850 scenes of 20-second driving sequences sampled at 2 Hz.
- A. Dataset: Qualitative comparisons show large VLMs can reason about scenes but miss future waypoints, whereas smaller fine-tuned VLMs predict accurately but reason less strongly.
- A. Dataset: Each extracted clip contains 2 seconds of historical context and a 6-second future prediction horizon.The clips are produced using an 8-second sliding window.
- A. Dataset: The derived dataset contains 16.54k training samples and 3618 test samples with recorded waypoints, vehicle states, and images.
B. Training
Training uses a shared Qwen3 VLM backbone with fine-tuning applied to the Driver-VLM. Qualitative scenarios and the end-to-end planning table support evaluation of the framework and its training design.
- B. Training: Only Driver-VLM is fine-tuned using supervised fine-tuning, while Qwen3 is used as the backbone for both models.
- B. Training: Additional qualitative scenes evaluate NaviDriveVLM when waiting at a red light, following another vehicle, and braking.Predicted and ground-truth waypoints are compared using minimum average L2 error over 6 seconds.
- B. Training: Table I reports end-to-end motion planning experiments on nuScenes.
- B. Training: The 8B-scale model uses 8-bit quantization and LoRA adaptation with rank 64, alpha 128, and dropout 0.05.
C. Metrics
Evaluation focuses on open-loop motion planning using multiple candidate waypoint sequences and qualitative comparisons across driving scenarios. Results show benefits from explicit reasoning and the decoupled framework.
- C. Metrics: The model generates six candidate future waypoint sequences, and evaluation selects the prediction with the smallest minimum Average L2 Error at the 6-second horizon.
- C. Metrics: In qualitative comparisons, an unfine-tuned large VLM reasons semantically but predicts inaccurate waypoints, while a fine-tuned smaller VLM predicts more accurately with degraded reasoning.
- C. Metrics: NaviDriveVLM combines reliable semantic reasoning with accurate waypoint prediction across representative driving scenarios.
- C. Metrics: The qualitative and tabulated evaluations cover waypoint and action prediction performance and input-component ablations.
- C. Metrics: NaviDriveVLM outperforms representative prior methods at the 3-second horizon on the nuScenes open-loop motion planning task.The cited comparison includes ST-P3, Ego-MLP, and UniAD.
- C. Metrics: Adding the Navigator improves planning beyond supervised fine-tuning of the single-model Driver-VLM baseline.This comparison isolates the contribution of explicit reasoning guidance.
E. Waypoints vs. Control Actions
NaviDriveVLM supports both waypoint and control-action outputs, with waypoint prediction stronger at short horizons and direct action prediction better for long-term average error.
- The Driver-VLM predicts either future waypoints or driving actions, with actions derived from waypoint supervision because nuScenes lacks ground-truth control labels.Ground-truth waypoints are converted into continuous, kinematically feasible control sequences using Tikhonov-regularized least squares.
- Waypoint-based prediction achieves lower short-term L2 error at the 1s, 2s, and 3s horizons.
- Direct action prediction achieves superior overall average L2 for long-term prediction.
F. Ablation Studies
Ablations show that high-level commands substantially improve planning, image inputs provide only marginal gains, and the complete reasoning-based configuration achieves the best reported average L2.
- Adding high-level commands reduces average L2 from 1.515 to 1.288, a decrease of 0.227.The authors attribute this improvement to explicit intention information guiding action prediction.
- Adding image inputs changes average L2 from 1.515 to 1.476, providing only a marginal gain of 0.039.The passage suggests many image tokens may be redundant or weakly task-relevant.
- The final NaviDriveVLM configuration combining reasoning, high-level commands, and images achieves a 6-second average L2 of 1.285.
- The decoupled framework preserves large-model reasoning while enabling efficient motion-prediction adaptation and improves planning over single VLM baselines.Ablations further indicate that Navigator reasoning contributes to the planning gains.