MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Driving

Lingjun Zhang; Yujian Yuan; Changjie Wu; Xinyuan Chang; Xin Cai; Shuang Zeng; Linzhe Shi; Sijin Wang; Hang Zhang; Mu Xu

Source-linked AI summary

MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Driving

Lingjun Zhang, Yujian Yuan, Changjie Wu, Xinyuan Chang, Xin Cai, Shuang Zeng, Linzhe Shi, Sijin Wang, Hang Zhang, Mu Xu

arXiv:2602.21952v1Feb 25, 2026cs.CV

TL;DR

MindDriver addresses the space misalignment of textual CoT and the unguided image prediction of image-based reasoning in autonomous driving. It progressively connects semantic understanding, imagined future scenes, and physical trajectory planning using aligned data annotation and progressive reinforcement fine-tuning. The framework achieves superior performance across nuScenes open-loop and Bench2Drive closed-loop evaluations.

Problem
Textual CoT misaligns semantic reasoning with physical-space trajectories, while image reasoning lacks planning-oriented guidance for accurate scene evolution.
Method
MindDriver progressively performs semantic understanding, semantic-to-physical-space imagination, and physical-space trajectory planning, trained with feedback-guided annotation and progressive reinforcement fine-tuning.
Results
MindDriver achieves superior performance in nuScenes open-loop and Bench2Drive closed-loop evaluations, including a 39.55% success rate without ego-status in Bench2Drive.
Takeaways & Limitations
Progressive multimodal reasoning provides a framework for aligning semantic understanding, imagined scenes, and physical planning in autonomous driving.
Takeaways & Limitations
MindDriver is highly GPU-dependent and currently generates only front-view images.

Abstract

from arXiv · show

Vision-Language Models (VLM) exhibit strong reasoning capabilities, showing promise for end-to-end autonomous driving systems. Chain-of-Thought (CoT), as VLM's widely used reasoning strategy, is facing critical challenges. Existing textual CoT has a large gap between text semantic space and trajectory physical space. Although the recent approach utilizes future image to replace text as CoT process, it lacks clear planning-oriented objective guidance to generate images with accurate scene evolution. To address these, we innovatively propose MindDriver, a progressive multimodal reasoning framework that enables VLM to imitate human-like progressive thinking for autonomous driving. MindDriver presents semantic understanding, semantic-to-physical space imagination, and physical-space trajectory planning. To achieve aligned reasoning processes in MindDriver, we develop a feedback-guided automatic data annotation pipeline to generate aligned multimodal reasoning training data. Furthermore, we develop a progressive reinforcement fine-tuning method to optimize the alignment through progressive high- level reward-based learning. MindDriver demonstrates superior performance in both nuScences open-loop and Bench2Drive closed-loop evaluation. Codes are available at https://github.com/hotdogcheesewhite/MindDriver.

1. Introduction

MindDriver addresses the mismatch between textual semantic reasoning and physical-space trajectory planning, while improving on image reasoning that lacks planning-oriented guidance. It progressively connects semantic understanding, imagined future scenes, and trajectory prediction, supported by aligned data annotation and reinforcement fine-tuning.

Motivation: Traditional textual CoT creates space misalignment because semantic reasoning must ultimately produce physical-space trajectories.Image-based intermediaries better combine semantic and physical information but lack clear planning-oriented guidance for future scene generation.
Proposed framework: MindDriver progressively reasons from textual scene understanding through imagined future images to physically grounded trajectory prediction.Its components cover semantic understanding, visual imagination, and physical trajectory prediction.
Training alignment: A feedback-guided annotation framework generates aligned multimodal reasoning data for transitions between semantic understanding, visual imagination, and trajectory planning.The training process emphasizes gradual alignment across the reasoning stages.
Evaluation: Experiments evaluate MindDriver on open-loop and closed-loop trajectory planning and future-frame generation.The reported evaluations support the effectiveness of progressive reasoning, automatic annotation, and reinforcement fine-tuning.

2. Related Work

Related work combines end-to-end autonomous driving with multimodal language-model reasoning, but semantic reasoning remains difficult to align with precise action execution. MindDriver extends this line with process-oriented reinforcement fine-tuning for multimodal reasoning alignment.

End-to-end driving: End-to-end driving systems jointly optimize perception and planning by mapping raw sensor inputs directly to trajectories.Representative systems integrate perception, prediction, and planning within unified architectures.
MLLMs for driving: MLLM-based driving methods use architectures, visual question answering, or structured reasoning to connect world knowledge with trajectory planning.Examples include dual-system trajectory refinement, VQA-based planning, and hierarchical or adaptive Chain-of-Thought.
Reinforcement learning: Existing reinforcement approaches often reward final outcomes without effectively optimizing intermediate reasoning processes.This limitation motivates process-level alignment for multimodal reasoning.
Reinforcement learning: MindDriver uses progressive reinforcement fine-tuning to reward the reasoning process while enhancing scene understanding and future-image imagination.GRPO is used to support faster convergence and more stable training dynamics.

3. MindDriver

MindDriver combines textual scene understanding, imagined future images, and physical trajectory prediction in a unified multimodal reasoning framework. It supports this progression with automatically filtered training data and staged reinforcement fine-tuning.

Progressive Multimodal Reasoning: MindDriver progressively maps semantic text reasoning to imagined future scenes and then to physically grounded trajectory predictions.The framework uses future-scene imagination to connect planning-oriented textual analysis with trajectory outputs.
Model Inputs: MindDriver uses temporal surround-view inputs, recent front-view history, driving commands, ego-vehicle state, and language instructions for planning.The visual inputs include six surround-view cameras and four recent front-view frames.
Unified Text Reasoning and Visual Generation: The model unifies textual reasoning and visual generation through a shared language-model prediction head over text and discrete visual tokens.A VQ-VAE tokenizer encodes images into discrete indices, while a decoder maps generated vision tokens back to pixels.
Training Data Annotation: The feedback-guided annotation pipeline generates raw CoT, filters it for format, decision, and logic, and re-annotates failed samples using error feedback.The resulting text reasoning is merged with future scene images and trajectories for supervised fine-tuning.
Progressive Reinforcement Fine-tuning: Progressive reinforcement fine-tuning first rewards semantically consistent dreamed images, then optimizes trajectory planning using L2 geometric distance and ADE.Stage 1 uses CLIP-image similarity, while Stage 2 evaluates average L2 distances over trajectory time steps.
Progressive Reinforcement Fine-tuning: GRPO optimizes the current policy from grouped candidate outputs using normalized group-relative advantages and a clipped objective.The objective includes KL-divergence regularization relative to the SFT reference policy.
Experiments: The framework is evaluated on end-to-end trajectory planning using nuScenes L2 and collision metrics under the evaluation methodologies specified for ST-P3 and UniAD.The table caption also identifies ego-status variants, BEV-Planner-derived results, and re-implemented Qwen2.5-VL-3B results.

4. Experiments

MindDriver is evaluated through open-loop nuScenes trajectory and image-generation tasks, closed-loop Bench2Drive driving, and ablations of its multimodal reasoning and training design. Results show improved trajectory planning, future-frame quality, closed-loop driving, and progressive reasoning components.

Experiment settings: nuScenes evaluation covers open-loop trajectory planning and future-frame generation, while Bench2Drive evaluates closed-loop driving performance.nuScenes provides 28,130 training and 6,019 validation samples; Bench2Drive evaluation uses 220 official short routes.
Main results: MindDriver outperforms prior methods on both ST-P3 and UniAD trajectory metrics without ego status, while reducing collisions against image-only and text-only CoT approaches.The comparison includes non-autoregressive, autoregressive, image-only CoT, and text-only CoT baselines.
Main results: MindDriver achieves superior future-frame FID to specialized diffusion models and image-only CoT, indicating improved generated-scene quality after textual reasoning.Frames are generated at 128x192 resolution to balance quality and generation speed.
Main results: 39.55% success rate is reported for MindDriver on closed-loop Bench2Drive, alongside a higher driving score and success rate without ego status.The paper describes performance as competitive with DriveAdapter and ReasonPlan in complex interactive scenarios.
Qualitative results: Figure 4 qualitatively compares predicted and ground-truth trajectories in nuScenes and shows performance variation over timestamps in closed-loop Bench2Drive.The nuScenes examples use red predicted and green ground-truth trajectories.
Ablation Study: Text-to-image multimodal CoT outperforms image-to-text reasoning, while pure image CoT offers limited benefit because it lacks planning-oriented reasoning.The ablation supports semantic planning before future-scene imagination.
Ablation Study: Dreaming the next 0.5s scene image performs best, whereas 1s and 1.5s horizons increase uncertainty and reduce accuracy.The shorter horizon matches the 0.5s/frame input-history sampling interval.
Ablation Study: Feedback-guided filtering improves reasoning quality over raw CoT and baseline training, while two-stage progressive RFT achieves the best performance.Progressive RFT first improves scene-image alignment and then further aligns trajectory planning with generated images.

5. Conclusion

MindDriver uses progressive multimodal reasoning to move from semantic understanding through imagined future scenes to physical-space trajectory prediction. It combines aligned data annotation and progressive reinforcement fine-tuning, while remaining GPU-dependent and limited to front-view image generation.

MindDriver unifies semantic-space text reasoning, future-scene imagination, and physical-space trajectory prediction in a progressive multimodal framework.The framework is designed to imitate progressive human-like reasoning for autonomous driving.
A feedback-guided annotation pipeline and progressive reinforcement fine-tuning optimize alignment across the multimodal reasoning process.
MindDriver is evaluated in both open-loop and closed-loop settings, where experiments validate its effectiveness.
MindDriver achieves 1 Hz inference speed on an NVIDIA RTX 4090 GPU but remains highly GPU-dependent and computationally demanding.
The current system generates only front-view images, leaving richer and more detailed visual outputs for future work.

6. Reasoning Annotation Pipeline

The annotation pipeline constructs progressive multimodal reasoning data by filtering and refining model-generated reasoning with feedback. It then combines text reasoning, future-scene imagery, and trajectories using explicit stage markers.

The pipeline uses three filtering processes and feedback-guided re-annotation to control data quality and iteratively refine rejected samples.
Qwen2.5-VL-72B first generates raw textual Chain-of-Thought from the current camera, history video, driving command, and instruction.
Format, decision, and logic checks identify structural omissions, incorrect driving decisions, and reasoning errors for feedback-based correction.
The annotated sample concatenates text CoT, a ground-truth future scene image, and trajectory with <think>, <dream>, and <answer> markers.
Qwen3-235B-A22B-Instruct is used for logical verification of preliminary reasoning chains.

7. Implementation Details

Implementation uses Qwen2.5-VL-3B as the base VLM, trains across multiple NVIDIA H20 GPUs, and applies separate supervised and progressive reinforcement fine-tuning configurations.

Experiments use Qwen2.5-VL-3B as the base VLM on 16 NVIDIA H20 GPUs with 96 GB each.
Supervised fine-tuning uses a 1 × 10−4 learning rate, batch size 32, and 12 nuScenes or 6 Bench2Drive epochs.
The feedback annotation pipeline allows at most 3 iterative rounds, freezes the vision encoder, and fully fine-tunes the LLM during SFT.
Progressive reinforcement fine-tuning uses a 3 × 10−6 learning rate and batch size 16.

8. Visualization of Dreamed Images

Visualization examples show MindDriver progressing from textual scene reasoning to imagined future images that represent motion and potential collision states. The qualitative analysis compares predicted values with ground truth.

MindDriver’s visualized outputs show a progressive reasoning process across two representative cases.
In the first case, textual reasoning identifies pedestrian motion trends and safety risks before generating a future-scene visualization.
The imagined scene depicts changed pedestrian positions, demonstrating modeling of future spatiotemporal relationships.
Figure 7 represents predictions in red and ground truth in green for qualitative comparison.
In the second case, the system predicts that a black SUV reaches the road center after 0.5 seconds when the ego vehicle maintains its speed.

9. More Visualization

MindDriver is evaluated through closed-loop CARLA testing, where visual history is converted into a predicted trajectory and vehicle controls. Visualizations show robust behavior in challenging nuScenes and Bench2Drive scenarios, including complex maneuvers, dynamic agents, and traffic-rule compliance.

Closed-loop evaluation: In CARLA closed-loop testing, four front-facing RGB images covering the previous two seconds are used to predict a two-second trajectory.A PID controller converts the predicted trajectory into throttle, brake, and steering signals.
nuScenes results: MindDriver avoids collisions across challenging nuScenes scenarios including nighttime, heavy rain, and high-curvature roads.Figure 11 compares the predicted red trajectory with the green ground-truth trajectory.
nuScenes results: The dynamics-driven approach produces smooth trajectories that nearly overlap with ground truth on sharp turns under kinematic constraints.The passage attributes this capability to dynamics-related rewards.
Bench2Drive results: On Bench2Drive, MindDriver handles pedestrians, cutting-in vehicles, narrow roads, nighttime, and other extreme conditions in closed-loop scenarios.The model is described as anticipating and reacting to dynamic agents such as jaywalking pedestrians and vehicles cutting in.
Bench2Drive results: In complex intersections, MindDriver follows traffic lights and STOP signs while stopping for multiple pedestrians, including children.The passage characterizes these decisions as socially compliant and improved relative to previous SOTA methods such as VAD.