Source-linked AI summary
FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving
Shuang Zeng, Xinyuan Chang, Mengwei Xie, Xinran Liu, Yifan Bai, Zheng Pan, Mu Xu, Xing Wei, Ning Guo
TL;DR
FSDrive addresses the modality gap created when textual CoT compresses visual driving information and obscures spatio-temporal relationships. It uses a VLA as both a world model generating a structured future frame and an inverse-dynamics model planning trajectories from that visual CoT, supported by unified VQA and visual-generation pre-training. The framework improves trajectory planning and collision outcomes while remaining effective across future-frame generation and scene-understanding evaluations.
Problem
Textual CoT converts rich visual driving data into symbolic representations that can obscure spatio-temporal relationships, discard fine-grained cues, and create a modality gap between perception and planning.
Method
FSDrive generates a structured future frame with lane dividers and 3D boxes as visual spatio-temporal CoT, then plans trajectories using current observations and the predicted scene.
Results
FSDrive demonstrates effectiveness across trajectory planning, future-frame generation, and scene understanding, including lower L2 and collision values than Doe-1 in reported settings and a 31% collision improvement over text CoT.
Takeaways & Limitations
The results support using unified visual intermediate representations to connect future-scene prediction with trajectory planning in autonomous-driving VLAs.
Takeaways & Limitations
The current system generates future frames only for the front view, while broader environmental awareness may require surround-view generation.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models offer significant potential for end-to-end driving, yet their reasoning is often constrained by textual Chains-of-Thought (CoT). This symbolic compression of visual information creates a modality gap between perception and planning by blurring spatio-temporal relations and discarding fine-grained cues. We introduce FSDrive, a framework that empowers VLAs to "think visually" using a novel visual spatio-temporal CoT. FSDrive first operates as a world model, generating a unified future frame that combines a predicted background with explicit, physically-plausible priors like future lane dividers and 3D object boxes. This imagined scene serves as the visual spatio-temporal CoT, capturing both spatial structure and temporal evolution in a single representation. The same VLA then functions as an inverse-dynamics model to plan trajectories conditioned on current observations and this visual CoT. We enable this with a unified pre-training paradigm that expands the model's vocabulary with visual tokens and jointly optimizes for semantic understanding (VQA) and future-frame prediction. A progressive curriculum first generates structural priors to enforce physical laws before rendering the full scene. Evaluations on nuScenes and NAVSIM show FSDrive improves trajectory accuracy and reduces collisions, while also achieving competitive FID for video generation with a lightweight autoregressive model and advancing scene understanding on DriveLM. These results confirm that our visual spatio-temporal CoT bridges the perception-planning gap, enabling safer, more anticipatory autonomous driving. Code is available at https://github.com/MIV-XJTU/FSDrive.
1 Introduction
FSDrive replaces lossy textual reasoning with visual spatio-temporal CoT for autonomous-driving planning. It combines future-scene generation, visual understanding, and trajectory planning in a unified VLA framework.
- Motivation: Textual CoT compresses continuous visual data into symbols, obscuring spatio-temporal relationships, fine-grained details, and the connection between perception and planning.
- Spatio-temporal CoT: FSDrive uses a unified future frame with future lane dividers, 3D detection boxes, and ordinary visual content to represent spatial structure and temporal evolution.The structural cues guide attention toward drivable areas and critical future objects while imposing physically plausible constraints.
- Spatio-temporal CoT: The framework equips the VLA to reason visually about trajectory planning rather than converting future scenarios into language descriptions.
- Unified pre-training: FSDrive jointly preserves semantic understanding through VQA and activates visual generation through a shared image-text vocabulary with minimal data and limited architectural changes.The pre-training paradigm is designed to support both current-scene comprehension and future-frame generation.
- Evaluation: Evaluations across trajectory planning, future-frame generation, and scene understanding demonstrate the effectiveness of FSDrive and its visual reasoning approach.
2 Related work
Prior work integrates language models with autonomous driving for explanation, prediction, and control, while world models support future prediction and planning. FSDrive contributes an image-form CoT that unifies the reasoning modality for trajectory planning.
- Multimodal generation and understanding: Unified multimodal models use discrete image tokens or continuous vision features, but prior approaches can lose semantic information or require costly training from scratch.
- Language models for driving: Autonomous-driving systems have integrated VLMs and LLMs for behavior explanation, control prediction, and trajectory or 3D-position output, often using natural-language representations.
- FSDrive: FSDrive instead uses a spatio-temporal CoT with a unified image form so the model can think visually about trajectory planning.
- World models: Driving world models infer dynamic environments and ego status to support future prediction and planning, with prior applications spanning generation, planning, and representation learning.
3 Proposed method: FSDrive
FSDrive unifies visual generation and understanding in an existing MLLM, using future visual representations as intermediate reasoning for trajectory planning. Its progressive training first develops structural scene predictions before generating future frames, while jointly supporting VQA and driving tasks.
- 3.1 Preliminary: FSDrive maps surround-view images and optional navigation or ego-status inputs to BEV trajectories through an MLLM.The model predicts waypoint sequences from current observations and task-related inputs.
- 3.2 Unified pre-training paradigm for visual generation and understanding: FSDrive expands an existing MLLM’s vocabulary with VQ-VAE image tokens while preserving its architecture and pretrained weights.This enables the model to predict image tokens without training the entire multimodal model from scratch.
- 3.2 Unified pre-training paradigm for visual generation and understanding: The unified paradigm combines VQA for current-scene comprehension with autoregressive future-frame prediction for visual generation.Future visual tokens are converted back into pixels by a VQ-VAE detokenizer, allowing unlabeled video data to support generation training.
- 3.2 Unified pre-training paradigm for visual generation and understanding: Progressive generation predicts structural future-scene elements before complete visual tokens, guiding geometric layout and enforcing physical constraints.The intermediate reasoning steps use annotated lane-divider and 3D-detection data before future-frame generation.
- 3.3 Spatio-temporal chain-of-thought: For planning, FSDrive integrates the holistic future scene, lane dividers, and 3D detections into one unified future frame as visual spatio-temporal CoT.The model then generates trajectories from current observations and the predicted future representation.
- 3.4 Training strategy: The training strategy initializes FSDrive from existing MLLMs, fully fine-tunes LLM parameters, freezes encoders, and uses two training stages.Unified pre-training is followed by supervised fine-tuning for scene understanding and trajectory planning.
4 Experiments
FSDrive is evaluated across trajectory planning, future-frame generation, and scene understanding using nuScenes, NAVSIM, and DriveLM. The experiments and ablations report stronger planning, competitive generation quality, and improved scene understanding, while testing the contributions of pre-training and visual CoT.
- Experimental settings: FSDrive evaluates trajectory planning on nuScenes and NAVSIM, future-frame generation on nuScenes, and scene understanding on DriveLM.Trajectory planning uses L2 displacement error and collision rate; generation uses FID; DriveLM uses language metrics including BLEU, ROUGE_L, and CIDEr.
- Trajectory planning: FSDrive surpasses prior methods in nuScenes trajectory planning under the reported ego-status and non-ego-status comparisons.The paper prioritizes non-ego-status evaluations and reports advantages over non-autoregressive and autoregressive methods.
- NAVSIM: 85.1 PDMS: FSDrive outperforms prior camera-only methods on NAVSIM’s pseudo closed-loop evaluation.All approaches use camera input without lidar data.
- Future-frame generation: FSDrive 2B achieves competitive FID against specialized diffusion models for 128×192 future-frame generation.The lightweight autoregressive model is reported to outperform Doe-1 7B in the cited comparison while using minimal data.
- Scene understanding: FSDrive achieves 0.57 on the DriveLM GVQA benchmark, outperforming recent methods including Cube-LLM and OmniDrive.The result is presented as evidence for the unified generation-and-understanding pre-training paradigm.
- Ablation study: Future-frame generation pre-training improves L2 by 16.4% and collisions by 15.8%, while the combined understanding-and-generation setup performs best overall.The ablation attributes these gains to world-model-based prediction, physical-dynamics learning, and stronger scene representation.
- Ablation study: 31% improvement: spatio-temporal CoT yields larger collision-rate gains than pure text CoT’s 8.6% improvement.Image-text CoT shows insignificant gains, while unified image-based reasoning is reported to identify future collision risks.
5 Conclusion
FSDrive unifies future scene generation and perception through image-form reasoning, using one VLA as both world model and inverse dynamics model. The current system generates only front-view future frames, leaving surrounding-view generation for future work.
- FSDrive uses intermediate image-form reasoning to unify future scene generation and perception in an end-to-end visual reasoning pipeline.
- The VLA serves as a world model for future frames and an inverse dynamics model for trajectory planning.
- The framework combines unified visual generation-and-understanding pretraining with progressive easy-to-hard visual CoT generation.
- The current system generates future frames only for the front view because of real-time efficiency considerations.
- Future work targets surround-view generation and improved visual quality through larger datasets and more advanced unified paradigms.