PaperlayerView exact revision on arXiv

Source-linked AI summary

LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving

Yuechen Luo, Fang Li, Shaoqing Xu, Yang Ji, Zehan Zhang, Bing Wang, Yuannan Shen, Jianwei Cui, Long Chen, Guang Chen, Hangjun Ye, Zhi-Xin Yang, Fuxi Wen

arXiv:2603.01928v2cs.CV

TL;DR

VLA models face semantic-perceptual conflicts from textual CoT and insufficient physical constraints in latent reasoning. LaST-VLA grounds latent spatio-temporal reasoning with geometric and dynamic priors, progressive training, and GRPO, achieving state-of-the-art performance across NAVSIM, SURDS, and NuDynamics.

  • Problem

    Existing VLA reasoning suffers from semantic-perceptual conflict, while unsupervised latent CoT lacks physical constraints and can degrade planning performance.

  • Method

    LaST-VLA aligns latent hidden states with geometric and dynamic priors, uses progressive SFT from spatial-temporal understanding to planning, and refines trajectories with GRPO.

  • Results

    LaST-VLA achieves state-of-the-art performance on NAVSIM, SURDS, and NuDynamics benchmarks.

  • Takeaways & Limitations

    Aligning latent reasoning with physical reality enhances the robustness, efficiency, and safety of VLA-based planning.

Abstract

from arXiv · show

While Vision-Language-Action (VLA) models have revolutionized autonomous driving by unifying perception and planning, their reliance on explicit textual Chain-of-Thought (CoT) leads to semantic-perceptual decoupling and perceptual-symbolic conflicts. Recent shifts toward latent reasoning attempt to bypass these bottlenecks by thinking in continuous hidden space. However, without explicit intermediate constraints, standard latent CoT often operates as a physics-agnostic representation. To address this, we propose the Latent Spatio-Temporal VLA (LaST-VLA), a framework shifting the reasoning paradigm from discrete symbolic processing into a physically grounded Latent Spatio-Temporal CoT. By implementing a dual-feature alignment mechanism, we distill geometric constraints from 3D foundation models and dynamic foresight from world models directly into the latent space. Coupled with a progressive SFT training strategy that transitions from feature alignment to trajectory generation, and refined via Reinforcement Learning with Group Relative Policy Optimization (GRPO) to ensure safety and rule compliance. \method~setting a new record on NAVSIM v1 (91.3 PDMS) and NAVSIM v2 (87.1 EPDMS), while excelling in spatial-temporal reasoning on SURDS and NuDynamics benchmarks.

1. Introduction

VLA models unify perception and driving policy but face interpretability, semantic-perceptual, efficiency, and physical-grounding limitations. LaST-VLA addresses these issues with supervised latent spatio-temporal reasoning and progressive training for planning and safety.

  • Existing limitations: Existing VLA approaches trade off interpretability, efficiency, and physical reliability: direct generation lacks reasoning, while textual CoT introduces latency and hallucinations.Explicit textual reasoning can cause planners to follow linguistic guidance inconsistent with visual evidence.
  • Existing limitations: Unsupervised latent reasoning improves computational efficiency but remains physically ungrounded and can become unstable during training.The paper contrasts this instability with supervised latent physical grounding.
  • LaST-VLA: LaST-VLA supervises latent reasoning with representations from geometry and video world models, embedding spatial-temporal perceptual capabilities into the reasoning process.The supervisory signals are temporally stable and step-level.
  • LaST-VLA: LaST-VLA unifies instruction following and dynamic prediction through a latent spatiotemporal Chain-of-Thought.The design is intended to overcome the precision limitations of explicit reasoning and instability of unsupervised latent reasoning.
  • Training strategy: Its progressive training first develops spatial-temporal understanding and then enables planning, with reinforcement learning further refining driving decisions.The training sequence moves from understanding capabilities to task-specific planning before RL refinement.

2. Related Work

Related work moves from textual to latent reasoning, but unsupervised latent CoT can degrade planning because it lacks physical constraints. LaST-VLA supervises latent reasoning with geometric and dynamic priors to improve trajectory-planning fidelity and robustness.

  • Textual reasoning: Textual CoT can suppress visual information and reduce efficiency and accuracy when intermediate reasoning becomes excessive.These concerns motivate alternatives to language as the medium for structured reasoning.
  • Latent reasoning: LaST-VLA constructs latent CoT by aligning hidden states with dynamic and geometric priors distilled from foundation models.The framework uses specialized adapters to transfer these priors into the latent representation.
  • Latent reasoning: Unsupervised continuous latent CoT can degrade planning performance because it lacks physical constraints.The proposed Think with Latent SpatioTemporal mechanism supervises latent CoT with geometric and dynamic priors.

3. Method

LaST-VLA bridges perception and planning with a latent spatio-temporal CoT that separates dynamic and geometric reasoning. It distills physical priors into latent states, progressively trains planning, and refines action generation with reward-based optimization.

  • Latent Spatio-Temporal CoT: LaST-VLA formulates end-to-end driving as conditional trajectory generation augmented by a latent reasoning bridge.The model first generates latent reasoning states and then predicts trajectory waypoints conditioned on them.
  • Latent Spatio-Temporal CoT: The latent CoT separates dynamic states for temporal evolution from geometric states for spatial structure, grounding reasoning in physical properties.Dynamic states capture world-model temporal dynamics, while geometric states encode 3D information.
  • Latent Spatio-Temporal CoT: Foundation-model teachers distill world dynamics and 3D geometry into continuous latent tokens instead of requiring pixel-level reconstruction.The Dynamics Adapter aligns with Cosmos representations, while the Geometry Adapter aligns with VGGT features for spatial priors such as depth and occupancy.
  • Latent Spatio-Temporal CoT: The latent CoT sequence is generated from multimodal visual and language embeddings, then partitioned into dynamic and geometric streams for specialized alignment.Random masking of visual embeddings is used during training to prevent shortcut alignment through raw pixel patterns.
  • Progressive Two-Stage SFT Strategy: Progressive two-stage supervised fine-tuning first emphasizes physical alignment, then combines trajectory learning with geometric and dynamic feature objectives.Phase I uses λWM = λ3D = 1.0 ≫ λaction = 0.01 and structured causal masking to encourage reliance on grounded reasoning.
  • Latent-Grounded Trajectory Refinement via GRPO: GRPO refines action generation with rewards for trajectory quality, output structure, and endpoint goal alignment.The reward design targets safe, compliant, and precise driving behavior through PDMS, format, and endpoint-distance components.

4. Experiment

Experiments evaluate LaST-VLA on planning, spatial reasoning, and dynamic understanding benchmarks, showing strong gains from grounded latent reasoning, progressive training, and structured masking. The method achieves state-of-the-art planning results and improved geometric, relational, and safety-related behavior.

  • NAVSIM benchmark: LaST-VLA-8B achieves 91.3 PDMS on NAVSIMv1, exceeding Recogdrive-2B by 0.5 PDMS.
  • NAVSIM benchmark: 87.1 EPDMS on NAVSIMv2 establishes a new state of the art, surpassing DriveVLA-W0-7B by 1.0 EPDMS.
  • Spatial and dynamic reasoning: 43.49% improvement over SURDS-3B and 7.68% over InternVL3-8B are achieved by LaST-VLA-8B on SURDS.
  • Latent spatio-temporal reasoning: The full geometric-and-dynamic latent CoT improves PDMS by 3.4 during SFT and 4.1 after RL relative to the baseline.
  • Reasoning ablations: Physical supervision yields 1.5 PDMS over unsupervised latent CoT and raises DAC to 97.9 and TTC to 95.6.
  • Reasoning ablations: Structured causal masking increases PDMS by 2.0, while N3D = 12 and NWM = 3 × 12 produces the optimal 91.3 PDMS.
  • Qualitative analysis: Qualitative comparisons show LaST-VLA maintaining lane compliance and anticipating rear-end collisions more effectively than textual CoT.

5. Conclusion

The conclusion presents LaST-VLA as a shift from explicit textual reasoning to physically grounded latent spatio-temporal reasoning. Combined with progressive training and GRPO, the framework achieves state-of-the-art results across the reported autonomous-driving benchmarks.

  • LaST-VLA shifts autonomous-driving reasoning from explicit text to a continuous latent spatio-temporal space.
  • The framework distills physical priors from 3D and video foundation models and combines them with progressive training and GRPO.
  • LaST-VLA achieves state-of-the-art performance on NAVSIM, SURDS, and NuDynamics.

B. More Results

Additional experiments on nuScenes evaluate open-loop planning, while geometric-supervision ablations examine which VGGT feature layers best support spatial planning. The reported results indicate accurate and safe trajectory prediction alongside benefits from deeper geometric features.

  • nuScenes benchmark: 0.38m average L2 error is the lowest among compared VLM-based methods on the nuScenes open-loop planning evaluation.
  • nuScenes benchmark: 0.18% collision rate accompanies LaST-VLA-8B's nuScenes trajectory-precision result.
  • nuScenes benchmark: LaST-VLA-2B also delivers robust nuScenes performance, supporting effectiveness across model sizes.
  • Geometric supervision ablation: Shallow VGGT features 4 and 11 are suboptimal, while deeper Feature 17 improves geometric-supervision performance.

C.1. nuScenes

The nuScenes evaluation uses a large-scale, multimodal driving dataset to assess generalization, while the accompanying ablation compares geometric supervision from different VGGT aggregator layers. The cited materials define the dataset setting and the feature-layer comparison rather than reporting additional numerical outcomes.

  • Dataset: nuScenes contains 1,000 complex driving scenarios of approximately 20 seconds each, captured with 32-beam LiDAR and six surrounding cameras.
  • Dataset: The standard nuScenes split is used for evaluation in the extended experiments.
  • Geometric supervision: Table 9 compares supervision from different VGGT aggregator layers, with Feature 23 identified as the last layer.

C.2. NuDynamics

NuDynamics is built from SURDS to evaluate dynamic scene understanding using clearly visible targets and behavior labels derived from nuScenes annotations.

  • NuDynamics extends the SURDS data basis to evaluate dynamic scene understanding.
  • The dataset selects clearly visible targets from nuScenes and uses Qwen2.5-VL-32B to generate concise descriptions.
  • Behavior labels cover stopped crossing, same-direction motion, opposite-direction motion, and diagonal movement, mapped from native nuScenes annotations.

D. Experimental Details

The experimental setup uses InternVL3-8B, benchmark-specific planning metrics, and reinforcement-learning rewards that jointly target trajectory quality, structural validity, and endpoint precision.

  • Model Architecture: InternVL3-8B connects a 300M-parameter InternViT encoder with Qwen2.5-7B as the vision-language backbone.
  • Metrics: NAVSIMv1 uses PDMS and NAVSIMv2 uses EPDMS for closed-loop planning, while nuScenes uses L2 trajectory metrics and collision rate.
  • Metrics: PDMS combines No At-Fault Collision, Drivable Area Compliance, Time-to-Collision, Comfort, and Ego Progress into one closed-loop planning score.
  • Metrics: EPDMS evaluates NAVSIMv2 using collision, drivable-area, direction, traffic-light, progress, collision-time, lane-keeping, and comfort-related components.
  • Reward Formulation: The reinforcement-learning reward combines trajectory quality, structural validity, and goal alignment.
  • Reward Formulation: The trajectory reward uses continuous PDMS, format reward checks tags and waypoint syntax, and goal reward uses endpoint L1 distance.

E. Visualization Analysis

Visualization analyses show LaST-VLA’s geometric and dynamic reasoning across autonomous-driving tasks, while failure analysis identifies limited front-camera coverage as a key boundary.

  • Failure Analysis: The reported limitation is tied to reliance on a single front-view camera, with surround-view inputs and temporal information proposed for future expansion of perceptual range.
  • SURDS: SURDS visualizations cover yaw angle, pixel location, depth range, distance, left/right, and front/behind determination tasks.
  • NAVSIM Comparison: LaST-VLA maintains lane-compliant trajectories and anticipates collisions more effectively than the textual CoT baseline in qualitative NAVSIM comparisons.
  • Failure Analysis: Failures predominantly occur when planned paths extend beyond the front-view camera’s field of view, causing spatial-grounding errors and possible collisions or drivable-area deviations.
  • NuDynamics: NuDynamics visualization includes motion-state estimation examples for dynamic-scene reasoning.
Loading 2603.01928v2…