PaperlayerView exact revision on arXiv

Source-linked AI summary

VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving

Jie Wang, Guang Li, Zhijian Huang, Chenxu Dang, Hangjun Ye, Yahong Han, Long Chen

arXiv:2602.20794v1cs.CV

TL;DR

VLMs lack the cross-view 3D geometric modeling required for fine-grained autonomous driving, and existing remedies provide limited geometric grounding. VGGDrive integrates frozen 3D features into a base VLM through CVGE and hierarchical adaptive injection, improving performance across five autonomous driving benchmarks and diverse evaluation protocols. The approach also relies on known camera intrinsic and extrinsic parameters for some 3D scene-mapping tasks.

  • Problem

    VLMs lack cross-view 3D geometry needed for fine-grained spatial understanding in autonomous driving, while existing integrations are limited in setting and effectiveness.

  • Method

    VGGDrive uses a frozen 3D foundation model and a plug-and-play CVGE with hierarchical adaptive injection to integrate 3D geometric features into a base VLM.

  • Results

    VGGDrive achieves comprehensive and significant improvements across five autonomous driving benchmarks covering scene understanding, cross-view risk perception, motion prediction, and trajectory planning.

  • Takeaways & Limitations

    The results validate the feasibility and potential of using 3D foundation models to provide VLMs with cross-view geometric capabilities for autonomous driving.

  • Takeaways & Limitations

    The approach explicitly uses camera intrinsic and extrinsic parameters as known prior information for trajectory-planning tasks requiring complete 3D scene mapping.

Abstract

from arXiv · show

The significance of cross-view 3D geometric modeling capabilities for autonomous driving is self-evident, yet existing Vision-Language Models (VLMs) inherently lack this capability, resulting in their mediocre performance. While some promising approaches attempt to mitigate this by constructing Q&A data for auxiliary training, they still fail to fundamentally equip VLMs with the ability to comprehensively handle diverse evaluation protocols. We thus chart a new course, advocating for the infusion of VLMs with the cross-view geometric grounding of mature 3D foundation models, closing this critical capability gap in autonomous driving. In this spirit, we propose a novel architecture, VGGDrive, which empowers Vision-language models with cross-view Geometric Grounding for autonomous Driving. Concretely, to bridge the cross-view 3D geometric features from the frozen visual 3D model with the VLM's 2D visual features, we introduce a plug-and-play Cross-View 3D Geometric Enabler (CVGE). The CVGE decouples the base VLM architecture and effectively empowers the VLM with 3D features through a hierarchical adaptive injection mechanism. Extensive experiments show that VGGDrive enhances base VLM performance across five autonomous driving benchmarks, including tasks like cross-view risk perception, motion prediction, and trajectory planning. It's our belief that mature 3D foundation models can empower autonomous driving tasks through effective integration, and we hope our initial exploration demonstrates the potential of this paradigm to the autonomous driving community.

1. Introduction

VLMs offer strong semantic reasoning but lack cross-view 3D geometric modeling needed for precise autonomous driving. VGGDrive addresses this gap by integrating frozen 3D features into a base VLM through CVGE and hierarchical adaptive injection.

  • VLMs inherently lack cross-view 3D geometry, limiting fine-grained spatial understanding for safe navigation in complex autonomous driving environments.
  • Q&A-based spatial training provides only limited improvements because it does not fundamentally establish solid geometric priors.
  • Existing VLM–3D foundation model integrations mainly target indoor, static, monocular settings and use simple schemes unsuitable for dynamic, multi-camera driving.
  • The paper asks how VGGT can provide VLMs with cross-view geometric grounding to compensate for their limitations in autonomous driving.
  • VGGDrive uses a frozen 3D foundation model to derive geometrically consistent features and a plug-and-play CVGE to inject them into 2D VLM embeddings hierarchically.

2. Related Work

Related work progresses from VLM-based scene understanding toward VLA models and the use of visual 3D foundation models. However, existing approaches remain limited by weak geometric integration and mismatched indoor or single-camera settings.

  • VLMs for Autonomous Driving: VLMs and VLA models use visual inputs for scene understanding, reasoning, and prediction of driving paths or waypoints.
  • VLMs for Autonomous Driving: Q&A supervision improves spatial concepts only limitedly, while separate action decoders improve trajectory prediction but disconnect scene understanding from action decision-making.
  • Visual 3D Foundation Models: Visual 3D foundation models pretrained on large-scale 3D datasets offer strong cross-view scene geometric modeling.
  • Visual 3D Foundation Models: VGGT processes multi-view images to reconstruct 3D scenes, including camera parameters, depth maps, and point clouds.
  • Visual 3D Foundation Models: Existing VLM–3D integrations primarily address indoor, static, single-camera scenarios and use simplistic strategies that do not meet complex driving requirements.

3. Methodology

VGGDrive augments Qwen2.5-VL with frozen VGGT cross-view 3D features through a hierarchical, layer-adaptive injection architecture. Its CVGE uses cross-attention to let 2D visual embeddings query and integrate shared geometric representations while preserving camera and multi-view information.

  • Architecture: VGGDrive combines a Qwen2.5-VL base model with a frozen VGGT model and a hierarchical adaptive injection mechanism.The architecture decouples the base LLM so hidden states and visual embeddings can be accessed at each decoder layer.
  • Base VLM Overview: The model accepts language instructions and surround-view images, using six views for nuScenes-based benchmarks and three front views for NAVSIM.Trajectory-planning prompts additionally include ego-vehicle state and navigation commands.
  • Cross-view 3D Geometric Enabler: The geometric representation retains VGGT camera and registration embeddings, while explicit camera parameters are encoded into attention keys and values for planning tasks.Intrinsic and extrinsic parameters are used because complete 3D scene mapping is important for NAVSIM and NuScenes-Plan.
  • Hierarchical Adaptive Injection Mechanism: CVGE uses independent but structurally consistent modules across layers so each layer can learn geometric information suited to its representation and 3D sensitivity.This layer-wise design provides adaptive extraction rather than applying one shared transformation to all decoder layers.

4. Experiment

VGGDrive is evaluated across five autonomous-driving benchmarks spanning language understanding and trajectory planning, with ablations examining its integration scheme and components. It improves cross-view perception, planning, and related metrics while preserving captioning performance.

  • Benchmark Setup: VGGDrive is evaluated on five benchmarks covering cross-view risk perception, state prediction, captioning, open-loop planning, and closed-loop planning.The benchmarks are NuInstruct, DriveLM, Omnidrive, NuScenes-Plan, and NAVSIM.
  • Trajectory Planning: 88.76 PDMS is achieved on NAVSIM, with a 2.72 improvement over the base VLM and nearly 2 points over comparable two-stage VLA methods.The comparison describes VGGDrive as optimal among VLM-based autoregressive trajectory-generation methods.
  • Cross-View Perception: 15.23 and 6.67 improvements in DriveLM Match and Average metrics exceed current SOTA methods by 2.12 and 1.24, respectively.DriveLM evaluates cross-view object perception, action prediction, and planning.
  • Trajectory Planning: 8% improvement in collision rate is reported on the nuScenes open-loop trajectory-planning benchmark over existing SOTA methods.VGGDrive also retains caption-related performance on Omnidrive despite adding cross-view capabilities.
  • Ablation Studies: Ablations show that the proposed integration scheme outperforms direct replacement, distillation, addition, and multi-head cross-attention alternatives across NAVSIM and NuInstruct.Further analyses examine adaptive injection, MHCA, residual injection, parameter sharing, and the 3D expert model; Fast3r improves performance but slightly underperforms VGGT.

5. Conclusion

The paper concludes that VGGDrive uses CVGE and hierarchical adaptive injection to integrate cross-view geometric features into VLMs. Results across five benchmarks support the feasibility and potential of using 3D foundation models for autonomous driving.

  • Conclusion: VGGDrive integrates cross-modal features through CVGE and a hierarchical adaptive injection mechanism to empower VLMs.The architecture is presented as an alternative to large-scale Q&A training or independent action decoders.
  • Conclusion: Experiments across five mainstream autonomous-driving benchmarks show comprehensive and significant improvements across diverse evaluation protocols.The evaluated capabilities include cross-view geometric grounding for autonomous-driving tasks.
  • Conclusion: VGGDrive validates the feasibility and potential of using 3D foundation models to provide cross-view geometric capabilities for autonomous driving.The paper characterizes this as a distinct technical pathway for deploying VLMs in autonomous driving.

Supplementary Material

The supplementary material clarifies benchmark applications and evaluation metrics, adds an ablation on injection layers, provides visual samples, and reports dataset statistics and evaluated capabilities.

  • Supplementary Organization: Section A clarifies VGGDrive’s application across five benchmarks and their corresponding evaluation metrics.
  • Supplementary Organization: Section B analyzes injecting 3D features into different VLM decoding layers on NAVSIM closed-loop trajectory planning.
  • Supplementary Organization: Section C provides additional visual samples, while Table S1 reports training and testing statistics and evaluated capabilities for five datasets.

A. Dataset and Metric

VGGDrive is evaluated across five autonomous-driving benchmarks spanning scene understanding, cross-view perception, prediction, dialogue, and trajectory planning, using task-specific metrics.

  • Benchmarks: Five benchmarks evaluate scene understanding, cross-view risk perception, action and state prediction, and trajectory planning.The datasets include NuInstruct, DriveLM, OmniDrive, NAVSIM, and nuScenes.
  • NAVSIM: NAVSIM uses eight cameras with 360° field of view and merged LiDAR from five sensors for complex dynamic driving scenarios.It contains 1,192 training scenarios and 136 testing scenarios.
  • Metrics: NAVSIM evaluates closed-loop planning with the Predictive Driver Model Score (PDMS).PDMS integrates No At-Fault Collision, Drivable Area Compliance, Time-to Collision, Comfort, and Ego Progress.
  • NuInstruct: NuInstruct covers cross-view risk perception, distance estimation, agent and ego state prediction, target motion prediction, and reasoning tasks.Its metrics include regression errors and classification accuracy.
  • DriveLM: DriveLM focuses on interactive urban-road scenarios involving vehicles, pedestrians, infrastructure, and changes in driving intentions.Its samples address both present-object perception and future object motion.
  • Other benchmarks: OmniDrive contributes scene description and general dialogue evaluation, while nuScenes supports open-loop trajectory planning with L2 and Collision Rate metrics.nuScenes contains 1,000 driving scenes captured by six cameras with a 360° horizontal field of view.

B. VLM Layer Injection Ablation

The ablation evaluates where cross-view 3D features should enter the VLM decoder and finds that adaptive injection across all layers outperforms single-layer alternatives.

  • Ablation setup: Cross-view 3D feature empowerment and adaptive injection are tested on individual decoding layers while preserving the final model configuration.The experiments target closed-loop trajectory planning on NAVSIM.
  • Single-layer injection: PDMS around 88 with single-layer injection exceeds the base VLM’s PDMS = 86.04 and existing VGGT-VLM integration solutions with PDMS < 87.The comparison uses the NAVSIM closed-loop planning task.
  • Full-layer injection: Full-layer adaptive injection achieves better performance than single-layer injection.The result supports the proposed combination of the Cross-View 3D Geometric Enabler and decoupled adaptive injection.
  • Layer analysis: Peak closed-loop planning performance occurs around decoding layer 11, while reasonable performance is maintained at both ends of the decoder.Performance varies substantially depending on the selected injection layer.

C. Qualitative Results

Qualitative examples illustrate VGGDrive’s trajectory-planning and action-or-state prediction capabilities across diverse autonomous-driving maneuvers and benchmarks.

  • Closed-loop planning: NAVSIM visualizations cover left turns, right turns, and straight driving in closed-loop trajectory planning.These examples are presented in Figures S2, S3, and S4.
  • Open-loop planning: Figure S5 presents qualitative examples of open-loop trajectory planning on nuScenes.The examples complement the closed-loop NAVSIM visualizations.
  • Action and state prediction: Figure S6 illustrates prediction of ego-vehicle and surrounding-agent actions and states after cross-view scene perception and modeling.The examples address action and state prediction in autonomous-driving scenarios.
  • Cross-view grounding: The visualizations are presented as evidence of improved VGGDrive performance from injecting VGGT cross-view 3D scene features into VLM-driven tasks.The section also states that the approach avoids large-scale VQA datasets and additional trajectory-generation action decoders.
Loading 2602.20794v1…