PaperlayerView exact revision on arXiv

Source-linked AI summary

Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving

Xingtai Gui, Yucheng Zhou, Dongqian Guo, Jiahao Gong, Feiyang Tan, Jianbing Shen

arXiv:2610.10390v1cs.CV

TL;DR

VLA driving models reason largely in a semantic space, although driving actions require precise 3D geometric cues. GeoCoTDrive grounds decision-critical 2D regions, retrieves localized 3D priors, and interleaves them into trajectory generation; experiments report improved planning across multiple benchmarks. The paper also introduces planning-relevant grounding and the PlanningGrounding dataset.

  • Problem

    VLA-based driving has a mismatch between geometric target actions and intermediate reasoning that is largely semantic and lacks explicit geometric cues.

    Source: §1 · pp. 1–2

  • Method

    GeoCoTDrive grounds decision-critical 2D regions, retrieves localized 3D priors within them, and interleaves geometry tokens into autoregressive trajectory generation, with PlanningGrounding annotations supervising region-level cues.

    Source: §1 · pp. 2–3

  • Results

    GeoCoTDrive improves planning performance across nuScenes, NAVSIM, and Bench2Drive, with experiments reporting gains in trajectory accuracy and safety-critical planning metrics.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The results highlight the importance of coupling planning-oriented 2D reasoning with localized 3D geometric priors for safer autonomous driving.

    Source: §6 · p. 10

    GeoCoTDrive primarily uses front-view images without explicit multi-view or temporal modeling, and localized geometry retrieval depends on grounding accuracy.

    Source: F Limitations · p. 20

Abstract

from arXiv · show

Vision-language-action~(VLA) models have emerged as a promising paradigm for autonomous driving. However, existing VLA models still suffer from a fundamental mismatch: driving actions require precise 3D geometric cues, while visual-language understanding and reasoning are largely conducted in a 2D semantic space. In this paper, we propose GeoCoTDrive, an explicit geometric chain-of-thought framework that grounds geometry in a planning-oriented manner. GeoCoTDrive follows a think with 2D first, drive with dedicated 3D priors paradigm. It first grounds 2D regions corresponding to decision-critical cues, and then retrieves localized 3D priors by sampling features from a geometric foundation model within the grounded regions. These localized geometric features are interleaved into the autoregressive context to support the trajectory generation. To supervise this process, we introduce planning-relevant grounding, a new region-level grounding task that focuses on local spatial cues directly affecting ego planning decisions, and construct the PlanningGrounding dataset to endow VLAs with planning-oriented grounding capability. Experiments across multiple end-to-end autonomous driving benchmarks show that GeoCoTDrive consistently improves safety-critical planning performance, demonstrating the effectiveness of the explicit geometric chain-of-thought process for VLA-based planning.

1 Introduction

GeoCoTDrive addresses the mismatch between geometric driving actions and semantic VLM reasoning by grounding decision-critical 2D regions before retrieving localized 3D priors. It introduces planning-relevant grounding and reports improved trajectory accuracy and safety-critical planning metrics across multiple benchmarks.

1 Introduction

  • VLA driving has a mismatch between geometric target actions and intermediate VLM reasoning that is largely semantic and lacks explicit geometric cues.The gap is especially relevant when decisions depend on fine-grained local spatial constraints, while prior geometry methods primarily use global scene-level representations.Source: §1 · pp. 1–2
  • GeoCoTDrive grounds sparse 2D regions for local decision-critical cues, retrieves 3D priors within them, and interleaves geometry tokens into the autoregressive context for trajectory generation.The method follows a “think with 2D first, drive with dedicated 3D priors” paradigm.Source: §1 · pp. 2–3
  • Planning-relevant grounding targets region-level cues affecting ego behavior, including lane boundaries, conflict regions, occlusions, and other local spatial constraints.The authors construct PlanningGrounding, a VQA-style dataset with 146K annotations using a semi-automatic annotation pipeline.Source: §1 · pp. 2–3
  • GeoCoTDrive generates grounding results and trajectories autoregressively, with PlanningGrounding annotations supervising grounding and expert trajectories supervising planning.At inference, both stages run sequentially in the same autoregressive pipeline.Source: §1 · p. 2
  • GeoCoTDrive consistently improves performance across multiple end-to-end autonomous driving benchmarks, including trajectory accuracy and safety-critical planning metrics.The framework explicitly grounds planning-relevant regions and retrieves dedicated local 3D features.Source: §1 · p. 2

2 Related Work

Prior VLA work uses language models for driving understanding and planning, while spatial-aware approaches add structured representations or geometric priors. GeoCoTDrive is distinguished by deriving localized geometry from explicit 2D reasoning about decision-critical regions.

2.1 VLAs for Autonomous Driving

2.2 Spatial-aware VLAs for Autonomous Driving

  • Prior studies incorporate geometric foundation models or geometry-aware representations through explicit inputs [16] [61] [50] [26] or representation alignment [68] [27] [20] [70] [28].These works validate geometric priors for spatial understanding and action generation [3] [64] [39] [47].Source: §2.2 · p. 3
  • Spatial-aware VLA methods use 3D-structured supervision [11] [73] [32] or pretrained geometric priors [54] [45] [37], but these approaches primarily introduce global geometry without 2D reasoning derivation.GeoCoTDrive instead identifies 2D decision-critical regions and retrieves localized 3D geometry.Source: §2.2 · p. 3

3 Method

GeoCoTDrive predicts future ego trajectories from visual observations and language context, using planning-relevant 2D regions to retrieve localized geometric features. Its training objective supervises grounding and trajectory responses while masking prompts, image placeholders, and inserted geometry features.

3.1 Preliminaries

  • Driving VLA models predict future ego trajectories from visual observations and language contexts, while geometric foundation models provide dense spatial features as complementary priors.The described geometric feature extractors include frozen Depth Anything V3 and VGGT models.Source: §3.1 · pp. 3–4

3.2 GeoCoTDrive

  • GeoCoTDrive generates planning-relevant regions, samples localized geometric tokens from a dense feature map, and interleaves them into the autoregressive context for trajectory decoding.The regions provide an intermediate reasoning trace that specifies where geometry should be retrieved.Source: §3.2 · pp. 4–5

3.3 Training Objective

  • Training uses an autoregressive language-modeling objective over grounding and trajectory responses, masking user prompts, image placeholders, and inserted geometry features from the loss.The multimodal sequence places grounding responses, inserted geometry tokens, and trajectory responses in one autoregressive context.Source: §3.3 · p. 5

4 Planning-oriented Grounding Dataset

PlanningGrounding targets compact regions that directly shape ego planning, including cues conventional object-centric grounding may miss. Its semi-automatic pipeline produces a 146K-annotation dataset with a manually refined test set.

4 Planning-oriented Grounding Dataset

  • Planning-relevant grounding localizes compact, decision-critical regions, including conflict areas, occlusions, and dense agent clusters that object-level grounding may overlook.The annotations are conditioned on the driving scene.Source: §4 · p. 5
  • A planning-oriented prompt identifies relevant regions and assigns categories; parsed boxes and category-reasoning pairs undergo automatic validation and human verification.Samples use front-view images, ego status, and high-level driving commands; source data comes from nuScenes, NAVSIM, and Bench2Drive.Source: §4 · pp. 5–6
  • PlanningGrounding contains 146K annotations across five planning-related categories, plus 0.6K manually refined samples for its test set.Categories include critical-object, road-boundary, conflict-area, occluded-unknown-area, and dense-object-area.Source: §4 · p. 6

5 Experiments

GeoCoTDrive is evaluated on planning and grounding tasks across nuScenes, NAVSIM, Bench2Drive, and PlanningGrounding. Results report improved planning performance, while ablations favor localized geometry retrieval guided by planning-relevant regions.

5.1 Experimental setup

  • Evaluation covers nuScenes open-loop planning, NAVSIM and Bench2Drive closed-loop planning, and PlanningGrounding localization performance.Planning metrics include trajectory error and safety, PDMS sub-scores, driving score, and success rate; grounding uses Recall@N, AP, and mIoU.Source: §5.1 · p. 6
  • Two-stage training adapts the VLM to driving, then freezes its vision encoder while aligning localized geometry features into the language space.The geometry foundation model is frozen, and benchmark configurations use different VLM backbones and trajectory-generation heads.Source: §5.1 · pp. 6–7

5.2 Main results

  • GeoCoTDrive improves over perception-centric and global geometry fusion methods on Bench2Drive, and improves NAVSIM safety-related submetrics over the Qwen2.5-VL baseline.The reported planning comparisons also describe favorable safety-related performance on nuScenes and competitive performance among VLA-based methods on NAVSIM.Source: §5.2 · p. 7, Table 3 · p. 7, Table 2 · p. 7
  • GeoCoTDrive achieves the best PlanningGrounding test-set results across all reported metrics, with especially clear gains at stricter IoU thresholds.The metrics are Recall at different IoU thresholds, AP, and mIoU.Source: §5.2 · p. 7, Table 4 · p. 7

5.3 Ablation study

  • Localized geometry retrieval from planning-grounded regions yields more consistent gains than directly interleaving global geometry tokens.The comparison indicates geometry is more effective when conditioned on decision-critical regions than injected as a global auxiliary stream.Source: §5.3 · p. 8, Table 5 · p. 8, Figure 4 · p. 8
  • PlanningGrounding annotations improve safety-related outcomes over perception labels, while both tested geometry backbones improve over baseline.DA3-LARGE has slightly better overall performance, whereas VGGT has marginally higher EP; the reported results support use of diverse pretrained geometric priors.Source: §5.3 · pp. 8–9, Table 6 · p. 9, Table 7 · p. 9

5.4 Qualitative Results

  • Qualitative results across nuScenes, NAVSIM, and Bench2Drive show stronger geometry-aware safety behavior than the baseline.The visualization covers multiple evaluation protocols and driving scenarios.Source: §5.4 · p. 9, Figure 6 · p. 9

6 Conclusion

GeoCoTDrive couples planning-relevant 2D grounding with localized 3D priors, and experiments across three benchmarks report improved planning performance. PlanningGrounding supplies region-level supervision for decision-critical cues that object-centric driving VQA datasets may not cover.

6 Conclusion

  • GeoCoTDrive grounds planning-relevant 2D regions, retrieves localized 3D priors from a frozen geometric foundation model, and interleaves geometry tokens into trajectory generation.The paper introduces PlanningGrounding as a VQA-style dataset supervising region-level decision-critical spatial cues.Source: §6 · p. 10
  • Experiments on nuScenes, NAVSIM, and Bench2Drive report improved planning performance and emphasize coupling planning-oriented 2D reasoning with localized 3D priors.Source: §6 · p. 10

Appendix

  • The appendix lists further dataset analysis, implementation details, ablations, qualitative results, and limitations sections.Source: Appendix · p. 16

A Autonomous Driving VQA

  • Existing driving VQA datasets generally ground object-level perception annotations, while PlanningGrounding targets regions directly affecting ego planning, including road boundaries, conflict zones, and uncertain areas.The appendix comparison concerns grounding supervision in existing driving VQA datasets.Source: A Autonomous Driving VQA · p. 16, Table 8 · p. 16

B Analysis of PlanningGrounding

PlanningGrounding localizes sparse, command-relevant regions that affect ego planning, including object-level and broader spatial constraints. It draws samples from nuScenes, NAVSIM, and Bench2Drive.

B Analysis of PlanningGrounding

C Implementation Details

GeoCoTDrive uses benchmark-specific VLM backbones and frozen geometric foundation models. Its implementation trains driving knowledge before geometry-interleaved planning, then samples localized priors during inference.

C Implementation Details

  • LLaVA-1.5-7B is the base VLM for nuScenes and Bench2Drive, while Qwen2.5-VL-3B is used for NAVSIM.Inputs are the front-view image, ego status, and navigation command; the model generates grounding results before the planning answer.Source: C Implementation Details · pp. 17–18
  • Frozen Depth Anything V3 METRIC-LARGE and vanilla VGGT provide geometric features sampled within grounded regions and projected into language embeddings.Source: C Implementation Details · p. 18
  • Stage-I updates the vision encoder and LLM using benchmark-specific driving QA and planning data alongside PlanningGrounding.It runs for three epochs with a cosine scheduler and peak learning rate 4 × 10−5.Source: C Implementation Details · p. 18, Table 10 · p. 18
  • Stage-II trains the model to generate grounding results before trajectories, interleaving localized geometric tokens at the <GEO_COT> position.Ground-truth boxes drive feature sampling during training; the stage runs for three epochs with peak learning rate 2 × 10−5.Source: C Implementation Details · p. 18, Table 10 · p. 18
  • At inference, predicted grounding boxes trigger localized feature sampling at <GEO_COT>, after which autoregressive decoding continues to the trajectory.Source: C Implementation Details · p. 18

D Additional Ablation Studies

Additional studies examine localized-geometry sampling, training stages, and NAVSIM turning scenarios. Qualitative planning comparisons are also provided for nuScenes and Bench2Drive.

D Additional Ablation Studies

E Additional Qualitative Results

Qualitative examples show GeoCoTDrive planning across varied road and visibility conditions, with grounded regions linked to localized geometric priors.

E Additional Qualitative Results

F Limitations

GeoCoTDrive’s front-view-only, non-temporal setup limits the traffic and interaction context it can account for, while localized geometry retrieval depends on grounding accuracy.

F Limitations

  • Front-view images without explicit multi-view or temporal modeling limit coverage of side and rear traffic, prolonged occlusions, and interaction histories.Future work will extend GeoCoTDrive to multi-view and temporal settings.Source: F Limitations · p. 20
  • Localized geometry retrieval depends on grounding accuracy; incorrect or incomplete regions may yield less relevant geometric cues and impair trajectory prediction.Source: F Limitations · p. 20
Loading 2610.10390v1…