Geo-VLA: Geometry-Aware Vision-Language-Action Planning via Internalization of Map Semantics

Ran Chen; Jiaxing Ren; Zhikun Zhang; Yunhao Hou; Junbao Zhuo; Bochao Zou

Source-linked AI summary

Geo-VLA: Geometry-Aware Vision-Language-Action Planning via Internalization of Map Semantics

Ran Chen, Jiaxing Ren, Zhikun Zhang, Yunhao Hou, Junbao Zhuo, Bochao Zou

arXiv:2608.21440v1Aug 18, 2026cs.ROcs.AI

TL;DR

VLA planners have limited representations of planning-relevant road geometry and topology, motivating supervision that addresses this gap without changing their inference interface. Geo-VLA uses Geo-QA for geometry-aware backbone pretraining and retains the original action decoder; on NAVSIM v1, it consistently improves two VLA baselines and reaches 92.1 PDMS, a new state of the art among single-camera VLA planners.

Problem
VLA models inadequately represent planning-relevant road geometry and topology from camera images, limiting their ability to capture structured road constraints.
Method
Geo-VLA adapts the vision-language backbone using Geo-QA map-derived supervision through contrastive learning and instruction tuning, then fine-tunes the original action decoder.
Results
92.1 PDMS: DynVLA + Geo-VLA establishes a new state of the art among single-camera VLA planners, while Geo-VLA improves both ReCogDrive and DynVLA.
Takeaways & Limitations
Geo-VLA internalizes planning-relevant map semantics during training while preserving the original action decoder, baseline inference interface, and no-map inference setting.
Takeaways & Limitations
Geo-VLA’s training-time supervision is bounded by the coverage and quality of offline map annotations, and evaluation is limited to single-camera planners under NAVSIM’s non-reactive protocol.

Abstract

from arXiv · show

Vision-language-action (VLA) models have advanced end-to-end autonomous driving by leveraging foundation models for semantic reasoning and long-tail generalization. However, their planning performance remains limited in complex driving environments because image-only representations inadequately capture planning-relevant road geometry and topology. In this paper, we propose Geo-VLA, a plug-and-play framework that enhances VLA models by learning geometry-aware visual representations. During training, Geo-VLA internalizes geometric map semantics to strengthen road-structure representations, while requiring no HD maps or additional lane information during inference. To support this approach, we introduce Geo-QA, a geometry-focused question-answering dataset that injects road geometry into vision-language representations through contrastive learning and instruction tuning. Experiments on NAVSIM v1 demonstrate that Geo-VLA consistently improves VLA planners with distinct action-generation architectures, achieving 92.1 PDMS and establishing a new state-of-the-art among single-camera VLA planners.

1 Introduction

Existing VLA planners struggle to represent road geometry and topology from front-view images, causing trajectory deviations in curves and intersections. Geo-VLA addresses this gap by internalizing map-derived geometry during training while preserving the baseline inference interface.

Existing VLA visual representations inadequately capture road geometry and topology needed for trajectory planning.Relevant constraints include lane boundaries, curvature, intersection connectivity, drivable-area boundaries, and lane-level topology.
Single front-view images make lane connectivity and turning restrictions difficult to infer reliably from road appearance and markings.
HD-map encoders provide structured road features but require collection, annotation, maintenance, localization, retrieval, and additional inference computation.They also change the original VLA input setting without addressing the backbone’s limited road-structure representation.
Geo-VLA adapts the vision-language backbone with map-derived geometric supervision, then connects it to each baseline’s original action decoder for trajectory fine-tuning.The action decoder is not redesigned, and inference uses the baseline interface without HD maps, map text, or additional map encoders.
Geo-QA contains 3,000 image-description samples covering lane structure, road direction and curvature, intersections, drivable boundaries, and topology.Image-text pairs support contrastive learning, while complete question-answer samples support instruction tuning.
Ablations compare static road-structure and dynamic-object supervision, with static supervision providing greater planning benefits.

2 Related Work

Prior autonomous-driving methods enhance scene representations, map reconstruction, language supervision, or action generation, but static road geometry and topology are not dedicated representation-learning targets. Geo-VLA instead internalizes map-derived road semantics before planning fine-tuning while retaining the original action decoder and inference interface.

Prior end-to-end driving methods improve spatiotemporal, multiview, planning-oriented, temporal, dynamics, or action-generation capabilities.
Existing approaches either represent map elements directly as structured planning features or reconstruct structured maps online from sensor observations.
Offline map-based approaches depend on collection, annotation, maintenance, and updates, whereas online reconstruction adds localization, map construction, and related inference operations.
Geo-VLA uses offline map annotations only as training supervision and remains compatible with each baseline’s original action decoder.
Geo-QA focuses language supervision on map-derived road semantics through image-text contrastive learning and question-answer instruction tuning.

3 Method

Geo-VLA adapts a VLA backbone with geometry-focused supervision from Geo-QA, then fine-tunes the original action decoder for planning. Offline map information guides training but is removed from the deployed inference interface.

Two-stage framework: Geo-VLA first adapts the vision-language backbone through geometry-aware pretraining, then fine-tunes the original action decoder under baseline imitation learning.The first stage leaves the action decoder fixed; the second fixes the adapted backbone and optimizes the decoder with expert trajectories.
Geo-QA construction: Geo-QA contains 3,000 balanced image-question-answer samples covering lane structure, road geometry, intersections, drivable-area boundaries, and topology.Each sample pairs a front-view image with an offline map annotation and a category-specific generation instruction for GPT-5.4.
Geometry-aware pretraining: Contrastive learning aligns images with map-semantic descriptions, while instruction tuning trains map-grounded question-answer responses.The same answer serves as the textual map-semantic description for contrastive learning.
Geometry-aware pretraining: The pretraining loss jointly combines image-to-text retrieval, text-to-image retrieval, and autoregressive answer prediction, weighted by λ.The contrastive term organizes road-structure representations across scenes, while instruction tuning makes them usable for geometry-focused QA.
Inference interface: At inference, the adapted backbone passes geometry-aware hidden states through the unchanged baseline interface without HD maps, map text, or contrastive projection heads.Map annotations and dataset-construction artifacts are used only during training, and the auxiliary projection heads are discarded afterward.

4 Experiments

Experiments evaluate Geo-VLA across VLA planners, map-encoding strategies, QA supervision types, and representative turning scenes on NAVSIM v1. Geo-VLA improves planning while retaining the baseline inference interface and better aligning trajectories with road geometry.

Main Results: 92.1 PDMS: DynVLA + Geo-VLA establishes a new state of the art among single-camera VLA planners.It exceeds the strongest prior result by 0.3 PDMS.
Main Results: 91.2 PDMS and 92.1 PDMS: Geo-VLA improves ReCogDrive from 90.8 and DynVLA from 91.0, respectively.
Map Encoding Comparison: Geo-QA matches lane detection on ReCogDrive and exceeds it by 0.4 PDMS on DynVLA while retaining baseline inference overhead.It remains within 0.7 and 0.3 PDMS of corresponding HD map variants.
Static versus Dynamic QA Supervision: 92.1 PDMS: static road-structure QA exceeds the 91.0 baseline and 90.7 dynamic-QA variant under equal-scale supervision.The comparison targets whether gains arise from static geometry rather than generic QA or sample count.
Qualitative Results: Geo-VLA follows turn headings and stays closer to expert lane corridors, while Geo-QA follows road curvature more closely than online lane detection.Figure 4 compares predicted and expert trajectories in representative turning and curved-road scenes.

5 Conclusion

Geo-VLA and Geo-QA address limited road-geometry and topology representations by internalizing map semantics during training while preserving the original inference interface. The method improves single-camera VLA planning, but its supervision and evaluation scope remain bounded.

Contributions: Geo-VLA combines contrastive learning and QA instruction tuning to internalize map semantics while retaining the original action decoder and requiring no HD maps at inference.
Results: 92.1 PDMS: Geo-VLA consistently improves ReCogDrive and DynVLA and establishes a new state of the art among single-camera VLA planners.
Results: Geo-VLA approaches HD map performance and surpasses lane detection on DynVLA while retaining baseline inference cost.
Limitations: Offline map-annotation coverage and quality bound Geo-QA training supervision, while evaluation is limited to single-camera planners under NAVSIM's non-reactive protocol.Behavior under interactive traffic remains to be established.

A Geo-QA Construction and Sampling

Geo-QA is built from NAVSIM front-view frames paired with local offline map annotations and balanced across five planning-relevant road-geometry categories. GPT-5.4 converts each pairing into map-grounded question-answer supervision and contrastive text.

Sampling: Each sample associates a NAVSIM front-view image Ii with a local map annotation Mi and category label ri.The category label controls generation and dataset balancing but is not used by the VLA planner at inference.
Generation: GPT-5.4 generates a geometry-focused question and map-grounded answer from Ii, Mi, and the category-specific instruction.The instruction targets planning-relevant road relations rather than generic captions or object lists.
Category design: The five categories cover complementary map semantics, including lateral constraints, direction and curvature, local intersection layouts, and connectivity relations.Topology questions address connectivity, merges, and splits, while each sample extracts a local relation affecting the planned trajectory’s feasible region or direction.

B Geo-QA Data Format

Geo-QA uses a conversation-style JSONL format that separates metadata from a two-turn image-question-answer interaction. The serialized answer serves as both the QA target and map-semantic text for pretraining.

Data format: The experimental Geo-QA file contains 3,000 records whose image fields reference sampled NAVSIM front-view frames under map_qa_images/.Each record includes sample metadata and a two-turn conversation.
Record contents: A representative record pairs a sampled front-view image with a geometry-focused question and a map-grounded answer.The example asks for distances from the ego vehicle to the left and right road boundaries.
Metadata: Category identifiers follow lane, geometry, intersection, boundary, and topology, while the category field is excluded from the deployed planner.Metadata is separated from the two-turn conversation in the serialized record.
Record contents: The schema stores the conversational input as an image and question, with the map-grounded answer as the prediction target.The underlying conversation uses human and gpt roles in standard from/value fields.

C Static and Dynamic QA Probe

The QA probe distinguishes static road-structure supervision from dynamic-object supervision using equal-sized training sets. Static QA produces a much larger average gain and improves both measured static and dynamic scores.

Probe results: 74.35 average GPT-Score is reached with static QA, compared with 67.00 using dynamic QA under the same data scale.The baseline average is 66.70.
Interpretation: The probe associates Geo-QA’s gain primarily with improved static road-structure understanding rather than generic QA supervision.The authors suggest dynamic content is already directly observable and well represented by the VLA backbone, whereas static constraints are harder to infer visually.

D Complete Objective Definitions

Geo-VLA’s complete objective combines image-text contrastive learning and QA instruction tuning for geometry-aware pretraining, then applies the selected planner’s original imitation-learning objective during planning fine-tuning. The two-stage protocol adapts the backbone first and optimizes the decoder second.

Contrastive pretraining: The image encoder and text path produce pooled features that projection heads map into a shared dz-dimensional embedding space.These projected features support scaled image-text similarity computation.
Contrastive pretraining: Matched image-text pairs use the same sample index as positives, while other in-batch pairs form negatives in the scaled similarity objective.The temperature τ is positive and controls similarity scaling.
Contrastive pretraining: Geo-VLA uses symmetric image-to-text and text-to-image InfoNCE losses for contrastive alignment.The symmetric objective averages the two directional losses.
Instruction tuning: QA instruction tuning applies an autoregressive token-level loss to the answer sequence, conditioning each token on the image, question, and preceding answer prefix.The conditional token distribution is denoted pθ, and ai,<ℓ is the prefix before token ℓ.
Combined objective: The complete pretraining objective balances contrastive alignment and QA instruction tuning with λ.The two objectives jointly adapt the vision-language backbone.
Planning fine-tuning: Planning fine-tuning preserves each planner’s input tuple, action representation, and original imitation-learning loss rather than adding a separate planner loss.The generic planning objective optimizes decoder parameters ψ on planning data Dplan.
Training protocol: The two-stage implementation trains LoRA adapters in the vision encoder and LLM backbone before fixing the adapted backbone and optimizing only decoder parameters.Projection heads are discarded between geometry-aware pretraining and planning fine-tuning.