PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning

Chun-Hsiao Yeh, Shengyi Qian, Manchen Wang, Yi Ma, Joseph Tighe, Fanyi Xiao

arXiv:2605.30231v1cs.CVcs.AI

TL;DR

VLMs struggle with robust 3D spatial reasoning, and existing solutions rely on biased VQA supervision or cumbersome specialized encoders. GASP injects geometric priors into transformer layers and improves correspondence matching and downstream spatial reasoning without 3D VQA training.

  • Problem

    VLMs struggle with robust 3D spatial reasoning, while prevailing VQA fine-tuning and specialized 3D encoders have important limitations.

  • Method

    GASP injects geometric priors into all LLM transformer layers using deep correspondence supervision with point-correspondence and depth-consistency objectives.

  • Results

    +18.2% on All-Angles Bench and +29.0% on VSI-Bench, while peak correspondence accuracy exceeds 70% and temporal robustness exceeds 85%.

  • Takeaways & Limitations

    Learning fundamental geometric priors is a promising pathway toward more reliable and generalizable 3D spatial reasoning in VLMs.

  • Takeaways & Limitations

    GASP trades about 1–2% general VQA accuracy for gains on spatial and temporal benchmarks, with a 1.9% NextQA drop reported.

Abstract

from arXiv · show

Vision-Language Models (VLMs) often struggle with robust 3D spatial reasoning. Prevailing methods that rely on fine-tuning with 3D visual question-answering (VQA) datasets may overfit dataset-specific biases, while integrating specialized 3D visual encoders is often inflexible and cumbersome. In this paper, we argue that genuine spatial understanding should emerge from learning fundamental geometric priors, not only from high-level VQA supervision. We propose GASP (Geometric-Aware Spatial Priors), a framework that injects these priors directly into the LLM's transformer layers. GASP employs a small correspondence head, applied as a deep supervision signal across all layers, and is trained with a dual objective leveraging ground-truth geometry from large-scale video scenes: a contrastive loss on ground-truth point correspondences enforces 2D view-invariance, while a depth consistency supervision resolves 3D geometric ambiguities. Our analysis first provides a diagnostic showing that standard VLMs' internal correspondence matching accuracy is very low (often below 5%). We then demonstrate that our training substantially improves this behavior, boosting peak layer-wise correspondence to over 70% and maintaining over 85% temporal robustness while baselines remain below 5%. These internal improvements translate to significant gains on downstream spatial benchmarks including +18.2% on All-Angles Bench and +29.0% on VSI-Bench, all without training on any 3D VQA data. Our findings indicate that learning from fundamental geometric priors is a promising and generalizable pathway towards VLMs with more reliable 3D spatial reasoning.

1 Introduction

GASP injects fundamental geometric priors into VLM transformer layers through correspondence and depth supervision, targeting view-invariant representations rather than dataset-specific 3D VQA patterns. It substantially improves internal correspondence matching and downstream spatial reasoning without training on 3D VQA data.

  • Introduction: Specialized 3D VQA-finetuned models can improve in-domain VSI-Bench performance but consistently drop on out-of-domain benchmarks, while specialized visual encoders remain rigid.GASP instead learns geometric priors without 3D VQA training and processes as a standard VLM during inference.
  • Introduction: View-invariant visual correspondence is proposed as the foundation for more generalizable 3D spatial reasoning across changing viewpoints.The approach emphasizes object constancy and internal geometric consistency rather than associating text with dataset-specific visual patterns.
  • Introduction: GASP injects geometric priors into every VLM transformer layer through a lightweight correspondence head with deep supervision.Its dual objective combines point-correspondence contrastive learning across frames with depth-consistency loss from large-scale video scenes.
  • Introduction: +18.2% on All-Angles Bench, +29.0% on VSI-Bench, and +15.0% on BLINK demonstrate downstream gains in camera pose estimation, object counting, and multi-view reasoning.These improvements accompany only minor changes in general video QA performance.
  • Introduction: Over 70% peak layer-wise correspondence matching accuracy and over 85% temporal robustness are achieved, versus below 5% for baseline VLMs.The analysis covers VLM backbones including Qwen2.5-VL-7B and LLaVA-NeXT-Video-7B.

2 Related Works

Related work pursues 3D-aware VLMs through explicit spatial representations or architectures that fuse 3D geometry with 2D semantics. Spatial-reasoning research has also centered on large-scale 3D-related VQA datasets and fine-tuned specialized models, raising concerns about superficial correlations and dataset-specific bias.

  • 3D-Aware VLMs: 3D-aware VLMs commonly process explicit point-cloud or pre-segmented-object data, or project multi-view images into voxel or BEV representations.These approaches make 3D structure explicit before or during multimodal processing.
  • 3D-Aware VLMs: Other 3D-aware methods use dual encoders or grounding agents to fuse 3D geometry features with 2D semantic features.This represents a complementary architectural strategy to explicit spatial representations.
  • Spatial Reasoning in VLMs: Large-scale 3D-related VQA datasets have become a dominant route for fine-tuning specialized models for spatial reasoning in VLMs.This paradigm was catalyzed by benchmarks such as VSI-Bench.
  • Spatial Reasoning in VLMs: Reliance on VQA fine-tuning may lead VLMs to learn superficial correlations and memorize dataset-specific biases, resulting in poor generalization.The passage contrasts this reliance with the paper’s alternative approach, but the provided text truncates before specifying it.

3 Preliminaries: Self-Attention in VLMs

VLMs concatenate visual and language tokens into a unified sequence processed by transformer self-attention, whose visual self-attention matrix directly exposes learned spatio-temporal correspondences. The section hypothesizes that geometrically consistent visual self-attention can provide an inductive bias for high-level spatial understanding.

  • Self-Attention Formulation: VLMs concatenate visual tokens V ∈ R^(N×d) and language tokens L ∈ R^(M×d) into X = Concat(V, L) ∈ R^((N+M)×d) for the LLM backbone.Each transformer layer projects X into queries, keys, and values for scaled dot-product attention.
  • Attention Decomposition: The attention similarity matrix S = QK^T partitions into visual self-attention, language self-attention, and cross-modal attention quadrants.Query and key matrices are partitioned according to whether tokens originate from visual or language inputs.
  • Geometric Correspondence: Visual self-attention Q_VK_V^T provides a direct window into learned spatio-temporal correspondence relevant to geometric reasoning.The section focuses on this QK matching rather than the other attention quadrants.
  • Geometric Correspondence: The authors hypothesize that explicitly learning visual self-attention representations to be geometrically consistent can unlock high-level spatial understanding.This hypothesis is motivated by the role of QK matching in temporal consistency and proposes geometric awareness as a robust inductive bias.

4 Learning Geometric Correspondence

GASP injects geometric priors into VLM transformer layers through a lightweight correspondence head supervised by 2D point correspondences and 3D depth consistency. Its combined objective jointly optimizes language, correspondence, and depth consistency without relying on 3D VQA datasets.

  • Geometric correspondence head: GASP attaches a lightweight correspondence head to an intermediate LLM transformer block, enabling direct supervision of internal geometric understanding.The head operates on visual tokens and minimally alters the base VLM architecture.
  • Geometric correspondence head: The two-layer MLP projects general-purpose visual features into a lower-dimensional correspondence embedding space, with weights initialized using SVD of the pretrained query projection matrix.Its layers map d → 2d_emb with GELU and then 2d_emb → d_emb.
  • 2D correspondence supervision: InfoNCE contrastive learning uses ground-truth point correspondences as positives and all other target-frame points as negatives to learn view-invariant embeddings.The method favors contrastive learning over coordinate regression because it supports view invariance and diverse negatives in high-dimensional feature spaces.
  • 3D depth consistency: Depth consistency supervision aligns geometrically valid correspondences across frames without directly regressing depth values.Soft matching weights produce differentiable expected depths, while the relative loss is scale-invariant across scenes with varying depth ranges.
  • Joint geometric supervision: The final multi-task objective combines LLM loss with correspondence and depth losses, jointly optimizing language, 2D correspondence, and 3D depth consistency without 3D VQA datasets.The depth term acts as a discriminative geometric regularizer rather than a depth estimator.

5 Experiments

Experiments show that GASP improves internal geometric representations and downstream spatial reasoning, with gains strongest on geometry-related tasks. The method incurs a modest general-VQA trade-off and reveals that supervision placement and LoRA rank affect the relationship between correspondence quality and benchmark performance.

  • Internal Representation Analysis: GASP evaluates layer-wise correspondence matching, confidence-accuracy correlation, and temporal robustness against pretrained Qwen2.5-VL-7B and LLaVA-NeXT-Video-7B baselines.The comparison includes full GASP with Lcorr + Ldepth and correspondence-only GASP with Lcorr.
  • Internal Representation Analysis: Near-zero PCK in baselines contrasts with improved matching from GASP, peaking at layers 20–25 for LLaVA and 25–28 for Qwen2.5-VL.The full model consistently outperforms correspondence-only GASP, supporting the contribution of depth consistency supervision.
  • Internal Representation Analysis: ρ ≈+0.62 for the full model contrasts with ρ ≈−0.22 for baselines, indicating improved confidence calibration and learned geometric reasoning.The baseline correlation is described as a positional-bias signature in which higher confidence predicts incorrect matches.
  • Internal Representation Analysis: Over 85% performance is maintained by GASP at 24-frame distances, whereas the correspondence-only baseline retains < 5% beyond an 8-frame gap.This pattern demonstrates graceful degradation and temporal-invariant geometric features.
  • Downstream Spatial Benchmarks: 34.1%→52.8% on Qwen2.5-VL Camera Pose Estimation and 24.7%→29.8% on LLaVA-NeXT Relative Direction show stronger All-Angles Bench geometric consistency.On VSI-Bench Object Counting, performance improves from 23.5% to 52.5%, consistent with better cross-view object identity.
  • General-Purpose Benchmarks: GASP loses 1.9% on NextQA, from 76.6% to 74.7%, but improves Video-MME from 59.3% to 61.6% and TempCompass from 68.4% to 70.3%.The reported trade-off favors applications where spatial geometry is more important than action-centric understanding.

6 Conclusion · Appendix · Overview

The paper proposes GASP, which injects fundamental geometric priors into LLM transformer layers and improves internal correspondence accuracy and downstream spatial performance. The supplementary material documents the data, implementation, evaluation, and VSI-Bench analyses supporting this approach.

  • 6 Conclusion: GASP injects fundamental geometric priors directly into the LLM’s transformer layers.The framework is designed to improve spatial intelligence without relying only on high-level VQA supervision.
  • 6 Conclusion: Over 70% internal correspondence accuracy demonstrates GASP’s correction of VLMs’ near-zero baseline behavior.The conclusion reports that GASP boosts internal correspondence accuracy from near zero to over 70%.
  • 6 Conclusion: GASP’s internal improvements generalize to significant gains on downstream spatial benchmarks.The conclusion connects improved internal correspondence behavior with downstream benchmark gains, without specifying additional values in this passage.
  • 6 Conclusion: The authors identify learning from geometric priors as a promising and generalizable path toward spatially intelligent VLMs.This is the paper’s stated overall conclusion regarding the proposed training strategy.
  • Appendix: The supplementary material details geometric training-data collection in Section A.These materials provide additional information about how the geometric training data were collected.
  • Appendix: Section B provides implementation details, including the correspondence head architecture Hc and all training hyperparameters.The appendix identifies both the architecture and optimization settings as part of the full implementation description.
  • Appendix: Section C details the evaluation protocol for measuring correspondence in GASP and the baselines.The evaluation protocol is applied to both the proposed model and comparison systems.
  • Overview: The supplementary material quantitatively analyzes VSI-Bench’s inherent biases and performance.This analysis is presented after the data, implementation, and correspondence-evaluation details.

A Training Dataset Collection

The training dataset uses multi-view video and depth maps to generate dense point correspondences, validating projections through depth consistency and balancing visible positive tracks with perturbed negatives.

  • Annotation generation: Dense point correspondence annotations are generated from DL3DV multi-view video sequences and depth maps using VGGT’s annotation recipe.The dataset leverages multi-view video sequences and depth maps from DL3DV.
  • Geometric construction: Correspondences are formed by back-projecting valid query-frame pixels into 3D world coordinates and projecting them into subsequent frames using camera intrinsics and extrinsics.The procedure uses COLMAP Structure-from-Motion camera parameters from DL3DV and VGGT.
  • Geometric validation: Projected correspondences are retained only when the depth difference satisfies |Dproj − Dmap| < 0.05 × min(Dproj, Dmap).This depth-consistency check compares projected depth with the depth-map value at the projected location.
  • Balanced sampling: Positive tracks come from validated 3D projections, prioritize points visible across at least 2 frames, and retain the top 50% ranked by visibility duration.The sampling targets 8 × 8 and 24 × 24 points per video frame.
  • Balanced sampling: Negative correspondences are generated through random spatial perturbations within 50% to balance the training signal.The dataset samples both positive and negative correspondences.

B Additional Implementation Details

The implementation uses an SVD-initialized correspondence head alongside LoRA-based training with specified optimization settings. Joint training combines 3D point correspondences with general video QA to preserve video understanding while improving spatial reasoning.

  • Correspondence Head Architecture: The correspondence head is a 2-layer MLP projecting dh to dh/2 and back, with GELU activation and SVD initialization from the attention query projection WQ.The hidden dimensions are dh = 3584 for Qwen2.5-VL-7B and dh = 4096 for LLaVA-NeXT-Video-7B.
  • Training Hyperparameters: LoRA finetuning uses rank r = 512 for LLaVA-NeXT-Video-7B and r = 128 for Qwen2.5-VL-7B on attention projections WQ, WK, WV, and WO.The correspondence head is trained in full precision with cosine scheduling, 10% warmup, and 3 epochs.
  • Training Hyperparameters: The loss weights are λc = 0.3 for contrastive loss and λd = 1.0 for distance loss.These settings correspond to the loss function in Equation 9.
  • Joint Training Data Composition: Joint training combines 1.75M DL3DV-derived point correspondence annotations with 100K general video QA samples from LLaVA-Video-178K.This composition is intended to maintain general video understanding while acquiring fine-grained spatial reasoning abilities.

C Correspondence Evaluation Protocol

The correspondence evaluation computes patch-level PCK for baseline and GASP models using transformer-layer query-key states, then assesses prediction accuracy and confidence calibration in 2D patch space.

  • Evaluation setup: The protocol evaluates correspondence accuracy (PCK) for LLaVA-NeXT-Video-7B, Qwen2.5-VL-7B, and GASP models.For baselines without explicit correspondence heads, query and key states are extracted from each transformer layer during the forward pass.
  • Correspondence matching: Predicted target patches are selected by maximizing cosine similarity between source-frame query features and target-frame key features.For source features Q_0 and target features K_j, the correspondence matrix is S = CosineSim(Q_0, K_j^T), with predicted patch p̂_i = arg max_j S_ij.
  • Calibration analysis: Calibration is measured by comparing Euclidean patch-space distances for correct predictions (d < 2) and incorrect predictions.Ground-truth and predicted patch indices are converted to 2D grid coordinates, and their distance defines the calibration gap between confidence on correct versus incorrect predictions.

D Analysis of VSI-Bench Dataset Bias

The analysis shows that VSI-Bench can be substantially improved using averaged dataset statistics, revealing non-visual shortcuts and susceptibility to dataset-specific bias. Cross-dataset results further indicate that 3D-VQA fine-tuning can overfit VSI-Bench while harming performance elsewhere.

  • Bias Hacking Experiment: VSI-Bench contains exploitable object- and room-size statistics, including average sofa size 181.30, bed size 216.06, and room size 20.5 square meters.These averages were extracted from VSI-Bench question-answer pairs and supplied as textual priors to baseline VLMs.
  • Bias Hacking Experiment: +0.47: LLaVA-NeXT-Video-7B’s Object Abs Distance score rises from 0.14 to 0.61 when given the textual prior.The corresponding LLaVA-NeXT-Video-72B score increases from 0.23 to 0.57 (+0.34).
  • Bias Hacking Experiment: 0.61 and 0.57: both baseline models with the textual prior outperform SFT-trained VLM-3R at 0.49 on this task.This demonstrates that averaged dataset statistics can solve a substantial portion of the benchmark without complex visual spatial reasoning.
  • Bias Hacking Experiment: Statistical regularities in VSI-Bench enable high performance even without visual input, validating the benchmark’s vulnerability to non-visual shortcuts.The experiment explicitly exploited statistical priors, mirroring the reported “Test-set Stress-Test” findings.
  • Generalization Analysis of 3D-VQA Models: +14.2% and +12.7%: SpaceR-7B and VILASR-7B improve on VSI-Bench, but performance drops by as much as -7.7% on other spatial benchmarks.The cross-dataset pattern indicates task-specific overfitting and memorization of dataset-specific distributions rather than robust generalized spatial reasoning.

E Analysis of Gradient Backpropagation

Geometric correspondence and depth losses backpropagate through the correspondence head into transformer Query-Key projectors, reshaping attention around 3D structure. Although the head is discarded at inference, the backbone retains spatial correspondence, view invariance, and depth awareness.

  • Gradient flow: The geometric losses Lcorr and Ldepth backpropagate through the correspondence head to update the transformer layers’ Query and Key projectors.The total objective is Ltotal = LLM + λcLcorr + λdLdepth.
  • Geometric gradient structure: Correspondence gradients are spatially localized, teaching attention to distinguish tokens by 3D position rather than semantic category alone.The correspondence contrastive gradient pulls matched embeddings together and pushes negative samples apart, creating view-invariance.
  • Impact on Query-Key similarity: Lcorr increases Query-Key similarity for spatially corresponding tokens and decreases it for geometrically distinct but semantically similar tokens.Accumulated updates make the projector product encode geometric correspondence alongside semantic similarity.
  • Depth-aware attention: Depth gradients suppress geometrically invalid matches and modulate attention to respect 3D boundaries, especially for midconfidence predictions.The softmax factor Aij(1 − Aij) amplifies gradients when Aij ≈ 0.5.
  • QK enhancement mechanism: After training, the discarded correspondence head leaves geometric priors baked into the backbone, yielding spatial correspondence, view invariance, and depth awareness.The resulting standard VLM performs spatial reasoning without auxiliary inputs because attention itself has been geometrically restructured.

F Relation to Positional Embeddings

RoPE and 3D-RoPE provide static coordinate information but do not encode visual correspondence or object permanence, and advanced-RoPE baselines achieve near-zero correspondence accuracy. GASP complements RoPE by supervising content-aware interactions and internalizing 3D consistency into the LLM’s 2D weights.

  • Limitations of Positional Embeddings: RoPE and 3D-RoPE encode static grid coordinates, but not visual correspondence or object permanence.3D-RoPE extends positional encoding with temporal or depth dimensions.
  • Limitations of Positional Embeddings: Near-zero correspondence accuracy in Qwen2.5-VL and LLaVA-NeXT shows advanced RoPE is insufficient for cross-frame object identity.RoPE cannot establish that objects at different coordinates across frames are the same entity.
  • GASP: From Coordinates to Correspondence: GASP operates on the model’s QK^T interaction mechanism rather than providing only an input-level positional signal.This distinguishes GASP from RoPE’s coordinate injection.
  • GASP: From Coordinates to Correspondence: GASP’s contrastive loss makes visual features content-aware, matching an object’s query representation to its key representation across views despite differing positional encodings.RoPE is content-agnostic, whereas GASP distinguishes visual content such as blank walls and complex faces.
  • GASP: From Coordinates to Correspondence: GASP internalizes 3D consistency into the LLM’s 2D weights, learning to respect occlusion boundaries from 2D RGB inputs during inference.Unlike 3D-RoPE approaches, this does not require explicit depth maps or point clouds at inference.
Loading 2605.30231v1…