PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation

Seonho Lee, Jiho Choi, Inha Kang, Jiwook Kim, Junsung Park, Hyunjung Shim

arXiv:2506.09883v2cs.CVcs.AI

TL;DR

Vision-language models remain limited in grounded 3D spatial reasoning, despite their performance on visual and linguistic tasks. Geometric Distillation transfers geometric cues from pretrained 3D foundation models into VLMs without explicit 3D annotations or architectural changes. The authors report consistent improvements over prior approaches across spatial reasoning tasks, with greater scalability and generalization to real-world scenes.

  • Problem

    VLMs remain limited in grounded spatial reasoning, including depth ordering, occlusion, and object layout, despite their performance on diverse multimodal tasks.

    Source: §1 · p. 1

  • Method

    Geometric Distillation fine-tunes VLMs using sparse correspondences, relative depth relations, and dense cost volumes from pretrained 3D foundation models, without explicit 3D annotations.

    Source: §1 · p. 2

  • Results

    The method consistently outperforms prior approaches across spatial reasoning tasks, including semantic correspondence, depth estimation, and 3D visual question answering.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The authors present geometric distillation as a scalable approach for bridging pretrained 2D VLMs with 3D spatial awareness while retaining natural image-text compatibility.

    Source: §5 · p. 8

    The method assumes access to multi-view imagery during training, relies on 3D foundation models whose biases limit control of distilled signals, and does not directly generalize to point clouds or meshes.

    Source: §6 · p. 9

Abstract

from arXiv · show

Vision-Language Models (VLMs) have shown remarkable performance on diverse visual and linguistic tasks, yet they remain fundamentally limited in their understanding of 3D spatial structures. We propose Geometric Distillation, a lightweight, annotation-free fine-tuning framework that injects human-inspired geometric cues into pretrained VLMs without modifying their architecture. By distilling (1) sparse correspondences, (2) relative depth relations, and (3) dense cost volumes from off-the-shelf 3D foundation models (e.g., MASt3R, VGGT), our method shapes representations to be geometry-aware while remaining compatible with natural image-text inputs. Through extensive evaluations on 3D vision-language reasoning and 3D perception benchmarks, our method consistently outperforms prior approaches, achieving improved 3D spatial reasoning with significantly lower computational cost. Our work demonstrates a scalable and efficient path to bridge 2D-trained VLMs with 3D understanding, opening up wider use in spatially grounded multimodal tasks.

1 Introduction

Vision-language models struggle with grounded 3D spatial reasoning, while existing approaches can require costly reconstruction, object-centric data, or extensive synthetic training. Geometric Distillation uses human-inspired geometric cues from pretrained 3D models to improve spatial understanding without explicit 3D annotations.

1 Introduction

  • VLMs struggle with grounded depth ordering, occlusion, and object layout because they rely on 2D projections without depth cues and multiview supervision.These shortcomings hinder applications including navigation, scene understanding, and robotic planning.Source: §1 · p. 1
  • FiT3D incurs high computational cost, MEF relies on 3D object-centric datasets, and SpatialVLM requires extensive synthetic data and task-specific tuning.These drawbacks motivate more efficient and generalizable ways to provide VLMs with 3D awareness.Source: §1 · p. 2
  • Geometric Distillation transfers cues from pretrained 3D foundation models to VLMs, using sparse correspondences, relative depth relations, and dense cost volumes.Sparse, semantically meaningful locations act as geometric anchors across views, while relative depth uses ordinal comparisons within and across views and cost volumes capture soft correspondences.Source: §1 · p. 2
  • The approach draws on human perception of depth and structure from sparse relational cues and on language’s use of relative spatial terms.Examples of perceptual cues include occlusion, relative size, and perspective.Source: §1 · p. 2
  • The method outperforms strong baselines on PF-PASCAL, TAP-Vid, and ScanQA, illustrating its effectiveness and scalability.It improves spatial-relation inference, such as object proximity, without explicit 3D labels or costly reconstruction.Source: §1 · pp. 2–3

2 Related Work

Prior work brings 3D information into visual and vision-language models through reconstruction, cross-view consistency, synthetic spatial QA, and specialized representations. This paper positions geometric distillation as a way to inject 3D foundation-model priors into pretrained 2D VLMs using multi-view images.

2 Related Work

  • FiT3D aligns 2D features with rendered 3D views, MEF maximizes cross-view feature equivariance, and SpatialVLM trains on large synthetic spatial QA corpora.The cited approaches respectively incur reconstruction-related compute and possible semantic loss, require explicit 3D annotations without direct depth supervision, or depend on massive synthetic datasets.Source: §2 · p. 3
  • CroCo, DUSt3R, MASt3R, and VGGT provide multiview-consistent features or estimates of scene geometry, including point maps, correspondences, camera poses, depth maps, and point clouds.The paper uses these 3D foundation models as sources of geometric priors for VLMs.Source: §2 · p. 3
  • Vision-language pretraining alone may not capture comprehensive 3D concepts, while prior efforts add representations or training strategies such as prompt tuning and point-cloud alignment.The paper instead injects 3D knowledge into 2D VLMs using multi-view images, without explicit supervision from other 3D data modalities.Source: §2 · p. 3
  • The architecture uses three loss branches to distill sparse correspondences, depth maps, and dense cost volumes from a 3D foundation model into a frozen CLIP image encoder with a LoRA adapter.The caption states this provides 3D spatial awareness without explicit 3D annotations.Source: Figure 3 · p. 4

3 Proposed Method

Geometric Distillation transfers geometric cues from pretrained 3D foundation models into a VLM using paired views, without ground-truth 3D annotations.

3 Proposed Method

  • Geometric Distillation transfers sparse correspondences, ordinal depth relations, and viewpoint-induced disparities from paired views into a pretrained VLM without ground-truth 3D annotations.The framework uses geometric signals from high-performance 3D foundation models such as MASt3R and VGGT.Source: §3 · p. 4

3.1 Sparse Correspondences

  • A geometric teacher supplies sparse cross-view matches, and a bidirectional SmoothAP objective encourages higher similarity for true matches than non-matches.The matching objective promotes feature-level alignment within a spatial neighborhood.Source: §3.1 · pp. 4–5

3.2 Relative Depth Understanding

  • Teacher depth maps supervise intra-view ordinal rankings and bounded inter-view depth differences over correspondences, accounting for scale variation across views.The method combines local and cross-view depth supervision.Source: §3.2 · p. 5

3.3 Dense Cost Volume Alignment

  • Dense alignment matches VLM cross-view patch-similarity distributions to geometric-teacher distributions using temperature-scaled softmax and Jensen-Shannon divergence.Figure 4 shows the distilled model’s localized geometric similarity more closely aligning with MASt3R than vanilla CLIP.Source: Figure 4 · p. 5, §3.3 · p. 6

3.4 Overall Objective

  • The overall objective combines the proposed loss components into one jointly trained objective, with hyperparameters balancing the loss terms.Source: §3.4 · p. 6

4 Experiments

Experiments evaluate Geometric Distillation on correspondence, dense scene understanding, and 3D visual question answering, using CLIP fine-tuning and linear probing. The method improves results across these tasks, while ablations assess the depth and cost losses.

4.1 Experimental Setups

  • The experiments fine-tune ViT-based CLIP with LoRA on Objaverse or ScanNet++, evaluating three correspondence benchmarks, dense scene tasks, and SQA3D and ScanQA.MASt3R is the primary pretrained geometric teacher; appendix experiments also use VGGT.Source: §4.1 · pp. 6–7

4.2 Experimental Results 4.2.1 3D Visual Understanding

  • On ScanNet++ training, gains are +11.87% PCK@0.05 and +12.36% average Jaccard index; the method also outperforms MEF without 3D annotations.Figure 5 qualitatively shows more accurate, spatially consistent semantic transfers than MEF.Source: §4.2.1 · p. 7, Figure 5 · p. 7
  • On ScanNet++, depth RMSE falls from 0.432 to 0.367; semantic segmentation is competitive, with up to 54 times less computation than FiT3D on one GPU.FiT3D requires about three days of training on four NVIDIA A6000 GPUs.Source: §4.2.1 · p. 7, Table 3 · p. 8

4.2.2 3D Vision-Language Understanding

  • On SQA3D and ScanQA, the method outperforms CLIP and MEF across metrics and datasets, reaching 48.6% EM-1 on SQA3D and raising ScanQA EM-1 from 19.6% to 20.7%.All baselines were fine-tuned on Objaverse for this comparison.Source: §4.2.2 · p. 8, Table 4 · p. 8

4.3 Ablation Study

  • Adding Ldepth to Lmatch improves all correspondence metrics; Lcost further raises PCK@0.05 by +4.69% and video-tracking position accuracy by +4.07%.Pose accuracy dips at some thresholds, but gains +2.18% at the challenging 1cm-1deg threshold.Source: §4.3 · p. 8, Table 5 · p. 8

5 Conclusion

Geometric Distillation transfers geometric signals from pretrained 3D foundation models into pretrained VLMs without architectural changes or explicit 3D annotations. The paper reports improvements across spatial reasoning tasks and frames the approach as a path from 2D vision-language understanding toward 3D perception.

5 Conclusion

  • Geometric Distillation uses multiview correspondences, relative depth relations, and dense cost volumes from models such as MASt3R and VGGT to improve pretrained 2D VLMs.The framework requires neither architectural modifications nor explicit 3D annotations.Source: §5 · p. 8
  • The method improves results across semantic correspondence, depth estimation, and 3D visual question answering, with reported scalability and generalization to real-world scenes.The conclusion characterizes it as a pathway bridging 2D vision-language understanding and 3D perception.Source: §5 · p. 8

6 Limitations & Future Work

The authors identify limits in training data access, teacher supervision, and modality coverage, and describe possible risks from transferred bias or inaccurate geometric signals. They report using existing public resources and describe planned artifact documentation and release.

6 Limitations & Future Work

  • Training assumes access to multi-view imagery; teacher-model supervision may introduce bias and limits control over geometric signals, and the framework does not directly generalize to point clouds or meshes.Future work targets monocular settings and self-supervised alternatives to reduce reliance on external teachers.Source: §6 · p. 9

A Potential Risks

  • Teacher-data biases may transfer to the VLM, while inaccurate depth maps or correspondences could produce incorrect spatial reasoning or degraded performance.The authors also note potential misuse in surveillance, military, or other ethically sensitive scenarios.Source: A Potential Risks · p. 13

B Use or Create Scientific Artifacts

  • The study introduces no new datasets and instead uses public pretrained models and benchmarks, including MASt3R and VGGT as geometric teachers.The artifact discussion says datasets and models are used under their respective license terms for non-commercial research.Source: B Use or Create Scientific Artifacts · p. 13, §B.1 · p. 13

B.1 Discuss The License for Artifacts

  • VGGT is distributed under CC BY-NC 4.0; MASt3R and DUSt3R are licensed under CC BY-NC-SA 4.0.The cited datasets also have specified licenses and are used for non-commercial research.Source: §B.1 · p. 13

B.2 Documentation of Artifacts

  • The authors state that code, checkpoints, and evaluation scripts will be publicly released with documentation and reproduction instructions.For datasets that cannot be redistributed, the planned release will provide source links and download scripts.Source: §B.2 · p. 13

B.3 Statistics for Dataset

The appendix describes the benchmark datasets and splits used across correspondence, dense scene understanding, and 3D question answering. It also reports the CLIP backbone, LoRA setup, and the method’s training-time comparison with FiT3D.

B.3 Statistics for Dataset

  • The benchmarks cover PF-PASCAL, TAP-Vid-DAVIS, and OnePose-LowTexture for correspondence; ScanNet++ and KITTI for depth; ADE20K and VOC2012 for segmentation; and SQA3D and ScanQA for 3D VQA.The appendix provides dataset statistics and split details for these downstream tasks.Source: §B.3 · pp. 13–14, Table 6 · p. 14

C Computational Experiments

  • The computational experiments describe model scale, training and hyperparameter choices, reported metrics, and implementation packages.The section frames these choices as supporting performance and computational efficiency.Source: C Computational Experiments · p. 14

C.1 Model Size and Budget

  • Geometric distillation takes approximately 1 hour and 20 minutes per model on a single NVIDIA A6000 GPU, versus FiT3D’s up to three days on four NVIDIA A6000 GPUs.The vision-language backbone contains approximately 93 million parameters; LoRA uses roughly 6 million parameters, about 6.5% of the total.Source: §C.1 · p. 14

C.2 Experimental Setup and Hyperparameters

The appendix specifies a fixed LoRA-based training and evaluation setup and reports smoother, more coherent cross-view features after geometric distillation.

C.2 Experimental Setup and Hyperparameters

  • Training uses AdamW at 1 × 10−5 for up to 500 epochs, rank-4 LoRA adapters, equally weighted losses, and cost-loss temperature annealing from 1.0 to 0.5.Hyperparameters were held fixed across datasets after empirical tuning on the ScanNet++ validation split; extensive search was not performed.Source: §C.2 · pp. 14–15
  • ScanNet++ distillation samples 10,000 views from 100 scenes, then selects 100 random pairs sharing overlapping 3D regions.The resulting dataset size matches the Objaverse view pairs used in MEF.Source: §C.2 · p. 15

C.3 Descriptive Statistics

  • Results are full-test-set means, but evaluations are deterministic single runs without reported error bars or variances unless otherwise specified.The stated metrics include PCK, Jaccard index, positional accuracy, RMSE, relative error, mIoU, and mAcc.Source: §C.3 · p. 15

C.4 Parameters for Packages

  • The pipeline uses PyTorch, Transformers, and PEFT for CLIP and LoRA, with vision and NLP evaluation tools used at default settings unless specified.External evaluation functions were not additionally tuned or modified.Source: §C.4 · p. 15

D.1 Information About Use Of AI Assistants

  • The authors report using ChatGPT-4o for grammar and style edits, while technical content, experiment design, and conceptual development were performed by the authors.They report no AI-generated content for core research contributions or evaluations.Source: §D.1 · p. 17

E.1 Feature Visualization

  • The method’s feature maps are smoother and more coherent than CLIP, FiT3D, and MEF while preserving semantic correspondences across viewpoints.The visualization uses PCA projections of features from randomly sampled Objaverse objects.Source: Figure 6 · p. 15, §E.1 · p. 17

E.2 More Qualitative Results

Qualitative TAP-Vid-DAVIS examples compare predicted object trajectories with ground truth and MEF.

E.2 More Qualitative Results

  • Our method tracks object parts more accurately and coherently than MEF, with trajectories closer to ground truth.In the car example, MEF confuses the rear wheel with the front wheel, while our method distinguishes and consistently tracks the parts.Source: §E.2 · p. 17

E.3 Example Results of 3D VQA

The SQA3D examples visualize lifted and clustered features to compare vanilla CLIP with geometrically distilled CLIP on spatial questions.

E.3 Example Results of 3D VQA

  • Distilled features show clearer spatial coherence than vanilla CLIP and correctly identify relative object distances in challenging SQA3D questions.Examples include questions about which object is farther or closer, where vanilla CLIP can misidentify spatial relationships.Source: §E.3 · p. 17

F.1 Comparison of Absolute and Relative Depth Understanding

On ScanNet++ correspondence tasks, relative depth loss outperforms absolute depth loss across the reported metrics and is associated with more generalizable geometric representations.

F.1 Comparison of Absolute and Relative Depth Understanding

  • The ScanNet++ comparison fine-tunes models with either log-scale absolute-depth regression or the proposed relative-depth loss, then evaluates 3D correspondence tasks.The absolute loss directly predicts depth values.Source: §F.1 · p. 17
  • 28.48% vs 27.04% PCK@0.05 (different views) and 42.16% vs 37.45% PCK@0.05 (same views) favor relative over absolute depth loss, which also improves tracking and pose accuracy.The reported increases are from 39.27% to 40.09% average Jaccard index and from 9.46% to 10.96% pose accuracy at 1cm–1deg.Source: §F.1 · p. 17, Table 7 · p. 18
  • The authors report that relative depth modeling yields more generalizable geometric representations and reduces the risk of overfitting to the training dataset’s depth distribution.Source: §F.1 · p. 18

F.2 Ablation on Loss Components with Different Training Dataset

The ScanNet++ ablation evaluates how relative-depth and cost-volume losses affect 3D correspondence tasks, and the teacher comparison favors MASt3R over VGGT. The method’s geometric guidance remains limited when input views have little or no shared 3D content.

F.2 Ablation on Loss Components with Different Training Dataset

  • On ScanNet++, Ldepth raises different-view PCK@0.10 from 41.76% to 43.43%; adding Lcost lifts different-view PCK@0.05 from 26.32% to 28.48% and pose accuracy to 10.96% at 1cm–1deg.The ablation also reports same-view PCK@0.05 rising from 37.45% to 42.16%, and average Jaccard index from 37.78% to 40.09%.Source: §F.2 · pp. 18–19, Table 8 · p. 19

F.3 Comparison of MASt3R and VGGT as a Teacher Model

  • On ScanNet++, MASt3R exceeds VGGT on different-view PCK@0.05 (28.48% vs. 24.22%), same-view PCK@0.05 (42.16% vs. 30.79%), video tracking (40.09% vs. 37.28%), and pose accuracy (10.96% vs. 8.15%).The authors attribute the difference partly to VGGT’s anchor-view selection, whereas MASt3R directly predicts dense, consistent semantic correspondences.Source: Table 9 · p. 19, §F.3 · p. 19

G Failure Cases

  • When input views have minimal or no overlapping 3D regions, foundation models may infer geometry inaccurately, passing erroneous guidance into distilled features and degrading downstream performance.The authors suggest overlap-aware sampling and stronger single-image 3D inference as possible mitigations.Source: G Failure Cases · pp. 19–20
Loading 2506.09883v2…