Source-linked AI summary
G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
Wenbo Hu, Jingli Lin, Yilin Long, Yunlong Ran, Lihan Jiang, Yifan Wang, Chenming Zhu, Runsen Xu, Tai Wang, Jiangmiao Pang
TL;DR
VLMs lack robust spatial intelligence because they do not explicitly learn geometry that reconstructs 3D space from 2D inputs. G2VLM unifies geometric reconstruction and multimodal reasoning through learned 3D features, achieving competitive 3D reconstruction and better-or-comparable spatial reasoning, while remaining limited by training instability at large scale.
Problem
VLMs show a significant gap in robust spatial understanding because standard training relies mainly on language and 2D visual priors rather than explicit visual geometry learning.
Method
G2VLM integrates geometric and semantic perception experts to predict 3D attributes and enhance spatial reasoning through in-context learning, interleaved reasoning, and shared attention.
Results
G2VLM is competitive with state-of-the-art feed-forward 3D reconstruction models and performs better than or comparably to larger models across spatial reasoning benchmarks, including an 18.5-point advantage over GPT-4o on SPAR-Bench.
Takeaways & Limitations
The unified model provides a strong baseline spanning low-level visual geometry and high-level spatial reasoning, with potential applications including 3D scene editing.
Takeaways & Limitations
Training instability with large-scale models requires advanced optimization techniques, careful data curation, and significant computational resources.
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) still lack robustness in spatial intelligence, demonstrating poor performance on spatial understanding and reasoning tasks. We attribute this gap to the absence of a visual geometry learning process capable of reconstructing 3D space from 2D images. We present G$^2$VLM, a geometry grounded vision-language model that bridges two fundamental aspects of spatial intelligence: spatial 3D reconstruction and spatial understanding. G$^2$VLM natively leverages learned 3D visual geometry features to directly predict 3D attributes and enhance spatial reasoning tasks via in-context learning and interleaved reasoning. Our unified design is highly scalable for spatial understanding: it trains on abundant multi-view image and video data, while simultaneously leveraging the benefits of 3D visual priors that are typically only derived from hard-to-collect annotations. Experimental results demonstrate G$^2$VLM is proficient in both tasks, achieving comparable results to state-of-the-art feed-forward 3D reconstruction models and achieving better or competitive results across spatial understanding and reasoning tasks. By unifying a semantically strong VLM with low-level 3D vision tasks, we hope G$^2$VLM can serve as a strong baseline for the community and unlock more future applications, such as 3D scene editing.
1. Introduction
G2VLM addresses weak spatial understanding in VLMs by integrating visual geometry learning with multimodal reasoning in a unified architecture. It uses complementary geometric and semantic experts and achieves strong results across 3D reconstruction and spatial reasoning tasks.
- Current VLMs often treat multiple images and video frames as flat 2D sequences, lacking explicit visual geometry learning for coherent 3D representations.
- G2VLM unifies spatial 3D reconstruction and high-level spatial understanding within one vision-language model.
- The model uses dedicated geometric “where” and semantic “what” experts whose shared self-attention links visual geometry learning with multimodal understanding.
- G2VLM learns 3D geometry from pure 2D inputs, reducing reliance on difficult-to-collect depth maps and camera poses while using abundant multi-view images and videos.
- A two-stage training strategy first learns geometry-rich representations, then jointly trains both experts on spatial understanding data.
- G2VLM achieves competitive feed-forward 3D reconstruction results and the best SPAR-Bench performance, surpassing GPT-4o by 18.5 points.On Sintel monocular depth estimation, its Absolute Relative Error is 0.297 versus VGGT’s 0.335.
2. Related Works
Related work spans unified vision-language models, spatial reasoning VLMs, and feed-forward visual geometry, while G$^2$VLM combines geometry learning with multimodal reasoning.
- VLMs as Unified Foundation Models: G$^2$VLM differs from unified foundation-model designs by requiring distinct architectures, objectives, and joint-training strategies for geometry learning and spatial reasoning.
- Spatial Reasoning VLMs: Spatial reasoning is central to spatial intelligence for world understanding, robotics, and embodied AI, motivating specialized spatial VLMs and benchmarks.
- Spatial Reasoning VLMs: Many spatial VLMs retain standard designs that treat images as flat 2D data, limiting their geometric grounding.
- Feedforward Visual Geometry: Feed-forward visual geometry methods directly predict pixel-aligned 3D point maps without known camera parameters and extend efficiently to multi-view inputs.
- Feedforward Visual Geometry: Although these geometry methods achieve accurate reconstruction and surpass optimization-based pipelines, they often neglect higher-level scene understanding needed for spatial reasoning.
3. Unified Spatial Vision-Language Model
G2VLM unifies visual geometry learning and multimodal understanding through specialized transformer experts and a staged training process. Its geometry objectives reconstruct 3D points, camera poses, and surface normals, while joint training incorporates these representations into spatial reasoning.
- Model Architecture: G2VLM uses two transformer experts: a geometry perception expert for visual geometry learning and a semantic perception expert for multimodal understanding.The experts are designed to interact through shared visual and language representations.
- Model Architecture: The geometry expert maps RGB images to hidden states decoded by lightweight heads for pixel-aligned 3D points and camera poses.The architecture uses a DINOv2 vision encoder and includes local point, camera, and global point heads.
- Implementation Details: The pretrained semantic expert is based on Qwen2-VL and can leverage off-the-shelf pretrained VLMs, while the architecture removes camera tokens and uses permutation-equivariant design for scalability.The joint-training implementation uses spatial-understanding and general VQA datasets, including SPAR-7M, Omnispatial, Mindcube, OST-Bench, and LLaVA-One-Vision.
- Visual Geometry Learning: The visual geometry objective combines point reconstruction, camera pose, and normal losses, with weighted terms controlled by hyperparameters.Point reconstruction uses an optimal scale factor, camera supervision combines rotation and translation terms, and normal supervision encourages locally smooth surfaces.
- Spatial Reasoning Learning: Joint training optimizes the semantic expert with cross-entropy so it can use learned geometric representations through in-context learning and interleaved reasoning.The paper compares three supervision strategies for this stage; the VG + CE Loss approach yields the best overall performance in Figure 4.
4. Experiments
G2VLM is evaluated on visual geometry, spatial understanding, and reasoning tasks, with ablations examining encoder and attention choices. It performs competitively on 3D reconstruction and strongly across spatial benchmarks, while dual encoders, global attention, and geometry pretraining improve results.
- Spatial Understanding & Reasoning Results: G2VLM-SR achieves the best results among existing works on four spatial benchmarks and surpasses GPT-4o by 18.48 points on SPAR-Bench.
- Spatial Understanding & Reasoning Results: G2VLM-SR leads open-source models on SPAR-Bench, MindCube, and OmniSpatial, while Qwen2.5-VL-72B leads OST-Bench.The comparison suggests online spatio-temporal scene understanding benefits from larger architectures, and the authors leave scaling for future work.
- Encoder Design: The dual-encoder design using DINO for geometry and CLIP for multimodal understanding performs best on both visual geometry and spatial understanding tasks.The comparison is shown in Figure 6(a).
- Attention Mechanism: Global attention consistently outperforms frame and mixed attention for geometric perception and also produces the strongest spatial reasoning results.Table 2 reports the corresponding positive interplay between geometric and semantic representations.
- Impact of Geometry Pretraining: Geometry pretraining is essential: the full model significantly outperforms a variant fine-tuned only on spatial-understanding data.
5. Conclusion
G2VLM unifies 3D reconstruction and spatial understanding by using learned 3D visual geometry to support spatial reasoning. The authors report strong generalization while identifying training instability with large-scale models as a limitation.
- 5. Conclusion: G2VLM bridges 3D reconstruction and spatial understanding by directly predicting 3D attributes and using learned geometry for spatial reasoning.It supports in-context learning and interleaved reasoning across both aspects of spatial intelligence.
- 5. Conclusion: Training instability with large-scale models remains a limitation requiring advanced optimization, careful data curation, and substantial computational resources.
A. Architecture Details
The architecture uses lightweight transformer-based geometry heads to decode camera poses and point maps from geometry features. Compared with VGGT, it avoids a computationally intensive DPT head and uses task-specific output heads.
- A. Architecture Details: Lightweight transformer-based geometry heads replace VGGT’s computationally intensive DPT head for camera poses and point-map prediction.The decoders share a transformer architecture but not weights, and apply self-attention independently to each image’s features.
- A. Architecture Details: Task-specific heads decode local point maps with an MLP and pixel shuffle, while camera poses use MLPs, pooling, and SVD-based rotation orthogonalization.The global point head stabilizes training but is excluded during inference.
B. Training Details
Training proceeds through staged geometric pretraining and joint training, using different resolutions, learning rates, and compute budgets. Loss clipping and gradient norm clipping are used to improve stability under noisy annotations and large-scale optimization.
- B. Training Details: Geometric pretraining runs at 224x224 for 100K iterations, then 518x518 for 20K steps with randomized aspect ratios.The stages use learning rates of 2e-4 and 5e-4, respectively, with randomly sampled batches containing 2–24 frames.
- B. Training Details: The low-resolution stage uses 32 A800 GPUs for 7 days, while the high-resolution stage uses 64 A800 GPUs for 3 days.
- B. Training Details: Visual geometry losses above 10 are clipped and smoothed to 0 because noisy 3D annotations produce loss spikes.The authors note that further data cleaning could reduce these instabilities.
- B. Training Details: Joint training uses AdamW for 16K iterations at a learning rate of 2e-5 on 64 A800 GPUs over 3 days.Gradient norm clipping at 1.0, bfloat16 precision, and gradient checkpointing support stability and computational efficiency.
C. More results
On SPAR-Bench, G2VLM-SR achieves the best performance consistently across the reported subtasks. The results also indicate that it surpasses human performance in the low category.
- C. More results: G2VLM-SR demonstrates the best performance consistently across all SPAR-Bench tasks.Table 3 reports comparisons across the benchmark’s subtasks, with scores visually marked by rank.
- C. More results: G2VLM-SR surpasses human performance in the low category of SPAR-Bench.