Source-linked AI summary
IGGT: Instance-Grounded Geometry Transformer for Semantic 3D Reconstruction
Hao Li, Zhengyu Zou, Fangfu Liu, Xuanyang Zhang, Fangzhou Hong, Yukang Cao, Yushi Lan, Manyuan Zhang, Gang Yu, Dingwen Zhang, Ziwei Liu
TL;DR
Existing methods often separate geometric reconstruction from semantic understanding or tightly align representations with one language model, limiting joint scene reasoning and adaptability. IGGT uses a unified transformer with 3D-consistent contrastive learning and instance-grounded masks, reports superior task performance and 3D coherence, and supports plug-and-play integration with diverse VLMs and LMMs.
Problem
Prior methods decouple 3D reconstruction from semantic understanding or align spatial models to specific VLMs, limiting mutual context, geometric detail, model adaptability, and instance discrimination.
Method
IGGT jointly learns geometric and instance-level semantic representations with a unified transformer and 3D-consistent contrastive learning, then uses instance masks to connect diverse VLMs and LMMs.
Results
IGGT outperforms state-of-the-art methods in downstream task performance and 3D scene coherence across spatial tracking, open-vocabulary segmentation, and scene grounding.
Takeaways & Limitations
The framework unifies spatial reconstruction and contextual understanding while enabling plug-and-play use of different vision-language and large multimodal models.
Takeaways & Limitations
The model is trained on self-annotated open-source images and evaluated in controlled settings, with potential for bias or unexpected behavior.
Abstract
from arXiv · showhide
Humans naturally perceive the geometric structure and semantic content of a 3D world as intertwined dimensions, enabling coherent and accurate understanding of complex scenes. However, most prior approaches prioritize training large geometry models for low-level 3D reconstruction and treat high-level spatial understanding in isolation, overlooking the crucial interplay between these two fundamental aspects of 3D-scene analysis, thereby limiting generalization and leading to poor performance in downstream 3D understanding tasks. Recent attempts have mitigated this issue by simply aligning 3D models with specific language models, thus restricting perception to the aligned model's capacity and limiting adaptability to downstream tasks. In this paper, we propose InstanceGrounded Geometry Transformer (IGGT), an end-to-end large unified transformer to unify the knowledge for both spatial reconstruction and instance-level contextual understanding. Specifically, we design a 3D-Consistent Contrastive Learning strategy that guides IGGT to encode a unified representation with geometric structures and instance-grounded clustering through only 2D visual inputs. This representation supports consistent lifting of 2D visual inputs into a coherent 3D scene with explicitly distinct object instances. To facilitate this task, we further construct InsScene-15K, a large-scale dataset with high-quality RGB images, poses, depth maps, and 3D-consistent instance-level mask annotations with a novel data curation pipeline.
1 INTRODUCTION
IGGT addresses fragmented 3D reconstruction and semantic understanding by jointly learning geometric and instance-level representations, then connecting them to diverse vision-language models for downstream scene understanding.
- Prior pipelines separate 3D reconstruction from semantic understanding, propagating errors and failing to exploit their mutual context.
- Aligning spatial models with specific VLMs can oversmooth geometric detail, restrict model integration, and fail to distinguish objects sharing a semantic category.
- IGGT jointly trains a unified transformer to encode 3D geometry and instance-level semantics, decoding geometric point maps and instance clustering fields.
- Instance masks connect IGGT with diverse VLMs and extend compatibility to multimodal models for broader scene-understanding applications.
- Experiments on spatial tracking, open-vocabulary segmentation, and scene grounding report higher task performance and 3D scene coherence than state-of-the-art methods.
2 INSSCENE-15K DATASET
InsScene-15K combines multiple scene sources with RGB, depth, pose, and 3D-consistent instance annotations, using source-specific curation procedures to improve mask quality and consistency.
- InsScene-15K provides RGB images, depth maps, poses, and 3D-consistent instance masks with unique instance IDs across views.
- The curation pipeline combines synthetic, video-captured, and RGBD-captured scenes to provide comprehensive and diverse data.
- Figure 3 compares annotations from three sources and, for RGBD scans, contrasts vanilla ScanNet++ masks with refined masks, matched IDs, and mIoU scores.
- For video scenes, SAM2 propagates masks temporally, while iterative keyframes and bidirectional propagation address new objects, drift, and temporal consistency.
- For RGBD scans, SAM2 proposals are aligned with projected coarse ground-truth masks to combine fine shape detail with consistent object identities.
3 METHODOLOGY
IGGT jointly reconstructs 3D geometry and predicts instance-discriminative features from multi-view images through unified tokens, cross-modal heads, and 3D-consistent supervision. Its instance-grounded outputs support multi-view tracking, open-vocabulary segmentation, and language-model-based scene grounding.
- Instance-Grounded Scene Understanding: IGGT jointly predicts geometric structures and instance features from unified multi-view token representations, then clusters the features into consistent instance masks for downstream understanding.The pipeline combines a unified foundation model with unsupervised instance clustering and mask-based vision-language or language-model interaction.
- Large Unified Transformer: A 1B-parameter transformer uses DINOv2 image tokens, camera tokens, and intra-view plus global-view attention to produce globally consistent unified tokens.The architecture captures local and global context while supporting arbitrary multi-view inputs and permutation equivariance.
- Downstream Heads and Cross-Modal Fusion Block: Geometry and Instance Heads decode the unified tokens into camera, depth, point, and dense instance features, with DPT-like prediction and cross-modal fusion for spatially aware boundaries.The sliding-window cross-attention block injects spatial structure into instance representations without the quadratic complexity of global attention.
- 3D-Consistent Contrastive Supervision: 3D-consistent contrastive supervision pulls same-instance features together across views and pushes different instances apart using a multi-task loss with pose, depth, and point-map terms.The objective structures instance representations according to scene geometry and is trained alongside geometry supervision.
- Evaluation: The capability overview reports spatial tracking, reconstruction, and 2D/3D open-vocabulary segmentation results on ScanNet and ScanNet++, while the framework is designed to support multiple downstream tasks.The supplied table passages identify the evaluated task categories but do not provide individual metric values.
- Instance-Grounded Scene Understanding: HDBSCAN clusters multi-view instance features into object identities, whose reprojected masks enable dense tracking and segmentation across views, including large camera motion.The resulting instance masks also prompt off-the-shelf VLMs for open-vocabulary segmentation and support object-centric QA through LMM queries.
4 EXPERIMENTS
IGGT is evaluated on instance tracking, open-vocabulary segmentation, and QA scene grounding, with results indicating strong task performance, 3D consistency, and flexible VLM integration.
- Evaluation Setup: The experiments use ScanNet and ScanNet++ scenes with multiple overlapping views selected to preserve spatial coverage and cross-view consistency.Ten scenes per dataset are sampled with 8–10 images per scene.
- Instance Spatial Tracking: IGGT surpasses competing methods on instance spatial tracking, achieving nearly 100% T-SR accuracy and over 60% T-mIoU while baselines remain below 30% T-mIoU.The method maintains object identity under large camera motions, whereas competing approaches lose track.
- Instance Spatial Tracking: IGGT produces 3D-consistent instance features that remain discriminative across views and similar object instances.PCA visualizations show similar-category instances with distinguishable feature colors, supporting tracking and segmentation under large motions.
- Open-Vocabulary Segmentation: On ScanNet++, IGGT leads open-vocabulary segmentation by 8.34% in mIoU and object localization by 7.88% in mAcc.
- QA Scene Grounding: Instance-grounded querying enables accurate segmentation for complex QA prompts and better multi-view consistency than the compared Gemini 2.5 Pro approach.The evaluation uses the Teatime scene from the LERF-OVS dataset.
- Ablation Study: Cross-modal fusion improves instance-head convergence and high-resolution geometric detail, while different VLMs favor different query types.LSeg and OpenSeg perform better on background classes, whereas CLIP performs better on complex categories.
5 CONCLUSION
The paper concludes that IGGT unifies spatial reconstruction with contextual understanding through joint geometric and instance-semantic representations. InsScene-15K and plug-and-play VLM/LMM integration support broader 3D-scene applications, while experiments report strong task performance and 3D coherence.
- 5 CONCLUSION: IGGT jointly trains geometric and instance-level semantic features in a unified representation for spatial reconstruction and contextual understanding.
- 5 CONCLUSION: InsScene-15K provides high-quality RGB images, poses, depth maps, and 3D-consistent instance masks for the proposed framework.
- 5 CONCLUSION: Instance-grounded scene understanding supports plug-and-play integration of diverse VLMs and LMMs, broadening the framework’s applications.
- 5 CONCLUSION: Extensive experiments report superior task performance and 3D coherence compared with recent state-of-the-art methods.
6 ETHICS STATEMENT
The ethics statement limits the work to academic exploration and acknowledges potential biases or unexpected behaviors from systems trained on self-annotated datasets and tested in controlled settings.
- 6 ETHICS STATEMENT: The model is trained on self-annotated datasets and evaluated in controlled settings, with potential biases or unexpected behaviors acknowledged.
- 6 ETHICS STATEMENT: The authors state that the research is intended for academic exploration and support ethical, safe, and socially aligned AI development.
A.1 USE OF LARGE LANGUAGE MODELS
Large Language Models were used only for manuscript grammar correction and stylistic polishing, not for the scientific work.
- A.1 USE OF LARGE LANGUAGE MODELS: Large Language Models contributed only minor grammar corrections and stylistic polishing, excluding methodology, experiments, analysis, and scientific contributions.
A.2 RELATED WORK
Prior work separates 3D reconstruction from semantic understanding or tightly couples them to specific vision-language models. IGGT instead learns a unified representation and uses instance masks to connect diverse VLMs plug-and-play.
- Existing methods often require dense multi-view inputs and per-scene optimization, limiting scalable semantic reasoning over reconstructed 3D scenes.
- IGGT unifies reconstruction and understanding while using instance masks to connect diverse VLMs and LMMs without tight coupling to one model.
A.3 TRAINING DETAILS
IGGT is initialized from VGGT and fine-tuned on InsScene-15K with specified hardware, optimizer, learning rates, and frame sampling. The section also visualizes manually annotated tracking ground truth and tracking predictions.
- IGGT is initialized from VGGT and fine-tuned on 15,000 InsScene-15K scenes using 8 NVIDIA A800 GPUs for 2 days.Training uses AdamW, with learning rates of 1 × 10−6 for the backbone and 1 × 10−5 for the geometry and instance heads.
- Each training batch randomly samples 1–12 frames from a scene, producing 24 images per batch.
- Figure 12 visualizes manually annotated tracking ground truth alongside the model’s tracking results.
A.4 METRICS FOR DIFFERENT TASKS
The evaluation defines temporal tracking metrics, constructs 3D semantic segmentation scores by lifting labeled points into voxels, and visualizes qualitative mask and point comparisons. Refined instance masks improve segmentation and tracking, while clustering remains boundary-limited.
- Instance Spatial Tracking: Temporal mIoU averages per-view mask intersection-over-union for each object across T views, while T-SR measures successful tracking across all views.
- 3D Semantic Segmentation: Figure 14 compares RGB and semantic 3D points for ground truth, IGGT, LSM, and Feature-3DGS.
- 3D Semantic Segmentation: 3D semantic mIoU is computed by labeling aligned RGB 3D points from 2D segmentation, voxelizing them, and comparing voxel representations.
- Mask Refinement: Refined InsScene-15K masks contain fewer unannotated regions and align more closely with objects, supporting more accurate instance segmentation and tracking.
- Limitations: Unsupervised instance-grounded clustering still produces object boundaries below state-of-the-art segmentation quality such as SAM2.