Source-linked AI summary
SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding
Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, Siyuan Huang
TL;DR
Grounded 3D vision-language learning is limited by scarce paired data and the difficulty of representing complex spatial scenes. The paper introduces SceneVerse and GPS, combining million-scale scene-language data with unified multi-level pre-training. GPS achieves state-of-the-art performance across existing 3D-VL grounding benchmarks and demonstrates zero-shot transfer, while the dataset currently covers only object and scene descriptions.
Problem
Grounded 3D vision-language learning lacks sufficient paired scene-language data and struggles to capture spatial information that 2D-VL models miss.
Method
SceneVerse combines diverse indoor scenes, human referrals, and automated scene-graph- and LLM-generated descriptions, while GPS performs unified multi-level scene-language pre-training.
Results
GPS achieves state-of-the-art results on existing 3D-VL grounding and question-answering benchmarks, with zero-shot transfer and scaling benefits beyond GPS.
Takeaways & Limitations
Scaling SceneVerse supports grounded 3D understanding across grounding, question answering, zero-shot transfer, and semantic segmentation settings.
Takeaways & Limitations
SceneVerse currently contains only object and scene descriptions, leaving question-answer pairs and dialogues as future language types.
Abstract
from arXiv · showhide
3D vision-language grounding, which focuses on aligning language with the 3D physical environment, stands as a cornerstone in the development of embodied agents. In comparison to recent advancements in the 2D domain, grounding language in 3D scenes faces several significant challenges: (i) the inherent complexity of 3D scenes due to the diverse object configurations, their rich attributes, and intricate relationships; (ii) the scarcity of paired 3D vision-language data to support grounded learning; and (iii) the absence of a unified learning framework to distill knowledge from grounded 3D data. In this work, we aim to address these three major challenges in 3D vision-language by examining the potential of systematically upscaling 3D vision-language learning in indoor environments. We introduce the first million-scale 3D vision-language dataset, SceneVerse, encompassing about 68K 3D indoor scenes and comprising 2.5M vision-language pairs derived from both human annotations and our scalable scene-graph-based generation approach. We demonstrate that this scaling allows for a unified pre-training framework, Grounded Pre-training for Scenes (GPS), for 3D vision-language learning. Through extensive experiments, we showcase the effectiveness of GPS by achieving state-of-the-art performance on all existing 3D visual grounding benchmarks. The vast potential of SceneVerse and GPS is unveiled through zero-shot transfer experiments in the challenging 3D vision-language tasks. Project website: https://scene-verse.github.io.
1 Introduction
3D vision-language learning remains less developed than 2D learning because 3D data collection and paired scene-language data are scarce. SceneVerse addresses this gap with a million-scale dataset and GPS, whose scaling supports strong 3D-VL performance and generalization.
- 3D grounded understanding remains in an incipient stage despite rapid progress in 2D vision-language learning.
- 3D scene datasets usually contain only thousands of scenes because acquisition is more complex and expensive than collecting 2D images.
- SceneVerse combines 68K 3D scenes with 2.5M scene-language pairs from human annotation and automated generation.
- GPS provides unified scene- and object-level alignment through large-scale pre-training on SceneVerse without auxiliary losses.
- GPS achieves state-of-the-art results on existing 3D-VL grounding and question-answering benchmarks, with zero-shot transfer and broader benefits for semantic segmentation.
2 Related Work
Prior grounded 3D understanding datasets are difficult to build because scene acquisition and annotation are demanding, and most remain limited in scene and language scale. Existing approaches use scene graphs and 2D-VL knowledge, but still lack sufficient paired 3D scene-language data and 3D spatial information.
- Grounded 3D scene datasets require extensive acquisition and annotation, supporting tasks such as referral, captioning, navigation, and reasoning.
- SceneVerse compares against prior 3DVL datasets across visual grounding, question answering, pre-training, and multi-tasking settings using human-annotated and synthetic language.
- Most existing datasets remain constrained in both scene and language scale, motivating larger aligned resources.
- 2D-VL progress has been driven by transformer pre-training, large-scale image-language datasets, and contrastive learning with data scaling.
- 3D grounding models often rely on task-specific architectures, loss designs, or advanced optimization because paired 3D scene-language data are limited.
- Pre-trained 2D-VL models provide object attributes and affordances but miss crucial 3D information such as spatial relationships.
3 SceneVerse
SceneVerse unifies diverse real and synthetic indoor scenes with human referrals and an automated scene-graph- and LLM-based language-generation pipeline. Its descriptions cover object attributes, spatial relations, and global scene content across multiple granularities.
- SceneVerse combines real datasets with synthetic environments to address the scarcity of available 3D scene data.
- Human annotation contributes 96,863 context-rich object referrals, each verified by two additional reviewers.
- 3D Scene Graph Construction: The hierarchical scene graph represents object instances using centroids and bounding-box sizes, with edges encoding spatial relationships.
- Language Generation: SceneVerse language covers object attributes, object-to-object spatial relations, and global scene descriptions.
- Language Generation: Templates and LLMs generate object captions, relationship referrals, and scene captions, while LLM rephrasing increases description diversity and naturalness.
- Data Quality: Human annotations passed reference verification with a 4.8% re-annotation rate, while the automatic pipeline underwent quality checks.
- Statistics: The dataset contains 68,406 room-level scans, 1.5M object instances, 2,290 categories, 21 relationship types, and roughly 2M generated texts.
4 Grounded Pre-training for Scenes
GPS aligns 3D scenes and language at object, referral-object, and scene levels using contrastive objectives, with masked language modeling added during training. Its referral-object alignment uses intra-scene contrast to support fine-grained grounding.
- Multi-level alignment: GPS forms contrastive objectives at object, referral-object, and scene levels to align 3D scenes with language.The framework is described as a transformer-based model using multi-level contrastive losses.
- Object-level grounding: An off-the-shelf 3D segmentation model decomposes each scene point cloud into a bag of objects for object-level grounding.Object point-cloud features are aligned with text features derived from object captions.
- Scene-level grounding: GPS encodes object features with spatial locations, then projects and max-pools them to obtain a scene feature for scene-level alignment.Scene features are aligned with scene-caption features using a scene-level contrastive objective.
- Referral-object-level grounding: A self-attention reasoning transformer models relationships between referral text and scene objects before referral-object contrastive alignment.The model produces aligned referred-object and sentence-level referral features.
- Referral-object-level grounding: Intra-scene contrast selects positives within the same scene, distinguishing referred objects from other objects for fine-grained grounding.Object- and scene-level alignment instead use inter-scene contrast, while referral alignment restricts comparisons to objects in the same scene.
- Training objective: Training combines object, scene, referral, and masked language modeling losses to learn multi-level scene-language alignment.The point-cloud encoder is first trained with object-level grounding, followed by joint scene-grounding objectives and masked language modeling.
5 Experiments
The experiments evaluate GPS and SceneVerse across grounding, zero-shot transfer, question answering, semantic segmentation, scaling, generated-text quality, domain transfer, and alignment ablations. Results show strong benchmark performance, transfer gains, scaling benefits, and sensitivity to multi-level alignment.
- 3D Visual Grounding: GPS trained on SceneVerse achieves state-of-the-art results on all existing 3D-VL grounding benchmarks.Pre-training substantially improves performance over training from scratch, while fine-tuning consistently outperforms existing baselines with only a projection MLP added.
- Zero-Shot Transfer: GPS consistently outperforms 3D-VisTA in zero-shot transfer across established benchmarks and SceneVerse-val.The results associate this generalization with contrastive alignment rather than traditional classification objectives.
- Zero-Shot Transfer: SceneVerse improves zero-shot grounding particularly when training data are limited, including on SceneVerse-val.Automatically generated scene-text pairs provide knowledge about scene distributions and contribute substantially to zero-shot improvement.
- Additional 3D-VL Tasks: SceneVerse pre-training improves existing 3D backbone models on open-vocabulary semantic segmentation and benefits semantic segmentation beyond 3D-VL grounding.The experiments also report consistent performance improvement as the amount of SceneVerse data increases in pre-training and zero-shot transfer settings.
- Generated Data: Generated template-based and LLM-refined texts improve ScanRefer performance without fine-tuning, while adding human annotations yields only a relatively marginal further improvement.The generated-text variants already achieve state-of-the-art results compared with previous baselines.
- Ablative Studies: Removing scene-level, object-level, or masked-language objectives reduces SceneVerse-val performance by about 5%, 2%, or 1.5%, respectively.The ablation results identify scene-level alignment as especially important for referral-object grounding.
6 Conclusion
The work scales 3D vision-language learning for grounded scene understanding through SceneVerse and GPS. Experiments show strong performance across common 3D-VL tasks and improved zero-shot generalization.
- SceneVerse scales 3D-VL with diverse scenes and multi-level descriptions from human annotation and automated scene-text generation.
- GPS uses multi-level scene-language contrastive alignment for grounded 3D-VL learning.
- GPS achieves state-of-the-art results on common 3D-VL tasks, including grounding and question answering.
- SceneVerse-trained GPS improves zero-shot generalization over previous baselines and benefits existing 3D models on semantic segmentation.
A The SceneVerse Dataset
SceneVerse unifies real and synthetic indoor 3D scenes, standardizes their representations, and generates object- and scene-level language through scene graphs, templates, and LLMs.
- Scene curation: SceneVerse combines real-world datasets with synthetic environments to address limited 3D scene data.
- Preprocessing: Room-level segmentation and semantic-label mapping align heterogeneous scenes with existing benchmark settings.
- Preprocessing: 68,406 3D scenes are represented as point clouds containing coordinates, RGB color, instance IDs, and semantic labels.
- Scene graphs: The scene graph captures 21 relationship types organized across four categories and hierarchical support levels.
- Language generation: Scene-language pairs are generated with relationship templates, multi-object and star-reference descriptions, and LLM rephrasing.
- Language generation: Object captions combine rendered multi-view evidence, BLIP2 proposals, CLIP scoring, and LLM refinement.
B.2 Pre-training Details
GPS is pretrained in two stages, first aligning object point clouds with captions and then aligning scene representations with captions and referring expressions.
- Two-stage pre-training: GPS first pretrains the object point-cloud encoder with an object-level grounding objective.
- Two-stage pre-training: The second stage freezes the object encoder and performs scene-level pre-training with grounding and referral-object objectives.
- Scene-level pre-training: Scene-level encoding uses a BERT language encoder, a spatial transformer, and feature aggregation for alignment.
- Optimization: Training uses AdamW, cosine annealing for object pre-training, and separate learning rates for language and spatial components.
C Experimental Details
The experiments evaluate grounding, question answering, zero-shot transfer, and open-vocabulary segmentation under dataset-specific and transfer-oriented settings.
- Grounding: Grounding experiments compare scratch, pre-train, and fine-tuned settings using validation sets and fixed object proposals.
- Zero-shot transfer: Zero-shot transfer uses held-out ScanNet and MultiScan referrals while training on other SceneVerse datasets and text pairs.
- 3D question answering: Question-answering experiments evaluate ScanQA and SQA3D after fine-tuning a question-answering module on GPS representations.
- Open-vocabulary segmentation: Open-vocabulary segmentation follows an annotation-free setup with SparseUNet, CLIP text features, and a vision-language adapter.
D.1 Semantic Segmentation
SceneVerse pre-training improves 3D semantic segmentation, demonstrating that scaling the pre-training data benefits tasks beyond 3D visual grounding. Further fine-tuning on ScanNet yields additional gains.
- Comparison: SceneVerse pre-training improves Swin3D-S semantic segmentation performance by approximately 6%.The comparison evaluates replacing Structured 3D pre-training data with SceneVerse.
- Comparison: SceneVerse pre-training consistently outperforms Structured 3D pre-training for semantic segmentation.The reported comparison supports a scaling effect from the larger SceneVerse pre-training set.
- Comparison: Fine-tuning on ScanNet after SceneVerse pre-training further improves semantic segmentation performance.
D.2 Qualitative Results
The appendix presents qualitative visualizations of GPS grounding predictions and open-vocabulary 3D semantic segmentation. Grounding figures distinguish correct predictions from errors, while segmentation figures distinguish known from unseen classes.
- 3D Visual-Language Grounding: Figure A.5 visualizes GPS results for 3D visual-language grounding, marking incorrect predictions in red and correct predictions or ground truths in green.
- Open-Vocabulary Semantic Segmentation: Figure A.6 visualizes open-vocabulary 3D semantic segmentation, marking ScanNet-20 vocabulary classes in blue and unseen classes in green.