PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounded 3D-LLM with Referent Tokens

Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, Jiangmiao Pang

arXiv:2405.10370v2cs.CV

TL;DR

Existing 3D scene-understanding models are often specialized or require task-specific fine-tuning, leaving unified 3D perception and reasoning underexplored. Grounded 3D-LLM uses referent tokens, phrase-level scene-text alignment, curated grounded data, and instruction-following templates to unify tasks, achieving leading performance and broad applicability across 3D benchmarks.

  • Problem

    Existing 3D scene-understanding models are typically customized for individual tasks or require task-specific fine-tuning, while unified 3D perception and reasoning remains scarce.

  • Method

    Grounded 3D-LLM represents scene objects or regions with referent tokens, applies CLASP phrase-level alignment using curated grounded scene-text data, and converts tasks into instruction-following formats.

  • Results

    Grounded 3D-LLM achieves leading performance and broad applicability across open-ended and close-ended 3D vision tasks, particularly grounding, without task-specific fine-tuning.

  • Takeaways & Limitations

    A single generative 3D multimodal model can connect natural-language phrases with scene regions while supporting diverse 3D vision-language tasks.

Abstract

from arXiv · show

Prior studies on 3D scene understanding have primarily developed specialized models for specific tasks or required task-specific fine-tuning. In this study, we propose Grounded 3D-LLM, which explores the potential of 3D large multi-modal models (3D LMMs) to consolidate various 3D vision tasks within a unified generative framework. The model uses scene referent tokens as special noun phrases to reference 3D scenes, enabling it to handle sequences that interleave 3D and textual data. Per-task instruction-following templates are employed to ensure natural and diversity in translating 3D vision tasks into language formats. To facilitate the use of referent tokens in subsequent language modeling, we provide a large-scale, automatically curated grounded scene-text dataset with over 1 million phrase-to-region correspondences and introduce Contrastive Language-Scene Pre-training (CLASP) to perform phrase-level scene-text alignment using this data. Our comprehensive evaluation covers open-ended tasks like dense captioning and 3D question answering, alongside close-ended tasks such as object detection and language grounding. Experiments across multiple 3D benchmarks reveal the leading performance and the broad applicability of Grounded 3D-LLM. Code and datasets are available at the https://groundedscenellm.github.io/grounded_3d-llm.github.io.

1 INTRODUCTION

Grounded 3D-LLM addresses the scarcity of unified 3D perception and reasoning models by representing scene referents with special tokens and integrating multiple tasks into language modeling. It combines phrase-level scene-text alignment, curated grounded data, and instruction-following conversions to support broad 3D vision-language capabilities.

  • Grounded 3D-LLM targets the open problem of unifying diverse 3D perception and reasoning tasks within one model.
  • Referent tokens represent singular or plural phraseable scene objects and connect noun phrases with external 3D regions.They can substitute for or accompany text phrases while being decoded into corresponding scene objects.
  • CLASP aligns noun phrases with corresponding 3D scene embeddings, while G-SceneCap supplies over 1 million phrase-level correspondences for this training.The dataset contains 107K scene captions and averages one object every nine tokens.
  • The model converts grounding, detection, dense captioning, and 3D question answering into instruction-following formats using diverse task-specific templates.Referent tokens are supervised to decode into specified object embeddings during multi-task instruction fine-tuning.
  • Without specialized models or task-specific fine-tuning, Grounded 3D-LLM achieves top-tier performance among generative models, particularly on grounding problems.The reported contribution emphasizes a single model spanning multiple 3D vision-language problems.

2 RELATED WORKS

Prior 3D scene-understanding research spans foundational learning, detection, visual grounding, and multimodal task models. These systems commonly use detectors as foundations for scene-language tasks, while multimodal models organize capabilities across several 3D outputs.

  • 3D multimodal models: 3D multimodal models are evaluated across instance segmentation, object detection, single- and multi-object grounding, question answering, and dense captioning.
  • 3D scene understanding: Prior 3D scene understanding includes foundational unsupervised learning, closed- and open-set detection, and visual grounding of objects from language.
  • 3D scene understanding: Detection systems typically provide the groundwork for subsequent scene-language tasks, including 3D visual grounding.

3 METHOD

Grounded 3D-LLM aligns noun phrases with 3D objects, then uses referent tokens and task-specific instruction formats to unify diverse 3D vision-language tasks.

  • 3.1 Preliminary: Vision-Language Correspondence: Phrase-level alignment links noun phrases in contextual descriptions to specific 3D objects, addressing the scarcity of grounded 3D scene-text data.The method models phrase-to-region correspondences rather than only sentence-level or word-level alignment.
  • 3.2 Grounded 3D-LLM: CLASP pre-trains a point-cloud encoder and cross-modal interactor on grounded scene-text before language-model instruction fine-tuning.The two-stage pipeline first aligns scene and text representations, then trains the LLM with interleaved referent tokens.
  • 3.2 Grounded 3D-LLM: The cross-modal interactor uses learnable object queries to combine point-cloud embeddings with text descriptions through one-way or bi-directional cross-attention.It produces object and text embeddings used for textual similarity and per-point mask classification.
  • 3.2 Grounded 3D-LLM: Referent tokens act as soft textual placeholders for phraseable objects, allowing language inputs and outputs to reference corresponding scene objects.Output referents are decoded into embeddings that retrieve object queries and associated point-level masks.
  • 3.3 Generating Grounded Scene-Text Data for Pretraining: The automatically generated G-SceneCap data captures noun relationships and global spatial relations, yielding over 1.02 million phrase-level correspondences at one correspondence per nine tokens.The data-generation pipeline combines dense 3D annotations, multi-view information, GPT-based processing, and rule-based spatial programming.
  • 3.4 Converting to Grounded Instruction-Following Format for LLM fine-tuning: Existing 3D tasks such as grounding, detection, dense captioning, and 3D QA are converted into instruction-following data using task-specific templates with referent correspondence.The conversion supports interleaved grounded responses and extends to embodied dialogue and planning.

4 EXPERIMENTS

Grounded 3D-LLM is evaluated across grounding, detection, captioning, and question-answering benchmarks as a generalist model without task-specific fine-tuning. It outperforms previous LLM-based models on most metrics, while ablations show benefits from diverse instruction templates, grounded scene-text pre-training, and one-to-many referent tokens.

  • Experimental setup: The evaluation covers single- and multi-object grounding, object detection and instance segmentation, dense object captioning, and 3D question answering on ScanNet-based benchmarks.The benchmarks use ScanNet’s 1,513 scenes, split into 1,201 training and 312 validation scenes.
  • Overall results: Grounded 3D-LLM outperforms previous LLM-based models on most metrics without task-specific fine-tuning, supporting its use as a unified 3D scene-understanding framework.Table 3 compares specialist and generalist models across the evaluated tasks.
  • Instruction templates: Diverse instruction-following templates significantly improve dense captioning and 3D QA, with Scan2Cap CIDEr@0.25 and CIDEr@0.5 each increasing by more than 8 points.Using a single template also produces unstable results across repeated experiments.
  • Referent-token ablations: Grounded scene-text pre-training raises grounding performance to 43.2 Acc@0.25 and 40.2 F1@0.25, while G-SceneCap alone achieves results comparable to all pre-training data.Directly adapting existing closed-set detectors is reported to be ineffective for integrating referent tokens.
  • Referent-token ablations: One-to-many referent tokens generally perform better than one-to-one tokens, where each object receives a unique token through Hungarian matching.The comparison evaluates alternative referent-token assignments for grounding.
  • CLASP evaluation: CLASP performs strongly on phrase grounding and detection, including ScanRefer, Multi3DRef, and ScanNet-200, while joint dataset training reaches 53.2 ScanRefer Acc@0.25 and 51.5 Multi3DRef F1@0.25.Adding G-SceneCap further improves ScanRefer Acc@0.25 by 2.1 points in the reported comparison.

5 CONCLUSION

Grounded 3D-LLM uses special language tokens to connect 3D scene objects or regions with natural language across multiple downstream tasks. Its broad effectiveness is accompanied by lower performance than expert models, motivating larger scene-text datasets.

  • Conclusion: Grounded 3D-LLM connects 3D scene objects or regions with language through scene referents represented as special language tokens.The model explores unified language modeling for multiple downstream 3D vision tasks.
  • Conclusion: The model remains limited relative to previous expert models, and scaling scene-text data is identified as future work to narrow this gap.

Supplementary Files for Grounded 3D-LLM

The supplementary materials provide additional ablations, training details, visualizations, dataset-generation information, and multi-task instruction templates.

  • Supplementary Files: Supplementary materials include additional ablations, hyperparameters, downstream-task visualizations and failure cases, grounded scene-text data-generation details, and multi-task instruction-following templates.They also include dataset statistics, generation prompts, and extensions to embodied tasks.

6 ADDITIONAL ABLATION STUDIES

Additional ablations examine how LLM scale and trainable components affect Grounded 3D-LLM. Larger models offer limited language-task gains, while semantically rich scene embeddings can be extracted with minimal trainable layers.

  • Effects of LLM model sizes: Grounded 3D-LLM models from 1B to 13B parameters outperform GPT-2, but larger models yield only marginal Scan2Cap gains and slight ScanQA declines.The current task scope may not fully challenge larger-scale LLMs.
  • Effects of LLM model sizes: A Vicuna-7B model training only the projection layer achieves comparable performance without LoRA, indicating that input scene embeddings are semantically rich.

7 IMPLEMENTATION DETAILS

Training uses separate CLASP pre-training and instruction fine-tuning configurations, with both stages optimized by AdamW and cosine learning-rate schedules.

  • CLASP pre-training runs for 600 epochs with 150 queries, batch size 5, and learning rate 1e-4.
  • Instruction fine-tuning runs for 50 epochs with 100 queries, batch size 5×200, and learning rate 8e-4.The instruction fine-tuning batch comprises five scenes, each containing up to 200 instructions.
  • Both training stages use cosine learning-rate schedules and AdamW optimization across 8 NVIDIA A100 GPUs.

8 RESULT VISUALIZATIONS

The visualizations show one Grounded 3D-LLM addressing multiple downstream tasks, while grounding failures arise from missing or incorrect objects and detection or segmentation errors.

  • Grounded 3D-LLM is visualized as one model solving multiple downstream benchmarks instead of using per-task fine-tuning.The visualization is presented in Fig. A-7.
  • Grounding failures include missing grounding objects, grounding semantically incorrect objects, and detection or segmentation errors.These failure categories are illustrated in Fig. A-8.
  • The examples cover ScanRefer single-object grounding, Multi3DRefer multiple-object grounding, ScanQA visual question answering, and Scan2Cap dense captioning.
  • The displayed outputs pair scene questions or grounding prompts with predicted object references, captions, or answers.

9 GROUNDED SCENE CAPTION DATASET

The Grounded Scene Caption dataset is automatically generated from ScanNet-200 using local object neighborhoods, language-model captioning, and program-derived spatial relations.

  • The pipeline selects anchor objects and nearby objects from ScanNet-200, limiting each local scene to at most 15 objects within an adaptively chosen radius.Local captions are constrained to fewer than 256 words.
  • Dataset statistics summarize spatial preposition–noun and noun–adjective relationships alongside noun diversity.
  • Grounded phrases are marked with brackets and linked visually to segmented point clouds in the dataset examples.
  • GPT-4 condenses object captions into natural local scene descriptions while retaining object IDs, and rule-based spatial relations are then incorporated.

10 TASK-SPECIFIC INSTRUCTION-FOLLOWING TEMPLATES

Task-specific instruction templates translate detection, grounding, question answering, and dense captioning into varied natural-language interactions with referent tokens.

  • Object detection: Detection templates vary questions and answer forms for single-object, multi-object, and no-object cases.
  • Data-generation prompts: Caption-generation prompts require natural captions containing key objects, attributes, unique IDs, and supplied spatial relationships.
  • Single and multi-object grounding: Grounding templates accommodate single, multiple, or absent referred objects, including ScanRefer and Multi3DRef settings.
  • 3D Question answering: ScanQA templates append instructions requiring answers as a single word or phrase, matching the dataset annotation guideline.
  • Dense captioning: Scan2Cap templates ask for object appearance followed by spatial relations.

11 EXTENSION TO EMBODIED DIALOGUE AND EMBODIED PLANNING

The extension applies Grounded 3D-LLM to embodied dialogue and planning by prompting GPT-4 with grounded scene captions and object identifiers. Visualized examples show referent-linked conversations and step-by-step plans for manipulating and describing scene objects.

  • Evaluation: The appendix reports 11.9K dialogue examples and 4.4K planning examples, with qualitative visualizations used because no well-defined benchmark jointly evaluates phrase grounding and language understanding.Figures A-20 and A-21 visualize embodied dialogue and planning outputs rather than reporting standardized benchmark scores.
  • Embodied dialogue: Grounded scene captions are converted into concise robot-user dialogues covering object appearance, spatial layout, design purpose, and simple scene-related activities.The dialogue prompt requires 3–5 rounds, role labels, natural exchanges, and no objects or details absent from the caption.
  • Embodied planning: Embodied planning prompts generate a JSON dictionary containing 2–4 high-level robot tasks, each expressed as short step-by-step actions grounded in uniquely identified scene objects.Each step must be simple and under 10 words, use the prescribed OBJ-ID format, and avoid adding unsupported objects or details.
  • Embodied dialogue: Dialogue examples interleave object descriptions with referent tokens, enabling responses about backpacks, desks, lamps, monitors, beds, boxes, pianos, doors, and other scene entities.The examples include object relocation, description, spatial relations, and explanations of object placement or purpose.
  • Embodied planning: Planning examples produce grounded action sequences for disposing of trash, preparing a piano area, polishing a mirror and rail, and arranging a bed.The plans explicitly reference objects such as tables, trash cans, piano benches, mirrors, rails, pillows, bedding, beds, and pianos.
Loading 2405.10370v2…