PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-LLM: Injecting the 3D World into Large Language Models

Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, Chuang Gan

arXiv:2307.12981v1cs.CVcs.AIcs.CLcs.LGcs.RO

TL;DR

LLMs and VLMs are not grounded in the richer concepts of the 3D physical world. This paper injects 3D representations into LLMs through 3D-LLMs trained on generated 3D-language data, using 2D VLM backbones and localization. The models outperform state-of-the-art baselines on ScanQA and 2D VLMs on held-in tasks, while relying on rendered multi-view images for their 3D features.

  • Problem

    LLMs and VLMs lack grounding in 3D physical-world concepts such as spatial relationships, affordances, and physics.

    Source: §1 · pp. 1–2

  • Method

    The paper trains 3D-LLMs on generated 3D-language data, using features from rendered multi-view images, pretrained 2D VLM backbones, and a 3D localization mechanism.

    Source: §1 · pp. 2–3

  • Results

    3D-LLMs outperform state-of-the-art ScanQA baselines and held-in 2D VLM baselines; BLEU-1 is ∼9% higher than the state-of-the-art ScanQA model on validation.

    Source: §5.1 · p. 7, §5.2 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    The paper reports that 3D-LLMs handle a diverse set of 3D-related tasks, including tasks beyond the scope of vanilla LLMs or 2D-LLMs.

    Source: §1 · p. 3

    The 3D feature extractor relies on 2D multi-view images, so scenes need to be rendered for training.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs. Project Page: : https://vis-www.cs.umass.edu/3dllm/.

1 Introduction

Existing LLMs and VLMs handle language and 2D images but lack grounding in the richer concepts of the 3D physical world. The paper introduces 3D-LLMs, broad 3D-language data, and a training approach that uses 2D VLM backbones and spatial localization.

1 Introduction

  • LLMs and VLMs lack grounding in 3D physical-world concepts such as spatial relationships, affordances, physics, and interaction.The introduction contrasts this limitation with their strengths in communication, commonsense reasoning, and 2D image understanding.Source: §1 · pp. 1–2
  • The proposed 3D-LLMs take 3D point clouds with features and language prompts as input and address diverse 3D-related tasks.The paper frames holistic scene understanding, spatial relationships, affordances, and 3D planning as task areas beyond vanilla LLMs or 2D-LLMs.Source: §1 · pp. 2–3
  • Three prompting procedures generate more than 300k 3D-language examples spanning tasks from captioning and question answering to grounding, dialogue, and navigation.The paper identifies scarcity of paired 3D data and language descriptions as a major data-acquisition challenge.Source: §1 · p. 2
  • The training approach derives 3D features from rendered multi-view images, uses 2D VLMs as backbones, and adds a localization mechanism for spatial information.The localization mechanism appends position embeddings to 3D features and trains output location tokens from object descriptions.Source: §1 · p. 3
  • 3D-LLMs outperform state-of-the-art ScanQA baselines, with a reported 9% BLEU-1 advantage.The paper also reports that the model outperforms 2D VLMs on held-in 3D captioning, task composition, and 3D-assisted dialogue datasets.Source: §1 · p. 3

2 Related Works

This section situates the work among language models, vision-language models, and 3D-language tasks. It distinguishes the paper's multi-task 3D model from task-specific 3D systems.

Large Language Models

  • LLMs are trained on large text corpora, while instruction tuning aims to align their predictions with human instructions and improve generalization to unseen tasks.The paper positions its work as injecting 3D concepts such as spatial relations, affordances, and physics into LLMs.Source: §2 · p. 3

Vision-Language Pre-trained Models

  • Vision-language models connect images and natural language, using either large image-language datasets or modules that link pretrained vision models with pretrained LLMs.The paper says its aim is an assistant that understands the 3D world and performs corresponding reasoning and planning.Source: §2 · p. 3
  • The paper identifies data sparsity, alignment between the 3D world and 2D images, and capture of 3D spatial information as obstacles.These obstacles are presented as challenges to building an assistant that understands and reasons about 3D.Source: §2 · pp. 3–4

3D & Language

  • Prior 3D-language models typically address individual tasks and may not generalize beyond their training distributions; this work targets multiple tasks, including dialogue and task decomposition.The passage names ScanQA, ScanRefer, and 3D captioning as examples of existing task-specific work.Source: §2 · p. 4

3 3D-Language Data Generation

The paper addresses the limited quantity and task diversity of existing 3D-language data by generating data from 3D assets with three prompting approaches. It draws on object and indoor-scene resources for the dataset.

3 3D-Language Data Generation

  • Existing 3D-language datasets are limited in quantity and diversity, often covering only one task, while paired 3D assets and language are difficult to obtain.The passage identifies generating a dataset usable across 3D-related tasks as an open need.Source: §3 · p. 4
  • Three prompting approaches generate 3D-language data: AABB-and-demonstration instructions, multi-view ChatCaptioner prompting, and revision between data types.The AABB approach provides room and object boxes and 0–3 few-shot examples; ChatCaptioner combines GPT questions with BLIP-2 answers across scene views to form a global description.Source: §3 · pp. 4–5
  • The prompting pipelines generate examples across multiple 3D-related tasks.Figure 1 is identified as showing examples from the generated data.Source: §3 · p. 5
  • The dataset is built mainly from Objaverse, ScanNet, and Habitat-Matterport resources, with semantic and bounding-box annotations available for the indoor-scene datasets.Objaverse descriptions are often noisy or absent, so ChatCaptioner is used to generate descriptions for its scenes.Source: §3 · p. 5

4 3D-LLM

The 3D-LLM training design reuses pretrained 2D visual-language components by mapping rendered-image features into 3D features. It also adds spatial encoding and location-token outputs to support localization.

4.1 Overview

  • Training from scratch is considered data-inefficient and resource-heavy, so the framework combines mapped 3D features with pretrained 2D VLM backbones and a localization mechanism.The approach is motivated by the smaller collected 3D-language dataset and the lack of pretrained 3D encoders comparable to image encoders.Source: §4.1 · p. 5

4.2 3D Feature Extractor

  • The feature extractor renders scenes from multiple views, extracts image features, and constructs features for 3D points from those rendered views.Direct reconstruction suits RGB-D data with accurate camera parameters; feature fusion suits noisy depth or camera parameters; neural fields suit RGB-only renderings with noisy camera parameters.Source: §4.2 · pp. 5–6

4.3.1 2D VLMs as backbones

  • Because the extracted 3D features share the feature space of pretrained image encoders, variable-sized point-cloud features can be fed to perceiver-style components in pretrained 2D VLMs.The paper uses these VLMs as backbones to train 3D-LLMs on its collected 3D-language data.Source: §4.3.1 · p. 6

4.3.2 3D Localization Mechanism

  • The localization mechanism combines position embeddings on 3D features with discrete location tokens that represent object bounding boxes.The location tokens encode discretized AABB corner coordinates, and their input and output embedding weights are unfrozen.Source: §4.3.2 · p. 6

5 Experiments

The experiments evaluate 3D-LLMs on held-out ScanQA and held-in tasks. The models outperform ScanQA baselines and outperform 2D VLMs and language-only LLMs on the reported held-in tasks.

5 Experiments

  • Training mixes held-in task datasets after an 8:1:1 train/validation/test split; ScanQA and 3DMV-VQA are held out from foundation-model training.The models use 2D VLM backbones, with most LLM parameters frozen for BLIP-2 and selected modules finetuned for Flamingo.Source: §5 · p. 7

5.1 Held-Out Evaluation

  • ∼9% on BLEU-1 over state-of-the-art ScanQA on validation and ∼7% on test; CIDER gains ∼5% over ScanQA.Table 1 covers ScanQA validation and Table 2 covers its test set.Source: §5.1 · p. 7, Table 1 · p. 8, Table 2 · p. 8
  • Multi-view image inputs perform much worse than 3D-LLMs; the authors suggest disorganized image features may lose 3D-related information.The authors contrast these inputs with holistic 3D features, which do not require explicit object representations.Source: §5.1 · pp. 7–8

5.2 More Extensive Evaluation

  • 3D-LLMs outperform 2D VLMs and language-only LLMs on held-in 3D captioning, 3D-assisted dialogue, and task decomposition.These held-in evaluations use the task-specific datasets and report response-quality metrics.Source: §5.2 · p. 8, Table 3 · p. 8
  • Figure 4 illustrates varied predictions, including dialogue, stepwise task decomposition, and scene descriptions.The examples include locating objects, responding in dialogue, and generating multi-step instructions.Source: §5.2 · p. 9

6 Conclusion

The paper introduces 3D-LLMs trained on a 300K-pair 3D-language dataset, using 2D pretrained VLM backbones and a 3D localization mechanism. It reports strong ScanQA results and diverse 3D-task performance, while noting that scene rendering is required.

6 Conclusion

  • The paper introduces 3D-LLMs that take 3D representations as input and generate responses across diverse 3D-related tasks.Its 300K 3D-language pairs cover tasks including dense captioning, question answering, task decomposition, grounding, dialogue, and navigation.Source: §6 · p. 9
  • The models use 2D pretrained VLM backbones and a 3D localization mechanism, and outperform state-of-the-art baselines on ScanQA.The paper also reports performance on a diverse set of 3D-related tasks.Source: §6 · p. 9
  • The 3D feature extractor relies on rendered multi-view images, so scenes must be rendered before training.The authors identify this as an additional rendering process.Source: §6 · p. 9

A 3D-Language Data

The appendix presents example prompts for generating task-decomposition and 3D-assisted-dialog data. The prompts use scene bounding boxes and demonstration responses to query ChatGPT.

A.1 Prompts

  • Figures 5 and 6 show exemplar prompts for generating task-decomposition and 3D-assisted-dialog data.The examples use the boxes-demonstration-instruction method.Source: §A.1 · p. 13, Figure 5 · p. 13, Figure 6 · p. 13
  • Few-shot examples pair scene bounding boxes with human-written responses, while each query provides scene boxes for ChatGPT to answer.The prompt content includes bounding boxes for rooms and objects in the scene.Source: §A.1 · p. 13

A.2 Data Distribution

The appendix points to Figure 7 as showing the distribution of the paper’s data.

A.2 Data Distribution

  • Figure 7 presents the distribution of the paper’s data.The supplied text does not specify the distribution’s categories or values.Source: §A.2 · p. 13

B.1 Implementation Details

On 3DMV-VQA, 3D-LLMs outperform the state-of-the-art baseline overall and on concept and relation questions. The experiments use pretrained 3D-LLMs and compare their performance with 3D and 2D baselines.

B.1 Implementation Details

  • BLIP-2-based 3D-LLMs train for 100K steps on 8 nodes with 8 V100s each, with batch size 16 per node; Flamingo-based 3D-LLMs train on 8 A100s with batch size 16.BLIP-2 uses AdamW with linear warmup over the initial 1K steps (10−8 to 10−5) and cosine decay to a minimum learning rate of 0; Flamingo uses AdamW, raising the learning rate linearly from 0 to 10−4 over 5,000 steps and holding it constant thereafter.Source: §B.1 · pp. 13–14

B.2.1 3DMV-VQA

  • The evaluation finetunes pretrained 3D-LLMs on 3DMV-VQA and compares them with neuro-symbolic, 3D-feature-plus-LSTM, and 2D VLM baselines.3D-CLR is described as extracting 3D feature fields before neuro-symbolic reasoning.Source: §B.2.1 · p. 14
  • 3D-LLMs outperform the state-of-the-art baseline overall and on concept and relation questions in 3DMV-VQA.They also outperform 3D-Feature+LSTM and their corresponding 2D VLMs with image input.Source: §B.2.1 · p. 14, Table 4 · p. 14

B.2.2 3D Grounding (Referring) on ScanRefer

On ScanRefer, 3D-LLMs predict object locations directly and outperform most baselines despite not using explicit object proposals or ground-truth boxes. The adjoining sections describe waypoint-based navigation, dense captioning, a Flamingo ablation, and qualitative examples across tasks.

B.2.2 3D Grounding (Referring) on ScanRefer

  • ScanRefer evaluation finetunes on its training set and reports validation results using ACC@0.25 IoU, average IoU, and center-to-ground-truth distances.Baselines include oracle- or VoteNet-proposal methods and 2D referring methods projected into 3D.Source: §B.2.2 · p. 15
  • 3D-LLMs outperform most ScanRefer baselines while predicting boxes directly with language-model losses, without explicit object proposals or ground-truth boxes.The authors report that predicted boxes are close to ground truth by average distance.Source: §B.2.2 · p. 15, Table 5 · p. 15

B.2.3 Object Navigation

  • For object navigation, the 3D-LLM predicts waypoints from partial-scene features, agent location, and history; a local policy converts waypoints into actions, and the model predicts when to stop.The illustrated episode explores when the target is unseen and directs the agent toward it after it becomes visible.Source: §B.2.3 · p. 15, Figure 8 · p. 16

B.3.1 3D Dense Captioning

  • In held-in 3D dense captioning, models caption regions specified by 3D bounding boxes, and 3D-LLMs outperform image-based baselines.Table 6 reports the dense-captioning results.Source: §B.3.1 · p. 16, Table 6 · p. 16

B.4.1 Ablative Studies on Flamingo Perceiver

  • Removing Flamingo’s perceiver resampler reduces its benefit for 3D-LLM training, supporting the module’s value in this setup.Table 7 presents the ablation.Source: §B.4.1 · p. 16, Table 7 · p. 16

B.5 More Qualitative Examples

Loading 2307.12981v1…