PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment

Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, Qing Li

arXiv:2308.04352v1cs.CV

TL;DR

3D-VL seeks to connect physical 3D scenes with natural language, but existing models often depend on sophisticated task-specific designs and training techniques. 3D-VisTA addresses this with a simple self-attention Transformer and ScanScribe pre-training, achieving state-of-the-art results across multiple 3D-VL tasks with strong data efficiency.

  • Problem

    Existing 3D-VL models rely heavily on sophisticated task-specific modules, auxiliary losses, and optimization tricks, while large-scale 3D-VL pre-training has been rarely explored.

  • Method

    3D-VisTA uses self-attention for single-modal modeling and multimodal fusion, and is pre-trained on the 278K-pair ScanScribe dataset with masked language/object modeling and scene-text matching.

  • Results

    3D-VisTA achieves state-of-the-art results across visual grounding, dense captioning, question answering, and situated reasoning, with strong performance under limited downstream annotations.

  • Takeaways & Limitations

    A simple pre-trained Transformer can support diverse 3D-VL tasks without auxiliary losses or sophisticated optimization techniques.

Abstract

from arXiv · show

3D vision-language grounding (3D-VL) is an emerging field that aims to connect the 3D physical world with natural language, which is crucial for achieving embodied intelligence. Current 3D-VL models rely heavily on sophisticated modules, auxiliary losses, and optimization tricks, which calls for a simple and unified model. In this paper, we propose 3D-VisTA, a pre-trained Transformer for 3D Vision and Text Alignment that can be easily adapted to various downstream tasks. 3D-VisTA simply utilizes self-attention layers for both single-modal modeling and multi-modal fusion without any sophisticated task-specific design. To further enhance its performance on 3D-VL tasks, we construct ScanScribe, the first large-scale 3D scene-text pairs dataset for 3D-VL pre-training. ScanScribe contains 2,995 RGB-D scans for 1,185 unique indoor scenes originating from ScanNet and 3R-Scan datasets, along with paired 278K scene descriptions generated from existing 3D-VL tasks, templates, and GPT-3. 3D-VisTA is pre-trained on ScanScribe via masked language/object modeling and scene-text matching. It achieves state-of-the-art results on various 3D-VL tasks, ranging from visual grounding and dense captioning to question answering and situated reasoning. Moreover, 3D-VisTA demonstrates superior data efficiency, obtaining strong performance even with limited annotations during downstream task fine-tuning.

1. Introduction

3D-VisTA addresses the complexity of existing 3D-VL models with a simple Transformer architecture and ScanScribe pre-training dataset. The resulting model supports multiple downstream tasks and achieves strong performance with limited annotations.

  • Existing 3D-VL models commonly use task-specific modules, auxiliary losses, and optimization tricks, motivating a simpler unified approach.Prior systems often focus on only one or two tasks and require specialized designs for grounding, captioning, or multimodal learning.
  • 3D-VisTA uses self-attention for both single-modal modeling and multimodal fusion, while encoding pairwise object relations into attention weights.This design targets 3D spatial comprehension without sophisticated task-specific modules.
  • ScanScribe contains 278K 3D scene-text pairs from 2,995 RGB-D scans covering 1,185 indoor scenes and 56.1K unique object instances.The dataset combines scans, object replacements, existing 3D-VL text, scene graphs, templates, and GPT-3-generated descriptions.
  • 3D-VisTA is pre-trained with masked language modeling, masked object modeling, and scene-text matching to align 3D point clouds with text.The pre-training procedure is intended to reduce the need for auxiliary losses and optimization tricks during downstream fine-tuning.
  • Fine-tuned 3D-VisTA achieves state-of-the-art performance across visual grounding, dense captioning, question answering, and situated reasoning, while remaining data-efficient.The reported tasks include ScanRefer, Nr3D, Sr3D, Scan2Cap, ScanQA, and SQA3D.

2. Related Work

Related work establishes 3D-VL benchmarks and models while highlighting the field’s reliance on task-specific designs and sophisticated optimization. 3D-VisTA instead combines multiple tasks through simple Transformer-based modeling and introduces a larger pre-training dataset.

  • 3D-VL benchmarks span visual grounding, question answering, situated reasoning, and related tasks that connect 3D scenes with natural language.Examples include ScanRefer, ReferIt3D, and ScanQA.
  • Prior 3D-VL models use cross attention, spatial-relation information, task-specific modules, auxiliary losses, or sophisticated optimization techniques.These design choices vary across grounding, captioning, and other benchmark tasks.
  • 3D-VisTA unifies visual grounding, question answering, and situated reasoning through a simple Transformer architecture without auxiliary losses or sophisticated optimization techniques.Table 1 compares models by supported tasks, auxiliary losses, and task-specific architectures.
  • Large-scale pre-training has advanced NLP, computer vision, and 2D vision-language tasks, but has rarely been explored in 3D-VL.The authors identify the lack of a large-scale pre-training dataset as a possible hindrance.
  • ScanScribe is introduced as the first large-scale 3D scene-text dataset for 3D-VL pre-training and is described as larger and more text-diverse than existing 3D-VL datasets.Table 2 compares ScanScribe with other datasets using task coverage and vocabulary-related fields.

3. 3D-VisTA

3D-VisTA is a simple, unified Transformer that encodes text and 3D scenes, fuses their tokens, and learns alignment through self-supervised pre-training. It can then be adapted to diverse 3D-VL tasks with lightweight heads and task losses alone.

  • Architecture: 3D-VisTA encodes sentence tokens and 3D object tokens before combining them in a unified Transformer for multi-modal fusion.The text module uses a four-layer Transformer, while object tokens combine point features, semantic classes, and locations.
  • Scene Encoding: Pairwise object distances and horizontal and vertical angles modulate self-attention to provide spatial context during scene encoding.The spatial features are mapped to attention scores through a learned projection and sigmoid function.
  • Self-supervised Pre-training: 3D-VisTA pre-trains with masked language modeling, masked object modeling, and scene-text matching to learn local and global scene-text alignment.Masked modeling targets fine-grained correspondence, while scene-text matching uses the [CLS] representation for global fusion.
  • Self-supervised Pre-training: The final pre-training objective simply adds the losses from the three proxy tasks, with negative scene-text pairs included for matching.Thirty percent of training-batch samples are negative pairs formed by replacing the scene or text with a random sample.
  • Adaptation: Self-supervised, task-agnostic pre-training avoids the task supervision used by prior supervised multi-task learning.This design is intended to eliminate auxiliary losses and optimization tricks during downstream fine-tuning.
  • Adaptation: The model adapts to visual grounding, dense captioning, question answering, and situated reasoning by adding lightweight task heads.Downstream fine-tuning uses the task loss only, without auxiliary losses or optimization tricks.

4. ScanScribe

ScanScribe is a large-scale 3D scene-text dataset built to support 3D-VL pre-training. It combines RGB-D indoor scans with descriptions derived from existing datasets, templates, and GPT-3.

  • Dataset Construction: ScanScribe addresses the limited availability of 3D-VL pre-training data with a large-scale collection of 3D scene-text pairs.The dataset is described as the first large-scale 3D scene-text pairs dataset for 3D-VL pre-training and is larger and more text-diverse than existing 3D-VL datasets.
  • 3D Scenes: 2,995 RGB-D scans cover 1,185 indoor scenes and 56.1K unique object instances collected from ScanNet and 3R-Scan.Ten percent of object instances in each scene are randomly replaced with category-matched Objaverse objects to increase object diversity.
  • Text: Scene descriptions combine transformed ScanNet text with template- and GPT-3-generated descriptions for 3R-Scan scenes.The source material includes ScanQA question-answer pairs, ScanRefer and ReferIt3D referring expressions, and 3R-Scan scene-graph annotations.

5. Experiments

Experiments show that 3D-VisTA is competitive even without pre-training or auxiliary optimization techniques, while ScanScribe pre-training improves results across diverse 3D-VL tasks and enables strong performance with limited annotations. Ablations and qualitative analyses indicate benefits from the combined objectives, expanded data, and improved spatial, visual, situational, and long-text understanding.

  • 5.2. Downstream Task Results: 57.5% and 69.6% overall accuracy on Nr3D and Sr3D, respectively, make 3D-VisTA trained from scratch competitive with most previous models.The scratch model also reaches 25.2% EM@1 on ScanQA, 1.7% above the previous SOTA, using only task losses.
  • 5.2. Downstream Task Results: 6.7%/6.8% accuracy gains on Nr3D/Sr3D and improvements across ScanRefer, ScanQA, Scan2Cap, and SQA3D show broad benefits from ScanScribe pre-training.Reported gains include 4.7%/4.3% on ScanRefer acc@0.25/0.5, 1.8%/2.6% on ScanQA EM@1, 4.2% on Scan2Cap C@0.25, and 1.8% on SQA3D average accuracy.
  • 5.2. Downstream Task Results: 3D-VisTA outperforms cited SOTA methods across Sr3D, ScanRefer, ScanQA, Scan2Cap, and SQA3D after pre-training.The reported margins are 3.6% on Sr3D, 2.7%/8.1% on ScanRefer, 3.5%/2.1 on ScanQA EM@1, 10.1%/19.2% on Scan2Cap C@0.25/0.5, and 1.9% on SQA3D average accuracy.
  • 5.2. Downstream Task Results: 30% and 40% of ScanRefer and ScanQA annotations suffice for pre-trained 3D-VisTA to outperform scratch training with full data.This result is presented as evidence of strong data efficiency during downstream fine-tuning.
  • 5.3. Ablation Studies: Using all three pre-training objectives together yields the best VG and QA performance, while adding 3R-Scan and Objaverse further boosts both tasks.The ablation describes MLM, MOM, and STM as complementary, and reports the best results when all three data sources are used.
  • 5.4. Qualitative Studies and Additional Results: Pre-training improves spatial relations, visual concepts, situations, and alignment with longer queries in qualitative and additional analyses.The reported effects include distinguishing same-class object instances, understanding colors and shapes, and larger gains for longer ScanRefer queries.

6. Conclusion

3D-VisTA is presented as a simple, adaptable 3D-VL architecture using self-attention without auxiliary losses or optimization tricks. Its pre-training and data efficiency support strong results across diverse 3D-VL tasks, while current object detection and dataset scale remain boundaries.

  • 3D-VisTA uses self-attention layers and adapts to various 3D-VL tasks without auxiliary losses or optimization tricks.
  • ScanScribe is introduced as the first large-scale 3D scene-text pairs dataset for 3D-VL pre-training.
  • The pre-trained model achieves state-of-the-art results across varied 3D-VL tasks and shows superior data efficiency.
  • Future Works: An offline 3D object detection module may bottleneck improvement, while ScanScribe remains insufficiently large for large-scale pre-training.

A.1. Downstream Tasks

The downstream evaluation covers 3D visual grounding and 3D question answering, using established datasets, task definitions, and metrics. These tasks emphasize identifying described objects and answering questions involving spatial relations.

  • 3D Visual Grounding: ScanRefer evaluates finding the target object described by a sentence in 3D scenes.Its dataset contains 51,583 human-written sentences describing 800 ScanNet scenes.
  • 3D Question Answering: ScanQA evaluates 3D question answering with 41,363 questions and 58,191 answers, focusing more on spatial relations than 2D QA.
  • 3D Question Answering: ScanQA uses exact-match metrics EM@1 and EM@10, alongside text-similarity metrics such as BLEU-4 and ROUGE.

A.2. Model Architecture

The model architecture combines a PointNet++ scene encoder with unified multimodal Transformer processing and task-specific prediction heads.

  • Scene Encoder: A three-layer PointNet++ scene encoder aggregates point features into a 768-dimension representation.The layers use radii 0.2, 0.4, and all points.
  • Multimodal Fusion: Text and object tokens use dimension 768 in the multimodal fusion layers.
  • Prediction Heads: The unified encoder uses 12 attention heads and 2048-dimensional feedforward layers, with an MLP grounding head.The visual grounding head has two layers and hidden dimension 384.

A.3. Training settings

Training settings define task-specific fine-tuning procedures and describe ScanScribe text generation from scene relations. Template-based and GPT-3-generated descriptions are balanced during pre-training.

  • Fine-tuning: Grounding tasks use only cross-entropy loss during fine-tuning on ScanRefer, Nr3D, and Sr3D.
  • Fine-tuning: ScanQA combines answer-classification and visual-grounding losses, while SQA3D uses answer-classification loss.
  • Scene-text Generation: ScanScribe generation supports 40 relations and maps scene relations to descriptions through templates.
  • Scene-text Generation: GPT-3 increases text diversity, and its generated texts are duplicated 15 times to balance template and GPT-3 text proportions in 3R-Scan pre-training.

B. Additional Results

Additional results examine ScanScribe text sources, relation descriptions, and qualitative model behavior. GPT-3-generated text improves spatial-relation benchmarks, while qualitative examples show strengths alongside limitations on complicated spatial relations and semantics.

  • Ablation Studies: GPT-3-generated text improves Sr3D by 1.0% and Nr3D by 1.5%, with little impact on ScanRefer and ScanQA.The ablation compares template-generated and GPT-3-generated text using results from 3R-Scan.
  • Relation Descriptions: ScanScribe maps spatial and object relations such as left, right, front, inside, attached to, and standing on to textual descriptions.The mapping includes comparative, spatial, containment, symmetry, and support relations.
  • Dataset Text: GPT-3-generated ScanScribe descriptions are more natural than template-generated descriptions.Figure A1 provides examples of both text-generation approaches.
  • Qualitative Results: Qualitative results show advantages in spatial reasoning, concept grounding, and situation understanding, but failures remain for some complicated spatial relations and semantic cases.Figure A2 reports qualitative results on ScanRefer, ScanQA, and SQA3D, with green ground-truth and red predicted object boxes.
Loading 2308.04352v1…