PaperlayerView exact revision on arXiv

Source-linked AI summary

LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning

Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, Tao Chen

arXiv:2311.18651v1cs.CV

TL;DR

LL3DA targets the challenge of understanding, reasoning, and planning over complex 3D environments represented by permutation-invariant point clouds, where prior multi-view approaches incur computational overhead. It directly encodes point clouds and combines textual instructions with visual prompts through interaction-aware features before a frozen LLM, achieving strong results across 3D Dense Captioning and 3D Question Answering.

  • Problem

    General-purpose 3D language models must handle complex environments and permutation-invariant point-cloud representations, while multi-view feature extraction introduces computational overhead.

  • Method

    LL3DA directly encodes 3D point clouds and aggregates textual instructions, visual prompts, and scene features into interaction-aware representations for a frozen LLM.

  • Results

    LL3DA surpasses various state-of-the-art 3D vision-language models on 3D Dense Captioning and 3D Question Answering benchmarks.

  • Takeaways & Limitations

    Visual interactions help remove ambiguities in cluttered 3D environments and support instruction following.

  • Takeaways & Limitations

    The LLM parameters and embedding layers are kept frozen to save memory cost.

Abstract

from arXiv · show

Recent advances in Large Multimodal Models (LMM) have made it possible for various applications in human-machine interactions. However, developing LMMs that can comprehend, reason, and plan in complex and diverse 3D environments remains a challenging topic, especially considering the demand for understanding permutation-invariant point cloud 3D representations of the 3D scene. Existing works seek help from multi-view images, and project 2D features to 3D space as 3D scene representations. This, however, leads to huge computational overhead and performance degradation. In this paper, we present LL3DA, a Large Language 3D Assistant that takes point cloud as direct input and respond to both textual-instructions and visual-prompts. This help LMMs better comprehend human interactions and further help to remove the ambiguities in cluttered 3D scenes. Experiments show that LL3DA achieves remarkable results, and surpasses various 3D vision-language models on both 3D Dense Captioning and 3D Question Answering.

1. Introduction

LL3DA addresses the difficulty of general-purpose 3D understanding in diverse environments and sparse point clouds by combining textual instructions with visual interactions. It uses interaction-aware 3D features with a frozen LLM and reports strong performance across 3D captioning and question answering.

  • 3D scene understanding supports autonomous driving and embodied AI but must handle diverse environments and sparse 3D points.
  • Existing 3D specialists are difficult to scale with limited supervision, while multi-view feature extraction adds computational overhead and overlooks 3D geometry.
  • LL3DA aggregates textual instructions, visual prompts, and 3D scenes into fixed-length querying tokens that prefix a frozen LLM.
  • Visual interactions help LL3DA address ambiguities in cluttered 3D environments during instruction following.
  • LL3DA achieves state-of-the-art results on widely used 3D Dense Captioning and 3D Question Answering datasets.

2. Related Work

Related work spans task-specific and multi-task 3D vision-language systems, LMMs that use visual tokens, and emerging LLM-based 3D understanding. LL3DA extends this direction by jointly handling textual and visual interactions with interaction-aware 3D representations.

  • 3D Dense Captioning translates a 3D scene into instance coordinates and natural-language descriptions.
  • Prior 3D vision-language research includes specialists for question answering, visual grounding, and dense captioning, alongside shared-structure multi-task models.
  • Large multimodal models commonly project global image features or extract region features as tokens for instruction reasoning.
  • LL3DA handles both visual prompts and textual instructions while extracting interaction-aware 3D scene representations for instruction following.

3. Methodology

LL3DA directly processes point-cloud scenes alongside textual instructions and visual interactions. Its Interactor3D combines frozen scene features, prompt encodings, and a multimodal transformer before a frozen LLM generates responses.

  • Model I/O: The model input comprises a point-cloud scene, textual instruction, and optional visual interactions; output is free-form language that may encode 3D coordinates.
  • Coordinate Representations: 3D points and boxes are serialized as coordinate text, with numerical values discretized to unsigned integers from 0 to 255.
  • Interactor3D: Interactor3D projects fixed-length interaction-aware scene embeddings as a textual prefix for a frozen LLM.
  • Scene Encoder: The scene encoder uses a masked transformer pretrained on ScanNet detection and outputs embeddings for uniformly down-sampled scene points.
  • Visual Prompt Encoder: User clicks are encoded with 3D Fourier positional embeddings, while box annotations use ROI features from a pretrained 3D object detector.
  • Multi-Modal Transformer: The multimodal transformer uses 32 learnable querying tokens to reconcile permutation-invariant scene embeddings with position-sensitive causal LLMs.
  • LLM: The LLM and its embedding layers remain frozen to save memory, and inference uses beam search with beam size 4.

4. Multi-modal Instruction Tuning

LL3DA frames multiple 3D tasks as autoregressive instruction following, varying textual instructions and visual prompts to distinguish tasks and provide auxiliary supervision. Training maximizes target-response likelihood with token-wise cross-entropy.

  • 4. Multi-modal Instruction Tuning: The general-purpose agent is trained to handle varied tasks in complex 3D scenes through multimodal instruction tuning.
  • 4.1. Tasks and Instructions.: All tasks are modeled as autoregressive generation following the “### assistant:” identifier.
  • 4.1. Tasks and Instructions.: 3D Dense Captioning uses clicks or box annotations and asks the model to describe or describe and localize an object.
  • 4.1. Tasks and Instructions.: 3D Question Answering uses answer or answer-and-localize instructions, with visual prompts randomly applied to mentioned objects during training.
  • 4.1. Tasks and Instructions.: Scene Description asks the model to translate global 3D-scene knowledge into natural language.
  • 4.1. Tasks and Instructions.: Embodied Conversation and Planning use human and assistant identifiers to distinguish information sources in multi-turn interactions.
  • 4.2. Instruction Following Tuning: Training optimizes target-response likelihood conditioned on the point cloud and textual and visual interactions.
  • 4.2. Instruction Following Tuning: Token-wise cross-entropy trains prediction of each response token from the preceding tokens, where |s| denotes the target response length.

5. Experiments

The experiments evaluate LL3DA on 3D dense captioning, 3D question answering, model design, auxiliary interactions, and generalist performance. Results show strong benchmark performance and benefits from interaction-aware visual prompting and textual instructions.

  • Experimental Setup: LL3DA is evaluated on ScanNet-derived datasets covering instance descriptions, scene descriptions, conversations, embodied planning, and question answering.The evaluation includes ScanRefer, Nr3D, ScanQA, and 3D-LLM annotations.
  • Comparison with SoTA Specialists: LL3DA consistently outperforms existing methods on ScanRefer and Nr3D for 3D Dense Captioning.Using Vote2Cap-DETR box predictions as visual prompts, it achieves 65.19% C@0.5 on ScanRefer and 51.18% on another reported setting.
  • Comparison with SoTA Specialists: LL3DA outperforms previous methods on the ScanQA validation set and two test sets for 3D Question Answering.The comparison includes both classification-based and generation-based methods.
  • Ablation Studies: Early fusion improves ScanRefer captioning by 3.45% C@0.5 over directly injecting visual prompts after scene embeddings.The unified transformer can aggregate instruction and visual-prompt information before producing interaction-aware features.
  • Ablation Studies: Textual instructions and visual prompts improve task diversity and performance when used as auxiliary tasks for 3D Dense Captioning and 3D Question Answering.For question answering, related-object clicks improve answer quality by +6.12% C.
  • Ablation Studies: As a generalist, LL3DA distinguishes 3D Dense Captioning and 3D Question Answering using shared weights, achieving 62.98% C@0.5 on ScanRefer and 75.67% CiDEr on ScanQA.Generalist weights also provide a strong initialization for fine-tuning; the fine-tuned ScanRefer model reaches 65.19% C@0.5, +2.35% above training from scratch.

6. Conclusions

LL3DA directly uses 3D point clouds and human textual and visual interactions to support understanding, reasoning, and planning in complex 3D environments. Visual interactions help remove ambiguities in cluttered scenes, while experiments show strong performance across 3D vision-language benchmarks.

  • LL3DA accepts textual and visual human interactions while directly encoding 3D point clouds for complex-environment understanding, reasoning, and planning.
  • Visual interactions remove ambiguities in cluttered 3D environments and demonstrate strong instruction-following capabilities.
  • Experimental results show remarkable performance across various 3D vision-language benchmarks.

Supplementary Material

The supplementary material extends the paper with visualizations, quantitative evaluations, implementation details, data statistics, training samples, and released resources. It also includes example code and video demonstrations.

  • The supplementary material provides visualizations, quantitative evaluations, coordinate studies, data statistics, training samples, and additional implementation details.
  • Code and pre-trained weights are intended for full release, together with example training and evaluation files.
  • Video clips provide additional illustrative demonstrations.

A. More Visualizations

Additional visualizations cover scene descriptions, 3D dense captioning, 3D question answering, embodied dialogue, and embodied planning. The planning examples also show feedback generation after completed actions.

  • Additional visualizations cover scene descriptions, 3D dense captioning, 3D question answering, embodied dialogue, and embodied planning.
  • Scene-description examples highlight phrases that mention instances in the 3D environment.
  • Results for dense captioning, question answering, and embodied dialogue mark wrong answers in red.
  • Embodied-planning results are described as commonsense-consistent and include both direct plans and feedback based on completed actions.

B. More Evaluations

The supplementary evaluations use custom ScanNet splits and report language-generation metrics for scene description, embodied dialogue, and embodied planning. They compare sampling strategies for LL3DA against zero-shot frozen LLM baselines.

  • B. More Evaluations: Because 3D-LLM did not release its validation set, the authors create a split for scene descriptions, embodied dialogue, and embodied planning.
  • B. More Evaluations: Scenes with ids less than 600 form training data, while the remaining scenes form validation data.
  • B. More Evaluations: Multi-turn dialogues are decomposed into 8,490 training and 1,222 validation samples.
  • B. More Evaluations: Embodied-planning data is decomposed into 16,972 training and 2,282 validation samples.
  • B. More Evaluations: Generated responses are evaluated with CiDEr, BLEU, METEOR, and Rouge-L.
  • B. More Evaluations: The multi-turn dialogue and embodied-planning decomposition pipelines produce separate training examples while preserving human and assistant identifiers for dialogue data.
  • B. More Evaluations: The ScanNet split reports annotation and scene counts for each task and split.
  • B. More Evaluations: Table 10 evaluates sampling strategies on the ScanNet portion of 3D-LLM while applying a fourth-order n-gram repetition penalty.

C. Additional Studies: Generating 3D Bounding Boxes with LLM

This study evaluates LL3DA's ability to generate 3D bounding boxes from language-model outputs for detection and dense captioning. The results indicate reasonable localization, though a gap remains relative to specialized 3D localizers.

  • Quantitative Findings: LL3DA achieves performance comparable to a 3D detector specialist at IoU=0.5 in open-vocabulary detection.The evaluation treats 3D detection as a 3D dense-captioning problem and decodes both boxes and category names from generated text.
  • Evaluation: The dense-captioning evaluation uses m@kIoU metrics based on caption quality and whether predicted boxes meet the specified IoU threshold.Metrics include CiDEr, BLEU-4, METEOR, and Rouge-L.
  • Box Generation: LL3DA reconstructs 3D bounding boxes by decoding center coordinates and dimensions from generated <obj> tokens.The decoded format is “<obj>cx, cy, cz, w, h, l</obj>”.
  • Quantitative Findings: LL3DA produces reasonable bounding-box estimates, although its localization remains below specialized 3D localizers.This comparison is reported for dense captioning with boxes generated by LL3DA.
  • Qualitative Findings: Qualitative visualizations show tight reconstructed boxes close to objects across diverse and complex 3D environments.The visualization simulates user clicks with vote-query spatial locations.

E. Additional Implementation Details

The implementation uses a pretrained masked transformer for point-cloud encoding and a six-layer multimodal transformer for integrating model inputs. Training updates about 111M parameters while keeping the OPT-1.3B language-model backbone frozen.

  • Scene Encoder: The scene encoder tokenizes each input point cloud into 2,048 point tokens using a set-abstraction layer.It then applies three cascaded masked transformer blocks with radii 0.16, 0.64, and 1.44.
  • Multi-Modal Transformer: The multimodal transformer has feature dimension 768, uses up to 12 attention heads, and contains six transformer layers.It loads pretrained word and positional embeddings from BERT.
  • Trainable Parameters: About 111M parameters are trainable, representing less than 10% of the frozen OPT-1.3B language-model backbone.The implementation therefore trains a relatively small parameter subset compared with the backbone.
Loading 2311.18651v1…