PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness

Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, Xihui Liu

arXiv:2409.18125v3cs.CV

TL;DR

LMMs have strong 2D visual capabilities but lack 3D spatial intelligence needed for physical-world interaction. LLaVA-3D adapts LLaVA with 3D patches and decoding components, achieving state-of-the-art performance across 3D tasks while preserving 2D capabilities.

  • Problem

    LMMs excel at 2D visual understanding but lack 3D spatial intelligence needed for physical-world interaction.

  • Method

    LLaVA-3D enhances 2D CLIP patches with 3D position embeddings, compresses 3D patches when needed, and uses a grounding decoder for 3D scene understanding and grounding.

  • Results

    LLaVA-3D achieves state-of-the-art performance across a wide range of 3D scene understanding benchmarks and tasks.

  • Takeaways & Limitations

    LLaVA-3D extends LLaVA to 3D scene understanding and grounding while efficiently preserving 2D visual understanding and reasoning capabilities.

  • Takeaways & Limitations

    Directly outputting 3D target positions remains difficult, with text-format and special-token approaches achieving 7.8 and 8.2 Acc@0.25, respectively.

Abstract

from arXiv · show

Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D scene understanding capabilities has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D visual understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we utilize the 3D position embeddings to enhance the 2D CLIP Patches with 3D spatial context information and construct 3D patches. By integrating the 3D position embeddings into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D visual understanding and 3D scene understanding. In contrast to previous 3D LMMs, LLaVA-3D supports decoding accurate 3D spatial perception outputs, e.g., 3D bounding boxes, directly from these 3D patches, without relying on the time-consuming off-the-shelf 3D segmentors. Experimental results show that LLaVA-3D converges 3.5x faster than existing 3D LMMs when trained on 3D vision-language datasets. Moreover, LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks but also maintains comparable 2D visual understanding and vision-language conversation capabilities with LLaVA.

1. Introduction

LLaVA-3D addresses the limited 3D spatial intelligence of LMMs by extending LLaVA with 3D capabilities while preserving robust 2D multimodal understanding. It uses 3D position embeddings and achieves strong 3D performance with efficient training and minimal preprocessing.

  • LMMs excel at 2D visual tasks but lack the 3D spatial intelligence needed for physical-world interaction.
  • 3D LMM development is constrained by scarce large-scale 3D datasets and the absence of powerful pretrained 3D point-cloud encoders.
  • Existing multi-view approaches rely on segmentation-based CLIP feature extraction, creating a complex and computationally intensive pipeline.
  • LLaVA-3D incorporates 3D spatial coordinates into 2D CLIP patches, producing 3D patches for 3D understanding while retaining LLaVA’s 2D capabilities.
  • LLaVA-3D achieves state-of-the-art performance across diverse 3D scene-understanding benchmarks with less training time and fewer epochs than existing 3D LMMs.

2. Related Work

Prior work extends LLMs with visual inputs through 2D LMMs, point-cloud or multi-view 3D representations, and joint 2D–3D feature modeling. LLaVA-3D builds on these directions by integrating 3D position-aware features into a 2D LMM.

  • 2D LMMs: LLaVA aligns 2D images with language models using an image encoder and projection layer, while BLIP2 uses a Q-Former to compress visual features with textual cues.
  • 2D LMMs: Video LMMs use multi-image inputs to capture temporal dynamics, while multi-view images provide a basis for representing 3D scenes.
  • Injecting 3D into LLMs: 3D scene-understanding methods integrate point clouds or multi-view images into LLMs, differing primarily in how they construct 3D scene representations.
  • Joint Modeling of 2D and 3D: Joint 2D–3D methods combine 2D foundation-model features with 3D position embeddings to create position-aware features for detection and segmentation.
  • Joint Modeling of 2D and 3D: LLaVA-3D extends this modeling approach to 2D LMMs, enabling them to perform 3D scene understanding through 3D position-aware features.

3. Method

LLaVA-3D constructs 3D-aware visual tokens from multi-view image patches, compresses them spatially when needed, and decodes coordinate-aware 3D outputs through a grounding decoder.

  • 3D Patch Construction: The method bridges 2D image features with 3D spatial context, then applies 3D-aware pooling and position encoding and decoding.
  • 3D Patch Construction: LLaVA-Video’s CLIP encoder splits each frame into patches and projects their visual features into the LLM space.
  • 3D Patch Construction: 3D patches combine projected multi-view 2D patch features with embeddings of their corresponding 3D positions.
  • 3D Patch Pooling: When patch counts threaten the LLM context length, 3D-aware pooling reduces tokens while preserving scene structure as completely as possible.
  • 3D Patch Pooling: Voxelization pooling averages patches within occupied voxels, whereas FPS pooling samples a fixed number of representative 3D patches.
  • 3D-aware Position Encoding & Decoding: The grounding decoder uses sampled instance queries, cross-attention, location tokens, and distance-adaptive self-attention to produce accurate 3D grounding outputs.

4. Training

LLaVA-3D uses two-stage joint instruction tuning to add 3D task abilities while preserving 2D capabilities and improving grounding. Its training uses 2D and 3D instruction data followed by decoder-focused fine-tuning.

  • Training Strategy: Two-stage training first teaches joint 2D and 3D instruction following, then further enhances 3D grounding.
  • Stage 1: Multi-Task Instruction Tuning: Stage 1 jointly fine-tunes on LLaVA-Video 2D data and the LLaVA-3D-Instruct-86K 3D instruction dataset.
  • Stage 2: Decoder-only Fine-tuning: Stage 2 trains the grounding decoder for additional epochs while freezing other components because it does not converge within one first-stage epoch.
  • Stage 2: Decoder-only Fine-tuning: The location token is further trained with the grounding decoder without influencing the model’s strong 2D and 3D scene-understanding capabilities.
  • Training Data: The LLaVA-3D-Instruct-86K collection organizes 3D instruction data by categories, subsets, and detailed dataset quantities.

5. Experiments

LLaVA-3D is evaluated across 3D question answering, dense captioning, visual grounding, and 2D video understanding, alongside analyses of its architecture and components. Results show strong 3D performance, direct single-stage grounding, preserved 2D capabilities, and benefits from 3D patches and 2D LMM initialization.

  • 3D Question Answering: LLaVA-3D achieves state-of-the-art performance across ScanQA and SQA3D, while the benchmarks may not fully assess 3D spatial understanding.LLaVA-Video can surpass task-specific methods zero-shot on these benchmarks, motivating caution about their diagnostic value.
  • 3D Question Answering: LLaVA-3D significantly outperforms LL3DA and LEO on MMScan QA, which requires coordinate-based 3D reasoning beyond text descriptions.The comparison uses models further fine-tuned on the full MMScan QA dataset.
  • 3D Question Answering: LLaVA-3D surpasses Claude3 and Gemini-Pro and performs comparably to GPT-4V on OpenEQA with significantly fewer model parameters.OpenEQA evaluates spatial understanding and embodied reasoning using more than 1,600 human-generated questions from diverse real-world environments.
  • 3D Dense Captioning: LLaVA-3D consistently outperforms existing methods on Scan2Cap, while using object proposals and 3D coordinate tokens to guide dense captioning.Scan2Cap requires object appearance, spatial-relation descriptions, and corresponding 3D bounding boxes.
  • 3D Visual Grounding: 49.8 Acc@0.25 on Multi3DRefer is achieved by directly decoding accurate 3D bounding boxes from 3D patches in a single-stage grounding manner.Unlike prior approaches, this method does not rely on constructed point clouds or an off-the-shelf 3D segmentor.
  • 2D Retention and More Analysis: LLaVA-3D retains performance comparable to LLaVA-Video across 2D video understanding and conversation benchmarks, while 3D patches yield 13.3% and 54.4% gains on MMScan QA and Scan2Cap.Developing from a 2D LMM improves ScanRefer by 4.1% Acc@0.25, and CLIP-only features reduce scene construction time from 900s to 0.2s.

6. Conclusion

LLaVA-3D extends LLaVA with 3D patches and a grounding decoder for 3D scene understanding while preserving 2D visual understanding and reasoning. It achieves state-of-the-art performance across various 3D tasks and benchmarks.

  • LLaVA-3D extends LLaVA’s capabilities to 3D scene understanding and grounding using 3D patches and a grounding decoder.
  • The framework efficiently preserves LLaVA’s 2D visual understanding and reasoning capability.
  • LLaVA-3D achieves state-of-the-art performance on various 3D tasks and benchmarks.

A. Implementation Details

LLaVA-3D uses a two-stage training process with successive weight initialization, specified optimization settings, and a second stage that trains only the grounding decoder.

  • LLaVA-3D is built on LLaVA-Video-7B and uses pretrained HuggingFace weights in a two-stage training process.Each stage builds upon weights learned in the previous stage.
  • Stage 1 trains for one epoch with Adam, total batch size 16, warmup ratio 0.03, and language-modeling plus grounding-decoder losses.Peak learning rates are 1e-5 for the LLM, 3D position encoding layer, and grounding decoder, and 2e-6 for the vision encoder.
  • Stage 2 freezes all components except the grounding decoder and trains for 40 epochs on 16 A100 GPUs.The peak learning rate is 1e-4.

B. Training Convergence Speed

The convergence comparison uses LLaVA-1.5 and LEO under matched Vicuna-7B settings, while evaluating performance across different training-data ratios.

  • LLaVA-1.5 is used as the base model, with LLaVA-3D-Instruct-86K replaced by MMScan QA training data in Stage 1.
  • LLaVA-3D performance is recorded and evaluated under different training-data ratios.
  • LEO is further fine-tuned on the full MMScan QA training data from its officially released checkpoint, with both models using Vicuna-7B.

C. More Architecture Details

The grounding decoder connects LLM localization tokens to 3D instance queries, using multi-scale spatial attention and a box head to predict 3D bounding boxes. Figure 5 reports higher data efficiency and faster convergence than LEO.

  • Training convergence comparison: Figure 5 reports higher data efficiency and faster convergence for LLaVA-3D than the existing 3D LMM LEO during instruction tuning.
  • Grounding decoder connection: The LLM predicts a special <LOC> token whose final embedding conditions the grounding decoder for 3D bounding-box prediction.
  • Query initialization: The decoder initializes 512 instance queries using farthest point sampling from 3D patches.Query values are initialized to zero, while sampled 3D coordinates determine learnable positional encodings.
  • Multi-scale cross-attention: Multi-scale cross-attention restricts each instance query to k nearest 3D patch neighbors, using k values of 16, 32, 64, and 128 across decoder layers.Relative query-neighbor positions are encoded with a learnable MLP and added to the attention inputs.
  • Distance-adaptive self-attention: Distance-adaptive self-attention models spatial relationships among queries and supports visual-language feature communication.Its bias uses Euler distances between instance queries; the location-token bias is set to zero.
  • Box prediction and losses: A two-layer MLP box head converts updated instance queries into corresponding 3D bounding boxes, optimized with DIOU and InfoNCE losses.

D. More Components Analysis

The component analysis examines pooling, view sampling, and practical 3D interaction. Results indicate that pooling choice and view count affect efficiency and performance, while the model supports diverse 3D understanding and video-based inputs.

  • Impact of Pooling Strategy: Voxelization pooling outperforms FPS pooling in the reported ScanQA and SQA3D ablation.The study evaluates pooling strategies using LLaVA-1.5 as the foundation model.
  • Multi-View Images Sampling Strategy: Uniform and text-guided view sampling achieve similar performance, so the authors adopt uniform sampling for simplicity.Text-guided sampling selects frames using CLIP image-text similarity to the input instruction.
  • Number of Views: 3D QA Exact Match increases only marginally as the number of sampled views grows, and excessive views can degrade performance.The comparison covers ScanQA and SQA3D under different multi-view image settings.
  • 3D Scene Understanding: LLaVA-3D demonstrates comprehensive scene understanding, accurate object recognition, precise 3D localization, and 2D image-based interaction.Visual examples include 2D click-based dense captioning, question answering, and 3D bounding-box prediction.
  • 3D Scene Understanding: The framework converts ordinary videos into model inputs by uniformly sampling 32 frames and using DUST3R to estimate depth, camera parameters, and poses.DUST3R completes this offline multi-view reconstruction within one minute.
Loading 2409.18125v3…