Source-linked AI summary
3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer
Jiajun Deng, Tianyu He, Li Jiang, Tianyu Wang, Feras Dayoub, Ian Reid
TL;DR
Existing 3D LMMs have difficulty providing fine-grained scene understanding and flexible interaction without complex auxiliary pipelines, while broader task generalization remains limited. 3D-LLaVA introduces an integrated point-cloud-only architecture centered on the multifunctional OST, and unified instruction tuning yields strong results across benchmarks, including 92.6% CiDEr on ScanQA. Its demonstrated scope still includes dependence on additional modules in related segmentation approaches, highlighting the broader challenge of precise, general 3D understanding.
Problem
Existing 3D LMMs rely on auxiliary prompt encoders or offline region processing, while flexible grounding and segmentation across tasks remain insufficiently supported.
Method
3D-LLaVA uses an integrated point-cloud-only architecture whose OST selects visual tokens, encodes visual prompts, and decodes referring 3D masks for an LLM.
Results
3D-LLaVA achieves state-of-the-art performance on five datasets, including 92.6% CiDEr on ScanQA, 4.9% above the previous best result.
Takeaways & Limitations
A versatile OST can unify 3D vision-centric dialogue, flexible prompting, and language-grounded point-cloud mask prediction within one architecture.
Takeaways & Limitations
Related referring-segmentation systems still depend on additional modules and have not demonstrated effectiveness across VQA and captioning.
Abstract
from arXiv · showhide
Current 3D Large Multimodal Models (3D LMMs) have shown tremendous potential in 3D-vision-based dialogue and reasoning. However, how to further enhance 3D LMMs to achieve fine-grained scene understanding and facilitate flexible human-agent interaction remains a challenging problem. In this work, we introduce 3D-LLaVA, a simple yet highly powerful 3D LMM designed to act as an intelligent assistant in comprehending, reasoning, and interacting with the 3D world. Unlike existing top-performing methods that rely on complicated pipelines-such as offline multi-view feature extraction or additional task-specific heads-3D-LLaVA adopts a minimalist design with integrated architecture and only takes point clouds as input. At the core of 3D-LLaVA is a new Omni Superpoint Transformer (OST), which integrates three functionalities: (1) a visual feature selector that converts and selects visual tokens, (2) a visual prompt encoder that embeds interactive visual prompts into the visual token space, and (3) a referring mask decoder that produces 3D masks based on text description. This versatile OST is empowered by the hybrid pretraining to obtain perception priors and leveraged as the visual connector that bridges the 3D data to the LLM. After performing unified instruction tuning, our 3D-LLaVA reports impressive results on various benchmarks.
1. Introduction
3D-LLaVA addresses the complexity and limited task flexibility of existing 3D LMMs with an integrated architecture centered on the multifunctional OST. It supports diverse 3D interaction and understanding tasks while achieving strong benchmark performance.
- 3D LMMs support applications requiring robust reasoning, precise 3D scene comprehension, and human-agent interaction.
- Existing systems often require extra prompt encoders, offline region proposals, or feature extraction, creating complex pipelines that complicate deployment and limit accessibility.
- 3D-LLaVA uses OST as a shared visual connector to unify interactive 3D dialogue and point-level scene comprehension without auxiliary modules or complicated steps.
- OST selects informative visual tokens, encodes visual prompts, and supports mask decoding using a superpoint representation.
- 92.6% CiDEr on ScanQA improves the previous best result by absolutely 4.9% CiDEr score after end-to-end instruction tuning.
2. Related Work
Prior 3D vision-language methods address grounding, referring segmentation, captioning, and question answering, but many struggle to generalize across tasks. 3D-LLaVA's framework combines point-cloud processing, visual prompts, language instructions, and mask generation for broader interaction.
- 3D vision-language understanding includes grounding, referring segmentation, captioning, and question answering.
- Earlier methods often achieve success on particular tasks but fall short of generalizing across different 3D understanding tasks.
- Unified and pretrained models improve diverse 3D tasks, yet task-specific heads and limited reasoning constrain broader general-purpose applications.
- The framework accepts point clouds, language instructions, and optional visual prompts, producing LLM text outputs and 3D masks through OST.
3. Approach
3D-LLaVA uses an integrated point-cloud-only architecture centered on the Omni Superpoint Transformer (OST), which connects 3D scene features with language while supporting visual-token selection, visual-prompt encoding, and referring-mask decoding. Its pipeline combines superpoint-based scene encoding, hybrid pretraining, joint instruction tuning, and selective parameter updates.
- 3D Scene Encoder: The scene encoder converts 3D points with coordinates and RGB values into voxels, applies a Sparse 3D U-Net, and pools features into superpoints.Superpoint pooling reduces the visual embedding count to hundreds or a few thousand while avoiding the information loss associated with farthest-point sampling.
- Omni Superpoint Transformer: OST uses superpoint features as queries, keys, and values, retaining correspondence while providing mask, classification, and alignment heads.Its mask head predicts binary masks, its classification head predicts categories, and its alignment head produces visual embeddings for the language interface.
- Details in Pipeline: OST selects top-K superpoints by objectness, encodes visual prompts through a parameter-free sampler, and projects visual and prompt embeddings into language tokens.Click prompts use three-nearest-neighbor interpolation, whereas boxes or masks use average pooling over points inside the prompt; masked attention prevents prompt features from influencing superpoints.
- Details in Pipeline: Instructions replace <PC> and <Visual Prompt> placeholders with visual and prompt embeddings, while frozen OST predicts referred-object masks from a concatenated segmentation query.Mask attention prevents information flow from the segmentation query to superpoints during referring segmentation.
- Training Scheme: Hybrid pretraining combines instance segmentation with 2D-to-3D knowledge distillation before joint instruction tuning across six 3D vision-language datasets.Instruction tuning jointly optimizes text generation and referring segmentation, updating the visual projector, SEG projector, and LoRA parameters while keeping the scene encoder, OST, and main LLM body frozen.
4. Experiments
3D-LLaVA is evaluated on ScanNet-based 3D vision-language benchmarks using unified instruction tuning, with metrics spanning question answering, captioning, and referring segmentation. It achieves strong results across tasks, including state-of-the-art performance and effective visual-prompt and grounding behavior.
- Datasets and Metrics: The experiments use ScanNet scans, with 1,201 training scenes and 312 validation scenes, while ScanNet200 annotations support visual-encoder pretraining.The reported evaluation follows pretraining and instruction-tuning stages on ScanNet-derived data.
- Datasets and Metrics: Evaluation covers 3D question answering, dense captioning, situated question answering, referring segmentation, and box-level visual grounding.Metrics include CiDEr, BLEU-4, METEOR, Rouge-L, extract-match accuracy, refined extract-match accuracy, and mIoU.
- Comparison with SoTA Models: 3D-LLaVA is the only compared model covering both text generation and point-level understanding across the benchmark categories.The comparison distinguishes specialist models, dataset-finetuned 3D LMMs, and unified multi-task 3D LMMs.
- Comparison with SoTA Models: 43.3% mIoU on ScanRefer and 42.7% mIoU on Multi3DRefer improve SegPoint’s previous best records by 1.6% and 6.6% mIoU, respectively.Multi3DRefer includes expressions referring to one, multiple, or zero objects; absent objects produce no segmentation token and background masks.
- Experimental Analysis: 51.2% accuracy at an IoU threshold of 0.25 demonstrates competitive box-level grounding from foreground masks, although Chat-Scene performs better.The boxes are obtained by filtering foreground masks with DBSCAN and taking minimum and maximum coordinates.
- Experimental Analysis: Reusing OST as the visual prompt encoder achieves the best visual-prompt result while avoiding additional learnable parameters and enabling deeper superpoint-feature abstraction.The ablation compares different prompt-embedding paradigms using boxes as visual prompts, with masks and clicking points represented as box variants.
5. Conclusion
3D-LLaVA combines a streamlined architecture with the multifunctional OST connector to support dialogue, visual interaction, and language grounding in 3D masks. The authors report strong results across multiple benchmarks while identifying 3D data collection and configuration as the next development challenge.
- 3D-LLaVA uses a streamlined architecture centered on the multifunctional Omni Superpoint Transformer for visual token selection, prompt encoding, and mask decoding.These capabilities support 3D vision-centric dialogue and grounding language expressions into 3D point-cloud masks.
- 3D-LLaVA enables flexible interaction and language grounding through a universal architecture.
- Extensive experiments report impressive results across multiple benchmarks.
- 3D data remains the main obstacle to developing 3D LMMs, making data collection and configuration the authors’ next step.