PaperlayerView exact revision on arXiv

Source-linked AI summary

Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning

Hanxun Yu, Wentong Li, Song Wang, Junbo Chen, Jianke Zhu

arXiv:2503.00513v2cs.CV

TL;DR

Existing 3D scene-understanding methods do not jointly capture fine-grained instance semantics, geometry, and spatial relations efficiently. Inst3D-LMM combines multimodal fusion, spatial-relation modeling, and end-to-end multitask tuning, achieving strong performance across grounding, understanding, and reasoning tasks.

  • Problem

    Prior methods do not jointly represent 3D instances’ geometric and semantic properties together with pairwise spatial relationships in a unified model.

  • Method

    Inst3D-LMM uses Multi-view Cross-Modal Fusion, 3D Instance Spatial Relation modeling, and end-to-end multi-task instruction tuning.

  • Results

    Inst3D-LMM outperforms previous state-of-the-art methods across 3D visual grounding, question answering, and dense captioning tasks.

  • Takeaways & Limitations

    The model provides a unified generalist framework for 3D scene understanding, grounding, and reasoning without task-specific fine-tuning.

Abstract

from arXiv · show

Despite encouraging progress in 3D scene understanding, it remains challenging to develop an effective Large Multi-modal Model (LMM) that is capable of understanding and reasoning in complex 3D environments. Most previous methods typically encode 3D point and 2D image features separately, neglecting interactions between 2D semantics and 3D object properties, as well as the spatial relationships within the 3D environment. This limitation not only hinders comprehensive representations of 3D scene, but also compromises training and inference efficiency. To address these challenges, we propose a unified Instance-aware 3D Large Multi-modal Model (Inst3D-LMM) to deal with multiple 3D scene understanding tasks simultaneously. To obtain the fine-grained instance-level visual tokens, we first introduce a novel Multi-view Cross-Modal Fusion (MCMF) module to inject the multi-view 2D semantics into their corresponding 3D geometric features. For scene-level relation-aware tokens, we further present a 3D Instance Spatial Relation (3D-ISR) module to capture the intricate pairwise spatial relationships among objects. Additionally, we perform end-to-end multi-task instruction tuning simultaneously without the subsequent task-specific fine-tuning. Extensive experiments demonstrate that our approach outperforms the state-of-the-art methods across 3D scene understanding, reasoning and grounding tasks. Source code is available at https://github.com/hanxunyu/Inst3D-LMM

1. Introduction

Inst3D-LMM addresses the limits of existing 3D LMMs by combining instance-level 2D semantics with 3D geometry and modeling scene-level spatial relations. It supports multiple 3D-language tasks through unified instruction tuning while improving reported efficiency and accuracy.

  • Motivation: Existing 3D scene methods target individual tasks, whereas LMMs can handle multiple 3D perception tasks within one model.The cited methods include 3D visual grounding, question answering, and dense captioning.
  • Approach: Inst3D-LMM generates instance-level representations that combine geometric and semantic properties with scene-level representations of pairwise object relations.This design uses 2D Vision Foundation Models and 3D specialist models to enrich instance features.
  • Training and efficiency: Inst3D-LMM tackles multiple 3D-language tasks without task-specific fine-tuning and uses minor token costs for the LLM.The framework is designed for end-to-end multi-task instruction tuning.
  • Reported outcome: Inst3D-LMM improves the LLM’s reported 3D-scene comprehension in both efficiency and accuracy.The paper reports this outcome across its proposed instance-aware methodology.
  • Approach: The MCMF module injects multi-view 2D features into coarse-semantic 3D features, while a 3D-ISR module captures pairwise spatial relationships among objects.MCMF uses a learnable [CLS] token to aggregate characteristics from each 2D view.

2. Related Works

Prior 3D language understanding methods commonly specialize in individual tasks or rely on closed-set models and task-specific fine-tuning. Inst3D-LMM instead presents a unified pipeline that combines multi-view semantic features, 3D instance information, and spatial relations for multi-task understanding.

  • 3D Scene Understanding with Language: 3D visual grounding localizes target objects from language queries, while 3D question answering and dense captioning require spatial reasoning or object localization and description.These tasks motivate language-driven 3D scene understanding.
  • 3D Scene Understanding with Language: Conventional methods typically focus on a specific task, while universal frameworks attempt to pre-align 3D scenes with corresponding textual descriptions.The cited universal frameworks include 3D-VLP and 3D-VisTA.
  • 3D Large Multi-modal Models: Most existing methods remain focused on closed-set scene understanding and require task-specific fine-tuning or specialized models.This contrasts with Inst3D-LMM’s generalist multi-task framing.
  • Inst3D-LMM pipeline: Inst3D-LMM extracts multi-view 2D semantic features and 3D proposals, then forms instance-level and relation-aware scene-level tokens before multi-task instruction tuning.The overview identifies MCMF as the instance-token module and 3D-ISR as the spatial-relation module.

3. Methodology

Inst3D-LMM builds instance-level and scene-level 3D representations by combining 3D geometry, multi-view 2D semantics, and pairwise spatial relations, then supports multiple tasks through end-to-end instruction tuning.

  • Instance-Level Feature Extraction: The framework extracts instance-level 3D features from segmented point-cloud instances and multi-view 2D semantic features from projected instance regions.It selects visible views, generates masks with sampled prompts, and produces multi-level image crops for feature extraction.
  • 3D Instance Spatial Relation: 3D-ISR computes pairwise distances and horizontal and vertical angles from instance centers to represent spatial relations among scene objects.These positional parameters form five-dimensional pairwise spatial features for the instance tokens.
  • 3D Instance Spatial Relation: Spatial-conditioned self-attention combines positional embeddings, pairwise spatial features, and instance tokens to encode scene-wide relations for each instance.The resulting attention map is integrated with instance-level visual tokens before scene-level aggregation.
  • 3D Instance Spatial Relation: A transformer encoder, max-pooling layer, and two-layer MLP generate relation-aware scene tokens alongside the final instance-level visual tokens.These representations are passed to the LLM for downstream processing.
  • End-to-End Multi-task Instruction Tuning: End-to-end multi-task instruction tuning updates MCMF, 3D-ISR, and the LLM while freezing the pretrained 3D object encoder, enabling multiple tasks without task-specific fine-tuning.The resulting model handles varied 3D vision-language tasks simultaneously.

4. Experiments

Experiments evaluate Inst3D-LMM on multiple 3D-language benchmarks and ablations of its fusion, spatial-relation, and efficiency designs. The model reports improvements across grounding, question answering, and dense captioning while reducing token counts.

  • Experimental Setting: Experiments use ScanNetv2, with 1,201 training scenes and 312 validation scenes, evaluating ScanRefer, Multi3DRefer, ScanQA, and Scan2Cap.The benchmarks cover single- and multi-object grounding, question answering, and dense captioning.
  • Main Results: +2.3% Acc@0.25 and +1.2% F1@0.25 over Chat-Scene on ScanRefer and Multi3DRefer, respectively.The generalist model achieves competitive performance against specialist and closed-set methods.
  • Main Results: 38.3% B-4@0.50 and 79.7% C@0.50 on Scan2Cap exceed 3D-VisTA by +4.3% and +12.8%, and LL3DA by +2.4% and +14.5%.These results concern dense-captioning localization and description.
  • Ablation Study: MCMF outperforms concatenation, parallel projection, and vanilla cross-attention on ScanRefer, ScanQA, and Scan2Cap.Adding multi-view 2D CLIP information also improves performance over using only 3D geometric features.
  • Ablation Study: 3D-ISR consistently surpasses prior spatial-relation methods and significantly improves grounding on ScanRefer and Multi3DRefer.Its position embeddings capture spatial relationships more accurately than sentence tokens used in ViL3DRel.
  • Ablation Study: Combining MCMF and 3D-ISR consistently outperforms either module alone, while distance benefits grounding, orientation benefits question answering, and both yield the best overall performance.MCMF reduces tokens to one per object proposal, and 3D-ISR uses eight tokens per 3D scene, lowering computational costs.

5. Conclusion and Limitations

Inst3D-LMM combines multi-view semantic fusion with spatial-relation modeling for 3D vision-language understanding and reasoning. The authors identify limited high-quality 3D-text data as a remaining barrier to embodied action control and note safety concerns around hallucinations.

  • Conclusion: Inst3D-LMM injects multi-view 2D semantic priors into 3D geometry features and uses spatial condition attention to capture pairwise relations.These modules generate fine-grained instance-level tokens and support understanding across multiple 3D vision-language tasks.
  • Limitations: Limited high-quality 3D-text datasets leave a gap between 3D LMM learning and real-world embodied action control, including robotic manipulation and navigation.The authors plan to scale diverse 3D vision and language data to improve reasoning and planning.
  • Limitations: Ethical safety concerns and potential hallucinatory outputs in LLM applications remain areas requiring attention.
Loading 2503.00513v2…