PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-SceneQ: Empowering 3D LLM With Query-Guided Adaptive Pruning and Multi-Modal Feature Enhancement

Yongqi Shan, Lu Zhang, Jiazuo Yu, Yunzhi Zhuge, Huchuan Lu

arXiv:2610.37202v1cs.CV

TL;DR

Existing 3D-LLMs may include many task-irrelevant object tokens and rely on point clouds that lack rich semantic cues. 3D-SceneQ combines query-guided object pruning with 2D semantic feature integration, and reports state-of-the-art performance across several 3D understanding tasks.

  • Problem

    Existing 3D-LLMs may include many irrelevant object tokens and rely on point clouds that lack rich semantic cues such as color, material, and high-level context.

    Source: §I · pp. 1–2

  • Method

    3D-SceneQ combines question-guided adaptive object-token pruning with object-level integration of projected 2D semantic features and 3D representations.

    Source: §I · p. 2

  • Results

    78.1 CIDEr on Scan2Cap, 104.7 CIDEr on ScanQA, and 67.0 EM@1 on SQA3D are reported as state-of-the-art results across three 3D understanding tasks.

    Source: §I · p. 2

  • Takeaways & Limitations

    The reported results span 3D captioning, 3D question answering, and situated question answering, supporting the authors' claim of strong reasoning capacity in complex 3D scenes.

    Source: §I · p. 2

    3D-SceneQ is optimized for indoor scenes, has limited generalization to outdoor settings, and lacks explicit temporal modeling for dynamic scenes with moving agents.

    Source: §V · pp. 10–11

Abstract

from arXiv · show

Current 3D scene understanding pipelines typically concatenate vast numbers of 3D object tokens with text tokens and feed the resulting sequence to a Large Language Model (LLM). However, existing 3D-LLMs for scene understanding encounter two major limitations: (i) excessive inclusion of task-irrelevant object data, introducing noise that reduces reasoning accuracy and increases hallucinations; and (ii) reliance solely on point cloud data, which inherently lacks the rich semantic information available in complementary 2D modalities, such as color, material properties, texture, and high-level contextual relationships. To address these challenges, we introduce 3D-SceneQ, a 3D LLM that unifies adaptive token pruning with multimodal semantic enrichment, markedly advancing scene understanding, reasoning, and grounding. Specifically, we propose a Query-Guided Adaptive Pruning (QGAP) module that adaptively selects task-relevant objects based on the intended meaning of user instructions, leveraging a learnable latent query to retain critical information while effectively suppressing irrelevant noise. In addition, we introduce a Multi-modal Object-level Feature Enhancement (MOFE) module, which integrates 2D feature embeddings from pre-trained models into 3D representation to enrich object-level semantic information. Evaluations show that 3D-SceneQ outperforms current state-of-the-art methods across multiple benchmarks (e.g., +5.7 CIDEr on Scan2Cap and +17.0 EM@1 on SQA3D), demonstrating its remarkable capabilities in various 3D language reasoning tasks.

I. INTRODUCTION

3D-SceneQ addresses task-irrelevant object-token noise and missing 2D semantic cues with adaptive pruning and multimodal feature integration. It reports state-of-the-art results across 3D captioning, question answering, and situated question answering.

I. INTRODUCTION

  • Existing 3D-LLMs concatenate text and many 3D object tokens, although only a small subset may be task-relevant and point clouds lack rich semantic cues such as color, material, and contextual relationships.The text identifies these issues as sources of noise and limitations on spatial-semantic reasoning.Source: §I · pp. 1–2
  • QGAP uses a learnable latent query to vary how many objects are retained for a question, suppressing irrelevant tokens to mitigate hallucinations and improve reasoning accuracy.MOFE projects multi-view 2D features into 3D space, pools them into object-level embeddings, and fuses them with 3D tokens through cross-attention.Source: §I · p. 2
  • MOFE enriches task-relevant 3D objects with semantic features from multi-view images processed by pre-trained models, complementing point-cloud geometry.The contribution summary describes the goal as richer object-level semantics for deeper scene understanding and robust reasoning.Source: §I · p. 2
  • 78.1 CIDEr on Scan2Cap, 104.7 CIDEr on ScanQA, and 67.0 EM@1 on SQA3D are reported as state-of-the-art results across captioning, question answering, and situated question answering.The authors describe the results as evidence of generalization and reasoning capacity in complex 3D scenes.Source: §I · p. 2

II. RELATED WORKS

Prior work progresses from point-cloud representation learning and task-specific 3D-language systems to multimodal language models and 3D-LLMs. The paper identifies semantic sparsity and structural redundancy in point clouds as unresolved challenges for downstream reasoning.

A. Point Cloud Understanding

  • Point-cloud methods including PointCLIP, CLIP2Point, ULIP, and OpenShape align 3D data with image or language representations, but remain constrained to object-level tasks and struggle with scene-level understanding.The subsection also describes PointGT and DT-Net as foundational point-cloud representation-learning architectures.Source: A. Point Cloud Understanding · p. 2
  • 3D question answering, visual grounding, and captioning are often handled by task-specific methods, while 3D-LLMs unify these tasks and can also support dialogue and task decomposition.The passage notes that task-specific systems struggle to generalize beyond their training distributions.Source: A. Point Cloud Understanding · p. 2

B. Multi-Modal Large Language Models

  • Multimodal LLMs bridge language with images, video, and audio using aligned representations, with approaches including CLIP, BLIP-2, and gated cross-attention models such as Flamingo.The described applications include visual question answering, image captioning, and multimodal instruction following.Source: 3D-SceneQ · p. 3

C. 3D Multi-Modal Large Language Models

  • 3D-LLMs extend multimodal models through point-cloud inputs, feature alignment, and cross-modal fusion, but point clouds remain semantically sparse and structurally redundant.The review notes that LEO uses visual inputs primarily as low-level control signals rather than integrating high-level image semantics into a unified 3D representation.Source: C. 3D Multi-Modal Large Language Models · pp. 3–4

III. METHODOLOGY

3D-SceneQ uses a two-stage learning scheme: first aligning 3D object features with language, then instruction-tuning with pruned and multimodally enhanced object tokens. The resulting second-stage sequence is processed by the LLM to generate responses.

A. 3D-SceneQ

  • The first stage aligns 3D object features with linguistic representations, and the second fine-tunes the model with task-specific instructions for 3D understanding and reasoning.The first stage uses PointNet++ [12] and a Spatial Transformer to produce object-centric tokens, then projects them to the LLM-compatible feature dimension.Source: A. 3D-SceneQ · p. 4
  • The first-stage input combines system messages, the user's question, and 3D object tokens, which the pretrained LLM processes to generate a response.The 3D representation contains N object tokens with feature dimensionality compatible with the LLM.Source: A. 3D-SceneQ · p. 4
  • Instruction tuning refines 3D features through QGAP pruning and MOFE enrichment, producing enhanced object tokens for the LLM input sequence.K denotes the number of objects remaining after QGAP pruning.Source: A. 3D-SceneQ · p. 4

B. Query-Guided Adaptive Pruning

QGAP ranks scene objects against the user’s question and adaptively determines how many to retain. A learnable latent query helps set the filtering ratio, and the highest-ranked objects proceed downstream.

C. Multi-modal Object-level Feature Enhancement

MOFE projects multi-view image features onto 3D points, pools them by object, and fuses the resulting object-level features with 3D representations. The fused features are passed to the LLM for multimodal scene understanding.

A. Experimental Setups

Experiments use captioning, referring-expression, scene-summary, question-answering, and instruction-tuning data, with standard generation and QA metrics. The setup combines a frozen pretrained backbone with parameter-efficient fine-tuning and lightweight proposed modules.

A. Experimental Setups

  • Alignment training uses object captions, object-in-context referring expressions, and scene-level summaries; instruction tuning covers captioning and question answering on ScanNet and 3RScan.The QA tasks ask about object properties, locations, and relationships.Source: A. Experimental Setups · p. 6
  • Language generation is evaluated with CIDEr, BLEU, METEOR, and ROUGE; open-ended generation also uses sentence similarity, while factual QA uses EM@1.EM@1 is the proportion of predictions exactly matching ground truth.Source: A. Experimental Setups · p. 6
  • The language backbone is Vicuna-7B, initialized from LEO’s first-stage model; the 3D encoder, MOFE segmentation model, tokenizer, backbone, and QGAP BERT encoders remain frozen.LoRA with rank 16 fine-tunes the language backbone; QGAP, MOFE, feature mapping, and the spatial transformer are trained from scratch for 10 epochs using AdamW at 3 × 10^-5.Source: A. Experimental Setups · p. 6
  • 3D-SceneQ has about 7 billion parameters, with 122.6 million tunable; QGAP contributes 7.1 million parameters and MOFE 70.3 million.The prompt combines a role description, scene-object context, and user instructions.Source: A. Experimental Setups · p. 6, Table II · p. 6

B. Comparison with State-of-the-Arts

3D-SceneQ achieves state-of-the-art performance across Scan2Cap, ScanQA, and SQA3D. The results are reported for a unified LLM framework augmented with QGAP and MOFE.

B. Comparison with State-of-the-Arts

  • On Scan2Cap, METEOR reaches 30.5 and ROUGE reaches 65.3, exceeding the previous best by 2.6 and 7.2 points, respectively.On ScanQA, CIDEr is 104.7 and BLEU-4 is 13.6.Source: B. Comparison with State-of-the-Arts · p. 6
  • The unified LLM-based framework uses lightweight QGAP and MOFE components rather than task-specific architectures.The authors report flexible adaptation to diverse 3D reasoning tasks and competitive quantitative performance.Source: B. Comparison with State-of-the-Arts · p. 6
  • Across Scan2Cap, ScanQA, and SQA3D, 3D-SceneQ consistently achieves state-of-the-art performance; on the SQA3D test set, EM@1 is 67.0%, 17 points above the strongest prior method.Source: B. Comparison with State-of-the-Arts · p. 6

1) Effectiveness of Core Components:

Ablations compare random pruning, query-guided pruning, and multimodal feature enhancement. Fixed-ratio filtering degrades as pruning increases, whereas QGAP adapts retention to query demands.

1) Effectiveness of Core Components:

  • Random pruning raises SQA3D EM@1 from 50.0 to 68.9 but drops Scan2Cap CIDEr from 72.4 to 52.3.The authors attribute the trade-off to indiscriminate pruning removing salient objects and attributes.Source: 1) Effectiveness of Core Components · p. 7
  • QGAP raises Scan2Cap CIDEr to 78.4 and SQA3D EM@1 to 70.1 by conditioning its object-token filtering ratio on the user’s question.The passage says this targeted sparsification outperforms both Baseline and RandFilter.Source: 1) Effectiveness of Core Components · p. 7
  • +5.7 CIDEr on Scan2Cap and +20.0 EM@1 on SQA3D over Baseline accompany the full model’s combined QGAP and MOFE strategy.The authors report consistent improvements across all benchmarks and describe a compact, information-dense scene representation.Source: 1) Effectiveness of Core Components · p. 7
  • MOFE raises Scan2Cap CIDEr to 77.8 and SQA3D EM@1 to 66.7 by fusing geometric, visual, and linguistic signals into object embeddings.Its ScanQA improvement is described as particularly pronounced, with cross-modal representations helping resolve spatial references and attribute dependencies.Source: 1) Effectiveness of Core Components · p. 7

2) Impact of Filtering Strategies and Ratios in QGAP:

3) Robustness of MOFE to Sensor Calibration and Registration Errors:

The section reports robustness under simulated camera noise and benefits of joint training, alongside qualitative analyses of query-adaptive filtering, hallucination reduction, and unseen-scene generalization.

D. Visual Analysis

  • 3D-SceneQ improves color perception and gives more spatially justified placement recommendations than the baseline in the qualitative comparisons.The examples include correctly identifying a brown chair and green plants, and grounding a recommendation in room geometry.Source: Figure 5 · p. 9, D. Visual Analysis · p. 9

3) Robustness of MOFE to Sensor Calibration and Registration Errors:

  • 78.0 (mild) and 77.4 (severe) Scan2Cap CIDEr remain close to clean 77.7; SQA3D EM@1 is 68.7 and 68.1 versus 66.7.MOFE pools 2D features across full object masks; the authors note that evaluation datasets have camera poses more accurate than the simulated noise.Source: 3) Robustness of MOFE to Sensor… · pp. 8–9

4) Influence of Training Setups:

  • ScanNet-only training underperforms the full model and struggles on 3RScan cross-domain tasks, while 3RScan-only has limited zero-shot generalization to ScanNet.The authors report that joint training broadens scene and task coverage and improves cross-domain generalization.Source: 4) Influence of Training Setups · p. 9

D. Visual Analysis

  • 3D-SceneQ avoids absent-object inventions, preserves object attributes, and distinguishes lookalikes where the baseline hallucinates or mislabels them.Examples include nonexistent objects, altered shapes or materials, and confusion between a whiteboard and bulletin board.Source: Figure 7 · p. 10, D. Visual Analysis · p. 10
  • For the generic instruction to tidy and arrange the room, QGAP removes 11% of object tokens; query-specific pruning rises to 18% for refrigerator and 22% for furnishings, matching intent.The refrigerator query removes windows and walls; the furnishings query removes plants and pillows.Source: D. Visual Analysis · pp. 9–10
  • On unseen Matterport3D scenes, zero-shot results include salient-object identification, material and shape awareness, cluttered-scene grounding, and color perception.The dataset has no language annotations for captioning or QA, so the authors manually construct prompts.Source: D. Visual Analysis · p. 10

V. LIMITATIONS AND FUTURE WORK

3D-SceneQ performs strongly across 3D understanding benchmarks, but its scope is constrained by indoor optimization, limited robustness in dynamic scenes, and reasoning depth without post-training signals.

V. LIMITATIONS AND FUTURE WORK

  • Outdoor generalization is limited because 3D-SceneQ is optimized for indoor scenes.Future work proposes more general-purpose encoders and balanced indoor–outdoor datasets to narrow the performance gap.Source: §V · pp. 10–11
  • Qualitative Matterport3D comparisons depict out-of-distribution scene descriptions, material and shape perception, grounding, and color sensitivity.The figure also shows examples of salient-object identification and description.Source: Figure 8 · p. 11
  • Without explicit temporal modeling, robustness decreases in dynamic scenes with moving agents; reasoning depth is constrained without post-training signals.The authors propose temporal modeling and post-training signals, including chain-of-thought supervision and reinforcement learning, as future directions.Source: §V · p. 11

VI. CONCLUSION

  • 3D-SceneQ consistently outperforms state-of-the-art approaches across benchmark tasks in captioning, question answering, and task planning.The authors describe the results as evidence of promising generalization and enhanced reasoning in challenging 3D environments.Source: §VI · p. 11
  • The authors position 3D-SceneQ as a foundation for future research and applications in intelligent interaction with complex 3D environments.The conclusion frames the system as a 3D LLM for scene understanding, reasoning, and planning.Source: §VI · p. 11

Related papers

Loading 2610.37202v1…