PaperlayerView exact revision on arXiv

Source-linked AI summary

LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences

Hongyan Zhi, Peihao Chen, Junyan Li, Shuailei Ma, Xinyu Sun, Tianhang Xiang, Yinjie Lei, Mingkui Tan, Chuang Gan

arXiv:2412.01292v2cs.CV

TL;DR

Large-scene 3D-VLMs face difficulty locating task-relevant visual information amid dense features, while existing benchmarks leave cross-room understanding underexplored. LSceneLLM uses LLM visual preferences to select and magnify relevant scene regions, and the authors report strong results across large-scene and existing benchmarks.

  • Problem

    In large scenes, task-agnostic object features can obscure task-relevant information and sparse inputs can lose small-object details, while cross-room understanding remains underexplored.

    Source: §1 · pp. 1–2

  • Method

    LSceneLLM uses LLM attention to select task-relevant regions, gathers dense visual details, and fuses them with coarse scene information.

    Source: §1 · p. 2

  • Results

    LSceneLLM reports superior performance on indoor and outdoor large-scene benchmarks and state-of-the-art results across single-room and large-scene benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    XR-Scene adds cross-room question answering, embodied planning, and scene-captioning tasks to large-scene 3D-VLM evaluation.

    Source: §1 · p. 2

Abstract

from arXiv · show

Research on 3D Vision-Language Models (3D-VLMs) is gaining increasing attention, which is crucial for developing embodied AI within 3D scenes, such as visual navigation and embodied question answering. Due to the high density of visual features, especially in large 3D scenes, accurately locating task-relevant visual information is challenging. Existing works attempt to segment all objects and consider their features as scene representations. However, these task-agnostic object features include much redundant information and missing details for the task-relevant area. To tackle these problems, we propose LSceneLLM, an adaptive framework that automatically identifies task-relevant areas by leveraging LLM's visual preference for different tasks, followed by a plug-and-play scene magnifier module to capture fine-grained details in focused areas. Specifically, a dense token selector examines the attention map of LLM to identify visual preferences for the instruction input. It then magnifies fine-grained details of the focusing area. An adaptive self-attention module is leveraged to fuse the coarse-grained and selected fine-grained visual information. To comprehensively evaluate the large scene understanding ability of 3D-VLMs, we further introduce a cross-room understanding benchmark, XR-Scene, which contains a series of large scene understanding tasks including XR-QA, XR-EmbodiedPlanning, and XR-SceneCaption. Experiments show that our method surpasses existing methods on both large scene understanding and existing scene understanding benchmarks. Plunging our scene magnifier module into the existing 3D-VLMs also brings significant improvement.

1. Introduction

Large-scene 3D-VLMs struggle to focus on task-relevant information and retain fine details, while cross-room understanding remains underexplored. LSceneLLM adaptively selects and magnifies relevant regions, and XR-Scene benchmarks cross-room tasks; the paper reports strong performance across large-scene and existing benchmarks.

1. Introduction

  • The paper frames scene-level 3D understanding as important for embodied tasks such as robotic manipulation, navigation, and long-horizon planning.It situates this work amid growing efforts to use LLM reasoning and summarization for understanding 3D point clouds.Source: §1 · p. 1
  • Existing object-centric methods use task-agnostic scene features, making task-relevant information hard to locate and risking the loss of small-object details through sparse inputs.Most 3D-VLM work focuses on object-level understanding, while large-scene and cross-room settings remain underexplored.Source: §1 · pp. 1–2
  • LSceneLLM uses coarse scene understanding and LLM attention to select task-relevant areas, extract dense details, and fuse them with coarse features.The framework is motivated by the idea that models, like people processing complex visual input, can first identify relevant regions and then inspect details.Source: §1 · p. 2
  • XR-Scene benchmarks cross-room understanding through question answering, embodied planning, and scene captioning tasks.Its average scene area is 132 m2, compared with 29 m2 for ScanQA.Source: §1 · p. 2
  • LSceneLLM demonstrates superior performance on indoor and outdoor large-scene benchmarks, and its scene magnifier significantly improves existing 3D-VLMs.The authors also report state-of-the-art performance across single-room and large-scene benchmarks.Source: §1 · p. 2

2. Related Work

Prior 3D-VLM work includes object-level models and object-centric scene representations, while task-relevant selection remains challenging in large scenes. Existing 3D understanding benchmarks largely cover small, single-room settings, motivating a cross-room benchmark based on HM3D.

2.1. 3D Vision Language Models

  • Object-centric scene methods detect or segment objects and model their spatial relationships, but task-agnostic features make relevant information difficult to locate among many objects.LL3DA uses learnable QFormer queries to summarize task-relevant object features, but this extraction occurs outside the LLM and is described as shallow and data-demanding.Source: §2.1 · pp. 2–3

2.2. 3D Understanding Benchmarks

  • Existing 3D understanding benchmarks are largely limited to small, single-room scenes, and the paper constructs a cross-room large-scene benchmark based on HM3D.HM3D is described as 1,000 high-resolution scans of building-scale residential, commercial, and civic spaces.Source: §2.2 · p. 3

3. LSceneLLM: Adaptive Framework For Large Scene Understanding

LSceneLLM combines a coarse scene overview with a scene magnifier that identifies task-relevant regions and adds fine-grained visual information. Its selector uses LLM attention to choose dense tokens, which adaptive self-attention fuses with the existing hidden states.

3.1. Overall Architecture

  • The architecture encodes a downsampled point cloud into sparse vision tokens, then extracts dense features near preferred-region centers for fine-grained tokens.The coarse module samples points and groups nearby features; local dense features are also processed through the SA module.Source: §3.1 · p. 3
  • The scene magnifier replaces self-attention in the LLM’s last NMA layers, adding selected dense vision tokens while retaining sparse vision and text tokens.The first NSA layers remain unchanged and use only sparse vision and text tokens to reduce computational complexity.Source: §3.1 · p. 4

3.2. Dense Vision Tokens Selector

  • The dense vision token selector uses LLM attention to identify regions of interest and select dense tokens rather than inputting all dense tokens.The selected dense tokens are incorporated in the later layers of the LLM.Source: §3.1 · p. 4, §3.2 · p. 4

3.3. Adaptive Self-attention Module

  • Adaptive self-attention fuses dense vision tokens with hidden states, while its attention map guides dense-token selection in the subsequent layer.Each text token interacts only with selected dense vision tokens, enforced by an attention mask.Source: §3.3 · p. 4, §3.1 · p. 4

4. XR-Scene: Cross-Room Scene Understanding Benchmark

XR-Scene is a cross-room benchmark proposed to evaluate large-scene understanding across question answering, embodied planning, and scene captioning. It is generated from HM3D and includes large scenes, object-diverse questions, and a fine-grained XR-QA subset.

4. XR-Scene: Cross-Room Scene Understanding Benchmark

  • XR-Scene evaluates cross-room scene understanding through XR-QA, XR-EmbodiedPlanning, and XR-SceneCaption.XR-QA requires locating queried objects and understanding their surrounding relationships; planning produces subtasks from high-level goals, while captioning describes scenes or rooms.Source: §4 · pp. 5–6
  • XR-Scene scenes are generated from HM3D using room positions to select nearby rooms, with SceneVerse annotations assisting QA-pair generation.GPT-4o is prompted with room annotations and a top-down view of the whole scene.Source: §4 · p. 5

4.1. Analysis of XR-Scene

  • XR-Scene contains more than 1,000 scenes averaging 132 m2, compared with 29 m2 for ScanQA [2].XR-QA has a more balanced distribution of object inquiries than ScanQA [2], and XR-QA-S uses a 0.05 m2 question-object bounding-box threshold to assess fine-grained understanding.Source: §4.1 · p. 6

5. Experiment

Across indoor and outdoor large-scene evaluations, LSceneLLM outperforms existing methods, while further analyses examine fine-grained understanding, module integration, ablations, and attention visualizations. Experiments use ScanNet and HM3D data, with several datasets and tasks represented in training and evaluation.

5.1. Datasets, Metrics and Implementation Details

  • Experiments use ScanNet and HM3D data, and evaluate textual responses with CiDEr, METEOR, Rouge, and accuracy.The implementation samples 40k points per scene and uses a fully fine-tuned Llama2-7b backbone; baselines are trained on the same data for comparison.Source: §5.1 · p. 6

5.2.1 Indoor Scene Understanding

  • LSceneLLM outperforms existing methods by a large margin across XR-QA, XR-SceneCaption, and XR-EmbodiedPlanning.The authors attribute prior methods’ weaker results to difficulty focusing amid many objects or to neglect of region–object dependencies.Source: §5.2.1 · p. 6
  • On ScanQA, LSceneLLM achieves optimal performance with limited training data on small-scale scenes.The authors present this as further validation of their hierarchical scene understanding approach.Source: §5.2.1 · pp. 6–7

5.2.2 Outdoor Scene Understanding

  • LSceneLLM achieves state-of-the-art NuscenesQA accuracy across generative methods, increasing accuracy by 3.0 over the previous best method.This result does not require multi-view image input or specialized framework design.Source: §5.2.2 · p. 7, Table 3 · p. 7

5.3. More Insights into Fine-Grain Large Scene Understanding

  • On XR-QA-S, LSceneLLM performs best and exceeds existing methods on CIDEr by a large margin.The authors connect other methods’ weaker small-object performance to detail loss during downsampling.Source: §5.3 · p. 7, Table 4 · p. 7

5.4. Module Plug-and-Play Analysis

  • Adding the scene magnifier to Ll3da improves XR-QA by 4.44 and XR-QA-S by 11.09, while Leo’s improvement is smaller.The authors attribute the comparatively smaller gains to Ll3da’s Qformer detail loss and Leo’s lack of a coarse scene understanding module.Source: §5.4 · pp. 7–8, Table 4 · p. 7

5.5. Ablation Studies

  • Using LLM attention to select areas of interest and activating 10%-20% of those areas yields the best XR-QA performance in the reported ablations.The ablations examine vision-token counts, selection strategy and threshold, and interaction between dense and sparse vision tokens.Source: Table 5 · p. 7, §5.5 · p. 8

5.6. Qualitative Analysis

  • In qualitative examples, LSceneLLM locates a magazine and small objects more accurately than Ll3da and Leo.For the magazine question, the other methods roughly focus on its location but answer incorrectly; for small objects, they fail to locate them accurately.Source: §5.6 · p. 8

6. Conclusion

The paper proposes an adaptive framework for locating task-relevant areas and magnifying their fine-grained details, and introduces XR-Scene for cross-room evaluation. The authors report significant improvements on large-scene benchmarks.

  • LSceneLLM uses LLM visual preferences to identify task-relevant areas and a plug-and-play scene magnifier to capture fine-grained details in focused areas.Source: §6 · p. 8
  • The authors report significant performance improvements on large 3D scene understanding benchmarks.Source: §6 · p. 8
  • XR-Scene is introduced as a cross-room understanding benchmark to extend evaluation of 3D-VLMs in large-scale environments.Source: §6 · p. 8

A. More Details on Generation of XR-Scene

XR-Scene generation selects nearby rooms from HM3D and uses their annotations to form cross-room QA and planning tasks. Its captioning and planning generation uses room-specific top-down views and GPT-4o prompts.

A. More Details on Generation of XR-Scene

  • Cross-room scenes are formed by selecting a random HM3D room and including its nearest N rooms by Euclidean distance between room centers.The scenes use HM3D room positions and SceneVerse annotations for room objects and relationships.Source: A. More Details on Generation of… · p. 11
  • XR-QA questions cover object properties and spatial relationships, generated from annotations after filtering objects that appear only once in the scene.GPT-4 generates the two question types for each retained annotated object.Source: A. More Details on Generation of… · p. 11

XR-Planning and XR-EmbodiedPlanning Generation

  • Cross-room planning and captioning require room–object relations and spatial understanding, including captions for whole scenes or specific rooms.GPT-4o is prompted with top-down views and room bounding boxes; a rendered RGB top-down view is also provided to reduce hallucinations.Source: XR-Planning and XR-EmbodiedPlanning… · p. 11

B. Ablation Study

The attention-confidence threshold ablation finds that a threshold of 96 selects 10%-20% of vision tokens for interaction with fine-grained scene features. Performance is weaker when selection includes too many or too few tokens.

  • A threshold of 96 performs best, selecting 10%-20% of vision tokens to interact with corresponding fine-grained scene features.Source: B. Ablation Study · p. 11
  • Selecting too many tokens makes it harder for the model to focus accurately on local areas.Source: B. Ablation Study · p. 11
  • Selecting too few tokens provides insufficient fine-grained scene information and limited help in understanding the scene.Source: B. Ablation Study · p. 11

Numbers of Dense Vision Token Interact With Sparse

The token-count ablation favors four dense vision tokens per sparse token, while attention-based selection outperforms random selection. Increasing token count alone helps less than adding the LSceneLLM module.

Numbers of Dense Vision Token Interact With Sparse

  • Four dense vision tokens per sparse vision token provide the greatest benefit in the local-region representation ablation.The experiment samples dense point-cloud features around each sparse token’s center point and aggregates them.Source: Numbers of Dense Vision Token Interact… · p. 11
  • Using four times as many vision tokens improves performance, but less than incorporating the LSceneLLM module.The comparison is reported in Table 7; the passage gives no metric values.Source: Numbers of Dense Vision Token Interact… · pp. 11, 13
  • Attention-weight-based dense-token selection outperforms random selection, while random selection provides little benefit.The ablation tests whether the self-attention map reflects the visual information used to answer questions.Source: Numbers of Dense Vision Token Interact… · p. 13

C. More Scene Understanding Results on ScanNet

On ScanNet-based scene captioning, embodied planning, and embodied QA, LSceneLLM outperforms current methods on most metrics. Its XR-QA attention visualizations show adaptive visual preferences locating task-relevant features.

C. More Scene Understanding Results on ScanNet

  • LSceneLLM outperforms current state-of-the-art approaches on most metrics across scene captioning, embodied planning, and embodied QA.The tasks use ScanNet data from 3D-LLM, organized by Ll3da; embodied QA takes the agent’s position and orientation into account.Source: C. More Scene Understanding Results on… · p. 13
  • These ScanNet tasks require holistic understanding of the entire scene, including agent-perspective QA informed by position and orientation.The datasets are sourced from the ScanNet part of 3D-LLM and organized by Ll3da.Source: C. More Scene Understanding Results on… · p. 13

D. More Attention Visualization of LSceneLLM on XR-QA

E. Computational Complexity Analysis

LSceneLLM uses a fixed number of visual tokens as scenes grow, placing its computational complexity between Ll3da and Leo. Unlike the object-centric baseline, it can maintain constant complexity while preserving scene details as scene size increases.

E. Computational Complexity Analysis

Loading 2412.01292v2…