PaperlayerView exact revision on arXiv

Source-linked AI summary

Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning

Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, Wenhan Xiong

arXiv:2403.11401v2cs.CVcs.AI

TL;DR

Existing visual-language models have limited persistent 3D spatial understanding for indoor-scene tasks, while many 3D approaches primarily handle static scenes. Scene-LLM combines hybrid 3D features, scene-level and egocentric information, and LLM reasoning, achieving strong results across 3D question answering and interactive planning.

  • Problem

    Existing visual-language models have limited persistent 3D spatial understanding, and recent 3D approaches primarily address static scenes rather than interactive planning.

  • Method

    Scene-LLM uses a hybrid point-voxel representation with state updates, integrates scene-level and egocentric 3D information, and aligns features through a lightweight projection layer using 3D frame data.

  • Results

    Scene-LLM achieves state-of-the-art results on ScanQA and SQA3D without additional fine-tuning and outperforms other LLM-based models on Alfred when fine-tuned.

  • Takeaways & Limitations

    Scene-LLM provides a 3D-visual-language model for reasoning in interactive 3D scenes, supporting scene changes and interactive planning in dynamic environments.

  • Takeaways & Limitations

    The approach assumes spatial coordinates are retained to partially provide spatial-coordinate understanding.

Abstract

from arXiv · show

This paper introduces Scene-LLM, a 3D-visual-language model that enhances embodied agents' abilities in interactive 3D indoor environments by integrating the reasoning strengths of Large Language Models (LLMs). Scene-LLM adopts a hybrid 3D visual feature representation, that incorporates dense spatial information and supports scene state updates. The model employs a projection layer to efficiently project these features in the pre-trained textual embedding space, enabling effective interpretation of 3D visual information. Unique to our approach is the integration of both scene-level and ego-centric 3D information. This combination is pivotal for interactive planning, where scene-level data supports global planning and ego-centric data is important for localization. Notably, we use ego-centric 3D frame features for feature alignment, an efficient technique that enhances the model's ability to align features of small objects within the scene. Our experiments with Scene-LLM demonstrate its strong capabilities in dense captioning, question answering, and interactive planning. We believe Scene-LLM advances the field of 3D visual understanding and reasoning, offering new possibilities for sophisticated agent interactions in indoor settings.

1 Introduction

Scene-LLM targets interactive indoor-scene reasoning by combining egocentric and scene-level 3D information with LLMs. Its hybrid representation, lightweight projection, frame-based alignment, and paired dataset support broad understanding and planning tasks.

  • Core approach: Scene-LLM combines egocentric information for localization and immediate updates with scene-level information for persistent, map-reliant planning.The two information sources jointly support interactive planning that requires both local state and comprehensive scene context.
  • Training: Ego-centric 3D frames in camera and world coordinates are used during projection-layer fine-tuning to improve alignment of small-object details.The training strategy first uses conceptual annotations with the LLM frozen, then fine-tunes with instructional-following annotations and the LLM.
  • Results: Scene-LLM achieves state-of-the-art results on 3D-VQA and interactive-planning benchmarks while supporting applications from dense captioning to interactive planning.The introduction specifically reports state-of-the-art performance on 3D-VQA and interactive planning benchmarks.
  • Core approach: Its hybrid 3D representation preserves fine-grained spatial information, supports state changes, and can be incorporated into an LLM through a lightweight projector.The representation addresses dense 3D feature alignment while keeping the interface to the pretrained language model simple.
  • Data and applications: 190k egocentric 3D-visual-language pairs and about 500k scene-level pairs form the large-scale alignment dataset.The dataset is designed to provide both frame-based and scene-level supervision.

2 Related Works

Related work spans 3D-language tasks, interactive indoor planning, and several classes of 3D scene representation. Scene-LLM is positioned at their intersection, combining 3D scene understanding and reasoning with linguistic modalities.

  • 3D-language tasks: Prior 3D-language benchmarks cover scene synthesis, captioning, grounding, 3D-VQA, interactive planning, and visual-language navigation.These tasks investigate different forms of language interaction with 3D scenes.
  • Interactive planning: Interactive indoor-planning methods map language to actions, use hierarchical strategies, or employ LLMs for concept generation, direct planning, and embodied-data fine-tuning.Hierarchical approaches separate high-level strategy from low-level execution.
  • 3D scene representations: Object-centric representations are intuitive and efficient but may overlook dense information and require additional design for tracking interactive objects.Their limitations motivate alternatives for dynamic indoor environments.
  • 3D scene representations: Dynamic graphs efficiently track scene changes but have complex topologies that make intuitive integration with LLMs difficult.They represent a different trade-off from object-centric and dense representations.
  • 3D scene representations: Dense representations include spatial maps, voxel grids, point-wise formats, and field-like formats, while emerging 3D-VLMs connect such representations with language.The related work frames Scene-LLM within expanding 3D visual-language research.

3 3D-Visual-Language Data Generation

The dataset generation pipeline creates complementary egocentric frame-language and scene-level language data from indoor environments. A two-stage process uses a 2D VLM for frame descriptions and an LLM for scene instruction-following data, which then supports Scene-LLM’s projection bridge.

  • Dataset: The dataset covers about 9,000 indoor scenes from real scans, HM3D single rooms, and expert-designed iTHOR scenes.These sources provide varied indoor environments for 3D-visual-language alignment.
  • 3D frame-language generation: Frame data is generated by rendering multiple camera views, capturing RGB and ground-truth depth, and projecting pixels into spatial coordinates as 3D point sets.Each resulting point set is called a 3D frame.
  • Scene-LLM bridge: Scene-LLM’s bridge concatenates tokenized 3D semantic features with coordinates and projects them into textual feature space.This architecture follows the two-stage data-generation process with a simple projection layer.
  • 3D frame-language generation: Approximately 100k 3D frames yield around 190k frame-language pairs with captions and object descriptions generated by MiniGPT-V2.These pairs provide fine-grained concept understanding and egocentric awareness.
  • 3D scene-language generation: Scene-level data aggregates 3D frames using camera poses, preserving feature consistency between frame and scene representations.The scene data supplies broader contextual information than individual frames.
  • 3D scene-language generation: Llama-2-chat-70B generates scene annotations from frame captions, object descriptions, object lists, and bounding boxes for diverse instructional data.The resulting supervision targets scene-level understanding and instruction following.

4 Scene-LLM

Scene-LLM aligns dense 3D visual features with an LLM while combining scene-level and egocentric information for interactive reasoning and updates.

  • 3D Visual Feature: Scene-LLM extracts semantic 3D point features by aggregating pixel-wise CLIP features into a point set.
  • 3D Visual Feature: Its hybrid point-voxel representation balances dense spatial information, interactive updates, and manageable token lengths for LLM input.Visible voxels provide the visual tokens after voxelization and KNN-based point clustering.
  • 3D Visual Feature: The model updates scene features by replacing voxel features where newly rendered frame data is visible, keeping scene semantics synchronized with state changes.
  • 3D-Visual-Language Alignment: A linear projection layer maps 3D visual features into the LLM’s feature space, with joint fine-tuning of the projection layer and LLM.
  • Inference: ScanQA and SQA3D evaluations compare task-specific and non-task-specific settings across benchmark metrics and question types.
  • Inference: Interactive inference first describes the current egocentric frame, then updates scene features and generates a response from the updated scene and instructions.The instruction includes the current frame description, task description, completed steps, and a next-step identifier.

5 Experiments

Experiments evaluate Scene-LLM on 3D visual question answering and interactive planning, using benchmark comparisons, ablations, and resolution studies. Results support the model’s effectiveness across tasks and indicate that spatial resolution matters more than view count for QA performance.

  • Interactive Planning: Scene-LLM performs best among goal-instruction-only methods on ALFRED, with the strongest high-level planning accuracy across methods.Table 3 reports success rate, goal-conditioned success rate, and high-level planning accuracy on seen and unseen splits.
  • Benchmarks: Scene-LLM is evaluated on ScanQA, SQA3D, and ALFRED, covering 3D question answering and interactive planning.ScanQA and SQA3D assess 3D scene understanding; ALFRED evaluates planning from high-level goals in interactive environments.
  • Benchmark Results: Scene-LLM achieves strong ScanQA performance over competing methods, including without task-specific fine-tuning.The evaluation reports superiority on most metrics and notes particular strength on “What” and “Which” question types.
  • Ablation Studies: Pretraining substantially improves performance, while frame data matches or slightly exceeds scene data during pretraining.The ablation attributes the pretraining benefit to concept alignment and associates frame-data effectiveness with richer conceptual content.
  • Resolution Analysis: Higher voxel resolution improves QA performance, whereas increasing the number of views produces only a minor EM increase.The experiments identify voxel resolution as the main bottleneck in scene feature extraction under the tested settings.
  • Ablation Studies: Removing egocentric information or scene-state updates reduces high-level planning accuracy, supporting both components in interactive inference.The reported decline is associated with egocentric information’s role in representing the current state and with the value of information updates.

6 Conclusion

The conclusion presents Scene-LLM as a 3D-visual-language model combining egocentric and geo-centric understanding for interactive 3D scenes. It reports strong performance in 3D VQA and interactive planning while identifying input length, dynamic-scene processing, geometry, and hallucination limitations.

  • Contribution: Scene-LLM combines egocentric and geo-centric 3D spatial understanding for reasoning in interactive 3D scenes.Its hybrid representation captures comprehensive spatial information and supports dynamic state updates.
  • Results: The model achieves top-tier performance in 3D visual question answering and interactive planning.The conclusion links these results to Scene-LLM’s integrated 3D spatial understanding.
  • Limitations: Scene-LLM remains limited by LLM input-token length, dynamic scenes without a state detector, missing geometry features, and language hallucinations.These limitations define important boundaries for processing dense, changing indoor environments.
  • Conclusion: The paper positions Scene-LLM as an advancement toward more complex agent interactions in indoor settings.

Supplementary for Scene-LLM

The supplementary material extends the main paper with additional examples, benchmark results, and ablation studies for Scene-LLM.

  • Supplementary Content: The supplement provides extended interactive-planning examples and detailed dense-caption benchmark results.
  • Supplementary Content: Additional ablations examine modalities, frame-data usage, and related design choices.

7 Ablation Studies comparing different modalities.

The modality ablations compare text, video, bird’s-eye-view, and 3D inputs on ScanQA and SQA3D, alongside analyses of scene and frame data. Results favor 3D inputs overall and indicate that frame data provides greater lexical and conceptual diversity.

  • Modality Comparisons: The modality study compares textual, video, bird’s-eye-view, and 3D point-set inputs on ScanQA and SQA3D.Text experiments include zero-shot, scene-data fine-tuning, and task-specific tuning.
  • Modality Comparisons: Video and bird’s-eye-view modalities outperform text on ScanQA but underperform it on SQA3D, while both underperform 3D data on both benchmarks.The authors suggest spatial downsampling may be less suitable for video and image patches than for 3D data.
  • Concept Diversity: Frame data contains more unique words and nouns than scene data, indicating greater lexical and conceptual diversity.The comparison uses captions from 200,000 scene examples and 190,000 frame examples, excluding stopwords for noun counts.
  • Concept Diversity: Training on frame data generates more nouns during inference, suggesting broader fine-grained conceptual coverage.The output comparison uses 142 validation scenes from ScanNet V2.

9 Ablation Studies comparing convergence speed.

Frame data converges faster than scene data during projection-layer training. The supplied benchmark passages also report strong task-specific results for dense captioning and 3DMV-VQA.

  • Convergence Speed: 190,000 textual annotations accompany frame data, while scene data contains 200,000 textual annotations.
  • Convergence Speed: Frame data produces a quicker reduction in training loss than scene data during the first 6,000 pretraining iterations.The comparison concerns concept alignment during projection-layer pretraining.
  • Dense Captioning: Scene-LLM outperforms the compared methods across all evaluated Scan2Cap dense-captioning metrics after task-specific fine-tuning.The comparison includes a modular method and another 3D-visual-language model.

11 Result on 3DMV-VQA.

Scene-LLM is evaluated on a single-room 3DMV-VQA subset using both zero-shot and task-specific settings. Its performance is reasonable without additional fine-tuning and improves notably with task-specific tuning.

  • Evaluation Setup: The evaluation uses a single-room 3DMV-VQA subset containing 1,212 scenes from the Habitat Matterport 3D Dataset.The benchmark tests understanding and reasoning about 3D scenes.
  • Evaluation Setup: Zero-shot evaluation uses the instruction “Answer the question using one word or one phrase” to match the benchmark’s concise answers.
  • Results: Scene-LLM demonstrates reasonable performance on 3DMV-VQA without additional fine-tuning.
  • Results: Task-specific tuning produces a notable improvement over the zero-shot setting.

14 Scene Data Generation Details.

Scene-level instructional data is generated from multiple context types and organized into 12 instruction-following annotation types. The resulting data includes question answering, captions, task decomposition, dialogue, and functionality-improvement examples.

  • Prompt Structure: Scene-data prompts contain Pcontext for scene context and Pinstruction for the requested instructional-data type.
  • Prompt Structure: The four scene-context types are dense caption, object caption, object entities, and object bounding boxes.
  • Annotation Types: The generation process produces 12 instruction-following annotation types and restricts context types for each data-generation task to reduce hallucination.
  • Annotation Types: The annotation types include scene and object captions, general and concept question answering, task decomposition, and functionality improvement.
  • Data Post-processing: LLama-2chat-70b post-processes generated instructional data by removing redundancy and inaccurate information, correcting errors, and completing incomplete sentences.
  • Annotation Distribution: The distribution of instructional-following annotations includes question answering, object caption, task decomposition, dense caption, human-robot dialogue, and function improvement.

15 Training Detail

The projection layer maps 3D features into the LLM’s feature space and is trained in staged configurations. Training uses AdamW with specified learning rates, warmup schedules, batch size, and GPU resources.

  • Projection Layer Structure: The projection layer maps 1,030-dimensional point features to the LLM’s 768-dimensional input space through fully connected layers and GELU activation.The layers have matrix shapes [1030, 768] and [768, 768].
  • Projection Layer Training: Projection-layer training uses 32 NVIDIA A100 GPUs, batch size 64, AdamW, learning rate 1e-5, and 1,000 warmup steps at 1e-6.This stage takes 5 hours.
  • Joint Fine-tuning: Joint fine-tuning uses 32 NVIDIA A100 GPUs, batch size 64, AdamW, learning rate 2e-5, and 2,000 warmup steps at 1e-6.This stage takes 8 hours.
  • Task-specific Tuning: Task-specific tuning reports results at 1,500 steps for ScanQA and SQA3D, 3,500 for Alfred, 6,000 for Scan2Cap, and 5,000 for 3DMV-VQA.
Loading 2403.11401v2…