PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models

Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, Hengshuang Zhao

arXiv:2501.01428v4cs.CV

TL;DR

VLMs remain limited in 3D spatial comprehension because they lack global-local correspondence between whole scenes and individual frames. GPT4Scene builds that correspondence with BEV images and consistent object markers, while ScanAlign fine-tuning improves smaller VLMs across indoor 3D tasks and can preserve gains with raw video inputs.

  • Problem

    VLMs have limited effectiveness on immersive 3D indoor scenes because they lack global scene representations and spatial-temporal context for local frames.

  • Method

    GPT4Scene reconstructs BEV images and aligns video frames with consistent Spatial-Temporal Object markers, while ScanAlign enables single-stage VLM fine-tuning.

  • Results

    GPT4Scene achieves state-of-the-art 3D understanding across indoor tasks, including zero-shot gains for large VLMs and fine-tuned Qwen2-VL-7B performance improvements.

  • Takeaways & Limitations

    Fine-tuned models retain strong 3D understanding with raw video inputs, indicating that the paradigm supports visual scene comprehension beyond explicit BEV and marker prompting.

Abstract

from arXiv · show

In recent years, 2D Vision-Language Models (VLMs) have made significant strides in image-text understanding tasks. However, their performance in 3D spatial comprehension, which is critical for embodied intelligence, remains limited. Recent advances have leveraged 3D point clouds and multi-view images as inputs, yielding promising results. However, we propose exploring a purely vision-based solution inspired by human perception, which merely relies on visual cues for 3D spatial understanding. This paper empirically investigates the limitations of VLMs in 3D spatial knowledge, revealing that their primary shortcoming lies in the lack of global-local correspondence between the scene and individual frames. To address this, we introduce GPT4Scene, a novel visual prompting paradigm in VLM training and inference that helps build the global-local relationship, significantly improving the 3D spatial understanding of indoor scenes. Specifically, GPT4Scene constructs a Bird's Eye View (BEV) image from the video and marks consistent object IDs across both frames and the BEV image. The model then inputs the concatenated BEV image and video frames with markers. In zero-shot evaluations, GPT4Scene improves performance over closed-source VLMs like GPT-4o. Additionally, we prepare a processed video dataset consisting of 165K text annotation to fine-tune open-source VLMs, achieving state-of-the-art performance on all 3D understanding tasks. Surprisingly, after training with the GPT4Scene paradigm, VLMs consistently improve during inference, even without object marker prompting and BEV image as explicit correspondence. It demonstrates that the proposed paradigm helps VLMs develop an intrinsic ability to understand 3D scenes, which paves the way for a seamless approach to extending pre-trained VLMs for 3D scene understanding.

1. Introduction

3D scene understanding is important for spatially aware applications, but vision-language models struggle with indoor scenes because they lack global scene representations and global-local frame correspondence. GPT4Scene addresses this gap using BEV context and spatial-temporal object markers, while ScanAlign fine-tuning yields strong performance gains.

  • 3D scene understanding captures overall layouts and object relationships for applications including embodied intelligence, virtual reality, and smart cities.
  • Vision-only 3D understanding is motivated by the difficulty of aligning point clouds, images, and text despite their richer visual cues.
  • VLMs struggle with immersive indoor scenes because they lack a global scene representation and alignment between local frames and spatial-temporal context.
  • GPT4Scene reconstructs a BEV image and places consistent spatial-temporal object markers across the BEV and 2D frames to bridge global and local information.
  • 48% relative improvement (40.7 →60.7 EM-1 score) is achieved by fine-tuning Qwen2-VL-7B on ScanAlign, a dataset with 165K aligned annotations.
  • GPT4Scene combines BEV global context, STO-markers, and ScanAlign to improve VLM-based 3D understanding in zero-shot and fine-tuning settings.

2. Related Work

Related work develops 3D vision-language understanding through point clouds, multimodal inputs, and unified language interaction. GPT4Scene is situated against these approaches by pursuing direct visual processing with VLMs.

  • 3D indoor scene understanding combines 3D perception and language models to identify object positions, structures, and relationships for question answering.
  • 3D vision-language research includes task-specific and unified models, with 3D pretraining methods combining 2D visual cues and point clouds for complementary information.
  • 3D point cloud LLMs use point clouds as input to support natural-language generation and interaction in 3D scenes.
  • VLMs integrate visual and language processing to understand and generate combined image-text information, building on 2D image-text pretraining.

3. Methodology

GPT4Scene combines a reconstructed bird’s-eye scene layout with spatial-temporal object markers to connect global context and local video observations for VLM-based 3D understanding. It supports zero-shot prompting for large VLMs and ScanAlign fine-tuning for smaller models.

  • GPT4Scene Framework: GPT4Scene samples video frames, reconstructs the complete sequence into a 3D point cloud, and renders a holistic BEV image.Sampling reduces VLM token and computation costs while reconstruction uses the full temporal sequence.
  • GPT4Scene Framework: The resulting marked video frames and BEV image are concatenated as visual inputs for VLM training and inference.The framework provides global scene layouts alongside object-level details and uses stitched marked frames to reduce input cost during zero-shot evaluation.
  • GPT4Scene Framework: STO-markers align object locations across 2D frames and the BEV image while preserving consistent object identities across frames.3D instance masks are projected onto frames and the BEV image; frame markers use 2D object-mask centroids.
  • Unlocking VLMs with Zero-shot Prompts: Zero-shot evaluation tests 3D question answering, dense captioning, and visual grounding with marked frames and the BEV image.The prompts cover scene questions, object descriptions identified by markers, and object IDs inferred from descriptions.
  • Enhancing VLMs with ScanAlign Fine-Tuning: ScanAlign contains 165K diversified text annotations paired with egocentric frames and BEV images, enabling single-stage instruction fine-tuning of smaller VLMs.The dataset is based on ScanNet annotations and uses STO-marker-annotated visual inputs.
  • Enhancing VLMs with ScanAlign Fine-Tuning: Fine-tuning with ScanAlign yields significant gains for smaller VLMs, whereas zero-shot GPT4Scene enhancements are limited for 2B, 7B, and 8B models.Zero-shot improvements are reported as more pronounced for larger models, motivating fine-tuning for smaller architectures.

4. Experiments

Experiments show that GPT4Scene improves 3D scene understanding through zero-shot prompting and ScanAlign fine-tuning, while performance depends on model scale and input configuration. The method achieves state-of-the-art results across indoor question answering, dense captioning, and visual grounding, with robustness to marker removal and reconstruction quality.

  • 4.1. Zero-shot Setting Results: GPT4Scene’s zero-shot benefits are limited for smaller VLMs but substantial for larger models, with GPT-4o and Gemini-1.5-Pro reaching performance comparable to Chat-Scene.Qwen2-VL-72B also shows significant improvement, whereas smaller models require fine-tuning through ScanAlign.
  • 4.3. Fine-tuning Main Results: Fine-tuned Qwen2-VL-7B achieves state-of-the-art performance across all indoor 3D understanding tasks on ScanNet.The benchmark covers question answering, dense captioning, and visual grounding.
  • 4.3. Fine-tuning Main Results: 56.1% BLEU-1 and 68.6% CIDEr improvements on ScanQA, plus a 41.0% EM-1 increase on SQA3D, result from GPT4Scene fine-tuning over baseline Qwen2-VL-7B.ScanQA changes are 27.8 → 43.4 for BLEU-1 and 53.9 → 90.9 for CIDEr; SQA3D EM-1 changes from 40.7 → 57.4.
  • 4.3. Fine-tuning Main Results: GPT4Scene-finetuned Qwen2-VL-7B substantially improves dense captioning and visual grounding, reaching new state-of-the-art results under HD and HDM configurations.The reported gains exceed existing methods across all evaluation metrics in these configurations.
  • 4.4. Ablation Study: Removing 30% of STO-markers or adapting their sizes has minimal impact, and BEV reconstruction quality negligibly affects performance because BEV mainly supplies global scene context.Across tested reconstruction methods and frame intervals, geometry precision has only a slight influence on results.
  • 4.4. Ablation Study: BEV images and STO-markers both contribute to spatial-understanding gains, while STO-markers are essential for visual grounding evaluation.Removing STO-markers prevents validation on the visual grounding task, leaving question answering as the evaluable task.

5. Conclusion

GPT4Scene enables VLMs to understand 3D scenes from visual inputs by connecting global BEV layouts with spatial-temporal object-marked video frames. It achieves state-of-the-art 3D question-answering performance with zero-shot GPT-4o and fine-tuned smaller VLMs.

  • GPT4Scene combines reconstructed BEV images with spatial-temporal object markers to connect global scene layouts and video-frame details.The framework uses STO markers to align video frames with the BEV representation.
  • Fine-tuned models achieve strong 3D question-answering performance using the ScanAlign dataset.The conclusion reports state-of-the-art performance with zero-shot GPT-4o and fine-tuned smaller VLMs such as Qwen2-VL.
  • Fine-tuned models also perform effectively with raw video inputs for 3D understanding.

A. Prompts of Closed-source VLMs

GPT4Scene’s closed-source VLM prompting supplies a stitched video view and a BEV image, followed by task-specific instructions and answer processing. The prompts constrain object-ID use and standardize concise responses.

  • The prompt provides a stitched 2D video view and a BEV image before task-specific evaluation.The system prompt describes the two-image input, while benchmark prompts specify the task.
  • Object IDs are supplied for reference but cannot be directly used in answers.
  • ScanQA prompts target concise answers of 1-5 words in a standardized format.Generated answers are later cleaned by removing regularized formatting and correcting singularization-related issues.

B. 2D Multi-modal Benchmark

Fine-tuning Qwen2-VL with ScanAlign improves selected 2D multimodal capabilities, especially object and action understanding, without reducing overall 2D understanding.

  • ScanAlign fine-tuning improves MVBench object and action metrics.The reported gains indicate better handling of spatial variations and scene-object information.
  • Fine-tuning preserves the model’s ability to understand images and video across other 2D multimodal benchmarks.

C. Qualitative Results

Qualitative and benchmark results examine GPT4Scene across 2D and 3D understanding tasks. Fine-tuned Qwen2-VL handles 3D queries from visual inputs and retains 2D multimodal performance.

  • 2D Multi-modal Benchmark: Fine-tuned Qwen2-VL improves 2D understanding on MVBench, particularly for object and action metrics.
  • 2D Multi-modal Benchmark: ScanAlign fine-tuning does not reduce the model’s 2D understanding capabilities on the reported multimodal benchmark.
  • Qualitative Results: Qualitative examples show 3D question answering from unannotated video and dense captioning and visual grounding from annotated inputs.The examples cover ScanQA, SQA3D, 3D dense captioning, and 3D visual grounding.

D. Full Quantitive Results

The paper reports complete evaluations across five 3D understanding benchmarks, covering question answering, dense captioning, and visual grounding. The accompanying prompts and qualitative results illustrate how labeled 2D frames and BEV images are supplied for analysis.

  • Benchmark coverage: Full results are reported for ScanQA, SQA3D, Scan2Cap, ScanRefer, and Multi3DRef.Tables 12–16 cover question answering, dense captioning, and visual grounding benchmarks.
  • Benchmark coverage: Table 14 evaluates 3D dense captioning on Scan2Cap.
  • Benchmark coverage: Tables 15 and 16 evaluate 3D visual grounding on ScanRefer and Multi3DRef.
  • Prompt and qualitative setup: The analysis prompt provides a labeled 2D image and a labeled BEV image, with eight video frames arranged in a 2x4 grid.Matching object numbers in the 2D and BEV images indicate the same objects.
  • Prompt and qualitative setup: Qualitative question-answering examples use videos without object annotations, whereas dense-captioning and visual-grounding examples use videos with object annotations.
Loading 2501.01428v4…