PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding

Duo Zheng, Shijia Huang, Liwei Wang

arXiv:2412.00493v2cs.CVcs.CL

TL;DR

MLLMs trained mainly on 2D data struggle to represent the spatial complexity of 3D scenes. Video-3D LLM treats scenes as videos, injects 3D position encoding, and uses maximum coverage sampling; it achieves state-of-the-art results across five benchmarks while using less data than LLaVA-3D.

  • Problem

    MLLM representations remain mismatched with complex 3D scenes because they are primarily trained on 2D images and labeled 3D data is limited.

  • Method

    Video-3D LLM processes video frames with global 3D coordinates, injects position encoding into video representations, and uses maximum coverage sampling in a multi-task model.

  • Results

    Video-3D LLM achieves state-of-the-art performance on ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D, surpassing LLaVA-3D with 26% of its 3D data.

  • Takeaways & Limitations

    The results support adapting video models to 3D scene understanding by combining video-based representations with explicit spatial information.

  • Takeaways & Limitations

    The evaluation uses Mask3D-generated object proposals for both 3D visual grounding and dense captioning.

Abstract

from arXiv · show

The rapid advancement of Multimodal Large Language Models (MLLMs) has significantly impacted various multimodal tasks. However, these models face challenges in tasks that require spatial understanding within 3D environments. Efforts to enhance MLLMs, such as incorporating point cloud features, have been made, yet a considerable gap remains between the models' learned representations and the inherent complexity of 3D scenes. This discrepancy largely stems from the training of MLLMs on predominantly 2D data, which restricts their effectiveness in comprehending 3D spaces. To address this issue, in this paper, we propose a novel generalist model, i.e., Video-3D LLM, for 3D scene understanding. By treating 3D scenes as dynamic videos and incorporating 3D position encoding into these representations, our Video-3D LLM aligns video representations with real-world spatial contexts more accurately. In addition, we have implemented a maximum coverage sampling technique to optimize the trade-off between computational cost and performance. Extensive experiments demonstrate that our model achieves state-of-the-art performance on several 3D scene understanding benchmarks, including ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.

1. Introduction

Video-3D LLM addresses the gap between 2D-trained MLLM representations and the spatial complexity of 3D scenes by using video frames with 3D position information. A multi-task model and maximum coverage sampling achieve state-of-the-art results across five 3D scene understanding benchmarks with less training data than LLaVA-3D.

  • Motivation: 2D-focused MLLM pretraining leaves a representation gap for complex 3D scenes, while limited labeled 3D data constrains straightforward 3D fine-tuning.Existing approaches commonly adapt point clouds, voxels, lifted image features, or recognized-object information.
  • Approach: Video-3D LLM directly processes video frames and corresponding 3D global coordinates, injecting position information into video representations for 3D scene understanding.Coordinates are obtained by backprojecting depth images, and position-aware representations connect visual appearance with spatial context.
  • Approach: Maximum coverage sampling selects informative frames to improve spatio-temporal coverage while reducing inference cost.The strategy treats frame selection as a coverage problem and uses a greedy algorithm.
  • Results: 4.1% Acc@0.25 on ScanRefer, 4.6 CIDEr@0.5IoU on Scan2Cap, 2.9% EM on ScanQA, and 3.0% EM on SQA3D over LLaVA-3D using 26% of its 3D data.The comparison uses 223k versus 859k 3D training examples.
  • Approach: The model is trained jointly across 3D question answering, dense captioning, and visual grounding tasks as a single generalist model.This multi-task setup supports the five reported benchmark evaluations.

2. Related Work

Prior work adapts MLLMs to 3D scenes through point clouds, lifted multi-view features, object representations, or 3D task instruction data. Video-based approaches show promise but generally lack detailed spatial information and broad 3D-task capability.

  • LLMs for 3D Scene Understanding: Existing 3D LLMs incorporate point clouds, lifted multi-view image features, or recognized-object characteristics to build scene-level representations.These approaches differ in how they convert or augment visual information for 3D understanding.
  • Video-Language Models for 3D Understanding: Video LLMs have been adapted to 3D question answering and robotic manipulation by leveraging large video datasets and pretrained video models.These methods demonstrate the potential of video representations for 3D-related tasks.
  • Video-Language Models for 3D Understanding: Existing video-based models do not capture detailed 3D spatial information, limiting their performance on tasks requiring precise spatial alignment.Their early adaptations also remain short of handling diverse 3D scene understanding tasks.

3. Method

Video-3D LLM represents 3D scenes as video streams paired with global coordinate maps, preserving temporal and spatial context while adding position awareness. It samples frames by maximum voxel coverage, encodes patch coordinates, and trains one model across multiple 3D tasks.

  • Model Architecture: Video-3D LLM directly processes temporal video frames and associated back-projected 3D coordinate maps instead of converting frames into point clouds or voxels.This design preserves temporal and spatial contextual information in the original video streams.
  • Frame Sampling: Maximum coverage sampling greedily selects frames that add the largest number of previously uncovered voxels until the frame budget or coverage threshold is reached.The formulation seeks a small frame subset maximizing the union of covered voxels; the greedy solver has approximation ratio 1 - 1/e.
  • Position-Aware Video Representation: Depth images are back-projected using camera intrinsics and extrinsics to produce global coordinates for each sampled frame.The resulting coordinate maps remain paired with their corresponding RGB frames.
  • Position-Aware Video Representation: The model averages coordinates within image patches, applies sinusoidal encoding to the 3D coordinates, and adds the resulting coordinate embeddings to visual embeddings.Patch-level pooling retains positional information before the coordinate and visual embeddings are combined.
  • Multi-Task Training: A single model is trained across multiple 3D scene understanding tasks using task-specific supervision, including cross-entropy for language-generation tasks and InfoNCE for visual grounding.Dense captioning also adds the encoded center position of the target bounding box to a special coordinate token.

4. Experiments

Video-3D LLM is evaluated as a multi-task 3D generalist across visual grounding, dense captioning, and question answering, with comparisons against expert and LLM-based baselines. It achieves strong benchmark results while maximum coverage sampling reduces inference time and ablations examine coordinate encoding, scene representation, and sampling choices.

  • Frame Sampling: Uniform sampling achieves state-of-the-art performance across varied 3D tasks, while maximum coverage sampling matches similar results with inference time reduced from 1050ms to 527ms.The maximum-coverage configuration uses a 95% coverage ratio and at most 32 frames.
  • 3D Visual Grounding: 58.1% Acc@0.25 and 51.7% Acc@0.5 on ScanRefer, plus 58.0% and 52.7% F1 on Multi3DRefer, establish strong 3D visual-grounding performance.Compared with ChatScene using the same object proposals, the model improves Acc@0.25 by 2.6% on ScanRefer and F1@0.25 by 0.9% on Multi3DRefer.
  • 3D Dense Captioning: Scan2Cap performance reaches 41.3 B-4@0.5 and 83.8 C@0.5, with 3D position encoding connecting video content to position information.The dense-captioning evaluation generates captions for detected object proposals.
  • 3D Question Answering: 30.1% EM on ScanQA and 58.6% EM on SQA3D show that Video-3D LLM also performs strongly on 3D question answering.The reported outcome is attributed to representations inherited from Video LLMs.
  • 3D Scene Representation: Compared with voxel modeling using the same backbone, the video representation gains 2% accuracy on ScanQA, indicating use of spatiotemporal priors beyond stronger visual features.The comparison also reports Scan2Cap C@0.5 values of 83.8 and 54.9, though the passage describes the difference as a notable improvement of 28.9.

5. Conclusion

The paper presents Video-3D LLM as a paradigm for using Video LLMs in 3D scene understanding, combining 3D position encoding with multi-task training and maximum coverage sampling.

  • 5. Conclusion: Video-3D LLM incorporates 3D position encoding into video representations and uses multi-task training to support multiple 3D scene understanding tasks.The model is designed to facilitate visual grounding, dense captioning, and question answering.
  • 5. Conclusion: Maximum coverage sampling optimizes the trade-off between computational cost and model performance.
  • 5. Conclusion: Extensive experiments demonstrate the superiority of the proposed method.

A. Implementation Details

The evaluation follows established dataset splits and task-specific procedures for prediction and caption generation. Data statistics are reported using the LLaVA format.

  • A. Implementation Details: The study reports training and testing statistics in separate tables using data converted to LLaVA format.Validation sets are used for ScanRefer, Multi3DRefer, Scan2Cap, and ScanQA, while SQA3D uses its test set.
  • A. Implementation Details: ScanRefer predictions select the object with the highest similarity, while Multi3DRefer selects objects whose cumulative probability exceeds threshold p=0.25.
  • A. Implementation Details: Scan2Cap evaluation adds “sos” and “eos” tokens around generated captions before measuring captioning performance.

B. Detailed Comparison

Video-3D LLM achieves strong results across detailed evaluations of 3D question answering, captioning, and visual grounding. Its results lead several benchmarks while exposing weaker performance on multi-target grounding.

  • B. Detailed Comparison: 58.86% average EM on SQA3D is the best performance across all question categories, exceeding LLaVA-3D by 2.94%.
  • B. Detailed Comparison: 83.77 CIDEr and 42.43 BLEU-4 are achieved on Scan2Cap, establishing state-of-the-art captioning results.
  • B. Detailed Comparison: 58.12% Acc@0.25 and 51.72% Acc@0.5 on ScanRefer surpass ChatScene by 2.6% and 1.5%, respectively.
  • B. Detailed Comparison: Video-3D LLM outperforms previous methods on several Multi3DRefer categories but remains below ChatScene on multi-target cases.
  • B. Detailed Comparison: 10.7% relative improvement on EM@1 and 11.9% on CIDEr are reported for ScanQA against previous top-tier models.
Loading 2412.00493v2…