PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors

Duo Zheng, Shijia Huang, Yanyang Li, Liwei Wang

arXiv:2505.24625v3cs.CVcs.AI

TL;DR

Existing video-based MLLMs for 3D understanding often require dense 3D inputs, while frame-wise visual tokens struggle with geometry across frames. VG LLM adds a 3D visual geometry encoder whose features are fused with visual tokens, achieving strong video-only 3D understanding and spatial reasoning, including 47.3% average accuracy on VSI-Bench for its 4B model, above Gemini-1.5-Pro.

  • Problem

    Prior approaches depend on dense 3D data, and standard MLLM visual representations struggle to capture inter-frame geometry needed for 3D reasoning.

  • Method

    VG LLM uses a pretrained 3D visual geometry encoder to extract geometry priors from image sequences, fuse them with conventional visual tokens, and feed them into an MLLM.

  • Results

    VG LLM outperforms larger spatial-enhanced models across multiple video-based 3D scene understanding and spatial reasoning tasks; its 4B model reaches 47.3% average accuracy on VSI-Bench, surpassing Gemini-1.5-Pro.

  • Takeaways & Limitations

    The results support effective 3D scene understanding and spatial reasoning from monocular RGB videos without explicit dense 3D scene input.

  • Takeaways & Limitations

    Performance is constrained by backbone capacity, and ScanNet-based evaluations may overestimate performance because VGGT was trained on commonly used 3D scene datasets.

Abstract

from arXiv · show

Previous research has investigated the application of Multimodal Large Language Models (MLLMs) in understanding 3D scenes by interpreting them as videos. These approaches generally depend on comprehensive 3D data inputs, such as point clouds or reconstructed Bird's-Eye View (BEV) maps. In our research, we advance this field by enhancing the capability of MLLMs to understand and reason in 3D spaces directly from video data, without the need for additional 3D input. We propose a novel and efficient method called the Video-3D Geometry Large Language Model (VG LLM). Our approach utilizes a 3D visual geometry encoder to extract 3D prior information from video sequences. This information is then integrated with visual tokens and input into the MLLM. Extensive experiments have shown that our method has achieved substantial improvements in various tasks related to 3D scene understanding and spatial reasoning, all directly learned from video sources. Impressively, our 4B model, which does not rely on explicit 3D data inputs, achieves competitive results compared to existing state-of-the-art methods, and even surpasses the Gemini-1.5-Pro in the VSI-Bench evaluations.

1 Introduction

Prior video-based MLLM approaches for 3D understanding depend on dense 3D inputs, while ordinary visual representations struggle to preserve geometry across frames. VG LLM addresses this by integrating geometry priors from video and improves 3D understanding and spatial reasoning without explicit dense 3D input.

  • Research gap: Dense 3D inputs such as depth maps, point maps, or reconstructed BEV maps constrain prior video-based approaches.These data can be difficult to acquire in real-world scenarios.
  • Research gap: Frame-wise visual encoding misses inter-frame correspondences, forcing the MLLM backbone to infer 3D structure from visual tokens.This inference is challenging, resource-intensive, and can require extensive supervision and careful fine-tuning design.
  • Approach: VG LLM integrates a 3D visual geometry encoder with a conventional visual encoder, fusing their features before the MLLM backbone.The geometry encoder is pretrained on point-map prediction and captures correspondences across frames, improving robustness to viewpoint transformations.
  • Results: VG LLM improves 3D scene understanding and spatial reasoning from video input without explicit dense 3D data.Evaluations cover 3D grounding, dense captioning, video object detection, VSI-Bench, and CV-Bench.
  • Results: 47.3% average accuracy on VSI-Bench is achieved by the 4B model, surpassing Gemini-1.5-Pro.The 8B model improves 3D video object detection by 11.9% precision and 10.7% F1.

2 Related Work

Related work enhances MLLM spatial understanding through improved visual representations, 3D scene data, and spatially focused benchmarks. These efforts motivate methods that capture geometry and relational reasoning directly from video.

  • MLLMs: MLLM research has expanded from 2D understanding to 3D scene understanding, embodied navigation, and learning 3D knowledge from videos.
  • 3D Large Language Models: 3D LLMs use point clouds, lifted multi-view features, video sequences, positional information, or reconstructed BEV maps to represent scenes.Video-3D LLM and GPT4Scene are identified as closely related approaches.
  • Spatial Reasoning: Spatial reasoning methods often rely on synthesized VQA data for depth estimation but primarily focus on static images.VSI-Bench evaluates relational reasoning and egocentric-allocentric transformation abilities.
  • Architecture context: The VG LLM architecture combines globally geometry-aware features from image sequences with semantic-aware features from individual images.
  • Geometry representations: Latent 3D geometry tokens can recover a 3D scene when paired with a dense prediction head.

3 Method

VG LLM extracts latent geometric information from RGB image sequences, fuses it with conventional visual tokens, and feeds the result to an MLLM. The framework supports video-only 3D understanding and spatial reasoning tasks, with task-specific formulations and spatial instruction tuning.

  • Architecture: VG LLM processes image sequences with a conventional 2D visual encoder and a 3D visual geometry encoder before producing the MLLM response.The 3D encoder models inter-frame correspondences and extracts geometric features jointly across images.
  • Architecture: The 3D visual geometry encoder uses models pretrained to capture inter-frame correspondences and reconstruct 3D scenes without additional 3D priors.Its components include per-image feature extraction, cross-frame fusion, and task-specific prediction heads; VG LLM uses the feature-extraction capability.
  • Task Applications: The model relies solely on RGB images during training and inference for its 3D scene understanding applications.The demonstrated tasks are 3D visual grounding, 3D dense captioning, and 3D video object detection.
  • 3D Visual Grounding: 3D visual grounding predicts an object’s frame index and oriented 3D bounding box in that frame’s coordinate system.The box includes center, size, and three rotation angles.
  • 3D Dense Captioning: 3D dense captioning detects object proposals first, then generates descriptions from their coordinates.The model is prompted to describe the object at a specified box center, evaluating positional and spatial understanding.
  • 3D Video Object Detection: 3D video object detection requires detecting all objects across a video in one unified coordinate system without explicit camera parameters or depth information.
  • Instruction Tuning: Instruction tuning uses SPAR-7M, a spatial reasoning dataset spanning 33 tasks, together with LLaVA-Hound data to preserve general visual instruction capabilities.

4 Experiments

Experiments evaluate VG LLM across 3D scene understanding, spatial reasoning, and generic multimodal benchmarks. Results show strong grounding, detection, spatial reasoning, robustness, and feature-fusion performance from video-based 3D geometry.

  • 3D Visual Grounding: VG LLM improves 3D visual grounding from monocular RGB videos, reaching 41.6% Acc@0.25 versus SPAR’s 31.9%, and 57.6% with proposal refinement.The direct prediction achieves a 9.7-point improvement over SPAR; refinement substantially increases Acc@0.25.
  • 3D Dense Captioning: VG LLM captions 3D object proposals using RGB images and center coordinates, achieving 80.0 C@0.5 and 41.5 B-4@0.5.The model does not require explicit depth or geometric data as additional inputs for caption generation.
  • 3D Video Object Detection: Adding 3D geometry raises average F1 by 8.2% for the 4B model and 8.7% for the 8B model in the 4-frame setting.The gains are reported as increases from 30.0% to 38.2% and from 32.5% to 41.2%, respectively, and are attributed to improved egocentric-allocentric transformation understanding.
  • 3D Video Object Detection: VG LLM maintains strong performance on longer inference sequences despite training on 4-frame sequences, whereas the baseline drops noticeably.This result indicates robustness to variations in frame count during inference.
  • Spatial Reasoning Benchmarks: VG LLM achieves 47.3% average accuracy with its 4B model and 50.7% with its 8B model on VSI-Bench, outperforming the listed competitors.The 4B model outperforms all competitors including Gemini-1.5 Pro.
  • Ablation Analysis: Directly adding geometry features performs best among the compared fusion strategies, while predicted camera or depth information alone provides no clear benefit.Visual geometry features outperform predicted spatial information because they combine spatial signals while reducing noise impact.

5 Conclusion

The framework enhances MLLMs’ 3D spatial understanding using latent 3D geometric information from video alone, without explicit 3D scene input.

  • VG LLM incorporates a 3D visual geometry encoder that provides latent 3D geometric information from video inputs.
  • The model outperforms larger spatial-enhanced models on various 3D scene understanding tasks and spatial reasoning benchmarks without explicit 3D scene input.

A.1 Evaluation Details.

Evaluation uses standardized MLLM benchmarks and task-specific procedures for 3D grounding, dense captioning, and video object detection.

  • LMMs-Eval standardizes MLLM evaluation, and inference uses greedy sampling to generate outputs.
  • VSI-Bench, Video-MME, and TempCompass use original configurations, while BLINK and CV-Bench use customized official task configurations and metrics.
  • 3D visual grounding transforms predicted boxes into world coordinates before calculating Intersection-over-Union against oriented ground-truth boxes.
  • 3D dense captioning uses Mask3D-detected object proposals and generates descriptions from object center coordinates transformed by the extrinsic matrix.
  • 3D video object detection greedily matches predicted boxes to unused same-category ground-truth boxes when their IoU exceeds a threshold such as 0.25.

B Data preparation

The data preparation uses publicly available datasets, task-specific video sampling, and structured prompts for grounding, captioning, and object detection.

  • All datasets used in the research are publicly available, with data-preparation details provided in the section.
  • SPAR-7M data are mixed using the official codebase, visual markers are drawn on input images, and navigation annotations are discarded.
  • LLaVA-Video-178K frames are sampled at 2 FPS with adaptive sampling constrained to between 4 and 8 frames.
  • For ScanRefer and Scan2Cap, each scene uses 32 uniformly sampled frames, while detection entries contain 4 consecutive frames sampled at 1 FPS.
  • 3D visual grounding prompts the model to localize the first clear frame showing a described object and output its frame-indexed 3D bounding box as JSON.
  • 3D dense captioning prompts the model to describe an object at specified coordinates using the video images and natural-language output.
  • 3D video object detection prompts the model to detect 3D bounding boxes in the first frame’s camera coordinate system and return object labels and box parameters.

C Detailed Results on 3D Video Object Detection

The method substantially improves 3D video object detection over a baseline without visual geometry, although small-object detection remains difficult.

  • The method significantly outperforms the baseline without visual geometry across most 3D video object detection categories.
  • Detecting small objects such as pillows, lamps, and backpacks remains challenging.

D More Visualization

Additional visualizations show that VG LLM improves 3D video object detection, while 3D visual grounding remains vulnerable to orientation and depth-localization errors.

  • 3D visual grounding: VG LLM successfully locates 3D bounding boxes for chairs, cabinets, and document organizers in positive grounding examples.The visualizations also show that object orientation can remain inaccurate.
  • 3D visual grounding: Negative grounding cases reveal that predictions may look correct in 2D projection while remaining inaccurate in 3D.
  • 3D video object detection: 3D geometry significantly improves detection precision and recall over the baseline in 3D video object detection.In one example, VG LLM corrects the baseline’s desk prediction box mismatch with the ground truth.
  • 3D video object detection: Tables 10 and 11 report F1 scores for all categories in six-frame and four-frame 3D video object detection settings.

E Limitations

The paper identifies limitations related to backbone capacity, training methodology, benchmark distribution, and inherited MLLM challenges.

  • Model capacity: VG LLM’s capabilities remain constrained by the inherent capacity of its 3B and 7B MLLM backbones.The authors suggest that scaling model size could improve fundamental abilities and generalization.
  • Training methodology: The work uses supervised fine-tuning exclusively, leaving reinforcement learning for future investigation.The paper notes that emerging research reports reasoning improvements from reinforcement learning.
  • Evaluation scope: Evaluating VGGT on benchmarks built from the same datasets used for training, such as ScanNet, risks overestimating performance.The authors identify out-of-distribution evaluation as a future direction.
  • Inherited MLLM challenges: Because VG LLM is built on Qwen2.5-VL, it inherits challenges including visual hallucinations, base-model biases, and vulnerability to adversarial inputs.
Loading 2505.24625v3…