PaperlayerView exact revision on arXiv

Source-linked AI summary

3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding

Xiaohu Huang, Jingjing Wu, Qunyi Xie, Kai Han

arXiv:2506.01946v2cs.CV

TL;DR

MLLMs lack explicit 3D training signals, leaving their 3D-aware representation learning insufficiently understood. 3DRS addresses this by aligning MLLM visual features with pretrained 3D foundation-model features, and experiments show consistent gains across MLLMs and scene-understanding benchmarks.

  • Problem

    MLLMs lack explicit 3D pretraining data, and systematic investigation of their 3D-aware representation learning remains limited.

  • Method

    3DRS provides direct 3D-aware supervision by aligning MLLM visual features with features from pretrained 3D foundation models.

  • Results

    Experiments demonstrate consistent performance improvements across multiple MLLMs and 3D scene-understanding benchmarks.

  • Takeaways & Limitations

    Higher multi-view correspondence scores consistently accompany better downstream scene-understanding performance.

  • Takeaways & Limitations

    The method's improvements may be limited by the relatively small finetuning dataset and the quality of its teacher 3D foundation model.

Abstract

from arXiv · show

Recent advances in scene understanding have leveraged multimodal large language models (MLLMs) for 3D reasoning by capitalizing on their strong 2D pretraining. However, the lack of explicit 3D data during MLLM pretraining limits 3D representation capability. In this paper, we investigate the 3D-awareness of MLLMs by evaluating multi-view correspondence and reveal a strong positive correlation between the quality of 3D-aware representation and downstream task performance. Motivated by this, we propose 3DRS, a framework that enhances MLLM 3D representation learning by introducing supervision from pretrained 3D foundation models. Our approach aligns MLLM visual features with rich 3D knowledge distilled from 3D models, effectively improving scene understanding. Extensive experiments across multiple benchmarks and MLLMs -- including visual grounding, captioning, and question answering -- demonstrate consistent performance gains. Project page: https://visual-ai.github.io/3drs

1 Introduction

The paper investigates how well MLLMs learn 3D-aware representations and shows that stronger multi-view correspondence aligns with better scene understanding. It introduces 3DRS, which distills supervision from pretrained 3D foundation models into MLLM visual features.

  • MLLMs lack explicit 3D pretraining data, while scarce paired 3D-text datasets constrain cross-modal feature alignment.
  • The study evaluates 3D awareness through view equivariance, measuring feature similarity between corresponding voxel pairs across views.
  • Higher correspondence scores consistently accompany better downstream performance, revealing a strong positive correlation between 3D-aware representation quality and scene understanding.
  • A handcrafted view-equivalence objective provides some gains but offers limited supervision for comprehensive 3D learning.
  • 3DRS aligns MLLM visual features with pretrained 3D foundation-model features encoding correspondence, depth, and camera knowledge.
  • Experiments across multiple MLLMs and 3D scene-understanding benchmarks show consistent performance improvements and support the framework's generality.

2 Method

The method evaluates MLLM 3D-aware representations through multi-view correspondence and improves them by distilling features from pretrained 3D foundation models. 3DRS combines this alignment with standard language-generation training and consistently improves correspondence learning across evaluated MLLMs and benchmarks.

  • 2.1.1 Preliminaries: MLLMs receive multi-view images with per-pixel 3D coordinates, which are positionally encoded and added to image features before text decoding.This preserves pretrained 2D perceptual knowledge while injecting explicit 3D priors.
  • 2.1.2 Assessing 3D Feature Learning via Multi-View Correspondence: Multi-view correspondence is evaluated by grouping pixels from different views into shared 3D voxels and measuring cosine similarity between their final visual features.The resulting sequence-level correspondence score averages similarities across all correspondence pairs.
  • 2.1.3 Quality of 3D Feature vs. Downstream Task Performance: Across five RGB-D scene-understanding benchmarks and three MLLMs, downstream performance consistently improves from the lowest to highest correspondence-score quartiles.The evaluated tasks cover visual grounding, captioning, and question answering.
  • 2.2.1 Correspondence-based 3D Supervision Loss: A correspondence loss encourages features from the same voxel to be similar and features from different voxels to be dissimilar, but its view-equivariance focus limits the range of captured 3D properties.The loss supplements the standard cross-entropy objective and yields downstream gains.
  • 2.2.2 3D Foundation Model-Guided Feature Distillation: 3DRS distills features from pretrained 3D foundation models, whose multi-task geometric pretraining provides targets encoding correspondence, camera, depth, and point-cloud information.The foundation-model outputs are pooled to the MLLM feature-map size, projected with an alignment MLP, and matched using a similarity-based distillation loss.
  • 2.2.2 3D Foundation Model-Guided Feature Distillation: The final objective combines language-generation cross-entropy with the 3D foundation-model alignment loss, and 3DRS substantially improves correspondence learning across all evaluated MLLMs and benchmarks.Figure 3 compares correspondence scores before and after applying 3DRS with VGGT as the foundation model.

3 Experiments

Experiments evaluate 3DRS across diverse 3D scene-understanding benchmarks, MLLM backbones, and representation supervisors. The results show consistent gains from 3D foundation-model supervision, including stronger performance than specialist, generalist, and 2D-supervised alternatives.

  • Experimental Setup: The evaluation covers object localization, multi-object referring, dense captioning, and question answering across six 3D scene-understanding benchmarks.Metrics include accuracy, F1, CIDEr, BLEU-4, and exact match, depending on the benchmark.
  • Comparison with State-of-the-Art Models: 3DRS substantially improves performance over task-specific specialists and consistently outperforms point-cloud-based and other MLLM-based 3D generalists.The comparison includes LL3DA, Chat-3D, Grounded 3D-LLM, 3D-LLaVA, Inst3D-LLM, LLaVA-3D, and Video-3D LLM.
  • Comparison with State-of-the-Art Models: 3DRS achieves the best open-source results on most VSI-Bench metrics, including object count, absolute distance, room size, and appearance order.It remains competitive with proprietary models while covering diverse spatial-reasoning tasks.
  • Effectiveness with Different MLLMs: 58.1 to 62.9: LLaVA-Next-Video with 3DRS improves ScanRefer Acc@0.25, while 58.0 to 60.4 improves Multi3DRef F1@0.25.LLaVA-OneVision and Qwen2-VL also improve on every evaluated dataset and metric.
  • Diagnostic Study: 3D foundation models FLARE and VGGT outperform 2D supervisors MAE, Siglip2, and Dinov2 across almost every metric.The reported explanation is that 3D models encode geometric structure, spatial relationships, and depth from large-scale 3D pretraining.
  • Diagnostic Study: 3D foundation-model supervision improves performance across all metrics beyond correspondence-loss supervision, while supervision at deeper layers, especially the last layer, performs best.The deeper-layer result is attributed to greater proximity to output predictions and greater capacity to fit 3D features.

4 Related Work

Prior scene-understanding methods use point clouds, multi-view images, and 2D-derived 3D priors, but 3D-awareness research has largely focused on vision models rather than MLLMs. 3DRS addresses this gap by using 3D foundation models to supervise MLLM representations.

  • 3D Scene Understanding with LLMs: Earlier methods align point-cloud and text features, while later approaches use multi-view images and inject 3D priors into 2D representations.These methods leverage MLLM pretraining on large-scale image-text data for 3D scene comprehension.
  • 3D-Awareness in Vision Models: Most prior 3D-awareness studies examine pure vision models and rely primarily on 2D foundation models rather than 3D foundation models.The cited examples include FiT3D, Probe3D, Lexicon3D, CUA-O3D, and related ViT-based work.
  • 3D-Awareness in Vision Models: Distilling 3D representations into MLLMs remains largely unexplored because MLLMs require integrated visual, linguistic, and spatial information.This distinguishes MLLM-focused 3D-awareness research from prior representation-distillation strategies.
  • 3DRS: 3DRS introduces 3D foundation models as supervisors to directly guide MLLM 3D representation learning instead of relying on 2D foundation-model supervision.The framework specifically targets the 3D-awareness of MLLMs.

5 Conclusion

The paper studies MLLM 3D representations and proposes 3DRS, which uses pretrained 3D foundation models as teachers for direct 3D-aware supervision. Across diverse benchmarks, 3DRS consistently improves object localization, spatial reasoning, and 3D question answering.

  • Conclusion: 3DRS provides direct 3D-aware supervision by using pretrained 3D foundation models as teachers for MLLMs.The framework is designed to produce richer geometric and spatial representations.
  • Conclusion: 3DRS consistently improves performance across object localization, spatial reasoning, and 3D question answering benchmarks.The conclusion reports gains across a variety of 3D scene-understanding tasks.

6 Limitation

The approach is limited by the relatively small fine-tuning dataset and by the quality and errors of the teacher 3D foundation model.

  • Limitations: The relatively limited fine-tuning dataset may restrict the full realization of 3DRS’s potential compared with the scale of MLLM pretraining data.The authors identify incorporating 3D-awareness learning during MLLM pretraining as a future direction.
  • Limitations: Because 3DRS distills from a teacher, inaccurate correspondence, depth estimation, or geometric representations may propagate to the student MLLM.The method’s performance is therefore upperbounded by the quality of the teacher 3D foundation model.

A.1 World Coordinate Computation

The method converts multi-view image pixels into global 3D coordinate maps using depth, camera intrinsics, and camera-to-world extrinsics.

  • Each image is paired with a depth map, camera intrinsics, and camera-to-world extrinsics to support 3D coordinate computation.
  • A pixel at (u, v) is mapped to its corresponding 3D coordinate in the global coordinate system.
  • Repeating the computation for every pixel produces one per-pixel 3D coordinate map for each image.

A.2 Datsests for Training

Training uses established 3D vision-language datasets spanning localization, captioning, and question answering, with varied sample counts and scan coverage.

  • Fine-tuning uses validation splits of ScanRefer, Multi3DRefer, Scan2Cap, and ScanQA, plus the SQA3D test split.
  • Dataset sizes range from 26,515 ScanQA instances to 79,445 SQA3D samples, while ScanRefer and Scan2Cap each contain 36,665 samples.
  • Most datasets derive from 562 unique scans, whereas SQA3D includes 518 scans.
  • The datasets vary in question and answer length, with SQA3D averaging 37.8-question and 1.1-answer words.
  • VSI-Bench evaluation uses the pre-training data from VG-LLM.

A.3 Detailed Comparison

3DRS achieves strong benchmark performance across localization, referring-expression understanding, question answering, and captioning tasks.

  • 3DRS achieves the best overall performance on the ScanRefer validation set, particularly in the challenging Multiple scenario.
  • 3DRS obtains the best overall Multi3DRefer results, with top F1 scores in standard and challenging zero-target and distractor settings.
  • 3DRS achieves the best ScanQA validation performance across almost all metrics, including EM, BLEU, METEOR, and CIDEr.
  • 3DRS achieves the highest SQA3D test-set scores on almost every question type and in the overall average.
  • 3DRS performs best on Scan2Cap CIDEr and remains highly competitive on BLEU-4, METEOR, and ROUGE-L.

A.4 Ablation Study

Ablation and qualitative studies examine supervision design and show that 3DRS supports accurate grounding, detailed captioning, and reliable question answering while raising broader-impact considerations.

  • Ablation Study: Supervising only the last network layer consistently performs best across object localization, captioning, and question answering benchmarks.Adding intermediate supervised layers degrades results, suggesting that multi-layer supervision may over-constrain geometric features and weaken semantic representations.
  • Ablation Study: Euclidean, cosine, and combined distillation losses yield very similar performance across the evaluated benchmarks.The feature-distance metric therefore has limited influence in this setting.
  • Qualitative Results: Grounding predictions align more closely with ground-truth boxes than baseline predictions in the shown examples.
  • Qualitative Results: Generated object captions are more accurate, detailed, precise, and faithful, with higher reported CIDEr scores than the baseline.
  • Qualitative Results: The question-answering examples include correct responses to counting and object-location questions, often supported by clear visual evidence.
  • Qualitative Results: Qualitative visualizations show more accurate grounding, richer object descriptions, and more reliable answers than the baseline across the evaluated tasks.
  • Broader Impacts: Enhanced 3D perception has potential benefits for assistive robotics, autonomous navigation, medical imaging, and immersive education.
  • Broader Impacts: Reconstructing and analyzing people’s environments in surveillance contexts raises privacy concerns requiring anonymization, consent, access controls, security, and regulatory compliance.
Loading 2506.01946v2…