Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness

Haochen Wang; Yucheng Zhao; Tiancai Wang; Haoqiang Fan; Xiangyu Zhang; Zhaoxiang Zhang

Source-linked AI summary

Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness

Haochen Wang, Yucheng Zhao, Tiancai Wang, Haoqiang Fan, Xiangyu Zhang, Zhaoxiang Zhang

arXiv:2504.01901v1Apr 2, 2025cs.CVcs.AIcs.CLcs.RO

TL;DR

Large-scale 3D vision-language data is scarce, while input-level modifications alone do not provide sufficient 3D awareness for adapting 2D LMMs. ROSS3D adds cross-view and global-view reconstructive visual supervision during training, achieving strong results across benchmarks and showing potential to use unlabeled 3D data.

Problem
Limited 3D vision-language datasets and insufficient 3D awareness from input-level modifications hinder adapting 2D LMMs to 3D scene understanding.
Method
ROSS3D introduces cross-view reconstruction of masked views and global-view reconstruction of Bird’s-Eye View images as 3D-aware visual supervision.
Results
ROSS3D substantially improves performance across multiple 3D scene-understanding benchmarks and, in certain settings, surpasses the 100% text-supervised baseline using additional unlabeled 3D data.
Takeaways & Limitations
The results demonstrate the potential of 3D-aware visual supervision and large amounts of unlabeled 3D visual data for 3D LMMs.
Takeaways & Limitations
The reconstructive objective described in the preliminaries does not itself introduce 3D awareness, and the overview simplifies some inputs to focus on essential aspects.

Abstract

from arXiv · show

The rapid development of Large Multimodal Models (LMMs) for 2D images and videos has spurred efforts to adapt these models for interpreting 3D scenes. However, the absence of large-scale 3D vision-language datasets has posed a significant obstacle. To address this issue, typical approaches focus on injecting 3D awareness into 2D LMMs by designing 3D input-level scene representations. This work provides a new perspective. We introduce reconstructive visual instruction tuning with 3D-awareness (Ross3D), which integrates 3D-aware visual supervision into the training procedure. Specifically, it incorporates cross-view and global-view reconstruction. The former requires reconstructing masked views by aggregating overlapping information from other views. The latter aims to aggregate information from all available views to recover Bird's-Eye-View images, contributing to a comprehensive overview of the entire scene. Empirically, Ross3D achieves state-of-the-art performance across various 3D scene understanding benchmarks. More importantly, our semi-supervised experiments demonstrate significant potential in leveraging large amounts of unlabeled 3D vision-only data.

1. Introduction

ROSS3D addresses the limits of input-level 3D modifications by adding 3D-aware visual supervision through cross-view and global-view reconstruction. It improves performance across multiple 3D scene-understanding benchmarks and shows potential for learning from unlabeled 3D data.

Motivation: 3D scene understanding supports embodied agents by requiring both object-level spatial relationships and comprehensive environmental layouts.These capabilities are relevant to navigation, manipulation, and complex tasks in physical environments.
Motivation: Limited 3D datasets and the lack of strong language-aligned 3D encoders motivate adapting 2D LMMs for 3D scene interpretation.Prior approaches use 3D point-cloud features or strong 2D priors, but the available 3D resources remain comparatively limited.
Motivation: Input-level modifications alone are insufficient for genuine 3D awareness because LMMs retain an inductive bias toward 2D data.The paper argues that visual pretext tasks should guide models toward spatial relationships and comprehensive layouts.
Method: ROSS3D adds 3D-aware vision-centric supervision during training through cross-view and global-view reconstructive pretext tasks.The method supervises visual outputs of LMMs directly using input video frames.
Method: Cross-view reconstruction recovers masked views from overlapping information, while global-view reconstruction aggregates all views to recover comprehensive Bird’s-Eye View images.The two tasks target detailed inter-view relationships and whole-scene layout, respectively.
Method: ROSS3D uses a denoising network to reconstruct masked views and BEV images from LMM visual outputs, providing the stated 3D-aware supervision.The reconstruction targets correspond to the cross-view and global-view tasks.
Results: 63.0 EM on SQA3D, 107.0 CIDEr on ScanQA, 66.9 ROUGE on Scan2Cap, 61.1 Acc@0.25 on ScanRefer, and 59.6 F1@0.25 on Multi3DRefer exceed Video-3D-LLM by +4.4 EM, +4.9 CIDEr, +5.2 ROUGE, +3.0 Acc@0.25, and +1.6 F1@0.25, respectively.The reported gains span question answering, captioning, and visual grounding benchmarks.
Results: Using 50% text-labeled data with 50% additional unlabeled 3D vision-only data can surpass the 100% text-supervised baseline in certain settings.This semi-supervised result demonstrates potential for leveraging large-scale unlabeled 3D visual data.

2. Related Works

Prior 3D scene-understanding work adapts multimodal models through point clouds, voxelized 2D features, multi-view videos, and BEV representations. Related research also motivates vision-centric supervision and scalable self-supervised learning from visual data.

3D Scene Understanding: 3D scene understanding includes visual grounding, dense captioning, and question answering, which support identifying objects, structures, and inter-object relationships.These tasks address complementary aspects of understanding environments for embodied agents.
Large Multimodal Models for Scene Understanding: Prior 3D LMMs use point-cloud features, voxelized CLIP patches, multi-view images treated as videos, or additional BEV images to represent scenes.These approaches aim to align 3D scene information with 2D LMMs.
Vision-Centric Designs in LMMs: Typical LMMs use visual encoders and multimodal projectors to convert images into visual tokens that serve as prefix tokens for language generation.The related-work discussion characterizes this design as primarily LLM-centric because supervision comes from text tokens.
Visual Self-Supervised Learning: Visual self-supervised learning uses pretext tasks to learn scalable representations from large-scale data without human annotations.Prior work applies this strategy to images, videos, and 3D scenarios, often focusing on point clouds in 3D perception.

3. Preliminaries

The preliminaries describe autoregressive visual instruction tuning, its text-only supervision, and reconstructive visual instruction tuning with a denoising network. The paper identifies the latter's lack of 3D awareness as the motivation for ROSS3D.

Visual Instruction Tuning: Autoregressive LMMs model each text token conditioned on preceding tokens and visual features supplied as prefix tokens.Visual features are produced by a visual encoder and multimodal projector.
Visual Instruction Tuning: The standard multimodal training objective maximizes the likelihood of text outputs, while only tokens after the visual-token prefix are supervised.The visual tokens occupy the first N positions and text outputs begin afterward.
Reconstructive Visual Instruction Tuning: Reconstructive visual instruction tuning adds a denoising network that recovers fine-grained latent visual tokens from visual outputs.The recovered tokens are derived from teacher-encoded visual inputs such as VAE representations.
Reconstructive Visual Instruction Tuning: The existing reconstructive objective is effective for 2D understanding but does not introduce 3D awareness.This limitation motivates adding 3D-aware visual supervision in ROSS3D.

4. Method

ROSS3D adds 3D-aware visual supervision to LMM training rather than relying only on input-level modifications. It uses reconstructive objectives for masked views and whole-scene BEV images, alongside text and grounding objectives.

ROSS3D introduces 3D-aware vision-centric supervision for visual outputs, contrasting with conventional methods that supervise only text outputs.
The method applies input and output transformations to video frames, with 3D awareness determined by reconstructing targets that differ from the inputs.
Cross-View Reconstruction: Cross-view reconstruction randomly masks views and reconstructs them from overlapping information in other views, modeling relationships and alignment across viewpoints.
Global-View Reconstruction: Global-view reconstruction converts masked-view inputs into whole-scene BEV images using video frames, camera extrinsics, and intrinsics.
Training Objective: ROSS3D uses a denoising objective in which visual outputs condition recovery of clean latent tokens from noisy tokens, with diffusion denoising as the default process.
Training Objective: Training also includes standard cross-entropy and a grounding loss, while masking is applied every Δt = 4 steps with a default mask ratio of 25%.

5. Experiments

ROSS3D is evaluated across five 3D scene-understanding benchmarks and multiple experimental comparisons. It consistently outperforms prior methods, while ablations and semi-supervised experiments examine the contribution of its reconstructive supervision and use of unlabeled visual data.

Evaluation Setup: ROSS3D is evaluated on SQA3D, ScanQA, Scan2Cap, ScanRefer, and Multi3DRefer, covering situated reasoning, spatial understanding, captioning, and single- or multi-target grounding.All datasets derive from ScanNet, which contains 1,513 annotated RGB-D scans.
Comparison with State-of-the-Arts: 63.0 EM on SQA3D and 107.0 CIDEr on ScanQA exceed Video-3D-LLM by +4.4 EM and +4.9 CIDEr, respectively.These results are reported for 3D question answering.
Comparison with State-of-the-Arts: 66.9 ROUGE, 43.4 BLEU-4, and 30.3 METEOR on Scan2Cap exceed Video-3D-LLM by +4.6, +1.0, and +1.4, respectively.The comparison concerns 3D dense captioning.
Comparison with State-of-the-Arts: 61.1 Acc@0.25 and 54.4 Acc@0.5 on ScanRefer, plus 59.6 F1@0.25 and 54.3 F1@0.5 on Multi3DRefer, outperform Video-3D-LLM across the reported grounding metrics.The gains are +3.0 and +2.7 on ScanRefer, and +1.6 on both Multi3DRefer thresholds.
Ablation Studies: Ablations find that 3D-aware pretext tasks matter, each task is effective, and their combination improves different input representations.Cross-view reconstruction is particularly effective for ScanRefer, while global-view reconstruction is effective for 3D question answering; ROSS3D also outperforms simply adding 3D features.
Ablation Studies: Using ROSS3D on 50% unlabeled 3D sequences surpasses the 100% text-supervised baseline on ScanQA, reaching 103.2 versus 102.1 CIDEr.The semi-supervised setting significantly outperforms the corresponding 50%-text alternatives.

6. Conclusion

ROSS3D improves LMM-based 3D scene understanding by using 3D-aware visual supervision. Its semi-supervised setup also shows potential for exploiting unlabeled 3D vision-only data.

6. Conclusion: ROSS3D uses cross-view and global-view reconstruction as 3D-aware supervision signals for spatial relationships and comprehensive scene layouts.Cross-view reconstruction targets spatial relationships, while global-view reconstruction targets whole-scene layout comprehension.
6. Conclusion: ROSS3D substantially improves performance across various 3D scene understanding benchmarks compared with previous state-of-the-art techniques.
6. Conclusion: Using 50% text-labeled data and 50% vision-only data, ROSS3D surpasses the 100% text-supervised baseline in certain settings.The result indicates potential for leveraging large amounts of unlabeled 3D data.

A. More Implementation Details

The implementation injects 3D information into video-frame features through absolute-coordinate position encoding and trains a multi-task generalist model on several 3D datasets.

A. More Implementation Details: Each pixel’s absolute 3D coordinates are computed from depth maps, camera extrinsics, and camera intrinsics before position encoding.The representation uses coordinates (x, y, z), depth maps D, extrinsic matrix T, and intrinsic matrix K.
A. More Implementation Details: The encoded 3D positions are added to video features extracted by a vision backbone such as CLIP.
A. More Implementation Details: ROSS3D is trained as a generalist model on 3D question answering, dense captioning, and visual grounding datasets using one set of learned parameters.The combined training data contains 223K samples converted to the LLaVA format.

A.3. Training Objectives

Training objectives supervise textual outputs, visual outputs, or grounding predictions according to the 3D task being performed.

A.3. Training Objectives: 3D question answering and dense captioning use cross-entropy loss for text outputs and denoising loss for visual outputs.
A.3. Training Objectives: 3D visual grounding uses only the 3D visual grounding loss to improve localization accuracy.
A.3. Training Objectives: Grounding object features are formed by averaging patches whose points overlap each proposal by more than 50%, then adding the center-coordinate 3D position embedding.

A.4. Evaluation Details

The evaluation details specify task-specific prediction procedures and report supplementary data and ablation configurations for training-data statistics, masking, and reconstruction settings.

A.4. Evaluation Details: For Multi3DRefer, objects are selected until their cumulative probability exceeds 25%.
A.4. Evaluation Details: The supplementary evaluation materials include training-data statistics and ablations on masking ratio, interval ∆t, and global-view reconstruction filtering.The masking-ratio ablation states that ROSS3D is robust against γ.
A.4. Evaluation Details: Scan2Cap evaluation uses special <sos> and <eos> tokens, while dense captioning and question answering use greedy sampling.

B.1. More Ablation Studies

ROSS3D remains robust across tested reconstruction design choices, while complete benchmark comparisons show consistent gains across 3D scene-understanding tasks.

Cross-View Reconstruction: A 25% masking ratio with interval 4 performs best for cross-view reconstruction, and ROSS3D remains robust to these design choices.The masking ratio and temporal interval are ablated separately to reduce train-test discrepancy.
Global-View Reconstruction: ROSS3D is robust to changes in BEV resolution and filtering technique for global-view reconstruction.
Full Benchmark Comparisons: ROSS3D significantly outperforms previous approaches across all reported benchmarks, supported by complete metric comparisons for SQA3D, ScanQA, ScanRefer, and Multi3DRefer.The supplementary tables provide full comparisons for 3D question answering and visual grounding settings.