Source-linked AI summary
VLM3: Vision Language Models Are Native 3D Learners
Zhipeng Cai, Zhuang Liu, Yunyang Xiong, Zechun Liu, Vikas Chandra, Yangyang Shi
TL;DR
Existing VLMs struggle with fine-grained 3D understanding, which remains dominated by complex expert vision models. VLM3 uses standard VLMs with focal-length unification, text-based pixel references, and mixed-task training, achieving accurate understanding across diverse 3D tasks while matching expert models.
Problem
Existing VLMs struggle with fine-grained 3D understanding, while expert vision models rely on complex task-specific designs.
Method
VLM3 trains standard VLMs with focal-length unification, normalized text-based pixel references, and diverse-task text-based supervised fine-tuning.
Results
VLM3 achieves accurate understanding across diverse fine-grained 3D tasks and improves DepthLM-7B’s depth estimation accuracy from 0.84 to 0.9.
Takeaways & Limitations
VLM3’s simplicity, flexibility, and scalability support a new way to build generalist 3D foundation models.
Takeaways & Limitations
VLM3 falls behind UFM, though further scaling and data-mixture tuning may improve performance.
Abstract
from arXiv · showhide
Vision Language Models (VLMs) enable a unified model to solve various vision tasks through prompting. They have shown promising performance in semantic understanding. However, 3D understanding still largely relies on expert vision models with complex task-specific designs. The key argument this work wants to make is that VLMs are native 3D learners. Our in-depth large scale study shows that 1) focal length unification, 2) text-based pixel reference and 3) data mixture and scaling, are all you need for effective 3D learning. Model architecture changes, large models, heavy data augmentations, and complex losses including the regression formulation, many of which form the foundation of expert vision models, are actually not necessary conditions. As a result, we propose VLM3, a scalable method with the simplest design that enables standard VLMs to master diverse 3D tasks. VLM3 not only advances the VLM depth estimation accuracy by a large margin (0.84 -> 0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.
1 Introduction
VLM3 argues that standard VLMs can become effective learners of diverse, fine-grained 3D tasks without complex task-specific designs. Its simple framework matches or surpasses expert-model performance across object-level understanding, depth, pixel correspondence, and camera pose estimation.
- Motivation: Existing VLMs struggle with fine-grained 3D understanding, motivating the search for standard architectures that can match expert vision models across diverse tasks.Prior approaches often focus on coarse object-level understanding or require extra encoders and modules.
- Core finding: Large-scale study shows that standard VLMs are effective 3D learners without changing architectures or losses or adding heavy data augmentations.The work concludes that most task-specific designs are not necessary conditions for effective 3D learning.
- Method: VLM3 uses focal-length unification through image resizing and text-based pixel or region references with normalized horizontal and vertical ranges to support mix-data training without visual prompting.These components address camera ambiguity and remove the need for visual prompting used in prior work.
- Scope: VLM3 enables standard VLMs to learn object-level 3D understanding, metric depth estimation, pixel correspondence estimation, and camera pose estimation.The framework is presented as a scalable and simple approach spanning diverse, fine-grained 3D tasks.
- Results: 0.84 to 0.9: VLM3-4B improves DepthLM-7B depth accuracy to match UnidepthV2, while reducing base-VLM correspondence EPE by 10x and raising camera-pose AUC30 from 5% to 94%.VLM3-4B also improves over SpatialRGPT-8B on SpatialRGPT-Bench without extra encoders; its camera-pose result surpasses VGGT and matches DA3-Giant.
2 Related Work
Prior 3D vision systems typically depend on task-specific architectures, while recent work has begun using VLMs for object-level and fine-grained 3D understanding. This work argues that standard VLMs without such specialized designs can learn 3D effectively and match heavily designed expert models.
- Task-specific Design in Expert Vision Models: Expert vision models commonly combine pretrained vision encoders with multiple decoders and task-specific routing for different 3D tasks.Decoder designs include DPT, FPN, Gaussian Process, and self-attention with linear layers.
- Task-specific Design in Expert Vision Models: Monocular depth, pixel correspondence, and camera pose estimation often rely on multiple prediction heads, multi-scale warping, or supervision from several related tasks.These designs may jointly use depth, camera rays, point tracks, and poses.
- Task-specific Design in Expert Vision Models: This work challenges the necessity of task-specific designs by showing that standard VLMs can learn 3D effectively and match heavily designed expert models.The claim applies to the generalist-model setting described in the passage.
- VLMs for 3D understanding: Earlier VLM studies converted expert predictions into text prompts or added object-reference encoders to support object-level 3D understanding.These approaches addressed object-level understanding and references involving multiple objects.
- VLMs for 3D understanding: Recent work extended VLMs to fine-grained 3D understanding, with DepthLM demonstrating pixel-level metric depth estimation at accuracy comparable to expert vision models.This observation motivates a broader and deeper study aimed at a simpler, more scalable method.
3 Method
VLM3 enables scalable 3D learning with standard VLMs by unifying focal lengths, using normalized text-based spatial references, and scaling carefully mixed data without task-specific architectural designs. The method supports diverse 3D tasks across single- and multi-view settings through a common text-based interface.
- Focal length unification: VLM3 resizes input images to a 1000-pixel focal length, addressing camera ambiguity without changing standard VLM architectures.For images lacking intrinsics, pretrained single-image calibration models estimate them before focal-length unification.
- Text-based pixel reference: Normalizing both pixel coordinates to [0, 2000) enables text-based pixel reference to match visual prompting accuracy.The normalized representation uses text prompts rather than rendered markers, improving scalability across multiple queried pixels and tasks.
- Text-based pixel reference: Text-based references let VLM3 pack multiple questions per image and train on 10 labeled pixels per sample with negligible computation overhead.The same mechanism refers to objects for object-level 3D understanding and to query and output pixels for correspondence estimation.
- Data mixture and scaling: Careful data mixture and scaling are sufficient for accurate 3D learning once camera ambiguity and pixel reference are solved, making complex task-specific designs unnecessary.When datasets differ greatly in size, naive scaling can saturate or reduce performance; weighting by dataset size is a reasonable cross-task baseline.
- Task coverage: VLM3 covers metric depth estimation, object-level 3D understanding, pixel correspondence estimation, and camera pose estimation across single- and multi-view settings.Implementations include 26M training images for depth, text bounding boxes for objects, roughly 10M image pairs for correspondence, and a simple approach that achieves SOTA pose-estimation accuracy.
4 Experiment
Experiments show that VLM3 achieves strong performance across depth estimation, object-level 3D understanding, pixel correspondence, and camera pose estimation while retaining a simple standard-VLM design. Analyses further indicate that text-based pixel reference, careful data weighting, and data scaling are more important than larger models or architectural complexity.
- Quantitative results: VLM3 improves average metric-depth accuracy from 0.84 to 0.9 using a 4B model, surpassing DepthLM-7B despite roughly half the model size.It also achieves new SOTA on the NuScenes and iBims1 datasets against expert vision models.
- Quantitative results: VLM3 surpasses SpatialRGPT-8B on qualitative and quantitative object-level 3D understanding while preserving the base architecture without extra object-reference encoders.The model supports both spatial relationships and metric-scale object properties.
- Quantitative results: VLM3 reduces baseline-VLM pixel-correspondence EPE by an order of magnitude and achieves lower EPE than DKM and RoMa.It remains behind UFM, leaving room for improvement through further scaling and data-mixture tuning.
- Quantitative results: VLM3 improves baseline-VLM camera-pose accuracy and reaches 94.0 versus DA3-Giant’s 94.7, surpassing models including VGGT and MapAnything.It predicts rotation and translation directions as well as metric-scale translation distance.
- Ablation analyses: Text-based pixel reference performs similarly to visual prompting under the same model and data, supporting a simpler and more scalable training interface.The comparison uses two models trained on 8M images plus 1 QA per image.
- Ablation analyses: Careful data weighting improves depth accuracy from 0.84 to 0.88 with dataset-size weighting and to 0.9 with VLM3 weighting, whereas naive scaling performs worse.At 26M images, data scaling remains more important than model scaling, and larger models can overfit.
5 Conclusion
VLM3 presents a minimal, scalable method showing that standard VLMs can learn accurate 3D understanding across diverse tasks using standard architecture and text-based SFT. It matches expert vision models consistently and suggests a path toward generalist 3D foundation models.
- Conclusion: VLM3 uses a minimal design to establish VLMs as native 3D learners.The method is described as scalable and minimal.
- Conclusion: With standard architecture and text-based SFT, VLMs achieve accurate 3D understanding across highly diverse tasks.The conclusion emphasizes that no specialized architecture is required for this broad 3D capability.
- Conclusion: VLM3 consistently matches expert vision models and offers a route toward generalist 3D foundation models.The authors highlight the method’s simplicity, flexibility, and scalability as enabling factors.
Appendix · A Further Implementation Details
The appendix specifies the implementation setup and training-data construction used across tasks. It details optimization, distributed-training choices, multiview sampling, and scene-level evaluation isolation.
- A Further Implementation Details: Tables 4 and 5 report task-specific hyperparameters and training-data statistics.
- A Further Implementation Details: Training uses cosine learning-rate decay with linear warmup at a 0.1 warmup ratio.
- A Further Implementation Details: The optimizer is AdamW with the default settings from the Transformers library.
- A Further Implementation Details: Training employs FSDP hybrid shard, gradient clipping of 0.02, gradient checkpointing, bfloat16, and Flash Attention 2.
- A Further Implementation Details: For multiview tasks, including pixel correspondence and camera pose estimation, image pairs are randomly sampled with > 25% covisibility.
- A Further Implementation Details: Evaluation holds out 30 ScanNet++ scenes so that evaluation data come from unseen scenes.