Source-linked AI summary
Unlocking Dense Metric Depth Estimation in VLMs
Hanxun Yu, Xuan Qu, Yuxin Wang, Jianke Zhu, Lei Ke
TL;DR
Current VLMs remain limited in complex 3D understanding, especially dense metric depth estimation. DepthVLM adds a lightweight depth head and two-stage unified vision-text training, outperforming existing VLMs and leading pure vision models while improving 3D spatial reasoning.
Problem
Current VLMs perform strongly on 2D tasks but remain limited in complex 3D understanding, which matters for applications such as AR/VR, autonomous driving, and robotics.
Method
DepthVLM attaches a lightweight depth head to the VLM backbone and uses two-stage unified vision-text training for joint dense depth and language prediction.
Results
DepthVLM outperforms existing VLMs and leading pure vision models on metric depth estimation while improving complex 3D spatial reasoning.
Takeaways & Limitations
A single VLM can jointly support dense geometry prediction and multimodal interaction with higher inference efficiency.
Takeaways & Limitations
The reported limitation is that text-dominant single-pixel supervision can cause DepthLM to emit a constant depth value and fail standard VQA evaluation.
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) excel at 2D tasks such as grounding and captioning, yet remain limited in 3D understanding. A key limitation is their text-only supervision paradigm, which under-constrains fine-grained visual perception and prevents the recovery of dense geometry. Prior methods either distill geometry from external vision models, introducing error accumulation, or enable direct prediction with inefficient per-pixel query or coarse token-level outputs. In this paper, we propose DepthVLM, a simple yet effective framework that transforms a single VLM into a native dense geometry predictor while preserving its multimodal capability. By attaching a lightweight depth head to the LLM backbone and training under a unified vision-text supervision paradigm with a two-stage schedule, DepthVLM generates full-resolution depth maps alongside language outputs in a single forward pass. We further introduce a unified indoor-outdoor metric depth benchmark in a VLM-compatible format. Experiments show that DepthVLM significantly outperforms existing VLMs with higher inference efficiency, surpasses leading pure vision models, and improves complex 3D spatial reasoning, moving toward a truly unified multimodal foundation model. The project page is available at https://depthvlm.github.io/
1 Introduction
DepthVLM turns a VLM into a native dense metric-depth predictor by adding a lightweight depth head and jointly generating pixel-level depth maps with language outputs. A two-stage training strategy and unified indoor–outdoor benchmark preserve multimodal capability while enabling efficient evaluation and improved 3D reasoning.
- Motivation: Current VLMs perform strongly on 2D visual reasoning and captioning but remain limited in complex 3D understanding needed for AR/VR, autonomous driving, and robotics.
- Motivation: Text-only supervision under-constrains fine-grained visual perception, preventing explicit modeling of dense scene geometry.
- Method: DepthVLM attaches a lightweight depth head to a single VLM backbone, enabling joint generation of dense pixel-level depth maps and language responses.
- Method: Its two-stage strategy first trains only the depth head, then fine-tunes the full model end-to-end to preserve the VLM’s original multimodal capability.
- Benchmark: DepthVLM-Bench unifies public indoor and outdoor depth datasets in a VLM-compatible format for training and direct comparison with pure vision models.
- Results: Experiments show that DepthVLM outperforms existing VLMs with higher inference efficiency, surpasses state-of-the-art pure vision models on metric depth estimation, and improves 3D spatial reasoning.
2 Related Work
Prior depth-estimation methods face domain-generalization and metric-scale limitations, while spatial-enhanced and geometry-generative VLMs trade off data requirements, scene coverage, accuracy, or inference efficiency. These approaches motivate unified dense metric geometry prediction from RGB inputs.
- Dense Metric Depth Estimation: Dense metric depth estimation recovers per-pixel absolute depth from RGB images, but early single-domain methods specialize to indoor or outdoor scenes with limited cross-domain generalization.MiDaS and DPT improve robustness across diverse datasets but provide relative depth without metric scale.
- Spatial-Enhanced VLMs: Spatial-enhanced VLMs inject explicit 3D data such as point clouds, voxels, or depth maps into LLMs, improving 3D VQA while relying on sparse, costly data and mainly indoor scenes.These methods use projectors to connect sensor-derived geometric signals with LLMs.
- Geometry-Generative VLMs: Geometry-generative VLMs directly generate dense geometry from RGB inputs, but existing methods lag behind pure vision models, focus on relative depth, or predict only one pixel per inference.Multi-SpatialMLLM and Seed1.5-VL explore pixel-level metric depth; G2VLM uses Mixture-of-Experts modeling for relative depth; DepthLM matches advanced vision models in accuracy but predicts one pixel per inference.
3 Methodology
DepthVLM turns a single VLM into a unified dense metric-depth and language predictor by adding a lightweight depth head and using coordinated architecture, training, and data strategies. Focal-length normalization and the diverse DepthVLM-Bench support cross-dataset generalization and consistent evaluation across indoor and outdoor domains.
- Unified Architecture: DepthVLM augments a standard VLM with a lightweight DPT-style depth head to jointly produce dense metric depth maps and language responses.The model uses a unified foundation-model backbone for low-level geometry prediction and high-level multimodal understanding.
- Unified Architecture: The depth head combines three intermediate ViT feature maps with LLM final hidden states at image-token positions, forming visual and vision-language representations.The resulting features capture increasing visual abstraction while incorporating contextualized vision-language information.
- Unified Architecture: Because VLM visual tokens are already downsampled, DepthVLM constructs an upsampled bottom-up pyramid that assigns higher spatial resolution to earlier ViT layers.Each feature is projected with a 1×1 convolution and resampled to a layer-specific resolution to preserve finer spatial details.
- Two-Stage Training: DepthVLM trains in two stages: first the depth head alone with SILog supervision, then the unfrozen LLM backbone end-to-end with combined language-modeling and depth losses.This schedule initializes depth prediction while reducing disruption to pretrained multimodal knowledge, then integrates geometry with language reasoning.
- Camera Normalization: Focal-length normalization rescales images and depth maps to a unified virtual camera, removing cross-dataset scale discrepancies and enabling focal-invariant mappings.Given focal length f and target focal length fc, the scale is s = fc / f, followed by isotropic bilinear resizing.
- DepthVLM-Bench: DepthVLM-Bench unifies public metric-depth datasets for VLM training and consistent comparison with pure vision models across indoor and outdoor domains.The training split mixes 8 datasets, while the evaluation split covers 9 disjoint datasets and samples 1k images and 10 pixels per image.
4 Experiment
DepthVLM outperforms existing VLMs and leading pure vision models on dense metric depth estimation while preserving multimodal understanding. Ablations show that multi-scale feature fusion, two-stage training, focal-length normalization, and direct dense decoding are central to its accuracy, stability, and efficiency.
- Comparison with Other VLMs: General-purpose VLMs perform poorly on metric depth, with Qwen3-VL-32B reaching δ1 = 0.21 and GPT-5.5 only 0.41 on average.Spatially enhanced VLMs also underperform a constant baseline, revealing a substantial gap to reliable 3D understanding.
- Comparison with Pure Vision Models: DepthVLM significantly outperforms most specialized pure vision models while retaining strong multimodal capabilities in a unified model.It is evaluated against pure vision methods on the same sampled pixels for a fair indoor-outdoor comparison.
- Evaluation on General Visual Benchmarks: Dense geometry prediction does not compromise multimodal understanding, as DepthVLM matches its original VLM backbones and improves OCRBench and POPE.DepthLM instead often overfits to text-heavy supervision and loses general-purpose capabilities.
- Evaluation on Spatial Reasoning Tasks: Native dense geometry prediction improves complex spatial reasoning, supporting downstream 3D tasks beyond metric depth estimation.The results indicate that strong dense geometry prediction provides a foundation for high-level spatial reasoning in VLMs.
- Qualitative Visualizations: DepthVLM produces finer structural detail and stronger semantic consistency than Youtu-VL and DepthLM in depth maps and reconstructed 3D point clouds.Youtu-VL generates noisy, fragmented point clouds, while DepthLM preserves semantic coherence but loses fine structural details.
- Ablation and Efficiency Analysis: Ablations identify multi-scale ViT–LLM feature fusion, full two-stage training, focal-length normalization, and direct pixel-aligned decoding as key design choices.The combined features integrate semantic information with fine-grained geometry; two-stage training balances depth and multimodal performance, normalization resolves scale ambiguity, and direct decoding avoids per-pixel queries and sparse-patch upsampling.
5 Conclusion and Limitations
DepthVLM unifies dense geometry prediction with high-level multimodal understanding by adding a lightweight depth head and using two-stage unified vision-text supervision. Experiments report leading performance across diverse datasets.
- DepthVLM jointly supports low-level dense geometry prediction and high-level multimodal understanding as a unified foundation model.
- A lightweight depth head is integrated into the VLM backbone and trained with a two-stage strategy under unified vision-text supervision.
- DepthVLM enables geometry prediction and language response in a single forward pass.
- Extensive experiments show leading performance across diverse datasets.
Technical Appendices and Supplementary Material · A Statistics of DepthVLM-Bench
The appendix reports DepthVLM-Bench’s data composition for training and evaluation. The benchmark combines diverse indoor and outdoor public datasets, with redundancy reduction for training and balanced sampling plus additional indoor sets for evaluation.
- A Statistics of DepthVLM-Bench: DepthVLM-Bench data composition is summarized in Tables 9 and 10.Table 9 covers training-data statistics, while Table 10 covers benchmark-data statistics.
- A Statistics of DepthVLM-Bench: Training samples span autonomous-driving and indoor datasets, including Argoverse2, Waymo, DDAD, NuScenes, ScanNet++, Taskonomy, HM3D, and Matterport3D.The training set samples from the training splits of multiple public benchmarks.
- A Statistics of DepthVLM-Bench: Uniform sampling reduces redundancy among near-duplicate RGB video frames in the training data.The paper notes that many RGB images are extracted from videos and therefore contain highly redundant frames.
- A Statistics of DepthVLM-Bench: Evaluation uses validation or test splits from the same public datasets to cover both outdoor and indoor scenarios.The evaluation benchmark is assembled from the corresponding splits of the public datasets used for training-data construction.
- A Statistics of DepthVLM-Bench: Each evaluation dataset contributes around 1K images when possible, supporting a balanced and consistent protocol across sources.The approximate 1K-image contribution applies when dataset sampling permits it.
- A Statistics of DepthVLM-Bench: Additional indoor benchmarks—ETH3D, sunRGBD, IBims-1, and NYUv2—extend evaluation across diverse environments and varying d.These standard indoor benchmarks are included to further assess generalization.
B Evaluation of VLMs on Metric Depth Estimation
The evaluation protocol tests VLM metric depth estimation by marking a query pixel with a visible red arrow and prompting models to predict its camera-axis distance in meters. Images are standardized across models by enlarging the marker and limiting the longest edge to 1024 pixels.
- Evaluation Protocol: The protocol asks each VLM to estimate the metric depth, in meters, of a pixel indicated by a red arrow on a single RGB image.Metric depth is defined as the distance along the camera’s optical axis from the marked point to the camera.
- Visual Marker Size: 20-pixel markers replace DepthLM’s 5-pixel default because untrained off-the-shelf VLMs cannot reliably detect the smaller marker.This prevents evaluation failures caused by marker detection rather than depth estimation.
- Image Resolution Handling: Images with a longest edge over 1024 pixels are downscaled to that limit before marker insertion, while smaller images remain unchanged.The procedure standardizes input conditions and preserves marker visibility across datasets and models.
- Evaluation Protocol: All VLMs are evaluated under identical conditions, with provided scripts used for DepthLM and Youtu-VL because both were trained specifically for metric depth estimation.The prompt requires only a depth number in meters, rounded to two decimal places.
C Asset License and Consent
The paper promotes openness and transparency by providing licenses and URLs for all public datasets and benchmarks used. These resources cover multimodal tasks ranging from depth estimation to hallucination evaluation.
- Openness and Transparency: The authors provide licenses and URLs for all public datasets and benchmarks used in the study.This openness and transparency effort addresses concerns about misinformation.
- Resource Coverage: The resources support depth estimation, general multimodal understanding, spatial reasoning, document understanding, and multi-image reasoning.
- Resource Coverage: They also cover visual grounding and hallucination evaluation, with the complete resource list summarized in Table 11.