Source-linked AI summary
DepthLM: Metric Depth From Vision Language Models
Zhipeng Cai, Ching-Feng Yeh, Hu Xu, Zhuang Liu, Gregory Meyer, Xinjie Lei, Changsheng Zhao, Shang-Wen Li, Vikas Chandra, Yangyang Shi
TL;DR
VLMs are flexible but struggle with 3D understanding, whereas pure vision models achieve strong metric depth using specialized architectures and losses. DepthLM addresses this gap with visual prompting, intrinsic-conditioned augmentation, and sparse text-based fine-tuning, achieving over 2x accuracy gains over advanced VLMs and matching expert pure vision models while extending to diverse 3D tasks.
Problem
VLMs remain far behind pure vision models on metric depth despite their flexibility, motivating expert-level 3D understanding without architecture or loss changes.
Method
DepthLM combines visual prompting, intrinsic-conditioned augmentation, and sparse text-based supervised fine-tuning with standard VLMs.
Results
Over 2x δ1 improvement across four indoor and outdoor datasets lets the 3B DepthLM surpass most larger VLMs and match advanced pure vision models.
Takeaways & Limitations
A single standard VLM framework can support accurate metric depth and diverse 3D tasks without dense prediction heads or complex losses.
Abstract
from arXiv · showhide
Vision language models (VLMs) can flexibly address various vision tasks through text interactions. Although successful in semantic understanding, state-of-the-art VLMs including GPT-5 still struggle in understanding 3D from 2D inputs. On the other hand, expert pure vision models achieve super-human accuracy in metric depth estimation, a key 3D understanding task. However, they require task-specific architectures and losses. Such difference motivates us to ask: Can VLMs reach expert-level accuracy without architecture or loss change? We take per-pixel metric depth estimation as the representative task and show that the answer is yes! Surprisingly, comprehensive analysis shows that text-based supervised-finetuning with sparse labels is sufficient for VLMs to unlock strong 3D understanding, no dense prediction head or complex regression/regularization loss is needed. The bottleneck for VLMs lies actually in pixel reference and cross-dataset camera ambiguity, which we address through visual prompting and intrinsic-conditioned augmentation. With much smaller models, our method DepthLM surpasses the accuracy of most advanced VLMs by over 2x, making VLMs for the first time comparable with pure vision models. Interestingly, without explicit enforcement during training, VLMs trained with DepthLM naturally avoids over-smoothing, having much fewer flying points at boundary regions than pure vision models. The simplicity of DepthLM also enables a single VLM to cover various 3D tasks beyond metric depth. Our code and model will be released at the link below.
1 Introduction
DepthLM addresses the gap between flexible VLMs and expert pure vision models by using standard text-based supervision for metric depth without architectural or loss changes. Visual prompting, intrinsic-conditioned augmentation, sparse labels, and a curated benchmark enable strong 3D understanding and broader task coverage.
- 1 Introduction: DepthLM turns VLMs into strong metric-depth estimators without adding dense prediction heads or changing the architecture or loss.The method uses pixel-level metric depth as a representative 3D understanding task.
- 1 Introduction: Visual prompting replaces text coordinates with image markers, while intrinsic-conditioned augmentation unifies focal lengths to resolve pixel-reference and camera-scale ambiguity.These components address fine-grained location errors and cross-dataset metric-scale differences.
- 1 Introduction: Sparse text-based supervised fine-tuning can learn strong 3D understanding from as little as one labeled pixel per training image.The paper also finds reinforcement learning can learn 3D understanding, although supervised fine-tuning is more efficient.
- 1 Introduction: With a 3B model, DepthLM achieves over 2x δ1 improvement across four indoor and outdoor datasets and surpasses most larger VLMs, including GPT-5.It also outperforms DepthPro and Metric3Dv2, matching advanced pure vision models.
- 1 Introduction: DepthLM extends beyond metric depth to unified VLM training for diverse 3D tasks involving reasoning, multiple points, and multiple images.The framework is evaluated through the DepthLMBench suite, which compares VLMs directly with pure vision models.
2 Related Work
Prior metric-depth systems rely on task-specific architectures, camera handling, or specialized pipelines, while earlier VLM work focuses mainly on object-level 3D understanding and text-based supervision. These differences leave direct comparison between VLMs and expert metric-depth models unclear.
- 2 Related Work: Metric-depth methods use separate indoor and outdoor heads, camera-intrinsic normalization, or architectures that predict intrinsics to establish a unified metric scale.The paper positions camera ambiguity as a central issue for combining data from different cameras.
- 2 Related Work: Existing VLM research on 3D understanding therefore does not establish how closely VLMs can approach advanced pure vision models on metric depth.The paper addresses this gap with direct metric-depth comparisons.
- 2 Related Work: Earlier VLM approaches distill outputs from pure vision models or use language models to generate spatial reasoning prompts, but such pipelines can accumulate training-data errors.Their object-level focus also makes comparison with pure vision metric-depth models difficult.
3 DepthLM
DepthLM analyzes why VLMs lag on metric depth and addresses pixel reference and camera ambiguity without changing the model architecture or loss. Its sparse-label, text-based SFT framework achieves strong metric depth and extends to multiple 3D tasks.
- 3 DepthLM: DepthLMBench combines public indoor and outdoor datasets, while DepthLM uses visual prompting, focal-length augmentation, sparse labels, and text-based SFT for unified 3D training.The framework is also applied to principal-axis distance, speed, time, two-point distance, and metric-scale camera pose.
- 3.1 Prompt Design: Marker-based visual prompting substantially outperforms text-based pixel coordinates, with a 0.15 indoor accuracy gap on ScanNet++ and robustness across marker shapes.VLMs struggle to map textual coordinates to pixel locations, especially around indoor boundaries and occlusions.
- 3.2 Loss: SFT and RL achieve similar accuracy at equal sample counts, but SFT is more efficient because GRPO requires 8–16 times more per-sample compute.Negative L1 is identified as a simple effective GRPO reward.
- 3.3 Mix-data Training: Intrinsic-conditioned augmentation resolves cross-camera ambiguity by unifying focal lengths, whereas VLMs otherwise struggle to distinguish cameras without architecture changes.Accuracy remains stable across a wide range of unified focal lengths and changes by less than 3%.
- 3.3 Mix-data Training: One labeled pixel per training image already exceeds 0.8 δ1, and image diversity matters more than label density for VLM-based 3D learning.This sparse-label setting later matches pure vision model accuracy.
4 Results
DepthLM brings VLM metric-depth accuracy close to expert pure vision models while retaining a unified, text-interactive architecture. It also extends to multiple 3D tasks and produces sharper object boundaries without explicit smoothing control.
- Main results: Below 0.4 δ1 leaves GPT-5 and other advanced VLMs behind DepthLM, which achieves over 2x improvement despite being orders of magnitude smaller.Table 1 reports the VLM comparison, while the 3B model is contrasted with much larger baselines.
- Scaling and architecture: Over 0.9 δ1 on ibims1 and NYUv2 is reached by the 7B model, while larger models help but are not necessary for accurate 3D understanding.A separately fine-tuned 12B architecture also has reasonable, though slightly lower, accuracy, supporting cross-architecture applicability.
- Visualization: Stable point-cloud scale remains close to ground truth across indoor and outdoor scenes, while DepthLM avoids boundary flying points without post-processing.Pure vision models produce smoother surfaces but can over-smooth boundaries; DepthLM instead preserves clearer object separation with somewhat more smooth-region noise.
- Multi-task results: More than 0.8 δ1 across diverse 3D tasks lets DepthLM outperform baselines by over 3.8x with one unified architecture and training framework.The multi-task evaluation includes metric depth and pose-related tasks, with pose trained and evaluated on specified datasets.
5 Conclusion
The paper concludes that VLM 3D-understanding deficits stem from pixel reference and camera ambiguity rather than missing dense heads or complex losses. DepthLM addresses these issues with visual prompting and intrinsic-conditioned augmentation, while leaving finer-grained improvements for future work.
- Conclusion: DepthLM uses visual prompting and intrinsic-conditioned augmentation to make standard VLMs strong pixel-level metric-depth estimators.The framework supports text-based supervised fine-tuning without adding dense prediction heads or complex training losses.
- Conclusion: The authors identify finer-grained data filtering and diverse complementary-task training as future directions for improving generalization and surpassing pure vision models.These directions are presented as limitations of the paper’s simplest design rather than established results.
A Examples of Rendered Markers and Text-based Pixel Reference
The appendix illustrates the marker prompts used for visual pixel reference and gives the corresponding text-based pixel-reference query. Marker design and size are varied to improve VLM recognition and baseline accuracy.
- Rendered markers: Roughly 5-pixel-wide markers are sufficient for VLMs to recognize the queried pixel location.Three marker types are shown, and different marker designs and sizes were tested for baseline evaluation.
- Text-based reference: Text-based pixel reference asks how far pixel (X, Y) is from the camera in an image with width W and height H.This is the direct coordinate-based query used in the pixel-reference experiment.
B Statistics of DepthLMBench
DepthLMBench reports available and used image counts for each training dataset, with redundant video frames subsampled before training.
- Dataset statistics: DepthLMBench statistics report each dataset’s available-image count alongside the subset used for training.The table format is available images divided by images used.
- Dataset statistics: Argoverse2 and ScanNet++ frames are subsampled because highly similar video images provide little additional training benefit.Only a subset of frames is retained for training from these datasets.
C Cross-Dataset Evaluation for SFT vs GRPO Experiments
Cross-dataset evaluation shows that SFT and GRPO follow similar performance trends to same-dataset evaluation, while the experiments use matched training conditions and specified GRPO settings.
- Cross-Dataset Results: SFT and GRPO show similar accuracy trends when trained on Argoverse2 and evaluated on NuScenes, as in same-dataset evaluation.The cross-dataset experiment tests whether GRPO provides a zero-shot benefit.
- Comparison Setup: SFT is adjusted toward GRPO with batch size 32 and a correspondingly reduced learning rate using square-root scaling.The comparison also reports standard hyper-parameters in Table 5.
E Prompts for Camera Ambiguity Analysis
The camera-ambiguity analysis compares explicit intrinsic prompting with predicted camera-ray information, finding that direct intrinsic values do not help metric-depth prediction.
- Explicit Intrinsic Prompting: The model is prompted with image dimensions and camera intrinsics to condition metric-depth prediction explicitly.The prompt follows a format similar to Seed1.5-VL.
- Predicted Intrinsic Conditioning: The answer includes camera-ray direction and distance before metric depth so predictions can condition on inferred intrinsics.The response describes horizontal and vertical angular offsets together with meters from the camera.
- Analysis Result: Adding exact camera-intrinsic values to the input prompt does not improve performance.The method therefore does not directly predict fx, fy, cx, and cy.
F Hyper-parameters for DepthLM Training
The appendix reports the hyper-parameters used for DepthLM training, including a unified focal length of 750 pixels for the 12B model because of memory limits.
- Training Configuration: The 12B model uses a unified focal length of 750 pixels because of memory constraints.Other SFT hyper-parameters are listed in Table 5.
G DepthLM Multi-task Performance on Each Dataset
DepthLM’s multi-task performance is reported separately for each individual dataset.
- Per-Dataset Evaluation: Table 6 reports DepthLM’s multi-task performance on individual datasets.