Source-linked AI summary
Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Kaixuan Wang, Hao Chen, Gang Yu, Chunhua Shen, Shaojie Shen
TL;DR
Monocular models struggle to recover metric depth across camera settings and lack large-scale supervision for surface normals. Metric3D v2 addresses these gaps with canonical camera transformation and joint depth-normal optimization, achieving strong zero-shot performance across depth and normal benchmarks and metric 3D reconstruction from internet images.
Problem
Existing monocular methods lack robust, generalizable geometry foundation models, while varying focal lengths create metric-depth ambiguity and normal labels remain scarce.
Method
Metric3D v2 combines canonical camera-space transformation for metric-depth recovery with joint depth-normal optimization that transfers knowledge from large-scale depth annotations.
Results
The models rank first across diverse depth and surface-normal benchmarks and reconstruct metric 3D from randomly collected internet images.
Takeaways & Limitations
Metric3D v2 supports zero-shot monocular metric perception and downstream applications including metrology, monocular SLAM, and large-scale 3D reconstruction.
Takeaways & Limitations
ScanNet is a non-zero-shot testing dataset for the ViT models.
Abstract
from arXiv · showhide
We introduce Metric3D v2, a geometric foundation model for zero-shot metric depth and surface normal estimation from a single image, which is crucial for metric 3D recovery. While depth and normal are geometrically related and highly complimentary, they present distinct challenges. SoTA monocular depth methods achieve zero-shot generalization by learning affine-invariant depths, which cannot recover real-world metrics. Meanwhile, SoTA normal estimation methods have limited zero-shot performance due to the lack of large-scale labeled data. To tackle these issues, we propose solutions for both metric depth estimation and surface normal estimation. For metric depth estimation, we show that the key to a zero-shot single-view model lies in resolving the metric ambiguity from various camera models and large-scale data training. We propose a canonical camera space transformation module, which explicitly addresses the ambiguity problem and can be effortlessly plugged into existing monocular models. For surface normal estimation, we propose a joint depth-normal optimization module to distill diverse data knowledge from metric depth, enabling normal estimators to learn beyond normal labels. Equipped with these modules, our depth-normal models can be stably trained with over 16 million of images from thousands of camera models with different-type annotations, resulting in zero-shot generalization to in-the-wild images with unseen camera settings. Our method enables the accurate recovery of metric 3D structures on randomly collected internet images, paving the way for plausible single-image metrology. Our project page is at https://JUGGHM.github.io/Metric3Dv2.
1 INTRODUCTION
Metric3Dv2 addresses zero-shot monocular metric depth and surface normal estimation by resolving camera-dependent metric ambiguity and exploiting metric-depth knowledge for normal learning. Trained on 16 million images, it achieves state-of-the-art performance across more than 16 benchmarks and supports metric 3D recovery from internet images.
- Motivation: Metric depth and surface normals provide complementary 3D representations: depth captures metric scale, while normals preserve local geometry without metric ambiguity.Both representations are therefore important for metric 3D recovery from a single image.
- Metric depth: The method addresses metric-depth ambiguity across camera settings with a canonical camera transformation that enables large-scale zero-shot metric-depth training.The approach analyzes pixel size, focal length, and sensor size, identifying focal length as critical for accurate metric recovery.
- Large-scale training: The model trains stably on 16 million images from 16 datasets spanning diverse scenes, camera models, and annotation categories.The training data include indoor and outdoor scenes, real and synthetic data, and tens of thousands of different cameras.
- Surface normals: A joint depth-normal optimization module distills metric-depth knowledge to learn surface normals on large-scale datasets without normal annotations.This weak supervision helps prevent normal-estimator collapse and supports generalization to large-scale unlabeled data.
- Results: Metric3Dv2 ranks first across a wide variety of depth and surface normal benchmarks and enables high-quality metric 3D structure recovery in the wild.The model also benefits downstream tasks including mono-SLAM, 3D scene reconstruction, and metrology.
2 RELATED WORK
Related work spans single-image 3D reconstruction, supervised monocular depth and surface-normal estimation, and iterative geometric refinement. Prior methods address object reconstruction, metric-depth generalization, normal-based geometry, and refinement, but face limitations involving detail, camera variation, labels, computation, or coupling.
- 3D reconstruction from a single image: Single-image 3D reconstruction methods generate high-fidelity models across diverse objects, but must improve detail recovery, memory-efficient representations, and generalization.Examples include cars, planes, tables, and human bodies.
- Supervised monocular depth estimation: Supervised monocular depth methods regress or discretize continuous depth, while metric recovery remains challenged by diverse scenes and camera settings.Relative-depth datasets improve scene generalization but lose geometric structure information.
- Deep iterative refinement for geometry: Iterative refinement supports coarse-to-fine geometry estimation, but existing approaches can be computationally expensive or optimize depth and normals independently.Prior joint depth-normal optimization requires multi-view priors and per-sample post-optimization, whereas the described approach tightly couples the two quantities.
3 METHOD
Metric3D v2 resolves monocular metric-depth ambiguity by transforming images or labels into a canonical camera space, then jointly optimizes metric depth and surface normals. Depth-normal consistency and large-scale depth annotations extend normal estimation beyond limited normal-label supervision.
- Metric Ambiguity: Unknown focal length causes metric ambiguity because different camera-distance combinations can produce identical imaging sizes, whereas sensor-size variation does not affect metric-depth estimation.The method therefore assumes known focal lengths for training and test images.
- Canonical Space Transformation: Canonical camera space transformation uses camera intrinsics to normalize training data, enabling direct metric-depth prediction and de-canonicalization at inference.The method offers label transformation (CSTM label) and input-image resizing (CSTM image) variants.
- Joint Depth-Normal Optimization: The pipeline predicts canonical metric depth and metric-agnostic normals, supervises depth with transformed ground truth, and trains normals using ground-truth normals when available plus depth-normal consistency.At inference, predicted depth is restored to the original camera space using camera intrinsics.
- Training Objectives: Random proposal normalization loss improves depth estimation by applying normalization-based supervision to randomly cropped ground-truth-depth patches.The loss is introduced alongside canonical-space supervision and depth-normal consistency.
4 EXPERIMENTS
Metric3Dv2 is trained on over 16 million samples from diverse RGB-D datasets and achieves leading zero-shot metric depth and surface-normal performance across unseen benchmarks. Qualitative and reconstruction experiments further demonstrate detailed predictions and promising zero-shot metric 3D recovery.
- Experimental setup: Training uses over 16 million samples from 16 RGB-D datasets, including approximately 10 million normal annotations spanning diverse indoor and outdoor scenes.The main training run uses AdamW, balanced dataset sampling, 48 A100 GPUs, and 800k iterations.
- Metric depth evaluation: Metric3Dv2 achieves the best zero-shot metric-depth results on NYUv2 and KITTI, surpassing overfitting and robust monocular methods, while fine-tuning ranks first.Table 1 compares zero-shot and fine-tuned performance; the reported ranking is supported by the table description and accompanying results discussion.
- Surface normal evaluation: The model ranks first across NYUv2, iBIMS-1, and ScanNet surface-normal benchmarks in both zero-shot and fine-tuning comparisons.Table 2 reports the benchmark-wide ranking, with zero-shot testing and post-fine-tuning distinguished.
- Affine-invariant depth evaluation: Metric3Dv2 significantly outperforms prior affine-invariant depth methods and establishes new state-of-the-art results on five zero-shot transfer benchmarks after scale-shift alignment.The affine-invariant benchmark follows the standard manual scale-and-shift alignment protocol.
- Metric depth evaluation: On five unseen metric-depth benchmarks, Metric3Dv2 is evaluated against state-of-the-art methods using indoor and outdoor models across varied scenes and camera models.The experiment uses unseen datasets spanning rooms, buildings, and driving scenes, with varied camera models.
- Qualitative results and applications: Qualitative results show finer depth details, improved foreground/background separation, and more detailed surface normals than competing methods, while zero-shot reconstructions contain less noise and fewer outliers.These comparisons cover multiple indoor, outdoor, and in-the-wild datasets, as well as unseen NYUv2 scenes for metric 3D reconstruction.
5 CONCLUSION
Metric3D v2 introduces geometric foundation models for zero-shot monocular metric depth and surface normal estimation, addressing camera-induced depth ambiguity and limited outdoor normal labels. Trained on over 16 million samples from more than 10,000 cameras, the models support robust metric 3D reconstruction and metrology from single internet images.
- 5 CONCLUSION: The proposed models address depth ambiguity from varying focal lengths with canonical camera space transformation and limited outdoor normal labels with joint depth-normal optimization.These modules target the distinct challenges of metric depth and surface normal estimation.
- 5 CONCLUSION: Zero-shot evaluations demonstrate the method’s effectiveness and robustness, while its models reconstruct metric 3D and enable metrology from randomly collected internet images.The downstream capability is metric 3D recovery from a single view.
Supplementary Materials for Metric3D v2: A Versatile Monocular
This section is the supplementary material for Metric3D v2, a geometric foundation model addressing zero-shot metric depth and surface normal estimation. The document is identified as arXiv version 4, dated 3 January 2025.
- The supplementary materials accompany Metric3D v2, titled “A Versatile Monocular Geometric Foundation Model.”
- The work concerns zero-shot metric depth estimation.
- The work also concerns surface normal estimation.
- The document is arXiv:2404.15506v4, dated 3 January 2025.
1 Details for Models
The models combine ConvNeXt encoders, AdaBins decoders, multiscale ConvGRU refinement, and projection heads for depth and normal updates. Training uses large multi-dataset collections, while evaluations compare resources, zero-shot performance, canonical-space settings, and qualitative predictions.
- Model architecture: The ConvNet models use ImageNet-22k-pretrained ConvNeXt encoders, AdaBins decoders, 256 depth bins over [0.3m, 150m], four flip connections, and an hourglass head.The added connections merge low-level encoder features, while the hourglass subnetwork enhances background predictions.
- Resource comparison: The model families trade performance against deployment cost: larger ViT backbones perform better, while classical encoder-decoder models run faster than diffusion models.Resource measurements use original PyTorch implementations on an NVIDIA A100 40G GPU without engineering optimization; Table 1 covers speed, memory, and benchmark performance.
- Model architecture: The update block uses three multiscale ConvGRU sub-blocks to refine hierarchical features, followed by depth and normal projection heads that predict updates.Refinement proceeds from coarse to fine scales, and each projection head uses two linear layers with a sandwiched ReLU activation.
- Data and evaluation: Over 16M images from 18 public datasets support training, including an 8M-image, 11-dataset collection for ConvNeXt-backbone models.Testing uses high-quality sensor data and provided camera intrinsics for canonical-space transformation.
- Canonical-space ablation: A 1000-pixel canonical focal length achieves the lowest depth error among the tested settings and is used in subsequent experiments.The ablation compares 250, 500, 1000, 1500, and 2000-pixel canonical focal lengths using absolute relative error.
- Qualitative comparison: Qualitative comparisons show finer-grained depth and normal maps with fewer artifacts than the cited competing methods across indoor and outdoor visualizations.Comparisons include ZoeDepth, Bae et al., Omnidata, AdaBins, and NewCRFs using ViT-g and ConvNeXt-L CSTM label models.