Source-linked AI summary
Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image
Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, Chunhua Shen
TL;DR
Single-image metric 3D reconstruction is limited by camera-dependent metric ambiguity: metric methods do not generalize across cameras, while affine-invariant methods lose real-world scale. Metric3D introduces canonical camera space transformation and trains on over 8 million images from diverse cameras, achieving state-of-the-art results on 7 zero-shot benchmarks and supporting metric reconstruction in the wild. Its metric recovery also benefits applications including metrology and monocular SLAM mapping.
Problem
Existing monocular metric-depth methods handle limited camera models, whereas large mixed-dataset methods learn affine-invariant depths that cannot recover real-world metrics.
Method
Metric3D uses canonical camera space transformation to address focal-length-induced ambiguity and combine mixed data from thousands of cameras for metric-depth training.
Results
Metric3D achieves state-of-the-art performance on 7 zero-shot benchmarks and recovers metric 3D structures from randomly collected Internet images.
Takeaways & Limitations
The model supports plausible single-image metrology and improves downstream tasks including monocular SLAM, 3D scene reconstruction, and dense metric mapping.
Takeaways & Limitations
The approach assumes that the focal length of each training and test image is available.
Abstract
from arXiv · showhide
Reconstructing accurate 3D scenes from images is a long-standing vision task. Due to the ill-posedness of the single-image reconstruction problem, most well-established methods are built upon multi-view geometry. State-of-the-art (SOTA) monocular metric depth estimation methods can only handle a single camera model and are unable to perform mixed-data training due to the metric ambiguity. Meanwhile, SOTA monocular methods trained on large mixed datasets achieve zero-shot generalization by learning affine-invariant depths, which cannot recover real-world metrics. In this work, we show that the key to a zero-shot single-view metric depth model lies in the combination of large-scale data training and resolving the metric ambiguity from various camera models. We propose a canonical camera space transformation module, which explicitly addresses the ambiguity problems and can be effortlessly plugged into existing monocular models. Equipped with our module, monocular models can be stably trained with over 8 million images with thousands of camera models, resulting in zero-shot generalization to in-the-wild images with unseen camera settings. Experiments demonstrate SOTA performance of our method on 7 zero-shot benchmarks. Notably, our method won the championship in the 2nd Monocular Depth Estimation Challenge. Our method enables the accurate recovery of metric 3D structures on randomly collected internet images, paving the way for plausible single-image metrology. The potential benefits extend to downstream tasks, which can be significantly improved by simply plugging in our model. For example, our model relieves the scale drift issues of monocular-SLAM (Fig. 1), leading to high-quality metric scale dense mapping. The code is available at https://github.com/YvanYin/Metric3D.
1. Introduction
Single-image metric 3D reconstruction remains difficult because existing methods either require matched camera models or recover depth only up to an unknown scale and shift. The paper addresses this with canonical camera transformations and large-scale mixed-camera training, enabling zero-shot metric reconstruction.
- Single-image reconstruction is difficult because multi-view geometry cannot be applied without additional views or a prior.
- Affine-invariant methods generalize across diverse data but recover depth only up to an unknown offset and scale.
- Metric-depth methods can be accurate on a single dataset but struggle to generalize across unseen cameras and scenes.
- The proposed canonical camera transformation addresses metric ambiguity caused by varying camera parameters, especially focal length.The method transforms camera-dependent inputs and supervision into a canonical camera space for metric prediction.
- The model achieves state-of-the-art performance on 7 zero-shot benchmarks and supports metric 3D recovery in the wild and downstream applications.Reported applications include metrology, monocular SLAM, and 3D scene reconstruction.
2. Related Work
Prior monocular 3D methods address either object-specific reconstruction, relative depth, or affine-invariant depth, while metric recovery remains challenged by camera variation. Large-scale mixed-data methods improve scene generalization but discard absolute metric information.
- Object reconstruction methods can produce detailed models for specific classes but rely on class- or instance-specific priors and do not target full-scene reconstruction.
- Monocular depth methods include metric, relative, and affine-invariant depth prediction categories.
- Metric-depth methods require matching camera intrinsics during training and testing, limiting the size and diversity of their training datasets.
- Relative depth improves generalization but loses geometric structure information.
- Affine-invariant methods mix large-scale data to improve scene generalization, but recover 3D scenes only up to unknown scale and shift.
3. Method
The method resolves monocular metric ambiguity by transforming images or depth labels into a canonical camera space, then applying a standard depth model and reversing the transformation at inference. It combines this strategy with mixed-camera training and local proposal normalization.
- 3.2. Canonical Camera Transformation: The canonical-space pipeline transforms the input, predicts canonical metric depth, supervises it with transformed ground truth during training, and converts predictions back using camera intrinsics.Random cropping changes field of view and optical center but does not introduce metric ambiguity.
- 3.1. Metric Ambiguity Analysis: Metric depth recovery depends critically on focal length, while sensor size and pixel size do not affect metric estimation under the stated analysis.The method assumes focal length is available because estimating it from a single image remains challenging and ill-posed.
- 3.1. Metric Ambiguity Analysis: Two cameras with f̂1 = 2f̂2 at distances d1 = 2d2 can produce the same imaging size, creating identical appearances with different metric-depth labels.This appearance-supervision conflict motivates canonical camera transformation.
- 3.2. Canonical Camera Transformation: CSTM maps inputs or ground-truth depth labels to a canonical camera model, enabling a depth estimator to predict metric depth in canonical space before de-canonicalization.For image transformation, images and labels are resized; for label transformation, depths are scaled while the input image remains unchanged.
- 3.3. Training: Training mixes 11 datasets containing over 10K cameras, using paired intrinsics for canonical transformation and a random proposal normalization loss to enhance local depth contrast.The objective combines LPWN, LVNL, Lsilog, and LRPNL; proposal crops number 32 and cover 0.125 to 0.5 of the original image size.
4. Experiments
Experiments show that Metric3D generalizes metric depth prediction across unseen datasets and camera models, supports accurate 3D reconstruction and downstream applications, and benefits from canonical transformation and random proposal normalization.
- 4.1. Zero-shot Generalization Test: The model achieves comparable performance with state-of-the-art methods on unseen NYUv2 and KITTI metric-depth benchmarks.The same model is evaluated without fine-tuning or metric adjustment.
- 4.1. Zero-shot Generalization Test: On six unseen benchmarks with diverse indoor, outdoor, and camera settings, existing metric-depth methods degrade noticeably, whereas Metric3D remains more robust.The compared methods use NYUv2 and KITTI models separately, while Metric3D uses one model for zero-shot testing.
- 4.1. Zero-shot Generalization Test: Metric3D outperforms prior affine-invariant depth methods by a large margin on most of five zero-shot datasets after scale-and-shift alignment.The comparison follows affine-invariant benchmark settings.
- 4.2. Applications Based on Our Method: Metric3D reconstructs more accurate 3D scenes than competing methods on nine unseen NYUv2 scenes, without per-frame scale alignment.Its metric depths are fused using provided camera poses; qualitative reconstructions contain less noise and fewer outliers.
- 4.3. Ablation Study: Removing CSTM prevents convergence on mixed metric datasets and eliminates zero-shot metric prediction ability, whereas either input or output transformation enables it.The two CSTM variants achieve comparable performance in the ablation.
- 4.2. Applications Based on Our Method: Metric3D depths improve Droid-SLAM trajectory accuracy and enable denser, more accurate metric mapping, with larger gains on KITTI than on ETH3D.The ETH3D scenes are small-scale indoor environments, while KITTI contains larger-scale scenes.
- 4.3. Ablation Study: Using different canonical focal lengths shows that excessively large or small values reduce performance.The experiment identifies sensitivity to the selected canonical focal length.
- 4.3. Ablation Study: Random proposal normalization loss further improves performance, while whole-image scale-shift invariant loss yields only slight improvement.The comparison uses the same baseline on KITTI and NYUv2.
5. Conclusion
The paper addresses single-image metric 3D reconstruction by transforming images and depth labels into a canonical camera space, enabling large-scale mixed-camera training and robust zero-shot evaluation.
- 5. Conclusion: Zero-shot evaluations demonstrate the method’s effectiveness and robustness, including metric reconstruction, in-the-wild metrology, and dense mapping on large-scale scenes.The conclusion highlights applications beyond single-image depth prediction.
6. Appendix
The appendix documents the datasets, supervision sources, testing protocol, reconstruction evaluation, and additional visual results used to assess Metric3D.
- Dataset details: Autonomous-driving datasets provide LiDAR and camera parameters for projected depth supervision, while several stereo or SfM datasets use pseudo-ground-truth depths.Noisy SfM metric information is excluded from the metric-depth loss for Mapillary PSD.
- Evaluation protocol: Testing uses provided camera intrinsics for canonical transformation, and reconstruction evaluation fuses predicted depths with poses before ICP-based point-cloud matching.Meshlab is used to measure structures in reconstructed point clouds.
- Additional visual results: Metric3D obtains consistent fused point clouds across six NuScenes ring cameras despite their different intrinsic parameters.The appendix also reports fewer qualitative depth artifacts than Adabins, NewCRFs, and Omnidata.