PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera

Yuliang Guo, Sparsh Garg, S. Mahdi H. Miangoleh, Xinyu Huang, Liu Ren

arXiv:2501.02464v2cs.CVcs.AIcs.RO

TL;DR

Metric-depth models struggle to generalize to large-FoV fisheye and 360° cameras. DAC trains on perspective images using an ERP-based framework, and reports state-of-the-art large-FoV results, including up to 50% higher δ1 accuracy on indoor fisheye and 360° datasets.

  • Problem

    Zero-shot metric-depth models often perform poorly on large-FoV fisheye and 360° cameras.

    Source: §1 · p. 1

  • Method

    DAC uses an ERP-based framework with pitch-aware Image-to-ERP conversion to generalize a model trained exclusively on perspective images across cameras with varied fields of view.

    Source: §1 · p. 2

  • Results

    DAC reports state-of-the-art zero-shot results on large-FoV datasets, with up to 50% improvement in δ1 accuracy on indoor fisheye and 360° datasets.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    DAC’s framework is intended to keep previously collected 3D data useful across camera types in new applications.

    Source: §6 · p. 8

    Camera orientations may be dataset-provided or estimated, and DAC does not require highly accurate pose estimates for depth estimation.

    Source: §7.2 · p. 12

Abstract

from arXiv · show

While recent depth foundation models exhibit strong zero-shot generalization, achieving accurate metric depth across diverse camera types-particularly those with large fields of view (FoV) such as fisheye and 360-degree cameras-remains a significant challenge. This paper presents Depth Any Camera (DAC), a powerful zero-shot metric depth estimation framework that extends a perspective-trained model to effectively handle cameras with varying FoVs. The framework is designed to ensure that all existing 3D data can be leveraged, regardless of the specific camera types used in new applications. Remarkably, DAC is trained exclusively on perspective images but generalizes seamlessly to fisheye and 360-degree cameras without the need for specialized training data. DAC employs Equi-Rectangular Projection (ERP) as a unified image representation, enabling consistent processing of images with diverse FoVs. Its core components include pitch-aware Image-to-ERP conversion with efficient online augmentation to simulate distorted ERP patches from undistorted inputs, FoV alignment operations to enable effective training across a wide range of FoVs, and multi-resolution data augmentation to further address resolution disparities between training and testing. DAC achieves state-of-the-art zero-shot metric depth estimation, improving $\delta_1$ accuracy by up to 50% on multiple fisheye and 360-degree datasets compared to prior metric depth foundation models, demonstrating robust generalization across camera types.

1. Introduction

Perspective-trained metric-depth models struggle on fisheye and 360° cameras, while undistorting inputs can sacrifice field of view. DAC uses an ERP-based framework to generalize from perspective training data to cameras with varied fields of view.

1. Introduction

  • Prior zero-shot metric-depth models degrade on large-FoV cameras, and perspective-friendly undistortion can lose substantial field of view.The introduction identifies unified camera representation, unseen large-FoV content, varying training image sizes, and train–test resolution mismatch as challenges.Source: §1 · pp. 1–2
  • DAC uses ERP as a shared representation, with pitch-aware Image-to-ERP conversion, FoV alignment, and multi-resolution training to address varied cameras and resolutions.Pitch-angle augmentation projects perspective images into high-distortion ERP regions to simulate observations associated with large-FoV cameras.Source: §1 · p. 2
  • Perspective-trained Metric3Dv2 [20] degrades on fisheye images; the degradation is less pronounced for a limited-FoV undistorted crop or its ERP conversion.The comparison illustrates the challenge motivating a method that retains large-FoV input.Source: Figure 2 · p. 2

2. Related Works

Prior zero-shot metric-depth methods address inconsistent depth scaling, while large-FoV methods have often relied on camera-specific distortions or representations. The paper identifies a lack of a unified model that can learn from mixed perspective datasets and generalize zero-shot to large-FoV images.

2.1. Zero-Shot Monocular Depth Estimation

  • Metric3D and UniDepth canonicalize camera space to address inconsistent scaling, but prior methods lack satisfactory zero-shot performance on large-FoV images.Metric3D uses camera intrinsics for preprocessing; UniDepth estimates and converts intrinsics with a network branch.Source: §2.1 · p. 2

2.2. Depth Estimation from Large FoV Cameras

  • Large-FoV depth methods address position-dependent distortion with deformable CNNs [42, 50, 63], ERP segmentation, and transformers.Transformer approaches improved in-domain performance, but the passage identifies limited large-scale FoV-specific training data as a generalization bottleneck.Source: §2.2 · pp. 2–3
  • No current method described enables one depth model trained on mixed large-scale perspective datasets to generalize zero-shot to ERP or fisheye images.The passage distinguishes this gap from the in-domain performance of transformer-based approaches.Source: §2.2 · p. 3

3. Notations and Preliminaries

The section describes depth scaling for camera conversion and resizing, and ERP as a shared spherical image representation. It also introduces Gnomonic Projection as the mapping used between tangent images and ERP coordinates.

3. Notations and Preliminaries

  • Depth scaling compensates for changes in focal length when converting to a canonical camera model and for apparent object-size changes caused by image resizing.The figure describes focal-length scaling as Ẑ = f̂_x Z / f_x and resizing as Z′ = uZ / u′.Source: §3 · p. 3, Figure 3 · p. 3
  • ERP is a spherical representation in which pixels correspond to latitude and longitude, spanning 180° in latitude and 360° in longitude.ERP height alone defines the space, allowing images with different original fields of view to be converted consistently.Source: §3 · p. 3
  • Gnomonic Projection maps tangent-image coordinates to spherical coordinates using a tangent plane centered on the sphere.The angular distance between the spherical point and the tangent-plane center is represented by c.Source: §3 · p. 3

4. Depth Any Camera

DAC converts images from diverse cameras into canonical ERP space so a model trained on perspective images can estimate metric depth across varying fields of view. Its training pipeline combines pitch-aware conversion, FoV alignment, and multi-resolution augmentation.

4. Depth Any Camera

  • Images from different camera types and FoVs are transformed into canonical ERP space for training and testing, allowing a perspective-trained model to predict metric depth consistently.Training uses perspective datasets converted into ERP patches; large-FoV test images are converted into the same representation.Source: §4 · p. 3

4.1. Pitch-Aware Image-to-ERP Conversion

  • Pitch-aware conversion maps ERP grid points to input-image locations and varies the patch-center latitude to simulate regions unique to large-FoV cameras.Online augmentation can add noise to the center latitude to generate ERP patches with varying shapes.Source: §4.1 · p. 4

4.2. FoV Alignment

  • FoV Alignment scales each input to a predetermined crop-area FoV, fitting diverse ERP regions to one patch size while retaining content and reducing background computation.The challenge arises when ERP regions vary in size: one crop size can either omit content or waste computation on padding.Source: §4.2 · p. 5

4.3. Multi-Resolution Training

  • Multi-resolution training resizes each ERP patch to two additional lower resolutions and sums losses across three image-resolution batches.The scheme addresses performance degradation when test resolution differs from training patch size.Source: §4.3 · p. 5

5. Experiments

DAC is evaluated zero-shot on large-FoV datasets against metric-depth baselines, then ablated to examine its components and training data. Results show strong indoor gains, while outdoor gains are more limited.

5.1. Experimental Setup

  • Training uses 670K indoor perspective images and 130K outdoor images; zero-shot tests cover two 360° datasets and two fisheye datasets.Models are trained separately for indoor and outdoor experiments; evaluations use metric-depth metrics.Source: §5.1 · p. 5, Table 1 · p. 6

5.2. Comparison with the SoTA

  • DAC significantly outperforms UniDepth and Metric3Dv2 indoors, with nearly 50% improvement over the next-best method on δ1.It also outperforms iDisc across all metrics on all indoor datasets.Source: Table 2 · p. 6, §5.2 · pp. 6–7
  • Outdoors, DAC significantly outperforms Metric3Dv2 and UniDepth but only marginally improves over iDisc under the same network configuration.The authors attribute the smaller gains partly to limited training-pitch variance and KITTI360 LiDAR concentration in less-distorted regions.Source: §5.2 · p. 7

5.3. Ablation Study

  • Removing FoV Align or Multi-Reso Training significantly reduces DAC performance, especially on zero-shot 360° images.The ablation uses HM3D-tiny indoor training and tests on Pano3D-GV2 and ScanNet++; pitch-aware ERP conversion is pivotal for large-FoV generalization.Source: §5.3 · p. 7, Table 3 · p. 7
  • For Pano3D-GV2, broader HM3D FoV and pitch coverage improves generalization despite rendering artifacts; for ScanNet++, Hypersim’s image quality appears more influential than FoV diversity.The single-camera Hypersim training set outperforms HM3D on ScanNet++.Source: §5.3 · p. 8
  • Mixed training with diverse datasets significantly enhances DAC’s generalization to large-FoV datasets compared with individually trained models.Source: §5.3 · p. 8

6. Conclusion

DAC combines pitch-aware Image-to-ERP transformation, FoV alignment, and multi-resolution training to estimate metric depth across camera types. The framework significantly outperforms state-of-the-art methods and is intended to keep previously collected 3D data useful across camera types.

  • DAC targets zero-shot metric depth estimation across perspective, fisheye, and 360° cameras.Source: §6 · p. 8
  • Pitch-aware Image-to-ERP transformation, FoV alignment, and multi-resolution training address varying FoVs and resolution inconsistencies.Source: §6 · p. 8
  • The authors report significant gains over state-of-the-art methods and robust generalization across backbone networks.Source: §6 · p. 8

7. Supplemental Experiments

Supplemental experiments examine perspective-dataset performance, DAC’s components and training data, and perspective models on distorted inputs. They report dataset- and test-domain-dependent effects, alongside limitations of perspective models on fisheye imagery.

7.1. Full Zero-Shot Metric Depth Experiments

  • DAC also generalizes to NYUv2 and KITTI, outperforming Metric3Dv2-pipeline iDisc, though it remains behind state of the art.The authors attribute the remaining gap likely to DAC’s smaller training set and Swin-L backbone relative to competing models.Source: §7.1 · p. 12

7.2. Full Modular Ablation Study

  • Pitch-aware ERP conversion is pivotal for large-FoV generalization; ERP-space pitch augmentation adds little when HM3D-tiny already spans a broad pitch range.The conversion projects inputs to different ERP latitudes, simulating regions typically visible only in large-FoV data.Source: §7.2 · p. 12

7.3. Full Ablation Study on Training Dataset

  • Training-dataset effects vary by test data, and DAC can leverage dataset diversity to boost overall performance across the test datasets.Pitch-aware conversion is crucial across almost all tested training and testing configurations; pitch augmentation matters chiefly when training pitch diversity is limited.Source: §7.3 · pp. 12–13

7.4. Zero-Shot Test of Perspective Depth Model on Distorted Images

  • Perspective-trained Metric3D performs poorly on fisheye data; undistortion sacrifices FoV or introduces interpolation artifacts, while ERP conversion or raw fisheye input degrades performance further.ERP conversion is also applied when testing perspective models for a fair comparison, since neither raw fisheye nor ERP is their original camera domain.Source: §7.4 · p. 13

8. On Applying Camera Distortion Models

The paper describes applying its image-to-ERP conversion to KB and MEI fisheye distortion models. The KB formulation is adjusted for numerical stability near 180° FoV, while the MEI formulation incorporates its model-specific shift and distortion terms.

8. On Applying Camera Distortion Models

  • The conversion framework is applied to two fisheye models: KB (OpenCV Fisheye) and MEI.Source: §8 · p. 13

8.1. KB Model

  • For the KB model, the distortion parameters k1–k4 are applied after computing θ_d; near 180° FoV, using the numerators avoids numerical issues from division by cos c.The remaining process follows the original KB model, with projection coordinates computed from intrinsic parameters.Source: §8.1 · p. 14

8.2. MEI Model

  • The MEI model adds a shift parameter ξ for larger FoV and tangential parameters p1 and p2; its projection and distortion stages produce coordinates for the later KB-style projection.Source: §8.2 · pp. 14–15

9. Efficient Up-Projection from Distorted Cameras via Lookup Table Approximation

Fisheye up-projection is computationally demanding because it requires inverting distortion, so the section describes lookup tables and DAC’s ERP-based alternative. It also covers converting Z-value ground truth to Euclidean distance and mapping results back for visualization.

9. Efficient Up-Projection from Distorted Cameras via Lookup Table Approximation

  • Fisheye up-projection requires inverting distortion, often by solving a high-order polynomial per pixel, making it computationally expensive and impractical for real-time applications.For perspective or ERP images, pixel rays can instead be computed in closed form.Source: §9 · p. 16
  • Precomputed lookup tables map 2D image coordinates to 3D ray directions for real-time up-projection, and the Scannet++ tables use a grid-search approach.The tables can also be generated with gradient-based numerical methods; grid search is suggested when tangential distortion is negligible.Source: §9 · p. 16
  • DAC avoids approximate fisheye up-projection by converting fisheye images to ERP patches, whose depth maps use unit-sphere ray directions for 3D projection.The conversion relies only on the forward distortion model.Source: §9 · p. 16
  • Lookup tables also support visualization by mapping ERP patches and predicted ERP depth maps back to the original fisheye image space.ERP-to-image conversion uses grid sampling to map each fisheye coordinate to its ERP location.Source: §9 · p. 16
  • For datasets such as ScanNet++, ground-truth Z-values are converted to Euclidean distances by dividing Z by the ray direction’s z-component.The converted distances can be used for evaluation or DAC training.Source: §9 · p. 16

10. Additional Visual Results

The additional visual comparisons cover four large-FoV test sets and include Unidepth alongside the methods shown earlier. DAC’s visualizations show sharper depth boundaries, more consistent scale, and an advantage in A.Rel maps over previous state-of-the-art methods.

10. Additional Visual Results

  • The additional comparisons cover Matterport3D, Pano3D-GV2, Scannet++, and KITTI360, and add Unidepth to the methods shown earlier.Source: §10 · p. 16
  • DAC’s visual results show sharper depth boundaries, more consistent scale, and a significant A.Rel advantage over previous state-of-the-art methods.The comparisons cover Matterport3D, Pano3D-GV2, Scannet++, and KITTI360.Source: §10 · p. 16
  • Each dataset example spans two rows: the left column shows the image and ground-truth depth, while method columns show A.Rel above predicted depth.The last column indicates the color range for depth and A.Rel maps.Source: Figure 8 · p. 17, Figure 9 · p. 18, Figure 10 · p. 19
Loading 2501.02464v2…