PaperlayerView exact revision on arXiv

Source-linked AI summary

MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimation

Reiner Birkl, Diana Wofk, Matthias Müller

arXiv:2307.14460v1cs.CV

TL;DR

Monocular depth estimation is underconstrained and must generalize across environments. MiDaS v3.1 evaluates new encoder backbones within the MiDaS architecture and reports a 28% average accuracy improvement for BEiT512-L over MiDaS v3.0 at non-square resolutions, while offering different quality-runtime tradeoffs.

  • Problem

    Monocular depth estimation is underconstrained from a single image, while metric-depth approaches can overfit narrow depth ranges and generalize poorly across environments.

  • Method

    MiDaS v3.1 integrates and compares transformer and convolutional encoder backbones within the MiDaS architecture and describes a strategy for adding future backbones.

  • Results

    28% average depth estimation accuracy improvement is reported for BEiT512-L over MiDaS v3.0 at non-square resolutions.

  • Takeaways & Limitations

    MiDaS v3.1 provides models with different depth-quality and runtime tradeoffs, including efficient options for downstream tasks requiring high frame rates.

  • Takeaways & Limitations

    MiDaS outputs are accurate only up to scale and shift, so the models alone are insufficient for applications requiring metric depth.

Abstract

from arXiv · show

We release MiDaS v3.1 for monocular depth estimation, offering a variety of new models based on different encoder backbones. This release is motivated by the success of transformers in computer vision, with a large variety of pretrained vision transformers now available. We explore how using the most promising vision transformers as image encoders impacts depth estimation quality and runtime of the MiDaS architecture. Our investigation also includes recent convolutional approaches that achieve comparable quality to vision transformers in image classification tasks. While the previous release MiDaS v3.0 solely leverages the vanilla vision transformer ViT, MiDaS v3.1 offers additional models based on BEiT, Swin, SwinV2, Next-ViT and LeViT. These models offer different performance-runtime tradeoffs. The best model improves the depth estimation quality by 28% while efficient models enable downstream tasks requiring high frame rates. We also describe the general process for integrating new backbones. A video summarizing the work can be found at https://youtu.be/UjaeNNFf9sE and the code is available at https://github.com/isl-org/MiDaS.

1. Introduction

Monocular depth estimation is difficult because a single image underconstrains pixel-level depth, yet learning-based MiDaS methods improve robustness through dataset mixing and scale-and-shift-invariant losses. MiDaS v3.1 integrates new backbones, compares their tradeoffs, and describes how to extend the architecture.

  • Monocular depth estimation regresses dense depth from a single image or camera view, supporting applications including generative AI, 3D reconstruction, and autonomous driving.
  • A single image provides insufficient information to determine depth at individual pixels, making monocular depth estimation underconstrained.
  • Dataset mixing and scale-and-shift-invariant losses have enabled MiDaS to produce robust and generalizable monocular depth estimates.
  • Transformers motivate the backbone expansion, alongside convolutional encoders pursued as competitive alternatives.
  • MiDaS v3.1 offers depth models with varied state-of-the-art backbones and aims to compare them while guiding future backbone integration.

2. Related Work

Related work frames monocular depth as an ill-posed, scale-ambiguous problem and motivates relative depth estimation through heterogeneous training data and invariant losses. MiDaS builds on dataset mixing and encoder-decoder architectures, progressing from convolutional to transformer backbones.

  • Metric depth regression can overfit homogeneous indoor or outdoor datasets, narrowing depth ranges and reducing generalization across environments.
  • Relative depth estimation instead targets depth up to scale and shift, using heterogeneous datasets with varying scales and camera parameters to improve cross-environment generalizability.
  • MiDaS mixes existing depth datasets with frames and disparity labels from 3D movies, while scale-and-shift-invariant losses address ambiguities in ground-truth labels.
  • MiDaS uses an encoder-decoder architecture whose encoders progressed from ResNet-based multi-scale models and EfficientNet-Lite to transformer-based ViT variants.

3. Methodology

The methodology reviews convolutional and transformer backbones, explains their integration into MiDaS, specifies training, and proposes a general strategy for adding future encoders.

  • The methodology surveys convolutional and transformer-based backbones considered for MiDaS v3.1.
  • It explains how encoder backbones are integrated into the MiDaS architecture.
  • It describes the training setup and a general strategy for adding new backbones in future extensions.

3.1. Overview of Encoder Backbones

MiDaS v3.1 selects encoder families to cover depth-quality and compute tradeoffs, emphasizing transformer backbones for released models while exploring additional transformer and convolutional alternatives.

  • Overview of Encoder Backbones: Depth estimation quality and compute requirements are treated as tradeoffs, motivating encoders optimized either for maximum quality or minimal resources.
  • Overview of Encoder Backbones: MiDaS v3.1 selects five promising encoder types for high depth quality or low-compute real-time applications.
  • Released Models: BEiT provides the highest depth estimation quality among released backbones, with BEiT512-L, BEiT384-L, and BEiT384-B variants.
  • Released Models: Swin and SwinV2 provide high-quality variants, while SwinV2-T targets downstream tasks with low compute resources.
  • Released Models: Next-ViT and LeViT are released for low-compute downstream tasks, with Next-ViT-L at 384x384 and MiDaS LeViT224 at 224x224.
  • Unpublished Models: Explored but unreleased models include Next-ViT-L, ViT-L Hybrid, DeiT3, and MobileViTv2 variants because their resulting depth models were less competitive.
  • Unpublished Models: The convolutional exploration includes ConvNeXt-L, ConvNeXt-XL, and EfficientNet-L2.
  • Legacy Models: Earlier MiDaS releases contribute legacy ResNeXt-101, EfficientNet-Lite3, ViT-L, and ViT-B Hybrid backbones to MiDaS v3.1.

3.2. Integration of Backbones into MiDaS

MiDaS v3.1 integrates classification backbones as feature extractors connected to a depth decoder, adapting hooks, hierarchy levels, resolutions, and channels to each architecture. Some implementation choices follow existing MiDaS designs, while others require backbone-specific modifications.

  • General integration: New backbones are created through timm when possible and connected to MiDaS’s depth decoder as feature-extraction encoders.Next-ViT is the exception to the common timm-based creation path.
  • BEiT: BEiT integration reuses MiDaS v3.0’s ViT hooking mechanism with absolute hook positions, but variable input resolutions require modified relative-position handling.The resolution-dependent relative position indices necessitate overriding several timm functions.
  • Swin and SwinV2: Swin and SwinV2 use hierarchical encoders whose successive levels reduce spatial resolution and increase embedding size, allowing some reshaping operators to be omitted.Unlike ViT and BEiT, their tensor shapes vary across hierarchy levels.
  • Hierarchical backbones: Hierarchical backbones constrain hooks to one per level; Swin and SwinV2 therefore use relative positions, whereas Next-ViT uses absolute positions because its blocks are sequential.Next-ViT uses four stages and selects hook positions 2, 6, 36, and 39.
  • LeViT: LeViT has three hierarchy levels, so its depth decoder is reduced to three levels and receives a deconvolutional decoder to recover the input resolution.LeViT-224’s convolutional stem reduces the resolution to 14x14 before attention.
  • Decoder adaptation: The decoder uses 256 channels in its hierarchical part across backbones and ultimately produces one channel for inverse relative depth.Other decoder stages reduce channels through 128 →32 →1.
  • Unreleased backbones: Several explored backbones were not released because their resulting depth models were less competitive, including transformer and convolutional alternatives.Examples include ConvNeXt, EfficientNet-L2, DeiT3, MobileViTv2, and an unreleased Next-ViT variant.

3.3. Training Setup

MiDaS v3.1 follows the MiDaS v3.0 training protocol while varying the dataset mix, including a broader option with indoor and outdoor datasets for metric-depth applications.

  • Training protocol: Training uses multi-objective optimization with Adam, learning rates of 1e-5 for encoders and 1e-4 for the decoder, ImageNet initialization for encoders, and random decoder initialization.The protocol follows MiDaS v3.0 and begins with pretraining on a dataset-mix subset for 60 epochs.
  • Dataset mixes: Dataset Mix 3+10 uses ten datasets, with ReDWeb, HRWSI, and BlendedMVS used for pretraining before training on the full mix.The ten-dataset mix includes ReDWeb, DIML, Movies, MegaDepth, WSVD, TartanAir, HRWSI, ApolloScape, BlendedMVS, and IRS.
  • Dataset mixes: Dataset Mix 5+12 adds NYUDepth v2 and KITTI, which were previously excluded to preserve zero-shot testing.The authors motivate their inclusion for metric-depth pipelines and observe improved generalizability across indoor and outdoor domains in those applications.

3.4. Discussion on using New Backbones

The paper presents a reusable strategy for extending MiDaS with new encoder backbones by connecting them to the decoder and adapting architecture-specific resolutions, hierarchy stages, operators, and channels.

  • Backbone integration: Using the PyTorch Image Models repository or a comparable framework can reduce the implementation effort for creating new encoder backbones.The backbone must then be connected appropriately to the depth decoder.
  • Decoder adaptation: Decoder modifications may change hierarchy stages or invert encoder operators, as in the LeViT integration, to obtain reasonable depth-decoding resolutions.The required changes depend on the backbone architecture.
  • Channel adaptation: Channel counts at selected network layers may also need adaptation, guided by the structure of similar backbones already integrated.This provides a practical guideline for future extensions.

4. Experiments

The experiments evaluate MiDaS v3.1 models across six datasets using zero-shot errors, relative improvement, runtime, and parameter counts. BEiT512-L provides the strongest released quality, while other models expose speed-quality tradeoffs and ablations reveal the effects of training data, resolution, and backbone choices.

  • 4.1. Evaluation: Six datasets are evaluated with dataset-specific errors, including WHDR, REL, δ1, and disparity RMSE for selected comparisons.The evaluation covers DIW, ETH3D, Sintel, KITTI, NYU Depth v2, and TUM.
  • 4.1. Evaluation: Relative improvement averages relative zero-shot error across six datasets against MiDaS v3.0’s largest ViT-L 384 model.Different inference resolutions must be compared separately because averaged pixel errors do not capture higher-resolution detail advantages.
  • 4.2.1 Published Models: 36% improvement makes BEiT512-L the best released model for square resolutions; it reaches 19% at height 512 and 28% at height 384.The paper reports these values for separate inference-resolution settings.
  • 4.2.1 Published Models: LeViT-224 runs at 73 FPS, while legacy EfficientNet-Lite3 reaches 90 FPS, extending MiDaS v3.1 toward lightweight downstream applications.The release includes more models to cover downstream tasks, including lightweight models.
  • 4.2.2 Unpublished Models: Increasing the dataset configuration raises Swin-L’s relative improvement from 2% to 21%, largely because KITTI and NYU Depth v2 become non-zero-shot datasets.Their δ1 scores decrease from 12.15 to 6.601 and from 6.571 to 3.343, respectively.
  • 4.2.2 Unpublished Models: MobileViTv2 variants reach relative improvements around -300%, while several other explored backbones are discarded for failing to surpass BEiT384-L.The rejected candidates include DeiT3, ViT-L Hybrid, and convolutional models such as ConvNeXt and EfficientNet-L2.
  • 4.3. Ablation Studies: Resolution-aware training modifications improve KITTI and overall scores, but BEiT384-L 5A+12A shows a possible resolution-overfitting tradeoff.The modified model reaches I = 37%, yet square-resolution improvement falls from 33% to 29%, below BEiT512-L’s 36%.

5. Applications

MiDaS v3.1 supports applications spanning relative-to-metric depth, depth-conditioned image generation, and joint image-depth diffusion. Its relative outputs can be integrated into downstream systems, but metric-depth use requires additional scale-and-shift alignment or metric fine-tuning.

  • Metric depth estimation: MiDaS models alone provide depth only up to scale and shift, so practical metric-depth systems require an additional metric-alignment component.Visual-inertial pipelines align non-metric maps against sparse metric depth, while ZoeDepth appends metric depth binning and fine-tunes prediction heads.
  • Metric depth estimation: MiDaS v3.1 models can improve metric depth accuracy when integrated into a modular visual-inertial depth pipeline.The pipeline performs global scale-and-shift alignment followed by learning-based dense alignment, and it allows different MiDaS models to be substituted.
  • Metric depth estimation: ZoeDepth combines MiDaS v3.1 relative-depth training with a metric depth binning module and metric fine-tuning for purely visual metric estimation.Its training uses the 5+12 dataset mix before fine-tuning on NYU Depth v2 and KITTI.
  • Depth-conditioned image diffusion: MiDaS relative depth outputs condition diffusion models to preserve semantic image shapes while allowing variation in artistic style.Stable Diffusion v2.0 used DPT-Hybrid from MiDaS v3.0 for this depth-guided image-to-image application.
  • Joint image and depth diffusion: LDM3D uses pretrained Stable Diffusion fine-tuned on caption, RGB-image, and depth-map tuples to generate joint image and depth data from text prompts.This application extends the use of depth information from conditioning image synthesis to RGBD generation.

6. Conclusion

MiDaS v3.1 introduces robust depth estimation models using new transformer backbones and provides guidance for integrating future backbones. BEiT512-L is reported as 28% more accurate than MiDaS v3.0 for non-square resolutions.

  • MiDaS v3.1 introduces depth models using BEiT, Swin, SwinV2, Next-ViT, and LeViT backbones.
  • 28% higher accuracy is reported for BEiT512-L at 512x512 than MiDaS v3.0 for non-square resolutions.
  • MiDaS training expands from 10 to 12 datasets by adding KITTI and NYU Depth V2 using the BTS split.
  • The paper details backbone integration and consolidates this experience into guidance for using future backbones with MiDaS.
Loading 2307.14460v1…