Source-linked AI summary
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, Vladlen Koltun
TL;DR
Monocular depth estimation is constrained by the lack of large, diverse dense-ground-truth datasets. The paper combines incompatible sources using robust losses and principled multi-objective training, including 3D movies, and finds improved zero-shot generalization across unseen datasets. It concludes that complementary data and systematic cross-dataset evaluation substantially advance generic monocular depth estimation.
Problem
Large-scale dense ground truth spanning diverse environments is difficult to acquire, leaving monocular-depth datasets with incompatible annotations and biases.
Method
The paper introduces scale- and shift-robust losses, Pareto-based multi-objective dataset mixing, auxiliary-task encoder pretraining, and a 3D-movies depth source.
Results
Models trained with complementary sources and appropriate procedures achieve state-of-the-art monocular-depth results across diverse environments under zero-shot cross-dataset transfer.
Takeaways & Limitations
Zero-shot testing on unseen datasets is presented as a better proxy for in-the-wild performance than held-out splits sharing dataset biases.
Takeaways & Limitations
3D-movie stereo data has scene-dependent depth budgets and film-specific camera or disparity characteristics that constrain its depth supervision.
Abstract
from arXiv · showhide
The success of monocular depth estimation relies on large and diverse training sets. Due to the challenges associated with acquiring dense ground-truth depth across different environments at scale, a number of datasets with distinct characteristics and biases have emerged. We develop tools that enable mixing multiple datasets during training, even if their annotations are incompatible. In particular, we propose a robust training objective that is invariant to changes in depth range and scale, advocate the use of principled multi-objective learning to combine data from different sources, and highlight the importance of pretraining encoders on auxiliary tasks. Armed with these tools, we experiment with five diverse training datasets, including a new, massive data source: 3D films. To demonstrate the generalization power of our approach we use zero-shot cross-dataset transfer}, i.e. we evaluate on datasets that were not seen during training. The experiments confirm that mixing data from complementary sources greatly improves monocular depth estimation. Our approach clearly outperforms competing methods across diverse datasets, setting a new state of the art for monocular depth estimation. Some results are shown in the supplementary video at https://youtu.be/D46FzVyL9I8
1 INTRODUCTION
Monocular depth estimation requires varied training data, but acquiring dense ground truth across environments is difficult. The paper develops robust objectives and dataset-mixing strategies, then evaluates generalization through zero-shot cross-dataset transfer.
- Monocular depth estimation is underconstrained and requires visual cues, long-range context, and learned representations.
- Dense ground-truth depth is difficult to acquire at scale because sensing technologies have limited range, operating conditions, cost, or scene coverage.
- The paper develops losses invariant to inconsistent depth scale and baselines, enabling training across stereo, laser-scanner, and structured-light data.
- A principled multi-objective dataset-mixing strategy improves results over naive mixing, while high-capacity encoders and auxiliary-task pretraining are empirically valuable.
- Zero-shot evaluation on unseen datasets is used to assess generalization, and experiments across six datasets outperform prior art quantitatively and qualitatively.
2 RELATED WORK
Prior monocular depth methods use geometric, probabilistic, non-parametric, or convolutional approaches, but their training data remains biased and limited. The paper positions controlled multi-source training as an unexplored direction for improving generalization.
- Earlier methods used MRFs, geometric assumptions, or non-parametric techniques, while newer approaches regress depth with convolutional networks.
- Self-supervised methods reduce annotation demands but generally require stereo images or apparent motion, limiting use in dynamic scenes.
- Existing datasets often contain homogeneous street or indoor layouts and few dynamic objects, biasing models toward constrained environments.
- Diverse dataset efforts include web-based ordinal labels, stereo-derived depth, and SfM/MVS reconstructions, but remain limited in scale or scene composition.
- Controlled mixing of multiple sources and optimal mixing strategies had not previously been explored for this monocular-depth setting.
3 EXISTING DATASETS
Depth datasets differ substantially in environments, annotations, accuracy, image properties, camera settings, and size. These differences create dataset-specific biases that can limit transfer to unseen data.
- Monocular-depth datasets vary in scene content, depth representation, annotation density and accuracy, image quality, camera settings, and dataset size.
- Each dataset has distinct characteristics and biases, while accurate dense data is difficult to collect at scale, especially for dynamic objects.
- The training collection combines complementary datasets including curated dynamic scenes, large mostly static reconstructions, and predominantly indoor RGB-D recordings.
- Generalization is benchmarked on six unseen datasets spanning diverse, accurate, synthetic, biased, and dynamic indoor environments.
4 3D MOVIES
The paper introduces 3D movies as a large, diverse source of stereo-derived relative depth from dynamic environments. It also documents film-specific limitations, preprocessing choices, and challenges for disparity extraction.
- 3D movies provide high-quality frames from diverse dynamic environments, including human-centered stories, landscapes, and animals.
- Movie stereo pairs yield relative rather than metric depth, but offer millions of carefully captured images without manual filtering of problematic content.
- Stereoscopic films impose scene-dependent depth budgets shaped by artistic and psychophysical considerations.
- Focal lengths, baselines, convergence angles, and screen-depth references vary across scenes, with both positive and negative disparities possible.
- The dataset uses 23 selected physical-stereo movies, high-resolution extraction, chapter and scene filtering, and temporal subsampling.
- Standard stereo matchers perform poorly on movie pairs because they assume positive disparities, while movie disparity ranges are also narrowed by limited depth budgets.
5 TRAINING ON DIVERSE DATA
The paper addresses incompatible depth representations, unknown scale, and disparity shifts by predicting in disparity space and using aligned, invariant losses. It also compares dataset-mixing strategies, including equal minibatch sampling and Pareto-optimal multi-objective learning.
- Challenges: Three incompatibilities complicate dataset mixing: direct versus inverse depth, unknown scale, and unknown disparity scale plus global shift.The shift ambiguity can arise from unknown stereo baselines and horizontal principal-point shifts during post-processing.
- Scale- and shift-invariant losses: The proposed framework predicts disparity and applies dense losses invariant to scale and shift across heterogeneous ground-truth representations.Disparity is inverse depth up to scale and shift; the loss aligns prediction and ground truth before applying a penalty.
- Scale- and shift-invariant losses: The alignment can use least squares, while the resulting scale- and shift-invariant MSE uses squared residuals after alignment.The scale and translation factors are obtained in closed form by rewriting the alignment objective as least squares.
- Scale- and shift-invariant losses: Robust alternatives use absolute deviations or trim the 20% largest residuals in each image to limit the influence of imperfect ground truth.The trimming threshold is set empirically, and the authors argue that ground-truth outliers should not influence training.
- Final loss: The final objective adds multiscale gradient matching in disparity space, using four levels with image resolution halved at each level.This term encourages sharp predicted discontinuities aligned with ground-truth discontinuities.
- Mixing strategies: Dataset mixing is evaluated through equal per-dataset minibatch sampling and a Pareto-based multi-objective strategy that treats each dataset as a separate task.The Pareto approach seeks a solution where improving one dataset’s loss requires worsening at least one other dataset’s loss.
6 EXPERIMENTS
Experiments examine loss functions, encoder pretraining, dataset selection and mixing, and zero-shot transfer. Diverse dataset mixing with Pareto-optimal optimization improves generalization, while specialized training can reduce performance on unseen domains.
- Experimental setup: The experiments use pretrained encoders, dataset-mixing studies, held-out validation sets, and six unseen test datasets for zero-shot evaluation.The unseen datasets are DIW, ETH3D, Sintel, KITTI, NYU, and TUM.
- Loss functions: The proposed trimmed MAE loss combined with gradient regularization yields the lowest validation error across datasets.All subsequent experiments use Lssitrim + Lreg.
- Encoders: Higher-capacity encoders improve performance, while random encoder initialization performs 35% worse than pretrained initialization on average.ResNeXt-101-WSL provides up to 15% relative improvement over the baseline and is used thereafter.
- Training datasets: Each dataset used alone generalizes worse on average than the small curated RW dataset, despite specialized datasets performing better on similar test sets.Compatible-dataset gains are offset by declines on other datasets.
- Dataset mixing: Mixing multiple training sets consistently improves baseline performance, but naive mixing can fail to benefit from added datasets.Pareto-optimal mixing improves over naive mixing and combining all five datasets produces the best model.
- Zero-shot transfer: The model outperforms baselines in zero-shot evaluation, and its smaller ResNet-50 variant also exceeds the state of the art.This indicates that performance is not attributable only to increased network capacity.
- Generalization limits: Single-dataset models can perform well on their target domains but significantly worse on other test sets because fine-tuning creates environment-specific priors.Such specialization may be desirable for some applications but is unsuitable when generalization is required.
- Qualitative results: Qualitative tests show plausible relative-depth estimates on abstract drawings when coarse cues such as shading or vanishing points are present.The reported examples include drawings and paintings with varying abstraction levels.
7 CONCLUSION
The paper introduces tools for combining complementary depth-data sources and evaluates robustness through zero-shot cross-dataset transfer. These ideas advance monocular depth estimation across diverse environments and improve its relevance to practical deployment.
- The paper introduces a flexible loss function and principled dataset-mixing strategy for combining complementary depth-data sources.
- Zero-shot cross-dataset testing is presented as a better proxy for in-the-wild performance than testing on held-out data from existing datasets.
- The work advances the state of the art in generic monocular depth estimation and substantially improves performance across diverse environments.