Source-linked AI summary
Single-Image Depth Perception in the Wild
Weifeng Chen, Zhao Fu, Dawei Yang, Jia Deng
TL;DR
Single-image depth estimation must work on unconstrained images, but existing RGB-D data are limited and metric depth from one image is ambiguous. The paper introduces DIW and a direct deep network trained with relative-depth annotations; it reports better performance than prior methods and improved in-the-wild estimation when combined with existing RGB-D data.
Problem
Existing RGB-D datasets are constrained in sensor characteristics and scene diversity, while metric depth from a single image is fundamentally ambiguous.
Method
The paper combines the 495K-image DIW relative-depth dataset with a single deep network that directly predicts pixel-wise metric depth using a relative-depth loss.
Results
The algorithm outperforms prior methods, and combining it with DIW annotations and existing RGB-D data significantly improves single-image depth estimation in the wild.
Takeaways & Limitations
Relative-depth annotations from diverse in-the-wild images can support metric-depth learning and serve as both a training resource and evaluation benchmark.
Abstract
from arXiv · showhide
This paper studies single-image depth perception in the wild, i.e., recovering depth from a single image taken in unconstrained settings. We introduce a new dataset "Depth in the Wild" consisting of images in the wild annotated with relative depth between pairs of random points. We also propose a new algorithm that learns to estimate metric depth using annotations of relative depth. Compared to the state of the art, our algorithm is simpler and performs better. Experiments show that our algorithm, combined with existing RGB-D data and our new relative depth annotations, significantly improves single-image depth perception in the wild.
1 Introduction
Single-image depth systems struggle to generalize from constrained RGB-D datasets to unconstrained images. The paper addresses this gap with relative-depth supervision, a large in-the-wild dataset, and a simpler direct prediction algorithm.
- Motivation: RGB-D datasets are constrained by sensor limitations and narrow scene diversity, leaving generalization to images in the wild unclear.NYU is mostly indoor, KITTI mostly road scenes, and Make3D mostly Stanford-campus scenes.
- Motivation: Metric depth is fundamentally ambiguous from one image, whereas humans are often better at judging which of two points is closer.This motivates collecting relative-depth annotations instead of absolute metric depth.
- Contributions: DIW contains 495K diverse in-the-wild images annotated with relative depth for randomly sampled point pairs.One point pair is sampled per image to reduce annotation redundancy, and the dataset serves both evaluation and training.
- Contributions: The paper introduces an algorithm that learns metric depth directly from relative-depth annotations using a single deep network.Unlike the prior pipeline, it directly predicts pixel-wise depth without intermediate ordinal classification or an additional optimization stage.
- Results: The proposed method outperforms prior methods, including a metric-depth system trained with ground-truth depth, while combining DIW, the algorithm, and existing RGB-D data improves in-the-wild estimation.The method produces more accurately ordered pixel-wise depth and the combined training strategy significantly improves single-image depth estimation in the wild.
2 Related work
Prior RGB-D datasets rely mainly on Kinect or LIDAR and are limited in scene coverage, while this work uses crowdsourced relative-depth annotations from diverse Internet images. Its approach also differs from earlier ordinal-relation methods through random point sampling and direct end-to-end prediction.
- RGB-D Datasets: Kinect-based RGB-D datasets are mainly indoor, whereas LIDAR-based datasets are biased toward man-made structures.The paper positions DIW as covering a wider variety of scenes and as readily expandable through crowdsourcing and Internet images.
- Intrinsic Images in the Wild: DIW samples random points rather than superpixel centers because constant depth within a superpixel is unreasonable.This design differs from crowdsourced intrinsic-reflectance annotation protocols.
- Intrinsic Images in the Wild: DIW samples one point pair per image instead of many to maximize the value of human annotations.The dataset therefore prioritizes annotation efficiency while retaining relative-depth supervision.
- Depth from a Single Image: Earlier image-to-depth networks were generally trained with metric-depth ground truth, whereas this approach uses relative depth.Zoran et al. is identified as the exception among the cited previous works.
- Depth from a Single Image: Zoran et al. classify sampled point pairs and reconstruct depth through optimization, while this work trains one end-to-end network to predict per-pixel metric depth directly.The direct formulation removes the intermediate ordinal classifier and optimization for resolving inconsistencies.
- Learning with Ordinal Relations: Ordinal relations have also been applied to surface reflectance, object recognition, and learning to rank.These applications provide the broader related-work context for relative comparisons.
3 Dataset construction
The dataset gathers relative-depth judgments for unconstrained Flickr images through carefully designed point-pair sampling and crowdsourced quality control. Its sampling strategy balances natural scene statistics against resistance to simple image-coordinate biases.
- Image and annotation collection: Images are collected from Flickr using random English-dictionary query keywords, excluding drawings and clip art.Workers compare two highlighted points and choose which is closer or whether the relation is hard to tell.
- Annotation budget: One point pair per image is chosen because additional pairs from the same image may provide less information than pairs from new images.The design is intended to maximize the information obtained from human annotators.
- Pair sampling: 85.8% agreement with humans is achievable by simply labeling the lower image point as closer, revealing a severe sampling bias.Uniformly random point pairs allow image coordinates to provide an overly strong shortcut.
- Pair sampling: 71.4% agreement is achievable by labeling the point nearer the image center as closer, motivating symmetric point sampling on random horizontal lines.Symmetry removes simple effective rules based purely on image coordinates.
- Pair sampling: The final dataset combines unconstrained and symmetric pairs in roughly equal proportions to balance natural scene statistics and performance differentiation.The two pair types therefore serve complementary dataset-design goals.
- Quality control: 1.24M processed images yielded 0.5M valid answers after worker filtering and agreement checks.Valid answers included 261K unconstrained pairs and 240K symmetric pairs; symmetric queries took 3.8 seconds medially and reached 32% worker agreement, versus 3.4 seconds and 52% for unconstrained pairs.
4 Learning with relative depth
The method replaces pairwise ordinal classification and post-hoc depth reconstruction with a single end-to-end network that predicts pixel-wise metric depth from relative-depth annotations. Its key design is the combination of multi-scale pixel-wise prediction with a ranking loss.
- Prior approach: Zoran et al. classify ordinal relations between superpixel centers, reconcile them with energy minimization, and interpolate depth within superpixels.This is the prior pipeline that the proposed method simplifies.
- Network approach: The proposed approach represents image-to-depth mapping as an end-to-end neural network that outputs depth at the input resolution.It requires a same-resolution network design and a training loss based on relative-depth annotations.
- Network design: The network uses an hourglass architecture with convolutions, downsampling, upsampling, and skip connections that restore high-resolution features.The implementation uses a variant of the Inception module, although the authors state that this particular architecture choice is not essential.
- Relative-depth loss: For each query, the loss uses the annotated relation between two image locations to constrain their predicted depths.The relation labels are closer (+1), further (−1), or equal (0), and ψk denotes the loss for query k.
- Relative-depth loss: The ranking loss encourages a large predicted depth difference for unequal relations and a small difference for equal relations.The formulation applies separate terms for closer, farther, and equal annotations.
- Core novelty: The method’s novelty is combining pixel-wise deep prediction with a ranking loss applied directly to the pixel-wise output.The authors identify this combination as novel for depth estimation.
5 Experiments on NYU Depth
On NYU Depth, the method is evaluated for ordinal and metric depth accuracy, compared with prior systems, and tested with random rather than superpixel-based training pairs. It achieves strong depth ordering while retaining meaningful metric estimates despite relative-only supervision.
- Evaluation setup: NYU Depth provides indoor scenes with Kinect ground-truth depth, and the evaluation follows Zoran et al.’s setup using sampled point pairs.The test protocol generates point pairs from the same NYU training and test data used by Zoran et al.
- Ordinal evaluation: Ordinal evaluation reports WKDR, WKDR=, and WKDR≠ for disagreement with ground-truth depth relations.The three measures distinguish overall disagreement, equality cases, and unequal closer/farther cases.
- Ordinal evaluation: The equality threshold τ changes all ordinal error measures because predicted equality depends on the permitted depth-difference range.Too small a threshold predicts equality for too few pairs, while the threshold must relax exact equality because ground-truth depths rarely match exactly.
- Results: The full-data model achieves superior depth-ordering performance while using only relative-depth supervision, rivaling the state of the art up to monotonic transformations.The comparison with Eigen et al. is qualified because Eigen’s system is optimized for a different supervision setting.
- Qualitative results: Qualitative NYU results are overall reasonable and qualitatively similar to a state-of-the-art system trained with ground-truth metric depth.The recovered depth maps are imperfect but visually comparable in the reported examples.
- Metric evaluation: Metric evaluation normalizes predicted depth to the training-set mean and standard deviation before computing measures such as RMSE.The method outperforms Zoran et al. on metric errors, but remains worse than the current state of the art while comparable to some earlier metric-supervised methods.
- Sampling ablation: Training with 3.3K randomly sampled pairs per image achieves comparable performance to Zoran et al., and more pairs improve results despite a train-test sampling mismatch.Testing still uses superpixel pairs, so the result demonstrates that superpixel segmentation is unnecessary for the method.
6 Experiments on Depth in the Wild
On DIW, the study compares five methods using WHDR and finds that pre-training on NYU Depth followed by DIW fine-tuning performs best. Fine-tuning improves qualitative predictions, particularly for outdoor scenes.
- 421K DIW images are used for training and 74K for testing.
- The evaluation reports Weighted Human Disagreement Rate (WHDR) for five methods, including Eigen et al., NYU-trained variants, DIW-trained variants, and a query-location baseline.
- Pre-training on NYU Depth and fine-tuning on DIW achieves the best result.NYU-only training performs worse, while DIW-only training achieves slightly better performance than NYU-only training despite using less supervision.
- DIW fine-tuning notably improves predicted depth quality, especially for outdoor scenes, and supports combining RGB-D data with crowdsourced annotations.
7 Conclusions
The paper studies depth recovery from unconstrained single images using relative-depth supervision. It introduces DIW and a relative-depth-trained metric-depth algorithm, reporting improved performance when combined with existing RGB-D data.
- The paper introduces a wild-image dataset with relative-depth annotations and an algorithm that learns metric depth from relative-depth supervision.
- The algorithm outperforms prior art, while combining existing RGB-D data with new relative-depth annotations significantly improves single-image depth perception in the wild.
Appendix
The appendix provides additional qualitative examples and training-result analyses for RGB-D datasets, DIW, NYU Depth, and point-pair sampling.
- Additional examples compare current RGB-D datasets with the Depth in the Wild dataset.
- Additional NYU Depth results compare the proposed method with Eigen et al. and Zoran et al., including ground-truth comparisons.
- Additional DIW qualitative results compare the proposed method with Eigen et al.
- Supplementary experiments train the network on full depth maps and distinguish NYU-trained subsets from full-training-set variants.
- Supplementary experiments vary the number and type of sampled point pairs used for training on NYU Depth.