Source-linked AI summary
MegaDepth: Learning Single-View Depth Prediction from Internet Photos
Zhengqi Li, Noah Snavely
TL;DR
Single-view depth prediction is constrained by limited training data, while existing sensor-based datasets suffer from indoor-only coverage, small scale, or sparse depth. The paper introduces MegaDepth, using Internet photo collections with SfM and MVS plus data refinement and ordinal relations, and reports strong generalization to novel scenes and diverse datasets.
Problem
Single-view depth prediction needs large, diverse training data, but existing sensor-based datasets are limited by indoor-only images, small training sets, or sparse depth.
Method
MegaDepth generates a large depth dataset from Internet photo collections using SfM and MVS, refining noisy outputs and adding semantic-segmentation-derived ordinal depth relations.
Results
Models trained on MegaDepth generalize to novel scenes and other datasets, including Make3D, KITTI, and DIW, without seeing those datasets during training.
Takeaways & Limitations
Internet-derived SfM and MVS data can support single-view depth prediction that generalizes across scenes and datasets.
Takeaways & Limitations
Cross-dataset testing has performance variance, so the evaluation averages results from four models trained on MegaDepth.
Abstract
from arXiv · showhide
Single-view depth prediction is a fundamental problem in computer vision. Recently, deep learning methods have led to significant progress, but such methods are limited by the available training data. Current datasets based on 3D sensors have key limitations, including indoor-only images (NYU), small numbers of training examples (Make3D), and sparse sampling (KITTI). We propose to use multi-view Internet photo collections, a virtually unlimited data source, to generate training data via modern structure-from-motion and multi-view stereo (MVS) methods, and present a large depth dataset called MegaDepth based on this idea. Data derived from MVS comes with its own challenges, including noise and unreconstructable objects. We address these challenges with new data cleaning methods, as well as automatically augmenting our data with ordinal depth relations generated using semantic segmentation. We validate the use of large amounts of Internet data by showing that models trained on MegaDepth exhibit strong generalization-not only to novel scenes, but also to other diverse datasets including Make3D, KITTI, and DIW, even when no images from those datasets are seen during training.
1. Introduction
MegaDepth addresses limited single-view depth training data by using overlapping Internet photos with SfM and MVS to create a large, diverse dataset. Models trained solely on this data generalize to novel scenes and datasets including Make3D, KITTI, and DIW.
- Existing depth datasets are constrained by indoor-only imagery, limited examples, sparse measurements, or specific collection scenarios.Kinect-based data is limited to indoor use; Make3D and KITTI have constrained scale, sparsity, or collection settings.
- MegaDepth uses overlapping Internet photos with structure-from-motion and multi-view stereo to automatically generate dense training depth.The dataset draws on diverse photos taken around the world and is made fully available to the community.
- MegaDepth combines MVS depth processing with a joint loss and semantic-segmentation-derived ordinal relations to address noisy reconstructions and unreconstructable dynamic objects.The ordinal augmentation targets objects such as people and cars that MVS often fails to reconstruct.
- Models trained on MegaDepth generalize to new scenes and diverse datasets, including Make3D, KITTI, and DIW, without seeing those datasets during training.The paper reports much better generalization than prior datasets and illustrates cross-dataset predictions from an MD-only model.
2. Related work
Prior single-view depth work relies mainly on sensor-derived RGB-depth pairs, view-synthesis supervision, or manually collected ordinal labels. Internet 3D reconstruction offers another data source, while some alternatives require additional test-time inputs or proxy supervision.
- Single-view depth prediction: Standard single-view depth methods train CNNs on RGB images paired with depth from datasets such as NYU, Make3D, and KITTI.These datasets inherit limitations from RGB-D sensors and laser scanning discussed in the introduction.
- Single-view depth prediction: Multiple-view methods can use view synthesis as supervision, but synthesized views are only a proxy for depth and may produce lower-quality learned depth.This line of work uses multiple views without directly providing depth ground truth.
- Depth estimation from Internet photos: One prior overlapping-image method requires two input images at test time rather than a single image.
- Ordinal depth prediction: Crowdsourced ordinal annotations provide relative depth judgments, supporting depth-order prediction when absolute depth is difficult to label.Depth in the Wild is an example of a large dataset built from such labels.
- Depth estimation from Internet photos: Internet photo collections have supported large-scale SfM and MVS reconstruction, but prior methods often require a detailed 3D model in advance or construction at run time.MegaDepth instead uses reconstructed Internet data to generate training data for single-view prediction.
3. The MegaDepth Dataset
MegaDepth is constructed from Flickr landmark photos reconstructed with SfM and MVS, then cleaned to reduce unreliable depths and augmented with semantic ordinal supervision. The resulting dataset retains Euclidean and ordinal training images after filtering.
- Photo calibration and reconstruction: The dataset begins with Flickr photos of well-photographed landmarks and reconstructs each collection using SfM and MVS.The process produces camera poses, sparse point clouds, and dense depth maps, with some pixels unreconstructed.
- Depth map refinement: Raw MVS depth contains transient-object errors, noisy discontinuities, and background-depth bleeding into foreground objects.These outliers negatively affect networks trained for single-view depth prediction.
- Depth map refinement: The refinement strategy favors conservative depth estimates, preferring less training data over unreliable depth.The method specifically targets the tendency of iterative MVS consistency to let background depths consume foreground objects.
- Depth enhancement via semantic segmentation: Semantic segmentation is used to filter spurious foreground depths, categorize images, and generate ordinal labels where MVS cannot reconstruct objects.Foreground, background, and sky categories support several filtering and labeling operations.
- Depth enhancement via semantic segmentation: Images with at least 30% valid nonsky depth are retained for Euclidean training, while foreground-heavy images provide ordinal supervision.
- Depth enhancement via semantic segmentation: The automatic ordinal procedure selects foreground and background regions whose geometry implies the foreground is closer, achieving over 95% pairwise ordinal accuracy.The regions are derived from semantic masks and reconstructed depth-range information.
- Dataset scale: The final construction reconstructs 200 landmark models covering about 150K images, retains 130K valid images, and assigns roughly 100K to Euclidean and 30K to ordinal data.The combined data forms the publicly available MegaDepth dataset.
4. Depth estimation network
The network predicts log-depth from a single photo using a scale-invariant loss that combines data, gradient-matching, and ordinal terms. The authors evaluate prior architectures and report that the hourglass network performs best.
- The hourglass network performs best among the evaluated VGG, hourglass, and ResNet architectures.
- Loss function: The scale-invariant loss operates in the log-depth domain because SfM+MVS depths are known only up to an unknown scale factor.The loss combines three terms: data, gradient matching, and ordinal depth constraints.
- Loss function: The scale-invariant data term measures mean squared error between differences of predicted and ground-truth log-depths across pixel pairs.Predicted and ground-truth log-depth values are indexed by pixel position, with residuals defined as their difference.
- Loss function: Lgrad uses multi-scale ℓ1 penalties on log-depth gradient differences to encourage smoother changes and sharper depth discontinuities.The loss captures depth gradients across large image distances and is evaluated at four scales.
- Loss function: Lord imposes robust, automatically labeled ordinal relations by encouraging likely ordered pixel pairs to have large, correctly ordered depth differences.It is designed to tolerate a small number of incorrectly ordered pairs and helps correct relations for difficult objects such as people and trees.
5. Evaluation
The evaluation tests MegaDepth models on unseen MD locations and on Make3D, KITTI, and DIW, examining architectures, loss terms, depth refinement, and cross-dataset generalization. MegaDepth training data and the proposed refinements and loss improve performance across these evaluations.
- Evaluation goals: The evaluation measures generalization to unseen Internet locations and to images from Make3D, KITTI, and DIW.The cross-dataset tests use models trained on MegaDepth without training images from the target datasets.
- MD test-set evaluation: The hourglass architecture achieves the best performance among the three evaluated architectures on the MD test set.The compared architectures are VGG, hourglass, and ResNet.
- MD test-set evaluation: Training with the full loss outperforms alternative loss variants, including the loss used by [6].The full loss combines scale-invariant data terms, multi-scale gradient matching, and ordinal depth supervision.
- MD test-set evaluation: Adding the ordinal loss significantly improves SDR̸= while increasing SDR= on the MD test set.The joint loss also helps preserve depth-map structure and capture nearby objects such as people and buses.
- Depth refinement: Depth refinement significantly boosts prediction performance across KITTI, Make3D, and DIW compared with training on raw MVS depth.Table 3 compares Raw MD and Clean MD, with lower error better for all measures.
- Generalization to other datasets: MegaDepth-trained models achieve the best performance among non-target-dataset-trained models on Make3D, KITTI, and DIW.The KITTI result holds despite the dataset's driving scenes and objects that are difficult to reconstruct with SfM/MVS; DIW predictions also work reasonably well on offices and close-ups.
6. Conclusion
Internet-derived SfM+MVS data can generate large-scale training data for single-view depth prediction and generalizes to unseen locations and other datasets, but retains important reconstruction and scale limitations.
- MegaDepth uses Internet-derived SfM+MVS data to generate large amounts of training data for single-view depth prediction.
- Models trained on MegaDepth predict state-of-the-art depth maps for locations never observed during training and generalize well to other datasets.
- MVS reconstruction remains imperfect for oblique surfaces, thin or complex objects, and difficult materials such as shiny glass.
- The method predicts relative rather than metric depth because the reconstructed 3D data has an unknown scale factor.
- The dataset is currently biased toward outdoor landmarks, although larger Internet photo collections may provide more diverse scenes.