Source-linked AI summary
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
David Eigen, Christian Puhrsch, Rob Fergus
TL;DR
Single-image depth prediction must reconcile global and local visual cues despite ambiguity in scene scale. The paper combines coarse global and fine local deep-network predictions with a scale-aware loss, achieving state-of-the-art results on NYU Depth and KITTI, including a 35% relative gain over the runner-up on NYU Depth.
Problem
Single-image depth estimation requires integrating global and local scene cues while addressing fundamental ambiguity in overall scene scale.
Method
The method uses a coarse network for global depth structure, a fine network for local refinement, and a loss accounting for depth relations beyond pointwise error.
Results
The system achieves state-of-the-art performance on both NYU Depth and KITTI, with a 35% average relative gain over the runner-up on NYU Depth.
Takeaways & Limitations
Combining global and local views enables reasonable single-image depth estimation while leveraging the full raw data distributions.
Takeaways & Limitations
On KITTI, uncorrected depth-image alignment from the rotating scanner limits the fine-scale network’s boundary improvements.
Abstract
from arXiv · showhide
Predicting depth is an essential component in understanding the 3D geometry of a scene. While for stereo images local correspondence suffices for estimation, finding depth relations from a single image is less straightforward, requiring integration of both global and local information from various cues. Moreover, the task is inherently ambiguous, with a large source of uncertainty coming from the overall scale. In this paper, we present a new method that addresses this task by employing two deep network stacks: one that makes a coarse global prediction based on the entire image, and another that refines this prediction locally. We also apply a scale-invariant error to help measure depth relations rather than scale. By leveraging the raw datasets as large sources of training data, our method achieves state-of-the-art results on both NYU Depth and KITTI, and matches detailed depth boundaries without the need for superpixelation.
1 Introduction
Monocular depth estimation is important for understanding scene geometry but is more ambiguous than stereo, especially because global scale is uncertain. The paper addresses this with a two-stage neural network and a scale-invariant, relation-aware loss.
- Motivation: Depth estimation supports richer scene representations and applications including recognition, 3D modeling, physics and support models, robotics, and occlusion reasoning.These benefits motivate depth prediction as a component of geometric scene understanding.
- Challenge: Unlike stereo, monocular depth estimation must integrate cues such as line angles and perspective because a single image lacks direct correspondence-based recovery.Stereo depth can be reduced to robust image-point correspondence when accurate correspondences are available.
- Challenge: Global scale remains a major ambiguity because infinitely many world scenes may produce the same image, despite physical plausibility constraints.Moderate variations in room and furniture sizes contribute to this uncertainty.
- Method: A scale-invariant error emphasizes spatial relations rather than general scale and suits applications such as 3D modeling where models are often rescaled during postprocessing.The scale-invariant error is used alongside more common scale-dependent errors.
- Method: The method directly regresses depth with two neural-network components: a global scene estimator followed by a local refinement stage.Its loss combines pointwise error with explicit depth relations between pixel locations.
2 Related Work
Prior work estimates monocular depth using engineered features, semantic labels, retrieval, or specialized hardware, while stereo methods exploit correspondence unavailable from a single RGB image. This work instead uses sensors only for training ground truth and predicts depth in software at test time.
- Monocular depth estimation: Earlier monocular methods use image features, linear regression, MRFs, or geometric-region labels, but Make3D depends on horizontal image alignment and struggles in less controlled settings.Saxena et al. predict depth from image features with linear regression and an MRF, while Hoiem et al. categorize regions into geometric structures rather than predicting depth explicitly.
- Monocular depth estimation: Later approaches combine semantic labels with handcrafted monocular features or transfer depths using SIFT Flow, with motion information improving estimates for moving foregrounds in videos.These methods can improve performance or alignment but retain dependence on handcrafted features, superpixels, retrieval, or motion cues.
- Stereo and multiview depth: Stereo and multiview methods estimate depth from correspondence across aligned or multiple views, including patch-based learning that relies on local stereo displacements.Stereo research spans matching, aggregation, and optimization, while multiview reconstruction matches uncalibrated consumer photographs of the same scene.
- Hardware-based depth estimation: Hardware-based systems obtain depth through defocus, active stereo, or time-of-flight, whereas this method uses such sensors only for training targets and predicts depth from RGB images at test time.The proposed system is purely software-based during testing.
3 Approach
The approach combines global coarse depth prediction with local refinement, while using a scale-invariant error and mixed training loss to emphasize depth relationships and retain absolute-scale accuracy.
- Two-scale architecture: The model uses two stacked networks: a coarse network predicts global scene depth, and a fine network edits it to incorporate local details.Both networks receive the original input, while the coarse output is passed to the fine network as additional first-layer features.
- Coarse-scale network: The coarse network integrates the entire scene through convolution, max-pooling, and fully connected layers to predict overall depth structure.Its global view captures cues such as vanishing points, object locations, and room alignment that local views may miss.
- Fine-scale network: The fine network uses convolutional layers to refine object and wall edges, with each output unit viewing 45x45 pixels and producing output at 1/4 input scale.The coarse prediction is concatenated with the first fine-scale feature map, and subsequent layers preserve the target size using zero-padded convolutions.
- Scale-invariant objective: The scale-invariant mean squared error in log space evaluates depth relationships independently of absolute global scale.It gives scalar multiples of a prediction the same error and can be interpreted as comparing pairwise depth differences between pixels.
- Training loss: The training loss averages elementwise l2 and scale-invariant error with λ = 0.5, improving qualitative output while preserving good absolute-scale predictions.The network predicts log depth, with λ = 0 corresponding to elementwise l2 and λ = 1 to the scale-invariant error.
4 Experiments
Experiments train the model on raw NYU Depth v2 and KITTI data, exploiting their large, minimally processed image distributions. Evaluation uses dataset-specific protocols and comparisons against Make3D, published methods, and a mean-depth baseline.
- Training data: The model is trained on raw NYU Depth v2 and KITTI data, retaining unfilled depth gaps that the model can naturally handle.The raw distributions provide additional same-scene images without preprocessing and suit the model’s need for large training sets.
- NYU Depth: The NYU training set contains 120K unique images, expanded to 220K after scene rebalancing; coarse and fine networks train for 2M and 1.5M samples, respectively.The model is evaluated on the 694-image NYU Depth v2 test set, with the fine network trained after fixing the coarse network.
- KITTI: KITTI uses 28 scenes for training and 28 for testing, with RGB inputs downsampled from 1224x368 to half resolution.The selected scenes come from the “city,” “residential,” and “road” categories of the raw data.
- KITTI: KITTI ground truth resolves conflicting LIDAR depths by selecting the value closest to RGB capture time, while the model receives the entire image despite depth covering only its bottom part.This construction addresses irregular sampling, differing capture times, and limited depth coverage.
- Comparisons: The method is compared with Make3D trained on the same datasets, published results from other methods, and a mean depth image baseline.Make3D on KITTI uses only 700 images because it could not scale beyond that size; predictions are compared over their intersecting crop regions after nearest-neighbor upsampling.
5 Results
The method achieves the best reported performance on both NYU Depth and KITTI, outperforming baselines across metrics while producing sharper depth boundaries through fine-scale refinement. Improvements include better scale-dependent and scale-invariant depth relations, although fine-scale refinement does not improve measured errors and remains limited by alignment issues on KITTI.
- NYU Depth: 35% relative gain over the runner-up on NYU Depth accompanies the best performance across all reported metrics.The comparison includes the data mean, Make3D, Karsch et al., and Ladicky et al.; the latter uses depth and semantic labels.
- NYU Depth: The method improves over Make3D on both scale-dependent and scale-invariant metrics, predicting better depth relations and means.Make3D performs relatively poorly because the dataset violates assumptions including horizontal ground-plane alignment.
- Qualitative results: Fine-scale refinement sharpens surface-boundary transitions and aligns them with local details, but can also include texture edges.These visual improvements are apparent in the depth maps despite no observed improvement in error measurements.
- KITTI: 31% relative gain over Make3D on KITTI includes a 25% gain in both scale-dependent and scale-invariant RMSE errors.Because Make3D is well-suited to horizontally aligned KITTI images, the gains indicate substantial improvement in predicted structure.
- KITTI qualitative results: Fine-scale refinement sharpens transitions near the road edge on KITTI, but its changes are limited by uncorrected depth-input alignment issues.Rotating-scanner misalignment dissociates edges from their true positions, causing the network to average over more random placements.
6 Discussion
The method makes single-image depth prediction feasible by combining global structure estimation with local refinement, achieving state-of-the-art results on NYU Depth and KITTI. Its fine-scale network improves alignment with object boundaries and wall edges, while future work targets surface normals and full-resolution depth maps.
- Method and results: Two deep networks combine global depth estimation with local refinement, enabling reasonably effective single-image depth prediction.The global network estimates overall depth structure, while the local network refines it at finer resolution.
- Method and results: The method achieves new state-of-the-art performance on both NYU Depth and KITTI by leveraging the full raw data distributions.The discussion attributes the result to combining global and local views and effectively using the datasets’ raw data.
- Future work: Future work will incorporate surface normals and extend depth maps to the full original input resolution through successively finer local networks.The authors identify joint normal-and-depth prediction as a potential way to improve overall performance.
- Qualitative analysis: Fine-scale refinement edits coarse predictions to better align with object boundaries and wall edges.Example predictions are shown for NYU Depth and KITTI, with outputs arranged from best to worst.