Source-linked AI summary
Depth from a Single Image by Harmonizing Overcomplete Local Network Predictions
Ayan Chakrabarti, Jingyu Shao, Gregory Shakhnarovich
TL;DR
Recovering depth from a single color image is useful when explicit depth measurements are unavailable or noisy. The paper predicts distributions over local depth derivatives and globalizes them into a consistent depth map; it reports state-of-the-art performance on NYUv2.
Problem
Recovering scene depth from one color image is useful when explicit depth measurements are unavailable and can help refine noisy or ambiguous measurements.
Source: §1 · p. 1Method
Results
The approach achieves state-of-the-art performance on the NYUv2 depth data set.
Source: §1 · p. 2Takeaways & Limitations
The distributional outputs can be combined with partial depth cues from other sources within a common globalization framework.
Source: §1 · p. 2The formulation represents the scene map as reciprocal metric depth, a choice motivated by perspective-camera geometry.
Source: §4 · p. 6
Abstract
from arXiv · showhide
A single color image can contain many cues informative towards different aspects of local geometric structure. We approach the problem of monocular depth estimation by using a neural network to produce a mid-level representation that summarizes these cues. This network is trained to characterize local scene geometry by predicting, at every image location, depth derivatives of different orders, orientations and scales. However, instead of a single estimate for each derivative, the network outputs probability distributions that allow it to express confidence about some coefficients, and ambiguity about others. Scene depth is then estimated by harmonizing this overcomplete set of network predictions, using a globalization procedure that finds a single consistent depth map that best matches all the local derivative distributions. We demonstrate the efficacy of this approach through evaluation on the NYU v2 depth data set.
1 Introduction
The paper addresses depth recovery from a single color image, where geometric cues occur in different regions and reveal different aspects of scene structure. It predicts distributions over local depth derivatives and combines them into a consistent depth map; on NYUv2, the approach achieves state-of-the-art performance.
1 Introduction
- Geometric cues occur in different image regions and indicate different aspects of 3D structure, motivating a local-structure approach to monocular depth.Depth inferred from monocular images can serve as a proxy when explicit measurements are unavailable and help refine noisy or ambiguous measurements.Source: §1 · p. 1
- The network predicts distributions over depth-map derivatives of different orders, scales, and orientations at every image location.These distributions represent local geometric structure while allowing different levels of certainty about different coefficients.Source: §1 · p. 2
- An efficient globalization procedure finds a single consistent depth map that best agrees with the local distributions.Source: §1 · p. 2, Figure 1 · p. 2
- On NYUv2, the approach achieves state-of-the-art performance.The authors suggest its distributional output could also combine monocular cues with sparse or noisy depth measurements.Source: §1 · p. 2
2 Related Work
Prior neural methods often regress depth directly, predict smoothness or surface normals, or estimate pairwise depth ordering. This paper instead predicts distributions over diverse coefficients in an overcomplete representation, with globalization that harmonizes local predictions without a restrictive local model.
2 Related Work
- Prior neural approaches include direct depth regression [1] [2], CRF-based smoothing [4], surface-normal prediction [6], and pairwise depth ordering.Pairwise ordinal predictions carried limited information for accurate absolute depth in the globalization scheme of.Source: §2 · pp. 2–3
- Unlike lower-dimensional basis methods [5], this approach uses distributions over an overcomplete set of coefficients, expecting precise predictions for only a subset.Source: §2 · p. 3
- Compared with Chakrabarti et al. [15], this method predicts independent local distributions over depth derivatives rather than imposing a restrictive local model, and uses Fourier-domain computations for efficient inference.Source: §2 · p. 3
3 Proposed Approach
The approach predicts distributions over an overcomplete set of local depth derivatives, then harmonizes them into a consistent scene map. It combines local image features with scene-level context and uses iterative optimization to reconcile the predictions.
3 Proposed Approach
- Each location is represented by 64 derivative filters spanning three scales, multiple orientations, and zeroth-, first-, and second-order responses.The bank also includes an impulse filter whose coefficient is the depth itself; first- and second-order filters are unit-normalized, while zeroth-order filters are normalized to balance the representation.Source: §3 · p. 3
3.1 Parameterizing Local Distributions
- Each coefficient distribution is a Gaussian mixture whose component means and shared per-derivative variance are fixed from training coefficients, while the network predicts mixture weights.The means come from one-dimensional K-means; the variance averages in-cluster variance over clusters with enough assignments.Source: §3.1 · p. 4
3.2 Neural Network-based Local Predictions
- The network predicts K × M mixture weights per image location and trains them against best-fit distributions for ground-truth coefficients using a variance-weighted KL-divergence loss.Its architecture combines local patch features with scene-level features before producing the coefficient distributions.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
3.3 Global Scene Map Estimation
- The scene map is estimated by alternating between coefficient and depth-map updates, progressively enforcing consistency between auxiliary coefficients and filtered depth values.The non-convex objective is handled with auxiliary variables; the depth-map step is a least-squares minimization, and the method uses a regularizer to increase the penalty faster and converge in fewer iterations.Source: §3.3 · pp. 5–6
4 Experimental Results
Experiments on NYU v2 evaluate the network’s local derivative predictions, the effects of filter subsets, and final depth estimation. The proposed method outperforms other state-of-the-art approaches on most test-set metrics.
4 Experimental Results
- Training uses roughly 56,000 NYU v2 color-depth pairs, while evaluation uses the valid-depth crop of the images.The scene map is reciprocal metric depth, y(n) = 1/z(n); training uses augmentation and a fully convolutional network with an eight-pixel stride.Source: §4 · p. 6, §4.1 · p. 6
4.1 Network Training
- The network is trained with SGD for 14 epochs, using one image per batch and a learning rate reduced at five specified epochs.Training augmentation includes rotations, horizontal flips, and random image contrast changes.Source: §4.1 · p. 6
4.2 Evaluation
- High-confidence local coefficient predictions have low errors, and globalization produces an accurate depth map despite low-confidence regions in individual predictions.The reported findings suggest that predictions from different coefficients provide complementary information.Source: Figure 3 · p. 7, §4.2 · p. 7
- Removing subsets of derivative filters degrades global estimation; second-order derivatives contribute least, and multi-scale zeroth-order-only filters perform worse than point-wise depth distributions.The ablation uses 100 validation images; the supplied passages do not include the table’s numerical values.Source: §4.2 · p. 7, Table 1 · p. 7
- The method yields superior performance on most metrics on the NYU v2 test set.Example predictions often better reproduce local geometric structure and usually estimate large, texture-less planar regions correctly, though some object positions are mis-estimated and a planar-region failure is shown.Source: §4.2 · p. 7, Table 2 · p. 8
- Overall inference, including network predictions and globalization, takes 24 seconds per image on an NVIDIA Titan X GPU.Source: §4.2 · p. 7
5 Conclusion
The conclusion presents probabilistic local depth-coefficient predictions and globalization as an alternative to point-wise depth regression. It identifies other depth-sensing settings and additional vision tasks as directions for future exploration.
5 Conclusion
- The approach predicts local depth-map coefficients probabilistically in an overcomplete representation, then reconciles them into a scene-depth estimate through globalization.Source: §5 · p. 8
- Future work will explore using these predictions for stereo reconstruction and improving active and passive depth-sensor measurements.Source: §5 · p. 8
- The paper also proposes exploring overcomplete probabilistic local predictions for motion estimation and intrinsic image decomposition.Source: §5 · p. 8