Source-linked AI summary
Monocular Depth Estimation with Hierarchical Fusion of Dilated CNNs and Soft-Weighted-Sum Inference
Bo Li, Yuchao Dai, Mingyi He
TL;DR
Monocular depth estimation remains challenging in complex scenes containing multiple objects at diverse scales. The paper uses an end-to-end classification-based framework with dilated convolutions and hierarchical fusion, and reports state-of-the-art performance on indoor and outdoor benchmarks. It also notes that the network cannot distinguish very detailed distance variation, even when trained with detailed labels.
Problem
State-of-the-art monocular depth methods still fall short on complex real-world compositions with multiple objects at diverse scales.
Source: §1 · pp. 2–3Method
An end-to-end framework recasts depth estimation as classification and uses dilated convolutions with hierarchical fusion of multi-scale depth cues.
Source: §1 · p. 3Results
The method achieves state-of-the-art performance on indoor and outdoor benchmarks, including NYU V2 and KITTI.
Source: §1 · p. 4, §6 · p. 17Takeaways & Limitations
The authors report that the classification-based framework performs strongly across both indoor and outdoor benchmarking datasets.
Source: §1 · p. 4The network cannot distinguish very detailed distance variation even when trained with a very detailed label.
Source: §5.2 · p. 16
Abstract
from arXiv · showhide
Monocular depth estimation is a challenging task in complex compositions depicting multiple objects of diverse scales. Albeit the recent great progress thanks to the deep convolutional neural networks (CNNs), the state-of-the-art monocular depth estimation methods still fall short to handle such real-world challenging scenarios. In this paper, we propose a deep end-to-end learning framework to tackle these challenges, which learns the direct mapping from a color image to the corresponding depth map. First, we represent monocular depth estimation as a multi-category dense labeling task by contrast to the regression based formulation. In this way, we could build upon the recent progress in dense labeling such as semantic segmentation. Second, we fuse different side-outputs from our front-end dilated convolutional neural network in a hierarchical way to exploit the multi-scale depth cues for depth estimation, which is critical to achieve scale-aware depth estimation. Third, we propose to utilize soft-weighted-sum inference instead of the hard-max inference, transforming the discretized depth score to continuous depth value. Thus, we reduce the influence of quantization error and improve the robustness of our method. Extensive experiments on the NYU Depth V2 and KITTI datasets show the superiority of our method compared with current state-of-the-art methods. Furthermore, experiments on the NYU V2 dataset reveal that our model is able to learn the probability distribution of depth.
1. Introduction
Monocular depth estimation remains difficult in complex scenes with multiple objects at different scales, and existing methods struggle with such compositions. The paper recasts depth prediction as dense labeling, combines dilated convolutions with hierarchical multi-scale fusion, and uses soft-weighted-sum inference; it reports state-of-the-art results on NYU V2 and KITTI.
1. Introduction
- The paper identifies data imbalance, rapid depth changes, and the need for long-range context as challenges in complex scenes with varied object scales.Perspective effects produce many more small-depth than large-depth samples.Source: §1 · pp. 2–3
- The framework recasts monocular depth estimation as multi-category dense labeling rather than regression, using classification to predict depth labels.The authors describe an end-to-end mapping from a color image to its corresponding depth map.Source: §1 · p. 3
- Dilated convolutions expand the receptive field, while hierarchical feature fusion combines multi-scale depth cues for scale-aware estimation.The network is based on a deep residual network.Source: §1 · p. 3
- The method achieves state-of-the-art performance on the indoor and outdoor benchmarks NYU V2 and KITTI.The conclusion also reports superiority over current state-of-the-art methods on both indoor and outdoor benchmarking datasets.Source: §1 · p. 4
- Soft-weighted-sum inference converts discrete depth labels into continuous depth values to reduce quantization error and improve robustness.The network is trained with multinomial logistic loss, and experiments indicate it learns a probability distribution among depth categories.Source: §1 · pp. 3–4
2. Related work
Prior monocular-depth work spans hand-crafted geometric and MRF/CRF approaches, deep CNNs, unsupervised learning from image sequences, and fully convolutional dense labeling. This paper's context places its approach among CNN-based depth estimation and dense-labeling methods.
2. Related work
- Earlier geometric and MRF/CRF methods used scene assumptions or hand-crafted features, which limited their applicability to complex scenarios.Examples include Blocks World, Origami World, shape from shading, and repeated structures.Source: §2 · pp. 4–5
- Reported deep-CNN approaches include Eigen et al.’s hierarchical deep framework, a patch-based CNN and hierarchical-CRF model, a CRF-CNN joint model, and CNN architecture for joint semantic labeling and monocular depth prediction.Laina et al. proposed using the Huber loss.Source: §2 · p. 5
- Garg et al. trained monocular depth estimation through image reconstruction; Godard et al. replaced explicit training depth with binocular stereo footage enforcing consistency; another method learned depth semi-supervisedly from sparse ground-truth depth [27].Source: §2 · p. 6
- Fully convolutional networks and dilated convolutions provide precedents for dense labeling and expanded receptive fields while retaining feature-map resolution,,.The paper describes its hierarchical fusion strategy as simpler than while achieving comparable results.Source: §2 · pp. 6–7
3. Our Framework
The framework maps color images to depth through a dense-labeling formulation, hierarchical multi-scale feature fusion, and soft-weighted-sum inference. Its network design, inference procedure, data augmentation, and training setup are described as parts of this approach.
3. Our Framework
- Monocular depth estimation is reformulated as multi-class labeling, with a hierarchical dilated CNN mapping RGB images to depth scores and soft-weighted-sum inference producing depth values.The framework is trained with classification loss and uses soft-weighted-sum inference at test time.Source: §3 · p. 7, §3.2 · p. 10
3.1. Network Architecture
- The ResNet-based architecture removes fully connected layers, expands receptive fields through dilation, and concatenates intermediate features for multi-scale fusion and boundary preservation.The paper reports that fully connected layers are unnecessary in this network because of dilated convolution.Source: §3.1 · pp. 7–9
3.2. Soft-Weighted-Sum Inference
- Depth is quantized into labels, then soft-weighted-sum inference converts predicted bin scores into continuous depth values to reduce quantization error and improve robustness.The experiments use 200 bins.Source: §3 · p. 7, §3.2 · p. 10
3.3. Data Augmentation
- The training datasets are augmented fourfold using random color changes, scaling and center crops, horizontal flips, and rotations.Color factors range from 0.9 to 1.1, and scale factors from 1.3 to 1.5.Source: §3.3 · p. 11
3.4. Implementation details
- Training initializes from ResNet and runs for 50k iterations, using a 0.001 learning rate for the first 30k before reducing it tenfold every 10k iterations.The implementation uses stochastic gradient descent with momentum 0.9 and weight decay 0.0004.Source: §3.4 · p. 11
4. Experimental Results
The experiments evaluate monocular depth estimation on the NYU V2 indoor dataset and KITTI outdoor dataset, comparing against published methods. The reported qualitative comparisons describe high visual quality without post-processing on both datasets.
4. Experimental Results
- The evaluation covers indoor and outdoor scenes, using NYU V2 and KITTI and comparing with published state-of-the-art methods.The reported quantitative evaluation uses established error metrics.Source: §4 · pp. 11–12
4.1. NYU V2 dataset
- NYU V2 split consists of 249 training and 215 testing scenes; equally sampled training-sequence frames yield approximately 24k unique images, and offline augmentation produces approximately 96k RGB items.Depth values are quantized into 200 bins, and network input is 240 × 320.Source: §4.1 · p. 12
4.2. KITTI dataset
- KITTI evaluation uses a lower-image crop, while training inputs the full image for additional context.The dataset split contains 22,600 training images and 697 testing images; depth is quantized into 50 bins.Source: §4.2 · p. 13
- The paper reports that its results are of high visual quality, although it applied no post-processing methods, presenting this as a qualitative assessment.Source: §4.2 · p. 13
4.1. NYU V2 dataset
- On NYU V2, Figure 4 qualitatively compares estimated depth maps, and the authors report their results have high visual quality despite applying no post-processing methods.Source: §4.1 · p. 12, Figure 4 · p. 14
5. Performance Analysis
The analysis tests architecture components and examines the network’s depth-score distributions and inference choices. It reports that dilated convolution and hierarchical fusion matter, while detailed distance variation remains difficult to distinguish.
5.1. Effect of Architecture Design
- Both dilated convolution and hierarchical fusion play important roles in the architecture evaluation.The component evaluation is reported in Table 3.Source: Table 3 · p. 15, §5.1 · p. 15
5.2. Effect of Soft-Weighted-Sum Inference
- The network’s output probabilities roughly follow a symmetric, Gaussian-like distribution and become more concentrated during training.The authors use multinomial logistic loss without explicitly distinguishing distances between nearby and farther classes.Source: §5.2 · p. 16
- Under hard-max inference, most prediction errors occur between nearby depth classes.The confusion matrix is described as diagonally dominant and symmetric.Source: §5.2 · p. 16
- With more depth bins, pixel accuracy drops sharply while relative error remains stable, indicating difficulty distinguishing fine distance variations.The authors describe the network’s depth-perception ability as limited even when trained with detailed labels.Source: §5.2 · p. 16
6. Conclusions
The proposed classification-based framework combines dilated convolution with hierarchical multi-scale feature fusion for monocular depth estimation. The authors report superiority on indoor and outdoor benchmarks and say the model learns distributions over depth labels.
6. Conclusions
- The proposed end-to-end classification-based framework uses dilated convolution and hierarchical fusion of multi-scale features for monocular depth estimation.The authors present this combination as addressing real-world difficulties in multi-scale depth estimation.Source: §6 · p. 17
- Experiments on indoor and outdoor benchmarks show superiority over current state-of-the-art methods.Source: §6 · p. 17
- The model learns a probability distribution among depth labels, which inspires soft-weighted-sum inference.Source: §6 · p. 17
- Figure 7 shows an NYU2 confusion matrix after merging 200 depth bins into 50.Source: Figure 7 · p. 18