Source-linked AI summary
Deeper Depth Prediction with Fully Convolutional Residual Networks
Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, Nassir Navab
TL;DR
The paper tackles the ambiguity of predicting scene depth from one RGB image, a task for which accurate depth supports several computer-vision applications. It introduces an end-to-end fully convolutional residual network with learned up-sampling and reverse Huber optimization. The resulting model uses fewer parameters and training data while achieving state-of-the-art depth-estimation results on two benchmark datasets.
Problem
Single-image depth estimation is ill-posed because color or intensity measurements map ambiguously to depth, although accurate depth benefits several RGB-based vision tasks.
Method
The paper uses an end-to-end fully convolutional residual CNN with up-projection blocks for learned feature-map up-sampling and a reverse Huber loss for training.
Results
The model achieves higher-quality, state-of-the-art results on two depth-estimation benchmarks while using fewer parameters and less training data than existing methods.
Takeaways & Limitations
A single-scale fully convolutional architecture can deliver simpler, faster-to-train monocular depth estimation without the multi-step refinement used by typical CNN approaches.
Abstract
from arXiv · showhide
This paper addresses the problem of estimating the depth map of a scene given a single RGB image. We propose a fully convolutional architecture, encompassing residual learning, to model the ambiguous mapping between monocular images and depth maps. In order to improve the output resolution, we present a novel way to efficiently learn feature map up-sampling within the network. For optimization, we introduce the reverse Huber loss that is particularly suited for the task at hand and driven by the value distributions commonly present in depth maps. Our model is composed of a single architecture that is trained end-to-end and does not rely on post-processing techniques, such as CRFs or other additional refinement steps. As a result, it runs in real-time on images or videos. In the evaluation, we show that the proposed model contains fewer parameters and requires fewer training data than the current state of the art, while outperforming all approaches on depth estimation. Code and models are publicly available.
1. Introduction
Monocular depth estimation is ill-posed because image appearance ambiguously maps to depth, yet accurate depth benefits several vision tasks. The paper proposes a fully convolutional CNN with efficient up-sampling and reverse Huber optimization, reporting strong benchmark performance with fewer parameters and training data.
- Single-image depth estimation is ill-posed because intensity or color measurements can correspond to ambiguous depth values.
- Accurate depth information improves RGB-only performance in reconstruction, recognition, semantic segmentation, and human pose estimation.
- The proposed fully convolutional architecture uses novel up-sampling blocks to produce higher-resolution dense outputs with fewer parameters and one order of magnitude less training data than the state of the art.
- Residual learning and up-projection blocks provide an efficient scheme for up-sampling feature maps.
- The network is trained with a reverse Huber loss that the authors argue is better suited to depth-map value distributions.
- The trained model is also evaluated in a 3D reconstruction scenario using RGB sequences and predicted depth maps for SLAM.
2. Related Work
Prior monocular-depth methods used geometric assumptions, feature matching, CNNs, and graphical-model refinement. This work instead uses a fully convolutional CNN with efficient residual up-sampling blocks to reduce parameter costs while targeting high-dimensional depth regression.
- Classic monocular-depth methods used hand-crafted features and probabilistic graphical models, often relying on strong scene-geometry assumptions.
- Non-parametric depth-transfer methods matched RGB images to RGB-D repositories, then warped or combined retrieved depth maps.
- The architecture builds on ResNet-50 and replaces its fully-connected layer with up-sampling blocks that yield an output at roughly half the input resolution.
- CNN-based approaches learned mappings from color pixels to depth, often using AlexNet or VGG architectures and multi-scale refinement.
- Other methods combined CNNs with CRFs or regression forests, increasing pipeline complexity or limiting the need for large datasets.
- The proposed method replaces expensive fully-connected layers with efficient residual up-sampling blocks called up-projections.
3. Methodology
The method uses a fully convolutional, residual architecture to predict dense depth maps from single RGB images, with learned up-sampling and a reverse Huber loss. Its design addresses receptive-field limits, efficient high-resolution prediction, and depth-value distributions.
- CNN Architecture: A fully convolutional network replaces fully connected layers while preserving a large receptive field through a ResNet-50 contractive part.For 304 × 228 inputs, ResNet-50 captures 483 × 483 input regions; an equivalent fully connected layer would require 3.3 billion parameters.
- CNN Architecture: Residual skip layers support deeper networks by bypassing convolutions and summing their outputs, while also providing a large receptive field.ResNet-50 produces 2048 feature maps at 10 × 8 resolution when its last pooling layer is removed.
- Up-Sampling: Up-convolution blocks double feature-map resolution through unpooling, 5 × 5 convolution, and ReLU, and four such blocks are stacked.The architecture learns up-scaling through successive unpooling and convolutional layers before dropout and the final convolution.
- Up-Sampling: Up-projection blocks add a 3 × 3 convolution and a projection connection from the lower-resolution feature map to the up-convolution result.The projection branch uses another up-convolution, and chaining blocks progressively increases feature-map sizes while passing high-level information forward.
- Efficient Up-Sampling: Reformulated up-convolutions avoid computations on zeros introduced by unpooling while producing the same output through composed filters and interleaving.After unpooling, 75% of the resulting feature-map values are zeros; the reformulation decreases whole-network training time by around 15%.
- Loss Function: The reverse Huber loss uses L1 behavior for small residuals and L2 behavior for large residuals, matching the observed heavy-tailed depth-value distribution.The transition threshold is set to 20% of the maximal per-batch pixel error, balancing sensitivity to small residuals with greater weighting of high residuals.
4. Experimental Results
Experiments on NYU Depth v2 and Make3D evaluate architecture, loss, efficiency, qualitative quality, and SLAM reconstruction. The proposed ResNet-UpProj model achieves strong depth predictions with efficient computation and without post-processing.
- Datasets and evaluation: The evaluation compares CNN variants and state-of-the-art methods on NYU Depth v2 and Make3D, covering indoor and outdoor scenes.NYU Depth v2 provides indoor scenes, while Make3D provides outdoor scenes.
- Architecture evaluation: The proposed architecture replaces fully-connected layers with efficient residual up-projection blocks for high-resolution depth prediction.The architecture evaluation motivates this replacement because high-resolution fully-connected variants require many parameters and converge less easily.
- Architecture evaluation: ResNet-UpProj gives by far the best results among the evaluated architecture variants and preserves more output structure than the fully-connected variant.The model uses four successive up-sampling steps, doubling resolution per block.
- Loss evaluation: The berHu loss outperforms L2 in all shown experiments, with especially strong gains in relative error and the δ1 measure.The reported explanation is that berHu has larger gradients than L2 for small residuals, which matter more for nearby depths.
- Efficiency: A single-image depth prediction takes 55ms with up-projection on the reported setup, enabling real-time image processing.The corresponding up-convolution implementation takes 78ms per image.
- Qualitative results: On NYU Depth, the fully convolutional model improves edge quality and structure definition over coarse fully-connected predictions while using one end-to-end model without additional post-processing.The comparison includes AlexNet, VGG, ResNet-FC-64x48, and publicly available predictions from Eigen and Fergus.
- Make3D results: On Make3D, the method significantly outperforms previous works with both L2 and berHu losses, with berHu providing a larger advantage on this challenging dataset.Evaluation uses regions with depth below 70m and upsamples predictions to 345 × 460.
- Application to SLAM: Predicted depths also support qualitative 3D SLAM reconstruction, where improved accuracy and edge preservation produce reconstructions comparable to ground-truth-based results.The comparison includes reconstructions from AlexNet, VGG, the proposed architecture, and ground-truth depth.
5. Conclusion
The method combines a single-scale, fully convolutional residual CNN with learned up-projection layers for monocular depth estimation. It reduces model and data requirements while achieving higher-quality, state-of-the-art results on two benchmark datasets.
- A single-scale CNN uses residual learning and fully convolutional up-projection layers to predict depth without a multi-step refinement process.The architecture produces dense depth predictions directly rather than refining coarse outputs through additional stages.
- Learned up-convolutional layers provide a faster, more efficient approach to increasing feature-map resolution.
- The architecture supports deeper configurations while greatly reducing the number of learned parameters and required training samples.
- The model achieves higher-quality results and state-of-the-art performance on two benchmark depth-estimation datasets.