Source-linked AI summary
High Quality Monocular Depth Estimation via Transfer Learning
Ibraheem Alhashim, Peter Wonka
TL;DR
Monocular depth estimation remains limited by blurry, low-resolution maps despite its importance for scene understanding and reconstruction. The paper uses transfer learning in a simple encoder-decoder with composite training and augmentation strategies, achieving state-of-the-art results on NYU Depth v2 and the proposed Unreal-1K dataset while preserving object boundaries more faithfully.
Problem
Existing monocular depth methods leave the quality and resolution of estimated depth maps open for improvement, despite depth estimation’s importance for scene understanding and reconstruction.
Method
The method combines a pre-trained encoder, a simple encoder-decoder architecture, a composite loss, and efficient augmentation for single-RGB-image depth estimation.
Results
The approach outperforms state-of-the-art methods on standard depth-estimation datasets and achieves the best generalization on a novel dataset.
Takeaways & Limitations
A well-constructed, meaningfully initialized encoder can produce high-quality depth maps with more faithful object boundaries using a simple architecture.
Abstract
from arXiv · showhide
Accurate depth estimation from images is a fundamental task in many applications including scene understanding and reconstruction. Existing solutions for depth estimation often produce blurry approximations of low resolution. This paper presents a convolutional neural network for computing a high-resolution depth map given a single RGB image with the help of transfer learning. Following a standard encoder-decoder architecture, we leverage features extracted using high performing pre-trained networks when initializing our encoder along with augmentation and training strategies that lead to more accurate results. We show how, even for a very simple decoder, our method is able to achieve detailed high-resolution depth maps. Our network, with fewer parameters and training iterations, outperforms state-of-the-art on two datasets and also produces qualitatively better results that capture object boundaries more faithfully. Code and corresponding pre-trained weights are made publicly available.
1. Introduction
The paper targets high-quality, high-resolution monocular depth estimation for applications that require faithful depth discontinuities. It proposes a simple transfer-learning encoder-decoder that improves accuracy and visual quality while using fewer resources.
- Depth estimation supports scene understanding, reconstruction, navigation, augmented reality, image refocusing, and segmentation.
- Current CNN-based methods still produce depth maps with limited quality and resolution, including large perturbations near depth discontinuities.
- The proposed architecture repurposes high-performing pre-trained classification networks as deep-feature encoders within a modular transfer-learning design.
- The method captures object boundaries more faithfully with fewer parameters and fewer training iterations than existing methods.
- Experiments report higher performance on standard datasets and the best generalization on a novel photo-realistic synthetic indoor dataset.
2. Related Work
Prior work addresses monocular depth estimation, reconstruction, and transfer learning, but estimated depth quality and resolution remain open challenges. This paper focuses on improving boundaries using a straightforward encoder-decoder and transferred classification features.
- 3D reconstruction from RGB images is ill-posed because incomplete coverage, scale ambiguity, and translucent or reflective materials make geometry ambiguous.
- CNN methods for monocular depth estimation have improved, but the quality and resolution of their estimated depth maps still leave room for improvement.
- The paper seeks better single-image depth estimation by leveraging simple architectures that perform well on other computer-vision tasks.
- Transfer learning motivates using image encoders originally designed for image classification, especially encoders that preserve spatial resolution.
- Unlike larger systems that may use multiple encoder-decoder networks, the method employs one straightforward encoder-decoder with skip connections.
3. Proposed Method
The proposed method estimates a depth map from one RGB image with a simple encoder-decoder, a composite loss, reciprocal-depth targets, and task-appropriate augmentation. Its architecture combines a pre-trained DenseNet encoder with upsampling and skip connections.
- Architecture: The method uses a straightforward encoder-decoder architecture and evaluates how encoder and decoder complexity relates to depth-estimation performance.
- Architecture: A DenseNet-169 encoder pretrained on ImageNet converts the RGB image into features, while upsampling layers and skip connections reconstruct the depth map.
- Loss Function: The loss combines point-wise depth error, depth-gradient error, and Structural Similarity loss as L(y, ˆy) = λLdepth(y, ˆy) + Lgrad(y, ˆy) + LSSIM(y, ˆy).
- Loss Function: The depth loss weight is set to λ = 0.1, and reciprocal-depth targets compensate for losses becoming larger at greater ground-truth depths.
- Learning and Inference: Augmentation uses horizontal flipping with probability 0.5 and color-channel permutations with probability 0.25, while excluding rotations and vertical flips.
4. Experimental Results
Experiments evaluate the network on NYU Depth v2, KITTI, and Unreal-1k using quantitative and qualitative measures, plus ablations of architecture and training choices. The method achieves strong benchmark performance, better visual depth quality, and improved generalization, while sparse KITTI depths limit convergence.
- Datasets and evaluation: The evaluation compares the network with state-of-the-art methods on standard datasets and tests generalization on a newly proposed dataset.Experiments include NYU Depth v2, KITTI, and Unreal-1k.
- Qualitative evaluation: The evaluation measures depth-image similarity, edge agreement, and surface-normal accuracy using mSSIM, thresholded gradient F1, and mean cosine distance.Higher values are better for mSSIM and edge F1, while lower values are better for surface-normal error.
- Qualitative evaluation: The method produces depth edges that better match ground truth and significantly fewer artifacts than state-of-the-art methods in qualitative comparisons.The comparisons include depth maps, extracted normal maps, and surface-normal error visualizations.
- Quantitative results: NYU Depth v2 results achieve state-of-the-art on all but two quantitative metrics and outperform the existing state-of-the-art with fewer parameters, iterations, and training samples.The comparison uses 42.6M versus 110M parameters, 1M versus 3M training iterations, and 50K versus 120K samples.
- Quantitative results: KITTI results rank second on all standard metrics, while the predicted depth maps have better quality than state-of-the-art outputs.The authors attribute weaker quantitative performance partly to KITTI’s very sparse depth maps and the loss function’s region-based edge and appearance objectives.
- Ablation studies: Ablations show that DenseNet-201 lowers validation loss but more than doubles parameters, making its gains unjustified by slower learning and higher GPU-memory requirements.Halving decoder features reduces performance and increases instability, while removing color augmentation promotes rapid overfitting.
- Generalization: On Unreal-1k, the method outperforms the other two methods on average errors while also being evaluated with the mSSIM quality measure.Unreal-1k contains photo-realistic indoor scenes with nearly perfect ground-truth depths.
5. Conclusion
The paper concludes that transfer learning with a well-constructed encoder enables high-quality monocular depth estimation, achieving state-of-the-art performance on NYU Depth v2 and Unreal-1K. Ablations examine encoder depth, decoder width, and color-swapping augmentation.
- A well-constructed encoder initialized with meaningful weights can outperform methods using expensive multistage estimation or multiple feature-encoding layers.
- Figure 5 evaluates three standard-model variations: a deeper DenseNet-201 encoder, a half-width decoder, and disabled color-swapping augmentation.
- The method achieves state-of-the-art performance on the NYU Depth v2 and Unreal-1K datasets.
A.1. Network Architecture
The network uses a DenseNet-169-based encoder-decoder with bilinear upsampling and leaky-ReLU decoder layers. Inputs remain in [0, 1], while target depths are clipped to [0.4, 10] meters.
- The encoder is based on DenseNet-169 with classification-specific top layers removed.
- The decoder begins with a 1 × 1 convolution and uses successive 2× bilinear upsampling blocks followed by 3 × 3 convolutions.
- Decoder convolutions use leaky ReLU activations with parameter α = 0.2.
- Input colors are kept in [0, 1] without normalization, and target depth maps are clipped to [0.4, 10] meters.
A.2. The Unreal-1K Dataset
Unreal-1K is a photo-realistic synthetic indoor dataset designed to test depth-estimation generalization with near-perfect ground truth. It spans varied scenes, objects, viewpoints, and lighting conditions.
- The dataset contains 32 living-area, kitchen, and office scenes with realistic materials and different lighting scenarios.
- Images and corresponding depth maps are captured at 640 × 480 resolution while a virtual camera moves around selected objects.
- More than 20K images were collected, with 1K randomly selected as the Unreal-1K testing dataset.
A.3. Additional Ablation Studies
Additional ablations evaluate training choices using validation loss on NYU Depth v2. Transfer learning and skip connections improve performance, while batch size 8 performs best among the tested values.
- The visual comparison lists input RGB images, ground truth, Laina et al., the proposed method, and Fu et al., with each depth map normalized to its ground-truth range.
- The additional ablations examine validation loss over 750K iterations, corresponding to 15 epochs, on NYU Depth v2.
- Training from scratch greatly increases validation loss compared with ImageNet-pretrained initialization.
- Removing skip connections decreases validation loss relative to the standard model, indicating worse depth-estimation performance.
- Batch size 8 gives the best performance among batch sizes 2, 8, and 16 while maintaining reasonable training time.