Source-linked AI summary
From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation
Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, Il Hong Suh
TL;DR
Monocular depth estimation is ill-posed, and encoder-decoder networks can lose spatial resolution needed for dense prediction. The paper introduces multi-stage local planar guidance layers and reports state-of-the-art results with significant margins on NYU Depth V2 and KITTI, while KITTI experiments reveal artifacts in upper scenes due to sparse ground truth.
Problem
Single-image depth estimation is ill-posed, while encoder-decoder networks reduce feature-map resolution and require techniques to recover effective dense predictions.
Method
The network places local planar guidance layers at 1/8, 1/4, and 1/2 decoding resolutions, learning plane coefficients to guide feature maps toward full-resolution depth.
Results
The method achieves state-of-the-art performance with significant margins on NYU Depth V2 and KITTI, including more than 10% RMSE improvement in KITTI’s 0–50m range over Fu et al.
Takeaways & Limitations
Ablation results identify the local planar guidance layers as the largest source of improvement, requiring only 0.1M additional trainable parameters.
Takeaways & Limitations
KITTI results show frequent artifacts in upper scenes, attributed to the dataset’s highly sparse ground-truth depth.
Abstract
from arXiv · showhide
Estimating accurate depth from a single image is challenging because it is an ill-posed problem as infinitely many 3D scenes can be projected to the same 2D scene. However, recent works based on deep convolutional neural networks show great progress with plausible results. The convolutional neural networks are generally composed of two parts: an encoder for dense feature extraction and a decoder for predicting the desired depth. In the encoder-decoder schemes, repeated strided convolution and spatial pooling layers lower the spatial resolution of transitional outputs, and several techniques such as skip connections or multi-layer deconvolutional networks are adopted to recover the original resolution for effective dense prediction. In this paper, for more effective guidance of densely encoded features to the desired depth prediction, we propose a network architecture that utilizes novel local planar guidance layers located at multiple stages in the decoding phase. We show that the proposed method outperforms the state-of-the-art works with significant margin evaluating on challenging benchmarks. We also provide results from an ablation study to validate the effectiveness of the proposed method.
1. Introduction
Monocular depth estimation is ill-posed because many 3D scenes can produce the same image, while dense prediction networks lose spatial resolution during encoding. The paper proposes multi-stage local planar guidance to connect encoded features explicitly to full-resolution depth estimation.
- Single-image depth estimation matters for applications including robotics, autonomous driving, scene understanding, and 3D reconstruction.
- Infinitely many 3D scenes can project to the same 2D image, making accurate monocular depth estimation challenging.
- DCNN depth networks typically use an encoder for dense feature extraction and a decoder for prediction, but strided operations reduce transitional spatial resolution.
- The proposed architecture places local planar guidance layers at decoding resolutions 1/8, 1/4, and 1/2, combining their outputs for full-resolution depth prediction.
- Unlike separate multi-scale depth estimates, the layers learn 4-dimensional plane coefficients jointly to reconstruct the final full-resolution depth.
2. Related Work
Prior monocular depth work includes supervised, semi-supervised, weakly supervised, self-supervised, and sequential-data approaches. The paper’s architecture combines dense features, contextual extraction, and multi-scale local planar guidance for depth prediction.
- Supervised monocular methods learn mappings from visual cues to depth using measured depth from RGB-D cameras or multi-channel laser scanners.
- Semi-supervised and weakly supervised methods use relative depth, ranking losses, sparse LiDAR supervision, or image-alignment objectives to reduce reliance on dense ground truth.
- The proposed network combines a dense feature extractor, ASPP contextual extractor, local planar guidance layers, and dense connections for final estimation.
- Self-supervised methods train from rectified stereo pairs by synthesizing one view from the other and minimizing reconstruction error.
- Sequential-data approaches impose constraints through adversarial subnetworks, inferred 3D geometry, point-cloud consistency, or ego-motion.
3. Method
The method introduces multi-scale local planar guidance layers in the decoder to connect internal features directly to full-resolution depth prediction. Local planar cues from multiple resolutions are combined, allowing coarse structures and fine details to contribute to the final estimate.
- Multi-Scale Local Planar Guidance: The decoder combines guidance outputs from 1/8, 1/4, and 1/2 resolutions with other features to predict depth at full resolution.The proposed layer recovers an internal feature map to resolution H, allowing its output to participate in the final nonlinear depth combination.
- Multi-Scale Local Planar Guidance: The guidance layer predicts constrained plane coefficients through 1 × 1 convolutions and separate activations before computing locally defined relative depths.The coefficient representation uses a unit normal vector and a sigmoid-scaled perpendicular distance.
- Multi-Scale Local Planar Guidance: The network places local planar guidance layers at multiple decoding stages to guide feature maps toward full-resolution depth prediction.The layers operate during decoding and serve as skip connections that establish direct relations between internal features and the final prediction.
- Multi-Scale Local Planar Guidance: Each guidance layer converts estimated plane coefficients into k × k local depth cues using ray-plane intersection.The coefficients include a plane normal and perpendicular distance, while normalized patch coordinates define the local pixels.
- Multi-Scale Local Planar Guidance: Coarser scales learn global shapes, while finer scales learn local details and can compensate for erroneous estimations at other scales.The paper reports that fine-scale outputs recover details missing from coarser estimates, including details around object boundaries.
- Training Loss: Training uses a scale-invariant log-depth loss composed of error variance and a weighted squared mean, with λ set to 0.85 and α set to 10.The authors state that scaling the loss range improves convergence and the final training result.
4. Experiments
Experiments evaluate the proposed method on indoor and outdoor depth benchmarks, using multiple encoders, standard metrics, online KITTI evaluation, and ablations. The method achieves strong benchmark performance, while sparse KITTI ground truth is associated with artifacts in some scene regions.
- Experimental Setup: The experiments cover NYU Depth V2 and KITTI, including standard depth metrics, an online KITTI evaluation, encoder comparisons, and an ablation study.The evaluation uses threshold accuracy, Abs Rel, Sq Rel, log10, and RMSE metrics.
- NYU Depth V2 Results: On NYU Depth V2, Ours-DenseNet achieves state-of-the-art performance with a significant margin on inlier and accuracy metrics except RMSE.The reported inlier measures are δ < thr, while the accuracy metrics are AbsRel and log10.
- KITTI Results: On KITTI, the proposed method outperforms existing works with a significant margin, although its 0–80m RMSE trails Fu et al.’s result.Within the 0–50m range, Ours-ResNet achieves more than 10% RMSE improvement over Fu et al.’s result.
- KITTI Results: On the online KITTI benchmark server, the method outperforms all published works using a model trained with KITTI’s official split.Ours-DenseNet was trained for 50 epochs with 28,654 sampled image-ground-truth pairs.
- Ablation Study: Adding core modules improves overall performance, with the largest ablation gain attributed to local planar guidance layers requiring only 0.1M additional trainable parameters.The final improvement comes from the training loss defined in Equation 5.
- Qualitative Results: Qualitative KITTI results show more precise object boundaries but artifacts in sky and upper-scene regions caused by very sparse ground-truth depth.The authors attribute these artifacts to regions lacking valid depth values across the dataset.
5. Conclusion
The supervised monocular depth estimation network achieves state-of-the-art results using multi-stage local planar guidance, while qualitative results show clearer object boundaries. KITTI experiments also reveal artifacts linked to sparse ground-truth depth, motivating denser photometric supervision.
- The supervised monocular depth estimation network achieves state-of-the-art results on challenging benchmarks.
- Multi-stage local planar guidance provides an explicit relation between internal feature maps and desired depth predictions, improving network training.The layers are deployed at multiple decoding stages and guide feature maps toward depth prediction.
- KITTI experiments reveal frequent upper-scene artifacts, analyzed as an effect of highly sparse ground-truth depth.The authors plan to investigate photometric reconstruction loss for denser supervision and further improvement.
- Qualitative results show clearer vehicle and traffic-sign boundaries on the KITTI Eigen test split.
- On NYU Depth V2, the method produces more distinctive object boundaries, although Yin et al. show competitive results.