Source-linked AI summary
Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries
Junjie Hu, Mete Ozay, Yan Zhang, Takayuki Okatani
TL;DR
Single-image depth estimation remains limited by lost spatial resolution and inaccurate object boundaries in many CNN-based methods. The paper proposes a four-module architecture with flexible multi-scale feature fusion and combines depth, gradient, and surface-normal losses. On NYU Depth V2, the method outperforms previous methods, especially for small objects and object boundaries, although standard accuracy measures can miss spatial edge distortions.
Problem
Existing single-image depth methods often lose spatial resolution, producing distorted or blurry object boundaries and missing small objects.
Method
The paper combines an encoder, decoder, multi-scale feature fusion, and refinement modules with complementary depth, gradient, and surface-normal losses.
Results
On NYU Depth V2, the method outperforms previous methods using traditional and proposed step-edge measures, especially for small objects and object boundaries.
Takeaways & Limitations
Higher-resolution depth reconstruction can improve recovery of fine details and clear object boundaries without post-processing refinement.
Takeaways & Limitations
Traditional depth-accuracy measures are limited because they can fail to detect spatial distortion and blur of object edges.
Abstract
from arXiv · showhide
This paper considers the problem of single image depth estimation. The employment of convolutional neural networks (CNNs) has recently brought about significant advancements in the research of this problem. However, most existing methods suffer from loss of spatial resolution in the estimated depth maps; a typical symptom is distorted and blurry reconstruction of object boundaries. In this paper, toward more accurate estimation with a focus on depth maps with higher spatial resolution, we propose two improvements to existing approaches. One is about the strategy of fusing features extracted at different scales, for which we propose an improved network architecture consisting of four modules: an encoder, decoder, multi-scale feature fusion module, and refinement module. The other is about loss functions for measuring inference errors used in training. We show that three loss terms, which measure errors in depth, gradients and surface normals, respectively, contribute to improvement of accuracy in an complementary fashion. Experimental results show that these two improvements enable to attain higher accuracy than the current state-of-the-arts, which is given by finer resolution reconstruction, for example, with small objects and object boundaries.
1. Introduction
Single-image depth estimation has advanced with CNNs, but downsampling and conventional losses still leave distorted boundaries, missing small objects, and blurred depth maps. The paper addresses these issues with flexible multi-scale feature fusion and complementary depth, gradient, and normal losses.
- Motivation: CNN-based depth estimation has advanced, but downsampling commonly reduces estimated-map resolution relative to the input.This loss of spatial resolution is associated with degraded reconstruction quality.
- Motivation: Existing depth maps can distort object shapes, omit small objects, and exhibit mosaic patterns, including state-of-the-art results.Higher-resolution maps with precise boundaries matter for object recognition and depth-aware image re-rendering and editing.
- Proposed approach: Multi-scale fusion combines lower-layer shape details with higher-layer global depth information in a complementary fashion.Lower-layer features are up-scaled to a common resolution through skip connections before fusion.
- Proposed approach: The proposed architecture uses an encoder, decoder, multi-scale feature fusion module, and refinement module for final prediction.It supports several encoder backbones and is trained end to end without post-processing refinement.
- Training objective: Depth, gradient, and surface-normal loss terms are complementary and improve accuracy, particularly around object edges.The added gradient and normal terms target errors that depth-difference losses handle poorly at step edges.
2. Related Work
Related work improves single-image depth estimation through up-projection, encoder-decoder designs, CRF refinement, multi-task learning, and dilated convolutions. This paper instead emphasizes flexible fusion of up-scaled multi-scale features and describes the proposed network components and upsampling strategy.
- Feature fusion: The multi-scale feature fusion module directly merges up-scaled encoder features rather than using only symmetric encoder-decoder skip connections.This design is described as more flexible across decoder-encoder backbones.
- Resolution recovery: Up-projection was introduced to improve resolution recovery after CNN downsampling.It is also the upsampling strategy used in the proposed network.
3. Proposed Method
The method combines a four-module multi-scale network with complementary depth, gradient, and surface-normal losses to improve high-resolution depth estimation and object-boundary accuracy.
- Improved Network Design: The architecture uses an encoder, decoder, multi-scale feature fusion module, and refinement module for final depth prediction.The encoder extracts multi-scale features, while the decoder, MFF, and refinement modules progressively decode, fuse, and integrate them.
- Improved Network Design: The MFF module upsamples encoder features from four scales, concatenates them, and transforms them into a 64-channel fused representation.Upsampling factors are ×2, ×4, ×8, and ×16; lower-layer features retain finer spatial information useful for recovering details lost through downsampling.
- Improved Network Design: The refinement module concatenates decoder and MFF features before three convolutional layers produce the final prediction.MFF and refinement add only 4M parameters compared with 63.6M for the ResNet-50 encoder-decoder portion.
- Loss Functions: Conventional depth-difference losses are comparatively insensitive to positional shifts, distortion, and blur around step edges.Natural range images contain many piecewise-smooth regions separated by step edges, including object boundaries.
- Loss Functions: The training loss combines depth, gradient, and surface-normal terms because they respond complementarily to different depth-map errors.Gradient loss targets edge shifts, while surface-normal loss is sensitive to small depth structures; the combined loss is L = ldepth + λlgrad + µlnormal.
- Accuracy Measures for Depth Estimation: Edge accuracy supplements RMS, relative, log, and thresholded measures by detecting positional errors in estimated object boundaries.The proposed measure applies Sobel operators and evaluates edge precision, recall, and F1 score at thresholds 0.25, 0.5, and 1.
4. Experiments
Experiments on NYU-Depth V2 compare the proposed method with prior approaches and loss variants. The method achieves strong traditional metrics and sharper recovery of object boundaries and small structures.
- Implementation Details: The experiments use NYU-Depth V2 with official 249-scene training and 215-scene testing splits, resized and cropped to 304×228 pixels.Training depth maps are downsampled to 114×152 pixels.
- Performance Comparison: The comparison includes methods using RGB alone, partially known depths, and joint task learning, with the proposed method compared under equal RGB-only conditions.Methods using additional information are included for reference.
- Performance Comparison: Visual comparisons show that the proposed method more accurately recovers object edges and small structures, including bottles and lamp shades.Earlier methods exhibit shape distortion, sharp discontinuities, mosaic effects, or inaccurate weak edges.
- Performance Comparison: The proposed edge-accuracy measure is associated with visual quality: the proposed method achieves the strongest edge accuracy and the finest visual details.A similar relationship is observed for the strongest prior method.
- Ablation Study: With ResNet-50, progressively adding gradient and surface-normal losses to depth loss recovers increasingly finer object details.The full loss combines l_depth, λl_grad, and μl_normal; visual comparisons also show l_depth-only results are more distorted and blurry.
5. Conclusion
The paper presents architectural and loss-function improvements for single-image depth estimation, targeting sharper object boundaries and small-object reconstruction. Experiments on NYU Depth V2 show improved performance over previous methods under both traditional and step-edge measures.
- Architectural improvement: The proposed network uses an encoder, decoder, multi-scale feature fusion module, and refinement module, trained end to end without post-processing.The encoder can use base networks such as ResNet, DenseNet, or SENet.
- Problem addressed: Previous methods often distort object shapes, miss small objects, or produce mosaic patterns because estimated depth boundaries are inaccurate.The paper relates these failures to the loss functions used by earlier methods and to the challenge of reconstructing step edges.
- Loss-function improvement: A combined loss of depth, gradient, and surface-normal differences targets errors around sharp depth discontinuities.Depth differences are insensitive to edge shifts and blurring, whereas gradient differences are more sensitive; surface-normal differences add sensitivity to small surface changes.
- Experimental results: The method outperforms previous approaches on NYU Depth V2 using both traditional measures and a proposed step-edge reconstruction measure.The strongest improvements concern small-object reconstruction and object-boundary estimation, consistent with visual comparisons.