PaperlayerView exact revision on arXiv

Source-linked AI summary

Structure-Aware Residual Pyramid Network for Monocular Depth Estimation

Xiaotian Chen, Xuejin Chen, Zheng-Jun Zha

arXiv:1907.06023v1cs.CV

TL;DR

Monocular depth estimation must recover pixel-wise depth from an RGB image, yet CNN methods do not explicitly account for hierarchical scene structure across scales. SARPN predicts depth coarse-to-fine with residual refinement and adaptive multi-scale feature fusion; the paper reports state-of-the-art quantitative and qualitative performance on NYUD v2.

  • Problem

    CNN-based depth methods extract and fuse multi-scale features, but do not explicitly account for hierarchical scene structure spanning large layouts and local details.

    Source: §1 · pp. 1–2

  • Method

    SARPN uses a coarse-to-fine residual pyramid to add finer structures to coarser predictions, with ADFF adaptively selecting features from all scales.

    Source: §1 · p. 2

  • Results

    SARPN reports state-of-the-art quantitative and qualitative performance, with significantly improved visual quality on NYUD v2.

    Source: §1 · p. 2, §5 · p. 6

  • Takeaways & Limitations

    The method is reported to preserve hierarchical scene structures while producing accurate depth estimates for large shapes and fine details.

    Source: §1 · p. 2, §5 · p. 6

Abstract

from arXiv · show

Monocular depth estimation is an essential task for scene understanding. The underlying structure of objects and stuff in a complex scene is critical to recovering accurate and visually-pleasing depth maps. Global structure conveys scene layouts, while local structure reflects shape details. Recently developed approaches based on convolutional neural networks (CNNs) significantly improve the performance of depth estimation. However, few of them take into account multi-scale structures in complex scenes. In this paper, we propose a Structure-Aware Residual Pyramid Network (SARPN) to exploit multi-scale structures for accurate depth prediction. We propose a Residual Pyramid Decoder (RPD) which expresses global scene structure in upper levels to represent layouts, and local structure in lower levels to present shape details. At each level, we propose Residual Refinement Modules (RRM) that predict residual maps to progressively add finer structures on the coarser structure predicted at the upper level. In order to fully exploit multi-scale image features, an Adaptive Dense Feature Fusion (ADFF) module, which adaptively fuses effective features from all scales for inferring structures of each scale, is introduced. Experiment results on the challenging NYU-Depth v2 dataset demonstrate that our proposed approach achieves state-of-the-art performance in both qualitative and quantitative evaluation. The code is available at https://github.com/Xt-Chen/SARPN.

1 Introduction

CNN-based depth estimation has improved, but existing methods do not explicitly model hierarchical scene structure across large layouts and fine details. SARPN predicts structure coarse-to-fine and reports state-of-the-art performance on NYUD v2 with improved visual quality.

  • Existing CNN methods struggle to preserve scene structure across scales, including accurate large planar regions and fine boundaries or object parts.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • The Residual Pyramid Decoder represents global layout at coarser levels and local object structure at finer levels in a coarse-to-fine prediction process.Source: §1 · p. 2
  • Residual refinement modules progressively add finer structures to the coarser scene structure predicted at upper levels.Source: §1 · p. 2
  • ADFF adaptively selects features from all scales to predict residual depths at different structure scales.Source: §1 · p. 2
  • SARPN achieves state-of-the-art performance on NYUD v2, with significantly improved visual quality of recovered depth maps.Source: §1 · p. 2

2 Related Work

Prior work developed CNN architectures, feature-fusion strategies, geometry-focused losses, and joint-task learning for depth estimation. Despite these advances, complex scenes with structures spanning room layouts to fine details remain challenging.

  • Depth estimation moved from multi-scale CNNs to FCNs, with ResNet features and up-projection improving resolution and accuracy.Source: §2 · p. 2
  • CRF postprocessing or CNN integration [Xu et al., 2017], attention [Xu et al., 2018b], multi-scale architectures [Fu et al., 2018], and dilated convolutions target detail or feature recovery.Source: §2 · p. 2
  • Multi-scale feature fusion can sharpen boundaries [Hu et al., 2019], but methods still face inaccurate predictions when complex-scene structure ranges from room layouts to fine details.Source: §2 · p. 2
  • Geometry-focused losses address global layouts with order-sensitive softmax [Zheng et al., 2018] or ordinary regression [Fu et al., 2018], and local details with depth, normal, and gradient terms.Source: §2 · p. 2
  • Joint-task approaches share information across depth, segmentation, and normal estimation, using multi-task networks, auxiliary tasks, recursive refinement, or learned sharing [Eigen and Fergus, 2015; Xu et al., 2018a; Zhang et al., 2018; Jiao et al., 2018].Source: §2 · p. 2

3 Methodology

SARPN combines multiscale feature extraction, dense feature fusion, and a coarse-to-fine residual pyramid to predict depth. Its decoder progressively adds structure detail to coarse scene layouts.

3.1 Structure-Aware Residual Pyramid Network

  • The encoder uses SENet to extract multiscale features, which feed dense fusion; decoding starts from a coarse depth map representing scene layout.Source: §3.1 · pp. 2–3

3.2 Residual Pyramid Decoder

  • At each finer scale, the Residual Refinement Module adds a residual map to upsampled upper-level depth, then refines it to add detail while preserving global layout.Source: §3.2 · p. 3

3.3 Adaptive Dense Feature Fusion

  • ADFF builds a fused feature pyramid with one Multi-scale Feature Fusion module per level, adaptively selecting features from all scales for that scale's prediction.Source: §3.3 · p. 3

3.4 Loss Function

  • Training combines losses across all L scales, with per-scale depth, gradient, and normal terms targeting pixel differences, edge errors, and fine details.Source: §3.4 · p. 4

4 Experiments

The experiments evaluate SARPN on NYUD v2 against recent methods, using quantitative metrics and visual comparisons. They also test component contributions and generalization to ScanNet and SUN-RGBD when trained only on NYUD v2.

4.1 Experimental Setup

  • Training uses 50K images, ImageNet-pretrained SENet, and 20 epochs with batch size 6; predictions are 152 × 114 and bilinearly upsampled for testing.Inputs are downsampled and center-cropped before prediction.Source: §4.1 · pp. 4–5

Quantitative Evaluation

  • On NYUD v2 single-task evaluation, SARPN leads on REL, log10, and all three threshold accuracies, ranks third in RMS, and beats three joint-task methods while matching Jiao et al. [2018].Jiao et al. [2018] uses extra semantic-segmentation labels during training.Source: Quantitative Evaluation · p. 5
  • RPD reduces REL, RMS, and log10 error by 6.5%, 3.5%, and 3.8%; adding ADFF reduces them further by 3.5%, 2.7%, and 4%.The comparison adds RPD to a UNet-like baseline, then adds ADFF.Source: Quantitative Evaluation · pp. 5–6
  • SARPN’s edge F1 score surpasses all other methods under three thresholds.Table 2 reports recovered edge-pixel accuracy under different thresholds.Source: Table 2 · p. 5, Quantitative Evaluation · p. 6

Qualitative Evaluation

  • NYUD2 comparisons show preserved object details and large planar regions, including bookshelf, chair, and sofa details and a wall and floor.Figure 4 also compares SARPN with Jiao et al. [2018], highlighting object-boundary depth and finer structures.Source: Figure 4 · p. 4, Figure 5 · p. 5, Qualitative Evaluation · p. 6
  • In novel-view 3D projections, SARPN’s results are closest to ground truth, with flatter recovered walls than competing methods.The projections are used to illustrate preservation of large planar regions.Source: Qualitative Evaluation · p. 6, Figure 6 · p. 6

Model Generalization

  • Trained only on NYUD v2, SARPN recovers multi-scale structures on ScanNet and SUN-RGBD despite their different data distributions, and fills ground-truth depth holes while maintaining scene structure.Figure 7 shows results on ScanNet and SUN-RGBD.Source: Model Generalization · p. 6, Figure 7 · p. 6

5 Conclusion

SARPN achieves state-of-the-art quantitative and qualitative performance. Its residual pyramid decoder predicts multi-scale depth maps while adaptively fusing image features from all scales.

  • SARPN achieves state-of-the-art performance in both quantitative and qualitative evaluation.Source: §5 · p. 6
  • The residual pyramid decoder predicts multi-scale depth maps that account for hierarchical scene structure.Source: §5 · p. 6
  • The residual pyramid progressively adds finer structures while preserving the coarser layout predicted at upper levels.Source: §5 · p. 6
  • Adaptive dense feature fusion combines image features from all scales when predicting each scale’s residual depth map.Source: §5 · p. 6
Loading 1907.06023v1…