PaperlayerView exact revision on arXiv

Source-linked AI summary

FCFR-Net: Feature Fusion based Coarse-to-Fine Residual Learning for Depth Completion

Lina Liu, Xibin Song, Xiaoyang Lyu, Junwei Diao, Mengmeng Wang, Yong Liu, Liangjun Zhang

arXiv:2012.08270v2cs.CV

TL;DR

Depth-completion methods can struggle to recover details when feature extraction and supervision are insufficient. FCFR-Net uses a two-stage residual-learning framework with channel shuffle and energy-based fusion; the paper reports superiority across indoor and outdoor benchmarks, including state-of-the-art KITTI RMSE.

  • Problem

    One-stage image-guided depth-completion methods have insufficient feature extraction and supervision to recover depth details.

    Source: Introduction · p. 1

  • Method

    FCFR-Net first produces coarse dense depth and then refines it through residual learning with channel-shuffled, energy-fused color and depth features.

    Source: Introduction · p. 2

  • Results

    FCFR-Net is reported to surpass state-of-the-art methods across indoor and outdoor benchmarks; on KITTI, it achieves state-of-the-art RMSE and improves RMSE by about 10% over STD.

    Source: Introduction · p. 2, Evaluation on KITTI Dataset · p. 5, Conclusion · p. 7

  • Takeaways & Limitations

    The authors conclude that consecutive features from dense depth maps and fused color-depth features support more accurate depth completion.

    Source: Conclusion · p. 7

    The channel-shuffle design assumes an even number of feature channels, as current DCNN-based approaches have even channel counts.

    Source: Channel Shuffle · p. 4

Abstract

from arXiv · show

Depth completion aims to recover a dense depth map from a sparse depth map with the corresponding color image as input. Recent approaches mainly formulate depth completion as a one-stage end-to-end learning task, which outputs dense depth maps directly. However, the feature extraction and supervision in one-stage frameworks are insufficient, limiting the performance of these approaches. To address this problem, we propose a novel end-to-end residual learning framework, which formulates the depth completion as a two-stage learning task, i.e., a sparse-to-coarse stage and a coarse-to-fine stage. First, a coarse dense depth map is obtained by a simple CNN framework. Then, a refined depth map is further obtained using a residual learning strategy in the coarse-to-fine stage with a coarse depth map and color image as input. Specially, in the coarse-to-fine stage, a channel shuffle extraction operation is utilized to extract more representative features from the color image and coarse depth map, and an energy based fusion operation is exploited to effectively fuse these features obtained by channel shuffle operation, thus leading to more accurate and refined depth maps. We achieve SoTA performance in RMSE on KITTI benchmark. Extensive experiments on other datasets future demonstrate the superiority of our approach over current state-of-the-art depth completion approaches.

Introduction

Depth completion must recover dense depth from sparse sensor inputs, but one-stage image-guided methods extract and supervise features insufficiently to recover depth details. FCFR-Net uses a two-stage coarse-to-fine residual framework with channel shuffle and energy-based fusion, and reports state-of-the-art RMSE on KITTI.

  • One-stage image-guided methods have insufficient feature extraction and supervision, while sparse-depth methods can produce blurred maps with unclear boundaries.Source: Introduction · p. 1
  • FCFR-Net formulates depth completion as sparse-to-coarse interpolation followed by coarse-to-fine refinement in a residual-learning framework.Source: Introduction · pp. 1–2
  • Channel shuffle mixes color and depth features at multiple scales to extract representative fused features.Source: Introduction · p. 2
  • Energy-based fusion further combines the channel-shuffled features, and the residual-learning framework is intended to improve depth-completion performance.Source: Introduction · p. 2
  • FCFR-Net achieves state-of-the-art performance in RMSE on the KITTI benchmark, with results on NYUv2 also demonstrating superiority over current approaches.Source: Introduction · p. 2

Related Work

Prior depth-completion work combines sparse depth and color at the signal or feature level, while image fusion research offers varied feature-combination strategies. The paper identifies insufficient micro-level integration of color and depth as a remaining gap.

  • Sparse-depth approaches use zero-valued invalid inputs or gravity-based interpolation, but discontinuities limit performance and interpolation can lose depth detail and semantics.Source: Sparse Depth based Approaches · p. 2
  • Signal-level methods directly connect image and sparse depth inputs, with some approaches using post-processing to obtain sharper results.Source: Signal Level Fusion · p. 2
  • Feature-level depth-completion methods use separate image and depth encoders with approaches such as cross-guidance or spatial-pyramid fusion, but the surveyed methods ignore micro-level integration.Source: Feature Level Fusion · p. 2
  • Image-fusion methods include multiscale CNNs, addition, l1-norm and soft-max fusion, and wavelet-based architectures.Source: Image Fusion · p. 2
  • The paper identifies insufficient color-depth fusion as a gap and presents a two-stage architecture with channel shuffle and energy-based fusion.Source: Image Fusion · p. 2, Figure 2 · p. 3

Our Approach

FCFR-Net formulates depth completion as sparse-to-coarse estimation followed by coarse-to-fine refinement. The second stage fuses color and depth features using channel shuffle and energy-based fusion, then applies residual learning.

Our Approach

  • FCFR-Net uses a sparse-to-coarse stage followed by a coarse-to-fine stage containing channel shuffle, energy-based fusion, and residual learning.The coarse-to-fine stage takes the color image and coarse dense depth map as input.Source: Our Approach · p. 3, Coarse-to-Fine · p. 3

Coarse-to-Fine

Sparse-to-Coarse

  • The sparse-to-coarse stage produces a coarse dense depth map, using STD to reduce computational limitations.Other sparse-to-dense frameworks or handcrafted interpolation can also be used in this stage.Source: Sparse-to-Coarse · p. 3

Coarse-to-Fine

  • Channel shuffle mixes color and depth feature channels at multiple scales to produce new fusion features.The exchanged features are returned to subsequent convolution blocks.Source: Channel Shuffle · p. 4
  • Energy-based fusion selects the feature value with higher regional energy instead of adding or concatenating the color and depth features.The operation is inspired by max pooling and is intended to preserve high-frequency and texture information.Source: Energy based Fusion · p. 4

Loss Function

  • The coarse-to-fine loss compares the recovered depth map with ground truth only at valid ground-truth positions.The passage defines p as either 1 or 2 and uses the number of valid positions as the normalization count.Source: Loss Function · p. 5

Experiment

Experiments evaluate FCFR-Net on KITTI and NYUDv2 and ablate its input, learning, and feature-fusion choices.

KITTI Dataset and Implementation Details

  • KITTI contains 85k training, 6k validation, and 1k testing image-depth pairs; training inputs are bottom-cropped to 352 × 1216.STD is used as the pre-trained sparse-to-coarse network; models are trained for 20 epochs.Source: KITTI Dataset and Implementation Details · p. 5

NYUDv2 Dataset and Implementation Details

  • NYUDv2 uses 45K training images and 654 labeled evaluation images, with inputs down-sampled to 320 × 240 and predictions center-cropped to 304 × 228.The coarse network is frozen during fine-network training, and predictions are made end-to-end at evaluation.Source: NYUDv2 Dataset and Implementation… · p. 5

Evaluation Metrics

Evaluation on KITTI Dataset

Evaluation on NYUDv2 Dataset

  • 13% lower RMSE than STD is reported on NYUDv2, where FCFR-Net also outperforms state-of-the-art approaches across all metrics using 500 sampled points.Qualitative results are described as showing more depth detail and object-edge structure in indoor scenes.Source: Table 2 · p. 6, Evaluation on NYUDv2 Dataset · p. 6

Ablation Studies

Conclusion

FCFR-Net combines a two-stage depth-completion framework with channel shuffle and energy-based fusion; experiments across indoor and outdoor benchmarks demonstrate superiority over state-of-the-art approaches.

Conclusion

  • Experiments across indoor and outdoor benchmarks demonstrate FCFR-Net's superiority over state-of-the-art approaches.The paper reports more accurate depth-completion results from its framework and feature operations.Source: Conclusion · p. 7
  • FCFR-Net formulates depth completion as sparse-to-coarse estimation followed by coarse-to-fine refinement.The authors state that dense depth maps provide consecutive features, enabling better depth results.Source: Conclusion · p. 7
  • Channel shuffle and energy-based fusion extract and fuse color and depth features to obtain more representative features.Source: Conclusion · p. 7
Loading 2012.08270v2…