Source-linked AI summary
RigNet: Repetitive Image Guided Network for Depth Completion
Zhiqiang Yan, Kun Wang, Xiang Li, Zhenyu Zhang, Jun Li, Jian Yang
TL;DR
Depth completion must recover dense depth from sparse measurements, but blurry image guidance and unclear depth structures limit image-guided methods. RigNet applies repetition in both image guidance and depth generation, using a repetitive hourglass network and dynamic-convolution guidance modules. The method reports superior or competitive performance on KITTI and NYUv2 and outstanding performance across three datasets.
Problem
Blurry image guidance and unclear depth structures impede detailed depth recovery from sparse measurements.
Method
RigNet uses a repetitive hourglass network for image guidance and a dynamic-convolution repetitive guidance module for progressive depth refinement.
Results
RigNet performs better than other methods in reported experiments and achieves outstanding performance on three datasets.
Takeaways & Limitations
Repetitive image and depth representations provide progressively refined guidance for dense, structure-detailed depth recovery.
Takeaways & Limitations
The efficient guidance algorithm is motivated by the large GPU memory consumption of repeated dynamic convolution.
Abstract
from arXiv · showhide
Depth completion deals with the problem of recovering dense depth maps from sparse ones, where color images are often used to facilitate this task. Recent approaches mainly focus on image guided learning frameworks to predict dense depth. However, blurry guidance in the image and unclear structure in the depth still impede the performance of the image guided frameworks. To tackle these problems, we explore a repetitive design in our image guided network to gradually and sufficiently recover depth values. Specifically, the repetition is embodied in both the image guidance branch and depth generation branch. In the former branch, we design a repetitive hourglass network to extract discriminative image features of complex environments, which can provide powerful contextual instruction for depth prediction. In the latter branch, we introduce a repetitive guidance module based on dynamic convolution, in which an efficient convolution factorization is proposed to simultaneously reduce its complexity and progressively model high-frequency structures. Extensive experiments show that our method achieves superior or competitive results on KITTI benchmark and NYUv2 dataset.
1 Introduction
Depth completion methods use image guidance to recover dense depth, but blurry image guidance and unclear depth structures impede detailed prediction. RigNet addresses both issues with repetitive designs in its image and depth branches, achieving strong results across datasets.
- Motivation: Depth completion recovers dense depth from sparse measurements, with image-guided methods using color images to improve predictions.Reliable depth predictions support applications including autonomous driving, augmented reality, virtual reality, and 3D scene reconstruction.
- Approach: RigNet develops repetitive designs in both the image guidance and depth generation branches to address blurry guidance and unclear depth structures.The proposed mechanism aims to provide gradually refined image and depth guidance.
- Image guidance branch: The vertically repetitive hourglass network uses RGB features from multi-scale layers to provide clearer and richer image semantics for depth recovery.This design differs from tandem models that mainly use final-layer features and parallel models with limited interaction or single-unit guidance.
- Depth generation branch: The repetitive guidance module uses dynamic convolution, convolution factorization, repeated guidance units, and adaptive fusion to progressively refine high-frequency depth structures.An efficient guidance algorithm reduces the memory cost of repeated dynamic convolution while maintaining accuracy.
- Results: RigNet performs better than other methods in the reported experiments and achieves outstanding performance on three datasets.The paper attributes these results to gradually refined image and depth representations.
- Contributions: The proposed network combines a lightweight repetitive hourglass network with a repetitive guidance module containing adaptive fusion and an efficient guidance algorithm.These components are presented as contributions for clearer image guidance and precise depth representations.
2 Related Work
Depth completion research includes depth-only and multimodal approaches, while RigNet uses a repetitive image-guided architecture to produce clearer guidance and structure-detailed depth.
- Depth-only approaches: Depth-only methods recover depth from sparse measurements without using color images, but single-modal approaches are limited without other reference information.Other multimodal inputs include surface normals and optical-flow images.
- Proposed framework: The network overview contains an image guidance branch with a repetitive hourglass network and a depth generation branch with repetitive guidance refinement.The design also includes an efficient guidance algorithm and adaptive fusion mechanism.
- Repetitive image guidance: RigNet repeatedly processes image features in multi-scale layers and repeatedly applies a guidance module in the depth branch.The two repetitive processes target clearer guidance and structure-detailed depth generation.
3 Repetitive Design
RigNet uses repetition in both image guidance and depth generation to progressively refine semantic features and structure-detailed depth. Its repetitive guidance combines efficient dynamic-convolution guidance with adaptive fusion while reducing memory demands.
- Repetitive Hourglass Network: RigNet introduces repetitive hourglass units to progressively extract clearer, hierarchical image guidance semantics.The network repeatedly uses similar lightweight units and combines encoder-decoder features to improve contextual guidance.
- Repetitive Guidance Module: The depth branch uses repetitive guidance to progressively generate dense, structure-detailed depth features from sparse depth and image guidance.The branch has the same architecture as RHN1 and applies guidance at multiple stages.
- Efficient Guidance Algorithm: Efficient guidance factorizes dynamic convolution into channel-wise and cross-channel operations, with channel-wise complexity reduced to O(C × H × W), or 1/R2 of the original.The unit concatenates image and depth inputs, applies convolution and global average pooling, then performs pixel-wise dot products with depth.
- Efficient Guidance Algorithm: 42.75GB to 0.037GB: efficient guidance reduces GPU memory nearly 1155 times versus common dynamic convolution in one fusion stage.Compared with GuideNet's convolution factorization, memory decreases from 0.334GB to 0.037GB, nearly 9 times lower.
- Adaptive Fusion Mechanism: Adaptive fusion combines k coarse depth maps using softmax-derived weights after convolution and global average pooling.The weighted sum produces the refined depth output from the repeated intermediate predictions.
4 RigNet
RigNet consists of an image guidance branch for hierarchical, clear semantics and a depth generation branch for structure-detailed predictions. It trains the network with mean squared error over valid ground-truth pixels.
- Network Architecture: The image guidance branch uses repetitive hourglass networks to generate hierarchical and clear semantics.RHN1 uses a residual encoder-decoder with low- and high-level feature connections, while later lightweight units extract clearer guidance semantics.
- Network Architecture: The depth generation branch applies repetitive dynamic-convolution guidance to produce structure-detailed depth features at multiple stages.Its repetitive guidance module is described in Fig. 3 and includes the efficient guidance algorithm and adaptive fusion mechanism.
- Loss Function: Mean squared error averages squared differences between ground-truth and predicted depth over valid ground-truth pixels.GT and P denote ground-truth and predicted depth, while Qv contains valid pixels and m is their count.
5 Experiments
Experiments evaluate RigNet on KITTI and NYUv2, using benchmark comparisons, qualitative results, ablations, and robustness tests. RigNet achieves strong benchmark performance, improves detail recovery, and benefits from its repetitive hourglass, repetitive guidance, and adaptive fusion components.
- Evaluation on KITTI Dataset: RigNet ranks first among publicly published KITTI results, improving RMSE by 17.42mm over the second-ranked method.Its depth maps also show more detailed structures and more accurate object boundaries than competing methods.
- Evaluation on NYUv2 Dataset: RigNet achieves the best NYUv2 performance among traditional and recent approaches using 500 sparse samples without additional datasets.Qualitative results show improved recovery of detailed structures.
- Ablation Studies: Increasing backbone depth alone barely changes errors, whereas RHN-10-3 and RHN-18-3 improve over corresponding deeper baselines by 24mm and 10mm.The RHN-18-3 features also contain clearer and richer contexts than the Deeper-18-1 baseline.
- Ablation Studies: Parallel RHN outperforms tandem and deeper-more hourglass designs while using fewer parameters or achieving a 10mm lower error than the baseline.These comparisons support the effectiveness of the lightweight repetitive hourglass design.
- Ablation Studies: Repetitive guidance lowers errors by 6.3mm and 11.2mm as recursion increases, while adaptive fusion further improves the result by 5.3mm over direct output.The repetitive depth features model high-frequency components, and adaptive fusion aggregates multi-step features with about 0-0.06GB additional GPU memory.
- Generalization Capabilities: RigNet outperforms competing methods across sparsity levels and complex lighting and weather conditions, indicating robust generalization.Replacing repetitive guidance with addition produces substantially worse RMSE under varying conditions.
6 Conclusion
RigNet addresses blurry image guidance and unclear depth structure through repetition in both its image guidance and depth generation branches. Its repetitive hourglass, repetitive guidance, efficient guidance, and adaptive fusion designs support structure-detailed depth prediction, with experiments showing outstanding performance.
- Conclusion: RigNet uses repetitive design in both image guidance and depth generation to address blurry guidance and unclear depth structure.The image branch uses a repetitive hourglass network, while the depth branch uses repetitive guidance for gradual prediction.
- Conclusion: The efficient guidance algorithm models high-frequency components while reducing GPU memory consumption.Adaptive fusion combines multi-stage depth features for better predictions.
- Conclusion: Extensive experiments show that RigNet achieves outstanding performance.