Source-linked AI summary
CompletionFormer: Depth Completion with Convolutions and Vision Transformers
Zhang Youmin, Guo Xianda, Poggi Matteo, Zhu Zheng, Huang Guan, Mattoccia Stefano
TL;DR
Depth completion must propagate sparse, noisy measurements while capturing both local and global pixel relationships. CompletionFormer couples convolutional attention with Transformers in a pyramidal single-branch network, achieving state-of-the-art results with lower computational overhead than pure Transformer methods.
Problem
Depth completion must reconstruct whole depth maps from sparse RGBD measurements while detecting outliers and modeling local and global pixel correlations.
Method
CompletionFormer integrates convolutional attention and Vision Transformers through JCAT blocks in a pyramidal single-branch architecture for multimodal feature interaction.
Results
CompletionFormer delivers state-of-the-art depth-completion results on indoor and outdoor datasets, with 429.6G FLOPs versus 1259.3G FLOPs for MPViT-Base.
Takeaways & Limitations
The hybrid design combines local convolutional receptive fields with global Transformer representations while retaining computation comparable to CNN-based methods.
Takeaways & Limitations
The experiments manually generate sparse data and do not add extra noise, while the current model runs at about 10 FPS.
Abstract
from arXiv · showhide
Given sparse depths and the corresponding RGB images, depth completion aims at spatially propagating the sparse measurements throughout the whole image to get a dense depth prediction. Despite the tremendous progress of deep-learning-based depth completion methods, the locality of the convolutional layer or graph model makes it hard for the network to model the long-range relationship between pixels. While recent fully Transformer-based architecture has reported encouraging results with the global receptive field, the performance and efficiency gaps to the well-developed CNN models still exist because of its deteriorative local feature details. This paper proposes a Joint Convolutional Attention and Transformer block (JCAT), which deeply couples the convolutional attention layer and Vision Transformer into one block, as the basic unit to construct our depth completion model in a pyramidal structure. This hybrid architecture naturally benefits both the local connectivity of convolutions and the global context of the Transformer in one single model. As a result, our CompletionFormer outperforms state-of-the-art CNNs-based methods on the outdoor KITTI Depth Completion benchmark and indoor NYUv2 dataset, achieving significantly higher efficiency (nearly 1/3 FLOPs) compared to pure Transformer-based methods. Code is available at \url{https://github.com/youmi-zym/CompletionFormer}.
1. Introduction
Depth completion reconstructs dense depth from sparse measurements and RGB images, but reliable propagation requires both local and global pixel relationships. CompletionFormer addresses this gap by coupling convolutional attention with Transformer representations in a pyramidal network.
- Depth completion reconstructs whole depth maps from sparse depth measurements and corresponding RGB images.
- Reliable completion requires detecting outliers and propagating valid depths across both nearby and extremely distant pixels.
- CompletionFormer couples CNN-based local features with Transformer-based global representations in a pyramidal architecture.
- The JCAT block integrates Vision Transformer and convolutional attention layers to provide local and global receptive fields for multimodal interaction.Spatial and channel attention increase the capacity of the convolutional layers.
- CompletionFormer uses a single-branch structure with computation comparable to CNN methods and higher efficiency than pure Transformer methods.
- CompletionFormer yields substantial improvements over state-of-the-art depth-completion methods, especially with very sparse provided depth.
2. Related Work
Depth-completion research has progressed from convolutional and graph-based propagation toward Transformer-based modeling, but combining their complementary properties remains important. CompletionFormer strengthens a U-Net backbone with joint local-global context for multimodal depth completion.
- Depth Completion: U-Net encoder-decoder architectures and spatial propagation operations are established approaches for predicting and refining dense depth.
- Vision Transformer: The architecture uses a U-Net backbone with JCAT blocks for multiscale depth-image interaction, full-resolution fusion, initial prediction, and SPN refinement.
- Vision Transformer: CompletionFormer strengthens U-Net with coherent local and global context information for depth completion.
- Depth Completion: Multi-branch networks use concatenation, summation, or attention-based strategies to fuse RGB and depth information.
- Vision Transformer: Fully Transformer-based depth completion can provide global modeling but has shown worse results and higher computational cost than pure CNN methods.
- Vision Transformer: The proposed joint convolutional-attention and Transformer block achieves higher efficiency and performance on public benchmarks than the compared Vision Transformer approach.
3. Method
CompletionFormer combines early RGB-depth embedding, convolutional attention, and Transformer layers in a pyramidal encoder-decoder to capture local detail and global context for depth completion. JCAT blocks and spatial propagation support multiscale feature extraction, multimodal fusion, and final depth refinement.
- Overall architecture: CompletionFormer uses early RGB-depth embedding so a single following branch can fuse multimodal information efficiently.Pixels with invalid depth can be corrected using appearance similarity to reliable measurements.
- Encoder: The encoder uses five stages, with ResNet34 BasicBlocks initially and repeated JCAT blocks in the subsequent four stages.Patch embedding produces feature resolutions of 1/4, 1/8, 1/16, and 1/32 relative to the input image.
- JCAT block: Each JCAT block combines parallel or cascaded convolutional attention and Transformer streams to provide local and global receptive fields.The convolutional path uses spatial and channel attention, while the Transformer path uses spatial-reduction attention and a feed-forward layer.
- JCAT block: Transformer self-attention matches every token with tokens across the input feature space, extending the receptive field to the full image.Spatially reducing keys and values decreases memory consumption, while embedded RGB-depth tokens support similarity comparison across pixels.
- Decoder: The decoder concatenates multiscale encoder outputs through skip connections, upsamples features, and applies convolutional attention for feature fusion before initial depth prediction.The prediction head also receives raw RGB and depth embedding features.
- SPN refinement: Spatial propagation refines the predicted depth map using affinity weights and confidence modulation, while the enhanced U-Net backbone achieves nearly the same accuracy with fixed-local or non-local neighbors.The affinity matrix is modulated by a predicted confidence map to prevent less confident pixels from propagating into neighbors.
4. Experiments
Experiments evaluate CompletionFormer across indoor and outdoor datasets, sparsity levels, architectural variants, and computational costs. Ablations support parallel multimodal fusion, convolutional attention, the single-branch encoder, and local-global modeling.
- Ablation Studies: Parallel coupling outperforms cascaded coupling, with CMT-Base reaching RMSE 92.0 versus MPViT-Base at 91.0.The authors therefore adopt parallel connections for interaction between streams with different contents and semantics.
- Ablation Studies: Removing spatial and channel attention increases RMSE from 90.0 to 91.1 while adding only 0.2G FLOPs.This ablation supports increasing convolutional capacity within the JCAT block.
- Ablation Studies: The single-branch encoder outperforms a dual-branch design, indicating that early multimodal embedding is more effective and efficient.The dual-branch alternative separately encodes RGB and depth before communicating between branches.
- Evaluation Settings: CompletionFormer is evaluated on NYUv2 and KITTI DC using standard depth-completion metrics and varied sparse-depth settings.NYUv2 uses randomly sampled depth points, while KITTI experiments simulate LiDAR patterns with different scanning-line densities.
- Benchmark Results: CompletionFormer consistently outperforms competing methods on NYUv2 across 0, 50, 200, and 500 sampled depth points.On KITTI DC, it ranks first in RMSE when jointly minimizing L1 and L2 losses and performs better near missing-depth regions.
5. Conclusion and Limitations
CompletionFormer integrates convolutional attention and Transformers in a single-branch depth-completion network. It achieves state-of-the-art results on indoor and outdoor datasets, but its approximately 10 FPS runtime remains above real-time needs.
- Conclusion: CompletionFormer integrates convolutional attention and Transformers into one block within a single-branch depth-completion network.The design is evaluated for sparse-input depth completion on indoor and outdoor datasets.
- Conclusion: The model achieves state-of-the-art results on indoor and outdoor datasets, supported by extensive ablation studies of accuracy and efficiency.
- Limitation: CompletionFormer runs at about 10 FPS, and reducing runtime further to meet real-time requirements is identified as future work.
A.1. Qualitative Results on NYUv2 Dataset
NYUv2 qualitative comparisons show clearer predictions from CompletionFormer than NLSPN and the Transformer-only variant. The advantage is especially visible around transparent regions near windows.
- Qualitative Comparison: CompletionFormer produces improved qualitative results compared with NLSPN in both visualized NYUv2 cases.
- Qualitative Comparison: Around transparent window regions, CompletionFormer predicts clear object boundaries while NLSPN and Ours-ViT produce blurry estimates.Ours-ViT denotes the configuration using only the Transformer layer.
- Figure 6: Figure 6 presents RGB images, dense predictions, and error maps whose colder colors indicate lower errors.
A.2. Model Architecture Details
The paper provides implementation-level architecture details for CompletionFormer, including parameter specifications and prediction-head output dimensions.
- Architecture Specification: The network parameters of CompletionFormer are reported to support understanding of the architecture and reproducibility.
- Prediction Heads: Each prediction head uses nearly the same design, while output channel η depends on the prediction type, with η = 1 for initial depth prediction.The architecture uses concatenation along the channel dimension where specified.