Source-linked AI summary
DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimation
Zhenyu Li, Zehui Chen, Xianming Liu, Junjun Jiang
TL;DR
Monocular depth estimation is difficult because RGB images provide incomplete and ambiguous depth cues. DepthFormer combines Transformer and CNN branches with HAHI and deformable attention, and it reports significant improvements across major depth datasets, including the most competitive KITTI result.
Problem
Monocular depth estimation from RGB images is ill-posed because missing cues, scale ambiguities, and translucent or reflective materials make spatial structure ambiguous.
Method
DepthFormer combines Transformer and convolution branches with HAHI feature interaction and deformable attention to model long-range correlation, preserve local information, and reduce attention memory cost.
Results
DepthFormer significantly outperforms state-of-the-art methods on KITTI, NYU, and SUN RGB-D, achieving the most competitive result on the KITTI benchmark.
Takeaways & Limitations
Combining Transformer-based global modeling with CNN-based local information and hierarchical feature interaction supports accurate monocular depth estimation across the evaluated datasets.
Takeaways & Limitations
Dense global attention may be costly, and the input-agnostic HAHI module may benefit from additional modalities such as sparse LiDAR; theoretical guarantees remain limited.
Abstract
from arXiv · showhide
This paper aims to address the problem of supervised monocular depth estimation. We start with a meticulous pilot study to demonstrate that the long-range correlation is essential for accurate depth estimation. Therefore, we propose to leverage the Transformer to model this global context with an effective attention mechanism. We also adopt an additional convolution branch to preserve the local information as the Transformer lacks the spatial inductive bias in modeling such contents. However, independent branches lead to a shortage of connections between features. To bridge this gap, we design a hierarchical aggregation and heterogeneous interaction module to enhance the Transformer features via element-wise interaction and model the affinity between the Transformer and the CNN features in a set-to-set translation manner. Due to the unbearable memory cost caused by global attention on high-resolution feature maps, we introduce the deformable scheme to reduce the complexity. Extensive experiments on the KITTI, NYU, and SUN RGB-D datasets demonstrate that our proposed model, termed DepthFormer, surpasses state-of-the-art monocular depth estimation methods with prominent margins. Notably, it achieves the most competitive result on the highly competitive KITTI depth estimation benchmark. Our codes and models are available at https://github.com/zhyever/Monocular-Depth-Estimation-Toolbox.
1 Introduction
DepthFormer targets monocular depth estimation by combining Transformer-based long-range modeling with CNN-based local information. Its HAHI module connects the two branches and improves feature aggregation before decoding.
- Monocular depth estimation requires exploiting both long-range correlations among objects and local consistency within objects.
- CNNs struggle with global context, while ViT models long-range correlation but lacks spatial inductive bias for local information.
- DepthFormer combines Transformer and convolution branches to model long-range correlation and preserve local information.
- HAHI enhances hierarchical Transformer features through element-wise interaction and models affinity between Transformer and CNN features using set-to-set translation.
- The deformable attention scheme limits high-resolution attention to learnable sampling vectors, reducing the memory cost of global attention.
- DepthFormer significantly outperforms state-of-the-art methods on KITTI, NYU, and SUN RGB-D, with the most competitive result on KITTI.
2 Related Work
Monocular depth estimation is an ill-posed problem addressed through encoder-decoder architectures and increasingly through Transformer-based global modeling. Prior work modifies CNNs, combines CNNs with attention, or integrates ViT at different stages to balance global and local information.
- Estimating depth from RGB images is ill-posed because missing cues, scale ambiguities, and translucent or reflective materials create ambiguous spatial interpretations.
- DPT, Adabins, and Transdepth are identified as important competitors using ViT encoders, adaptive bins, or different Transformer placement and fusion strategies.
- CNN-based approaches improve limited receptive fields through multi-scale fusion, atrous convolutions, feature pyramids, and attention mechanisms.
- Encoder-decoder models use encoders for representations and decoders with convolutions and upsampling to recover resolution and estimate depth.
- HAHI addresses limitations of late fusion by enhancing multi-level hierarchical features and modeling affinity between heterogeneous branch features.
- DepthFormer leverages a Transformer encoder to model long-range correlations while retaining convolutional processing for local information.
3 Methodology
DepthFormer combines Transformer and convolution branches to capture long-range correlation and local information, then connects them through HAHI for stronger feature aggregation. Its deformable attention design limits high-resolution attention to learnable sampling points while supporting sharper depth predictions.
- Motivation: The pilot study motivates combining global and local modeling: CNNs struggle with long-range relationships, while ViT performs better on distant but worse on near-object depth.Near-object errors are especially consequential because near objects are more numerous under the long-tail depth distribution.
- Encoder: DepthFormer uses a Transformer branch for long-range correlation and a lightweight convolution branch for local information.The convolution branch uses only the first ResNet block to preserve low-level features and reduce computation.
- Transformer branch: Swin Transformer supplies hierarchical representations and reduces computational complexity through local attention, shifted windows, and patch merging.Transformer features are sampled and reassembled from selected layers into multi-level feature maps.
- HAHI module: HAHI enhances all hierarchical Transformer features and models affinity between Transformer and CNN features to address insufficient aggregation from independent branches.The module first projects hierarchical features to a common channel dimension, then unfolds and concatenates them into pixel-level feature vectors.
- Effect: DepthFormer preserves local information and enables sharper depth maps without artifacts compared with using Transformer features alone.The reported qualitative comparison specifically links the additional convolution branch with sharper predictions and artifact avoidance.
4 Experiment Results
DepthFormer is evaluated on KITTI, NYU-Depth-v2, and SUN RGB-D using standard depth-estimation protocols and comparisons with leading methods. Results show strong accuracy, speed, cross-dataset generalization, and improvements from the proposed components.
- Comparison to State-of-the-Art: DepthFormer outperforms all competitors across NYU-Depth-v2 metrics and produces sharper depth maps with accurate object boundaries.The predicted depths also support satisfying 3D scene reconstructions with reasonable depth estimates.
- Comparison to State-of-the-Art: 16% improvement in sqErrorRel accompanies DepthFormer's most competitive result on the KITTI depth-estimation benchmark.The benchmark test predictions are submitted online, and the comparison is made against leading methods.
- Comparison to State-of-the-Art: DepthFormer shows strong cross-dataset generalization when trained on NYU-Depth-v2 and evaluated on SUN RGB-D without fine-tuning.It improves all reported metrics and predicts accurate depths in extremely dark areas.
- Ablation Studies: The convolution branch and HAHI significantly improve overall performance, while larger-scale pre-training benefits the Transformer encoder more than the CNN encoder.The ablation compares reinforced ResNet-50 and Swin-T baselines and evaluates ImageNet-22K pre-training.
- Ablation Studies: Using one convolutional block gives the best performance, preserving local information while reducing complexity and cutting training time by 2.5× or more.Adding more blocks degrades performance, potentially because consecutive convolutions wash out low-level features and reduce spatial resolution.
- Ablation Studies: Multi-level DSA improves RMS by 4.9%, and adding DCA with multi-level DSA yields a further 2.2% RMS improvement.The results support the importance of both multi-level aggregation and deformable cross-attention for heterogeneous interaction.
5 Conclusion
DepthFormer combines Transformer and CNN branches to exploit long-range correlations and local information, using HAHI to improve feature interaction. The authors report significant improvements on challenging depth-estimation datasets while identifying several directions for future investigation.
- DepthFormer combines Transformer and CNN branches to model long-range correlations and preserve local information for monocular depth estimation.
- HAHI enhances multi-level features and models feature affinity to address insufficient aggregation caused by independent branches with late fusion.
- The method achieves significant improvements over state-of-the-art methods on popular and challenging datasets.
- Future work includes more dedicated attention mechanisms, multimodal inputs such as sparse LiDAR, and improved explainability and transparency.