Source-linked AI summary
Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction
Guanglei Yang, Hao Tang, Mingli Ding, Nicu Sebe, Elisa Ricci
TL;DR
CNNs have limited explicit long-range modeling, while pure Transformers lack the convolutional spatial inductive bias needed for continuous pixel-wise prediction. TransDepth combines a Transformer with ResNet and an attention-gated decoder, achieving state-of-the-art results across monocular depth and surface-normal benchmarks.
Problem
CNNs have limited explicit long-range dependency modeling, while pure Transformers are unsuitable for continuous pixel-wise prediction because they lack convolutional spatial inductive bias and resolution reduction.
Method
TransDepth combines a linear Transformer with a ResNet backbone and uses a unified attention-gated decoder to fuse multi-scale features and preserve local representations.
Results
TransDepth outperforms previous methods on KITTI and NYU depth, including 0.956 and 0.900 on δ<1.25, and achieves state-of-the-art NYU surface-normal results.
Takeaways & Limitations
Transformers can be applied to continuous pixel-wise depth and surface-normal prediction when combined with convolutional spatial inductive bias and gated multi-scale decoding.
Abstract
from arXiv · showhide
While convolutional neural networks have shown a tremendous impact on various computer vision tasks, they generally demonstrate limitations in explicitly modeling long-range dependencies due to the intrinsic locality of the convolution operation. Initially designed for natural language processing tasks, Transformers have emerged as alternative architectures with innate global self-attention mechanisms to capture long-range dependencies. In this paper, we propose TransDepth, an architecture that benefits from both convolutional neural networks and transformers. To avoid the network losing its ability to capture local-level details due to the adoption of transformers, we propose a novel decoder that employs attention mechanisms based on gates. Notably, this is the first paper that applies transformers to pixel-wise prediction problems involving continuous labels (i.e., monocular depth prediction and surface normal estimation). Extensive experiments demonstrate that the proposed TransDepth achieves state-of-the-art performance on three challenging datasets. Our code is available at: https://github.com/ygjwd12345/TransDepth.
1. Introduction
The paper addresses limited global-context modeling and local-detail loss in CNN-based pixel-wise prediction by combining Transformers with convolutional representations. TransDepth uses a Transformer-enhanced ResNet and attention-gated decoding, achieving state-of-the-art results across continuous prediction benchmarks.
- CNN encoders struggle to model global contexts explicitly because convolution is intrinsically local.
- Progressive down-sampling enlarges receptive fields but discards local information crucial for dense prediction.
- Prior pure Transformer segmentation approaches do not resolve the representation-learning problem for continuous pixel-wise prediction.
- TransDepth embeds Transformers into a ResNet backbone to model semantic pixel dependencies while using an attention-gated decoder to preserve local representations.
- 0.956 on KITTI and 0.900 on NYU depth δ<1.25 demonstrate performance exceeding previous methods, alongside new state-of-the-art NYU surface-normal results.
2. Related Work
Related work spans efficient and pure Transformer vision models, continuous pixel prediction, surface-normal estimation, and attention-based dense prediction. TransDepth combines a linear Transformer with ResNet and introduces spatial- and channel-level attention gates for continuous pixel-wise tasks.
- Transformers in Computer Vision: Vision Transformer research includes local or axial self-attention designs that reduce the computation of global attention.
- Transformers in Computer Vision: Pure patch-sequence Transformers are unsuitable for continuous pixel prediction because these tasks rely on deep representations and fully convolutional decoders.
- Transformer-Based Pixel Prediction: The proposed hybrid combines a linear Transformer with ResNet while retaining compatibility with dilated convolutions and attention modules.
- Surface Normal Estimation: Surface-normal estimation requires both global and local features, with prior approaches using high-capacity networks, specialized losses, or extra training phases.
- Attention Models: TransDepth adds spatial- and channel-level attention to an attention decoder and reports superiority over PGA-Net on challenging dense continuous pixel-level tasks.
3. The Proposed TransDepth
TransDepth combines a ResNet backbone with Transformer layers and an attention-gated decoder for continuous pixel-wise prediction. The design models semantic dependencies while using multi-scale fusion and attention mechanisms to preserve local representations.
- 3.1. Transformer for Depth Prediction: TransDepth adds Transformer layers to a ResNet backbone to address limited receptive fields and model semantic pixel dependencies.The hybrid structure retains convolutional representation learning while incorporating Transformer-based context modeling.
- 3.1. Transformer for Depth Prediction: The Transformer receives flattened patches from the CNN’s final feature map rather than directly reshaping the input image into an image-patch sequence.A p×p patch-embedding kernel flattens spatial dimensions of the CNN feature map and projects them into the Transformer dimension.
- 3.1. Transformer for Depth Prediction: Each Transformer layer uses multi-headed self-attention with query, key, and value projections, followed by a residual MLP block.The attention heads are computed independently using learnable weight matrices, and the output is transformed by an MLP with a residual skip.
- 3.2. Attention Gate Decoder: The attention-gated decoder learns to combine multi-scale feature maps using latent kernels instead of simple concatenation or weighted averaging.The deepest feature map serves as the receiving feature, while preceding feature maps serve as emitting features.
- 3.2. Attention Gate Decoder: The attention gate separates attention and message components, infers spatial and channel attention variables, and applies a local conditional kernel before generating attention.The message component is calculated by convolving the receiving feature with a learned kernel.
- 3.3. Optimization: For depth and surface-normal estimation, the network uses a depth optimization loss based on logarithmic depth differences and an angular loss for surface normals.The depth-loss parameters are set to λ = 0.85 and α = 10.
4. Experiments
Experiments evaluate TransDepth on KITTI, NYU depth, and surface normal datasets using standard protocols and compare its performance with prior methods. Results show strong benchmark performance, with attention-gated decoding and mixed ResNet–Transformer backbones supporting dense prediction.
- Datasets and protocols: Experiments use KITTI for outdoor depth estimation, NYU for indoor depth and surface normal prediction, and ScanNet for surface normal evaluation.KITTI uses 22,600 training and 697 testing frames; NYU depth uses 120K RGB-Depth pairs from 464 scenes; ScanNet has 189,916 training and 20,942 testing samples.
- Datasets and protocols: The evaluation follows standard monocular-depth metrics and reports median angle and 11.25° accuracy for surface normal estimation.Depth metrics include abs-rel, sq-rel, rms, log-rms, and threshold accuracy; the main paper reports two selected surface-normal metrics because of space limitations.
- Benchmark results: TransDepth performs favorably against prior KITTI depth methods, achieving the best results on most metrics in the supervised single-image setting.Compared with FAL-Net, BTS, and PGA-Net, the method is reported to improve by a large margin and operates end-to-end without extra post-processing.
- Benchmark results: 0.900 δ<1.25 and 0.365 root mean squared error are reported on NYU depth, improving over Huynh et al.’s 0.882 δ<1.25.The method also outperforms state-of-the-art and attention-based approaches in the indoor evaluation.
- Benchmark results: TransDepth outperforms state-of-the-art methods on the reported surface-normal metrics, while joint AGD and ViT use produces more precise object boundaries qualitatively.The comparison includes median angle and 11.25° accuracy; the method avoids the extra gravity labels and two-step pre-training required by Do et al.’s method.
- Ablation studies: The mixed ResNet–ViT backbone outperforms pure ResNet and pure Transformer encoders, while ablations show that adding more scale information does not always improve performance.The selected encoder is ResNet-50 with ViT-B/16, and the attention gate decoder is evaluated with different scale-fusion combinations.
5. Conclusions
TransDepth combines ResNet’s spatial-correlation inductive bias with Transformers’ global relationship modeling for continuous pixel-wise prediction. Its unified attention-gate decoder merges low-level information, and experiments report new state-of-the-art results on three datasets.
- TransDepth combines ResNet and Transformers to model spatial correlations and global relationships in continuous pixel-wise prediction.
- The decoder uses independent channel-wise and spatial-wise attention to merge more low-level information into deeper representations.
- 0.956 on δ<1.25 is reported for KITTI depth estimation.
- 0.900 on δ<1.25 is reported for NYU depth estimation.
- 61.7 on 11.25◦ is reported for NYU surface normal estimation.