Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction

• CNN encoders struggle to model global contexts explicitly because convolution is intrinsically local.Source: §1 · p. 1
• Progressive down-sampling enlarges receptive fields but discards local information crucial for dense prediction.Source: §1 · p. 1
• Prior pure Transformer segmentation approaches do not resolve the representation-learning problem for continuous pixel-wise prediction.Source: §1 · pp. 1–2
• TransDepth embeds Transformers into a ResNet backbone to model semantic pixel dependencies while using an attention-gated decoder to preserve local representations.Source: §1 · p. 2
• 0.956 on KITTI and 0.900 on NYU depth δ<1.25 demonstrate performance exceeding previous methods, alongside new state-of-the-art NYU surface-normal results.Source: §1 · p. 2

Transformers in Computer Vision

• Vision Transformer research includes local or axial self-attention designs that reduce the computation of global attention.Source: §2 · p. 2
• Pure patch-sequence Transformers are unsuitable for continuous pixel prediction because these tasks rely on deep representations and fully convolutional decoders.Source: §2 · p. 2

Transformer-Based Pixel Prediction

• The proposed hybrid combines a linear Transformer with ResNet while retaining compatibility with dilated convolutions and attention modules.Source: §2 · pp. 2–3

Surface Normal Estimation

• Surface-normal estimation requires both global and local features, with prior approaches using high-capacity networks, specialized losses, or extra training phases.Source: §2 · p. 3

Attention Models

• TransDepth adds spatial- and channel-level attention to an attention decoder and reports superiority over PGA-Net on challenging dense continuous pixel-level tasks.Source: §2 · p. 3

3.1. Transformer for Depth Prediction

• TransDepth adds Transformer layers to a ResNet backbone to address limited receptive fields and model semantic pixel dependencies.The hybrid structure retains convolutional representation learning while incorporating Transformer-based context modeling.Source: §3 · p. 3, §3.1 · p. 4
• The Transformer receives flattened patches from the CNN’s final feature map rather than directly reshaping the input image into an image-patch sequence.A p×p patch-embedding kernel flattens spatial dimensions of the CNN feature map and projects them into the Transformer dimension.Source: §3.1 · p. 4
• Each Transformer layer uses multi-headed self-attention with query, key, and value projections, followed by a residual MLP block.The attention heads are computed independently using learnable weight matrices, and the output is transformed by an MLP with a residual skip.Source: §3.1 · p. 4

3.2. Attention Gate Decoder

• The attention-gated decoder learns to combine multi-scale feature maps using latent kernels instead of simple concatenation or weighted averaging.The deepest feature map serves as the receiving feature, while preceding feature maps serve as emitting features.Source: §3.2 · p. 4
• The attention gate separates attention and message components, infers spatial and channel attention variables, and applies a local conditional kernel before generating attention.The message component is calculated by convolving the receiving feature with a learned kernel.Source: §3.2 · pp. 4–5

3.3. Optimization

• For depth and surface-normal estimation, the network uses a depth optimization loss based on logarithmic depth differences and an angular loss for surface normals.The depth-loss parameters are set to λ = 0.85 and α = 10.Source: §3.2 · p. 5

Datasets and protocols

• Experiments use KITTI for outdoor depth estimation, NYU for indoor depth and surface normal prediction, and ScanNet for surface normal evaluation.KITTI uses 22,600 training and 697 testing frames; NYU depth uses 120K RGB-Depth pairs from 464 scenes; ScanNet has 189,916 training and 20,942 testing samples.Source: §4.1 · p. 5
• The evaluation follows standard monocular-depth metrics and reports median angle and 11.25° accuracy for surface normal estimation.Depth metrics include abs-rel, sq-rel, rms, log-rms, and threshold accuracy; the main paper reports two selected surface-normal metrics because of space limitations.Source: §4.2 · p. 5

Benchmark results

• TransDepth performs favorably against prior KITTI depth methods, achieving the best results on most metrics in the supervised single-image setting.Compared with FAL-Net, BTS, and PGA-Net, the method is reported to improve by a large margin and operates end-to-end without extra post-processing.Source: §4.4 · p. 6
• 0.900 δ<1.25 and 0.365 root mean squared error are reported on NYU depth, improving over Huynh et al.’s 0.882 δ<1.25.The method also outperforms state-of-the-art and attention-based approaches in the indoor evaluation.Source: §4.4 · p. 7
• TransDepth outperforms state-of-the-art methods on the reported surface-normal metrics, while joint AGD and ViT use produces more precise object boundaries qualitatively.The comparison includes median angle and 11.25° accuracy; the method avoids the extra gravity labels and two-step pre-training required by Do et al.’s method.Source: §4.5 · p. 7

Ablation studies

• The mixed ResNet–ViT backbone outperforms pure ResNet and pure Transformer encoders, while ablations show that adding more scale information does not always improve performance.The selected encoder is ResNet-50 with ViT-B/16, and the attention gate decoder is evaluated with different scale-fusion combinations.Source: §4.6 · p. 8, Table 4 · p. 8, Table 5 · p. 8

• TransDepth combines ResNet and Transformers to model spatial correlations and global relationships in continuous pixel-wise prediction.Source: §5 · p. 8
• The decoder uses independent channel-wise and spatial-wise attention to merge more low-level information into deeper representations.Source: §5 · p. 8
• 0.956 on δ<1.25 is reported for KITTI depth estimation.Source: §5 · p. 8
• 0.900 on δ<1.25 is reported for NYU depth estimation.Source: §5 · p. 8
• 61.7 on 11.25◦ is reported for NYU surface normal estimation.Source: §5 · p. 8