PaperlayerView exact revision on arXiv

Source-linked AI summary

Attention Attention Everywhere: Monocular Depth Prediction with Skip Attention

Ashutosh Agarwal, Chetan Arora

arXiv:2210.09071v1cs.CV

TL;DR

MDE must combine global context with pixel-level resolution, yet conventional encoder-decoder skip fusion can produce incorrect depth labels. PixelFormer refines pixel queries with skip attention and predicts adaptive depth bins from coarse features, reporting state-of-the-art improvements on NYUV2 and KITTI and better SUNRGBD generalization.

  • Problem

    MDE must predict pixel-wise depth from one RGB image while combining global context with pixel-level resolution, and conventional skip fusion can leave depth labels incorrect.

  • Method

    PixelFormer treats MDE as pixel query refinement, using SAM to cross-attend decoder queries to higher-resolution encoder features and BCP to predict adaptive bin centers from initial queries.

  • Results

    PixelFormer achieves state-of-the-art performance, improving NYUV2 absolute relative error by 5.3%, KITTI Sq. Rel by 3.9%, and SUNRGBD absolute relative error by 9.4%.

  • Takeaways & Limitations

    Attention-based skip fusion and supervised coarse-level bin prediction provide the paper's proposed route to more accurate depth labels and improved cross-dataset generalization.

Abstract

from arXiv · show

Monocular Depth Estimation (MDE) aims to predict pixel-wise depth given a single RGB image. For both, the convolutional as well as the recent attention-based models, encoder-decoder-based architectures have been found to be useful due to the simultaneous requirement of global context and pixel-level resolution. Typically, a skip connection module is used to fuse the encoder and decoder features, which comprises of feature map concatenation followed by a convolution operation. Inspired by the demonstrated benefits of attention in a multitude of computer vision problems, we propose an attention-based fusion of encoder and decoder features. We pose MDE as a pixel query refinement problem, where coarsest-level encoder features are used to initialize pixel-level queries, which are then refined to higher resolutions by the proposed Skip Attention Module (SAM). We formulate the prediction problem as ordinal regression over the bin centers that discretize the continuous depth range and introduce a Bin Center Predictor (BCP) module that predicts bins at the coarsest level using pixel queries. Apart from the benefit of image adaptive depth binning, the proposed design helps learn improved depth embedding in initial pixel queries via direct supervision from the ground truth. Extensive experiments on the two canonical datasets, NYUV2 and KITTI, show that our architecture outperforms the state-of-the-art by 5.3% and 3.9%, respectively, along with an improved generalization performance by 9.4% on the SUNRGBD dataset. Code is available at https://github.com/ashutosh1807/PixelFormer.git.

1. Introduction

MDE requires both global context and pixel-level resolution, but conventional skip fusion can leave depth labels confused despite accurate boundary alignment. PixelFormer addresses this with attention-based pixel-query refinement and adaptive bin prediction, achieving reported gains across datasets.

  • Motivation: Encoder-decoder MDE architectures combine high-resolution encoder details with semantically rich decoder context, while convolutional locality limits long-range fusion.Self-attention can enlarge receptive fields, but the paper identifies feature fusion as a remaining source of depth-label confusion.
  • Proposed approach: PixelFormer uses a Skip Attention Module to cross-attend decoder-based pixel queries to neighboring higher-resolution encoder features within windows.The module replaces concatenation followed by convolution with window-based cross-attention and query refinement across scales.
  • Proposed approach: PixelFormer initializes pixel queries from the coarsest encoder features, then refines them to finer resolutions through SAM modules.This design treats monocular depth estimation as a pixel query refinement problem.
  • Proposed approach: Adaptive bin centers are predicted from initial pixel queries, embedding depth information through direct ground-truth supervision.The BCP module predicts image-adaptive bins at the coarsest level instead of decoding features and attending again.
  • Results: 5.3% improvement over SOTA is reported on NYUV2, while SUNRGBD generalization improves by 9.4%.The cited NYUV2 figure concerns absolute relative error, and the SUNRGBD figure concerns absolute relative error.

2. Related Works

Prior MDE work spans CNN, transformer, classification-regression, and skip-connection designs. PixelFormer builds on these lines by using global contextual queries to retrieve high-resolution encoder details through skip attention.

  • CNN based MDE Techniques: CNN-based MDE methods integrate global and local information using pyramids, multiscale fusion, geometric constraints, and edge-based regularization.The related work cites Eigen et al., Laplacian pyramids, CLIFFNet, virtual normals, and geometric-edge similarity.
  • Transformer based MDE Models: Transformer-based MDE models use vision transformers or Swin Transformers to aggregate global information and predict adaptive bins or multiscale depths.Examples include CNN-ViT hybrids, adaptive-bin prediction, and Swin Transformer models with multiscale CRFs.
  • MDE as Classification Vs Regression Task: Classification-regression formulations predict depth as a linear combination of discretized bin centers, with recent methods adapting bins per image.PixelFormer predicts bin centers from initial pixel queries rather than decoding features to high resolution before transformer processing.
  • Skip Connections: Skip connections transfer high-resolution encoder information to decoders, but naive fusion is hindered by the semantic gap between encoder and decoder features.Prior approaches use residual blocks or attention to address this gap.
  • Skip Connections: PixelFormer uses global contextual decoder queries to retrieve high-resolution details from encoder features through skip attention.This is the paper's connection between transformer-style context aggregation and skip-connection fusion.

3. Proposed Methodology

PixelFormer formulates monocular depth estimation as pixel-query refinement, using global coarsest-level features to initialize queries and attention-based fusion to refine them across resolutions. Adaptive bin prediction complements this design by producing per-image depth discretization from the initial queries.

  • Problem Formulation: MDE is modeled as classification-regression, with adaptive bin widths discretizing depth and pixel probabilities weighting the predicted bin centers.The depth at each pixel is computed as a linear combination of per-image bin centers and pixel-wise probability scores.
  • Pixel Query Initialisation: The Pixel Query Initialiser aggregates multiscale global information from coarsest-resolution features to create initial pixel queries.Pyramid spatial pooling uses scales 1, 2, 3, and 6 before upsampling, concatenation, and convolution.
  • Bin Center Prediction: The Bin Center Predictor uses global average pooling and an MLP to estimate 256 adaptive bin widths per image from the initial pixel queries.Direct ground-truth supervision is intended to embed depth information into the initial queries, while avoiding high-resolution ViT decoding.
  • Skip Attention Module: The Skip Attention Module replaces convolution-after-concatenation fusion with window-based cross-attention between decoder pixel queries and higher-resolution encoder features.The module attends within 7 × 7 windows, using query, key, and value projections plus relative position bias.
  • Decoder and Prediction: SAM hierarchically refines pixel queries through multiscale encoder features, then predicts per-pixel bin probabilities used to form final depth estimates.Attention and MLP residual updates produce refined queries, which are converted into depth embeddings and probability distributions over bin centers.
  • Training: Training uses a scaled Scale-Invariant loss based on logarithmic differences between predicted and ground-truth depth.The experiments use λ = 0.85 and α = 10, with n denoting pixels having available ground-truth values.

4. Datasets and Evaluation

The evaluation covers indoor, outdoor, and cross-dataset indoor depth estimation using established dataset splits, depth bounds, crops, and standard error and threshold metrics. NYUV2 and KITTI provide in-distribution benchmarks, while SUNRGB-D measures generalization without fine-tuning.

  • NYUV2: NYUV2 contains 120K RGB-depth pairs from 464 indoor scenes, with official training/testing splits and a 10-meter depth upper bound.The method is evaluated at 120 × 160 output resolution, upsampled by 4× to ground-truth resolution.
  • KITTI: KITTI contains stereo images and 3D scans from 61 outdoor scenes, using 26K training images, 697 test images, and an 80-meter depth upper bound.Evaluation uses the crop defined by Garg et al. and bilinear upsampling to ground-truth resolution.
  • SUNRGB-D: SUNRGB-D contains 10,335 indoor RGB-D room images, split into 5,285 training and 5,050 testing images, with an 8-meter depth upper bound.The official test set is used for evaluation.
  • Metrics: Evaluation uses Abs Rel, RMSE, log10, and threshold accuracy at 1.25, 1.25^2, and 1.25^3; KITTI additionally uses Sq Rel.These metrics support comparisons with prior monocular depth estimation work.

5. Experiments

PixelFormer is evaluated on NYUV2, KITTI, and SUNRGB-D using quantitative and qualitative comparisons against prior methods. It improves reported depth-estimation metrics and generalizes to SUNRGB-D without fine-tuning.

  • Implementation: Experiments use Adam optimization for 20 epochs on KITTI and NYUV2, with Swin-L pre-trained weights and data augmentation.Training uses four NVIDIA A100 GPUs and includes rotation, flipping, brightness changes, and Cut Depth augmentation.
  • NYUV2: 5.3% improvement in Absolute relative error over SOTA is reported on NYUV2 without additional training data.PixelFormer also improves RMSE by 9.6% over [23] and 3.5% over NeWCRFs.
  • NYUV2: PixelFormer produces more accurate NYUV2 depth maps than Adabins and NeWCRFs in qualitative comparisons.The authors attribute this to SAM capturing long-range dependencies and report depth estimates for missing objects.
  • KITTI: 3.9% improvement in Sq. Rel and 2.3% improvement in RMSE over NeWCRFs are reported on the KITTI Eigen Split.On the official KITTI benchmark, PixelFormer ranks first among previous peer-reviewed approaches, with further gains in Abs Rel and SILog against NeWCRFs.
  • SUNRGB-D: 9.4% improvement in Abs Rel and 7.4% improvement in RMSE over Adabins are reported on SUNRGB-D without fine-tuning.The evaluation measures generalization from models trained on NYUV2 to the SUNRGB-D test set.

6. Ablation Study

The ablation study evaluates the Skip Attention Module and depth information embedded in initial pixel queries on KITTI Eigen Split. SAM outperforms convolution-based fusion alternatives, while the pixel-query design uses already aggregated global information.

  • Efficacy of Skip Attention Module: The SAM ablation uses KITTI Eigen Split with Swin-T as the encoder, reporting higher-is-better and lower-is-better metrics.The table marks the best results in bold and the second-best results with underlining.
  • Efficacy of Skip Attention Module: SAM outperforms Add-Conv by 4.0% in Abs Rel and 4.2% in Sq. Rel on KITTI Eigen Split.Add-Conv uses pixel-wise addition followed by convolution, while Cat-Conv concatenates features before convolution.
  • Efficacy of Skip Attention Module: The addition-based Add-Conv approach outperforms concatenation-based Cat-Conv by a small margin.
  • Effectiveness of embedding the depth information into the pixel queries: Predicting bin centers from initial pixel queries provides no further benefit from adding mViT-First, because global information is already aggregated through the PQI module.

7. Conclusion

PixelFormer poses monocular depth estimation as pixel query refinement, refining global initial queries through Skip Attention and embedding depth information through Bin Center Prediction. The method improves state-of-the-art performance on NYUV2, KITTI, and SUNRGBD generalization.

  • Conclusion: PixelFormer poses monocular depth estimation as a pixel query refinement problem.Global initial pixel queries are refined to higher resolutions by querying multiscale encoder features through the Skip Attention Module.
  • Conclusion: Skip Attention fuses decoder features with long-range dependency, unlike convolution-based skip connections, leading to more accurate depth labels.
  • Conclusion: 5.3% and 3.9% improvements over state-of-the-art are reported on NYUV2 and KITTI, respectively.
  • Conclusion: SUNRGBD generalization performance improves by 9.4%.
  • Conclusion: The authors plan to apply SAM's content-adaptive fusion to other dense estimation tasks such as semantic segmentation.
Loading 2210.09071v1…