PaperlayerView exact revision on arXiv

Source-linked AI summary

DaViT: Dual Attention Vision Transformers

Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, Lu Yuan

arXiv:2204.03645v1cs.CV

TL;DR

Vision transformers face a tradeoff between global context and computational efficiency, especially for high-resolution inputs. DaViT addresses this with complementary spatial-token and channel-token self-attention, achieving consistent gains across vision benchmarks and tasks.

  • Problem

    Vision transformers can model global context, but their quadratic token-length complexity limits efficient scaling to high-resolution inputs.

  • Method

    DaViT alternates spatial window self-attention over spatial tokens with grouped channel self-attention over image-level channel tokens.

  • Results

    DaViT consistently outperforms state-of-the-art vision transformers across three benchmarks and four tasks with fewer computational costs.

  • Takeaways & Limitations

    The complementary attentions provide both local fine-grained features and global interactions while maintaining efficient computation.

Abstract

from arXiv · show

In this work, we introduce Dual Attention Vision Transformers (DaViT), a simple yet effective vision transformer architecture that is able to capture global context while maintaining computational efficiency. We propose approaching the problem from an orthogonal angle: exploiting self-attention mechanisms with both "spatial tokens" and "channel tokens". With spatial tokens, the spatial dimension defines the token scope, and the channel dimension defines the token feature dimension. With channel tokens, we have the inverse: the channel dimension defines the token scope, and the spatial dimension defines the token feature dimension. We further group tokens along the sequence direction for both spatial and channel tokens to maintain the linear complexity of the entire model. We show that these two self-attentions complement each other: (i) since each channel token contains an abstract representation of the entire image, the channel attention naturally captures global interactions and representations by taking all spatial positions into account when computing attention scores between channels; (ii) the spatial attention refines the local representations by performing fine-grained interactions across spatial locations, which in turn helps the global information modeling in channel attention. Extensive experiments show our DaViT achieves state-of-the-art performance on four different tasks with efficient computations. Without extra data, DaViT-Tiny, DaViT-Small, and DaViT-Base achieve 82.8%, 84.2%, and 84.6% top-1 accuracy on ImageNet-1K with 28.3M, 49.7M, and 87.9M parameters, respectively. When we further scale up DaViT with 1.5B weakly supervised image and text pairs, DaViT-Gaint reaches 90.4% top-1 accuracy on ImageNet-1K. Code is available at https://github.com/dingmyu/davit.

1 Introduction

DaViT addresses the tradeoff between global context and computational efficiency by combining spatial and channel self-attention. Its complementary attentions support local refinement and global interactions, with reported gains across vision tasks at comparable computation.

  • Vision transformers capture long-range dependencies, but quadratic complexity with token length limits scaling to high-resolution inputs.
  • Prior approaches trade off resolution, global context, and computational complexity, motivating efficient image-level self-attention.
  • DaViT introduces channel tokens by transposing the token matrix, making each channel token an abstract representation of the entire image.
  • Channel group attention reduces complexity to linear in both spatial and channel dimensions while retaining global interactions within channel groups.
  • DaViT alternates spatial window and channel group attention so spatial attention refines local features while channel attention models global representations.
  • DaViT consistently outperforms state-of-the-art vision transformers across three benchmarks and four tasks with fewer computational costs.

2 Related Work

DaViT differs from prior vision transformers and MLP-based models by applying transformer self-attention to both spatial tokens and their transposed channel tokens. Its channel group attention efficiently models global context while preserving fine-grained local modeling across multiple vision tasks.

  • Vision Transformers and MLPs: Prior vision transformers model global relationships over pixel-level or patch-level tokens, whereas DaViT also attends to their transpose as image-level channel tokens.
  • Vision Transformers and MLPs: MLP-Mixer repeatedly applies MLPs across spatial locations or feature channels but does not use transformer architectures or self-attention layers.
  • Hierarchical Vision Transformers: DaViT combines hierarchical architecture and fine-grained local attention with group channel attention for efficient global-context modeling.
  • Channel-wise Attentions: Channel group attention captures global information with linear complexity in both channel and spatial dimensions.
  • Channel-wise Attentions: The dual attention mechanism is analyzed for classification, object detection, and segmentation, demonstrating effectiveness across these tasks.

3 Methodology

DaViT alternates spatial window attention with channel group attention to combine local and global representations while keeping complexity linear in spatial size. Channel grouping controls channel-wise cost, and the model uses a four-stage hierarchical layout.

  • Architecture: DaViT stacks spatial window attention and channel group attention within four hierarchical stages, preserving stage resolution and feature dimension between patch embeddings.Each stage begins with patch embedding and contains dual attention blocks.
  • Complexity: Standard global self-attention has complexity O(2P^2C + 4PC^2), making its cost prohibitive when the number of patches P is large.The visual feature has P patches and C channels.
  • Spatial Window Attention: Window attention partitions P patches into Nw non-overlapping windows of Pw patches, reducing complexity to O(2PPwC + 4PC^2) with linear spatial scaling.The window attention operates independently within each local window.
  • Complementarity: Window attention loses global information, whereas channel attention supplies global spatial context through channel tokens and remains linear in spatial size.The two mechanisms are therefore arranged alternately to provide local and global features.
  • Channel Group Attention: Channel group attention transposes patch-level tokens, divides C channels into Ng groups of Cg channels, and performs image-level attention within each group.The grouping satisfies C = Ng × Cg, while the channel attention map has size Cg × Cg.
  • Model Instantiation: DaViT-Tiny, Small, and Base use four-stage configurations with progressively specified layer counts, channel groups, and attention heads.Tiny and Small start with C = 96, while Base starts with C = 128.

4 Analysis

The analysis contrasts DaViT with Swin and DeiT, showing how channel attention models global interactions while dual attention preserves both detail and object-focused representations. The reported evaluations cover ImageNet-1K classification and COCO detection comparisons.

  • Global Interactions: A single DaViT channel-attention block models global interactions by computing attention scores between channels using all spatial positions.This differs from patch-wise global interaction mechanisms used by several prior transformers.
  • Channel Attention Visualization: Channel attention fuses multiple input tokens, selects globally important regions, and suppresses unimportant regions in visualized feature maps.The visualization uses an output channel and its top-7 relevant input channels from the third network stage.
  • Feature Map Comparison: DaViT feature maps combine Swin's fine-grained details with DeiT's coarse global features across four network stages.The analysis reports that Swin initially lacks focus on the main object, while DeiT loses details.
  • Object Detection: COCO detection comparisons report box mAP and mask mAP for RetinaNet and Mask R-CNN under a 1× training schedule.FLOPs are measured at 800 × 1280, with additional 3×-schedule comparisons referenced separately.
  • Channel Attention Ablation: Replacing channel self-attention with SE or ECA blocks provides a quantitative comparison against alternative channel-wise operators.The comparison is conducted using the DaViT-Tiny model.

5 Experiments

DaViT is evaluated across image classification, object detection, instance segmentation, and semantic segmentation, consistently outperforming strong transformer baselines with comparable or lower computational costs. Ablations show that dual attention improves performance across stages and layouts.

  • Image Classification: 84.2% top-1 accuracy is reached by DaViT-Small with 49.7M parameters on ImageNet-1K, surpassing larger counterpart-Base models with near-half computations.DaViT-Tiny, Small, and Base improve over Swin Transformer by 1.5%, 1.1%, and 1.2%, respectively.
  • Image Classification: 90.2% and 90.4% top-1 accuracy are achieved by DaViT-Huge and DaViT-Gaint after pre-training with 1.5B weakly supervised image-text pairs.The models use 362M and 1.4B parameters, respectively.
  • Object Detection and Instance Segmentation: 1.5-2.9% gains on RetinaNet 3× and 1.0-1.4% gains on Mask R-CNN 3× over Swin Transformer models.These results use 3× training schedules and are reported for COCO object detection and instance segmentation.
  • Object Detection and Instance Segmentation: 1.8% and 0.9% improvements over the state-of-the-art on RetinaNet 3× and Mask R-CNN 3×, respectively, are achieved by the DaViT-Base model.The comparisons are reported for the base model under the 3× schedule.
  • Semantic Segmentation: 1.2-1.8% gains over Swin Transformers are obtained by DaViT-Tiny, Small, and Base on ADE20K semantic segmentation.The models use similar numbers of parameters and FLOPs, with 512 × 512 inputs.
  • Ablation Study: Dual attention consistently boosts performance at each network stage, with the best results obtained when added across all four stages.Among similar-computation layouts, window attention first performs slightly better than channel attention first or parallel attention.

6 Conclusion

DaViT combines spatial window attention and channel group attention to capture global context efficiently. The two attentions complement one another by pairing global channel interactions with fine-grained spatial refinement.

  • Conclusion: DaViT combines spatial window attention and channel group attention to model global context while maintaining computational efficiency.The mechanism is evaluated across multiple vision benchmarks.
  • Conclusion: Channel attention captures global interactions by computing attention between channels using information from all spatial positions.Each channel token contains an abstract representation of the entire image.
  • Conclusion: Spatial attention refines local representations through fine-grained interactions across spatial locations, supporting global information modeling in channel attention.The conclusion describes the two self-attentions as complementary.

A.1 Details of Model Configuration

DaViT uses a four-stage hierarchical architecture with stride-convolution patch embeddings and fixed attention-group configurations. Larger variants scale model capacity and adjust channel-attention dot products for larger resolutions and models.

  • Architecture: Four stages use stride-convolution patch embeddings with kernel sizes {7, 2, 2, 2} and strides {4, 2, 2, 2}.The stages maintain feature-map resolutions and dimensions while producing a hierarchical representation.
  • Attention Configuration: 49 tokens per spatial window and 32 channels per group and head are used across the three base model configurations.The window size is 7 × 7, and Cg = Ch = 32 for all blocks.
  • Attention Configuration: DaViT-Tiny and DaViT-Small use stage-wise head and group counts of {3, 6, 12, 24}.The provided configuration specifies these counts for the four stages.
  • FFN Ablation: Models without FFNs add dual attention blocks to keep total computation costs similar, using {2, 2, 11, 2} for Tiny and {2, 2, 28, 2} for Small and Base.The authors note that a better configuration may exist specifically for models without FFNs.

A.2 Channel Attention on Vanilla ViT

Applying channel group attention to vanilla DeiT tests whether the dual-attention mechanism transfers beyond the full DaViT architecture. The experiment keeps parameter counts and FLOPs comparable with vanilla DeiT.

  • Generalizability: Channel group attention is alternatively arranged with vanilla DeiT patch-level self-attention to test the mechanism's generalizability.The number of groups and channels is matched to DeiT's heads and channels for comparable parameters and FLOPs.

A.3 Transformer without FFNs

The study tests whether DaViT can retain effective dual-attention modeling without feed-forward networks. Removing FFNs and adding dual-attention blocks yields gains over single-attention alternatives, though the resulting model remains below the full DaViT and recent state-of-the-art models.

  • FFNs dominate DaViT’s FLOPs and parameter count, motivating an exploration of pure dual attention without FFNs.The authors remove FFNs and add more dual-attention blocks to match computational costs.
  • 1.5% and 1.7% higher Top-1 accuracy than pure window attention and channel attention, respectively, demonstrates the value of combining spatial-wise and channel-wise interactions.
  • The FFN-free model matches or exceeds models such as PVT and DeiT but remains inferior to Swin, Focal, and the full DaViT.

A.4 Model Capacity against Model Depth

DaViT’s dual spatial and channel attention provides modeling capacity that remains competitive with deeper transformer architectures. It also improves practical efficiency and outperforms conventional CNN channel-attention replacements in the reported comparisons.

  • Attention interactions: DaViT combines spatial-wise and channel-wise attention with local and global interactions within each dual-attention block.
  • Model depth: Using two fewer layers, DaViT achieves performance comparable to or better than Swin Transformer.The third stage is reduced from three layers to two and then one, compared with six and four corresponding layers in Swin and Focal.
  • Model depth: DaViT consistently outperforms existing state-of-the-art models at the same depth and surpasses Focal Transformer by a large margin with fewer computational costs.
  • Inference efficiency: DaViT has higher real-time throughput than Swin Transformer in the reported comparison.The authors attribute this efficiency advantage to a cleaner structure and group channel attention, including removal of shifted-window partitioning.
  • Channel attention: 1.6% superior performance than SE- and ECA-based variants shows the reported advantage of DaViT’s channel group self-attention.The comparison replaces channel self-attention in the tiny model with SE or ECA blocks; the proposed attention performs dynamic feature fusion across global tokens.
Loading 2204.03645v1…