PaperlayerView exact revision on arXiv

Source-linked AI summary

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo

arXiv:2103.14030v2cs.CVcs.LG

TL;DR

Adapting Transformers to vision is difficult because visual elements vary in scale, images have far higher resolution, and global attention scales quadratically with token count. Swin Transformer uses hierarchical representations and shifted local windows, achieving state-of-the-art results across classification, detection, and segmentation.

  • Problem

    Transformers need vision-specific designs because visual elements vary in scale, images have high resolution, and global self-attention has quadratic complexity in token count.

  • Method

    Swin Transformer builds hierarchical feature representations using efficient self-attention within non-overlapping windows whose partitions shift between consecutive layers to connect windows.

  • Results

    Swin Transformer achieves state-of-the-art performance across image classification, object detection, and semantic segmentation, including 58.7 box AP on COCO and 53.5 mIoU on ADE20K.

  • Takeaways & Limitations

    The results support Swin Transformer as a general-purpose vision backbone with linear computational complexity relative to input image size.

  • Takeaways & Limitations

    Sliding-window self-attention can have inefficient memory access in practice when different query pixels use different key sets.

Abstract

from arXiv · show

This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with \textbf{S}hifted \textbf{win}dows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at~\url{https://github.com/microsoft/Swin-Transformer}.

1. Introduction

Swin Transformer adapts Transformers to vision through hierarchical feature maps and shifted-window self-attention, addressing visual scale variation while maintaining linear complexity in image size. It delivers strong results across classification, detection, and segmentation, including state-of-the-art COCO performance.

  • Architecture: Swin Transformer builds hierarchical feature maps by merging image patches and computes self-attention within local windows, yielding linear computation complexity in input image size.This design supports both image classification and dense recognition tasks.
  • Motivation: The architecture targets vision’s varying visual-element scales, a challenge arising when transferring Transformers from language, where word tokens are the basic processing elements.The paper seeks a general-purpose computer-vision backbone analogous to Transformers in NLP and CNNs in vision.
  • Shifted Windows: Shifted window partitioning between consecutive layers connects neighboring windows and significantly enhances modeling power while preserving efficient hardware memory access.All query patches within a window share the same key set, facilitating memory access in hardware.
  • Shifted Windows: The shifted-window approach has much lower latency than sliding windows while offering similar modeling power, and it also benefits all-MLP architectures.The latency and modeling-power comparisons are reported in Tables 5 and 6.
  • Results: 58.7 box AP and 51.1 mask AP on COCO test-dev surpass prior state of the art by +2.7 box AP and +2.6 mask AP, respectively.The comparisons are against Copy-paste [26] without external data for box AP and DetectoRS [46] for mask AP.

2. Related Work

This section situates Swin Transformer among established CNN backbones, self-attention extensions, and Transformer-based vision models. It emphasizes CNNs’ continued dominance while identifying limitations of existing self-attention and ViT approaches for general-purpose, high-resolution vision.

  • CNN and variants: CNNs remain the primary computer-vision backbones, following advances from AlexNet and architectures including VGG, GoogleNet, ResNet [30], DenseNet, HRNet, and EfficientNet.Related architectural improvements also include depthwise convolution and deformable convolution [18].
  • Self-attention based backbone architectures: Existing self-attention backbones replace some or all spatial convolutions with local-window attention, achieving slightly better accuracy/FLOPs trade-offs than corresponding ResNet architectures.Sliding-window self-attention can nevertheless suffer from inefficient practical memory access when moving weights across a feature map.
  • Self-attention/Transformers to complement CNNs: Self-attention and Transformers also augment CNN backbones or heads to encode distant dependencies and heterogeneous interactions, and encoder-decoder Transformers have been applied to object detection and instance segmentation [7] [23] [74] [55] [32] [27] [8] [13] [56].
  • Transformer based vision backbones: ViT [20] and follow-ups [63] [72] [15] [28] [66] apply Transformers to non-overlapping image patches, but their low-resolution feature maps and quadratic image-size complexity limit general-purpose dense-vision use.Direct upsampling or deconvolution adaptations for detection and segmentation report relatively lower performance [2] [81].

3. Method

Swin Transformer combines hierarchical patch-based representations with shifted-window self-attention, achieving scalable local computation while connecting neighboring windows. Its multi-stage feature resolutions align with convolutional backbones, enabling replacement of existing vision-task backbones.

  • Architecture: Patch merging progressively reduces tokens across four stages, producing hierarchical feature maps at H/4 × W/4, H/8 × W/8, H/16 × W/16, and H/32 × W/32 resolutions.Each 2 × 2 patch-merging operation reduces resolution by 2× and changes the output dimension from 4C concatenated features to 2C.
  • Architecture: The resulting resolutions match typical convolutional backbones such as VGG and ResNet [30], allowing Swin Transformer to replace backbones in diverse vision methods.Figure 3 depicts the Swin-T architecture and its successive regular- and shifted-window blocks.
  • Swin Transformer block: Each Swin Transformer block replaces standard MSA with shifted-window MSA while retaining LayerNorm, residual connections, and a two-layer GELU MLP.The block applies LayerNorm before each attention and MLP sublayer.
  • Shifted window partitioning in successive blocks: Shifted-window self-attention alternates regular and displaced non-overlapping partitions to connect neighboring windows while retaining efficient local computation.Global self-attention is quadratic in the number of patches, whereas windowed attention is linear when the window size M is fixed, with M=7 by default.
  • Relative position bias: Learned relative position bias improves performance over omitting the bias or using absolute position embeddings, while adding absolute input embeddings further decreases performance slightly.The relative bias can also initialize fine-tuning with a different window size through bi-cubic interpolation [20] [63].
  • Model variants: The model family includes Swin-T, Swin-S, Swin-B, and Swin-L, spanning approximately 0.25× to 2× the base model’s size and computational complexity.Swin-T and Swin-S have complexity similar to ResNet-50 (DeiT-S) and ResNet-101, respectively, with default window size M=7 and head query dimension d=32.

4. Experiments

Experiments show that Swin Transformer achieves strong results across ImageNet-1K classification, COCO detection, and ADE20K segmentation, while shifted windows, relative position bias, and efficient implementations provide consistent benefits.

  • ImageNet-1K classification: 86.4% top-1 accuracy is achieved after ImageNet-22K pre-training, while Swin-T and Swin-B surpass comparable DeiT models by +1.5% and +1.5%/+1.4%.For Swin-B, ImageNet-22K pre-training adds 1.8%∼1.9% over ImageNet-1K training from scratch.
  • COCO object detection: +3.4∼4.2 box AP gains over ResNet-50 are obtained consistently across four detection frameworks, while the best model reaches 58.7 box AP and 51.1 mask AP.Against ResNeXt101-64x4d, Swin achieves +3.6 box AP and +3.3 mask AP with similar model size, FLOPs, and latency.
  • Results: 58.7 box AP and 51.1 mask AP on COCO test-dev surpass previous best results by +2.7 box AP and +2.6 mask AP.Swin-L with ImageNet-22K pre-training achieves 53.5 mIoU on ADE20K val, surpassing the previous best by +3.2 mIoU.
  • ADE20K semantic segmentation: +5.3 mIoU over DeiT-S, +4.4 mIoU over ResNet-101, and +2.4 mIoU over ResNeSt-101 are reported on ADE20K, while Swin-L reaches 53.5 mIoU.Swin-S achieves 49.3 versus 44.0 mIoU compared with DeiT-S at similar computation cost.
  • Ablations: +1.1% top-1 accuracy, +2.8 box AP/+2.2 mask AP, and +2.8 mIoU result from shifted windows across ImageNet-1K, COCO, and ADE20K.The gains indicate that shifted windows connect windows across preceding layers, with small latency overhead.
  • Ablations: Relative position bias improves classification, detection, and segmentation over no positional encoding and absolute position embedding, whereas absolute embeddings harm dense prediction.Relative position bias yields +1.2%/+0.8% top-1 accuracy, +1.3/+1.5 box AP, +1.1/+1.3 mask AP, and +2.3/+2.9 mIoU.
  • Efficiency: Cyclic self-attention implementation yields 13%, 18%, and 18% speed-ups for Swin-T, Swin-S, and Swin-B, respectively, over naive padding.Shifted-window architectures are 4.1/1.5, 4.0/1.5, and 3.6/1.5 times faster than sliding-window variants for Swin-T, Swin-S, and Swin-B.

5. Conclusion

Swin Transformer produces hierarchical representations with linear computational complexity in input image size and achieves state-of-the-art results on COCO object detection and ADE20K semantic segmentation. Its shifted-window self-attention is effective and efficient for vision, motivating investigation in natural language processing and unified vision-language modeling.

  • Swin Transformer produces hierarchical feature representations with linear computational complexity relative to input image size.
  • Swin Transformer achieves state-of-the-art performance on COCO object detection and ADE20K semantic segmentation, significantly surpassing previous best methods.
  • The paper motivates investigating shifted-window self-attention in natural language processing and pursuing unified modeling of vision and language signals.
  • Shifted-window self-attention is effective and efficient for vision problems.

A1. Detailed Architectures

Table 7 specifies the detailed architectures under a shared 224×224 input assumption, including patch-feature concatenation, 96-dimensional linear layers, and 7×7 attention windows.

  • A1. Detailed Architectures: Table 7 gives detailed architecture specifications for models using an input image size of 224×224.The input-size assumption applies to all architectures.
  • A1. Detailed Architectures: “Concat n × n” concatenates n × n neighboring patch features, downsampling the feature map by a factor of n.
  • A1. Detailed Architectures: “96-d” denotes a linear layer with output dimension 96, while “win. sz. 7 × 7” denotes multi-head self-attention with a 7 × 7 window.

A2. Detailed Experimental Settings

The experiments use task-specific training and evaluation protocols spanning ImageNet classification, object detection, and ADE20K segmentation. Classification uses global average pooling with single-crop evaluation, while larger-resolution models are fine-tuned from 224^2 inputs and dense prediction uses established frameworks and multi-scale procedures.

  • ImageNet classification: Classification applies global average pooling followed by a linear classifier, matching the accuracy of an additional ViT [20] or DeiT [63] class token, with single-crop top-1 evaluation.The default input resolution is 224^2; models evaluated at 384^2 are fine-tuned from 224^2 rather than trained from scratch.
  • ImageNet classification: From-scratch ImageNet-1K training uses AdamW [37] for 300 epochs with cosine decay, 20-epoch warm-up, batch size 1024, learning rate 0.001, weight decay 0.05, gradient clipping, and extensive augmentation and regularization.The listed strategies include RandAugment [17], Mixup [77], CutMix, random erasing, and stochastic depth.
  • ImageNet classification: Larger-resolution classification fine-tuning uses AdamW [37] for 30 epochs with constant learning rate 10^-5, weight decay 10^-8, unchanged augmentation and regularization, and stochastic depth ratio 0.1.ImageNet-22K pre-training uses 14.2 million images across 22K classes in a two-stage procedure before ImageNet-1K fine-tuning.
  • Object detection: Object-detection ablations evaluate Cascade Mask R-CNN [6], ATSS, RepPoints v2 [12], and Sparse RCNN [56] under shared multi-scale training, AdamW, batch size 16, and a 3x 36-epoch schedule.System-level comparison uses an improved HTC [9] with stronger multi-scale training, a 6x 72-epoch schedule, soft-NMS [5], an extra global self-attention layer, and ImageNet-22K initialization; Swin models use stochastic depth ratio 0.2.
  • Semantic segmentation: ADE20K segmentation uses UperNet in mmsegmentation for 150 categories, with AdamW, linear decay and warm-up, 160K iterations on 8 GPUs, standard augmentations, and multi-scale inference.Training uses 2 images per GPU, while inference evaluates scales from 0.5× to 1.75× and reports scores after training on both training and validation images.

A3. More Experiments

Additional experiments show that larger input resolutions improve ImageNet-1K accuracy at the cost of slower inference, while AdamW generally improves COCO detection accuracy over SGD. Applying the hierarchical and shifted-window designs to MLP-Mixer yields substantially better accuracy and efficiency, indicating generalizability.

  • Input Resolution: Larger input resolutions generally improve ImageNet-1K top-1 accuracy but reduce inference speed.Table 8 evaluates input image sizes from 224^2 to 384^2.
  • Optimizer Comparison: Replacing SGD with AdamW generally improves COCO object-detection accuracy for ResNe(X)t backbones, especially smaller ones, motivating AdamW comparisons against Swin Transformer.The comparison uses Cascade Mask R-CNN.
  • Swin-Mixer: Applying the hierarchical design and shifted-window approach to MLP-Mixer architectures produces Swin-Mixer and supports the generalizability of these design choices.The experiments compare Swin-Mixer with MLP-Mixer and ResMLP on ImageNet-1K classification.
  • Swin-Mixer: 81.3% vs. 76.4%: Swin-Mixer significantly outperforms MLP-Mixer using less computation, while also offering a better speed–accuracy trade-off than ResMLP.Swin-Mixer uses 10.4G computation versus 12.7G for MLP-Mixer.
Loading 2103.14030v2…