PaperlayerView exact revision on arXiv

Source-linked AI summary

Swin Transformer V2: Scaling Up Capacity and Resolution

Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo

arXiv:2111.09883v2cs.CV

TL;DR

Large vision models face training instability, resolution gaps between pre-training and fine-tuning, and heavy labeled-data requirements. Swin Transformer V2 addresses these issues with architectural changes, continuous position bias, and SimMIM, reaching state-of-the-art results across four benchmarks with a 3-billion-parameter model.

  • Problem

    Large vision models face instability during training, large window-resolution gaps between pre-training and fine-tuning, and increasing dependence on huge labeled datasets.

  • Method

    The paper combines res-post-norm and scaled cosine attention, Log-CPB for variable window sizes, and SimMIM self-supervised pre-training.

  • Results

    Swin Transformer V2 sets state-of-the-art results on four representative vision benchmarks using a 3-billion-parameter model trained with 40× less labeled data than JFT-3B practice.

  • Takeaways & Limitations

    The approach scales Swin Transformer V2 to 3 billion parameters and images up to 1,536×1,536 resolution while transferring across window resolutions.

  • Takeaways & Limitations

    Comparisons did not particularly align model complexities or pre-training data.

Abstract

from arXiv · show

Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities like that of human beings. This paper aims to explore large-scale models in computer vision. We tackle three major issues in training and application of large vision models, including training instability, resolution gaps between pre-training and fine-tuning, and hunger on labelled data. Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images. Through these techniques, this paper successfully trained a 3 billion-parameter Swin Transformer V2 model, which is the largest dense vision model to date, and makes it capable of training with images of up to 1,536$\times$1,536 resolution. It set new performance records on 4 representative vision tasks, including ImageNet-V2 image classification, COCO object detection, ADE20K semantic segmentation, and Kinetics-400 video action classification. Also note our training is much more efficient than that in Google's billion-level visual models, which consumes 40 times less labelled data and 40 times less training time. Code is available at \url{https://github.com/microsoft/Swin-Transformer}.

1. Introduction

Swin Transformer V2 addresses the instability, resolution-transfer, and labeled-data challenges that limit scaling vision models. It scales capacity and resolution to a 3-billion-parameter model achieving records across four representative vision benchmarks.

  • Res-post-norm and scaled cosine attention improve capacity scaling, while Log-CPB transfers models across varying window resolutions.
  • Large vision models had reached about 1–2 billion parameters, but existing models were mainly applied to image classification.
  • Training instability arises because residual outputs accumulate activation amplitudes across layers in large models.
  • A 3-billion-parameter Swin Transformer V2 trains with images up to 1,536×1,536 resolution and 40× less labeled data than JFT-3B practice.
  • The model achieves 84.0% top-1 accuracy on ImageNet-V2, 63.1 / 54.4 box / mask AP on COCO, 59.9 mIoU on ADE20K, and 86.8% top-1 accuracy on Kinetics-400.

2. Related Works

Related work traces scaling progress in language and vision networks, the rise of vision Transformers, and methods for transferring positional information across resolutions. Swin Transformer V2 builds on these directions with continuous bias generation for arbitrary window sizes.

  • Language models scaled from BERT-340M to 530B dense and 1.6T sparse parameters, improving language benchmarks and zero-shot or few-shot performance.
  • Vision networks progressed from CNN scaling to vision Transformers that established strong results across image, region, pixel, and video benchmarks.
  • Local vision Transformers can change window size during fine-tuning, creating a need to transfer positional information across resolutions.
  • Relative positional bias is common in vision Transformers, while directly learning bias values as weights limits flexibility across window sizes.
  • Log-CPB uses a meta network and log-spaced coordinates to transfer relative position biases across arbitrary window sizes.

3. Swin Transformer V2

Swin Transformer V2 addresses instability when scaling model capacity and degraded transfer across window resolutions. It combines residual post-normalization, scaled cosine attention, and log-spaced continuous position bias to support larger, higher-resolution vision models.

  • Swin Transformer is a general-purpose hierarchical, local, translation-invariant vision backbone for classification, detection, and segmentation.
  • Scaling challenges: Scaling the original Swin Transformer exposes activation instability and degraded performance when transferring across window resolutions.At huge scale, activation-amplitude discrepancies reach 10^4, and the original model cannot complete training; bicubic bias interpolation loses accuracy at larger resolutions.
  • Scaling model capacity: Residual post-normalization normalizes each residual output before merging, preventing deeper-layer amplitude accumulation.An additional main-branch layer normalization is used every six Transformer blocks in the largest model.
  • Scaling model capacity: The combination of post-normalization and scaled cosine attention stabilizes training in huge models.Activation values remain milder at deeper layers, and self-supervised pre-training succeeds where the original Swin Transformer diverges.
  • Scaling model capacity: Scaled cosine attention replaces query-key dot products with normalized cosine similarities and a learnable, head- and layer-specific temperature.The cosine function produces milder attention values, addressing attention maps dominated by a few pixel pairs.
  • Scaling window resolution: Log-spaced continuous position bias uses a meta network over log-spaced relative coordinates to generate bias values for arbitrary window sizes.The generated values can be transferred to fine-tuning tasks with varying window sizes and pre-computed for inference.
  • Scaling window resolution: Log-spaced CPB performs best among the compared position-bias approaches, particularly when transferred to larger window sizes.
  • Implementation: The implementations enable training a 3B model for COCO at 1,536×1,536 resolution and Kinetics-400 at 320 × 320 × 8 resolution.

4. Experiments

Experiments evaluate SwinV2 across image classification, detection, segmentation, and video recognition while scaling model capacity and resolution. The results support stable training, transfer across window resolutions, and strong performance from a 3-billion-parameter model.

  • Experiments cover ImageNet-1K classification, COCO detection, ADE20K segmentation, and Kinetics-400 action classification.
  • Image classification: 84.0% top-1 accuracy on ImageNet-V2 is +0.7% above the previous best, while ImageNet-1K V1 reaches 90.17% versus 90.88%.The authors note fewer training iterations and lower image resolutions than previous efforts.
  • Dense vision recognition: 63.1/54.4 box/mask AP on COCO is +1.8/1.4 above the previous best, and 59.9 mIoU on ADE20K is +1.5 higher.Larger test-time windows provide additional gains in both tasks, probably attributed to Log-CPB.
  • Video action classification: 86.8% top-1 accuracy on Kinetics-400 is +1.4% above the previous best, with larger test-time windows adding +0.2%.The additional gain is probably attributed to the effective Log-CPB approach.
  • Ablation study: Res-post-norm and scaled cosine attention improve accuracy by +0.2%, +0.4%, and +0.5% at tiny, small, and base sizes, respectively.Their combination stabilizes training, including self-supervised pre-training of a huge model that diverges with the original Swin Transformer.
  • Ablation study: Continuous position bias outperforms SwinV1’s parameterized bias after resolution transfer, with Log-CPB marginally better than the linear-spaced version.Its benefit increases as the resolution change between pre-training and fine-tuning grows.
  • Ablation study: Without fine-tuning, enlarging the window from 8 to 24 retains 78.9% versus 81.8%, while the original approach falls from 81.7% to 68.7%.A previously unseen window size of 12 can reach +0.4% over the original accuracy.

5. Conclusion

Swin Transformer V2 scales vision models to 3 billion parameters and 1,536×1,536 training images, setting new records on four representative vision benchmarks.

  • 3 billion parameters and 1,536×1,536 training images demonstrate Swin Transformer V2’s scaled capacity and resolution.The approach combines residual-post-norm, scaled cosine attention, and log-spaced continuous relative position bias.
  • New performance records were set on four representative vision benchmarks.The benchmarks span image classification, object detection, semantic segmentation, and video action classification.

A1. Experimental Settings for Ablation

The ablation experiments use SwinV2-T, SwinV2-S, and SwinV2-B across ImageNet-1K classification, COCO detection, and ADE20K segmentation under specified training and fine-tuning settings.

  • Ablations evaluate SwinV2-T, SwinV2-S, and SwinV2-B on ImageNet-1K, COCO, and ADE20K.ImageNet-1K is used for pre-training, followed by task-specific fine-tuning where applicable.
  • ImageNet-1K ablations use 256×256 inputs with 8×8 windows, 300 epochs, AdamW, cosine decay, and 20-epoch warm-up.The setup uses batch size 1024, initial learning rate 1×10^-3, weight decay 0.05, and gradient clipping with max norm 5.0.
  • ImageNet-1K fine-tuning lasts 30 epochs when downstream resolution exceeds pre-training resolution.It uses AdamW, cosine decay, an initial learning rate of 4 × 10^-5, and weight decay of 1 × 10^-8.
  • COCO detection uses cascade mask R-CNN with multi-scale augmentation and 16×16 windows.The shorter image side ranges from 480 to 800, the longer side is 1333, and training uses a 3× scheduler.
  • ADE20K segmentation uses 512×512 images, 16×16 windows, batch size 16, and 160K training iterations.The setup includes linear learning-rate decay, 1,500-iteration warm-up, and stochastic depth ratio 0.3 across models.

A2. Experimental Settings for System-Level Comparison

System-level comparisons fine-tune SwinV2-B and SwinV2-L from ImageNet-22K pre-training across image, detection, segmentation, self-supervised, and video settings with task-specific resolutions.

  • SwinV2-B and SwinV2-L are first pre-trained on ImageNet-22K before fine-tuning on downstream recognition tasks.The system-level experiments include ImageNet classification, COCO detection, and ADE20K segmentation.
  • ImageNet-22K pre-training uses 192×192 inputs, 12×12 windows, 90 epochs, batch size 4096, and AdamW.The schedule includes cosine decay, 5-epoch warm-up, weight decay 0.1, and gradient clipping with max norm 5.0.
  • ImageNet-1K evaluation tests top-1 accuracy on both ImageNet-1K V1 and V2.
  • COCO detection uses 1,536×1,536 inputs with 32×32 windows and HTC++ after Objects-365 V2 intermediate fine-tuning.The COCO stage trains for 45,000 steps with batch size 96 and a multi-scale ratio of [0.1, 2.0].
  • ADE20K segmentation uses 640×640 images, 40×40 windows, batch size 64, and 40K iterations.The training uses AdamW, linear learning-rate decay, and a 375-iteration warm-up.
  • Self-supervised pre-training uses the 70-million-image ImageNet-22K-ext dataset for 20 epochs at 192×192 resolution.The stage uses a batch size of 9216 and AdamW with cosine decay and 30000 steps of linear warm-up.
  • Kinetics-400 fine-tuning uses two stages, increasing video resolution from 256×256×8 to 320×320×8.The corresponding window sizes increase from 16×16×8 to 20×20×8.

A3. Learnt Relative Position Bias by Different Approaches

Figure 4 visualizes learned relative position bias matrices across pre-training and fine-tuning resolutions, comparing 8×8 and 16×16 windows for three first-block heads.

  • Figure 4 shows relative position bias matrices learned by different approaches with a SwinV2-T model.The visualization covers the three heads in the first block.
  • The left panels correspond to 256×256 pre-training with 8×8 windows, while the right panels correspond to 512×512 fine-tuning with 16×16 windows.H-x denotes the x-th head.
Loading 2111.09883v2…