Source-linked AI summary
Self-Supervised Learning with Swin Transformers
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, Han Hu
TL;DR
ViT/DeiT-based self-supervised methods had limited evidence on dense prediction tasks, motivating a more general-purpose Transformer backbone. MoBY combines MoCo v2 and BYOL with Swin Transformer, reaching 75.0% ImageNet-1K top-1 accuracy with Swin-T and enabling downstream evaluation. Its downstream results are on par with supervised methods, while the authors identify room for improvement.
Problem
Recent Transformer-based self-supervised methods mainly reported ImageNet-1K linear evaluation because ViT/DeiT were not well suited to dense prediction tasks.
Method
MoBY combines MoCo v2 and BYOL components and adopts Swin Transformer as a general-purpose backbone.
Results
75.0% top-1 accuracy is achieved on ImageNet-1K with Swin-T after 300-epoch training, while downstream COCO detection and ADE20K segmentation performance is on par with supervised methods.
Takeaways & Limitations
Swin Transformer enables more comprehensive evaluation of Transformer-based self-supervised representations across classification, object detection, and semantic segmentation.
Takeaways & Limitations
The approach introduces no new inventions and performs comparably or slightly worse than supervised methods on downstream tasks.
Abstract
from arXiv · showhide
We are witnessing a modeling shift from CNN to Transformers in computer vision. In this work, we present a self-supervised learning approach called MoBY, with Vision Transformers as its backbone architecture. The approach basically has no new inventions, which is combined from MoCo v2 and BYOL and tuned to achieve reasonably high accuracy on ImageNet-1K linear evaluation: 72.8% and 75.0% top-1 accuracy using DeiT-S and Swin-T, respectively, by 300-epoch training. The performance is slightly better than recent works of MoCo v3 and DINO which adopt DeiT as the backbone, but with much lighter tricks. More importantly, the general-purpose Swin Transformer backbone enables us to also evaluate the learnt representations on downstream tasks such as object detection and semantic segmentation, in contrast to a few recent approaches built on ViT/DeiT which only report linear evaluation results on ImageNet-1K due to ViT/DeiT not tamed for these dense prediction tasks. We hope our results can facilitate more comprehensive evaluation of self-supervised learning methods designed for Transformer architectures. Our code and models are available at https://github.com/SwinTransformer/Transformer-SSL, which will be continually enriched.
1 Introduction
MoBY combines self-supervised learning with Transformer backbones, using Swin Transformer to support evaluation beyond ImageNet classification. It achieves strong linear-evaluation accuracy but leaves downstream transfer room for improvement.
- Downstream evaluation: Swin Transformer enables evaluation on COCO object detection and ADE20K semantic segmentation, unlike recent ViT/DeiT-based approaches focused on linear evaluation.Swin is adopted because it is designed as a general-purpose vision backbone for dense prediction tasks.
- Approach: MoBY combines MoCo v2 and BYOL into a self-supervised approach for Transformer backbones.The method combines prior components and uses tuned training choices rather than introducing new inventions.
- ImageNet-1K evaluation: 72.8% top-1 accuracy is achieved with DeiT-S after 300-epoch ImageNet-1K training, slightly exceeding MoCo v3 and DINO with lighter tricks.
- ImageNet-1K evaluation: 75.0% top-1 accuracy is achieved with Swin-T after 300-epoch training, 2.2% higher than DeiT-S.
- Downstream evaluation: On COCO detection and ADE20K segmentation, MoBY representations perform on par with supervised methods, indicating substantial room for improving Transformer-based self-supervised learning.
2 A Baseline SSL Method with Swin Transformers
MoBY combines MoCo v2 and BYOL components in a two-encoder architecture built around Transformer backbones. Swin Transformer supplies hierarchical, efficient representations suitable for downstream dense prediction evaluation.
- MoBY design: MoBY inherits MoCo v2’s momentum design, key queue, and contrastive loss, while adopting BYOL’s asymmetric encoders, augmentations, and momentum scheduler.
- MoBY design: The online encoder is gradient-updated, while the target encoder is a momentum-updated moving average whose momentum gradually increases to 1.The default starting momentum is 0.99.
- MoBY design: The contrastive loss compares an online feature with the target feature from the same image against target features stored in a key queue.The queue size is 4096 by default, and τ is the temperature term.
- Training: MoBY uses AdamW and asymmetric drop path regularization, with asymmetric rates reported as crucial for final performance.The training setup fixes the learning rate at 0.001 and weight decay at 0.05 while tuning queue size, momentum, temperature, and drop-path rates.
- Swin Transformer backbone: Swin Transformer uses hierarchical representations and shifted local windows to improve efficiency while enabling cross-window connections.
- Swin Transformer backbone: Swin-T is selected as the default backbone because its complexity is similar to ResNet-50 and DeiT-S while supporting object detection and semantic segmentation evaluation.
3 Experiments
The experiments evaluate MoBY on ImageNet-1K linear evaluation, downstream detection and segmentation, and targeted ablations. Results show strong linear-evaluation accuracy with lightweight settings, but limited gains over supervised Transformer pretraining on downstream tasks.
- Linear Evaluation on ImageNet-1K: 72.8% top-1 accuracy is achieved by MoBY with DeiT-S after 300-epoch training, slightly exceeding MoCo v3 and DINO without multi-crop.The comparison uses DeiT-S for fair evaluation against methods adopting ViT/DeiT backbones.
- Linear Evaluation on ImageNet-1K: MoBY mainly follows standard MoCo v2 and BYOL settings with batch size 512, while MoCo v3 and DINO use heavier tricks.The reported settings are intended to remain affordable for most labs.
- Linear Evaluation on ImageNet-1K: Replacing layer normalization before MLP blocks with batch normalization improves 100-epoch Swin-T accuracy from 70.9% to 72.0%, but changes the original Swin architecture.This modification is not used as the standard setting, and broader Transformer improvements are left for future study.
- Linear Evaluation on ImageNet-1K: 75.0% top-1 accuracy is achieved with Swin-T after 300-epoch training, surpassing DeiT-S by +2.2%.The Swin-T versus DeiT-S gap is larger than the corresponding supervised-learning gap of +1.5%.
- Transferring Performance on Downstream Tasks: MoBY and supervised pretraining perform similarly when transferred to COCO object detection and instance segmentation.The evaluation uses Mask R-CNN and Cascade Mask R-CNN under 1x and 3x settings.
- Transferring Performance on Downstream Tasks: MoBY performs slightly worse than supervised pretraining on ADE20K semantic segmentation, indicating room for improvement in Transformer-based self-supervised learning.The evaluation uses UPerNet on ADE20K.
- Ablation Study: Increasing online-encoder drop path from 0.05 to 0.1 benefits representation learning, whereas adding drop path to the target encoder causes a 1.9% accuracy drop.The target-encoder change reduces accuracy from 70.9% to 69.0%, motivating asymmetric drop path rates.
- Ablation Study: Performance remains stable for queue sizes from 1024 to 16384; the defaults are queue size 4096, temperature 0.2, and starting momentum 0.99.The temperature and starting momentum values are selected because they perform best in their respective ablations.
4 Conclusion
MoBY combines a Vision Transformer backbone with a self-supervised learning recipe, achieving strong ImageNet-1K linear-evaluation accuracy while enabling downstream-task evaluation.
- 72.8% and 75.0% top-1 accuracy are achieved with DeiT-S and Swin-T, respectively, after 300-epoch training.
- The general-purpose Swin Transformer backbone enables evaluation on object detection and semantic segmentation alongside ImageNet-1K linear evaluation.
- MoBY performs comparably or slightly worse than supervised methods on downstream tasks, indicating room to improve self-supervised learning with Transformer architectures.
- The paper aims to facilitate more comprehensive evaluation of self-supervised learning methods designed for Transformer architectures.