Source-linked AI summary
SiT: Self-supervised vIsion Transformer
Sara Atito, Muhammad Awais, Josef Kittler
TL;DR
Vision transformers have mainly relied on large-scale supervised pretraining or teacher guidance, motivating investigation of self-supervised pretraining for downstream classification. SiT uses transformer autoencoding and multiple self-supervised tasks, and outperforms existing self-supervised methods while supporting classification on small datasets.
Problem
Vision transformers generally require large-scale supervised data or teacher guidance, while manual annotation is expensive and their data-hungry nature limits training on small datasets.
Method
SiT pretrains vision transformers on unlabelled data through pretext tasks, using a transformer autoencoder with a lightweight decoder and multiple loss functions.
Results
SiT outperformed state-of-the-art self-supervised methods with wide margins across classification benchmarks and enabled transformer finetuning on datasets with only a few thousand images.
Takeaways & Limitations
SiT supports self-supervised vision-transformer pretraining for small-data classification and can learn useful representations for linear evaluation and clustering without supervision.
Takeaways & Limitations
The discussion identifies a limitation of contrastive learning: it often emphasizes global transformation invariance while disregarding contextual representations.
Abstract
from arXiv · showhide
Self-supervised learning methods are gaining increasing traction in computer vision due to their recent success in reducing the gap with supervised learning. In natural language processing (NLP) self-supervised learning and transformers are already the methods of choice. The recent literature suggests that the transformers are becoming increasingly popular also in computer vision. So far, the vision transformers have been shown to work well when pretrained either using a large scale supervised data or with some kind of co-supervision, e.g. in terms of teacher network. These supervised pretrained vision transformers achieve very good results in downstream tasks with minimal changes. In this work we investigate the merits of self-supervised learning for pretraining image/vision transformers and then using them for downstream classification tasks. We propose Self-supervised vIsion Transformers (SiT) and discuss several self-supervised training mechanisms to obtain a pretext model. The architectural flexibility of SiT allows us to use it as an autoencoder and work with multiple self-supervised tasks seamlessly. We show that a pretrained SiT can be finetuned for a downstream classification task on small scale datasets, consisting of a few thousand images rather than several millions. The proposed approach is evaluated on standard datasets using common protocols. The results demonstrate the strength of the transformers and their suitability for self-supervised learning. We outperformed existing self-supervised learning methods by large margin. We also observed that SiT is good for few shot learning and also showed that it is learning useful representation by simply training a linear classifier on top of the learned features from SiT. Pretraining, finetuning, and evaluation codes will be available under: https://github.com/Sara-Ahmed/SiT.
1 INTRODUCTION
The introduction motivates self-supervised pretraining for vision transformers, whose data hunger and reliance on labels limit their use on small datasets. It presents SiT and GMML as a transformer-based framework combining reconstruction and discriminative learning.
- Vision transformers require substantial data because they lack the inductive bias of conventional vision models.
- Single image labels can produce sub-expressive features because natural images contain multiple concepts that annotations do not fully capture.
- Contrastive learning obtains invariant image embeddings but can neglect contextual representations and requires mechanisms to avoid representation collapse.
- GMML learns visual representations by reconstructing up to 70% of missing visual information from visible-token context.
- SiT combines reconstruction and contrastive losses in a transformer-based autoencoder framework for self-supervised learning.
- The paper reports that SiT can train transformers on datasets containing only a few thousand samples and consistently outperform supervised pretraining for downstream classification.
2 RELATED WORKS
Related work covers discriminative self-supervised methods, especially contrastive and clustering approaches, and positions SiT as combining pretext reconstruction with contrastive learning. It also contrasts GMML with transformer masked-autoencoder methods and external-knowledge approaches.
- Contrastive approaches: Contrastive methods bring augmented views of the same image closer while separating views from different images.
- Contrastive approaches: Contrastive learning often relies on large batches, memory banks, or additional techniques to avoid representation collapse.
- Clustering approaches: Deep clustering methods generate representation-learning targets from image clusters but require computationally expensive clustering and precautions against trivial solutions.
- SiT relative to prior work: SiT combines pretext-task learning with contrastive learning to obtain useful and discriminative representations in a transformer-based framework.
- Comparison with post art: GMML shares the masked-autoencoder principle with SimMIM and MAE while using a lighter decoder than MAE.
- Comparison with post art: BeIT uses an unsupervised encoder to group visual patches into a vocabulary, introducing external knowledge into its training targets.
3 METHODOLOGY
SiT is a self-supervised vision transformer that learns representations by reconstructing corrupted local image content and estimating geometric transformations, then transfers learned features to downstream classification. Its GMML-based architecture combines transformer context with lightweight decoding and joint reconstruction and contrastive objectives.
- Self-supervised representation learning: SiT learns visual integrity by recovering partially masked or transformed image parts and estimating geometric transformations applied to the input.The learned weights can initialize downstream tasks such as image classification, object detection, and segmentation.
- Vision Transformer architecture: The vision-transformer backbone tokenizes images into patches, adds positional embeddings, and processes them through stacked multi-head self-attention and MLP blocks.Self-attention matches queries with keys and produces weighted combinations of value vectors.
- Self-supervised architecture: Unlike ViT, SiT replaces the classification token during pretraining with data tokens for reconstruction and a contrastive token for contrastive prediction.After pretraining, the contrastive head can be replaced with a classifier for downstream transfer learning.
- Pretraining objectives: SiT jointly performs GMML-based image reconstruction and contrastive learning using a transformer autoencoder with a lightweight decoder.Image reconstruction uses local transformations of spatially neighboring patches, while contrastive learning compares representations from augmented views.
- Image reconstruction: The reconstruction objective restores the original image from corrupted input using an ℓ1 loss, with reconstructed images produced from the backbone’s data-token outputs.The light decoder receives the data-token representations from the vision-transformer encoder.
4 EXPERIMENTAL RESULTS
Experiments evaluate SiT across classification, multi-label, and segmentation settings using ViT-S/16 and related protocols. Across small and large datasets, SiT pretraining improves downstream performance and often exceeds competing supervised or self-supervised methods.
- Experimental setup: SiT is evaluated on multi-class, multi-label, and video instance segmentation datasets using common self-supervised pretraining and downstream finetuning protocols.The experiments include dataset statistics, implementation settings, domain transfer, multi-label classification, and video object segmentation.
- Small-dataset classification: Self-supervised SiT pretraining consistently improves performance over training ViT-S/16 from scratch on small datasets, with gains up to 64.7% on Cars.The comparison uses target-dataset pretraining and finetuning without external data.
- Small-dataset classification: +3.9%, +11.7%, +11.8%, +8.8%, +1.1%, +1.1%, +0.8%, and +1.8% improvements are reported on Flowers, Pets, CUB, Aircraft, STL10, Cars, CIFAR10, and CIFAR100, respectively, over concurrent SSL methods.SimMIM is excluded from the small-dataset comparison because its pretrained models did not converge.
- Large-scale pretraining: With ImageNet-1K pretraining, SiT outperforms supervised pretraining on most downstream datasets and improves ImageNet-1K finetuning by 1.1% for ViT-S/16.SiT is also reported as outperforming or matching competing methods on most small and large datasets with ViT-B/16.
- Multi-label classification: SiT pretraining improves multi-label mAP by +40.6, +27.1, and +8.0 absolute points on PASCAL VOC, MS-COCO, and Visual-Genome without external data.ImageNet-1K-pretrained SiT further exceeds SSL state-of-the-art by +0.5 and +0.6 absolute mAP on MS-COCO and Visual-Genome.
- Ablation and transfer: Using reconstruction alone produced performance similar to training from scratch, while the combined SiT design supports improved downstream finetuning and strong comparisons against supervised and self-supervised methods.The reconstruction-only result is attributed to possible identity mapping without suitable constraints.
5 CONCLUSION
SiT uses self-supervised pretext tasks and transformer autoencoding to initialize downstream classification models. It combines reconstruction with other loss functions and reports wide-margin improvements over existing self-supervised methods.
- SiT is pretrained on unlabelled data for pretext tasks, then used to initialize finetuning for downstream classification.
- The transformer architecture enables SiT to function as an autoencoder and combine reconstruction with different loss functions.
- SiT adds an extra token for contrastive learning alongside reconstruction loss.
- SiT outperformed state-of-the-art self-supervised methods with wide margins on the studied task.
- The study focuses on image classification, while segmentation and detection remain proposed directions for future investigation.