PaperlayerView exact revision on arXiv

Source-linked AI summary

Escaping the Big Data Paradigm with Compact Transformers

Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, Humphrey Shi

arXiv:2104.05704v4cs.CVcs.LG

TL;DR

Large datasets and substantial computation have limited the use of Transformers in small-data and resource-constrained settings. The paper introduces compact Transformer variants that combine sequence pooling and convolutional tokenization, achieving strong results from scratch on small datasets and competitive performance with fewer parameters.

  • Problem

    Transformers are commonly associated with large datasets and computational resources, despite many scientific and medical domains having much smaller datasets.

  • Method

    The paper introduces ViT-Lite, CVT with sequence pooling, and CCT with a convolutional tokenizer for compact Transformer-based vision models.

  • Results

    CCT reaches 98% top-1 accuracy on CIFAR-10, while compact variants achieve competitive performance with substantially fewer parameters.

  • Takeaways & Limitations

    Compact Transformers can extend Transformer research to small-data regimes and be trained with limited computational resources.

Abstract

from arXiv · show

With the rise of Transformers as the standard for language processing, and their advancements in computer vision, there has been a corresponding growth in parameter size and amounts of training data. Many have come to believe that because of this, transformers are not suitable for small sets of data. This trend leads to concerns such as: limited availability of data in certain scientific domains and the exclusion of those with limited resource from research in the field. In this paper, we aim to present an approach for small-scale learning by introducing Compact Transformers. We show for the first time that with the right size, convolutional tokenization, transformers can avoid overfitting and outperform state-of-the-art CNNs on small datasets. Our models are flexible in terms of model size, and can have as little as 0.28M parameters while achieving competitive results. Our best model can reach 98% accuracy when training from scratch on CIFAR-10 with only 3.7M parameters, which is a significant improvement in data-efficiency over previous Transformer based models being over 10x smaller than other transformers and is 15% the size of ResNet50 while achieving similar performance. CCT also outperforms many modern CNN based approaches, and even some recent NAS-based approaches. Additionally, we obtain a new SOTA result on Flowers-102 with 99.76% top-1 accuracy, and improve upon the existing baseline on ImageNet (82.71% accuracy with 29% as many parameters as ViT), as well as NLP tasks. Our simple and compact design for transformers makes them more feasible to study for those with limited computing resources and/or dealing with small datasets, while extending existing research efforts in data efficient transformers. Our code and pre-trained models are publicly available at https://github.com/SHI-Labs/Compact-Transformers.

1. Introduction

The paper addresses whether compact Transformers can be trained from scratch on small datasets while combining attention with convolutional efficiency. It introduces progressively compact architectures and reports strong accuracy with modest computational and parameter requirements.

  • Motivation: Small datasets in scientific and medical domains make data-efficient and computationally accessible vision models important.The paper notes that such domains may lack ImageNet-scale datasets and that large infrastructure can limit reproducibility.
  • Approach: The proposed approach bridges Transformers and CNNs by combining attention with spatial invariance, sparse interactions, and weight sharing.This design targets training Transformer-based models from scratch on small datasets with fewer parameters and lower computational requirements.
  • Results: 98% top-1 accuracy on CIFAR-10 is achieved by CCT, making it substantially smaller than most comparable Transformer models.The paper also reports that CCT outperforms most comparable CNN-based models except certain NAS techniques.
  • Results: 0.28 million parameters are sufficient for a lightweight model to approach 90% top-1 accuracy on CIFAR-10.This demonstrates flexibility across model sizes.
  • Contributions: ViT-Lite extends Transformer research to small-data regimes by providing a compact model trainable from scratch on CIFAR-10.The contribution is framed as extending Transformer-based research beyond large-scale training settings.
  • Efficiency: 90% accuracy on CIFAR-10 with a GPU and 80% with a CPU are obtained in under 30 minutes.The result is reported for one NVIDIA 2080Ti GPU and an AMD 5900X CPU.

2. Related Works

Related work establishes that vision Transformers benefit from large-scale training, while later methods reduce reliance on pretraining or incorporate convolutional inductive biases. This paper focuses on training compact vision Transformers from scratch on much smaller datasets.

  • Vision Transformers: ViT showed that large-scale training can offset the inductive-bias advantages of CNNs and support competitive vision performance.The comparison emphasizes scalability and the role of training-set size.
  • Vision Transformers: ViT tokenizes images by flattening non-overlapping patches into latent vectors and adds positional embeddings to represent spatial relationships.Its architecture also uses a Transformer encoder and classification mechanism based on a class token or pooled outputs.
  • Data-Efficient Transformers: DeiT improved data efficiency through stronger augmentation, training techniques, and knowledge transfer without large-scale pretraining.Its convolutional-teacher variant transfers convolutional inductive biases to the student model.
  • Alternative Architectures: Other approaches reduce the need for large-scale pretraining by using attention-based tokenization or learnable convolutional inductive biases.T2T-ViT uses windowed attention for tokenization, while ConViT uses gated positional self-attention.
  • Research Gap: This paper asks whether vision Transformers can be trained from scratch on datasets orders of magnitude smaller than ImageNet.Its stated goal is a compact and efficient model that improves accessibility.

3. Method

The method develops ViT-Lite, CVT, and CCT by shrinking the Transformer, replacing the class token with sequence pooling, and adding convolutional tokenization. These components preserve local image information while improving flexibility and efficiency.

  • Model Variants: ViT-Lite is a smaller ViT variant adapted for small-scale learning, while CVT adds sequence pooling and CCT further adds a convolutional tokenizer.CCT’s tokenizer generates richer tokens and preserves local information.
  • Model Comparison: CVT uses sequence pooling without convolutional tokenization, whereas CCT combines sequence pooling with convolutional tokenization.The comparison identifies CVT as a simpler, faster patch-based alternative when compute is more limited.
  • Compact Design: Compact variants range from 0.22M to 3.8M parameters, using smaller depths, fewer heads, narrower hidden layers, and dataset-adjusted tokenizers.The architecture retains Transformer blocks with multi-head self-attention, MLP layers, normalization, activation, and dropout.
  • SeqPool: SeqPool maps the Transformer output sequence from Rb×n×d to Rb×d using learned attention weights over tokens.The weighted representation is flattened and passed to a classifier, while removing one forwarded token slightly decreases computation.
  • Convolutional Tokenizer: The convolutional tokenizer replaces patch embedding with convolution, ReLU, and max pooling to inject spatial inductive bias.It maintains local spatial information and can use overlapping operations.
  • Convolutional Tokenizer: Convolutional tokenization removes the requirement that input resolution be strictly divisible by a preset patch size.This gives CCT greater input-size flexibility than ViT.

4. Experiments

Experiments evaluate compact transformer variants across small-, higher-resolution, and medium-scale image datasets, including comparisons with CNNs and other transformers. CCT achieves strong accuracy with longer training, compact models, and limited or no pretraining.

  • Datasets: Experiments cover CIFAR-10, CIFAR-100, MNIST, Fashion-MNIST, Flowers-102, ImageNet-1k, and additional NLP classification tasks.The image datasets span small-resolution, higher-resolution, and medium-scale settings.
  • Small-scale training: 98% CIFAR-10 accuracy and 82.87% CIFAR-100 accuracy were achieved by CCT-7/3×1 after 5000 training epochs.The model was also evaluated at 300 and 1500 epochs.
  • ImageNet: ImageNet experiments compare CCT with ResNet50, ViT, and DeiT, including a CCT variant trained with hard knowledge distillation.The comparison includes fairer ResNet50 results using a training schedule similar to the authors’ schedule.
  • ImageNet: Distillation has less effect on CCT than on DeiT, which the authors attribute to inductive biases already provided by CCT’s convolutional tokenizer.The authors connect convolutional tokenization with transferred inductive bias in the student model.
  • Flowers-102: 99.76% top-1 accuracy on Flowers-102 was reported with ImageNet pretraining, while CCT was also effective without pretraining.The result exceeded larger models pretrained on JFT-300M, and CCT used at least a quarter of the next-best model’s size.

5. Conclusion

The conclusion presents CCT as a compact, flexible transformer design for small-data and resource-constrained settings. Ablations indicate that convolutional tokenization reduces reliance on positional encoding while sequence pooling contributes to accuracy.

  • Conclusion: CCT is presented as a transformer that can work in small-data regimes and outperform convolutional models of equivalent or larger sizes.The conclusion also reports reduced computational costs and memory constraints relative to other transformer-based models.
  • Conclusion: The smallest variants can run on a minimal GPU or CPU, supporting research with modest computational resources.The paper frames this capability as relevant to scientific domains with limited data.
  • Appendix analyses: The variants and tokenizers are documented in Tables 5 and 6, with architectural details provided in the appendix.The appendix also includes additional performance analyses.
  • Positional Embedding: CCT relies less on positional encoding than the other tested variants, and removing it can have little effect on accuracy.The ablation also finds that sequence pooling slightly helps in this respect while having a larger effect on total accuracy.

C.2. Performance vs Dataset Size

The dataset-size experiment compares ViT-Lite and CCT across progressively reduced CIFAR-10 training subsets. CCT is more robust in the low-sample regime.

  • Performance vs Dataset Size: CCT achieves higher accuracy than ViT-Lite with fewer CIFAR-10 samples per class, especially in the low-sample regime.The tested subsets contain 500, 1000, 2000, 3000, 4000, or 5000 samples per class.

C.3. Performance vs Dimensionality

CCT outperforms ViT-Lite across CIFAR-10 image sizes from 16×16 to 64×64, with the performance gap widening as pixel count increases.

  • CCT performs better than ViT-Lite at every tested image size from 16×16 to 64×64.The comparison uses downsampled and upsampled CIFAR-10 images.
  • The performance difference between CCT and ViT-Lite widens as the number of pixels increases.
  • CCT better utilizes image information density, whereas ViT-Lite shows no continued performance increase beyond the standard 32×32 size.

D. Dimensionality Experiments

The positional-embedding analysis compares training from scratch with inference from 32×32 pre-trained models, finding CCT less dependent on positional embeddings and more generalizable without them.

  • CCT shows very little training-performance difference when positional embeddings are used versus omitted.Figure 6 reports models trained from scratch.
  • Without positional embeddings, the CCT model has higher inference generalizability across image sizes than ViT-Lite.Figure 7 evaluates models pre-trained on 32×32 images.
  • Learnable positional embeddings prevent ViT-Lite inference on image sizes larger than the pre-trained image.

E. Hyperparameter tuning

The experiments use separate per-dataset hyperparameter sweeps for small-scale, small-resolution settings, with standardized training and optimizer choices for transformer and CNN models.

  • Small-scale, small-resolution experiments used a separate hyperparameter sweep for each model and dataset.
  • Models trained from scratch were trained for 300 epochs unless otherwise noted.
  • ViT, CVT, and CCT used weighted Adam with β1 = 0.9 and β2 = 0.999.
  • CNNs used SGD with momentum 0.9, and the authors planned to release checkpoints and YAML training settings for reproduction.

F. Ablation Study

The ablation study evaluates how convolutional blocks, kernel size, augmentation, and tuning affect compact transformer variants, using results reported as the best of four runs.

  • Table 8 varies convolutional-block count, convolution size, AutoAugment, and minor dropout, attention-dropout, or stochastic-depth tuning.
  • The first Table 8 row is essentially ViT, while the next three rows are more compact modified ViT variants with smaller patch sizes.
  • The reported ablation values are the best results out of four runs.

G. NLP experiments

The paper extends CCT to text classification and evaluates it on five datasets. CCT performs well on small NLP datasets, with the best models remaining substantially smaller than vanilla transformer counterparts.

  • NLP experiments: CCT is evaluated as a general-purpose classifier on five text classification datasets: AGNews, TREC, SST, IMDb, and DBpedia.The results are summarized in Table 10.
  • NLP experiments: CCT outperforms the vanilla transformer across the reported NLP classification results.The authors present this as evidence that their techniques also help with NLP tasks.
  • NLP experiments: The NLP adaptation uses pretrained frozen GloVe embeddings, treats text as single-channel data, and applies size-1 convolution kernels with masking.The embedding dimension is 300, and the word-embedding parameters are excluded from reported model sizes.
  • NLP experiments: Less than 1M parameters characterize the best-performing CCT models on the small NLP datasets.These models are described as significantly smaller than their vanilla counterparts while outperforming them.

H.1. Extended small-scale experiments

The extended experiments provide additional model comparisons and examine validation accuracy across image resolutions and positional-embedding settings. The reported table captions define the evaluation measures and parameter-count conventions.

  • Extended small-scale experiments: Table 10 reports top-1 validation accuracy on text classification datasets, excluding pretrained frozen word-embedding parameters from model-size counts.This convention applies because the word embeddings are pretrained and frozen during training.
  • Extended small-scale experiments: Table 11 presents extended top-1 comparisons, with some models trained longer than the standard setting.The caption directs readers to Table 2 for the longer-training condition.
  • Extended small-scale experiments: Figures 6 and 7 plot CIFAR-10 image resolution against top-1 validation accuracy for training-from-scratch and inference-only settings, respectively.The plotted images are square.
  • Extended small-scale experiments: The positional-embedding comparison includes learnable positional embeddings, with the captioned limitation that they are only possible up to 32x32 without changing weights.The figure labels also include a no-positional-embedding condition.
Loading 2104.05704v4…