PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Training of Visual Transformers with Small Datasets

Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, Marco De Nadai

arXiv:2106.03746v2cs.CVcs.LG

TL;DR

VTs can represent global relations but are more data-hungry than CNNs because they lack CNN inductive biases, and their small-dataset behavior varies despite comparable ImageNet accuracy. The paper compares VTs in scarce-data regimes and adds a self-supervised relative-localization loss that predicts distances between final token embeddings during supervised training. Across extensive evaluations, the loss consistently improves baseline accuracy, sometimes by up to 45 points.

  • Problem

    VTs lack typical CNN inductive biases, and the small-dataset behavior of second-generation VTs remains unclear despite comparable ImageNet results.

  • Method

    The paper compares VTs across scarce-data settings and jointly trains them with cross-entropy and a dense self-supervised loss predicting spatial distances between final token-embedding pairs.

  • Results

    Across 11 datasets, training scenarios, and three VTs, dense localization loss always improved baseline accuracy, sometimes by up to +45 points.

  • Takeaways & Limitations

    Dense relative localization is an easy-to-reproduce tool for improving VT performance when data or training time is limited.

  • Takeaways & Limitations

    The analysis mainly focuses on VTs approximately the size of ResNet-50, while the effect of fine-grained embedding grids remains insufficiently analyzed.

Abstract

from arXiv · show

Visual Transformers (VTs) are emerging as an architectural paradigm alternative to Convolutional networks (CNNs). Differently from CNNs, VTs can capture global relations between image elements and they potentially have a larger representation capacity. However, the lack of the typical convolutional inductive bias makes these models more data-hungry than common CNNs. In fact, some local properties of the visual domain which are embedded in the CNN architectural design, in VTs should be learned from samples. In this paper, we empirically analyse different VTs, comparing their robustness in a small training-set regime, and we show that, despite having a comparable accuracy when trained on ImageNet, their performance on smaller datasets can be largely different. Moreover, we propose a self-supervised task which can extract additional information from images with only a negligible computational overhead. This task encourages the VTs to learn spatial relations within an image and makes the VT training much more robust when training data are scarce. Our task is used jointly with the standard (supervised) training and it does not depend on specific architectural choices, thus it can be easily plugged in the existing VTs. Using an extensive evaluation with different VTs and datasets, we show that our method can improve (sometimes dramatically) the final accuracy of the VTs. Our code is available at: https://github.com/yhlleo/VTs-Drloc.

1 Introduction

VTs model global token relations but lack CNN inductive biases, making them more data-hungry. This paper evaluates their small-data robustness and introduces a self-supervised localization task for regularization.

  • VT attention models global token relations, whereas CNN receptive fields emphasize locality, translation invariance, and hierarchical visual structure.
  • Second-generation VTs mix convolution and attention, but their behavior on medium-small datasets remains unclear despite strong ImageNet results.
  • The study finds that VTs with similar ImageNet accuracy can differ substantially in classification accuracy on smaller datasets.
  • The proposed task samples final-token pairs and predicts their geometric distance, encouraging embeddings to encode local and contextual information.
  • Ldrloc jointly regularizes supervised training without additional annotations and improved tested baselines, sometimes by up to 45 points.
  • The contributions include comparing VT behavior in scarce-data regimes and proposing relative localization as a training-regularization task.

2 Related work

Related work spans visual-transformer architectures, positional representations, and self-supervised learning. The paper distinguishes its dense relative-localization regularizer from fully self-supervised and multi-view approaches.

  • Visual Transformers: Visual Transformers originated with fully transformer-based vision models such as iGPT and ViT, alongside earlier attention mechanisms inside CNNs.
  • Visual Transformers: Second-generation VTs combine convolutions with attention, using geometric token grids to introduce local processing while retaining long-range modeling.
  • Positional embeddings: Positional embeddings provide token-location information because attention and token-wise feed-forward layers are permutation invariant.
  • Self-supervised learning: ELECTRA replaces masked-token prediction with a discriminative task defined over all input tokens, inspiring dense sampling over VT embeddings.
  • Self-supervised learning: Unlike common vision pretext tasks, the proposal predicts relative positions of token pairs from one image and does not require multiple views or positive pairs.

3 Preliminaries

A VT converts an image into a token grid, processes tokens with attention-based layers, and uses a classification head for prediction. Existing architectures are retained when the proposed localization loss is added.

  • A VT splits an image into a K × K patch grid and linearly projects each patch into an input token.
  • Convolutions with stride or pooling can reduce the initial token-grid resolution, producing a final k × k embedding grid with k ≤ K.
  • The final embedding grid supports classification through either a class token or average global pooling, followed by an MLP producing class posteriors.
  • When localization is added, each tested VT keeps its native architecture, including its available class token, pooling layer, and positional embedding.

4 Dense relative localization task

The dense relative localization task trains VTs to predict spatial relationships between randomly sampled token embeddings, using a self-supervised loss added to standard classification training. The task is effective across architectures, but finer embedding grids slow convergence and can produce noisy gradients.

  • Task definition: The method samples embedding pairs from each image’s k × k final-embedding grid and predicts their normalized 2D relative translation offsets.The grid is denoted Gx = {ei,j}, and the task uses multiple randomly sampled pairs per image.
  • Task definition: A small two-layer MLP receives concatenated embedding pairs and predicts the relative distance between their grid positions.Its two output neurons correspond to the two spatial dimensions.
  • Optimization: The relative localization loss averages the L1 error between sampled target offsets and MLP predictions, then combines with cross-entropy as Ltot = Lce + λLdrloc.The loss samples m pairs uniformly for each image and averages their errors; λ is 0.1 for T2T and CvT and 0.5 for Swin.
  • Architectural comparison: Relative positional embeddings are not sufficient to solve the task, since adding Ldrloc to CvT generally produces a smaller relative accuracy boost than adding it to Swin.CvT has no positional embedding, whereas Swin uses relative positional embedding.
  • Architectural comparison: For T2T and CvT, the task uses a 7 × 7 grid because the original 14 × 14 grid caused slow convergence and likely noisy gradients.The authors attribute the difficulty to finer-grid localization and leave deeper analysis for future work.

5 Experiments

The experiments evaluate Visual Transformers across small-data, limited-epoch, from-scratch, and fine-tuning settings, comparing architectures, regularization, and ResNet baselines. Results show that architecture robustness varies substantially on smaller datasets, while Ldrloc consistently improves VT accuracy.

  • Experimental setup: Experiments cover 11 image-classification datasets, comparable-size VT and ResNet-50 models, and training from scratch or ImageNet-1K fine-tuning.The study also varies training duration and evaluates Ldrloc with fixed hyperparameters across datasets and protocols.
  • Training duration: Ldrloc improves accuracy for all tested VTs on IN-100, with larger gains when training uses fewer epochs.The authors interpret this pattern as consistent with a regularizer whose effects are more pronounced in shorter training regimes.
  • Training from scratch: VT accuracy varies far more across small datasets than ImageNet-1K results suggest: CvT and Swin differ by about 45–46 points on Quickdraw and Sketch, and 30 points on CIFAR-10.On ImageNet-1K, the reported top-1 accuracies are Swin-T 81.3, T2T-ViT-14 81.5, and CvT-13 81.6.
  • Training from scratch: Ldrloc improves every tested VT on every dataset in the from-scratch experiments, including almost 4 points on SVHN with CvT and more than 45 points on Quickdraw with Swin.The results indicate that the auxiliary self-supervised task supplies additional training signal when data are limited.
  • ResNet comparison: The best-performing VT is usually comparable with a same-size ResNet, while Ldrloc produces marginal but consistent ResNet gains on 9 of 10 datasets.The smaller ResNet improvement is attributed to convolutional architectures already embedding local inductive biases.
  • Fine-tuning: During fine-tuning, Ldrloc improves all tested VTs on every dataset, including more than 5 points for Swin and more than 7 points for T2T on Infograph.CvT is excluded from this analysis because publicly available ImageNet-1K pre-trained models were unavailable.

6 Conclusion

The paper finds substantial variation among VTs on small-medium datasets and proposes dense relative localization to regularize training. Across 11 datasets, the loss consistently improves baseline accuracy, sometimes by up to 45 points, while the study focuses mainly on approximately ResNet-50-sized models.

  • VT performance varies substantially on small-medium datasets, with CvT usually generalizing more effectively when less data are available.
  • Dense relative localization is a self-supervised auxiliary task that regularizes VT training and encourages learning spatial information from token pairs.
  • Up to +45 points: dense localization loss improved baseline accuracy across 11 datasets, different training scenarios, and three VTs.
  • The analysis mainly studies VTs approximately the same size as ResNet-50, while deeper analysis of fine-grained embedding grids remains future work.

A Pseudocode of the dense relative localization task

The section presents dense relative localization as an easily reproduced auxiliary task and describes several alternative loss implementations. These variants differ in target representation, prediction depth, and loss formulation, while most retain low computational cost.

  • Figure 2 presents PyTorch-like pseudocode for the dense relative localization task and its Ldrloc loss.
  • The first variant replaces the original target offsets with offsets that include negative values while keeping the remaining formulation unchanged.
  • The second variant converts offset regression into classification, predicting separate discrete offset distributions for the two spatial dimensions.
  • A Gaussian-prior variant minimizes normalized squared distance between predicted expectations and ground-truth offsets, with variance regularization weighted by α.
  • The very-dense variant computes localization loss at every transformer block, giving earlier layers accumulated gradients from later blocks.
  • Other proposed variants require one forward and one backward pass per image, plus m forward passes through the localization MLP.

B.1 Empirical comparison of the loss variants

On IN-100 with 100-epoch training, the standard Ldrloc variant outperforms the tested alternatives. The very-dense formulation is the only variant reported to underperform the Swin baseline, plausibly because finer intermediate grids make localization harder.

  • On IN-100, adding Ldrloc to Swin increases final top-1 accuracy by 1.26 points.All models are trained for 100 epochs, with top-1 test accuracy as the metric.
  • All other dense localization variants underperform Ldrloc in the comparison.
  • The very-dense Lall_drloc variant is the only tested variant that underperforms the baseline.
  • The authors presume finer intermediate embedding grids make localization harder, slowing convergence and likely providing noisy gradients.
  • Swin’s relative positional embedding is not sufficient for the auxiliary task because offset information is blended into value weighting rather than directly represented in token embeddings.

C Experiments with a larger training budget

Additional experiments test Ldrloc with longer schedules, larger datasets, and a substantially larger ViT-B/16 model. The loss remains beneficial, although gains on ImageNet-1K are slight and the margin depends on the training setting.

  • On ImageNet-1K with standard 300-epoch training, Ldrloc yields a slight improvement for both Swin and T2T.The experiments use publicly available baseline code and default hyperparameters.
  • The larger-budget experiments indicate that Ldrloc can remain effective with longer schedules and higher-capacity VT models.
  • +8.06 versus +2.62: T2T’s relative Ldrloc margin on Infograph is larger with 300 epochs than with 100 epochs.
  • Training from scratch with ViT-B/16 tests Ldrloc on a model with 86.4 million parameters, about four times the size of the other tested VTs and ResNets.

D Transfer to object detection and image segmentation tasks

The authors test whether Ldrloc-based pre-training transfers to object detection, instance segmentation, and semantic segmentation without changing task-specific architectures or fine-tuning protocols. Across these tasks, Ldrloc improves performance over baselines, with larger relative gains when pre-training uses IN-100.

  • Transfer setup: Ldrloc pre-training transfers to object detection and instance segmentation without modifying the task-specific architecture or fine-tuning protocol.The transfer experiments use Swin backbones with standard detection and segmentation architectures.
  • Object detection and instance segmentation: ImageNet-1K pre-training with Ldrloc achieves higher detection and instance segmentation accuracy than baselines across nearly all reported metrics.This comparison covers both Mask R-CNN and Cascade Mask R-CNN on COCO.
  • Object detection and instance segmentation: IN-100 pre-training produces an even higher relative improvement over baselines for object detection and instance segmentation.The comparison is reported in Tab. 11.
  • Semantic segmentation: Ldrloc-pre-trained models always outperform baselines on all reported semantic segmentation metrics after ImageNet-1K pre-training.The semantic segmentation experiments use UperNet with a Swin backbone on ADE20K.
  • Semantic segmentation: The relative semantic segmentation boost is higher when the models are pre-trained on IN-100 rather than ImageNet-1K.The comparison is reported in Tab. 13.

E Training efficiency

Training curves on CIFAR-100 show that Ldrloc benefits VT optimization throughout training and can accelerate learning. The auxiliary task also adds negligible training-time overhead.

  • Training curves: After 60 epochs, Swin with Ldrloc reaches 55.01 top-1 accuracy, exceeding the baseline’s 53.28 after 100 epochs.The curves cover CvT, Swin, and T2T trained from scratch on CIFAR-100.
  • Training curves: Ldrloc is beneficial over the whole training stage for VTs trained from scratch on CIFAR-100.The reported curves compare baseline training with training using the auxiliary loss.
  • Training overhead: The auxiliary localization task imposes negligible overhead relative to total training time.Training-time comparisons report seconds per batch with batch size 1024.

F Implementation details and an additional ablation study on the localization MLP

The implementation uses a simple localization MLP and integrates multiple VT implementations into a common training framework. An ablation varies the hidden-layer width of the localization head.

  • Localization MLP: The localization MLP has three fully connected layers with ReLU activations in its first two layers.It first projects the concatenated token embeddings into 512 dimensions, then applies a 512-dimensional linear layer before prediction.
  • Localization MLP: For Ldrloc, the MLP prediction layer outputs two values corresponding to d_u and d_v.The output design depends on the loss variant.
  • MLP ablation: The localization MLP ablation studies the number of neurons in each of its two hidden layers during 100-epoch CIFAR-100 training from scratch.The ablation is summarized in Tab. 16.
  • Implementation: The experiments use Swin’s official framework together with official T2T-ViT code and publicly released CvT code.The implementations are inserted into the Swin training framework for comparison.
  • Training protocol: Networks trained from scratch or fine-tuned for 100 epochs use AdamW, cosine decay, linear warm-up, batch size 1024, and weight decay 0.05.The initial learning rate is 0.001 from scratch and 0.0005 for fine-tuning.
Loading 2106.03746v2…