PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition

Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, Ping Luo

arXiv:2205.13535v3cs.CV

TL;DR

Adapting a single pretrained ViT to many visual tasks while minimizing task-specific tuning remains challenging. AdaptFormer adds lightweight, trainable modules to frozen ViTs, and experiments show comparable or better performance than full tuning across scalable image and video recognition tasks.

  • Problem

    Adapting a pretrained model to varied downstream tasks requires tuning as few parameters as possible while keeping the remaining parameters frozen.

  • Method

    AdaptFormer introduces AdaptMLP, a lightweight plug-and-play module that adapts pretrained ViTs by tuning small bottleneck modules while freezing most shared parameters.

  • Results

    AdaptFormer achieves comparable or better performance than full tuning across five image and video recognition datasets, with consistently higher accuracy as input frames increase.

  • Takeaways & Limitations

    AdaptFormer supports scalable adaptation across diverse vision recognition tasks while adding less than 2% of the pretrained model’s parameters.

  • Takeaways & Limitations

    The design assumes that effective ViT adaptation should focus on MLPs and use parallel feature integration rather than modifying attention mechanisms.

Abstract

from arXiv · show

Pretraining Vision Transformers (ViTs) has achieved great success in visual recognition. A following scenario is to adapt a ViT to various image and video recognition tasks. The adaptation is challenging because of heavy computation and memory storage. Each model needs an independent and complete finetuning process to adapt to different tasks, which limits its transferability to different visual domains. To address this challenge, we propose an effective adaptation approach for Transformer, namely AdaptFormer, which can adapt the pre-trained ViTs into many different image and video tasks efficiently. It possesses several benefits more appealing than prior arts. Firstly, AdaptFormer introduces lightweight modules that only add less than 2% extra parameters to a ViT, while it is able to increase the ViT's transferability without updating its original pre-trained parameters, significantly outperforming the existing 100\% fully fine-tuned models on action recognition benchmarks. Secondly, it can be plug-and-play in different Transformers and scalable to many visual tasks. Thirdly, extensive experiments on five image and video datasets show that AdaptFormer largely improves ViTs in the target domains. For example, when updating just 1.5% extra parameters, it achieves about 10% and 19% relative improvement compared to the fully fine-tuned models on Something-Something~v2 and HMDB51, respectively. Code is available at https://github.com/ShoufaChen/AdaptFormer.

1 Introduction

The paper targets efficient adaptation of pretrained vision Transformers across many visual tasks without maintaining a separate fully tuned model for each task. AdaptFormer addresses this by tuning small task-specific modules while sharing most pretrained weights, and reports strong downstream performance.

  • Motivation: Vision models with task-specific weights become difficult to maintain as the number of tasks and model capacity grow.The passage highlights ViT-G/14 with over 1.8 billion parameters as an example.
  • Approach: Less than 2% of parameters are tuned while over 98% remain shared across tasks, unlike full tuning with separate task-specific weights.
  • Motivation: The main adaptation challenge is tuning as few parameters as possible while keeping the remaining pretrained parameters frozen across downstream tasks.
  • Results: On SSv2 with a ViT-Base backbone, AdaptFormer surpasses full tuning using only 0.2% tunable parameters.
  • Approach: AdaptFormer adapts vision Transformers to diverse visual recognition tasks while avoiding catastrophic interference between tasks.
  • Results: Experiments across varied downstream tasks report that AdaptFormer significantly outperforms existing fine-tuning approaches.

2 Related Works

The related-work discussion situates AdaptFormer among vision Transformers and efficient transfer-learning methods. It emphasizes the dominance of full tuning in vision and the limited exploration of parameter-efficient adaptation for vision Transformers.

  • Vision Transformers: Vision Transformers have expanded from image classification to detection, segmentation, video understanding, point clouds, and 3D recognition.
  • Efficient transfer learning: Transfer learning reuses a pretrained model as the starting point before fine-tuning it for a new task.
  • Efficient transfer learning: Traditional downstream adaptation typically makes all network parameters learnable, while efficient adaptation has received comparatively little attention in computer vision.
  • Efficient transfer learning: Visual Prompt Tuning adapts ViTs by prepending learnable tokens while freezing the pretrained backbone.

3 Approach

AdaptFormer adapts pretrained vision Transformers by replacing their MLP blocks with lightweight, task-specific bottleneck branches while freezing the original model parameters. The module is designed for scalable image and video recognition and can be inserted into multiple Transformer architectures.

  • Architecture: AdaptFormer denotes the vision Transformer equipped with the proposed plug-and-play AdaptMLP bottleneck module.The module is introduced for efficient fine-tuning of pretrained vision Transformer models.
  • Architecture: AdaptFormer replaces each Transformer MLP block with AdaptMLP, combining the original frozen MLP branch with a trainable bottleneck branch.The bottleneck uses down-projection, ReLU, and up-projection layers, connected to the original branch through a residual connection and scale factor s.
  • Fine-tuning: During fine-tuning, only newly added parameters are optimized, while the pretrained model components remain frozen.At inference, task-specific extra-parameter weights are loaded alongside the shared frozen parameters, allowing one overall model to support multiple tasks.
  • Tunable parameters analysis: The AdaptMLP adds fewer than 2% of the pretrained model parameters, so total model size grows slowly as downstream tasks are added.Its per-layer parameter count is 2 × d × ˆd + ˆd + d, including biases, with a bottleneck dimension ˆd much smaller than d.
  • Applicability: AdaptMLP can be inserted into multiple vision Transformer architectures because they share similar MLP structures despite differing attention mechanisms.The paper also states that AdaptMLP can be applied to attention-free models, while prompt-based methods may be less straightforward for specialized attention variants.
  • Discussion: AdaptFormer targets scalable recognition across image and video domains with limited learnable parameters and comparable or better performance than full tuning.The design emphasizes MLP adaptation and parallel feature combination for domain-specific features.

4 Experiments

Experiments evaluate AdaptFormer across image and video recognition tasks, compare it with common adaptation baselines, and analyze parameter scaling and architectural choices. AdaptFormer generally improves performance while updating only a small parameter subset.

  • Experimental setup: AdaptFormer is evaluated on image and video recognition tasks using pre-trained ViT backbones, with linear probing, full fine-tuning, and VPT as baselines.The experiments cover downstream datasets in both visual domains and compare several fine-tuning strategies.
  • Main results: AdaptFormer-64 outperforms VPT by 3.46%, 2.87%, and 4.63% on CIFAR-100, SVHN, and Food-101, respectively.On Something-Something V2, it exceeds VPT by about 15% and full fine-tuning by about 5% Top-1 accuracy.
  • Parameter scaling: VPT performance is task-dependent: accuracy rises through 300K tunable parameters on SSv2 but drops beyond 50K on HMDB-51.VPT training also collapses around the tenth epoch when the token count reaches 8 or more, whereas AdaptFormer optimization remains stable over the tested settings.
  • Main results: AdaptFormer maintains a favorable multi-label classification trade-off, reducing updated parameters from 85.86 to 1.25M while achieving slightly lower mAP than fine-tuning.It also has an advantage over linear probing and VPT in this evaluation.
  • Ablation studies: Increasing AdaptFormer’s middle dimension improves SSv2 accuracy up to saturation near 64, while even dimension one reaches about 50.03% top-1 accuracy.Across datasets, the optimal middle dimension varies, motivating a trade-off between performance and parameter count.
  • Video experiments: With eight input frames, AdaptFormer exceeds linear fine-tuning by 30% and VPT by 14% top-1 accuracy.Increasing the number of frames benefits all three evaluated fine-tuning methods.
  • Video experiments: Using an ImageNet-21k pre-trained model for SSv2 action recognition, AdaptFormer reaches 46.06% top-1 accuracy, exceeding full fine-tuning by 4.56%.Linear probing reaches 6.56%, while VPT reaches 16.94% in the same setting.

5 Conclusion

AdaptFormer efficiently adapts pretrained ViT backbones to scalable vision recognition tasks through lightweight modules, with experiments across image and video datasets supporting improved transferability at little computational cost. The paper limits its scope to recognition tasks and leaves detection and segmentation for future study.

  • AdaptFormer uses AdaptMLP to fine-tune lightweight modules that produce features adapted to multiple downstream vision recognition tasks.The pretrained Vision Transformer backbone is adapted without requiring a complete task-specific model description here.
  • Experiments on five datasets spanning image and video domains validate improved ViT transferability with little computational cost.
  • AdaptFormer is evaluated only on recognition tasks, so its effectiveness for object detection and semantic segmentation remains unclear.

Checklist

The checklist reports that the paper addresses its contributions, limitations, societal impacts, reproducibility materials, training details, compute resources, and asset licensing. It also records that error bars were not applicable and human-subject procedures were not applicable.

  • The paper reports that its main claims, contributions, scope, and limitations are accurately described, with limitations shown in the conclusion.
  • The paper reports discussing potential negative societal impacts and states that the ethics review guidelines were followed.
  • The authors provide code, data, reproduction instructions, and training details, with the code URL shown in the abstract and details in supplementary materials.
  • Error bars and participant-related procedures are marked not applicable in the checklist.
  • The paper reports compute resources in Section 4.1 and states that existing assets were cited and their licenses documented in supplementary materials.
  • The datasets are publicly available for research use and are reported to contain no personally identifiable information or offensive content.

A Appendix

The supplementary material covers implementation details and additional evaluations intended to clarify AdaptFormer’s training and applicability.

  • The supplement includes pretraining and fine-tuning details, experiments on hierarchical vision transformers such as AdaptFormer-Swin, and PyTorch-like AdaptMLP pseudocode.

A.1.1 Pre-training Approaches

The paper uses MAE-based self-supervised checkpoints for image and video pretraining, with masking strategies tailored to each domain.

  • For images, MAE masks nearly 75% of input patches and reconstructs missing pixels; the study uses a ViT-B/16 checkpoint pretrained on ImageNet-1K for 800 epochs.
  • For videos, VideoMAE extends MAE with joint space-time attention and uses a 90% to 95% masking ratio; the study uses a checkpoint pretrained on Kinetics-400.

A.1.2 Implementation Details of Fine-tuning

Fine-tuning uses shared configurations and separate settings for image and video experiments, with SGD and linear learning-rate scaling specified for optimization.

  • Fine-tuning settings separate shared configurations from experiment-specific configurations for image and video tasks.Table 5 presents shared optimizer and base learning-rate settings in its upper part and separated settings below.
  • SGD is used for stable training on small-scale datasets such as CIFAR10.The implementation differs from prior work using LARS by selecting SGD.
  • The learning rate follows linear scaling as lr = base_lr×batchsize / 256.This rule scales the learning rate according to batch size relative to 256.
  • Image and video experiments follow MAE and VideoMAE settings, respectively.An affine-free BatchNorm layer is inserted before the final fully connected layer, and video fine-tuning uses no flip augmentation.

A.2.1 AdaptFormer with Supervised Pre-training

The experiments evaluate AdaptFormer across supervised pre-training, Swin Transformers, adapter designs, parameter dimensions, computational efficiency, and video pre-training choices. Results generally show strong transfer with few trainable parameters, while performance depends on architecture, dataset overlap, and attention type.

  • AdaptFormer with Supervised Pre-training: With supervised pre-training, AdaptFormer surpasses full-tuning on four benchmarks using only 1.46% tunable parameters.The four benchmarks are CIFAR100, SVHN, SSv2, and HMDB51; on Food-101 it is nearly comparable to full-tuning at 90.89% versus 90.96%.
  • AdaptFormer on Swin Transformer: AdaptFormer-Swin outperforms full-tuning on image benchmarks with only 1.43% of parameters.It also significantly surpasses linear probing, particularly on SSv2, supporting transfer across vision Transformer variants.
  • ImageNet-1K Evaluation: ImageNet-1K fine-tuning with ImageNet-21K pre-training is considered unreasonable for cross-dataset adaptation because ImageNet-1K is a subset of ImageNet-21K.The paper documents these experiments for completeness despite the dataset overlap.
  • Middle Dimension Ablation: 82.33% top-1 accuracy is achieved by AdaptFormer-1 on ImageNet-1K, the best result among tested middle dimensions.Increasing the middle dimension to 4, 16, or 64 produces progressively larger performance drops, with -0.47% at 64.
  • ImageNet and Kinetics Pre-training: For joint spatiotemporal attention, Kinetics-400 pre-training yields higher top-1 accuracy than ImageNet pre-training, 53.8 versus 52.0.The paper therefore uses joint spatiotemporal attention and considers Kinetics pre-training preferable for its video experiments.
  • Adapter Design: AdaptMLP uses a frozen original MLP alongside tunable down-projection, ReLU, and up-projection layers.The added module is designed as a plug-and-play replacement for the original MLP.
Loading 2205.13535v3…