PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Bypasses Are Better Vision Transformer Adapters

Shibo Jie, Zhi-Hong Deng

arXiv:2207.07039v3cs.CV

TL;DR

Full finetuning of increasingly large ViTs creates storage demands, while existing PETL modules lack visual inductive bias. The paper proposes convolutional bypasses that adapt pretrained ViTs with few trainable parameters, and reports stronger performance than language-oriented PETL methods across visual and low-data evaluations.

  • Problem

    Existing ViT PETL methods lack visual inductive biases such as spatial locality and 2D neighborhood structure, while full finetuning is storage-inefficient for increasingly large models.

  • Method

    Convpass inserts trainable convolutional bottleneck blocks parallel to ViT's MHSA or MLP blocks and updates only these modules plus the classification head during finetuning.

  • Results

    Convpass outperforms previous PETL methods overall, achieving the best average performance and a 1.1% improvement over NOAH on the evaluated tasks.

  • Takeaways & Limitations

    The results support tailoring PETL modules to visual inductive biases when adapting vision models, particularly in low-data regimes.

Abstract

from arXiv · show

The pretrain-then-finetune paradigm has been widely adopted in computer vision. But as the size of Vision Transformer (ViT) grows exponentially, the full finetuning becomes prohibitive in view of the heavier storage overhead. Motivated by parameter-efficient transfer learning (PETL) on language transformers, recent studies attempt to insert lightweight adaptation modules (e.g., adapter layers or prompt tokens) to pretrained ViT and only finetune these modules while the pretrained weights are frozen. However, these modules were originally proposed to finetune language models and did not take into account the prior knowledge specifically for visual tasks. In this paper, we propose to construct Convolutional Bypasses (Convpass) in ViT as adaptation modules, introducing only a small amount (less than 0.5% of model parameters) of trainable parameters to adapt the large ViT. Different from other PETL methods, Convpass benefits from the hard-coded inductive bias of convolutional layers and thus is more suitable for visual tasks, especially in the low-data regime. Experimental results on VTAB-1K benchmark and few-shot learning datasets show that Convpass outperforms current language-oriented adaptation modules, demonstrating the necessity to tailor vision-oriented adaptation modules for adapting vision models.

1 Introduction

Vision Transformer scaling makes full finetuning storage-intensive, motivating parameter-efficient adaptation. The paper argues that vision-oriented modules with convolutional inductive bias better suit visual transfer, especially with limited data.

  • 1 Introduction: 1843M parameters in ViT-G versus 58M in ResNet-152 illustrates the exponential growth driving parameter-efficient transfer learning for ViT.Full finetuning also requires storing a whole model for each downstream task.
  • 1 Introduction: Existing PETL methods freeze pretrained weights and finetune lightweight modules, but the paper questions whether language-designed modules are optimal for vision models.These methods were originally developed for pretrained language models and later applied to ViT.
  • 1 Introduction: Visual tasks benefit from spatial locality and 2D neighborhood structure, whereas language-oriented modules imply weaker spatial relations and variable-length input support.The paper labels these alternatives vision-oriented and language-oriented modules, respectively.
  • 1 Introduction: Downstream datasets are often limited or domain-specific, so adaptation modules should introduce additional inductive bias to improve data efficiency.Weakly biased models may need substantial data to learn visual structure from scratch.
  • 1 Introduction: 0.33M new parameters for an 86M-parameter ViT-B lets Convpass outperform full finetuning and language-oriented methods on the 19-task VTAB benchmark.The method updates only Convpass modules and the classification head during finetuning.
  • 1 Introduction: Convpass outperforms previous language-oriented methods, supporting the paper’s claim that ViT adaptation modules should reflect visual-task characteristics.The contribution summary identifies tailoring adaptation modules to vision models as necessary.

2 Related Work

This section reviews ViT architecture and language-oriented PETL methods, then motivates analyzing their path structures. These methods include adapters, low-rank updates, prompt tokens, and architecture search over their combinations.

  • 2 Related Work: ViT partitions images into patches, embeds them into 1D token sequences, and processes them with MHSA and MLP blocks.MHSA projects tokens into queries, keys, and values; MLP uses two fully connected layers with GELU.
  • 2 Related Work: Adapters use a bottleneck MLP with h << d and add its transformed output to the input through a residual connection.The residual computation is X′ ← X + φ(XW_down)W_up.
  • 2 Related Work: AdaptFormer is a parallel Adapter design for adapting MLP blocks, whereas the sequential Adapter placement follows the MLP blocks.The parallel formulation uses a scaling hyperparameter s.
  • 2 Related Work: LoRA learns low-rank approximations of updates to attention projections, adding scaled low-rank products to the query and key computations.Its factors have rank r << d.
  • 2 Related Work: VPT concatenates trainable prompt tokens with the input before each layer and removes those prompts at the layer’s end.The next layer receives newly concatenated prompts.
  • 2 Related Work: NOAH combines Adapter, LoRA, and VPT while searching over Adapter dimension, LoRA rank, and prompt length.Its search is performed through neural architecture search.
  • 2 Related Work: Although VPT and NOAH target vision, their components are fundamentally ported from NLP, so the paper classifies these PETL methods as language-oriented.BitFit and Sidetune are cited as methods that perform poorly on ViT.

3 Methodology

The methodology analyzes adapter paths in ViT and proposes convolutional bypasses that restore spatial structure and inject visual inductive bias while remaining storage-efficient.

  • 3.1 Rethinking Adapters from an Unraveled View: Unraveling ViT exposes frozen paths alongside trainable paths, clarifying how adapters alter the pretrained network.Adapter-equipped ViTs can be viewed as ensembles of frozen, MHSA-Adapter, and Adapter-MHSA paths.
  • 3.1 Rethinking Adapters from an Unraveled View: Language-oriented PETL methods adjust MHSA, whose limited visual inductive bias can be problematic when downstream visual data is scarce.LoRA and VPT tune MHSA-related components, while convolution supplies spatial locality and 2D neighborhood structure.
  • 3.2 Adapting ViT via Convolutional Bypasses: Convpass inserts parallel convolutional bottleneck bypasses, restoring token sequences to 2D and processing image tokens separately from the [cls] token.The module uses 1×1 channel reduction, 3×3 convolution, and 1×1 channel expansion, with ResNet-style residual structure.
  • 3.2 Adapting ViT via Convolutional Bypasses: Convpass(-attn) achieves 12 SOTA results among PETL methods across the 19-task VTAB-1K benchmark.The benchmark reports average results across Natural, Specialized, and Structured groups.
  • 3.2 Adapting ViT via Convolutional Bypasses: Convpass creates trainable paths containing convolution alone or convolution with MHSA, yielding transformer, ResNet-like CNN, and hybrid models.Because every trainable path contains Convpass, adaptation can use convolutional visual inductive bias and 2D neighborhood structure.
  • 3.2 Adapting ViT via Convolutional Bypasses: O(Ld) trainable parameters keep Convpass storage-efficient compared with ViT’s O(Ld^2) parameters when h << d.The paper gives d = 768 and h = 8 as an experimental example.

4 Experiments

Experiments evaluate Convpass across standard transfer learning, few-shot recognition, domain generalization, backbone comparisons, and module-placement designs. Across these settings, Convpass generally improves performance, with strongest benefits where visual inductive bias and data efficiency matter.

  • Transfer Learning: Convpass achieves the best average performance on VTAB-1K, exceeding previous SOTA PETL methods by 1.1%.Convpassattn also surpasses NOAH with half as many parameters, while Convpass is more training-efficient because NOAH trains an additional supernet.
  • Transfer Learning: Convpass has the best performance in all three VTAB groups, with especially significant advantages on Natural and Structured tasks.Its advantage is less pronounced on Specialized tasks, where it does not remarkably outperform NOAH and AdaptFormer.
  • Few-Shot Learning: Convpass leads the average results across all five few-shot settings, with highlighted advantages on FGVC-Aircraft and Stanford Cars.On Food101, NOAH is slightly better at 16 shots, but Convpass leads as training data decreases; simpler datasets show similar performance across methods.
  • Domain Generalization: ConvpassCLIP outperforms CLIP-specific baselines on the source domain and achieves SOTA performance on three of four target domains.It performs worse on ImageNet-A, plausibly because that dataset selects samples misclassified by ResNet.
  • Comparison with Other Backbones: With traditional transfer learning, Swin-B and ConvNeXt-B outperform ViT-B/16, but Convpass-equipped ViT-B/16 surpasses fully finetuned versions of both backbones on average.Applying Convpass to Swin preserves an advantage over full finetuning, although its improvement diminishes because Swin already has stronger visual inductive bias.
  • Module Placement and Design: Parallel Convpass designs outperform sequential designs, and placing modules alongside MHSA performs better than placing them alongside MLP.The final Convpass combines the best parallel MHSA- and MLP-side designs; replacing 3×3 convolutions degrades performance across VTAB groups.

5 Conclusion

The paper identifies insufficient visual inductive bias in current ViT PETL methods and proposes Convpass, a convolutional adaptation approach. Experiments show Convpass outperforms other PETL methods and has notable domain generalization.

  • Convpass adapts pretrained ViTs through trainable convolutional bypasses designed for visual tasks.
  • Convpass outperforms other parameter-efficient transfer learning methods on VTAB-1K and few-shot learning benchmarks.
  • The results indicate that visual-task characteristics should inform the design of PETL methods for ViT.

A Datasets

The experiments report dataset statistics for the datasets used in the study, including a benchmark of dataset configurations summarized in Table 6.

  • The paper reports validation results for ImageNet-1K because its test set is unavailable.
  • Table 6 summarizes statistics for the datasets used in the experiments.

B.2 Code Implementation

All experiments were implemented with PyTorch and used NVIDIA RTX3090 GPUs, with models based on timm.

  • The experiments use PyTorch and NVIDIA RTX3090 GPUs.
  • The models are implemented based on timm.

B.3 Data Augmentation

The experiments apply ImageNet normalization and task-specific resizing, cropping, flipping, color-jitter, and RandAugmentation procedures across training and evaluation data.

  • Images are resized to 224 × 224 and normalized with ImageNet’s mean and standard deviation.
  • Few-shot training samples use color-jitter and RandAugmentation.
  • Few-shot validation and test samples are resized to 256 × 256, center-cropped to 224 × 224, and normalized with ImageNet statistics.
  • Domain-generalization training uses random resizing and cropping plus random horizontal flipping, while validation and test images are resized to 224 × 224.

B.4 Hyperparameters

The hyperparameter section specifies the search space for s and refers readers to Table 8 for additional hyperparameters.

  • s is searched from {0.01, 0.1, 1, 10, 100}.
  • Additional hyperparameters are listed in Table 8.
  • Table 8 presents the paper’s hyperparameters.

B.5 Results of Adapter

The reported Adapter–VPT comparison depends on initialization: zero-initializing both Adapter layers blocks gradient backpropagation, whereas the authors’ Xavier-weight, zero-bias setup yields Adapter outperforming VPT.

  • Adapter significantly underperforms VPT under Jia et al.’s reported implementation.
  • Zero-initializing the weights of both Adapter FC layers blocks gradient backpropagation.
  • Using Xavier-initialized weights and zero-initialized biases, the authors report that Adapter outperforms VPT.
Loading 2207.07039v3…