PaperlayerView exact revision on arXiv

Source-linked AI summary

FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer

Shibo Jie, Zhi-Hong Deng

arXiv:2212.03145v2cs.CV

TL;DR

Large vision transformers remain costly to customize because conventional fine-tuning requires storing full models, motivating more storage-efficient PETL. FacT tensorizes ViT weights and decomposes their increments into trainable factors, achieving competitive benchmark performance with very small parameter budgets and stronger few-shot results than PETL baselines.

  • Problem

    Full fine-tuning stores an entire model for each downstream task, motivating PETL methods that reduce the parameters required to adapt large pre-trained ViTs.

  • Method

    FacT tensorizes ViT weights into a shared tensor and decomposes weight increments into lightweight factors that are tuned while the pre-trained backbone remains frozen.

  • Results

    FacT matches NOAH on VTAB-1K with 19% of its parameters, uses 8K parameters for an 85.8M-parameter ViT while outperforming full fine-tuning, VPT, and BitFit, and beats PETL baselines in few-shot learning.

  • Takeaways & Limitations

    FacT demonstrates that PETL storage efficiency can be improved by reducing both intra-weight and inter-weight redundancy in weight increments.

  • Takeaways & Limitations

    The approach assumes rank redundancy in pre-trained ViT weights and excludes irregular, few-in-number components such as patch embedding, normalization, and bias parameters from tensorization.

Abstract

from arXiv · show

Recent work has explored the potential to adapt a pre-trained vision transformer (ViT) by updating only a few parameters so as to improve storage efficiency, called parameter-efficient transfer learning (PETL). Current PETL methods have shown that by tuning only 0.5% of the parameters, ViT can be adapted to downstream tasks with even better performance than full fine-tuning. In this paper, we aim to further promote the efficiency of PETL to meet the extreme storage constraint in real-world applications. To this end, we propose a tensorization-decomposition framework to store the weight increments, in which the weights of each ViT are tensorized into a single 3D tensor, and their increments are then decomposed into lightweight factors. In the fine-tuning process, only the factors need to be updated and stored, termed Factor-Tuning (FacT). On VTAB-1K benchmark, our method performs on par with NOAH, the state-of-the-art PETL method, while being 5x more parameter-efficient. We also present a tiny version that only uses 8K (0.01% of ViT's parameters) trainable parameters but outperforms full fine-tuning and many other PETL methods such as VPT and BitFit. In few-shot settings, FacT also beats all PETL baselines using the fewest parameters, demonstrating its strong capability in the low-data regime.

Introduction

FacT targets the storage burden of adapting large pre-trained ViTs by decomposing weight increments into lightweight factors. It achieves competitive or stronger performance with substantially fewer trainable parameters.

  • Full fine-tuning requires storing an entire customized model for each downstream task, creating prohibitive storage costs as vision models grow.
  • PETL reduces storage overhead by tuning only a small subset of parameters, including added structures, selected ViT parameters, or low-rank weight-increment factors.
  • FacT tensorizes ViT weights and jointly decomposes their increments, allowing factors to be shared across weight matrices and updated while the backbone remains frozen.
  • After fine-tuning, FacT stores only the classification head and learned factors rather than complete downstream models.
  • FacT-TK matches NOAH with 19% of its parameters, while FacT-TT adapts an 85.8M-parameter ViT using only 8K parameters and outperforms full fine-tuning, VPT, and BitFit.
  • The framework includes Tensor-Train and Tucker variants, and experiments report competitive VTAB-1K performance and stronger few-shot results than PETL baselines.

Related Work

Related work reduces adaptation or model storage through parameter-efficient tuning and tensor decomposition, but these approaches target different objects and exploit different redundancies. FacT combines these ideas by compressing weight increments rather than pre-trained weights.

  • PETL adapts frozen pre-trained models by tuning small parameter subsets, through added structures or selective updates to existing parameters.
  • Adapters add bottleneck modules around feed-forward blocks, with sequential or parallel designs that transform and merge intermediate features.
  • LoRA represents query and value weight increments with low-rank matrix products, while VPT inserts trainable prompts either throughout the network or only at its input.
  • NOAH combines existing PETL methods using a trained supernet followed by architecture search over adapter width, LoRA rank, and prompt length.
  • Tensor decomposition approximates tensors with low-rank factors and has been used to compress convolutional, recurrent, and transformer networks.
  • Model compression reduces the whole model, whereas PETL reduces trainable parameters while retaining pre-trained weights; FacT therefore compresses weight increments.
  • ViT’s consistent transformer-layer dimensions enable tensorization into a single tensor, illustrated as a 12L × d × d representation.

Method

FacT tensorizes ViT Transformer weights into one 3D tensor and decomposes weight increments into trainable factors, reducing the storage needed for downstream adaptation. The framework supports Matrix-Batch, Tensor-Train, and Tucker formats while keeping most ViT parameters frozen.

  • Tensorizing Vision Transformer: A ViT’s Transformer-layer weights can be stacked into a single 12L × d × d tensor because its layers have consistent structure.The classification head, patch embedding, normalization, and biases are excluded from this tensorized format.
  • Factor-Tuning: a Unified Perspective: Traditional fine-tuning stores a dense weight increment or fine-tuned model for each task, creating O(Ld^2) storage overhead.The increment is defined as ∆W = Wft − W0, where W0 is the pretrained tensorized ViT.
  • Factor-Tuning: a Unified Perspective: The framework supports Matrix-Batch, Tensor-Train, and Tucker decompositions, with LoRA corresponding to the Matrix-Batch case.FacT-TT and FacT-TK optimize Tensor-Train and Tucker factors, respectively.
  • Factor-Tuning: a Unified Perspective: With the same rank r, FacT-TK uses fewer trainable parameters than FacT-TT, while LoRA is substantially larger than both.Figure 4 compares trainable-parameter counts as a function of decomposition rank.
  • Factor-Tuning: a Unified Perspective: On VTAB-1K, FacT-TK≤32 outperforms previous PETL methods while using significantly fewer trainable parameters.The reported average accuracy and parameter counts are averaged over group-wise values.

Experiments

Experiments evaluate FacT across VTAB-1K and fine-grained few-shot datasets, showing competitive or state-of-the-art performance with substantially fewer trainable parameters. Under extreme storage constraints, tiny FacT variants remain competitive with or outperform several baselines.

  • VTAB-1K: FacT-TT≤16 and FacT-TK≤32 use 37K and 69K trainable parameters, respectively, versus NOAH’s 361K, while outperforming NOAH on 11 of 19 VTAB-1K tasks.They also achieve new state-of-the-art results on 7 of 19 tasks.
  • VTAB-1K: FacT-TT≤16 and FacT-TK≤32 achieve new state-of-the-art results on 7 of 19 VTAB-1K tasks.NOAH additionally trains a large supernet for 500 epochs, making FacT more efficient in training.
  • VTAB-1K: FacT-TT and FacT-TK show no clear advantage over each other, although Tucker has a higher compression ratio.The results suggest that further compression beyond FacT-TT does not produce obvious improvement.
  • Extreme storage constraints: FacT-TT4 and FacT-TK8 use only 8K and 14K parameters, or 0.01% and 0.02% of ViT-B’s 85.8M parameters, under extreme storage constraints.Their trainable-parameter counts are comparable in magnitude to linear probing, while performance exceeds full fine-tuning and VPT.
  • VTAB-1K: FacT achieves state-of-the-art results on Natural and Specialized VTAB-1K groups but underperforms NOAH on Structured tasks.The methods remain much more lightweight and efficient than the other baselines.
  • Fine-Grained Few-Shot Learning: FacT-TT achieves state-of-the-art average accuracy on five fine-grained few-shot datasets while using the fewest trainable parameters.It performs best across all settings on four of five datasets, slightly trailing NOAH on Food-101 at 8-shot and 16-shot.

FacT for Hierarchical Transformers

FacT extends to hierarchical transformers by tensorizing each stage separately, addressing varying hidden dimensions. On Swin-B, FacT-TT16 outperforms other PETL baselines while retaining parameter efficiency; ablations favor MHSA tensorization and ranks around 16.

  • FacT for Hierarchical Transformers: FacT uses partitioned tensorization for hierarchical models whose hidden dimension varies across layers but remains consistent within each stage.Each stage is individually tensorized into a single tensor before decomposition.
  • FacT for Hierarchical Transformers: FacT-TT16 outperforms other PETL methods on VTAB-1K with Swin-B while using fewer parameters than VPT-Deep and BitFit.The experiment uses Swin-B pre-trained on supervised ImageNet21K.
  • Ablation Analyses: Tensorizing MHSA performs better than tensorizing FFN overall across Natural, Specialized, and Structured groups.This suggests MHSA blocks are more important than FFN blocks for downstream transfer in these ablations.
  • Ablation Analyses: Tensorizing all blocks performs best on Specialized and Structured groups, while MHSA-only tensorization is slightly better on Natural at larger ranks.The results compare All, MHSA-only, and FFN-only tensorization strategies across ranks 4, 8, 16, and 32.
  • Ablation Analyses: Average performance improves with rank, but FacT-TT becomes inefficient at very large ranks because the size of factor Σ grows with r^2.Performance is nearly saturated when r ≥16, so larger ranks are unnecessary.

Conclusion

FacT is presented as a tensorization-decomposition framework that reduces both intra- and inter-weight redundancy by updating and storing only increment factors. It achieves competitive VTAB-1K performance with fewer parameters and outperforms PETL baselines in few-shot learning.

  • Conclusion: FacT reduces intra- and inter-weight redundancy by tensorizing ViT weights and factorizing their increments.The framework updates and stores only the factors while adapting the model.
  • Conclusion: FacT achieves competitive VTAB-1K results with significantly fewer parameters and outperforms all PETL baselines on few-shot learning.The paper presents FacT as a promising framework for improving PETL storage efficiency.

Datasets

The evaluation uses VTAB-1K, a benchmark spanning 19 visual classification datasets grouped into Natural, Specialized, and Structured categories. Each dataset provides 1,000 training samples, with final evaluation using test-set top-1 accuracy.

  • Datasets: Each VTAB-1K dataset contains 1,000 training samples, and experiments report top-1 accuracy on test sets.The 1,000 samples are split into 800 training and 200 validation samples during hyper-parameter tuning, then recombined for final training.

Pre-trained Backbones

The paper's pre-trained backbones include ViT and Swin Transformer models, with referenced ImageNet21K checkpoint files for the evaluated backbones.

  • Pre-trained Backbones: The referenced pre-trained backbones use ImageNet21K checkpoints for ViT-B/16 and Swin-Base.The passage provides storage links for the ViT and Swin checkpoint files.
  • Pre-trained Backbones: The paper extends FacT beyond ViT to Swin Transformer as a representative hierarchical backbone.The extension addresses ViT's limited visual inductive bias and supports evaluation on hierarchical transformers.

Code Implementation

Experiments use standardized image preprocessing, augmentation, and hyperparameter settings, with baseline implementations reproduced from official codebases. For fair comparison with NOAH, the study retains normalized inputs despite finding that non-normalized inputs improve average VTAB-1K accuracy by about 1–2%.

  • Implementation: All experiments are implemented with PyTorch and timm and run on NVIDIA RTX 3090 GPUs.
  • Data preprocessing: Experiments resize images to 224 × 224 and normalize them with ImageNet’s mean and standard deviation.
  • Evaluation setting: 1–2% average VTAB-1K accuracy gains are observed with non-normalized inputs, but normalized inputs are retained for fair comparison with NOAH.
  • Data preprocessing: Training uses color-jitter and RandAugmentation, while validation and test images are center-cropped from 256 × 256 to 224 × 224 before normalization.
  • Hyperparameters: The scalar s is searched over {0.01, 0.1, 1, 10, 100}, with other hyperparameters largely following Zhang, Zhou, and Liu (2022).
  • Baseline reproduction: Baseline results are reproduced using the official VPT and NOAH codebases, with Adapter using Xavier initialization to avoid blocked gradient propagation.
Loading 2212.03145v2…