PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning

Dongze Lian, Daquan Zhou, Jiashi Feng, Xinchao Wang

arXiv:2210.08823v3cs.CV

TL;DR

Existing fine-tuning choices trade efficiency against accuracy, motivating a parameter-efficient method that avoids task-dependent inference overhead. The paper proposes SSF, which scales and shifts pre-trained features, and reports stronger benchmark performance with few trainable parameters while merging the added parameters at inference. SSF is evaluated across image-classification, robustness, and out-of-distribution datasets, with input-independent modulation as a design assumption.

  • Problem

    Full fine-tuning is inefficient, linear probing loses accuracy, and methods such as VPT introduce task-dependent parameters or additional inference computation.

  • Method

    SSF fine-tunes scale and shift parameters that linearly modulate deep features from a pre-trained model, using a unified parameter space and re-parameterization for inference.

  • Results

    Across 26 image-classification datasets and 3 robustness and out-of-distribution datasets, SSF achieves state-of-the-art performance with about 0.3M trainable parameters and improves over full fine-tuning by 2.46% on FGVC and 11.48% on VTAB-1k.

  • Takeaways & Limitations

    SSF provides a parameter-efficient fine-tuning approach that is plug-and-play across CNNs, Transformers, and MLPs without additional inference parameters or computational costs.

  • Takeaways & Limitations

    SSF assumes input-independent scale and shift parameters so they represent the distribution of the whole downstream dataset and remain re-parameterizable.

Abstract

from arXiv · show

Existing fine-tuning methods either tune all parameters of the pre-trained model (full fine-tuning), which is not efficient, or only tune the last linear layer (linear probing), which suffers a significant accuracy drop compared to the full fine-tuning. In this paper, we propose a new parameter-efficient fine-tuning method termed as SSF, representing that researchers only need to Scale and Shift the deep Features extracted by a pre-trained model to catch up with the performance of full fine-tuning. In this way, SSF also surprisingly outperforms other parameter-efficient fine-tuning approaches even with a smaller number of tunable parameters. Furthermore, different from some existing parameter-efficient fine-tuning methods (e.g., Adapter or VPT) that introduce the extra parameters and computational cost in the training and inference stages, SSF only adds learnable parameters during the training stage, and these additional parameters can be merged into the original pre-trained model weights via re-parameterization in the inference phase. With the proposed SSF, our model obtains 2.46% (90.72% vs. 88.54%) and 11.48% (73.10% vs. 65.57%) performance improvement on FGVC and VTAB-1k in terms of Top-1 accuracy compared to the full fine-tuning but only fine-tuning about 0.3M parameters. We also conduct amounts of experiments in various model families (CNNs, Transformers, and MLPs) and datasets. Results on 26 image classification datasets in total and 3 robustness & out-of-distribution datasets show the effectiveness of SSF. Code is available at https://github.com/dongzelian/SSF.

1 Introduction

Large pre-trained models make full fine-tuning costly and prone to overfitting, while linear probing is less accurate and other efficient methods add task-dependent complexity. SSF addresses these issues by scaling and shifting deep features, achieving strong accuracy with few trainable parameters and no extra inference parameters.

  • Motivation: Large pre-trained models encourage transfer learning, but full fine-tuning updates every parameter and can overfit small target datasets.It also requires dedicated parameters for each task, increasing storage demands for large models.
  • Motivation: Linear probing tunes only the last head layer but usually performs worse than full fine-tuning, while VPT improves accuracy by inserting learnable prompts.VPT requires task-specific prompt counts and can add redundant computation or reduce accuracy when poorly selected.
  • Results: SSF achieves state-of-the-art performance with about 0.3M average learnable parameters on FGVC and VTAB-1k comparisons using ViT-B/16.The cited comparison covers seven fine-tuning methods across both benchmarks.
  • SSF: SSF scales and shifts deep features from a pre-trained model to address upstream–downstream distribution differences using a unified parameter space.Its scale and shift parameters modulate downstream features toward a discriminative space.
  • Results: SSF improves Top-1 accuracy over full fine-tuning by 2.46% on FGVC and 11.48% on VTAB-1k while using about 0.3M trainable parameters.The reported pairs are 90.72% vs. 88.54% on FGVC and 73.10% vs. 65.57% on VTAB-1k.

2 Related Work

Related work spans convolutional and Transformer architectures, transfer learning with increasingly large models and datasets, feature-modulation methods, and re-parameterization for efficient inference.

  • Model Architectures: CNNs have long served as the primary architecture for image-feature extraction, while Vision Transformers established another major computer-vision architecture family.ViT adapted the Transformer paradigm from NLP to vision and achieved promising results.
  • Transfer Learning: Pre-training followed by downstream fine-tuning is used to improve convergence or performance, with recent work scaling both models and datasets.Examples include ViT and Swin Transformer V2 trained on ImageNet-21K and JFT-300M.
  • Feature Modulation: Normalization methods such as BN, LN, and GN modulate feature distributions through normalization followed by learned scale and shift factors.Other feature-modulation approaches include spatial transformation and adaptive instance normalization.
  • Re-parameterization: Model re-parameterization improves inference efficiency by merging parameters from batch-normalization layers or multiple network branches.This provides the background for merging linear transformations into existing model components.

3 Approach

SSF inserts lightweight scale-and-shift modules after network operations, updates only these factors during fine-tuning, and merges them into existing weights for inference without added parameters or computation.

  • Scaling and Shifting Your Features for Fine-tuning: The scale and shift factors linearly modulate deep features to match the distribution of a target dataset.The factors are input-independent so they represent the distribution of the downstream dataset.
  • Scaling and Shifting Your Features for Fine-tuning: SSF uses a task-independent tunable parameter space and can insert modules after each linear-coefficient operation without requiring task-specific prompt counts.The method is described as supporting independent downstream tasks and later multi-task or continual fine-tuning.
  • Scaling and Shifting Your Features for Fine-tuning: SSF inserts SSF-ADA after operations such as MSA, MLP, and LN while keeping the pre-trained operation weights frozen during fine-tuning.The SSF-ADA parameters are updated instead.
  • Scaling and Shifting Your Features for Fine-tuning: SSF computes the transformed feature as y = γ ⊙ x + β, where γ and β are the scale and shift factors.The output y becomes the input to the next operation.
  • Re-parameterization: SSF re-parameterizes the linear transformation by absorbing γ and β into the preceding layer’s weights and biases.This merging removes extra parameters and computational cost during inference.
  • Complexity Analysis: During training, SSF adds md parameters per layer, whereas Adapter adds 2dd′ and VPT adds nd under the stated dimensional assumptions.In practice, SSF is reported to outperform Adapter and VPT-Deep with slightly fewer training-stage parameters; its inference overhead is zero after merging.

4 Experiments

Experiments across classification, robustness, and OOD benchmarks evaluate SSF with diverse datasets, backbones, baselines, computational measures, and design ablations. SSF generally combines strong accuracy with few trainable parameters, unified task-independent tuning, and zero-overhead inference, while some large-dataset settings remain challenging.

  • Image classification: 11.48% (73.10% vs. 65.57%) Top-1 accuracy improvement over full fine-tuning is reported on VTAB-1k, while SSF uses fewer trainable parameters than VPT-Deep.The reported parameter comparisons are 0.24M vs. 0.60M on VTAB-1k and 0.39M vs. 0.85M on FGVC.
  • Parameter efficiency: SSF maintains a unified learnable parameter space across tasks, whereas VPT requires task-specific prompt counts and careful prompt-number design.The reported performance is sensitive to the number of prompts, while SSF scales and shifts extracted deep features.
  • Design ablations: 93.99% accuracy with 0.28M trainable parameters is obtained when SSF-ADA is inserted into all 12 vision-transformer layers; performance grows nearly linearly with inserted layers.Removing SSF-ADA from MLP operations performs worse than removing it from attention operations at comparable parameter counts.
  • Experimental settings: SSF is evaluated on 26 image-classification tasks and three robustness and out-of-distribution datasets using ViT, Swin Transformer, CNN, and MLP backbones.The main initialization is ViT-B/16 pretrained on ImageNet-21K; additional backbones include Swin-B, ConvNeXt-B, and AS-MLP-B.
  • Robustness and OOD: SSF outperforms other parameter-efficient methods on ImageNet-A, ImageNet-R, and ImageNet-C, despite lower ImageNet-1K accuracy than full fine-tuning.The authors associate this robustness and OOD result with freezing most pretrained parameters and preserving large-scale pretrained knowledge.
  • Visualization and analysis: 93.99% vs. 93.82% accuracy on CIFAR-100 shows SSF can outperform full fine-tuning despite producing a different weight distribution.The analysis reports that scale and shift parameters adjust original weights and biases, while SSF features become most similar to full fine-tuning in the last layer.

5 Conclusion

The paper concludes that SSF scales and shifts features from a pre-trained model, achieving strong parameter-efficient fine-tuning results while avoiding inference-time overhead through re-parameterization.

  • Figure 5 visualizes feature similarities between full fine-tuning and linear probing, VPT-Deep, and SSF across ViT-B/16 layers.
  • SSF scales and shifts features extracted by a pre-trained model for parameter-efficient fine-tuning.
  • 2.46% (90.72% vs. 88.54%) and 11.48% (73.10% vs. 65.57%) Top-1 accuracy improvements over full fine-tuning require about 0.3M trainable parameters.
  • Experiments across 26 image classification datasets, 3 robustness and out-of-distribution datasets, and CNN, Transformer, and MLP families show SSF’s effectiveness.
  • SSF’s scale and shift parameters can be merged into pre-trained weights via re-parameterization, avoiding extra inference parameters and FLOPs.

Checklist

The checklist records affirmative responses for claims, limitations, ethics, reproducibility, compute reporting, and asset attribution, while marking non-applicable items where appropriate.

  • The authors state that the paper’s contributions and scope are described in the abstract, introduction, and experiments.
  • The authors report discussing limitations and potential negative societal impacts in the appendix.
  • The checklist marks theoretical assumptions and proofs as not applicable because the paper reports experimental rather than theoretical results.
  • Code, data, training details, and compute resources are reported as available in the paper or supplementary materials.
  • The authors report addressing error bars where applicable and citing existing assets with their licenses.
  • The checklist reports no human-subject consent, participant-risk, instruction, or compensation documentation was applicable.

A Detailed Descriptions for the Evaluation Datasets

The evaluation covers fine-grained, diverse low-data, and general image-classification datasets, with dataset statistics and splits summarized in Table 8.

  • Table 8 reports statistics for the various evaluation datasets, including train/validation/test splits and classes.
  • FGVC: The FGVC evaluation uses CUB-200-2011, NABirds, Oxford Flowers, Stanford Dogs, and Stanford Cars.
  • VTAB-1k: VTAB-1k contains 19 tasks spanning natural, specialized, and structured images, with only 1,000 training samples per task.
  • General Image Classification Datasets: General image classification evaluation uses CIFAR-100 and ImageNet-1K, containing 60,000 images with 100 categories and 1.28M training images with 1,000 categories, respectively.

A.2 Robustness and OOD

Robustness and out-of-distribution evaluation uses ImageNet-A, ImageNet-R, and ImageNet-C to assess models fine-tuned on ImageNet-1K.

  • ImageNet-A contains real-world adversarial samples from 200 classes selected from ImageNet-1K’s 1,000 classes.
  • ImageNet-R contains 30,000 rendition images covering 200 ImageNet-1K classes.
  • ImageNet-C contains corrupted images involving noise, blur, weather, and other corruption types for robustness evaluation.

A.3 Detection and Segmentation

The paper evaluates fine-tuning methods on COCO and ADE20K for detection, instance segmentation, and semantic segmentation, with metrics defined for each task.

  • COCO and ADE20K provide evaluation settings for object detection, instance segmentation, and semantic segmentation, respectively.APb and APm denote average precision for object detection and instance segmentation; mIoU and MS mIoU denote single-scale and multi-scale semantic-segmentation inference.

B Experiments on Detection and Segmentation

SSF outperforms linear probing and VPT-Deep on detection and segmentation benchmarks, but remains below full fine-tuning on both datasets.

  • Object Detection and Instance Segmentation: SSF outperforms linear probing and VPT-Deep on COCO object detection and instance segmentation.Experiments use Mask R-CNN with a Swin-T ImageNet-1K-pretrained backbone and a 1x, 12-epoch schedule.
  • Semantic Segmentation: SSF outperforms linear probing and VPT-Deep on ADE20K semantic segmentation.The experiments use UperNet with a Swin-T ImageNet-1K-pretrained backbone.
  • Comparison with Full Fine-Tuning: SSF still has a large gap compared to full fine-tuning on both detection and segmentation datasets.The paper suggests this may be due to the characteristics of detection and segmentation tasks, but the supplied passage truncates that explanation.

C.1 Feature Distribution

SSF produces feature distributions and attention patterns that resemble full fine-tuning while improving over linear probing and VPT-Deep in the visualizations.

  • Feature Distribution: SSF achieves better feature clustering than linear probing and VPT-Deep in t-SNE visualizations on CIFAR-100.All methods use ViT-B/16 pretrained on ImageNet-21K.
  • Feature Distribution: 93.99% vs. 93.82% accuracy makes SSF and full fine-tuning difficult to distinguish by feature distribution.The comparison is reported for CIFAR-100.
  • Attention Map: SSF tends to produce attention similar to full fine-tuning, but it also generates a failure prediction in the second row.VPT-Deep concentrates attention on objects in some images but lacks suitable attention in others.

D Limitations and Societal Impacts

The paper’s scope is efficient backbone sharing across independently treated tasks, and it notes deployment and data-use implications of this approach.

  • Limitations: SSF treats tasks independently rather than modeling relationships among multiple tasks during fine-tuning.The authors identify efficient discovery of positive task relationships as a direction for future work.
  • Societal Impacts: SSF saves parameters and avoids changing the deployed backbone architecture by replacing only weights after re-parameterization.The paper associates these properties with easier transfer of large models and savings in computational resources and carbon emissions.
  • Societal Impacts: SSF, like other fine-tuning methods, may raise data-use concerns when its upstream pretrained model was trained on illegal data.The paper frames this as a potential violation of the use of fine-tuning methods.
Loading 2210.08823v3…