PaperlayerView exact revision on arXiv

Source-linked AI summary

Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning

Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, Bohan Zhuang

arXiv:2303.08566v2cs.CVcs.AIcs.LG

TL;DR

Existing visual PEFT methods use heuristic-selected trainable positions across tasks, without accounting for task-specific domain gaps. SPT identifies sensitive positions for each task and allocates a parameter budget across unstructured and structured tuning. Across downstream recognition tasks, it boosts existing PEFT methods, including a 4.2% mean Top-1 accuracy gain over Adapter on FGVC.

  • Problem

    Existing PEFT methods place trainable parameters in the same heuristic-selected positions across tasks, neglecting task-specific domain gaps and characteristics.

    Source: §1 · p. 2

  • Method

    SPT uses a one-pass sensitivity criterion to identify task-specific important parameters and adaptively combines unstructured and structured tuning under a parameter budget.

    Source: §1 · p. 2

  • Results

    Across 24 downstream recognition tasks, SPT boosts existing PEFT methods; it improves Adapter by 4.2% mean Top-1 accuracy and outperforms state-of-the-art PEFT methods on FGVC.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SPT is reported as complementary to existing PEFT methods across varied vision Transformer backbones and pre-training strategies.

    Source: §1 · p. 2

    SPT-LoRA has slightly higher fine-tuning memory than full fine-tuning and LoRA because it updates unstructured parameters with sparse gradients.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Visual Parameter-Efficient Fine-Tuning (PEFT) has become a powerful alternative for full fine-tuning so as to adapt pre-trained vision models to downstream tasks, which only tunes a small number of parameters while freezing the vast majority ones to ease storage burden and optimization difficulty. However, existing PEFT methods introduce trainable parameters to the same positions across different tasks depending solely on human heuristics and neglect the domain gaps. To this end, we study where to introduce and how to allocate trainable parameters by proposing a novel Sensitivity-aware visual Parameter-efficient fine-Tuning (SPT) scheme, which adaptively allocates trainable parameters to task-specific important positions given a desired tunable parameter budget. Specifically, our SPT first quickly identifies the sensitive parameters that require tuning for a given task in a data-dependent way. Next, our SPT further boosts the representational capability for the weight matrices whose number of sensitive parameters exceeds a pre-defined threshold by utilizing existing structured tuning methods, e.g., LoRA [23] or Adapter [22], to replace directly tuning the selected sensitive parameters (unstructured tuning) under the budget. Extensive experiments on a wide range of downstream recognition tasks show that our SPT is complementary to the existing PEFT methods and largely boosts their performance, e.g., SPT improves Adapter with supervised pre-trained ViT-B/16 backbone by 4.2% and 1.4% mean Top-1 accuracy, reaching SOTA performance on FGVC and VTAB-1k benchmarks, respectively. Source code is at https://github.com/ziplab/SPT

1. Introduction

Visual PEFT reduces the storage burden of full fine-tuning, but existing methods place trainable parameters in the same heuristic-selected positions across tasks. SPT identifies task-specific sensitive parameters and allocates a budget across unstructured and structured tuning, improving existing PEFT methods on downstream recognition benchmarks.

1. Introduction

  • Full fine-tuning stores a separate parameter instance for each task and deployment scenario, making storage costs grow with the number of cases.A single ViT-H model requires at least 2.3GB, compared with 2.2GB collectively for the Top-10 U.S. apps in May 2021.Source: §1 · p. 1
  • Existing PEFT methods use the same heuristic-selected trainable positions across tasks, neglecting task-specific domain gaps and characteristics.PEFT tunes a small parameter subset while freezing most parameters, reducing storage burden and optimization difficulty.Source: §1 · p. 2
  • SPT estimates task-specific parameter sensitivity from loss reduction using a first-order Taylor approximation in one forward and backward pass.Sensitivity calculation for ViT-B/16 takes 5.5 seconds on a single GPU for a VTAB-1k dataset.Source: §1 · p. 2
  • SPT directly tunes sensitive connections unless a weight matrix exceeds a sensitivity threshold, when structured tuning replaces unstructured tuning under a similar budget.Structured tuning can use existing methods such as LoRA or Adapter to boost representational capability.Source: §1 · p. 2
  • SPT improves Adapter by 4.2% mean Top-1 accuracy and outperforms state-of-the-art PEFT methods on FGVC.Experiments cover 24 downstream recognition tasks using plain and hierarchical vision Transformers under supervised and self-supervised pre-training.Source: §1 · p. 2

2. Related Work

Related PEFT methods either add trainable parameters to a backbone or reparameterize updates into it, while transfer-learning research selects task-suitable data, models, or weights. SPT differs by identifying task-specific positions before fine-tuning and combining fine- and coarse-grained allocation.

2. Related Work

  • Addition-based PEFT adds trainable parameters to the backbone, while reparameterization-based PEFT tunes parameters that are already in or can be merged into it.LoRA [23] optimizes two low-rank matrices that can be merged into weight matrices.Source: §2 · pp. 2–3
  • Task-specific transfer work selects pre-training data, models, or weights based on their relationship to the target task.Prior approaches select weights through policy networks, sparse masks, or learned binary gates, [15], [66].Source: §2 · p. 3
  • Unlike prior adaptive weight selection, SPT identifies important positions before fine-tuning with one forward and backward pass and uses fixed binary masks for insensitive gradients.SPT combines fine-grained unstructured and coarse-grained structured tuning, and is described as complementary to existing PEFT methods.Source: §2 · p. 3

3. Method

SPT measures task-specific parameter sensitivity, then allocates a fixed trainable-parameter budget across sensitive weights using unstructured or structured tuning. Its sensitivity scores use target-task loss changes approximated with a one-step update, while structured tuning replaces direct tuning in matrices with many sensitive parameters.

3. Method

  • SPT is organized in two stages: measure task-specific sensitivity, then allocate trainable parameters to important positions under the given budget.Directly tuning the top-τ connections uses a binary mask to update only the selected weights.Source: §3 · p. 3, §3.2 · p. 4

3.1. Task-specific Parameter Sensitivity

  • SPT scores each pretrained parameter by the target-task empirical-risk change from tuning it, using a one-step gradient update to approximate the required weight change.The method accumulates sensitivity over training samples before fine-tuning; 400 samples suffice for reasonable sensitivity estimates in 5.5 seconds on one GPU for VTAB-1k with ViT-B/16.Source: §3.1 · pp. 3–4

3.2. Adaptive Trainable Parameters Allocation

  • SPT selects the top-τ sensitive parameters for direct tuning, but switches to structured tuning in sensitive matrices that exceed a threshold to preserve the parameter budget.For LoRA, the threshold is set so its trainable-parameter count is no greater than the number of sensitive parameters in that matrix.Source: §3.2 · pp. 4–5
  • The method combines unstructured and structured tuning granularities, with structured tuning reported as important for tasks with larger domain gaps.The authors report that both granularities contribute to SPT's superior performance.Source: §3.2 · p. 5

4. Experiments

The experiments compare SPT with PEFT baselines across recognition tasks, backbones, and pre-training strategies. They examine sensitivity-based allocation, structured and unstructured tuning, computational costs, and task-specific sensitivity patterns.

4.1. Experimental Setup

  • Experiments evaluate SPT on 24 downstream tasks—five FGVC datasets and 19 VTAB-1k classification tasks—using ViT-B/16 with supervised, MAE, or MoCo v3 pre-training and supervised Swin-B; baselines include addition-based and reparameterization-based PEFT.Source: §4.1 · p. 5

4.2. Main Results

  • SPT variants outperform PEFT baselines across supervised and self-supervised ViT settings; with supervised ViT-B/16, SPT-ADAPTER exceeds NOAH by 0.9% mean top-1 accuracy and improves ADAPTER-8 and LORA-8 by 4.0% and 3.3% on FGVC.Results also extend to Swin-B, ConvNeXt-Base, and semantic segmentation; SPT variants outperform baselines on the reported tasks.Source: §4.2 · pp. 5–7, Table 1 · p. 6, Table 2 · p. 6

4.3. Ablation Study

  • Combining structured and unstructured tuning at task-specific positions achieves the highest performance across all dataset groups; structured tuning is especially important on Structured datasets with larger domain gaps.Applying the sensitivity criterion with structured methods also yields gains for PROMPT-DEEP, ADAPTER-32, and LORA-16.Source: §4.3 · p. 7, Figure 3 · p. 8, Table 6 · p. 8
  • Sensitivity estimation is robust to the sample count: 400 of 800 training images suffice, and calculating sensitivity takes 5.5 seconds on one GPU for ViT-B/16.SPT-LORA has slightly higher fine-tuning memory than full fine-tuning and LORA-16 because it updates unstructured parameters with sparse gradients; omitting unstructured tuning offers similar memory to LoRA and 0.8% higher VTAB-1k Top-1 accuracy.Source: §4.3 · p. 8, Table 8 · p. 8, Table 7 · p. 8

4.4. Observations on Sensitivity Patterns

  • Under a 0.4M parameter budget, sensitive-parameter proportions vary markedly by task across network depth, while query and key matrices have the lowest proportions across the three sampled tasks.The authors speculate that pre-trained patch relationships can be reused for downstream classification despite domain changes.Source: §4.4 · p. 9, Figure 4 · p. 9

5. Conclusion

SPT identifies task-specific sensitive parameters and allocates tuning under a desired parameter budget by combining unstructured and structured tuning. Experiments across 24 downstream recognition tasks show that it complements existing PEFT methods and improves their performance.

5. Conclusion

  • SPT measures task-specific parameter sensitivity and adaptively combines unstructured and structured tuning under a desired trainable-parameter budget.The paper describes this allocation strategy as providing flexibility and representational capability.Source: §5 · p. 9
  • Experiments across 24 downstream recognition tasks use plain and hierarchical vision Transformer backbones and different pre-training strategies.Source: §5 · p. 9
  • SPT is complementary to existing PEFT methods and improves their performance, while future work will consider dense prediction, vision-and-language tasks, and training efficiency for on-device use.Source: §5 · p. 9

Appendix

The supplementary material provides further contender details, sensitivity visualizations, dataset samples, and per-task results for SPT variants on two benchmarks.

Appendix

  • Section 6 provides additional details about the contender methods.Source: Appendix · p. 12
  • Section 7 presents more sensitivity patterns, and Section 8 shows sampled images from ImageNet and VTAB-1k.Source: Appendix · p. 12
  • Tables 9 and 10 report per-task results for SPT variants on FGVC and VTAB-1k, respectively.Source: Appendix · p. 12

6. More Details of Contenders

The appendix defines the compared tuning contenders by which backbone parameters they freeze and what trainable components they add.

6. More Details of Contenders

  • FULL tunes all backbone and classification-head parameters, while LINEAR freezes the backbone and tunes only a linear head.Source: §6 · p. 12
  • BIAS, PARTIAL-k, and MLP-k freeze most or all backbone parameters while tuning bias terms, the last k layers, or a k-layer classification MLP, respectively.Source: §6 · p. 12
  • PROMPT-SHALLOW and PROMPT-DEEP add trainable prompts at the input or within each ViT block, respectively.Source: §6 · p. 12
  • ADAPTER-k adds down-projection, ReLU, and up-projection layers in each block's FFN, whereas ADAPTFORMER adds a concurrent branch there.Source: §6 · p. 12
  • LORA-k adds two low-rank matrices to attention weight matrices, while NOAH searches configurations combining prompts, adapters, and LoRA modules.Source: §6 · p. 12

7. More Parameter Sensitivity Patterns

Additional ViT-B/16 visualizations show sensitivity patterns across datasets and pre-training strategies, with variation both across tasks and by pre-training type.

7. More Parameter Sensitivity Patterns

  • Sensitivity proportions vary by dataset across network depth but show similar patterns across datasets by operation type.Source: §7 · p. 12
  • Across 19 downstream tasks, self-supervised pre-trained backbones have higher sensitivity variance than the supervised pre-trained backbone.Source: §7 · p. 12
  • The sensitivity variance of MAE-pre-trained ViT-B/16 is twice that of supervised-pretrained ViT-B/16.Source: §7 · p. 12
  • The authors speculate that identifying task-specific positions may help SPT variants handle the larger sensitivity variances of self-supervised backbones.Source: §7 · p. 12

8. Dataset Samples for the Source and Target Domains

Figure 10 contrasts ImageNet samples with VTAB-1k domains: Natural-task samples look more similar to ImageNet than Structured-task samples. The authors connect this pattern to larger domain gaps in Structured tasks and speculate that structured tuning helps mitigate them.

8. Dataset Samples for the Source and Target Domains

  • Natural-task samples are relatively more similar to ImageNet than Structured-task samples, aligning with the observation that Structured tasks have larger domain gaps.Source: §8 · p. 12, Figure 10 · p. 16
  • Structured tuning improves Structured-dataset performance, and the authors speculate that it helps mitigate those larger domain gaps.Source: §8 · p. 12
  • Figure 10 shows samples from ImageNet and VTAB-1k Natural and Structured tasks.Source: Figure 10 · p. 16
Loading 2303.08566v2…