PaperlayerView exact revision on arXiv

Source-linked AI summary

AdapterTune: Zero-Initialized Low-Rank Adapters for Frozen Vision Transformers

Salim Khazem

arXiv:2603.14706v1cs.CVcs.AIcs.LG

TL;DR

Frozen-backbone ViT transfer needs both stable adapter optimization and principled capacity selection. AdapterTune uses zero-initialized low-rank residual adapters and a rank-based feature-shift analysis; across its benchmark, it improves over head-only tuning everywhere and beats full fine-tuning in many settings.

  • Problem

    Frozen-backbone transfer lacks stable adapter insertion and principled guidance for choosing adapter capacity, while head-only tuning can underfit task-specific shifts.

  • Method

    AdapterTune inserts zero-initialized low-rank residual adapters into frozen Vision Transformer blocks and formalizes rank as a capacity budget for approximating downstream feature shifts.

  • Results

    AdapterTune improves over head-only tuning on every dataset-backbone pair and outperforms full fine-tuning on 10 of 15 core settings while updating less than 1% of model parameters.

  • Takeaways & Limitations

    The method provides a parameter-efficient, theory-grounded approach for adapting frozen Vision Transformers across multiple datasets and architectures.

  • Takeaways & Limitations

    The analysis is limited when the target task requires a genuinely high-rank shift, where moderate-rank adapters may underperform full fine-tuning.

Abstract

from arXiv · show

Frozen-backbone transfer with Vision Transformers faces two under-addressed issues: optimization instability when adapters are naively inserted into a fixed feature extractor, and the absence of principled guidance for setting adapter capacity. We introduce AdapterTune, which augments each transformer block with a residual low-rank bottleneck whose up-projection is zero-initialized, guaranteeing that the adapted network starts exactly at the pretrained function and eliminates early-epoch representation drift. On the analytical side, we formalize adapter rank as a capacity budget for approximating downstream task shifts in feature space. The resulting excess-risk decomposition predicts monotonic but diminishing accuracy gains with increasing rank, an ``elbow'' behavior we confirm through controlled sweeps. We evaluate on 9 datasets and 3 backbone scales with multi-seed reporting throughout. On a core 5 dataset transfer suite, AdapterTune improves top-1 accuracy over head-only transfer by +14.9 points on average while training only 0.92 of the parameters required by full fine-tuning, and outperforms full fine-tuning on 10 of 15 dataset-backbone pairs. Across the full benchmark, AdapterTune improves over head-only transfer on every dataset-backbone pair tested. Ablations on rank, placement, and initialization isolate each design choice. The code is available at: https://github.com/salimkhazem/adaptertune

1 Introduction

AdapterTune adapts frozen Vision Transformers with lightweight residual adapters, preserving the pretrained function at initialization while addressing the limited flexibility of head-only tuning. Its rank-controlled design is theoretically motivated and evaluated across datasets and backbones with reproducible multi-seed experiments.

  • Full fine-tuning is expensive for repeated downstream adaptation, whereas head-only tuning is cheap but can underfit task-specific representation shifts.
  • AdapterTune trains low-rank residual adapters and a classification head while keeping the pretrained Vision Transformer backbone frozen.
  • Zero-initializing the adapter up-projection makes the initial adapted network exactly equal to the pretrained model, improving optimization stability.
  • The rank analysis models adapters as approximations of low-rank feature-space task shifts and predicts monotonic but saturating gains as rank increases.
  • +14.9 points in average top-1 accuracy over head-only tuning was achieved on the core benchmark while training 0.92% of full fine-tuning parameters.

2 Related Work

Prior transfer methods trade off expressivity, inference structure, sequence complexity, and parameter cost. AdapterTune targets frozen-backbone vision transfer while adding a rank-based account of the capacity required for feature shifts.

  • Full fine-tuning remains the dominant adaptation protocol, while this work studies permanently frozen backbones with lightweight updated adapters.
  • Vision adapter methods place, reparameterize, or search adapter modules in different ways, whereas AdapterTune zeroes the actual up-projection to guarantee zero initial output.
  • LoRA factorizes additive weight updates for attention matrices, while AdapterTune inserts residual modules without modifying backbone weights additively at inference.
  • Prompt tuning adds learnable tokens but increases attention complexity quadratically and can alter the forward pass affecting positional encodings.
  • If the required feature shift has rank r*, ranks below r* incur tail-eigenvalue approximation error, while ranks at least r* incur no further approximation loss.

3 Method

AdapterTune applies shared low-rank residual adapters within a frozen ViT, initialized so the adapter path is initially zero, and trains the adapters with a linear classification head. Placement can be reduced to every other block with nearly unchanged reported accuracy, while the parameter formulas make rank and model scale explicit.

  • 3.1 Preliminaries: The encoder is a pretrained ViT with fixed parameters, and the adapter is applied identically across all tokens using shared weights.
  • 3.2 Residual Adapter Module: The adapter uses a low-rank bottleneck with down-projection, GELU activation, and up-projection, then adds its output residually to the block representation.
  • 3.2 Residual Adapter Module: Zero-initializing the up-projection and its bias makes the adapter output zero for every input, preserving the pretrained forward pass at initialization.
  • 3.2 Residual Adapter Module: Every-block placement uses L adapters, while every-k placement uses floor(L/k) modules; on CIFAR-10/ViT-S, every other block differed by less than 0.1 points.
  • 3.4 Trainable Parameter Count: Each rank-r adapter contributes 2rd + r + d trainable parameters, and total adapter-plus-head parameters equal L times that count plus Cd.
  • 3.4 Trainable Parameter Count: Adapter training uses well under 1.5% of full fine-tuning parameters across the evaluated backbones.
  • 3.5 Training Objective and Protocol: The objective is cross-entropy from a linear classification head over the adapted encoder, with frozen backbone parameters and trainable adapter parameters.

4 Theoretical Analysis

AdapterTune models downstream adaptation as approximating a low-rank feature shift, yielding an excess-risk trade-off between truncation bias and finite-sample estimation error. The analysis predicts diminishing returns with rank, while its guarantees rely on linearization, low-rank task shifts, and blockwise treatment.

  • 4.1 Setup and Assumptions: AdapterTune’s linearized action is a rank-at-most-r feature shift, with the adapter applying Δ_rh to frozen features h.The bottleneck uses down- and up-projections, while the GELU contributes higher-order terms beyond the first-order approximation.
  • 4.2 Approximation Bound: The rank-r approximation error is bounded by B^2 times the squared singular-value tail beyond rank r.Truncated SVD provides the optimal rank-r approximation in Frobenius norm under the low-rank linearized-shift assumption.
  • 4.2 Approximation Bound: The excess-risk decomposition combines approximation error from the omitted singular-value tail with estimation error that scales as Õ(sqrt(Ldr/n)).Here L is the number of adapted blocks and n is the number of training samples.
  • 4.3 Diminishing Returns with Rank: When singular values decay polynomially with exponent p > 1/2, approximation error decreases sublinearly as O(r^(1/2−p)).The convergent singular-value tail yields the diminishing-returns rate predicted by the corollary.
  • 4.3 Diminishing Returns with Rank: +0.27 points separates ranks 8 and 32 on CIFAR-10/ViT-S, compared with +0.10 points between ranks 32 and 64.This sweep confirms the predicted elbow: larger gains at small rank and diminishing gains at moderate rank.
  • 4.4 Limitations of the Analysis: The analysis is limited by linearization, the requirement that the target shift be low-rank, and its independent treatment of transformer blocks.The paper does not claim the bound is tight in nonlinear regimes and leaves cross-block error propagation for future work.

5 Experiments

AdapterTune is evaluated across diverse datasets, backbones, training regimes, ranks, placements, and initialization choices. It consistently improves over head-only transfer, often matches or exceeds full fine-tuning with far fewer trainable parameters, while showing diminishing rank returns and specific weaknesses under large domain shifts.

  • Experimental setup: AdapterTune is evaluated on 9 datasets and 3 pretrained backbone scales using three adaptation regimes, a shared 50-epoch recipe, deterministic splits, and 3 random seeds.The benchmark reports top-1 test accuracy as mean ± standard deviation.
  • Main results: +14.7 points is AdapterTune’s average improvement over head-only tuning across every dataset-backbone pair.The reported gains range from +0.6 points on Oxford-IIIT Pet / DeiT-T to +50.8 points on SVHN / DeiT-T.
  • Main results: 10 of 15 settings favor AdapterTune over full fine-tuning, including all CIFAR-100 and Oxford-IIIT Pet configurations.On CIFAR-100 / ViT-B, AdapterTune reaches 91.21% versus 80.65% for full fine-tuning, a +10.6-point difference.
  • Failure cases and honest analysis: Full fine-tuning retains a 1.2–4.6-point lead on SVHN and Food101 with smaller backbones, where large domain shifts exceed the narrow adapter bottleneck’s capacity.The widest deficits occur on DeiT-Tiny; increasing rank from 16 to 64 closes roughly half the gap on the SVHN / ViT-S pair.
  • Rank ablation and theory validation: Rank sweeps show diminishing returns: on CIFAR-10 / ViT-S, gains are +0.27 points from r = 8 to r = 32 but only +0.10 points from r = 32 to r = 64.The broader sweep reports the same elbow across every dataset-backbone pair, while r = 16 remains an efficiency default and r = 32 captures most observable peak accuracy.
  • Extended benchmark: On the extended benchmark, AdapterTune improves over head-only transfer across all backbone scales for Flowers102, ImageNet-R, Tiny-ImageNet, and FGVC-Aircraft.On Flowers102 / ViT-B, it reaches 97.8%, surpassing full fine-tuning by +2.1 points; on ImageNet-R it recovers more than 95% of the full fine-tuning gap.

6 Discussion

AdapterTune performs best when task shifts are moderate and low-rank, while narrow backbones and severe domain gaps expose approximation limits. Its empirical behavior aligns with diminishing-return theory, narrower generalization gaps, and relatively robust hyperparameter choices, although rank selection and task scope remain limitations.

  • When AdapterTune excels: AdapterTune yields its largest gains on moderate domain shifts with low-rank feature requirements, but trails full fine-tuning by 1.9–4.6 points on severe shifts with narrow backbones.A rank-16 bottleneck spans only approximately 8% of DeiT-Tiny’s dimension in the cited failure cases.
  • Theory alignment: Accuracy gains diminish with rank, with gains from r = 8→32 exceeding those from r = 32→64 across every dataset-backbone pair.This pattern matches the predicted r^(1/2−p) decay law.
  • Generalisation: AdapterTune has average train-test gaps of 1.7–2.7%, compared with 11–13% for full fine-tuning and near-zero gaps for Head-Only.The comparison is based on training-versus-test accuracy across dataset-method combinations.
  • Sensitivity: Every tested hyperparameter configuration exceeds 97.4% top-1 accuracy, with less than 0.4 percentage points separating all 27 settings.The sweep varies learning rate, weight decay, and adapter scale on CIFAR-10/ViT-S/16.
  • Practical defaults & Limitations: The theory may loosen under feature-shift saturation, dense prediction may require direct attention updates, and optimal rank currently requires an empirical sweep.The paper recommends r = 16 for efficiency, r = 32 for peak performance, zero-initialization, and every-block placement.

7 Conclusion

AdapterTune adapts frozen Vision Transformers with zero-initialized, low-rank residual adapters and connects adapter rank to downstream feature shifts through a theoretical bound. Across a reproducible benchmark, it outperforms full fine-tuning in most core settings while updating less than 1% of model parameters.

  • 7 Conclusion: AdapterTune adapts frozen Vision Transformers using zero-initialized, low-rank residual adapters.Zero-initialization makes the adapted network match the pretrained function at initialization.
  • 7 Conclusion: Theoretical analysis connects adapter rank to downstream feature shifts and predicts the diminishing returns observed in empirical sweeps.
  • 7 Conclusion: AdapterTune outperformed full fine-tuning on 10 of 15 core settings while updating less than 1% of model parameters.The benchmark spans 9 datasets and 3 architectures with rigorous reproducibility.
Loading 2603.14706v1…