PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Efficient Visual Adaption via Structural Re-parameterization

Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, Rongrong Ji

arXiv:2302.08106v2cs.CV

TL;DR

PETL can reduce adaptation costs for large pretrained vision models, but many methods still incur inference latency. RepAdapter re-parameterizes sequential adapters into pretrained models and uses sparse design and placement. Across 27 benchmarks of three vision tasks, it outperforms most existing PETL methods and is validated across a range of vision models.

  • Problem

    PETL methods reduce the storage burden of adapting large pretrained models, but most still incur non-negligible inference latency.

    Source: §1 · pp. 1–2

  • Method

    RepAdapter inserts lightweight sequential adapters that can be re-parameterized into nearby pretrained projection weights after training.

    Source: §1 · p. 2

  • Results

    Across 27 benchmarks of three vision tasks, RepAdapter outperforms most existing PETL methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The paper validates RepAdapter’s generalization across ConvNeXt, ViT, Swin-Transformer, and CLIP.

    Source: §1 · p. 3

Abstract

from arXiv · show

Parameter-efficient transfer learning (PETL) is an emerging research spot aimed at inexpensively adapting large-scale pre-trained models to downstream tasks. Recent advances have achieved great success in saving storage costs for various pre-trained models by updating a small number of parameters instead of full tuning. However, we notice that most existing PETL methods still incur non-negligible latency during inference. In this paper, we propose a parameter-efficient and computational friendly adapter for giant vision models, called RepAdapter. Specifically, we first prove that common adaptation modules can also be seamlessly integrated into most giant vision models via our structural re-parameterization, thereby achieving zero-cost during inference. We then investigate the sparse design and effective placement of adapter structure, helping our RepAdaper obtain other advantages in terms of parameter efficiency and performance. To validate RepAdapter, we conduct extensive experiments on 27 benchmark datasets of three vision tasks, i.e., image and video classifications and semantic segmentation. Experimental results show the superior performance and efficiency of RepAdapter than the state-of-the-art PETL methods. For instance, RepAdapter outperforms full tuning by +7.2% on average and saves up to 25% training time, 20% GPU memory, and 94.6% storage cost of ViT-B/16 on VTAB-1k. The generalization ability of RepAdapter is also well validated by a bunch of vision models. Our source code is released at https://github.com/luogen1996/RepAdapter.

1. Introduction

PETL reduces adaptation costs by updating a small subset of parameters, but many methods still add inference latency. RepAdapter re-parameterizes sequential adapters into pretrained models and examines sparse design and placement.

1. Introduction

  • Full tuning of giant vision models can impose large storage costs, while PETL methods may slow inference through added tokens or network depth.Fully fine-tuning ViT-G on 19 VTAB-1k tasks requires storing over 35 billion parameters; with batch size 1, ViT latency increases by almost 20% for adapters.Source: §1 · pp. 1–2
  • LoRA avoids additional inference computation through re-parameterization, but the paper reports limited generalization on common vision models and says it cannot re-parameterize common sequential adapters.The cited comparison reports LoRA underperforms full tuning on ConvNeXt and adapters on ViT.Source: §1 · p. 2
  • Linear adaptation modules placed sequentially can be re-parameterized into a feed-forward structure without performance degeneration, keeping the inference network intact while retaining adapter effectiveness.This extends re-parameterization beyond parallel branches to sequential visual adapters.Source: §1 · p. 2
  • RepAdapter’s dense-sparse structure saves 25% of parameters, while sparse design and adapter placement improve parameter efficiency and performance.The introduction also identifies placement as essential for giant vision models.Source: §1 · pp. 2–3
  • Across 27 benchmarks of three vision tasks, RepAdapter outperforms most existing PETL methods and generalizes across ConvNeXt, ViT, Swin-Transformer, and CLIP.The evaluated tasks include image and video classification and semantic segmentation.Source: §1 · pp. 2–3

2. Related Work

Related work describes adapter and prompt-based PETL alongside structural re-parameterization, then distinguishes RepAdapter by its flexibility and ability to re-parameterize sequential structures.

2.1. Parameter-efficient Transfer Learning

  • Vision PETL mainly uses adapters inserted as small MLPs or prompt tuning with trainable tokens; LoRA learns low-rank parameters for frozen attention weights.The subsection also notes normalization-layer adaptation as another approach.Source: §2.1 · p. 3
  • RepAdapter differs from visual adapters in structure and placement, and its re-parameterization is not limited to simple modules such as normalization layers.The paper characterizes its trade-off among performance, efficiency, and generalization as better than these related works.Source: §2.1 · p. 3

2.2. Structural Re-parameterization

  • Existing structural re-parameterization commonly converts multi-branch structures into single-branch structures, as in RepVGG’s merging of convolution branches and an identity layer.Other cited work applies the approach to inception-like units and capacity improvements.Source: §2.2 · p. 3
  • RepAdapter is presented as flexible across parameterized modules such as convolutions and capable of re-parameterizing sequential structures.The authors describe it as a complement to existing structural re-parameterization research.Source: §2.2 · p. 3

3. Methods

The method adapts ViT with lightweight residual adapters, then removes their nonlinearities so sequential adapters can be merged into pretrained weights for inference. RepAdapter further uses grouped projections and empirically selected placements to reduce parameters and improve performance.

3.1. Preliminary

  • RepAdapter uses a bottleneck adapter with a residual connection, deployed either sequentially after the FFN or in parallel with it.The adapter’s downsampling and upsampling projections can be compact, with a hidden size such as 8.Source: §3.1 · p. 4

3.2.1 Sequentially Structural Re-parameterization

  • Removing the adapter nonlinearity does not degrade vision-task performance, allowing sequential adapters to simplify into linear projections and merge into nearby pretrained weights.The merged weights can be incorporated into MHA, FFN, or convolutional modules, avoiding additional inference cost.Source: §3.2.1 · p. 4

3.2.2 Adapter Structure

  • RepAdapter makes its up-projection group-wise, converting it to a dense projection by zero-padding for re-parameterization.With two groups, the sparse design saves about 25% of parameters compared with common visual adapters.Source: §3.2.2 · p. 5
  • RepAdapter is empirically more effective before neural modules, and applying it to both MHA and FFN is more beneficial in ViT.Four possible insertion locations are considered; the final deployments are marked in dark orange.Source: §3.2.2 · p. 5, Figure 4 · p. 5

4. Experiments

Across image and video classification and semantic segmentation, experiments evaluate RepAdapter on 27 datasets and find strong performance, efficiency, and generalization. Ablations examine design choices, while comparisons test inference speed and transfer across settings and architectures.

4.1. Datasets and Metrics

  • Evaluation covers image classification, video classification, and semantic segmentation: top-1 accuracy is reported for classification tasks, while mIoU is reported on ADE20K validation.The evaluation uses 19 VTB-1k datasets, ImageNet and three ImageNet domain-generalization sets, Something-Something V2 and HMDB51, and ADE20K.Source: §4.1 · p. 5

4.2. Implementation Details

  • The default setup uses ViT-B/16 for image classification, VideoMAE-pretrained ViT-B/16 for video, and ImageNet-21k-pretrained ViT-L/14 for segmentation.RepAdapter is inserted before MHA and FFN by default; the lighter RepAdapterattn variant is inserted only before MHA.Source: §4.2 · p. 5

4.3.1 Comparisons with the State-of-the-arts.

  • RepAdapter improves on state-of-the-art PETL methods on VTAB-1k; using it before both MHA and FFN raises average performance from 75.5 to 76.1, and it exceeds LoRA by +1.6% average accuracy.RepAdapterattn achieves state-of-the-art performance with 0.11M parameters; RepAdapter exceeds NOAH by +0.6%.Source: §4.3.1 · p. 6

4.3.2 Ablation Studies

  • On VTAB-1k, sparse structure and new placement improve ViT-B/16 accuracy by +0.6% and +1.0%, respectively; increasing groups can improve performance and efficiency, while more parameters do not always help.Pre-insertion beats post-insertion; a fully sparse structure loses performance, indicating the importance of RepAdapter’s dense component for information exchange.Source: §4.3.2 · pp. 6–7

4.3.3 Efficiency Analysis

  • At batch size 1, Adapter is 22.5% slower than RepAdapter despite adding only 0.03 GFLOPs, illustrating that GPU latency also depends on network topology.The reported efficiency comparison measures inference speed in images per second on a NVIDIA 3090 GPU and averages 100 trials.Source: §4.3.3 · p. 7, Table 4 · p. 7

4.3.4 Generalization Experiments

  • On CLIP, visual- and text-encoder RepAdapter improve over zero-shot CLIP on source and target datasets; RepAdapter-T exceeds CoCoOp using a handcrafted prompt, and RepAdapter gains +5% over full tuning on ConvNeXt.The study also evaluates CLIP on 16-shot ImageNet and domain generalization, and tests additional architectures on VTAB-1k.Source: §4.3.4 · p. 7, Table 5 · p. 8, Table 6 · p. 8
  • On video classification, RepAdapter-16 exceeds AdaptFormer-64 by +3.52% on HMDB51 while using more than 50% fewer parameters; on ADE20K, it beats VPT+Bias by +1.08 mIoU-MS with fewer parameters.VPT performs below full tuning on SSv2, while AdaptFormer’s best performance uses 1.26M parameters.Source: §4.3.4 · p. 8, Table 7 · p. 8, Table 8 · p. 8

5. Conclusions

The paper concludes that RepAdapter offers parameter-efficient transfer learning for giant vision models, with parameters mergeable into the pretrained model for zero extra inference cost. Experiments across 27 datasets and three vision tasks support its efficiency, performance, and generalization.

5. Conclusions

  • RepAdapter’s parameters merge completely into pretrained vision models through structural re-parameterization, adding zero inference cost.The authors attribute its effectiveness over existing PETL approaches to its sparse structure and careful deployment.Source: §5 · p. 8
  • Experiments across 27 datasets in image classification, video classification, and semantic segmentation confirm the method’s reported efficiency, performance, and generalization.Source: §5 · p. 8
  • The work proposes RepAdapter as a PETL method for giant vision models.Source: §5 · p. 8

Related papers

Loading 2302.08106v2…