PaperlayerView exact revision on arXiv

Source-linked AI summary

5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks

Dongshuo Yin, Leiyi Hu, Bin Li, Youqun Zhang, Xue Yang

arXiv:2408.08345v2cs.CV

TL;DR

Existing visual delta-tuning methods had not exceeded full fine-tuning on challenging recognition tasks. Mona adds multi-filter visual adapters, and the paper reports that it outperforms full fine-tuning across representative visual tasks, including a 1% AP gain on COCO instance segmentation.

  • Problem

    Existing visual delta-tuning methods had not outperformed full fine-tuning on challenging visual recognition tasks such as semantic and instance segmentation.

    Source: Introduction · pp. 1–2

  • Method

    Mona is an adapter-based tuning method that uses vision-friendly filters to process pretrained visual features from multiple cognitive perspectives.

    Source: Introduction · p. 2

  • Results

    Mona outperforms full fine-tuning across representative visual tasks and gains 1% AP over it on COCO instance segmentation.

    Source: Introduction · p. 2, Main Results · p. 5

  • Takeaways & Limitations

    The results suggest that full fine-tuning is no longer the optimal choice for representative visual tasks.

    Source: Main Results · p. 5, Conclusion · p. 7

    Adapter modules, including Mona, add structures and slightly increase inference cost.

    Source: Discussion · p. 7

Abstract

from arXiv · show

Pre-training & fine-tuning can enhance the transferring efficiency and performance in visual tasks. Recent delta-tuning methods provide more options for visual classification tasks. Despite their success, existing visual delta-tuning art fails to exceed the upper limit of full fine-tuning on challenging tasks like object detection and segmentation. To find a competitive alternative to full fine-tuning, we propose the Multi-cognitive Visual Adapter (Mona) tuning, a novel adapter-based tuning method. First, we introduce multiple vision-friendly filters into the adapter to enhance its ability to process visual signals, while previous methods mainly rely on language-friendly linear filters. Second, we add the scaled normalization layer in the adapter to regulate the distribution of input features for visual filters. To fully demonstrate the practicality and generality of Mona, we conduct experiments on multiple representative visual tasks, including instance segmentation on COCO, semantic segmentation on ADE20K, object detection on Pascal VOC, oriented object detection on DOTA/STAR, and image classification on three common datasets. Exciting results illustrate that Mona surpasses full fine-tuning on all these tasks, and is the only delta-tuning method outperforming full fine-tuning on the above various tasks. For example, Mona achieves 1% performance gain on the COCO dataset compared to full fine-tuning. Comprehensive results suggest that Mona-tuning is more suitable for retaining and utilizing the capabilities of pre-trained models than full fine-tuning. The code will be released at https://github.com/Leiyi-Hu/mona.

Introduction

Full fine-tuning has been widely used in vision, but existing delta-tuning methods have not surpassed it on challenging recognition tasks. Mona introduces visual adapters with vision-friendly filters and reports stronger performance than full fine-tuning across representative visual tasks.

Introduction

  • Full fine-tuning updates the pretrained backbone and task-specific heads or necks, motivating interest in more efficient transfer methods.Source: Introduction · p. 1
  • Existing delta-tuning methods can match or exceed full fine-tuning on simpler tasks but have not surpassed it on semantic or instance segmentation.Source: Introduction · pp. 1–2
  • Mona uses vision-friendly filters to replace traditional linear adapters and process visual signals through multiple cognitive perspectives.The paper describes convolution-based filters and multiple filters following the adapter reduction layer.Source: Introduction · p. 2
  • Mona-tuning outperforms full fine-tuning across representative visual tasks, including image classification, object detection, semantic and instance segmentation, and oriented object detection.The paper reports a 1% mAP gain over full fine-tuning on COCO.Source: Introduction · p. 2

Related Work

The related work describes three broad delta-tuning approaches and reviews their development in computer vision, where methods have not consistently replaced full fine-tuning.

Delta-tuning

  • Delta-tuning methods either update a small subset of pretrained parameters, reparameterize pretrained weights, or add trainable structures while fixing the backbone.The paper compares Mona with methods from all three groups.Source: Delta-tuning · p. 2

Computer Vision Meets Delta-tuning

  • Visual delta-tuning progressed from prompt tuning and adapters for classification to LoRand for dense prediction, but had not surpassed full fine-tuning across visual recognition tasks.Mona is presented as an alternative that outperforms full fine-tuning in performance and new-parameter size.Source: Computer Vision Meets Delta-tuning · p. 2

Methods

Mona tuning freezes pre-trained parameters while learning adapter parameters, adding input-distribution adjustment and visual convolutional filters to address limitations of linear adapters. Its parameter count is controlled by fixing the adapter’s down-projection dimension at 64.

Methods

  • Mona is inserted after MSA and MLP in each SwinBlock; pre-trained layers remain fixed while Mona parameters are updated.The adapter includes scaled LayerNorm, convolutional filters, an aggregation filter, and four internal skip-connections.Source: Figure 2 · p. 3

Adapter-tuning

  • Adapter-tuning fixes pre-trained parameters and updates adapter parameters, unlike full fine-tuning, which updates all backbone parameters.Adapter-tuning may also update parameters outside the backbone; θF denotes fixed parameters and ω denotes updated parameters.Source: Adapter-tuning · p. 3

Mona

  • Mona adjusts the input distribution with LayerNorm and learnable scaling, then processes features using averaged 3×3, 5×5, and 7×7 depth-wise filters and 1×1 aggregation.The design addresses biased inputs from fixed layers and the limits of linear filters for visual signals; it also uses skip-connections.Source: Mona · p. 3, Figure 2 · p. 3

Parameter Analysis

  • Each Mona module has (2n + 3)m + n^2 + 84n + 2 parameters, with n fixed at 64 to reduce parameter count.Here m is the input dimension and n is the dimension after down projection.Source: Parameter Analysis · p. 4

Experiments

The experiments evaluate Mona on representative visual recognition tasks, with dataset and metric choices varying by task. They include horizontal and oriented detection, segmentation, and image classification.

Experiments

  • The experiments section covers settings, results, convergence analysis, and ablation experiments; detailed training hyperparameters are in the Supplementary Material.Source: Experiments · p. 4

Datasets

  • Experiments cover instance and semantic segmentation, horizontal and oriented object detection, and image classification across representative datasets.The datasets are COCO, ADE20K, Pascal VOC 0712, DOTA, STAR, Oxford 102 Flower, Oxford-IIIT Pet, and VOC 2007 Classification.Source: Datasets · p. 4
  • Evaluation uses APbox for Pascal VOC detection and oriented detection, mIoU for ADE20K, APbox and APMask for COCO, and top-1, top-5, and average accuracy for classification.Pascal VOC 0712 uses Swin-Large with RetinaNet; ADE20K uses Swin-Large with UperNet; COCO uses Swin-Base with Cascade Mask RCNN.Source: Datasets · p. 4
  • The experiments include multiple detection frameworks on STAR, a more challenging oriented-detection dataset than horizontal detection because it requires angle information and more complex models.The paper describes oriented detection as relevant to fields such as remote sensing.Source: Datasets · p. 4

Pre-trained Models and Toolkits

The experiments use Swin Transformer backbones pre-trained on ImageNet-22k and verify results with task-specific toolkits.

Pre-trained Models and Toolkits

  • All experiments use Swin Transformer backbones pre-trained on ImageNet-22k at 224×224 resolution.Most tasks use Swin-Large; COCO, DOTA, and STAR use Swin-Base because of memory consumption.Source: Pre-trained Models and Toolkits · p. 4
  • The reported verification toolkits are MMDetection, MMSegmentation, MMRotate, and MMClassification.Source: Pre-trained Models and Toolkits · p. 4
  • Swin-Base is used for COCO, DOTA, and STAR, while most other tasks use Swin-Large.The paper attributes this backbone choice to memory consumption for those tasks.Source: Pre-trained Models and Toolkits · p. 4

Baselines

The comparisons include baselines that update existing framework parameters and methods that add lightweight structures. Extra-structure baselines use fixed pre-trained layers and adapter intermediate dimensions of 64.

Baselines

  • Without extra structure, baselines include FULL, FIXED, BITFIT, NORMTUNING, and PARTIAL-1, which vary which framework or backbone parameters are updated.FULL updates all framework parameters; FIXED updates non-backbone parameters; BITFIT updates backbone biases and non-backbone parameters; NORMTUNING updates norm layers and non-backbone parameters; PARTIAL-1 updates the last backbone block and non-backbone parameters.Source: Baselines · p. 4
  • Extra-structure baselines fix pre-trained layers and use adapter intermediate dimensions of 64, following AdaptFormer.They include ADAPTER, LORA, ADAPTFORMER, and LORAND; LoRand++ uses α=4 and β=16 after each SwinBlock’s MSA/MLP, and its best-performing variant is selected as the challenging comparison setting.Source: Baselines · p. 4

Main Results

Mona surpasses full fine-tuning across representative visual tasks, including instance and semantic segmentation, object detection, image classification, and oriented object detection. Results also indicate that stronger adapter design can improve transfer without requiring the largest parameter count.

Main Results

  • On COCO instance segmentation, Mona is the only evaluated method to outperform full fine-tuning, by 1%.The table reports COCO benchmark results, trainable-backbone parameters, and performance.Source: Main Results · p. 4, Table 1 · p. 5
  • Delta-tuning performance is not directly related to parameter count: Partial-1 has the most trainable parameters but performs significantly below adapter-based baselines.The authors interpret this as evidence that module design can improve transfer efficiency while reducing new parameters.Source: Main Results · p. 5
  • Mona outperforms full fine-tuning by 3.6% on Pascal VOC object detection and 0.18% on ADE20K semantic segmentation.Mona also outperforms the other methods reported for these tasks.Source: Main Results · p. 5, Table 2 · p. 6
  • Mona beats full fine-tuning and other efficient methods on DOTA and STAR with Oriented R-CNN, and beats all baselines on STAR with KLD and H2RBox-v2.STAR is described as more challenging than DOTA, and is tested with additional frameworks.Source: Main Results · p. 5
  • Mona outperforms all baselines on Flowers102 and OxfordPets and has the best average result across the three classification datasets.The average results of all delta-tuning methods also surpass full fine-tuning.Source: Main Results · p. 5
  • Mona surpasses full fine-tuning across representative visual tasks, with a 1% AP gain on COCO instance segmentation.The reported scope also includes semantic segmentation, object detection, image classification, and oriented object detection.Source: Main Results · p. 5

Loss Analysis

On Pascal VOC object detection, Mona converges faster than full fine-tuning and other delta-tuning methods, and its convergence advantage over full fine-tuning is linked to better performance.

Loss Analysis

  • Mona also converges faster than the other delta-tuning methods in the comparison.The authors suggest that multi-cognitive visual filters better process visual features and accelerate transfer-learning convergence.Source: Loss Analysis · p. 6
  • The authors link Mona’s convergence advantage over full fine-tuning to its better performance on VOC.Source: Loss Analysis · p. 6
  • Mona converges faster than full fine-tuning and significantly exceeds it in the loss-curve comparison.Figure 3 presents the loss curves for the object detection task on Pascal VOC.Source: Figure 3 · p. 6, Loss Analysis · p. 6

Ablations

The ablations examine adapter dimension and backbone size on Pascal VOC; 64-dimensional adapters perform best among the tested dimensions, while Mona surpasses full fine-tuning across three backbone sizes.

Ablations

  • Adapter performance is not proportional to intermediate dimension, so adding adapter parameters does not necessarily improve results.The authors note that the 64-dimensional AdaptFormer also exceeds its 32- and 256-dimensional versions in visual classification.Source: Ablations · pp. 6–7
  • Table 4 reports top-1 and top-5 accuracy on three classification datasets using Swin-L.Source: Table 4 · p. 7
  • With other settings fixed, Table 5’s ablation finds that Mona’s 64-intermediate-dimension adapters outperform the tested 32- and 128-dimensional options; the benchmark is not specified in the cited passages.Source: Ablations · p. 6, Table 5 · p. 7

(Mona)

  • Mona surpasses full fine-tuning across the 29M Swin-T, 88M Swin-B, and 197M Swin-L backbone settings.The authors report that Mona’s performance improves as model size grows.Source: (Mona) · p. 7, Table 6 · p. 8
  • From Swin-T to Swin-L, full fine-tuning gains 3.6% performance while Mona gains 3.8%.As the backbone grows, Mona’s parameters make up a smaller proportion of the model.Source: (Mona) · p. 7

Discussion

Adapter-based methods including Mona add a small inference-cost increase, while re-parameterization approaches such as LoRA avoid that cost but tend to perform poorly on visual tasks.

Discussion

  • Adapter modules add a small inference-cost increase, a practical concern for industry use.Source: Discussion · p. 7
  • Re-parameterization approaches such as LoRA avoid added inference cost but tend to perform poorly on visual tasks.Source: Discussion · p. 7
  • The authors identify combining the strengths of adapters and re-parameterization as future work.Source: Discussion · p. 7

Conclusion

The paper introduces Mona, an adapter-based visual fine-tuning method, and reports that it outperforms full fine-tuning and other delta-tuning methods across representative visual tasks. The authors conclude that full fine-tuning is no longer the optimal choice for visual tasks.

Conclusion

  • Mona is a visual fine-tuning method based on multi-cognitive visual adapters.Source: Conclusion · p. 7
  • Across instance and semantic segmentation, object detection, image classification, and oriented object detection, Mona outperforms full fine-tuning and other delta-tuning methods.Source: Conclusion · p. 7
  • The authors conclude that full fine-tuning is no longer the optimal choice for visual tasks.Source: Conclusion · p. 7

Related papers

Loading 2408.08345v2…