PaperlayerView exact revision on arXiv

Source-linked AI summary

A note on data-efficient finetuning of vision foundation models

Niclas Hergenröther, Antonio Orvieto

arXiv:2610.37510v1cs.CVcs.LG

TL;DR

The paper addresses the need for systematic, tuned evidence on how expanding foundation-model architectures interact with visual prompting and other transfer methods. It evaluates these choices and VP regularization, finding that outcomes vary by model and task; on DTD, LoRA outperforms FFT in five of six models.

  • Problem

    Prior evaluations of newer foundation models with VP had limited model selection or lacked hyperparameter tuning, leaving results ambiguous to interpret.

    Source: §2 · p. 2

  • Method

    The study evaluates ImageNet-pretrained models with AutoVP, compares VP, LP, FFT, and LoRA on DTD, and tests common VP regularization methods.

    Source: §2 · pp. 2–3

  • Results

    ConvNext was the best VP option in this setup, while LoRA achieved higher reported DTD accuracy than FFT for five of six evaluated models.

    Source: §4.2 · p. 4, Conclusion · p. 5

  • Takeaways & Limitations

    For VP, model selection should consider both pretraining accuracy and architecture; on data-scarce tasks, the authors identify LoRA as a lightweight alternative to FFT.

    Source: §4.2 · p. 4, Conclusion · p. 5

    The compute comparison assumes VMamba-B on DTD with 47 target classes, and reported parameter counts depend on the selected model and task.

    Source: §3 · p. 3

Abstract

from arXiv · show

Adapting foundation models to a new task within or beyond the training domain is the de facto standard for building specialized machine learning models. Recently, Visual Prompting (VP) has emerged as a promising technique for efficient knowledge transfer. As new foundation model families~(like Mamba) get introduced and VP pipelines such as AutoVP reach greater maturity, we find a growing need for a systematic evaluation of current approaches. In this work, we assess the performance of the latest models, comparing them to earlier architectures and alternative fine-tuning methods, to better understand the progress, challenges and opportunities in the field of efficient fine-tuning under resource limitations. Towards this goal, this paper provides a concise empirical overview of the interactions among foundation model families (Attention-, Convolution-, and Mamba-based) and transfer paradigms: VP, Linear Probing (LP), Full Fine-tuning (FFT) and Low Rank Adaptation (LoRA). Our work builds up on previous findings by broadening the selection of evaluated models and techniques while tuning hyperparameters. In the interest of delivering practical guidelines for the user, we also explore application of prevalent regularization techniques to boost performance in the context of VP.

1. Introduction

Efficient transfer is essential for foundation models, but expanding architectures and fine-tuning options call for systematic evaluation. The paper compares VP across model families and other methods, reporting differences by architecture, task data, and compute setting.

1. Introduction

  • Vision Transformers and Mamba models perform similarly under current VP practice, with a slight edge for Transformers; ConvNext is best among the evaluated VP choices.Figure 1 plots ImageNet pretraining accuracy against average VP accuracy across tasks.Source: §1 · p. 1, Figure 1 · p. 1
  • On DTD, LoRA achieves higher reported accuracy than FFT for five of six evaluated models while training fewer parameters.Source: §1 · p. 1
  • On data-scarce tasks, VP can underperform LP, FFT, and LoRA; under restricted compute, LP appears dominant relative to VP.Source: §1 · p. 1
  • Prevalent regularization techniques can boost VP generalization, with most gains from regularizing label-mapping weights rather than the prompt.Source: §1 · p. 2

2. Background

This section situates visual prompting among transfer methods and motivates a broader, tuned evaluation of models and techniques. It outlines the comparisons and regularization approaches used in the study.

2. Background

  • Visual prompting (VP) transfers a pretrained model using a universal input perturbation while keeping the source model frozen (Elsayed et al., 2018; Bahng et al., 2022).Source: §2 · p. 2
  • Earlier combinations of newer foundation models and VP had limited model selection or lacked hyperparameter tuning, making their results ambiguous to interpret.Source: §2 · p. 2
  • The study evaluates ImageNet1k-pretrained models with AutoVP on common classification tasks, examining performance and prompt resizing.It also compares the approaches on DTD, described as the benchmark's most challenging task with limited training data.Source: §2 · p. 2, Table 2 · p. 2
  • Linear probing trains a replacement classifier while freezing the pretrained model; full fine-tuning updates all model parameters and requires more compute and memory.Source: §2 · pp. 2–3
  • LoRA trains low-rank updates while keeping the base model frozen, reducing trainable parameters (Hu et al., 2022).Source: §2 · p. 3
  • The VP regularization study tests dropout, L1/L2 regularization, sharpness-aware minimization, and AutoAugment, including targeting prompt or label-mapping parameters.On DTD, all models reached at least 99% training accuracy despite weaker test performance than LP, FFT, or LoRA.Source: §2 · p. 3

3. Experimental setup

The experimental setup evaluates ImageNet-pretrained models on classification tasks, then compares transfer methods on DTD and tests VP regularization. The compute comparison is tied to a specific VMamba-B and DTD setting.

3. Experimental setup

  • The benchmark uses eight classification tasks and ImageNet-pretrained models from Mamba, Transformer, and Convolution families.The listed tasks are CIFAR10/100, Flowers102, GTSRB, DTD, EuroSAT, Food101, and OxfordPets.Source: §3 · p. 3
  • The experiments cover AutoVP across the models, compare LP, VP, FFT, and LoRA on DTD, and evaluate VP regularization with VMamba-B.Source: §3 · p. 3
  • The first and second experiment sets use one Nvidia A100 GPU; the remaining experiments use either a Tesla V100 or Quadro RTX 6000.Source: §3 · p. 3
  • For the compute comparison, trainable-parameter estimates assume VMamba-B on DTD with 47 target classes and LoRA rank 2.Reported values depend on the model and task; the authors note VMamba LoRA may be less optimized than Transformer implementations.Source: §3 · p. 3

4. Results

Visual-prompting results vary with model choice and adaptation design. On DTD, LoRA outperforms FFT in most evaluated models, while regularization improves VP but does not close every reported performance gap.

4.1. VP levels model performances

  • ImageNet pretraining accuracy generally correlated with VP performance, while ConvNext achieved the highest accuracy on 4 of 8 tasks.Performance differences were marginal on CIFAR10 and EuroSAT, but model selection was crucial on DTD.Source: §4.1 · p. 4
  • 54.77 percentage points: AutoVP improved VMamba performance on Flowers102 over ILM-VP, which reached 38.9%.ViT-B reached 83.1% with ILM-VP and was outperformed by VMamba under AutoVP.Source: §4.1 · p. 4
  • For VMamba-T on Flowers102, fully connected label mapping was the main source of improvement over ILM-VP.The ablation separates label-mapping and prompt-resizing effects.Source: §4.1 · p. 4, Figure 3 · p. 4
  • Prompt resizing varied across models: ConvNext retained the initial size, while Swin-B showed considerable variation; models differing only by size tended to behave similarly.Figure 2 reports mean prompt size for the best-performing run, normalized to an initial size of 1, averaged across tasks.Source: §4.1 · p. 4, Figure 2 · p. 4

4.2. Comparison training paradigms

  • In the specific setting described, LoRA outperformed FFT on 5 of the 6 evaluated models, while results showed no substantial differences between different LoRA rank configurations (see tab. 4).Source: §4.2 · p. 4
  • About 15% of FFT’s GPU memory was required for VP in the VMamba-B-on-DTD setting; LP was the lightest method, with VP only slightly more demanding.The table caption defines GPU-Mem as peak allocated GPU memory and runtime as the median across runs.Source: §4.2 · p. 4, Table 3 · p. 4

4.3. Regularization of VP

  • L1 regularization yielded similar enhancements on prompt generation and label mapping, whereas L2 was substantially more effective on label mapping; augmentation gave the smallest gains.VMamba also benefited from SAM and dropout applied to label mapping.Source: §4.3 · p. 5
  • On DTD, tested regularizers improved VMamba-B’s VP accuracy but left a gap to LoRA, and none fully closed the performance gap between VMamba and ConvNext.The comparison uses VMamba-B on DTD and reports unregularized VP in the first row.Source: §4.3 · p. 5, Table 5 · p. 5

Conclusion

The conclusion recommends considering both pretraining accuracy and architecture when selecting a model for VP, with ConvNext the best option in this setup. For data-scarce tasks, it identifies LoRA as a lightweight method that outperformed FFT.

Conclusion

  • For data-scarce tasks, LoRA appears to be the method of choice, outperforming FFT while remaining lightweight.For VP in this setup, model selection should consider both pretraining accuracy and architecture; ConvNext was the best option.Source: Conclusion · p. 5

A. Benchmarking Visual Prompting

The section reports prompt resizing across model–task combinations, alongside the experimental setup used for the AutoVP experiments. Prompt size can increase or decrease relative to its initial calibration.

A. Benchmarking Visual Prompting

B. Comparing Knowledge Transfer Methods

The section describes the implementation and training setup for comparing Linear Probing, Full Fine-tuning, LoRA, and Visual Prompting.

B. Comparing Knowledge Transfer Methods

C. Regularizing Visual Prompting

The section tests several regularization approaches for Visual Prompting, including strategies applied to the prompt or label-mapping parameters.

C. Regularizing Visual Prompting

Related papers

Loading 2610.37510v1…