Source-linked AI summary
Low-Rank Few-Shot Adaptation of Vision-Language Models
Maxime Zanella, Ismail Ben Ayed
TL;DR
Few-shot VLM adaptation has largely emphasized prompt learning and adapters, often with heavy procedures or task-specific hyperparameters. The paper applies LoRA to few-shot VLMs and evaluates CLIP-LoRA against existing methods across 11 datasets. CLIP-LoRA consistently outperforms prompt- and adapter-based methods with fixed hyperparameters, while the study shows that selecting adapted matrices and ranks remains nontrivial.
Problem
Few-shot VLM research has largely overlooked PEFT and often relies on computationally heavy procedures or task-specific hyperparameter searches.
Method
The paper introduces CLIP-LoRA for few-shot VLM adaptation and examines encoder, weight-matrix, and rank choices through comparisons across 11 datasets.
Results
CLIP-LoRA consistently outperforms prompt- and adapter-based few-shot methods while using fixed hyperparameters.
Takeaways & Limitations
CLIP-LoRA provides a strong simple baseline for evaluating future prompt-learning and adapter-based methods in few-shot VLMs.
Takeaways & Limitations
Selecting which matrices to adapt and which LoRA ranks to use for maximum performance is not trivial.
Abstract
from arXiv · showhide
Recent progress in the few-shot adaptation of Vision-Language Models (VLMs) has further pushed their generalization capabilities, at the expense of just a few labeled samples within the target downstream task. However, this promising, already quite abundant few-shot literature has focused principally on prompt learning and, to a lesser extent, on adapters, overlooking the recent advances in Parameter-Efficient Fine-Tuning (PEFT). Furthermore, existing few-shot learning methods for VLMs often rely on heavy training procedures and/or carefully chosen, task-specific hyper-parameters, which might impede their applicability. In response, we introduce Low-Rank Adaptation (LoRA) in few-shot learning for VLMs, and show its potential on 11 datasets, in comparison to current state-of-the-art prompt- and adapter-based approaches. Surprisingly, our simple CLIP-LoRA method exhibits substantial improvements, while reducing the training times and keeping the same hyper-parameters in all the target tasks, i.e., across all the datasets and numbers of shots. Certainly, our surprising results do not dismiss the potential of prompt-learning and adapter-based research. However, we believe that our strong baseline could be used to evaluate progress in these emergent subjects in few-shot VLMs.
1. Introduction
Few-shot VLM adaptation has expanded beyond zero-shot prediction, but existing prompt- and adapter-based approaches often overlook PEFT efficiency and require costly, task-specific tuning. This work introduces CLIP-LoRA and evaluates it across 11 datasets as a simple, fixed-hyperparameter alternative.
- Few-shot adaptation uses limited labeled data to specialize VLMs for downstream tasks.
- Larger VLM variants can improve accuracy but increase the need for efficient fine-tuning methods that scale to large models.In CLIP, ViT-L/14 surpasses ViT-B/16 by over 6% accuracy on ImageNet, including after few-shot adaptation.
- Existing adapters and prompt-learning methods can increase computational demand and model size, while adapters also require dataset-specific hyperparameter searches.
- Few-shot VLM research has mainly focused on prompt tuning and adapters while overlooking recent PEFT advances such as LoRA.
- The paper investigates LoRA design choices and compares CLIP-LoRA with prompt- and adapter-based methods across 11 datasets.The study varies encoder selection, adapted weight matrices, and matrix rank, using empirical ablations and comparisons.
- CLIP-LoRA improves over state-of-the-art few-shot VLM methods while reducing computational overhead and maintaining one hyperparameter configuration across target tasks.The authors position this fixed-hyperparameter approach as a baseline for evaluating future prompt-learning and adapter-based research.
2. Related work
PEFT methods reduce the parameters trained in large models through selective updates, added modules, learnable tokens, or low-rank weight changes. In VLM few-shot learning, prompt tuning and adapters dominate, while LoRA offers merged updates without added inference latency.
- PEFT reduces fine-tuning expense by training relatively few parameters, trading off memory footprint, computational overhead, and performance.
- Selective methods update subsets of existing weights, including transformer biases or task-specific pruned parameters.
- Adapters add trainable modules to frozen architectures, but their extra parameters can increase inference latency after adaptation.
- Prompt tuning adds learnable tokens to inputs or intermediate sequences and has become especially popular for few-shot and zero-shot VLM learning.
- LoRA represents weight changes with low-rank matrices while freezing original parameters, and the updates can be merged before inference without additional latency.
- VLM few-shot research includes joint vision-language prompt methods such as PLOT and MaPLe, which adapt both modalities for downstream flexibility.
- Adapter-based VLM methods include CLIP-Adapter, Tip-Adapter, and TaskRes, which use modified features, cache similarities, or embedding adjustments.
3. Few-shot fine-tuning for VLMs
Few-shot VLM classification uses normalized image and text embeddings, then adapts either prompts or additional parameters with limited labeled support data. Prompt tuning avoids manual prompt engineering but can be computationally heavy, while adapters reduce fine-tuning load yet depend on task-specific hyper-parameter searches.
- VLM prediction: Zero-shot classification pairs each image embedding with class-specific text embeddings and predicts the class with the highest posterior probability.Text and image representations are normalized and compared through cosine similarity before softmax prediction.
- Few-shot formulation: Few-shot VLM adaptation assumes N/K labeled support samples per target class, typically fewer than 16, and minimizes cross-entropy loss.The labels are one-hot encoded for the support images.
- Prompt tuning: Prompt tuning optimizes learnable text or visual tokens, replacing manually designed prompts with task-specific soft prompts.CoOp constructs text prompts from trainable tokens and fixed class-name tokens.
- Adapters: Adapters add trainable modules to frozen encoders and can reduce computational load compared with prompt tuning.Examples include feature-transforming multilayer modules and class-score modules based on similarities among labeled samples.
- Adapters: Adapter performance depends strongly on task-specific hyper-parameters, requiring validation-set searches that consume additional labeled samples and computational resources.The resulting tuning burden reduces portability to new downstream tasks.
4. CLIP-LoRA
LoRA represents updates to frozen VLM weights with low-rank matrices and applies them to selected transformer attention matrices. The CLIP-LoRA setup uses fixed low-rank, regularization, optimization, and prompting choices across experiments while targeting efficient few-shot adaptation.
- LoRA mechanism: LoRA models each fine-tuning update as the product of two small matrices while keeping the original weight matrix frozen.The update has rank r, typically much smaller than the original matrix dimensions, and is scaled by γ.
- Transformer placement: LoRA can target transformer key, query, value, and output weight matrices, with these matrices corresponding to attention components.The cited formulation identifies WKi, WQi, WVi, and Wo as the key, query, value, and output matrices.
- CLIP-LoRA design: CLIP-LoRA applies rank-2 low-rank matrices to query, key, and value matrices and regularizes their inputs with dropout probability 0.25.The method limits adaptation to selected attention matrices because few-shot supervision is relatively small.
- Training setup: CLIP-LoRA fixes 2 × 10^-4 learning rate, cosine scheduling, batch size 32, and 500 times N/K iterations across experiments.Training is configured to run on a single 24 Gb GPU, and the same prompt template is used for every dataset.
5. Few-shot learning
The study benchmarks CLIP-LoRA against prompt- and adapter-based methods on 11 few-shot classification datasets and finds stronger average performance with shorter training and lower memory requirements. Its results remain favorable across vision backbones, although performance is weaker on Food101 and OxfordPets.
- Evaluation setup: CLIP-LoRA is evaluated on 10 fine-grained classification datasets plus ImageNet using top-1 accuracy averaged over 3 random seeds.The benchmark covers scenes, aircraft, satellite imagery, automobiles, food, pets, flowers, objects, textures, human actions, and ImageNet.
- Comparative results: CLIP-LoRA outperforms, on average, adapter- and prompt-based few-shot methods.The comparison includes CoOp, CoCoOp, PLOT++, KgCoOp, MaPLe, ProGrad, Tip-Adapter-F, and TaskRes.
- Comparative results: PLOT++ is the strongest prompt-based competitor, but its two-stage training and dataset-specific textual templates reduce portability compared with CLIP-LoRA.The reported comparison particularly highlights ImageNet, UCF101, and Aircraft.
- Comparative results: CLIP-LoRA underperforms on Food101 and OxfordPets, where few-shot learning offers minimal improvement.The authors suggest this may reflect a lack of regularization under straightforward cross-entropy training.
- Backbone consistency: CLIP-LoRA surpasses other few-shot methods on average with both ViT-B/32 and ViT-L/14, supporting consistency across vision encoders.Figure 2 reports detailed results on the same 11 datasets and averages for ViT-B/16, ViT-B/32, and ViT-L/14.
- Efficiency: CLIP-LoRA achieves better performance with shorter training and avoids extra inference memory beyond the original model.LoRA merges adapted matrices at inference, whereas Tip-Adapter-F stores embeddings for all instances across classes.
6. How to apply LoRA for VLMs?
The paper studies how encoder choice, attention-matrix selection, rank, and module placement affect LoRA for VLMs. Adapting both encoders and placing modules across all layers are generally effective, while broader matrix adaptation can hurt at high rank.
- Design considerations: The design study varies encoder selection, adapted attention matrices, matrix rank, and LoRA module placement across ImageNet, Stanford Cars, and EuroSAT.Figure 3 reports top-1 accuracy with 4 shots, averaged over 3 random seeds.
- Encoder choice: Adapting both encoders leads to the best results on average, except that vision-only tuning is marginally more stable on EuroSAT.The authors relate this finding to approaches that add vision tokens beyond text-only prompt tuning.
- Attention matrices: Adapting value or output attention matrices, Wv and Wo, appears to provide the most consistent performance among the four matrices studied.Adapting more weight matrices can improve results, but the benefit is not uniform.
- Attention matrices: Increasing the number of adapted matrices can decrease performance on ImageNet and StanfordCars when the rank is high.This motivates interest in methods that dynamically adjust matrix rank.
- Module placement: LoRA placement across all layers is more effective than restricting modules to the upper or lower half, whose performances vary without a clear winner.The figure compares Bottom, Up, and All placements across encoder levels.
7. Conclusion
The work establishes CLIP-LoRA as a strong, simple baseline for few-shot VLM adaptation, outperforming prompt- and adapter-based methods with fixed hyper-parameters. It also identifies LoRA design choices and adaptable hyper-parameter guidance as directions for future research.
- CLIP-LoRA consistently outperformed prompt- and adapter-based methods in few-shot VLM adaptation using fixed hyper-parameters.
- The authors propose preserving this simplicity and efficiency with fixed hyper-parameters or providing clear guidelines for adaptable settings.
- Selecting LoRA matrices and determining their rank to maximize performance is not trivial.