CLIP-Adapter: Better Vision-Language Models with Feature Adapters

Peng Gao; Shijie Geng; Renrui Zhang; Teli Ma; Rongyao Fang; Yongfeng Zhang; Hongsheng Li; Yu Qiao

Source-linked AI summary

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, Yu Qiao

arXiv:2110.04544v2Oct 9, 2021cs.CVcs.CL

TL;DR

Few-shot adaptation of pretrained vision-language models must avoid difficult prompt engineering, excessive annotations, and overfitting from full-model fine-tuning. CLIP-Adapter addresses this gap with lightweight bottleneck feature adapters and residual blending of adapted and pretrained features. It outperforms competitive baselines across eleven image-classification datasets while retaining a simple design.

Problem
Prompt engineering is difficult, downstream tasks need annotations, and naive full-model fine-tuning can overfit and train slowly with few examples.
Method
CLIP-Adapter fine-tunes a small number of bottleneck adapter weights after CLIP’s visual or language encoder and blends adapted features residually with pretrained features.
Results
CLIP-Adapter outperforms competitive baselines on eleven image classification datasets under different few-shot setups.
Takeaways & Limitations
The results support lightweight feature adapters as a simple alternative to prompt-based adaptation for few-shot image classification.

Abstract

from arXiv · show

Large-scale contrastive vision-language pre-training has shown significant progress in visual representation learning. Unlike traditional visual systems trained by a fixed set of discrete labels, a new paradigm was introduced in \cite{radford2021learning} to directly learn to align images with raw texts in an open-vocabulary setting. On downstream tasks, a carefully chosen text prompt is employed to make zero-shot predictions.~To avoid non-trivial prompt engineering, context optimization \cite{zhou2021coop} has been proposed to learn continuous vectors as task-specific prompts with few-shot training examples.~In this paper, we show that there is an alternative path to achieve better vision-language models other than prompt tuning.~While prompt tuning is for the textual inputs, we propose CLIP-Adapter to conduct fine-tuning with feature adapters on either visual or language branch. Specifically, CLIP-Adapter adopts an additional bottleneck layer to learn new features and performs residual-style feature blending with the original pre-trained features.~As a consequence, CLIP-Adapter is able to outperform context optimization while maintains a simple design. Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness of our approach. Code is released at t https://github.com/gaopengcuhk/CLIP-Adapter.

1 Introduction

Existing visual systems require costly task-specific data and annotations, while CLIP enables zero-shot classification but prompt design remains difficult. CLIP-Adapter instead fine-tunes lightweight feature adapters, offering an alternative to prompt tuning for few-shot classification.

Large-scale dataset collection and downstream fine-tuning require substantial labor, cost, and annotations.
CLIP performs zero-shot visual classification by aligning images and texts and inserting categories into hand-crafted prompts.
Prompt engineering remains time-consuming and dependent on domain knowledge, motivating learned continuous prompts such as CoOp.
CLIP-Adapter fine-tunes lightweight bottleneck feature adapters rather than optimizing CLIP’s full parameter set.
CLIP-Adapter uses residual-style blending to combine adapted features with pretrained representations while reducing few-shot overfitting risk.
Compared with CoOp, CLIP-Adapter achieves better few-shot classification performance with a simpler design and is evaluated through ablations on eleven datasets.

2 Related Work

Prior work addresses data-efficient adaptation through model fine-tuning, adapters, and prompt design, while vision-language models learn multimodal representations. CLIP-Adapter extends this landscape by applying lightweight feature-adapter fine-tuning to CLIP-based classification.

Model Fine-Tuning: Adapter modules freeze backbone weights and insert learnable linear layers for data-efficient downstream fine-tuning.
Model Fine-Tuning: Related parameter-efficient methods extend adapters to parallel, language, multimodal, black-box, side-tuning, sparse, and scaling-based settings.
Prompt-Based Methods: Prompt-based approaches tune discrete or continuous prompts to adapt pretrained language and vision-language models without changing the base architecture.
Vision-Language Models: CLIP and ALIGN demonstrate the value of contrastive vision-language representation learning across diverse vision tasks.
Vision-Language Models: CLIP-Adapter differs from CoOp by improving vision-language models through lightweight feature-adapter fine-tuning rather than prompt design.

3 Our Approach

CLIP-Adapter adapts CLIP for few-shot classification by fine-tuning lightweight visual and/or textual feature adapters while preserving pretrained features and freezing the original backbone.

CLIP-Adapter: CLIP-Adapter fine-tunes additional feature adapters instead of using CoOp’s prompt tuning approach.The adapters are lightweight additions to CLIP’s visual or language branch.
Architecture: The method appends bottleneck linear layers while keeping CLIP’s original backbone frozen during few-shot fine-tuning.This reduces the number of optimized parameters and addresses overfitting risks associated with few-shot training.
Architecture: Visual and textual adapters transform the original image features and classifier weights, respectively, before classification.The resulting features are combined with the original representations through residual connections.
Feature blending: Residual ratios α and β control how strongly adapted features replace or retain the original pretrained knowledge.The paper notes that optimal values vary across datasets.
Optimization: During few-shot training, adapter weights are optimized with the original CLIP contrastive loss, while predictions use the highest category probability.The image and text branches can be tuned separately or jointly.

4 Experiments

Experiments evaluate CLIP-Adapter on 11 image-classification datasets across 1, 2, 4, 8, and 16 shots using lightweight adapters and handcrafted prompts. The reported figure shows consistent gains over previous baselines across training-shot settings.

Experimental setup: CLIP-Adapter is evaluated on 11 image-classification datasets using 1, 2, 4, 8, and 16 shots.The default configuration tunes the visual adapter while freezing the classifier weight.
Implementation: The setup uses ResNet-50 as the visual backbone and a 12-layer Transformer as the textual encoder.Training uses the same hyperparameters as CoOp except for the residual ratio α.
Experimental setup: The experiments use handcrafted hard prompts rather than CoOp’s learnable continuous prompts.The embedding dimensionality of each bottleneck layer is set to 256, one quarter of the original embedding dimensionality.
Preprocessing: CLIP-style preprocessing preserves cropped-image aspect ratios, and all methods improve under it relative to CoOp-style preprocessing.CLIP-Adapter still outperforms the other baselines across shot settings.
Main results: CLIP-Adapter consistently shows better performance than previous baselines across different training shots on the 11 datasets.The cited figure presents the main few-shot learning results.

4.2 Comparison on Few-Shot Learning

CLIP-Adapter consistently outperforms Zero-shot CLIP, Linear Probe CLIP, and CoOp across few-shot classification settings while retaining favorable efficiency. Ablations attribute this performance to lightweight residual adapters, especially when inserted at the encoder’s end.

Performance Comparison: CLIP-Adapter outperforms Zero-shot CLIP, Linear Probe CLIP, and CoOp across all 11 datasets and shot settings.
Performance Comparison: 20% to 50% absolute gains occur on the first five fine-grained datasets versus Zero-shot CLIP under 16-shot training.Gains are smaller on Caltech101 and ImageNet, while OxfordPets and Food101 show limited improvements because Zero-shot CLIP already performs well.
Performance Comparison: 53.6% and 42.16% margins on OxfordPets, and 37.17% and 27.58% on ImageNet, separate CLIP-Adapter from Linear Probe CLIP under 1-shot and 2-shot training.
Efficiency Comparison: CLIP-Adapter uses half as many parameters as Linear Probe CLIP, adds 7.89% accuracy, and achieves 16× less training time and 29× faster inference than CoOp.The comparison uses 16-shot ImageNet classification; the reported training-time increase over Linear Probe CLIP is 37 minutes.
Adapter Variants: Visual-only, text-only, and joint adapter variants all improve substantially over Zero-shot CLIP, while visual-only adaptation performs better than text-only adaptation.Learnable residual ratios α and β are used instead of manually selecting dataset-specific values.
Adapter Placement: End insertion achieves superior performance with 0.52M parameters, compared with 5.20M when adapters are inserted in all layers.Earlier insertion increases back-propagation costs and can harm pretrained knowledge, whereas end insertion helps preserve it and alleviate overfitting.
Adapter Comparisons: Residual adapters outperform existing adapter-based methods and ELEVATER projection baselines by preserving CLIP’s pretrained knowledge.

4.3 Visualization of Learned Manifold

The t-SNE visualization compares feature manifolds learned by CLIP, CoOp, and CLIP-Adapter variants on EuroSAT. CLIP-Adapter with residual connections produces clearer class separation and better same-class similarity structure.

The visualization covers CLIP, CoOp, and CLIP-Adapter with and without residual connections after EuroSAT training.
Categories 0 to 9 represent the ten EuroSAT classes, including AnnualCrop, Forest, Highway or Road, and Sea or Lake.
CLIP-Adapter with residual connections shows more obvious separation of image features across categories than the other methods.
For confusing categories such as Highway or Road, Permanent Crop Land, and Pasture Land, CLIP-Adapter better detects same-class image-manifold similarities.

4.4 Ablation Studies

Ablations examine bottleneck size, residual blending, prompt styles, visual backbones, distribution shift, and finetuning scope. The results identify settings that preserve generalization while showing CLIP-Adapter’s robustness and advantage over CoOp.

Dimension of Bottleneck Layer: The best bottleneck dimension is D/4; both overly small and overly large intermediate dimensions significantly deteriorate performance.D denotes the original image-feature dimension, and D/4 preserves sufficient semantics without redundancy.
Residual Ratio α: The optimal residual ratio is 0.6 on DTD and 0.2 on ImageNet, indicating different balances between adapted and original knowledge.α=0 corresponds to zero-shot CLIP, whereas α=1.0 relies fully on adapted features and is not optimal because of overfitting.
Influence of Prompt Styles: Prompt ensembling slightly outperforms hard and mixed hard-soft prompts on ImageNet, with hard prompts retaining useful and robust raw-text knowledge.The default hard prompt is “a photo of a {class}”, while soft prompts are described as lacking clear meaning in this comparison.
Ablation of Visual Backbones: CLIP-Adapter consistently outperforms CoOp across ResNet-50, ResNet-101, ViT-B/32, and ViT-B/16 on DTD and ImageNet.The comparison uses 16-shot few-shot learning.
Robustness under Distribution Shift: CLIP-Adapter consistently outperforms other baselines on ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R under distribution shift.The adapter is trained on ImageNet and evaluated on four out-of-distribution datasets.
Finetuning Whole CLIP vs. CLIP-Adapter: Finetuning either CLIP encoder hurts performance and takes more training time, supporting adapter finetuning instead of updating the huge-parameter CLIP.The encoder experiments use early stopping to obtain the highest accuracy.

5 Conclusions and Future Work

The paper presents CLIP-Adapter as an alternative to prompt-based few-shot image classification. It fine-tunes lightweight bottleneck layers with residual blending and outperforms competitive baselines across eleven datasets and varied few-shot setups.

CLIP-Adapter is presented as an alternative to prompt-based approaches for few-shot image classification.
The method fine-tunes a small number of additional bottleneck layers and uses residual connections to blend zero-shot knowledge with adapted features.
CLIP-Adapter outperforms competitive baselines on eleven image classification datasets under different few-shot setups.

Data Availability Statement

The study created no new data and conducted all training and evaluation on eleven publicly available image classification datasets.

No new data were created during the study.
All experiments were conducted on eleven publicly available image classification datasets.

Result Comparison under CLIP-Style Preprocessing.

Under CLIP-style preprocessing, the paper compares few-shot performance across eleven datasets. All methods improve relative to CoOp-style preprocessing, while CLIP-Adapter remains ahead of other baselines across shot settings.

Under CLIP-style preprocessing, all methods improve over CoOp-style preprocessing, and CLIP-Adapter still outperforms other baselines across different shot settings.