PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, Yu Qiao

arXiv:2110.04544v2cs.CVcs.CL

TL;DR

Few-shot adaptation of pretrained vision-language models must avoid difficult prompt engineering, excessive annotations, and overfitting from full-model fine-tuning. CLIP-Adapter addresses this gap with lightweight bottleneck feature adapters and residual blending of adapted and pretrained features. It outperforms competitive baselines across eleven image-classification datasets while retaining a simple design.

  • Problem

    Prompt engineering is difficult, downstream tasks need annotations, and naive full-model fine-tuning can overfit and train slowly with few examples.

    Source: §1 · pp. 1–2

  • Method

    CLIP-Adapter fine-tunes a small number of bottleneck adapter weights after CLIP’s visual or language encoder and blends adapted features residually with pretrained features.

    Source: §1 · p. 2, §5 · p. 13

  • Results

    CLIP-Adapter outperforms competitive baselines on eleven image classification datasets under different few-shot setups.

    Source: §1 · p. 3, §5 · p. 13

  • Takeaways & Limitations

    The results support lightweight feature adapters as a simple alternative to prompt-based adaptation for few-shot image classification.

    Source: §1 · p. 3, §5 · p. 13

Abstract

from arXiv · show

Large-scale contrastive vision-language pre-training has shown significant progress in visual representation learning. Unlike traditional visual systems trained by a fixed set of discrete labels, a new paradigm was introduced in \cite{radford2021learning} to directly learn to align images with raw texts in an open-vocabulary setting. On downstream tasks, a carefully chosen text prompt is employed to make zero-shot predictions.~To avoid non-trivial prompt engineering, context optimization \cite{zhou2021coop} has been proposed to learn continuous vectors as task-specific prompts with few-shot training examples.~In this paper, we show that there is an alternative path to achieve better vision-language models other than prompt tuning.~While prompt tuning is for the textual inputs, we propose CLIP-Adapter to conduct fine-tuning with feature adapters on either visual or language branch. Specifically, CLIP-Adapter adopts an additional bottleneck layer to learn new features and performs residual-style feature blending with the original pre-trained features.~As a consequence, CLIP-Adapter is able to outperform context optimization while maintains a simple design. Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness of our approach. Code is released at t https://github.com/gaopengcuhk/CLIP-Adapter.

1 Introduction

Existing visual systems require costly task-specific data and annotations, while CLIP enables zero-shot classification but prompt design remains difficult. CLIP-Adapter instead fine-tunes lightweight feature adapters, offering an alternative to prompt tuning for few-shot classification.

  • Large-scale dataset collection and downstream fine-tuning require substantial labor, cost, and annotations.Source: §1 · p. 1
  • CLIP performs zero-shot visual classification by aligning images and texts and inserting categories into hand-crafted prompts.Source: §1 · p. 2
  • Prompt engineering remains time-consuming and dependent on domain knowledge, motivating learned continuous prompts such as CoOp.Source: §1 · p. 2
  • CLIP-Adapter fine-tunes lightweight bottleneck feature adapters rather than optimizing CLIP’s full parameter set.Source: §1 · p. 2
  • CLIP-Adapter uses residual-style blending to combine adapted features with pretrained representations while reducing few-shot overfitting risk.Source: §1 · p. 2
  • Compared with CoOp, CLIP-Adapter achieves better few-shot classification performance with a simpler design and is evaluated through ablations on eleven datasets.Source: §1 · p. 3

2 Related Work

Prior work addresses data-efficient adaptation through model fine-tuning, adapters, and prompt design, while vision-language models learn multimodal representations. CLIP-Adapter extends this landscape by applying lightweight feature-adapter fine-tuning to CLIP-based classification.

Model Fine-Tuning

  • Adapter modules freeze backbone weights and insert learnable linear layers for data-efficient downstream fine-tuning.Source: §2 · p. 3
  • Related parameter-efficient methods extend adapters to parallel, language, multimodal, black-box, side-tuning, sparse, and scaling-based settings.Source: §2 · p. 3

Prompt-Based Methods

  • Prompt-based approaches tune discrete or continuous prompts to adapt pretrained language and vision-language models without changing the base architecture.Source: §2 · p. 3

Vision-Language Models

  • CLIP and ALIGN demonstrate the value of contrastive vision-language representation learning across diverse vision tasks.Source: §2 · p. 3
  • CLIP-Adapter differs from CoOp by improving vision-language models through lightweight feature-adapter fine-tuning rather than prompt design.Source: §2 · p. 3, Figure 2 · p. 4

3 Our Approach

CLIP-Adapter adapts CLIP for few-shot classification by fine-tuning lightweight visual and/or textual feature adapters while preserving pretrained features and freezing the original backbone.

CLIP-Adapter

  • CLIP-Adapter fine-tunes additional feature adapters instead of using CoOp’s prompt tuning approach.The adapters are lightweight additions to CLIP’s visual or language branch.Source: §3.2 · p. 5, §3.3 · p. 6

Architecture

  • The method appends bottleneck linear layers while keeping CLIP’s original backbone frozen during few-shot fine-tuning.This reduces the number of optimized parameters and addresses overfitting risks associated with few-shot training.Source: §3.2 · p. 5
  • Visual and textual adapters transform the original image features and classifier weights, respectively, before classification.The resulting features are combined with the original representations through residual connections.Source: §3.2 · p. 5

Feature blending

  • Residual ratios α and β control how strongly adapted features replace or retain the original pretrained knowledge.The paper notes that optimal values vary across datasets.Source: §3.2 · p. 5, §3.3 · p. 6

Optimization

  • During few-shot training, adapter weights are optimized with the original CLIP contrastive loss, while predictions use the highest category probability.The image and text branches can be tuned separately or jointly.Source: §3.2 · pp. 5–6, §3.3 · p. 6

4 Experiments

Experiments evaluate CLIP-Adapter on 11 image-classification datasets across 1, 2, 4, 8, and 16 shots using lightweight adapters and handcrafted prompts. The reported figure shows consistent gains over previous baselines across training-shot settings.

Experimental setup

  • CLIP-Adapter is evaluated on 11 image-classification datasets using 1, 2, 4, 8, and 16 shots.The default configuration tunes the visual adapter while freezing the classifier weight.Source: §4.1 · p. 6

Implementation

  • The setup uses ResNet-50 as the visual backbone and a 12-layer Transformer as the textual encoder.Training uses the same hyperparameters as CoOp except for the residual ratio α.Source: §4.1 · p. 6

Experimental setup

  • The experiments use handcrafted hard prompts rather than CoOp’s learnable continuous prompts.The embedding dimensionality of each bottleneck layer is set to 256, one quarter of the original embedding dimensionality.Source: §4.1 · p. 6

Preprocessing

  • CLIP-style preprocessing preserves cropped-image aspect ratios, and all methods improve under it relative to CoOp-style preprocessing.CLIP-Adapter still outperforms the other baselines across shot settings.Source: §4.1 · p. 6, Figure 3 · p. 7

Main results

  • CLIP-Adapter consistently shows better performance than previous baselines across different training shots on the 11 datasets.The cited figure presents the main few-shot learning results.Source: Figure 3 · p. 7

4.2 Comparison on Few-Shot Learning

CLIP-Adapter consistently outperforms Zero-shot CLIP, Linear Probe CLIP, and CoOp across few-shot classification settings while retaining favorable efficiency. Ablations attribute this performance to lightweight residual adapters, especially when inserted at the encoder’s end.

Performance Comparison

  • CLIP-Adapter outperforms Zero-shot CLIP, Linear Probe CLIP, and CoOp across all 11 datasets and shot settings.Source: §4.2.2 · p. 8
  • 20% to 50% absolute gains occur on the first five fine-grained datasets versus Zero-shot CLIP under 16-shot training.Gains are smaller on Caltech101 and ImageNet, while OxfordPets and Food101 show limited improvements because Zero-shot CLIP already performs well.Source: §4.2.2 · p. 8
  • 53.6% and 42.16% margins on OxfordPets, and 37.17% and 27.58% on ImageNet, separate CLIP-Adapter from Linear Probe CLIP under 1-shot and 2-shot training.Source: §4.2.2 · p. 8

Efficiency Comparison

  • CLIP-Adapter uses half as many parameters as Linear Probe CLIP, adds 7.89% accuracy, and achieves 16× less training time and 29× faster inference than CoOp.The comparison uses 16-shot ImageNet classification; the reported training-time increase over Linear Probe CLIP is 37 minutes.Source: §4.2.3 · pp. 8–9, Table 1 · p. 8

Adapter Variants

  • Visual-only, text-only, and joint adapter variants all improve substantially over Zero-shot CLIP, while visual-only adaptation performs better than text-only adaptation.Learnable residual ratios α and β are used instead of manually selecting dataset-specific values.Source: §4.2.5 · p. 9

Adapter Placement

  • End insertion achieves superior performance with 0.52M parameters, compared with 5.20M when adapters are inserted in all layers.Earlier insertion increases back-propagation costs and can harm pretrained knowledge, whereas end insertion helps preserve it and alleviate overfitting.Source: §4.2.6 · p. 9, Table 2 · p. 10

Adapter Comparisons

4.3 Visualization of Learned Manifold

The t-SNE visualization compares feature manifolds learned by CLIP, CoOp, and CLIP-Adapter variants on EuroSAT. CLIP-Adapter with residual connections produces clearer class separation and better same-class similarity structure.

  • The visualization covers CLIP, CoOp, and CLIP-Adapter with and without residual connections after EuroSAT training.Source: §4.3 · p. 10
  • Categories 0 to 9 represent the ten EuroSAT classes, including AnnualCrop, Forest, Highway or Road, and Sea or Lake.Source: §4.3 · p. 10
  • CLIP-Adapter with residual connections shows more obvious separation of image features across categories than the other methods.Source: §4.3 · p. 10
  • For confusing categories such as Highway or Road, Permanent Crop Land, and Pasture Land, CLIP-Adapter better detects same-class image-manifold similarities.Source: §4.3 · p. 10

4.4 Ablation Studies

Ablations examine bottleneck size, residual blending, prompt styles, visual backbones, distribution shift, and finetuning scope. The results identify settings that preserve generalization while showing CLIP-Adapter’s robustness and advantage over CoOp.

Dimension of Bottleneck Layer

  • The best bottleneck dimension is D/4; both overly small and overly large intermediate dimensions significantly deteriorate performance.D denotes the original image-feature dimension, and D/4 preserves sufficient semantics without redundancy.Source: §4.4 · p. 11

Residual Ratio α

  • The optimal residual ratio is 0.6 on DTD and 0.2 on ImageNet, indicating different balances between adapted and original knowledge.α=0 corresponds to zero-shot CLIP, whereas α=1.0 relies fully on adapted features and is not optimal because of overfitting.Source: §4.4 · p. 12

Influence of Prompt Styles

  • Prompt ensembling slightly outperforms hard and mixed hard-soft prompts on ImageNet, with hard prompts retaining useful and robust raw-text knowledge.The default hard prompt is “a photo of a {class}”, while soft prompts are described as lacking clear meaning in this comparison.Source: §4.4 · pp. 12–13

Ablation of Visual Backbones

  • CLIP-Adapter consistently outperforms CoOp across ResNet-50, ResNet-101, ViT-B/32, and ViT-B/16 on DTD and ImageNet.The comparison uses 16-shot few-shot learning.Source: §4.4 · p. 13

Robustness under Distribution Shift

  • CLIP-Adapter consistently outperforms other baselines on ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R under distribution shift.The adapter is trained on ImageNet and evaluated on four out-of-distribution datasets.Source: §4.4 · p. 13

Finetuning Whole CLIP vs. CLIP-Adapter

  • Finetuning either CLIP encoder hurts performance and takes more training time, supporting adapter finetuning instead of updating the huge-parameter CLIP.The encoder experiments use early stopping to obtain the highest accuracy.Source: §4.4 · p. 13

5 Conclusions and Future Work

The paper presents CLIP-Adapter as an alternative to prompt-based few-shot image classification. It fine-tunes lightweight bottleneck layers with residual blending and outperforms competitive baselines across eleven datasets and varied few-shot setups.

  • CLIP-Adapter is presented as an alternative to prompt-based approaches for few-shot image classification.Source: §5 · p. 13
  • The method fine-tunes a small number of additional bottleneck layers and uses residual connections to blend zero-shot knowledge with adapted features.Source: §5 · p. 13
  • CLIP-Adapter outperforms competitive baselines on eleven image classification datasets under different few-shot setups.Source: §5 · p. 13

Data Availability Statement

The study created no new data and conducted all training and evaluation on eleven publicly available image classification datasets.

Result Comparison under CLIP-Style Preprocessing.

Under CLIP-style preprocessing, the paper compares few-shot performance across eleven datasets. All methods improve relative to CoOp-style preprocessing, while CLIP-Adapter remains ahead of other baselines across shot settings.

Related papers

Loading 2110.04544v2…