PaperlayerView exact revision on arXiv

Source-linked AI summary

Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification

Renrui Zhang, Zhang Wei, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, Hongsheng Li

arXiv:2207.09519v1cs.CVcs.AIcs.CL

TL;DR

Existing CLIP adaptation methods improve few-shot classification by fine-tuning additional modules, but require extra training and computation. Tip-Adapter instead retrieves few-shot knowledge through a cache model without training, while Tip-Adapter-F fine-tunes cached keys and achieves state-of-the-art performance with fast convergence.

  • Problem

    Existing methods improve CLIP’s few-shot classification performance through fine-tuning, but require extra training time and computational resources.

  • Method

    Tip-Adapter appends frozen CLIP with a non-parametric key-value cache model constructed from few-shot data, while Tip-Adapter-F unfreezes cached keys for fine-tuning.

  • Results

    Across 11 few-shot classification datasets, Tip-Adapter performs comparably to training-required approaches, while Tip-Adapter-F achieves state-of-the-art performance with super-fast convergence.

  • Takeaways & Limitations

    Few-shot knowledge can be incorporated into CLIP through cache-based feature retrieval without conventional SGD-based training, with optional key fine-tuning for stronger performance.

  • Takeaways & Limitations

    Tip-Adapter-F keeps cache values and CLIP encoders frozen while learning cached keys, because the values encode ground-truth annotations.

Abstract

from arXiv · show

Contrastive Vision-Language Pre-training, known as CLIP, has provided a new paradigm for learning visual representations using large-scale image-text pairs. It shows impressive performance on downstream tasks by zero-shot knowledge transfer. To further enhance CLIP's adaption capability, existing methods proposed to fine-tune additional learnable modules, which significantly improves the few-shot performance but introduces extra training time and computational resources. In this paper, we propose a training-free adaption method for CLIP to conduct few-shot classification, termed as Tip-Adapter, which not only inherits the training-free advantage of zero-shot CLIP but also performs comparably to those training-required approaches. Tip-Adapter constructs the adapter via a key-value cache model from the few-shot training set, and updates the prior knowledge encoded in CLIP by feature retrieval. On top of that, the performance of Tip-Adapter can be further boosted to be state-of-the-art on ImageNet by fine-tuning the cache model for 10$\times$ fewer epochs than existing methods, which is both effective and efficient. We conduct extensive experiments of few-shot classification on 11 datasets to demonstrate the superiority of our proposed methods. Code is released at https://github.com/gaopengcuhk/Tip-Adapter.

1 Introduction

Existing CLIP adaptation methods improve few-shot classification but require additional training, while Tip-Adapter introduces a training-free cache-based adapter and a fine-tuned variant for efficient adaptation.

  • CoOp and CLIP-Adapter improve CLIP for few-shot classification but require training additional learnable modules.
  • Tip-Adapter appends weight-frozen CLIP with a non-parametric adapter built as a query-key cache model from few-shot data.
  • The cache stores CLIP visual features as keys and one-hot labels as values for retrieving few-shot knowledge.
  • During inference, similarity-based retrieval aggregates cached values into an adapter prediction, which is combined with CLIP’s original prediction through a residual connection.
  • Tip-Adapter-F unfreezes cache keys for fine-tuning, achieves state-of-the-art ImageNet performance with fast convergence, and both variants are evaluated on 11 datasets.

2 Related Work

Related work establishes CLIP as a source of transferable zero-shot features and describes additive-module adaptation and cache-based retrieval, positioning Tip-Adapter as a training-free alternative.

  • CLIP, DeCLIP, and ALIGN obtain transferable features for zero-shot recognition by learning from contrastive vision-language pairs.
  • CoOp, CLIP-Adapter, and WiSE-FT improve CLIP with limited training data by freezing pretrained weights and training additive learnable modules.
  • Tip-Adapter directly infuses few-shot supervision into pretrained CLIP in a training-free manner, targeting greater time and memory efficiency.
  • A cache model stores training-image features and labels as key-value pairs and retrieves information through similarity-based aggregation without parameter updates.

3 Method

Tip-Adapter adapts CLIP with a non-parametric cache built from few-shot visual features and labels, retrieving that knowledge during inference and combining it with CLIP’s prior predictions. Its fine-tuned variant updates cached keys while retaining frozen values and CLIP encoders, and the approach also relates visual and textual caches to prior adapter designs.

  • Training-free Adaption of CLIP: Tip-Adapter constructs a non-parametric key-value cache from few-shot CLIP visual features and one-hot ground-truth labels, avoiding extra training.The cached features serve as keys and the labels as values.
  • Training-free Adaption of CLIP: During inference, a test feature retrieves cached values through query-key similarities, producing an adapter prediction that is combined with CLIP’s original prediction by a residual connection.The similarity transformation uses φ(x) = exp(-β(1 − x)), with β controlling sharpness and α balancing the two predictions.
  • Tip-Adapter with Fine-tuning: Tip-Adapter-F fine-tunes cached keys with SGD while freezing cached one-hot values and both pre-trained CLIP encoders.The keys are initialized from the cache model to improve affinity estimation without changing category annotations.
  • Multi-modality Cache Model: Tip-Adapter’s multi-modality cache extends the visual cache with textual keys from CLIP’s classifier and identity-matrix textual values.The textual keys represent category text features produced by CLIP’s textual encoder.
  • Relations with Previous Models: Compared with CLIP-Adapter, Tip-Adapter retrieves few-shot knowledge directly from cached features and labels instead of learning an MLP-based adapter through SGD.CLIP-Adapter uses learnable parameters and a lower-dimensional bottleneck, whereas Tip-Adapter’s bottleneck dimension equals NK.

4 Experiments

Experiments across 11 datasets evaluate Tip-Adapter and its fine-tuned variant under few-shot, efficiency, ablation, prompt, and distribution-shift settings. Tip-Adapter improves CLIP without training, while Tip-Adapter-F delivers stronger performance with efficient fine-tuning.

  • Training Settings: Experiments cover 11 image classification datasets with 1-, 2-, 4-, 8-, and 16-shot training settings.
  • ImageNet Comparison: Tip-Adapter consistently surpasses Zero-shot CLIP without training, while Tip-Adapter-F achieves the best performance across few-shot settings.On ImageNet, Tip-Adapter exceeds Linear-probe CLIP by +38.53% in 1-shot and +29.06% in 2-shot settings.
  • Efficiency Comparison: Tip-Adapter improves Zero-shot CLIP by +1.70% without extra training, while Tip-Adapter-F reaches state-of-the-art accuracy using one-tenth of competing methods’ training epochs.CoOp and CLIP-Adapter require training, whereas Tip-Adapter uses the cache model for a performance-efficiency trade-off.
  • Other Datasets: Tip-Adapter-F consistently surpasses compared methods across the other 10 datasets after fewer-epoch cache-model fine-tuning.Tip-Adapter itself surpasses CoOp trained with 1 or 2 shots on most datasets but can trail methods trained with more shots.
  • Ablation Studies: With α=1.0, Tip-Adapter reaches 62.03% accuracy on ImageNet, balancing CLIP prior knowledge with few-shot cache knowledge.Increasing α from 0.0 to 1.0 improves accuracy; α=0.0 is equivalent to Zero-shot CLIP.
  • Ablation Studies: Constraining the cache size to 16 limits training-free gains as shots increase, whereas Tip-Adapter-F continues improving by fine-tuning cache keys.The performance boost gradually slows with more samples for training-free Tip-Adapter, but fine-tuning breaks this apparent cache-size limit.
  • Prompt Design: Prompt ensembling outperforms a single prompt, with Tip-Adapter and Zero-shot CLIP showing larger accuracy drops under prompt variation.Tip-Adapter-F and CLIP-Adapter are less affected by changing prompt designs.
  • Distribution Shift: Tip-Adapter shows superior robustness to distribution shift when trained on ImageNet and tested on ImageNetV2 or ImageNet-Sketch.

5 Visualization

The visualization tracks cache-key distributions before, during, and after fine-tuning. Fine-tuning makes same-category embeddings converge and different-category clusters separate more clearly.

  • Figure 7 uses t-SNE to visualize cache keys for 10 categories of 16-shot ImageNet across three fine-tuning stages.The stages are training-free Tip-Adapter, Tip-Adapter during fine-tuning, and final Tip-Adapter-F.
  • Before training, the cache construction already produces discriminative category distributions.
  • During fine-tuning, same-category embeddings converge while different-category clusters become more separated, supporting stronger classification capability.

6 Conclusions

Tip-Adapter retrieves few-shot knowledge through a cache model and combines it with CLIP’s pre-trained knowledge without training. Tip-Adapter-F further fine-tunes cached keys, achieving state-of-the-art performance while retaining a modest training requirement.

  • Tip-Adapter constructs a non-parametric adapter from the few-shot training set and incorporates retrieved knowledge with CLIP’s pre-trained knowledge training-free.
  • Tip-Adapter-F fine-tunes cached keys for a few epochs and achieves state-of-the-art performance among existing methods.
  • Tip-Adapter-F still requires 20-epoch fine-tuning on ImageNet to learn the best-performing cache model.

A Fine-tuning Settings

The fine-tuning experiments compare unfreezing different Tip-Adapter modules on 16-shot ImageNet. They preserve cached labels and encoders while fine-tuning cached keys for the main Tip-Adapter-F setting.

  • Tip-Adapter-F fine-tunes cache keys while freezing cache values, CLIP’s visual encoder, and textual encoder.
  • The experiments train seven module-unfreezing settings for 20 epochs on 16-shot ImageNet, using smaller learning rates for CLIP encoders.
  • Fine-tuning cached values reduces accuracy to 60.90%, while fine-tuning the entire cache model causes training collapse.

B Performance Gain without Training

Tip-Adapter improves Zero-shot CLIP without training across 11 classification datasets under 16-shot settings. The largest reported gains occur on EuroSAT and Flowers102, where domain differences require more few-shot knowledge.

  • Tip-Adapter’s absolute accuracy improvement over Zero-shot CLIP is evaluated on 11 classification datasets under 16-shot settings.
  • 33.02% improvement occurs on EuroSAT and 23.87% on Flowers102 without any training.
  • The authors associate larger gains with larger domain gaps between downstream datasets and CLIP’s pre-training data.

C Compared to Fully-trained Methods

Compared with conventional methods trained on full datasets, Tip-Adapter performs strongly using only 16-shot data and no parameters or training. Tip-Adapter-F improves further through efficient fine-tuning.

  • Using only a 16-shot training set, training-free Tip-Adapter outperforms ResNet-50 by +1.9% and DeiT-T by +3.9% on ImageNet.
  • Tip-Adapter-F achieves higher performance through efficient fine-tuning requiring 6 minutes.
  • The comparison covers Tip-Adapter, Tip-Adapter-F, and conventional methods trained by full training sets on ImageNet.
Loading 2207.09519v1…