Source-linked AI summary
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, Hongsheng Li
TL;DR
Few-shot improvements to CLIP-Adapter typically require costly SGD fine-tuning. Tip-Adapter instead builds adapter weights from a few-shot key-value cache, achieving performance on par with or better than CLIP-Adapter while enabling much faster fine-tuning.
Problem
CLIP-Adapter improves few-shot classification but requires substantial computation and GPU memory because large vision-language models are fine-tuned with slow-converging SGD.
Method
Tip-Adapter constructs a non-parametric key-value cache from few-shot features and labels, uses it to initialize adapter weights, and combines retrieved predictions with pre-trained CLIP predictions.
Results
Tip-Adapter achieves on-par or better results than trained CLIP-Adapter, while its fine-tuned variant requires 20 epochs rather than CLIP-Adapter’s 200.
Takeaways & Limitations
The cache-based initialization provides an efficient training-free alternative for few-shot classification and supports rapid performance improvements through limited fine-tuning.
Abstract
from arXiv · showhide
Contrastive Vision-Language Pre-training, known as CLIP, has provided a new paradigm for learning visual representations by using large-scale contrastive image-text pairs. It shows impressive performance on zero-shot knowledge transfer to downstream tasks. To further enhance CLIP's few-shot capability, CLIP-Adapter proposed to fine-tune a lightweight residual feature adapter and significantly improves the performance for few-shot classification. However, such a process still needs extra training and computational resources. In this paper, we propose \textbf{T}raining-Free CL\textbf{IP}-\textbf{Adapter} (\textbf{Tip-Adapter}), which not only inherits CLIP's training-free advantage but also performs comparably or even better than CLIP-Adapter. Tip-Adapter does not require any back propagation for training the adapter, but creates the weights by a key-value cache model constructed from the few-shot training set. In this non-parametric manner, Tip-Adapter acquires well-performed adapter weights without any training, which is both efficient and effective. Moreover, the performance of Tip-Adapter can be further boosted by fine-tuning such properly initialized adapter for only a few epochs with super-fast convergence speed. We conduct extensive experiments of few-shot classification on ImageNet and other 10 datasets to demonstrate the superiority of proposed Tip-Adapter. The code will be released at \url{https://github.com/gaopengcuhk/Tip-Adapter}.
1. Introduction
Tip-Adapter addresses the computational cost of fine-tuning CLIP-Adapter by initializing adapter weights from a few-shot cache model without SGD. It retains strong few-shot classification performance and can be further fine-tuned rapidly.
- Motivation and approach: CLIP-Adapter and Tip-Adapter use two linear layers with a residual connection, but only CLIP-Adapter trains its adapter with SGD.Tip-Adapter initializes its linear-layer weights from a cache model instead.
- Motivation and approach: Fine-tuning CoOp and CLIP-Adapter requires substantial computation because SGD converges slowly and large vision-language models consume considerable GPU memory.This motivates combining CLIP’s training-free property with CLIP-Adapter’s few-shot performance.
- Motivation and approach: Tip-Adapter constructs a query-key cache from few-shot image features and labels, then uses it to set the adapter weights without back propagation.The adapter retrieves few-shot knowledge from the cache rather than learning weights through conventional SGD fine-tuning.
- Inference and performance: During inference, Tip-Adapter combines adapted features with original CLIP features, leveraging both few-shot knowledge and CLIP’s pre-trained representation.The residual combination preserves information from both sources.
- Inference and performance: Tip-Adapter matches or can exceed trained CLIP-Adapter performance, while further fine-tuning reaches stronger performance in 20 epochs instead of CLIP-Adapter’s 200.The paper reports competitive performance across 11 few-shot classification datasets and reduced training time.
2. Related Work
The paper situates Tip-Adapter within data-efficient transfer learning and cache-based retrieval methods. Its key distinction is directly infusing few-shot supervision into CLIP without further training.
- Data-efficient Transfer Learning: Large-scale dataset collection is difficult because of long-tailed data distributions, noisy annotations, and increasing labeling costs.This motivates transfer-learning approaches that improve CLIP using limited training data.
- Data-efficient Transfer Learning: Unlike prompt or adapter fine-tuning, Tip-Adapter directly infuses few-shot supervision into pre-trained CLIP without additional training.It calculates and caches the few-shot training set once, requiring no further training.
- Cache Models: Transformer-based models use key-query attention to encode high-quality features across language and vision applications.Tip-Adapter’s cache retrieval is described as similar to transformer attention.
- Cache Models: A cache model stores training-image features and labels as key-value pairs, retrieving and aggregating information for each test query.This retrieval process is non-parametric and requires no parameter updates.
3. Method
Tip-Adapter extends CLIP-Adapter with a cache-derived, training-free adapter that incorporates few-shot visual features and labels while retaining CLIP’s prior knowledge. Its cached initialization can match CLIP-Adapter and can be fine-tuned efficiently for stronger performance.
- Training-free CLIP-Adapter: The method retains CLIP-Adapter’s two-layer MLP and residual design, combining adapted features with the original CLIP feature for classification.The residual ratio α balances the cache-based prediction with the pre-trained CLIP prediction.
- Training-free CLIP-Adapter: Tip-Adapter constructs adapter weights from a key-value cache of few-shot CLIP visual features and corresponding labels, avoiding conventional SGD fine-tuning.The cached features serve as keys or first-layer weights, while one-hot labels serve as values or second-layer weights.
- Training-free CLIP-Adapter: Tip-Adapter without fine-tuning performs comparably to fully fine-tuned CLIP-Adapter, while further fine-tuning its initialized adapter can substantially improve accuracy with only 20 epochs.During fine-tuning, the first-layer keys are updated while cached values and CLIP encoders remain fixed.
- Training-free CLIP-Adapter: For inference, Tip-Adapter uses the test image feature as a query, converts normalized-feature distances into affinities, and retrieves cached values through affinity-weighted combination.The exponential mapping produces nonnegative affinities whose sharpness is controlled by β, yielding retrieved class predictions from ALtrain.
- Training-free CLIP-Adapter: Tip-Adapter directly sets W1 to cached training features and W2 to transposed one-hot labels, using a bottleneck dimension of NK rather than CLIP-Adapter’s low-dimensional bottleneck.The paper associates this initialization with alleviated few-shot over-fitting and the use of high-dimensional linear layers.
- Training-free CLIP-Adapter: Tip-Adapter’s bounded distance-based activation differs from CLIP-Adapter’s ReLU activation for unbounded inputs.The activation is defined as ϕ(x) = exp(−β(1 − x)) and modulates normalized-feature distances.
4. Experiments
Experiments on ImageNet and 10 additional datasets compare Tip-Adapter and Tip-Adapter-F with established CLIP-based few-shot methods. Results show strong training-free performance, further gains from brief fine-tuning, and sensitivity to cache and residual settings.
- Comparison on ImageNet: In 1-shot and 2-shot settings, Tip-Adapter surpasses linear-probe CLIP by 38.53% and 29.06%, and CoOp by 13.08% and 10.08%, respectively.Tip-Adapter consistently surpasses zero-shot CLIP without training, though it remains behind well-tuned CLIP-Adapter.
- Efficiency comparison: Tip-Adapter requires no extra training time while providing a 1.70% boost, and Tip-Adapter-F reaches higher accuracy using only 1/10 of CLIP-Adapter’s epochs.CoOp takes the most training time, while CLIP-Adapter reduces training time with a larger improvement.
- Performances on Other Datasets: On 10 additional datasets, Tip-Adapter significantly improves over zero-shot CLIP, while Tip-Adapter-F eliminates gaps against CLIP-Adapter in higher-shot settings.The datasets span categories including cars, actions, scenes, textures, satellite imagery, aircraft, pets, and food.
5. Conclusion
Tip-Adapter constructs adapter weights non-parametrically from a few-shot key-value cache rather than SGD, achieving performance on par with or better than trained CLIP-Adapter. Fine-tuning can further improve it with a favorable efficiency–performance trade-off, though the best weights still require 20 epochs.
- Tip-Adapter obtains adapter weights from a cache model built from the few-shot training set instead of SGD training.The cache stores few-shot knowledge as features and labels for retrieval-based adapter processing.
- Tip-Adapter’s training-free initialization achieves results on par with or better than CLIP-Adapter with training.
- Fine-tuning Tip-Adapter for a few epochs further enhances performance while preserving a favorable efficiency–performance trade-off.
- The best-performing Tip-Adapter weights still require 20-epoch fine-tuning, motivating future work on improved adapter-weight construction.