Source-linked AI summary
Efficient Test-Time Adaptation of Vision-Language Models
Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, Eric Xing
TL;DR
Test-time adaptation for vision-language models must handle distribution shifts without the intensive computation used by prior prompt-tuning methods. TDA introduces a training-free dynamic key-value cache with progressive pseudo-label refinement and negative pseudo labeling. Across two benchmarks, it is reported to outperform state-of-the-art methods while substantially reducing testing time.
Problem
Prior vision-language test-time adaptation methods achieve promising performance but require computationally intensive prompt optimization, limiting efficient test-time use.
Method
TDA uses a lightweight dynamic key-value cache for progressive pseudo-label refinement and adds a negative cache to reduce the impact of noisy pseudo labels without backpropagation.
Results
TDA outperforms state-of-the-art test-time adaptation methods across two benchmarks while reducing ImageNet testing time from over 12 hours to 16 minutes.
Takeaways & Limitations
The training-free cache design provides an efficient and effective approach to test-time adaptation of vision-language models.
Takeaways & Limitations
The broader impact of test-time adaptation requires attention to transparency, fairness, accountability, and other ethical considerations during deployment.
Abstract
from arXiv · showhide
Test-time adaptation with pre-trained vision-language models has attracted increasing attention for tackling distribution shifts during the test time. Though prior studies have achieved very promising performance, they involve intensive computation which is severely unaligned with test-time adaptation. We design TDA, a training-free dynamic adapter that enables effective and efficient test-time adaptation with vision-language models. TDA works with a lightweight key-value cache that maintains a dynamic queue with few-shot pseudo labels as values and the corresponding test-sample features as keys. Leveraging the key-value cache, TDA allows adapting to test data gradually via progressive pseudo label refinement which is super-efficient without incurring any backpropagation. In addition, we introduce negative pseudo labeling that alleviates the adverse impact of pseudo label noises by assigning pseudo labels to certain negative classes when the model is uncertain about its pseudo label predictions. Extensive experiments over two benchmarks demonstrate TDA's superior effectiveness and efficiency as compared with the state-of-the-art. The code has been released in \url{https://kdiaaa.github.io/tda/}.
1. Introduction
Vision-language models face distribution shifts at test time, while existing prompt-tuning approaches require computationally intensive optimization. TDA addresses this with a training-free dynamic cache and negative pseudo labeling for efficient, robust adaptation.
- Motivation: Test-time prompt tuning adapts vision-language models by optimizing domain-specific prompts from test data, but its backpropagation-based optimization is computationally intensive.TPT and DiffTPT use augmented views and entropy minimization to learn prompts for test samples.
- TDA: TDA uses a lightweight key-value cache storing test-sample features as keys and few-shot pseudo labels as values, without test-time backpropagation.The cache progressively incorporates lower-entropy predictions and can be computed with simple matrix multiplications.
- Negative pseudo labeling: Negative pseudo labeling assigns certain classes as absent when predictions are uncertain, complementing the positive cache and reducing the impact of noisy pseudo labels.The combined caches improve tolerance to pseudo-label noise and generalization to testing data.
- Evaluation: TDA is evaluated on two test-time adaptation benchmarks and is reported to outperform state-of-the-art methods while improving testing efficiency.The reported ImageNet testing time decreases from over 12 hours to 16 minutes.
2. Related Work
Related work establishes test-time adaptation as a response to distribution shifts and surveys parameter, normalization, augmentation-consistency, prompt-tuning, and cache-based approaches. For vision-language models, prompt learning remains effective but computationally expensive, motivating more efficient cache-based adaptation.
- Test-time adaptation: Test-time adaptation adapts models to testing data whose distribution differs from training data, supporting deployment across diverse environments.Examples include autonomous driving under varied weather and medical diagnosis across hospitals.
- General methods: Existing test-time adaptation methods update model weights, normalization statistics, or both, while MEMO instead enforces invariant predictions across augmentations.These approaches differ in whether they update parameters or impose prediction consistency.
- Vision-language adaptation: TPT and DiffTPT adapt CLIP through test-time prompt learning, but prompt optimization is computationally expensive and time-consuming.DiffTPT increases test-data augmentation diversity using pre-trained diffusion models.
- Cache models: Cache models offer efficient inference and non-parametric processing without parameter updates, although large training-data caches can challenge memory efficiency.Unbounded Cache and PSMM capture long-term dependencies by storing large amounts of training data.
3. Method
TDA adapts CLIP at test time without backpropagation by maintaining dynamic positive and negative key-value caches of test features and pseudo labels. It progressively refines cache contents using prediction entropy and combines cache-based predictions with CLIP predictions.
- Motivation: TDA addresses the computational cost of test-time prompt tuning, which uses many augmentations and optimization steps for each test image.These requirements limit applicability in real-world settings.
- Architecture: TDA uses CLIP image features as keys and pseudo labels as values in two lightweight, dynamically updated caches.The caches support positive and negative predictions that are combined with the original CLIP prediction.
- Positive cache: The positive cache stores high-quality pseudo labels and corresponding test features, expanding dynamically while prioritizing lower-entropy predictions under a per-class shot capacity.When a class queue is full, a higher-entropy key-value pair can be replaced by a lower-entropy test prediction.
- Positive cache: TDA generates initial one-hot pseudo labels by applying softmax to CLIP image-text predictions before deciding whether to add or replace positive-cache entries.The cache is initialized empty and accumulates pairs during test-time adaptation.
- Prediction: At inference, TDA retrieves cache information using test features and combines positive-cache, negative-cache, and pre-trained CLIP predictions.The cache adaptation uses the same adaptation function defined for Tip-Adapter.
- Negative cache: The negative cache stores features with negative pseudo labels that mark likely absent classes in uncertain predictions, reducing reliance on noisy positive labels.Probabilities above the negative-label threshold are assigned -1, while other entries are assigned 0.
4. Experiments
Experiments evaluate TDA across OOD and cross-domain benchmarks, efficiency comparisons, and ablations of its cache designs and shot capacity. TDA improves accuracy over prior test-time adaptation methods while sharply reducing testing time, and combining positive and negative caches performs best in ablations.
- Experimental Setup: TDA is evaluated on OOD and cross-domain benchmarks against CLIP, train-time adaptation methods, and test-time adaptation methods including TPT and DiffTPT.The OOD benchmark covers ImageNet-derived datasets, while the cross-domain benchmark evaluates diverse class datasets.
- OOD Benchmark: 2.74% and 3.08% average OOD accuracy gains over TPT are reported for ResNet-50 and ViT-B/16, respectively.Against DiffTPT, the corresponding gains are 0.94% and 3.37%.
- Efficiency and Effectiveness: 16 minutes is TDA’s testing time, compared with 12h 50min for TPT and 34h 45min for DiffTPT.The efficiency comparison uses the ImageNet validation dataset with 50,000 images and a single NVIDIA Quadro RTX 6000 GPU.
- Cross-Domain Benchmark: 3.37% and 2.43% average-accuracy gains over TPT are achieved on the cross-domain benchmark with CLIP-ResNet-50 and CLIP-ViT-B/16, respectively.TDA also gains 1.18% and 2.06% over DiffTPT for the respective backbones.
- Ablation Studies: 61.03% cross-domain accuracy is achieved by combining positive and negative caches, versus 60.38% for the positive cache and 60.11% for the negative cache.On ImageNet, the ablation uses TDA’s default accuracy of 61.35% and finds that both cache types individually surpass CLIP while their combination performs best.
- Ablation Studies: Shot capacity 3 for the positive cache and 2 for the negative cache yields the best performance.Shot capacity is the maximum number of key-value pairs per class; overly low or high values reduce performance, with larger negative caches producing noisier pseudo labels.
5. Conclusion
TDA enables efficient test-time adaptation through a dynamic key-value cache and negative pseudo labeling. Experiments across two benchmarks show improved performance with substantially reduced testing time.
- TDA maintains test-sample features as keys and few-shot pseudo labels as values in a dynamic queue for progressive adaptation.
- Negative pseudo labels assign certain classes as absent when predictions are uncertain, reducing the effects of noisy pseudo labels.
- Extensive experiments over two benchmarks show that TDA outperforms state-of-the-art test-time adaptation methods while significantly reducing testing time.
6. Benchmark Details
The evaluation uses an OOD Benchmark for robustness to natural distribution shifts and a Cross-Domain Benchmark spanning ten image-classification datasets.
- The OOD Benchmark evaluates robustness on ImageNet and out-of-distribution versions containing varying styles and corruptions.
- Its datasets include ImageNet-V2, ImageNet-A, ImageNet-R, and ImageNet-S, covering updated natural images, perturbations, artistic styles, and sketches.
- The Cross-Domain Benchmark contains 10 datasets spanning general, fine-grained, satellite, action, texture, and scene image classification.
7. Parameter Studies on Thresholds
Parameter studies examine thresholds for negative pseudo-labeling and Negative Cache feature selection, along with residual and sharpness ratios. The reported settings favor stable negative-label selection and adapted-feature weighting.
- Negative pseudo-labeling threshold: The negative pseudo-labeling threshold p_l performs best at 0.03, while larger values remain stable without notable improvement or degradation.
- Negative Cache feature selection: Shifting the threshold range from [0.2, 0.5] to [0.3, 0.6] decreases performance by 0.48% through noisier early Negative Cache samples.
- Residual and sharpness ratios: The optimal residual ratio is 2.0 and the optimal sharpness ratio is 5.0 for TDA.
8. More Experimental Analysis
Additional analyses examine cache construction and shot capacity. Caches accumulate samples during inference, while larger positive-cache capacity can reduce adapted-prediction reliability through noise.
- Caches built for inference: Caches start empty and progressively accumulate samples during inference, with a 0.06% accuracy drop during the initial 1% of ImageNet testing.
- Caches built for inference: Bypassing cache usage early improves accuracy by 0.1% but is not adopted because it requires an additional timing hyperparameter.
- Class imbalance under high shot capacity: Increasing shot capacity from 3 to 6 reduces cache accuracy from 90.3% to 86.6% because larger caches accumulate noise.
- Class imbalance under high shot capacity: With a 6-shot positive cache, only 4 of 1000 ImageNet classes contain fewer than 6 samples, indicating minimal class imbalance.
9. Broader Impact
Test-time adaptation may improve the real-world applicability, accessibility, inclusivity, and versatility of vision-language models across domains. Its deployment also requires attention to transparency, fairness, and accountability.
- Test-time adaptation may enhance real-world applicability and make vision-language models more versatile across domains such as healthcare and assistive technologies.
- It may improve accessibility and inclusivity while offering opportunities to mitigate biases and personalize user experiences.
- Responsible development and deployment require transparency, fairness, and accountability in the adaptation process.