PaperlayerView exact revision on arXiv

Source-linked AI summary

To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models

Siru Jiang, Yuwei Liang, Jian Liang, Ran He, Tieniu Tan

arXiv:2609.08367v1cs.CVcs.LG

TL;DR

Vision-language models can degrade under distribution shift, and adaptation may be ineffective or harmful for individual samples. This paper introduces selective adaptation and proposes CAS, which uses prediction similarity across augmented views to select samples for adaptation. CAS preserves or improves full-adaptation accuracy while skipping up to 85% of adaptations across the evaluated benchmarks.

  • Problem

    Vision-language models are sensitive to distribution shifts, and existing TTA methods assume adaptation benefits every test sample despite unchanged or worsened predictions in many cases.

    Source: §1 · pp. 1–2

  • Method

    Selective adaptation detects whether to adapt or skip each test sample, and CAS scores prediction similarity across augmented views to guide that choice.

    Source: §1 · pp. 2–3

  • Results

    CAS achieves around 90% AUC and preserves or improves full-adaptation accuracy while skipping 85% of adaptations across ImageNet, its variants, and fine-grained benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Selective adaptation shifts TTA efficiency toward per-sample decisions, with CAS serving as a baseline for further work on the problem.

    Source: §1 · p. 3, §5 · p. 15

Abstract

from arXiv · show

Test-time adaptation (TTA) has emerged as a prominent strategy for adapting vision-language models to distribution shifts during inference. We conduct a per-sample analysis of model predictions before and after adaptation, and observe two failure modes in existing TTA methods that echo previous work. Adaptations are frequently negligible, yielding no change in the model's predictions, and more severely, they can be detrimental by flipping previously correct predictions to incorrect ones. This naturally raises a question: Can we identify and skip such negligible or harmful adaptations? In this work, we introduce a new problem of selective adaptation, which aims to determine whether a given test sample should undergo adaptation or be skipped. To this end, we propose Cross-Augmentation Similarity (CAS), a simple baseline that performs adaptation only when predictions across augmented views exhibit low similarity. Notably, CAS not only preserves but in some cases improves overall accuracy, even when skipping nearly 85% of the adaptation process. We hope other researchers will explore this new direction and surpass the performance of our baseline. Our code is available at https://github.com/sirujiang/selective-adaptation.

1 Introduction

Vision-language models can lose performance under distribution shift, while test-time adaptation may be unnecessary or harmful for individual samples. The paper frames selective adaptation and introduces CAS to skip ineffective adaptations while preserving accuracy.

1 Introduction

  • More than 90% of adaptation processes are negligible or harmful: many leave predictions unchanged, while some flip correct predictions to incorrect ones.The analysis is conducted under the classic TPT framework and is consistent with prior observations.Source: §1 · p. 2
  • Selective adaptation formulates a binary detection task that decides whether each test sample should undergo adaptation or be skipped.Higher scores identify cases tending toward negligible or harmful adaptation; those samples use zero-shot predictions, while lower-scored samples continue to adaptation.Source: §1 · p. 2
  • The proposed Accuracy Expectation with a Triangular Prior (AEP) measures expected accuracy under different skip ratios.The passage introduces AEP as a metric for the evaluation setup.Source: §1 · p. 3
  • CAS preserves or improves full-adaptation accuracy while skipping up to 85% of adaptations across ImageNet, its variants, and fine-grained benchmarks.The paper also reports an AUC of around 90% and maintained calibration performance in calibration-oriented TTA methods.Source: §1 · p. 3
  • CAS scores prediction similarity across augmented views to identify samples for adaptation, comparing it with random skipping and established OOD detection methods under representative TTA frameworks.The motivation is that test-time augmentation predictions may correlate with adaptation effectiveness.Source: §1 · p. 3
  • The paper presents selective adaptation as a direction for improving TTA efficiency and CAS as a baseline for future work.It calls for further investigation of the problem and extensions beyond image classification.Source: §1 · p. 3

2 Related Work

Prior work on vision-language model test-time adaptation spans training-based and training-free approaches, with efficiency work focused on optimization or retrieval. This paper distinguishes per-sample selective skipping from those approaches and from post-inference selective classification.

Test-time adaptation (TTA)

  • The paper focuses exclusively on episodic TTA, where each test sample is treated independently, rather than online TTA over streaming data.Online TTA updates parameters using historical knowledge from prior test samples.Source: §2 · p. 3

TTA for VLMs

  • VLM TTA includes training-based methods that optimize parameters, such as TPT's prompt optimization with entropy minimization, and training-free methods such as ZERO, MTA, and TPS.Recent work also examines calibration and adversarial robustness.Source: §2 · p. 3

Efficiency in TTA

  • This work improves episodic TTA efficiency by selectively skipping ineffective adaptations, unlike methods that streamline optimization or use key–value cache retrieval.Examples include TTL's low-rank adapter optimization, STS's limited parameter adaptation, and EATA's selection of informative samples for adaptation.Source: §2 · p. 4

Out-of-distribution (OOD) detection and selective classification

  • Selective classification rejects predictions after inference, whereas selective adaptation rejects samples before adaptation.OOD detection scoring functions are used as comparison strategies, and AUC evaluates the selective adaptation baseline.Source: §2 · p. 4

3 Method

The method formulates selective adaptation as deciding whether to adapt or skip each test sample. CAS uses agreement across augmented views to score samples, while the formulation also specifies evaluation measures for detection and accuracy across skip ratios.

3.1 Preliminaries

  • CLIP uses visual and text encoders to produce class probabilities, whose maximum defines the zero-shot prediction; the adapted model supplies a corresponding prediction.The class labels are encoded as textual prompts, and the visual encoder extracts image features.Source: §3.1 · p. 4

3.2 Problem Formulation: Selective Adaptation

  • Over 90% of test samples on ImageNet and EuroSAT are negligible or harmful adaptation cases; only beneficial adaptations improve accuracy.The formulation distinguishes harmful, negligible, and beneficial outcomes by whether correctness changes between zero-shot and adapted predictions.Source: §3.2 · p. 5
  • A sample is skipped when its score α(x) meets threshold γ; AUC measures detection, while AEP integrates accuracy across skip ratios under a triangular prior.AEP can also use other performance metrics, such as ECE, in place of accuracy.Source: §3.2 · pp. 5–6

3.3 Cross-Augmentation Similarity as a Simple Baseline

  • CAS combines distribution similarity with label agreement across entropy-selected views, allowing probability differences between same-label views to inform adaptation decisions.The selected views are those whose zero-shot prediction entropy falls below the threshold β.Source: §3.3 · pp. 6–7

4 Experiment

Across zero-shot episodic benchmarks, the experiments evaluate CAS with four TTA methods and find that it can preserve or improve accuracy while reducing adaptation cost. Further analyses examine calibration, component contributions, alternative metrics and augmentations, cutoff sensitivity, and prediction patterns across augmented views.

4.1 Experimental Setup

  • Experiments use CLIP-ViT-B/16 in a zero-shot episodic setting and compare CAS across four TTA methods against random skipping, Energy, and MCM.Evaluation covers ImageNet variants and fine-grained datasets; each image has 64 AugMix views and the cutoff percentile is fixed at 0.1.Source: §4.1 · pp. 8, 10

4.2 Results

  • CAS leads selection strategies across TTA methods: ZERO reaches 69.36% AEP and 94.58% AUC on ImageNet, while TPT averages 87.97% AUC and 64.95% AEP on fine-grained benchmarks.On ImageNet, CAS's ZERO AUC exceeds the second-best baseline, MCM, by 30.36%; on fine-grained datasets, its TPT AUC margins over MCM and Energy are 20.07% and 26.28%.Source: Table 2 · p. 9, §4.2 · p. 10

4.3 Impact on Calibration

  • Across calibration-oriented methods, CAS achieves the highest AEP and AUC while maintaining competitive EEP; under C-TPT, these averages are 61.33%, 85.67%, and 5.43%, respectively.For TPT, average EEP is 8.60% across ImageNet and its variants.Source: Table 3 · p. 10, §4.3 · p. 11

4.4 In-depth Analysis

  • CAS maintains or improves accuracy over full adaptation while skipping up to 85% of samples, and cuts TPT inference time from 7.76 to 2.55 hours on ImageNet.That is a 3.04× speedup, with accuracy changing from 68.88% to 69.03%; ZERO and STS overhead also falls by approximately 20% and 10%, respectively.Source: Figure 4 · p. 11, §4.4 · pp. 12–13, Table 5 · p. 12
  • CAS's consistency and similarity components each improve AUC by about 30% over MCM on ImageNet and its variants, while their combination is more stable overall, especially on fine-grained datasets.Under ZERO, combining the components raises AUC from 90.82% to 91.25%.Source: §4.4 · pp. 12–13, Table 4 · p. 12
  • CAS remains best with alternative AEP functions and resized-crop or horizontal-flip augmentations, reaching 62.19% average AEP with f(s) = 1 and 92.65% average AUC with the alternative augmentations.With f(s) = 2s, CAS obtains 61.94% average AEP; under the alternative augmentation setting, average AEP is 63.76%.Source: Table 6 · p. 13, Table 7 · p. 13, §4.4 · p. 14
  • AUC is robust across cutoff percentiles, but excessive selected views can slightly degrade it; AEP changes marginally and declines on EuroSAT as the cutoff grows.The authors set the cutoff percentile to 0.1 to maintain detection quality while minimizing computation.Source: §4.4 · p. 14, Figure 5 · p. 14

4.5 Case Study

  • Stable correct-to-correct samples offer limited adaptation gains, wrong-to-correct samples lie near decision boundaries, and correct-to-wrong samples are especially sensitive to perturbations.Wrong-to-wrong predictions are consistently incorrect, suggesting stable but biased representations that augmentation alone cannot rectify.Source: §4.5 · p. 15

5 Conclusion

The conclusion frames selective adaptation as deciding per test sample whether to adapt or skip, and presents CAS as a baseline that can reduce computational overhead while maintaining performance. Its algorithm scores high-quality augmented views and skips adaptation for samples with high CAS scores.

5 Conclusion

  • Figure 6 visualizes augmented views alongside their corresponding predictions.The conclusion encourages exploring selective adaptation beyond image classification.Source: Figure 6 · p. 15, §5 · p. 15
  • Selective adaptation makes a per-sample decision about whether a test sample should undergo adaptation or be skipped.The conclusion presents reduced computational overhead with maintained performance as the baseline's main practical benefit.Source: §5 · p. 15

A Algorithm

  • CAS scores prediction consistency and distribution similarity across high-quality augmented views, and high-scoring samples use zero-shot predictions instead of adaptation.The algorithm takes a test image, pretrained VLM, augmentation function, augmentation count, cutoff percentile, and threshold as inputs.Source: A Algorithm · p. 20

B Results on ResNet-50

On ResNet-50, CAS outperforms competing selection strategies on ImageNet variants and fine-grained datasets. It is also effective with the model-based MEMO and training-free MTA methods.

B Results on ResNet-50

  • With ZERO, CAS averages 87.96% AUC and 47.54% AEP on ImageNet variants, versus MCM’s 54.88% AUC.On ImageNet, CAS reaches 91.86% AUC, compared with MCM’s 61.44%.Source: B Results on ResNet-50 · p. 20, Table 8 · p. 21
  • On fine-grained datasets, CAS averages 87.64% AUC under STS, versus MCM at 58.56% and Energy at 52.30%.With STS, peak AUCs are 97.08% on Caltech101 and 93.30% on Food101.Source: B Results on ResNet-50 · pp. 20–21

C Results on Other TTA Methods

  • CAS remains effective when applied to the model-based MEMO method and the training-free MTA method.The paper presents these evaluations as evidence of CAS’s applicability beyond the training-based TTA methods emphasized in the main paper.Source: C Results on Other TTA Methods · p. 21

C.1 CAS for MEMO [75]

Under MEMO, CAS outperforms competing selection strategies on CIFAR-10, CIFAR-10-C, and ImageNet-R. It also maintains accuracy close to full adaptation across much of the skip-ratio range.

C.1 CAS for MEMO [75]

  • On CIFAR-10, CAS reaches 97.98% AUC versus MCM’s 87.22%; on ImageNet-R, it reaches 81.86% AUC, 30.09% above the second-best method.Table 10 compares AUC and AEP under MEMO on CIFAR-10 and ImageNet-R.Source: §C.1 · pp. 21–22, Table 9 · p. 22
  • On CIFAR-10-C, CAS achieves the highest AUC of 95.10% and AEP of 80.40%.The passage reports that CAS consistently outperforms Random, Energy, and MCM across AUC and AEP.Source: §C.1 · pp. 21–22
  • Across almost the entire skip-ratio range under MEMO, CAS beats Random, MCM, and Energy while keeping accuracy close to full adaptation.The accuracy–skip-ratio comparison shows larger accuracy drops for the other strategies as the skip ratio increases.Source: §C.1 · p. 22

C.2 CAS for MTA [72]

Under training-free MTA, CAS outperforms competing selection strategies on ImageNet variants and fine-grained datasets, with strong average AUC results across both groups.

C.2 CAS for MTA [72]

  • Under MTA on ImageNet variants, CAS averages 93.36% AUC, versus MCM’s 59.80%.Table 11 compares AUC and AEP across selection strategies on ImageNet and its variants.Source: §C.2 · pp. 22–23, Table 11 · p. 24
  • On ImageNet-A under MTA, CAS achieves 92.12% AUC, and its performance remains stable across the evaluated ImageNet variants.Source: §C.2 · p. 23
  • On fine-grained datasets under MTA, CAS averages 92.30% AUC, versus MCM at 66.50% and Energy at 61.03%.CAS also reaches 99.13% AUC on Caltech101 and 97.85% on Pets.Source: §C.2 · p. 23, Table 12 · p. 24

D More Baselines

CAS is evaluated against additional selection baselines under TPT with ViT-B/16, and achieves strong average AUC and AEP results.

D More Baselines

  • Under TPT with ViT-B/16, CAS averages 89.42% AUC, outperforming the second-best baseline by 22.63%, and yields 62.44% average AEP.Table 13 reports AUC and AEP on ImageNet and its variants.Source: D More Baselines · p. 24, Table 13 · p. 24
  • The additional baselines explored are Max Logits, Max Softmax, and GL-MCM.Source: D More Baselines · p. 24
  • The main text reports Energy and MCM because their performance was comparable and space was limited.Source: D More Baselines · p. 24

E Different VLMs

CAS retains strong detection performance with SigLIP, and its performance remains stable when using substantially fewer test-time augmentation views than the default 64.

E Different VLMs

F Impact of the Number of Test-Time Augmentations

  • CAS performance initially increases with more augmentation views, then rapidly plateaus at a relatively low view count.The comparison reduces the default 64 views to as few as 6.Source: F Impact of the Number of Test-Time… · p. 25
  • AUC and AEP remain largely stable across datasets even with substantially fewer augmentation views than the default 64.Figure 8 presents the effect of reducing augmentation views on ImageNet and its variants.Source: F Impact of the Number of Test-Time… · p. 25

G Adaptation Behaviors across Different TTA Methods

Different TTA methods show highly consistent judgments about which samples require adaptation. CAS performance also remains stable with substantially fewer augmentation views than the default.

G Adaptation Behaviors across Different TTA Methods

  • AUC and AEP remain largely stable across datasets when using substantially fewer augmentation views than the default 64.Figure 8 examines ImageNet and its variants, and performance rapidly plateaus at a relatively low view count.Source: Figure 8 · p. 26
  • Hamming similarity in selective-adaptation labels across TTA methods consistently exceeds 90% on evaluated benchmarks.The analysis compares TPT, R-TPT, STS, and ZERO.Source: G Adaptation Behaviors across… · pp. 25–26
  • Ineffective cases are largely shared across methods, suggesting adaptation necessity is determined largely by the sample rather than the TTA algorithm.Most samples are consistently identified as not requiring adaptation.Source: G Adaptation Behaviors across… · p. 26

H Statistics of Adaptation Cases

Across 15 benchmarks, negligible adaptation cases dominate while beneficial cases are typically around 5% or lower. CAS values distinguish beneficial from ineffective cases, and CAS performance is reported as stable across domains.

H Statistics of Adaptation Cases

  • Beneficial adaptations account for only around 5% or less across the 15 benchmarks, while most samples fall into negligible cases.The four transitions are Correct to Correct, Wrong to Wrong, Correct to Wrong, and Wrong to Correct; only Wrong to Correct is genuinely effective.Source: H Statistics of Adaptation Cases · pp. 26–27, Figure 10 · p. 28

I CAS Distribution

  • Beneficial cases cluster at small CAS values, whereas negligible and harmful cases tend toward larger values with similar high-region peaks.Figure 11 groups the cases into negligible, harmful, beneficial, ineffective, and effective categories.Source: I CAS Distribution · p. 29, Figure 11 · p. 29

J Standard Deviations

Loading 2609.08367v1…