PaperlayerView exact revision on arXiv

Source-linked AI summary

C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion

Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark Hasegawa-Johnson, Yingzhen Li, Chang D. Yoo

arXiv:2403.14119v3cs.CVcs.AIcs.CLcs.LG

TL;DR

Test-time prompt tuning for CLIP targets accuracy without labels, but calibration is important for quantifying prediction uncertainty and conventional calibration methods rely on labeled data. The paper uses text-feature dispersion to tune prompts for calibration, and reports improved calibration from C-TPT across datasets and CLIP models. The method may slightly reduce accuracy in some cases, and its balancing hyperparameter is fixed across test scenarios.

  • Problem

    CLIP prompt tuning has primarily targeted accuracy, while calibration matters for prediction uncertainty and conventional calibration methods require substantial labeled data.

    Source: §1 · pp. 1–2

  • Method

    C-TPT jointly tunes prompts at test time by maximizing ATFD, using text-feature dispersion as a calibration signal without labeled data.

    Source: §5.2 · p. 5, §5.1 · p. 5

  • Results

    C-TPT improves calibration across datasets and CLIP models; in fine-grained classification, average ECE falls from 11.7 to 6.20 for CLIP-RN50 with Hard Prompt initialization and from 11.6 to 5.13 for CLIP-ViT-B/16.

    Source: §1 · p. 2, §6.1 · p. 7

  • Takeaways & Limitations

    Prompts with greater text-feature dispersion are associated with better calibration, offering a label-free signal for test-time prompt tuning.

    Source: §5.2 · p. 5, §8 · p. 9

    The balancing hyperparameter λ is fixed at 50.0 across test scenarios and may not be optimal for every data sample.

    Source: §A.2 · p. 15

Abstract

from arXiv · show

In deep learning, test-time adaptation has gained attention as a method for model fine-tuning without the need for labeled data. A prime exemplification is the recently proposed test-time prompt tuning for large-scale vision-language models such as CLIP. Unfortunately, these prompts have been mainly developed to improve accuracy, overlooking the importance of calibration, which is a crucial aspect for quantifying prediction uncertainty. However, traditional calibration methods rely on substantial amounts of labeled data, making them impractical for test-time scenarios. To this end, this paper explores calibration during test-time prompt tuning by leveraging the inherent properties of CLIP. Through a series of observations, we find that the prompt choice significantly affects the calibration in CLIP, where the prompts leading to higher text feature dispersion result in better-calibrated predictions. Introducing the Average Text Feature Dispersion (ATFD), we establish its relationship with calibration error and present a novel method, Calibrated Test-time Prompt Tuning (C-TPT), for optimizing prompts during test-time with enhanced calibration. Through extensive experiments on different CLIP architectures and datasets, we show that C-TPT can effectively improve the calibration of test-time prompt tuning without needing labeled data. The code is publicly accessible at https://github.com/hee-suk-yoon/C-TPT.

1 INTRODUCTION

Test-time prompt tuning (TPT) targets accuracy without labels, but calibration—the alignment between predicted probabilities and outcomes—has been overlooked despite its importance for prediction uncertainty. The paper studies unlabeled calibration during prompt tuning by using CLIP’s text-feature dispersion and introduces C-TPT to improve calibration.

1 INTRODUCTION

  • Reducing prediction entropy can produce overconfident predictions, which worsens calibration.TPT uses entropy minimization as a self-supervision signal during test-time adaptation.Source: §1 · p. 1
  • Calibration matters because it measures whether predicted probabilities align with the true outcome probabilities, supporting trustworthy uncertainty estimates in applications such as healthcare and autonomous driving.Existing CLIP prompt-tuning work has primarily focused on classification accuracy.Source: §1 · p. 2
  • Traditional calibration techniques require substantial labeled data, limiting their use in test-time prompt tuning.The paper instead seeks to use CLIP’s intrinsic structure and properties to calibrate without labels.Source: §1 · p. 2
  • C-TPT jointly tunes prompts at test time by maximizing Average Text Feature Dispersion (ATFD), improving calibration across datasets and CLIP models without labeled data.The paper reports that this approach improves test-time prompt optimization and produces better-calibrated predictions.Source: §1 · p. 2

2 RELATED WORK

Prior prompt-tuning methods differ in their data requirements, while established calibration methods generally use labeled data. TPT adapts prompts without labels, motivating a calibration approach tailored to that setting.

2 RELATED WORK

  • TPT adaptively learns prompts from a single unlabeled test sample, unlike CoOp and CoCoOp, which require annotated training data.CoCoOp conditions prompts on input images to address CoOp’s difficulty generalizing to out-of-distribution data.Source: §2 · p. 2
  • Post-processing calibration methods such as temperature scaling and conformal prediction use a held-out validation dataset, while trainable methods add calibration objectives during training.The cited trainable approaches include MMCE, SB-ECE, S-AvUC, and ESD.Source: §2 · p. 2
  • Established calibration methods rely on substantial labeled data, restricting their use in few-shot or test-time CLIP prompt tuning.The paper proposes a prompt-tuning calibration technique designed to operate without labeled data.Source: §2 · p. 3

3 BACKGROUND AND PROBLEM SETUP

CLIP classifies images by comparing visual and text features for candidate classes, while prompt tuning can adapt prompts as trainable embeddings. The section also frames calibration through prediction confidence and ECE.

3 BACKGROUND AND PROBLEM SETUP

  • CLIP encodes an image and each prompted class description, then compares their features to produce class probabilities and a predicted class with confidence.The visual and text encoders produce the respective features, and the text descriptions use a prompt prefix paired with each candidate class.Source: §3 · p. 3
  • Prompt tuning replaces hand-crafted prompts with trainable embeddings optimized by gradient descent; TPT performs this adaptation without labeled data.The cited prior CLIP prompt-tuning studies use 16 labeled samples per class from ImageNet.Source: §3 · p. 3
  • Calibration requires prediction confidence to match the empirical correctness rate, and ECE measures confidence-accuracy discrepancies across bins.Lower ECE indicates better alignment between predicted probabilities and the true probabilities of being correct.Source: §3 · p. 3
  • Figure 1 illustrates TPT’s calibration impact, prompt-dependent ECE variation at similar accuracy, and text-feature clustering patterns across prompts.The figure covers reliability diagrams, 80 hard prompts, and a t-SNE visualization.Source: Figure 1 · p. 4

4 REVISITING THE CALIBRATION OF CLIP MODELS

The section reports that TPT can increase calibration error, and that prompts with similar accuracy can differ in calibration. It links better calibration to greater dispersion among class-embedded text features.

4 REVISITING THE CALIBRATION OF CLIP MODELS

  • TPT improves accuracy over hard prompts but increases calibration error, consistent with its entropy-minimization objective and the risk of overconfidence.The paper connects this pattern to test-time adaptation methods that use prediction confidence as a self-supervision signal.Source: §4 · p. 4
  • Across 80 hard prompts, calibration varies notably even when accuracy is similar, so some prompts are better calibrated at comparable accuracy.The authors identify prompt choice as a potential guide for improving calibration during prompt optimization.Source: §4 · p. 4
  • At comparable accuracy, poorly calibrated prompts cluster text features while well-calibrated prompts show wider dispersion, suggesting dispersion as a regularizer for test-time optimization.The image feature remains unchanged across prompts, leading the authors to examine class-embedded text features.Source: §4 · p. 4
  • For hard prompts within 3% of each dataset’s highest accuracy, ATFD and ECE have negative associations across CLIP models and datasets.The figure caption reports average Pearson correlations of -0.70 for CLIP-RN50 and -0.76 for CLIP-ViT-B/16.Source: Figure 2 · p. 5

5 C-TPT: CALIBRATED TEST-TIME PROMPT TUNING

Section 5 formalizes the relationship between text-feature dispersion and calibration, then uses it to add a calibration objective to test-time prompt tuning. C-TPT maximizes ATFD alongside accuracy-oriented tuning.

5 C-TPT: CALIBRATED TEST-TIME PROMPT TUNING

  • The method builds on observations linking text-feature dispersion with calibration and introduces ATFD to examine that relationship.C-TPT is then introduced as a way to jointly train prompts with accuracy-oriented tuning.Source: §5 · p. 5

5.1 CORRELATION BETWEEN CALIBRATION AND TEXT FEATURE DISPERSION

  • For a prompt, the method first computes a centroid from the text features associated with the class descriptions.These features are t[p;y1], t[p;y2], …, t[p;yN].Source: §5.1 · p. 5
  • ATFD measures the mean L2 distance between each class text feature and that centroid.A larger ATFD indicates more dispersed text features; a smaller value indicates closer clustering.Source: §5.1 · p. 5
  • Across multiple datasets, ATFD and ECE showed a negative correlation when comparing prompts with similar accuracy.The analysis covered different CLIP models and datasets.Source: §5.1 · p. 5

5.2 C-TPT: CALIBRATED TEST-TIME PROMPT TUNING

  • C-TPT tunes prompts at test time to enhance calibration by spreading the text features for possible classes.It can be used jointly with prompt-tuning methods designed for accuracy.Source: §5.2 · p. 5
  • The objective combines TPT with a weighted C-TPT loss, defined as the negative of ATFD.The C-TPT loss operates on the class-embedded text representations.Source: §5.2 · p. 5

6 EXPERIMENTS

The experiments evaluate C-TPT with TPT on fine-grained classification and natural distribution shifts. Across both settings, the reported results show lower calibration error while retaining TPT’s accuracy benefits.

6 EXPERIMENTS

  • The study evaluates two settings: fine-grained classification and natural distribution shift.The shift benchmarks include ImageNet-V2, ImageNet-A, ImageNet-R, and ImageNet-Sketch.Source: §6 · p. 6, §6.2 · p. 7

6.1 FINE-GRAINED CLASSIFICATION

  • Evaluation covers 11 fine-grained datasets, including plants and animals, scenes, textures, food, transportation, actions, satellite imagery, and general objects.The experiments use CLIP-RN50 and CLIP-ViT-B/16.Source: §6.1 · p. 6
  • The fine-grained setup compares hard-prompt initialization and a four-prompt ensemble, with TPT applied alone or jointly with C-TPT.For natural shifts, the same configurations are used with λ set to 20.0.Source: §6.1 · p. 6, §6.2 · p. 7
  • Average ECE fell from 11.7 to 6.20 on CLIP-RN50 and from 11.6 to 5.13 on CLIP-ViT-B/16 with hard prompts, reductions of 47% and 56%.Across all fine-grained settings, C-TPT improved calibration without compromising TPT’s accuracy benefits.Source: §6.1 · p. 7, Table 1 · p. 7

6.2 NATURAL DISTRIBUTION SHIFTS

  • Under natural distribution shifts, ECE dropped 28% and 52% with hard prompts, and 14% and 24% with ensembles, for CLIP-RN50 and CLIP-ViT-B/16, respectively.C-TPT improved calibration over TPT while retaining TPT’s accuracy increase.Source: §6.2 · p. 7, Table 2 · p. 8

7 ABLATION STUDY

C-TPT reduces calibration error relative to temperature-scaled TPT in most instances, including lower average error without labeled training data. The visual analyses connect better calibration with dispersed text features that cluster near their corresponding class labels.

7.1 COMPARISON WITH PREVIOUS CALIBRATION METHOD

  • C-TPT achieves lower average calibration error than temperature-scaled TPT, even without labeled training data.Figure 4 compares TPT, temperature-scaled TPT, and their joint use with C-TPT on CLIP-ViT-B/16.Source: Figure 4 · p. 8, §7.1 · p. 8

7.2 WHY DOES TEXT FEATURE DISPERSION INFLUENCE CALIBRATION?

  • Well-calibrated hard prompts have dispersed text features that cluster by class, while poorly calibrated prompts cluster together across classes.The comparison uses hard prompts with similar accuracies but varying calibration.Source: §7.2 · p. 9, Figure 5 · p. 9
  • As C-TPT’s influence increases with λ values 45, 50, and 55, tuned text features become more dispersed and cluster by their respective class labels.Standard TPT features instead cluster with one another.Source: §7.2 · p. 9, Figure 5 · p. 9
  • C-TPT guides prompt optimization toward better calibration by increasing text feature dispersion, a property difficult to control with embedding-based prompts.With hard prompts, users can select prompts that are not overly descriptive.Source: §7.2 · p. 9

8 CONCLUSION

The conclusion presents C-TPT as a way to improve calibration during CLIP prompt tuning without labeled data, using text-feature dispersion as its guiding signal.

8 CONCLUSION

  • Across various datasets, C-TPT prioritizes Average Text Feature Dispersion during tuning and refines prediction calibration without labeled data.The conclusion describes well-calibrated prompts as having high feature dispersion near class labels, unlike poorly calibrated prompts with low dispersion and clustered features.Source: §8 · p. 9

10 ETHICS STATEMENT

The ethics statement reports that the work is original, introduces no new datasets, and uses no demographic or identity characteristics in experiments.

10 ETHICS STATEMENT

  • The study introduces no new datasets and reports no experiments using demographic or identity characteristics.The authors state that the work is original and external contributions or sources are cited.Source: §10 · p. 10

11 REPRODUCIBILITY STATEMENT

The reproducibility statement points to the paper’s training procedures, hyperparameters, experimental settings, and appendix prompt list.

11 REPRODUCIBILITY STATEMENT

  • Training procedures, hyperparameters, and experimental settings are detailed in Sections 6.1 and 6.2, with hard prompts listed in Appendix A.3.Source: §11 · p. 10

A.1 BROADER IMPACT

The appendix examines C-TPT’s broader impact, limitations, and behavior across additional test-time and training-time settings. It reports calibration improvements alongside trade-offs and scope constraints.

A.1 BROADER IMPACT

  • The paper frames label-free calibration as relevant to trustworthiness and safety in settings where labeled data are scarce.Healthcare and autonomous-vehicle applications are cited as examples of decision-making uses of CLIP.Source: §A.1 · p. 15

A.2 LIMITATIONS

  • Regularization-based calibration can involve accuracy degradation in some cases, although the reported reductions are minor.The paper relates this trade-off to trainable calibration methods, which typically report less than 1–2% degradation.Source: §A.2 · p. 15
  • Limited access to training or validation data for hyperparameter tuning constrains test-time adaptation; the fixed λ may not be optimal for every sample.The study uses λ=50 as a pragmatic fixed choice and suggests per-sample adaptation as future work.Source: §A.2 · p. 15

A.4 CALIBRATION OF CLIP MODELS

  • Across 11 datasets, hard prompts can produce different calibration errors at similar accuracy levels.Figure 6 visualizes prompt-level ECE and accuracy for CLIP-ViT-B/16.Source: §A.4 · p. 16, Figure 6 · p. 16

A.6 INFORMATION ON THE CHOICE OF LAMBDA

  • Across validation datasets, ECE and ATFD appear to saturate at λ=50.0, while accuracy is not significantly affected, though some cases show minor degradation of approximately 1% (CLIP-RN50).Source: Figure 7 · p. 17

A.8 PARETO FRONT: VISUALIZING THE EFFECT OF C-TPT

  • C-TPT tends to form an accuracy–ECE Pareto front, while varying λ moves solutions along it and hard prompts lie outside it.Figure 8 compares hard prompts, TPT, and TPT+C-TPT, including one initialization under varying λ.Source: §A.8 · p. 19, Figure 8 · p. 19

A.10 ADDITIONAL EXPERIMENTAL RESULTS: APPLICATION TO DIFFERENT TEST-TIME PROMPT TUNING METHODS

  • C-TPT reduces PromptAlign’s calibration error while retaining its accuracy increase in the reported test-time evaluations.The appendix evaluates fine-grained classification and natural distribution shifts.Source: §A.10 · p. 22, Table 9 · p. 22, Table 10 · p. 22

A.12 APPLICABILITY OF C-TPT ON TRAINING-TIME PROMPT TUNING METHOD

  • ImageNet prompt-dispersion rankings correlate with rankings on other datasets (mean Spearman 0.92), and CoOp+C-TPT is reported to improve calibration in a training-time evaluation.The CoOp evaluation uses CLIP-RN50 and reports accuracy and ECE.Source: §A.12 · p. 25, Figure 9 · p. 25, Table 16 · p. 25

Related papers

Loading 2403.14119v3…