PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking the Test-Time Prompt Tuning Objective from the Perspective of Calibration

Jungwon Choi, Hyeonseo Jang, Kibok Lee, Eunwoo Kim

arXiv:2608.30230v1cs.AI

TL;DR

The paper addresses calibration degradation in test-time prompt tuning caused by entropy minimization’s tendency to produce overconfident predictions. It aligns original-view predictions with confidence-aware augmented-view targets, using target uncertainty and adaptive temperature scaling, and reports improved calibration with strong accuracy across benchmarks.

  • Problem

    Entropy minimization in test-time prompt tuning sharpens predictions regardless of sample-specific uncertainty, creating calibration degradation and misalignment between confidence and correctness.

  • Method

    The method aligns the original-view prediction with an augmented-view target, modulates matching using target entropy, and applies confidence-aware temperature scaling before aggregation.

  • Results

    The approach achieves state-of-the-art calibration and downstream performance, including 65.71% average accuracy and 3.65 average ECE compared with TPT’s 64.62% accuracy and 11.67 ECE.

  • Takeaways & Limitations

    The method improves calibration while maintaining strong recognition performance and does not require additional regularization to compensate for EM’s side effects.

  • Takeaways & Limitations

    Experiments primarily use CLIP-based vision-language models, leaving broader VLM architectures and extremely large-scale or real-time deployment for future study.

Abstract

from arXiv · show

Test-time prompt tuning (TPT) has emerged as a powerful paradigm, refining prompts for each test sample via entropy minimization (EM) over multiple augmented views. However, we identify a limitation in the standard EM-based adaptation: it inherently drives the model toward overconfident predictions disregarding sample-specific uncertainty, leading to significant calibration degradation. To address these limitations, we propose a new objective that replaces the conventional EM loss by aligning the original-view prediction with a target distribution derived from augmented views via cross-entropy, while adversarially incorporating the entropy of the target distribution to capture sample-specific uncertainty. Furthermore, to better construct this target distribution, we apply confidence-aware temperature scaling to each augmented-view prediction according to its confidence, sharpening confident predictions while softening uncertain ones. This formulation allows the model to increase confidence only when the target distribution is reliable, while preserving uncertainty when it reflects ambiguous or conflicting augmented-view predictions. Extensive experiments across diverse benchmarks demonstrate that our approach not only achieves state-of-the-art accuracy but also significantly improves model calibration.

1 Introduction

Test-time prompt tuning adapts vision-language prompts without labels, but its entropy-minimization objective can sharpen predictions indiscriminately and degrade calibration. The paper replaces this objective with uncertainty-aware alignment between original-view predictions and augmented-view targets.

  • Background: Test-time prompt tuning adapts prompts at inference using only unlabeled test samples and multiple augmented views.It builds on vision-language models that align image and text representations in a shared embedding space.
  • Problem: Entropy minimization consistently encourages sharper probabilities regardless of each test sample’s characteristics.This tendency may improve accuracy but can increase the mismatch between correctness and confidence.
  • Problem: Calibration degradation is concerning because confidence estimates should reflect the likelihood that predictions are correct, especially in safety-critical settings.
  • Problem: Prior calibration methods retain the original EM loss and mitigate its side effects with regularization rather than resolving its underlying source.Standard EM also operates on averaged augmented-view predictions, while test-time confidence is determined by the final prediction.
  • Method: The proposed objective aligns the original-view prediction with an augmented-view target while using augmentation-derived uncertainty to modulate adaptation strength.Confidence-aware temperature scaling is applied to augmented-view predictions to construct a more reliable target.
  • Contribution: Experiments report state-of-the-art calibration and downstream task performance without additional regularization for EM side effects.

2 Related Work

Related work covers prompt adaptation, neural-network calibration, and vision-language-model calibration. Test-time prompt tuning uses unlabeled augmented views, while existing calibration strategies include post-hoc, train-time, and regularization-based approaches.

  • Prompt Learning: Test-time prompt tuning updates prompt parameters using unlabeled test samples and multiple augmented views, unlike earlier prompt-learning methods that use labeled training data.
  • Calibration of Deep Neural Networks: Calibration measures how closely predicted confidence aligns with empirical accuracy and supports applications requiring reliable uncertainty estimates.
  • Calibration of Deep Neural Networks: Post-hoc calibration techniques cannot be directly applied to test-time adaptation when only unlabeled test samples are available.
  • Calibration of VLMs: Vision-language models often become overconfident after domain adaptation, and TPT can exacerbate this issue through entropy minimization.Recent methods address calibration through strategies such as text-feature dispersion and class-feature separation.

3 Preliminaries

The preliminaries define zero-shot classification with vision-language embeddings and describe TPT’s entropy minimization over selected augmented views. Figure 1 contrasts this procedure with the proposed confidence-aware objective.

  • Zero-shot Classification: Zero-shot classification embeds class names into textual prompts and compares them with visual image embeddings in a shared space.
  • Zero-shot Classification: The temperature parameter τ controls the sharpness of the prediction distribution.
  • Zero-shot Classification: The final predicted class is the class with maximum probability, and its maximum probability is the associated confidence.
  • Test-time Prompt Tuning: TPT adapts prompt parameters without labels by minimizing prediction entropy across multiple augmented views of one test image.It filters views using a confidence threshold and aggregates probabilities from selected low-entropy views.
  • Test-time Prompt Tuning: TPT minimizes the entropy of the aggregated augmented-view distribution over K classes.This objective encourages consistent and confident predictions across augmentations.
  • Comparison: Figure 1 contrasts EM’s possible misalignment and overconfident errors with the proposed method’s confidence-aware targets and smoother predictions for ambiguous samples.

4 Analysis of Uncertainty-Aware Adaptation Objective

The analysis identifies misalignment between TPT’s augmented-view objective and the original-view inference prediction. It therefore matches the original prediction to an uncertainty-aware augmented target, preserving softness when views are ambiguous.

  • Objective Mismatch: The original-view prediction p determines inference-time predictions and confidence, whereas p_aug is the averaged target from multiple augmented views.Standard TPT minimizes H(p_aug) without explicitly optimizing p, allowing the two distributions to remain misaligned.
  • Cross-Entropy Matching: Cross-entropy matching encourages the original prediction p to assign high probability to classes favored by p_aug.This directly transfers class-level information estimated from multiple augmented views into inference probabilities.
  • Target Uncertainty: Consistent augmented views produce sharp averaged targets, while ambiguous or inconsistent views produce soft targets that encode sample-dependent uncertainty.Preserving this uncertainty prevents ambiguous targets from making p overconfident.
  • Uncertainty Modulation: Unlike conventional EM, the proposed objective uses target entropy to reduce matching pressure for soft targets rather than sharpening them.Sharp targets retain a strong matching signal, while uncertain targets exert a weaker one.
  • Objective Formulation: The objective is H(p_aug,p) - H(p_aug), with target entropy counteracting cross-entropy’s tendency to produce overconfident predictions.The resulting formulation allows p to follow confident augmented targets while remaining softer for uncertain targets.
  • Comparison: Compared with regularized calibration methods that retain EM, this approach preserves uncertainty within the objective itself.

5 Confidence-aware Adaptive Temperature Scaling

Confidence-aware adaptive temperature scaling adjusts each augmented view according to its confidence before averaging, producing sharper targets for confident views and softer targets for uncertain ones.

  • Motivation: Directly averaging augmented-view probabilities treats all views equally despite differences in confidence.The method addresses this by adjusting each view before aggregation.
  • Confidence-aware scaling: Confidence-aware adaptive temperature scaling assigns view-specific temperatures based on each view’s maximum predicted probability.Confidence is estimated as α_i = max_k p(k), and temperature is linearly interpolated between Tmin and Tmax.
  • Confidence-aware scaling: Confident predictions receive sharper distributions, whereas uncertain predictions receive smoother distributions.This preserves differences in reliability across augmented views before constructing the target.
  • Target construction: The final target distribution is formed by averaging the calibrated soft labels from all augmented views.The number of augmented views is denoted by N.
  • Target alignment: The method minimizes KL divergence between the aggregated augmented-view target and the original-image prediction.The objective is written as LKLD = DKL(paug ∥ p).

6 Experiments

Across fine-grained classification, prompt initializations, DynaPrompt, and natural distribution shifts, the method improves calibration while maintaining or improving accuracy. Analyses attribute this behavior to uncertainty-sensitive confidence control rather than indiscriminate sharpening.

  • Fine-Grained Classification: 65.71% average accuracy and 3.65 average ECE outperform TPT, C-TPT, and O-TPT on ten fine-grained classification benchmarks.The corresponding baselines achieve 64.62%/11.67 ECE, 64.48%/5.32 ECE, and 64.12%/4.46 ECE, respectively.
  • Pre-trained Prompts: Under CoOp and MaPLe initializations, the method achieves the best reported calibration while also attaining the highest average accuracy in each setting.CoOp records 4.63 ECE and 63.99% accuracy; MaPLe records 3.28 ECE and 66.05% accuracy.
  • Different TPT Framework: 65.51% accuracy and 4.04 average ECE demonstrate strong performance within DynaPrompt, with calibration gains across most datasets.The method outperforms DynaPrompt, C-TPT, and O-TPT in accuracy by 1.29%, 1.75%, and 2.16%, respectively.
  • Natural Distribution Shifts: On natural distribution shifts, the method lowers average ECE from TPT’s 11.99 to 3.55 while closely matching the best-performing baseline in accuracy.The evaluation uses ImageNet and ImageNet-A, ImageNet-V2, ImageNet-R, and ImageNet-Sketch.
  • Ablation Study: The ablation shows complementary contributions from temperature scaling and the KLD objective, with their combination producing the best results.Temperature scaling alone changes accuracy from 64.52% to 64.77% and ECE from 9.80 to 8.70; KLD alone reaches 64.97% accuracy and 6.18 ECE.
  • Confidence and Entropy Analyses: Unlike EM, the method preserves confidence for correct samples and increases uncertainty for incorrect samples after adaptation.Mean entropy changes are approximately −0.25 for correct samples and +0.14 for incorrect samples, whereas EM reduces entropy for both groups.

7 Conclusion

The paper concludes that confidence-aware alignment improves reliability while maintaining competitive accuracy, but evaluation remains limited mainly to CLIP-based models and test-time inference.

  • Figure 6 examines prediction discrepancy between augmented and original views.
  • The method consistently achieves better reliability while maintaining competitive accuracy across diverse benchmarks.
  • The experiments primarily evaluate CLIP-based vision-language models, leaving broader VLM architectures for future work.
  • The adaptation process focuses on test-time inference, while scalability to extremely large-scale batches or real-time streaming data remains unexplored.
Loading 2608.30230v1…