PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling Image Prototypes for Guided Test-Time Adaptation

Liwen Wang, Xingbo Dong, Iman Yi Liao, Deyin Liu, Massimo Tistarelli, Lin Yuanbo Wu, Zhe Jin

arXiv:2609.09737v1cs.CVcs.AI

TL;DR

Test-time adaptation must limit pseudo-label error accumulation and catastrophic forgetting, but existing uncertainty estimates can be overconfident or costly, while static prototypes misalign during adaptation. DIPTTA introduces regenerative synthetic image prototypes for dynamic replay and source-calibrated uncertainty estimation, and experiments report strong performance across benchmarks, especially under severe domain shifts.

  • Problem

    TTA must address error accumulation from noisy pseudo-labels and catastrophic forgetting, while existing uncertainty and static-prototype methods have important limitations.

  • Method

    DIPTTA distills a compact set of synthetic source images that regenerates aligned feature prototypes and anchors source-calibrated uncertainty estimation.

  • Results

    Extensive experiments on multiple benchmarks show that DIPTTA significantly outperforms state-of-the-art methods, particularly under strong domain shifts.

  • Takeaways & Limitations

    DIPTTA provides a dynamic source-knowledge anchor for replay and uncertainty estimation without relying on static feature representations.

Abstract

from arXiv · show

Test-Time Adaptation (TTA) enhances the robustness of models against distribution shifts but faces two critical challenges: error accumulation from noisy pseudo-labels and catastrophic forgetting of source knowledge. Uncertainty-based approaches designed to mitigate error accumulation often yield overconfident or computationally expensive estimates, while strategies intended to prevent forgetting via prototype replay rely on static representations that easily become misaligned as the model adapts. To address these issues, this paper proposes a novel framework, Distilling Image Prototype for Guided Test-Time Adaptation (DIPTTA). The core of the proposed approach is the introduction of a Distill Image Prototype (DIP), a compact set of synthetic images that serves as a dynamic and regenerative anchor of source knowledge. This prototype enables a dynamic feature replay mechanism that continuously generates feature prototypes aligned with the current state of the model, thus effectively preventing catastrophic forgetting. Furthermore, the DIP anchors a source-calibrated uncertainty estimation method, which provides a less biased measure of sample reliability by leveraging stable source knowledge, thereby robustly suppressing error accumulation. Extensive experiments on multiple benchmarks demonstrate that DIPTTA significantly outperforms state-of-the-art methods, particularly under severe domain shifts. The source code is available at https://github.com/LiwenWang919/DIPTTA.

I. INTRODUCTION

DIPTTA addresses error accumulation and catastrophic forgetting in test-time adaptation by using a regenerative image-space source anchor. It combines dynamic feature replay with source-calibrated uncertainty estimation and reports strong benchmark performance, especially under severe shifts.

  • TTA must control noisy pseudo-label errors while preserving source knowledge under distribution shifts.
  • Static feature prototypes progressively misalign with the evolving feature space, weakening their role as forgetting-prevention anchors.
  • DIP is a compact set of synthetic images distilled from the source dataset that can regenerate features through the updated model.
  • DFPR regenerates feature prototypes for each target batch and uses them in contrastive learning to retain source knowledge during adaptation.
  • SCUE uses DIP-derived source knowledge to provide a more reliable signal for weighting target samples and reducing error accumulation.
  • Extensive experiments on multiple benchmarks report state-of-the-art performance, with consistent improvements under strong domain shifts.

B. Uncertainty Quantification in Test Time Adaptation

Uncertainty estimation in TTA must identify unreliable target predictions without becoming overconfident, computationally prohibitive, or biased by the changing target domain. DIPTTA introduces a source-calibrated reference through its image prototypes.

  • DIPTTA anchors uncertainty estimation in source knowledge represented by synthetic image prototypes.
  • Entropy-based uncertainty measures are lightweight but can be overconfident and miscalibrated on out-of-distribution data.
  • Bayesian uncertainty methods capture epistemic uncertainty more formally but are often computationally expensive for online TTA.
  • Target-only uncertainty estimates lack a stable source reference while both the model and target distribution change during adaptation.

1) Model Architecture:

DIPTTA combines mean-teacher online adaptation with image-space prototype distillation. Synthetic images are optimized through simulated training and source-data meta-optimization, then regenerated under the current model state.

  • The student adapts on each unlabeled target batch, while the teacher tracks the student through exponential moving average updates.
  • DIP shifts the knowledge anchor from frozen feature vectors to synthetic images whose features can be regenerated as the extractor changes.
  • A small, randomly initialized synthetic set is optimized separately for each class using a bi-level procedure.
  • The inner loop updates a virtual model on synthetic mini-batches using a simulated training loss.
  • The outer loop optimizes synthetic images according to the virtually trained model’s performance on source-data mini-batches.
  • The optimization can match feature distributions, enforce classifier consistency, and apply privacy-friendly pixel or smoothness regularization.

2) Dynamic Feature Prototype Generation:

DIPTTA dynamically regenerates class feature prototypes by passing distilled synthetic images through the current student model. These prototypes guide contrastive learning and anchor target features to source knowledge.

  • Dynamic prototype generation: At each adaptation step, the current student feature extractor processes the distilled synthetic images to generate class-wise feature prototypes.The prototypes are regenerated as the model changes, maintaining alignment with its evolving feature space.
  • Contrastive guidance: For each target sample, similarity is computed between its student features and all dynamically generated class prototypes.The pseudo-label predicted by the teacher identifies the target class used by the contrastive objective.
  • Contrastive guidance: The resulting contrastive learning strategy alleviates feature uncertainty in the target domain through dynamically updated prototypes.
  • Contrastive guidance: The contrastive loss pulls pseudo-labeled target features toward their assigned prototypes while pushing them away from other class prototypes.This anchors the adapted feature space to source knowledge represented through the current model.

C. Source-Calibrated Uncertainty Estimation

DIPTTA estimates target-sample reliability using source-calibrated uncertainty anchored by the DIP. It approximates this calibration efficiently and uses the resulting weights to reduce the influence of uncertain samples in adaptation losses.

  • Source-calibrated uncertainty: The DIP provides a stable source anchor for uncertainty estimation, unlike methods that rely solely on target predictions.The anchor is intended to provide less biased uncertainty quantification during domain-shifted adaptation.
  • Source-calibrated uncertainty: Source calibration evaluates target predictions together with their alignment to the source anchor, reducing confidence when target features diverge from source knowledge.Alignment with the source anchor instead increases calibrated confidence.
  • Source-calibrated uncertainty: The calibrated uncertainty U(x_t) is defined as the entropy of the source-conditioned predictive distribution and is intended to remain reliable under domain shift.
  • Efficient approximation: Because full parameter-space integration is computationally intractable online, DIPTTA uses a Laplace approximation centered at the MAP estimate and focuses Bayesian modeling on the classification head.The feature extractor remains deterministic, while Kronecker-factored Laplace approximation improves computational feasibility.
  • Uncertainty weighting: The exponentiated negative entropy of the mean predictive probability yields a sample weight that down-weights highly uncertain target examples.These weights are applied to consistency and contrastive loss components.

D. Overall Optimization Objective

DIPTTA optimizes a weighted combination of replay, contrastive, and consistency losses. The replay term preserves source knowledge while the other terms guide adaptation and prediction agreement.

  • Loss composition: The total objective combines Consistency Loss, Contrastive Loss, and Replay Loss.Replay operates on source-driven feature prototypes to prevent catastrophic forgetting.
  • Loss composition: The three losses are combined as Ltotal = λ1LReplay + λ2LContrastive + λ3LConsistency.The coefficients control each loss term's contribution to optimization.
  • Loss weighting: Across experiments, the selected weights are λ1 = 0.5, λ2 = 0.25, and λ3 = 0.15.

IV. EXPERIMENTS

DIPTTA is evaluated for continual test-time adaptation across corruption and disturbance benchmarks under an online, source-free setting. The experiments compare its classification error with existing TTA methods.

  • Benchmark evaluation: The evaluation covers CIFAR-10-C, CIFAR-100-C, ImageNet-C, ImageNet-R, and the CCC benchmark.These tasks assess robustness to corruptions and disturbances in input data.
  • Evaluation setting: All experiments perform online adaptation without source-data access and use corruption severity level five.The study employs standard pre-trained Wide-ResNet, ResNeXt-29, and ResNet-50 models.
  • Comparison with baselines: The comparison includes entropy minimization and test-time batch normalization among the state-of-the-art TTA methods.
  • Comparison with baselines: Table I reports classification error rates on CIFAR10-C, CIFAR100-C, and TinyImageNet-C under TTA, with bold values marking the best results.

1) Comparison on CIFAR10 to CIFAR10-C:

Across CIFAR corruption benchmarks, DIPTTA achieves the lowest reported error rates and consistently outperforms competing continual test-time adaptation methods, especially under severe shifts.

  • CIFAR10-C: DIPTTA attains a 16.4% mean classification error on CIFAR10-C, beating AEA at 17.2% and Source Only at 43.5%.It performs best on 10 of 15 corruption types, including Gaussian, Shot, and Impulse noise.
  • CIFAR100-C: DIPTTA achieves a 30.4% mean error on CIFAR100-C, surpassing AEA at 30.9% and EATA at 31.1%.It is particularly strong on Contrast at 21.6% and Elastic transform at 24.7%.
  • TinyImageNet-C: DIPTTA records a 60.1% mean error on TinyImageNet-C, improving over Source Only at 83.5% and AEA at 61.0%.The lowest errors occur across corruptions including Glass blur, Snow, and Pixelate.
  • Continual TTA: Under CTTA on CIFAR10-C, DIPTTA reaches a 13.9% mean error, outperforming RMT at 16.7%, BeCoTTA at 16.3%, and TCA at 14.7%.The comparison includes entropy regularization, contrastive learning, self-training, and pseudo-label filtering methods.
  • Continual TTA: On CIFAR100-C CTTA, DIPTTA reduces mean error to 27.8%, compared with Source Only at 46.4% and Tent at 60.9%.The result addresses degradation during long sequential adaptation and exceeds prior methods described in the passage.

3) Comparison on ImageNet to ImageNet-C:

DIPTTA remains competitive across ImageNet-scale and long-sequence evaluations, with leading results on ImageNet-C, CCC, and ImageNet-R and evidence supporting dynamic prototype replay.

  • ImageNet-C: DIPTTA achieves a 58.4% mean error on ImageNet-C, outperforming TCA at 59.3% and BeCoTTA at 60.9%.CoTTA reaches 62.7%, while the source-only model begins at 82.0%.
  • ImageNet-C: DIPTTA reduces ImageNet-C error to 58.4% despite the source-only model’s initial 82.0% error under severe corruptions.The benchmark evaluates scalability and robustness under large-scale corruption shifts.
  • CCC benchmark: On CCC, DIPTTA leads CCC-Easy at 51.2% and CCC-Medium at 40.7%, with 33.8% average accuracy across all three difficulties.TCA performs best on CCC-Hard, but DIPTTA has the best overall average.
  • ImageNet-R: On ImageNet-R, DIPTTA reaches 52.7% error with ResNet50 and 35.3% with ViT.The passage attributes this performance to distilled prototypes capturing abstract class knowledge rather than only superficial features.
  • Dynamic prototype replay: In CTTA ablations, DIPTTA reaches 13.9% error on CIFAR10-C versus 16.7% for RMT, which uses static prototypes.The comparison supports regenerating prototypes so they remain aligned with the evolving model and avoid an obsolete knowledge anchor.

2) Advantage of Source-Calibrated Uncertainty:

Source-calibrated uncertainty and the overall DIPTTA design improve reliability while preserving computational efficiency, with robustness across batch sizes.

  • Source-calibrated uncertainty: DIPTTA outperforms entropy-based methods such as Tent and EATA on distribution-shifted data, including ImageNet-R.The passage attributes the advantage to calibrating uncertainty against stable source knowledge.
  • Ablation: On CIFAR10-C, SCUE alone lowers error from 18.3% to 17.2%, while DFPR alone lowers it to 16.9%.The ablation evaluates the standalone contribution of each module.
  • Efficiency: On ImageNet-C, DIPTTA takes 12.235 seconds per batch versus 11.773 seconds for RMT.The small overhead contrasts with computationally intensive methods such as MEMO and AEA.

2) Sensitivity Analysis of Batch Size:

DIPTTA remains robust across batch sizes and shows stable performance across the tested range. Its source-aligned feature representations also cluster more tightly around source prototypes than competing methods.

  • Batch-size sensitivity: DIPTTA consistently outperforms competitors, including at batch size 1, demonstrating robustness to batch-size variation.On CIFAR benchmarks, its optimal error rates are 13.9% and 27.8% at batch size 200, with stability across batch sizes 50–300.
  • Batch-size sensitivity: DIPTTA achieves CIFAR error rates of 13.9% and 27.8% at batch size 200.The method remains stable across the 50–300 batch-size range, unlike volatile methods such as TCA.
  • Hyperparameter sensitivity: The selected hyperparameter set is (0.5, 0.25, 0.15) for all experiments because the CIFAR10-C and CIFAR100-C optima are close.The dataset-specific optima are (0.5, 0.25, 0.15) and (0.45, 0.2, 0.2), respectively.
  • Feature-space visualization: On CIFAR10-C, DIPTTA features form tighter clusters around source-domain prototypes than those of other methods.The comparison uses t-SNE visualizations of randomly selected features from Gaussian and Fog domains.
Loading 2609.09737v1…