PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Test-Time Model Adaptation without Forgetting

Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, Mingkui Tan

arXiv:2204.02610v2cs.LG

TL;DR

TTA must adapt models under distribution shifts without incurring prohibitive per-sample computation or forgetting in-distribution knowledge. EATA selectively updates on reliable, non-redundant samples and uses Fisher regularization to protect important parameters; experiments demonstrate improved efficiency and reduced ID performance degradation across benchmark datasets.

  • Problem

    Existing TTA methods require costly backward computation for each test sample and can degrade in-distribution performance while improving out-of-distribution performance.

  • Method

    EATA combines sample-adaptive entropy minimization that filters high-entropy and redundant samples with Fisher-based regularization of important model parameters.

  • Results

    Experiments on CIFAR-10-C, ImageNet-C, and ImageNet-R demonstrate that EATA improves TTA efficiency and alleviates catastrophic forgetting.

  • Takeaways & Limitations

    Selective adaptation and Fisher regularization make continual test-time adaptation more practical for settings containing both OOD and ID samples.

  • Takeaways & Limitations

    With batch size B = 1, directly applying Tent and ETA may fail, although a sliding window makes ETA work well and consistently outperform Tent.

Abstract

from arXiv · show

Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and testing data by adapting a given model w.r.t. any testing sample. This task is particularly important for deep models when the test environment changes frequently. Although some recent attempts have been made to handle this task, we still face two practical challenges: 1) existing methods have to perform backward computation for each test sample, resulting in unbearable prediction cost to many applications; 2) while existing TTA solutions can significantly improve the test performance on out-of-distribution data, they often suffer from severe performance degradation on in-distribution data after TTA (known as catastrophic forgetting). In this paper, we point out that not all the test samples contribute equally to model adaptation, and high-entropy ones may lead to noisy gradients that could disrupt the model. Motivated by this, we propose an active sample selection criterion to identify reliable and non-redundant samples, on which the model is updated to minimize the entropy loss for test-time adaptation. Furthermore, to alleviate the forgetting issue, we introduce a Fisher regularizer to constrain important model parameters from drastic changes, where the Fisher importance is estimated from test samples with generated pseudo labels. Extensive experiments on CIFAR-10-C, ImageNet-C, and ImageNet-R verify the effectiveness of our proposed method.

1. Introduction

Test-time adaptation addresses distribution shifts but must overcome costly per-sample backpropagation and catastrophic forgetting on in-distribution data. EATA addresses both through selective sample adaptation and Fisher-based parameter regularization.

  • Motivation: Test-time adaptation targets distribution shifts that arise when real-world test samples differ from training data.Such shifts include natural variations and corruptions such as lighting changes and sensor noise.
  • Challenges: Prior methods require backward computation for each test sample, making adaptation costly in latency-sensitive applications.Some methods perform multiple backward passes for a single sample.
  • Challenges: Existing methods can improve out-of-distribution performance while severely degrading performance on in-distribution samples.A practical method should handle streams containing both ID and OOD samples.
  • Proposed Method: EATA excludes high-entropy and redundant samples from optimization, reducing backward updates while improving OOD performance.The sample-efficient strategy gives reliable, non-redundant samples greater influence during adaptation.
  • Proposed Method: EATA uses a Fisher-information regularizer to constrain important weights and prevent performance degradation on ID samples.Fisher importance is estimated from a small set of test samples.

2. Related Work

Related work spans test-time adaptation, continual learning, and unsupervised domain adaptation. EATA performs online adaptation with selective backward propagation and an anti-forgetting regularizer while avoiding costly storage of all previous outputs.

  • Test-Time Adaptation: Test-time training methods modify source-model training with supervised and self-supervised objectives before adapting through self-supervision at test time.This pipeline may require training assumptions that are not controllable in practice.
  • Continual Learning: Continual learning shares EATA’s focus on preserving essential previously learned knowledge and mitigating catastrophic forgetting.In EATA, forgetting refers to degraded performance on in-distribution test samples.
  • Unsupervised Domain Adaptation: Unsupervised domain adaptation commonly optimizes a source model with labeled source data and unlabeled target data, whereas source-free methods avoid source-data access.The supplied related-work passage notes generative-modeling and information-maximization approaches.
  • EATA: EATA selects active samples for backward propagation and regularizes parameters to keep them close to their initial values.The method computes a sample-adaptive weight S(x) and constrains changes from Θo.
  • EATA: Unlike offline approaches that process datasets over multiple epochs, EATA adapts online with selective backward propagation for incoming target samples.This design is intended to reduce inference-time computation.

3. Problem Formulation

The problem formulation considers models trained on a distribution P(x) but tested under a potentially different distribution Q(x). TTA adapts the model using test data, yet existing approaches are costly and can harm ID accuracy when streams mix ID and OOD samples.

  • Setting: A base model trained on P(x) is expected to perform well on in-distribution test data drawn from the same distribution.The model parameters are denoted Θo.
  • Setting: Distribution shift occurs when test samples are drawn from Q(x) with Q(x) ≠ P(x), causing unreliable predictions and poor performance.The formulation attributes this mismatch to possible differences between training and test data.
  • Test-Time Adaptation: TTA adapts a model using test samples alone to improve prediction performance on an out-of-distribution test domain.Existing objectives include entropy minimization and prediction-consistency maximization under augmentations.
  • Limitations: Existing TTA methods may require one or multiple backward computations per sample, making them unsuitable for latency-sensitive applications.They also commonly assume that all test samples are out of distribution.
  • Limitations: Optimizing only on OOD samples can severely reduce accuracy on ID samples when a test set contains both domains.The updated model is reported to have consistently lower ID accuracy in the cited empirical validation.

4. Proposed Methods

EATA combines active test-sample selection with Fisher-weight regularization to make entropy-minimization adaptation more efficient while reducing catastrophic forgetting. It selects reliable, diverse samples for updates and constrains important parameters from changing excessively.

  • Sample-efficient entropy minimization: EATA selects reliable, non-redundant test samples for entropy-minimization updates, reducing unnecessary backward computation.Samples with high entropy or excessive similarity are excluded from optimization.
  • Reliable sample identification: Low-entropy samples contribute more to adaptation, whereas high-entropy samples may hurt performance because their gradients can be biased and unreliable.The preliminary study evaluates models adapted on subsets sorted by prediction entropy.
  • Reliable sample identification: The entropy-based weight excludes samples with entropy above E0 and assigns greater update weight to samples with lower prediction uncertainty without back-propagation.Sent(x) uses an indicator threshold and entropy-dependent weighting.
  • Non-redundant sample identification: EATA removes redundant samples by comparing each prediction with an exponential moving average of previous adaptation-sample outputs.Cosine similarity between the incoming prediction and the moving average determines a diversity-based weight.
  • Anti-forgetting weight regularization: A weighted Fisher regularizer limits changes to parameters important for the in-distribution domain, addressing forgetting without additional adaptation computation.Fisher importance is estimated with a diagonal Fisher information matrix and the regularizer penalizes deviations from the original parameters.

5. Experiments

Experiments across CIFAR-10-C, ImageNet-C, and ImageNet-R show that EATA improves test-time adaptation efficiency and OOD performance while mitigating ID forgetting. Ablations and additional analyses support its active sample selection and Fisher regularization strategies.

  • Comparisons of OOD Performance and Efficiency: EATA consistently improves ImageNet-C classification error across all 15 corruption types, while preserving OOD performance with anti-forgetting regularization.ETA improves over Tent on Gaussian noise from 71.6% to 65.0% error, and EATA maintains comparable OOD performance while preventing ID forgetting.
  • Comparisons of OOD Performance and Efficiency: 26,031 average backward passes make ETA more efficient than Tent, TTT, and MEMO on ImageNet-C.Compared with Tent’s 50,000 backward passes, ETA excludes high-entropy and similar samples from optimization.
  • Demonstration of Preventing Forgetting: EATA maintains clean ID accuracy while preserving corruption accuracy during both reset and lifelong adaptation on ImageNet-C.During lifelong adaptation, Tent’s clean and corruption accuracy rapidly degrades, whereas EATA remains stable across corruption types.
  • Ablation Studies: 65.7% error with 26,694 backward passes improves over the baseline’s 71.6% error and 50,000 backward passes, while redundancy filtering further reduces backward passes to 19,121.Removing similar samples preserves comparable OOD error, reducing it from 65.7% to 65.0%.
  • Ablation Studies: EATA performs best with E0 in [0.4, 0.5] × ln 103, and Fisher estimation is stable with at least 300 unlabeled ID samples.The authors set E0 to 0.4 × ln 103 for the efficiency-performance trade-off; with Q ≥ 300, clean accuracy is much higher than without regularization.
  • More Discussions: EATA achieves 113s to 102s wall-clock time and requires only 9.8 MB extra GPU memory on the reported ResNet-50 ImageNet-C setting.The reported comparison is 28.6% accuracy for Tent versus 35.1% for EATA; the implementation could be further accelerated by avoiding full-batch gradient computation.
  • More Discussions: EATA remains effective under mixed shifts, large backbone models, and varying numbers of test samples, with larger gains when more test samples are available.The mixed-distribution evaluation includes 15 corruption types and 750k images; EATA consistently outperforms Tent across tested sample counts.

6. Conclusion

EATA combines selective entropy-minimization updates with a Fisher-based anti-forgetting regularizer for test-time adaptation. Experiments on benchmark datasets support improved efficiency, OOD performance, and continual adaptation without ID performance degradation.

  • EATA selectively adapts models using reliable and non-redundant test samples, improving adaptation efficiency and OOD performance.The method uses a sample-efficient entropy minimization strategy.
  • A Fisher-based anti-forgetting regularizer constrains adaptation so models can continue adapting without performance degradation on in-distribution test samples.The regularizer is incorporated into test-time adaptation.
  • Extensive experiments on several benchmark datasets demonstrate the effectiveness of EATA.

Supplementary Materials for “Efficient Test-Time Model Adaptation without Forgetting”

The supplementary material provides additional experimental details, comparisons, anti-forgetting results, and discussion of related training-time robustification studies.

  • Section A: Section A provides additional experimental details for EATA.
  • Section B: Section B compares EATA’s OOD performance and efficiency with state-of-the-art methods on ImageNet-C across corruption types and severity levels.
  • Section C: Section C presents additional results demonstrating EATA’s anti-forgetting ability.
  • Section D: Section D discusses related training-time robustification studies.

A.1. More Details on Datasets

The experiments evaluate OOD generalization on CIFAR-10-C, ImageNet-C, and ImageNet-R. The corruption benchmarks vary corruption type and severity, while ImageNet-R contains artistic renditions of ImageNet classes.

  • Experiments use CIFAR-10-C, ImageNet-C, and ImageNet-R as benchmark datasets for OOD generalization.
  • CIFAR-10-C and ImageNet-C contain corrupted validation images spanning 15 corruption types, four categories, and five severity levels.The categories are noise, blur, weather, and digital.
  • ImageNet-R contains 30,000 images with artistic renditions of 200 ImageNet classes.The images were primarily collected from Flickr and filtered by Amazon MTurk annotators.

A.2. More Experimental Protocols

The experimental protocols use ResNet-26 for CIFAR-10 and ResNet-50 for ImageNet, with models trained on original datasets and tested on clean or OOD data. Compared methods follow their original settings with a reduced TTT augmentation size for ImageNet experiments.

  • ResNet-26 and ResNet-50 are used for CIFAR-10 and ImageNet experiments, respectively.Models are trained on the original training sets and tested on clean or OOD test sets.
  • Compared methods generally use hyperparameters from their original papers or MEMO.The protocol specifies augmentation sizes for TTA and modifies TTT’s ImageNet augmentation size.
  • TTT’s ImageNet augmentation size is reduced from 64 to 20 because size 64 is very time-consuming, with only a slight performance difference reported.The original setting required about 12 GPU hours for a specific ImageNet-C corruption and severity level.

B. More Results on Out-of-distribution Performance and Efficiency

EATA improves test-time adaptation efficiency and performance across ImageNet-C settings while remaining stable across random test-sample orders. Its gains are not explained merely by increasing Tent’s learning rate.

  • Out-of-distribution performance: ETA and EATA consistently outperform state-of-the-art methods in most ImageNet-C corruption types across severity levels 1-4.The compared methods include TTA, MEMO, and Tent.
  • Efficiency: 31,741 backward passes are required by ETA at severity level 3, compared with 50,000 for Tent and 50,000×64 for MEMO.The figure averages ETA’s count over 15 corruption types.
  • Out-of-distribution performance: EATA outperforms Tent across the tested learning rates, whereas Tent improves from 45.3% to 43.9% error before degrading at 20.0 × 10−4.The comparison indicates that simply enlarging Tent’s learning rate does not match the sample-adaptive method.
  • Evaluation setup: Table 10 evaluates Tent and EATA on ImageNet-C Gaussian noise using error percentage across different learning rates.The learning-rate values are scaled by 10−4.
  • Stability: EATA performs consistently across 10 randomly shuffled test-sample orders, demonstrating stability across the evaluated random seeds.The random seeds range from 2020 to 2029, and performance is reported using mean and standard deviation.

C. More Results on Prevent Forgetting

Additional ImageNet-C experiments show that forgetting becomes more severe for Tent under lifelong adaptation and stronger corruptions, while EATA maintains OOD accuracy and ID clean accuracy in both settings.

  • Lifelong adaptation: Tent suffers more severe ID performance degradation under lifelong adaptation than under reset adaptation.The paper connects the lifelong setting with more optimization steps.
  • Effect of corruption severity: Increasing corruption severity increases Tent’s ID clean accuracy degradation after OOD adaptation.The result associates more severe distribution shifts with more severe forgetting.
  • EATA performance: EATA achieves higher OOD corruption accuracy while maintaining ID clean accuracy competitive with the original accuracy in both reset and lifelong adaptation.The comparison covers both adaptation scenarios.

D. More Discussions on Related Training-Time Robustification

The paper situates test-time adaptation alongside training-time approaches that enlarge training distributions and reports additional ImageNet-C comparisons across corruption categories and adaptation settings.

  • Related training-time robustification: Prior training-time robustification studies enlarge the training data distribution to cover possible test-time shifts.The listed approaches include adversarial training, data augmentation, and searching or enhancing model subnetworks.
  • ImageNet-C comparisons: Table 12 compares state-of-the-art methods on ImageNet-C severity levels 1-4 using error percentage, distinguishing normalization and joint-training variants.GN and BN denote group and batch normalization; JT denotes joint supervised and rotation-prediction training.
  • Additional experimental views: Figures 5-7 compare backward-propagation counts and forgetting-related OOD and ID accuracy under lifelong and reset adaptation settings.Figure 5 varies corruption types and severity levels, while Figures 6 and 7 differ in whether parameters are reset between corruption types.
Loading 2204.02610v2…