PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Test-Time Adaptation in Dynamic Scenarios

Longhui Yuan, Binhui Xie, Shuang Li

arXiv:2303.13899v1cs.CV

TL;DR

PTTA addresses the gap between standard TTA assumptions and dynamic real-world streams, where distributions change while samples remain correlated over time. RoTTA combines robust batch normalization, category-balanced memory sampling, and time-aware teacher-student training; it achieves state-of-the-art benchmark results, reducing averaged classification error by over 5.9%, 5.5% and 2.2% on CIFAR-10-C, CIFAR-100-C and DomainNet, respectively. Its main limitations are the need to tune RBN's α, lack of recovery from model collapse, and incomplete coverage of correlation types.

  • Problem

    PTTA studies test-time adaptation when distribution changes and correlated sampling occur simultaneously, beyond prior settings that generally consider these factors separately.

  • Method

    RoTTA uses robust batch normalization, a category-balanced memory bank weighted by timeliness and uncertainty, and time-aware teacher-student robust training.

  • Results

    Over 5.9%, 5.5% and 2.2% reductions in averaged classification error versus the best baseline are reported on CIFAR-10-C, CIFAR-100-C and DomainNet, respectively.

  • Takeaways & Limitations

    RoTTA robustly adapts on practical correlated streams with continually changing distributions and is presented as a practical deployment option.

  • Takeaways & Limitations

    RBN requires careful tuning of α, RoTTA lacks a remedy for model collapse, and category similarity does not cover every form of correlation.

Abstract

from arXiv · show

Test-time adaptation (TTA) intends to adapt the pretrained model to test distributions with only unlabeled test data streams. Most of the previous TTA methods have achieved great success on simple test data streams such as independently sampled data from single or multiple distributions. However, these attempts may fail in dynamic scenarios of real-world applications like autonomous driving, where the environments gradually change and the test data is sampled correlatively over time. In this work, we explore such practical test data streams to deploy the model on the fly, namely practical test-time adaptation (PTTA). To do so, we elaborate a Robust Test-Time Adaptation (RoTTA) method against the complex data stream in PTTA. More specifically, we present a robust batch normalization scheme to estimate the normalization statistics. Meanwhile, a memory bank is utilized to sample category-balanced data with consideration of timeliness and uncertainty. Further, to stabilize the training procedure, we develop a time-aware reweighting strategy with a teacher-student model. Extensive experiments prove that RoTTA enables continual testtime adaptation on the correlatively sampled data streams. Our method is easy to implement, making it a good choice for rapid deployment. The code is publicly available at https://github.com/BIT-DA/RoTTA

1. Introduction

Practical test-time adaptation (PTTA) targets dynamic test streams where distributions change and samples are correlated, conditions that challenge prior TTA assumptions. RoTTA addresses these challenges with robust normalization, category-balanced memory sampling, and time-aware teacher-student training, achieving strong benchmark results.

  • Motivation: Deep models lose performance under distribution shifts, motivating adaptation methods that operate with only unlabeled online test streams.Unlike domain adaptation and generalization, TTA does not require access to raw source data during adaptation.
  • Problem setting: PTTA combines continually changing distributions with temporally correlated test samples, as encountered in autonomous driving and other dynamic environments.Prior TTA studies commonly assume independently sampled data from a fixed target distribution.
  • Challenges: Correlated streams can distort batch-normalization statistics, while changing distributions make accumulated error gradients and model collapse more likely.These effects create incorrect predictions and unstable adaptation.
  • RoTTA: RoTTA combines robust statistics estimation, category-balanced sampling based on timeliness and uncertainty, and time-aware robust training with a teacher-student model.The memory bank retains newer, less uncertain samples to form a snapshot of the current test distribution.
  • Results: RoTTA reduces averaged classification error by over 5.9%, 5.5% and 2.2% versus the best baseline on CIFAR-10-C, CIFAR-100-C and DomainNet, respectively.The paper reports state-of-the-art results across these benchmarks and emphasizes ease of implementation for deployment.

2. Related Work

Related adaptation fields address distribution shift, source-free adaptation, or sequential learning, but prior TTA work generally treats distribution change and sample correlation separately. PTTA instead targets their simultaneous occurrence in continual unlabeled streams.

  • Domain adaptation: Domain adaptation transfers knowledge from labeled source data to an unlabeled target dataset, whereas PTTA adapts on the fly from continual target streams.Standard domain adaptation may collapse when only continual streams from multiple target domains are accessible.
  • Domain generalization: Domain generalization trains across multiple source domains to generalize to unseen domains, while this work improves a source-pretrained model using unlabeled online data at test time.The two settings differ in when and how target-domain information is used.
  • Continual learning: Continual learning addresses sequential tasks without forgetting prior knowledge, and PTTA similarly faces catastrophic forgetting from correlated sampling.This connection motivates using continual-learning ideas for stable test-time deployment.
  • Test-time adaptation: Test-time adaptation uses only a source model and unlabeled target data, with prior methods including self-supervised adaptation, pseudo-labeling, and BatchNorm updates through entropy minimization.These methods established online adaptation but were mainly evaluated under simpler stream assumptions.
  • Gap addressed by PTTA: Prior work considered distribution changing and correlated sampling in isolation, leaving PTTA to address both simultaneously.Methods designed for correlated streams may still omit instability caused by long-term adaptation under changing distributions.

3. Method

PTTA models online adaptation when test distributions change continually and batches are highly correlated. RoTTA addresses this setting with robust normalization, category-balanced memory sampling, and stabilized training.

  • Problem Definition and Motivation: PTTA adapts a pretrained model to online unlabeled batches that are highly correlated while the test distribution changes continually.At each step, the model receives a correlated batch and predicts on the fly.
  • Problem Definition and Motivation: Smaller Dirichlet concentration parameter δ produces stronger correlation among sampled data and greater aggregation of categories.The experiments use δ = 0.1 as the default setting.
  • Robust Batch Normalization: Correlated test samples make current-batch normalization statistics unreliable, with BN performance decreasing rapidly as correlation increases.RoTTA replaces this approach with global robust statistics updated using an exponential moving average.
  • Category-Balanced Sampling with Timeliness and Uncertainty: The memory bank captures a more stable snapshot of the current distribution, reducing adaptation to the unreliable observed distribution caused by correlated samples.Correlated batches can distort marginal label frequencies and increase the risk of model collapse.
  • Category-Balanced Sampling with Timeliness and Uncertainty: RoTTA maintains a category-balanced memory bank that accounts for sample timeliness and prediction uncertainty when updating and sampling data.Capacity is distributed equally across categories, while older or uncertain samples receive lower priority through the heuristic score.
  • Robust Training with Timeliness: RoTTA further stabilizes adaptation with a teacher-student model and timeliness-aware reweighting of memory-bank instances.The method trains only affine parameters in robust batch normalization for time efficiency and stability.

8 else

When the memory bank has available category capacity, the current test sample is inserted with its predicted category, heuristic score, and uncertainty.

  • Memory Update: The current sample is added to the memory bank together with its predicted label, heuristic score, and uncertainty.This records the information used by category-balanced sampling with timeliness and uncertainty.
  • Memory Update: The insertion stores the sample's heuristic score for later comparison during memory-bank replacement.The score combines timeliness and uncertainty in the broader CSTU procedure.
  • Memory Update: The insertion also preserves the sample's uncertainty for subsequent memory-bank management.Uncertainty is computed from the model's prediction entropy in the algorithm.

12 else

RoTTA updates its student using memory-bank instances weighted by timeliness, then updates the teacher through exponential moving average. This procedure is intended to reduce harmful gradients and stabilize adaptation.

  • Robust Training with Timeliness: After inference and memory-bank updating, RoTTA begins updating the student model using stored instances and their ages.The student update follows inference with the teacher model on the current correlated batch.
  • Robust Training with Timeliness: The robust loss averages instance losses across the occupied memory bank and conditions each loss on the corresponding sample age.The memory-bank occupation is denoted by Ω.
  • Robust Training with Timeliness: RoTTA updates the teacher model by exponential moving average after optimizing the student.The procedure is designed to reduce error gradients from old or unreliable instances and stabilize adaptation.
  • Robust Training with Timeliness: Timeliness reweighting assigns each memory-bank instance an age-dependent weight that decreases as age increases relative to bank capacity.This weight is combined with a cross-entropy loss between teacher predictions on weak views and student predictions on strong views.
  • Robust Training with Timeliness: The per-instance training loss is the product of the timeliness weight and the cross-entropy loss between teacher and student predictions.This defines the loss used in the robust training objective.

4. Experiments

Experiments evaluate RoTTA on corruption and domain-shift streams with continual distribution changes and correlated sampling. RoTTA consistently outperforms prior methods, while ablations and stress tests support the roles of its components and robustness across stream conditions.

  • 4.1. Setup: Experiments use CIFAR10-C, CIFAR100-C, and DomainNet with continually changing corruptions or domains and correlatively sampled test streams.CIFAR corruption experiments use severity 5; DomainNet adapts a source-pretrained model to the remaining five domains.
  • 4.2. Comparisons with the State-of-the-arts: RoTTA achieves the best performance on CIFAR10→CIFAR10-C and CIFAR100→CIFAR100-C, improving over the second-best method by 5.9% and 5.5%, respectively.The experiments use shared test streams across compared methods.
  • 4.2. Comparisons with the State-of-the-arts: BN, PL, TENT, and CoTTA negatively adapt compared with Source, while PL, TENT, and NOTE can eventually collapse during long-term adaptation.The reported degradation against Source is −6.5 ∼−46.4%; collapse is reported with error rate > 97%.
  • 4.2. Comparisons with the State-of-the-arts: On DomainNet, RoTTA consistently achieves the best performance and gains 2.2% over the second method, LAME.Most competing methods perform worse than the Source model by 4.6 ∼22.8%.
  • 4.3. Ablation Study: Removing RBN, CSTU, or robust training causes significant performance degradation, with removal of RBN reducing performance by 50.2% on CIFAR10-C and 16.3% on CIFAR100-C.The ablation results attribute complementary roles to robust normalization, timely confident distribution snapshots, and timeliness-aware robust training.
  • 4.3. Ablation Study: RoTTA remains strong across ten corruption-change orders, varying correlation, and different batch sizes, including 4.3% and 4.7% gains on CIFAR10-C and CIFAR100-C.Higher correlation rapidly degrades BN, PL, TENT, and CoTTA; RoTTA achieves the best results across tested batch sizes.

5. Conclusion

The paper introduces RoTTA for practical test-time adaptation under continually changing distributions and correlated test sampling. It combines robust normalization, memory-based sampling, and time-aware teacher-student reweighting to stabilize adaptation.

  • RoTTA addresses PTTA, where test distributions change continually and samples are correlated over time.
  • Robust batch normalization estimates feature normalization statistics for complex test streams.
  • A memory bank captures the current test distribution through category-balanced sampling that considers sample timeliness and uncertainty.
  • A time-aware reweighting strategy with a teacher-student model stabilizes the adaptation process.
  • Extensive experiments and ablation studies verify RoTTA’s robustness and effectiveness under PTTA.

6. Appendix

The appendix reports RoTTA’s sensitivity to key hyperparameters and evaluates it across multiple continually changing corruption orders under PTTA. It also states limitations involving robust batch normalization and outlines future directions for more realistic test-time adaptation.

  • Future work: Future work includes replacing components of RoTTA and extending test-time adaptation toward real-world scenarios under the PTTA setup.The authors characterize RoTTA as a preliminary attempt for more realistic test streams.
  • Limitations: RoTTA’s robust batch normalization requires careful design of α because it is described as a naive solution for correlatively sampled batches.The limitations section also observes eventual model collapse for some methods, including PL and TENT, during adaptation.
  • Hyperparameter sensitivity: RoTTA’s classification error remains relatively stable at 35.0-36.9% when timeliness and uncertainty are jointly weighted.Using only one of these factors causes a significant performance drop.
  • Hyperparameter sensitivity: 35.0 −36.0% classification error is achieved for α ∈{0.1, 0.05, 0.01}, while overly aggressive or gentle updates produce unreliable statistics.α controls how quickly global normalization statistics are updated.
  • Hyperparameter sensitivity: ν = 0.001 gives the best performance, whereas updating the teacher model too quickly or slowly degrades performance.The sensitivity analysis varies ν across six values.
  • Additional experiments: Across ten distribution-changing orders, RoTTA shows consistently superior performance on CIFAR10→CIFAR10-C and CIFAR100→CIFAR100-C.The appendix reports detailed results for the corruption sequences and uses alphabetical domain order for DomainNet.
Loading 2303.13899v1…