PaperlayerView exact revision on arXiv

Source-linked AI summary

NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation

Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, Sung-Ju Lee

arXiv:2208.05117v3cs.LG

TL;DR

Test-time adaptation commonly assumes i.i.d. test samples, although real streams can be temporally correlated and undermine adaptation. NOTE addresses this with IABN and PBRS, and experiments report stronger non-i.i.d. performance with comparable i.i.d. performance while retaining efficient inference.

  • Problem

    Existing TTA methods typically assume i.i.d. test samples, although many application streams are temporally correlated.

  • Method

    NOTE combines Instance-Aware Batch Normalization for shifted instances with Prediction-Balanced Reservoir Sampling to simulate i.i.d.-like samples from non-i.i.d. streams.

  • Results

    NOTE outperforms baselines under non-i.i.d. or real-distribution settings while achieving comparable performance under the i.i.d. assumption.

  • Takeaways & Limitations

    NOTE provides a batch-free, single-forward-pass TTA design with negligible memory overhead, supporting latency-sensitive applications.

  • Takeaways & Limitations

    NOTE assumes backbone networks contain BN or IABN layers, limiting direct applicability to architectures such as LSTMs and Transformers.

Abstract

from arXiv · show

Test-time adaptation (TTA) is an emerging paradigm that addresses distributional shifts between training and testing phases without additional data acquisition or labeling cost; only unlabeled test data streams are used for continual model adaptation. Previous TTA schemes assume that the test samples are independent and identically distributed (i.i.d.), even though they are often temporally correlated (non-i.i.d.) in application scenarios, e.g., autonomous driving. We discover that most existing TTA methods fail dramatically under such scenarios. Motivated by this, we present a new test-time adaptation scheme that is robust against non-i.i.d. test data streams. Our novelty is mainly two-fold: (a) Instance-Aware Batch Normalization (IABN) that corrects normalization for out-of-distribution samples, and (b) Prediction-balanced Reservoir Sampling (PBRS) that simulates i.i.d. data stream from non-i.i.d. stream in a class-balanced manner. Our evaluation with various datasets, including real-world non-i.i.d. streams, demonstrates that the proposed robust TTA not only outperforms state-of-the-art TTA algorithms in the non-i.i.d. setting, but also achieves comparable performance to those algorithms under the i.i.d. assumption. Code is available at https://github.com/TaesikGong/NOTE.

1 Introduction

Test-time adaptation addresses distribution shifts using only unlabeled test streams, but temporal correlation violates common i.i.d. assumptions and can cause existing methods to overfit. NOTE combines IABN and PBRS to improve robustness while retaining strong i.i.d. performance and practical efficiency.

  • TTA adapts DNNs to unseen target domains using only unlabeled test streams, avoiding additional data acquisition and labeling costs.
  • Real-world test streams are often temporally correlated, so samples follow time-varying distributions rather than the i.i.d. assumption used by prior TTA methods.Examples include autonomous driving and human activity recognition.
  • Existing TTA methods overfit temporally imbalanced batches, increasing errors and harming generalization under non-i.i.d. streams.Figure 2 reports higher average classification error under non-i.i.d. conditions than under i.i.d. conditions.
  • NOTE combines Instance-Aware Batch Normalization, which corrects normalization for shifted instances, with Prediction-Balanced Reservoir Sampling, which simulates class- and time-balanced i.i.d. samples.PBRS stores simulated i.i.d.-like samples in memory for adaptation.
  • On CIFAR10-C, NOTE achieves 21.1% error under non-i.i.d. conditions versus an average 15.1% lower error than the state-of-the-art method, and 17.6% under i.i.d. conditions versus 17.8% for the best baseline.Evaluations also cover CIFAR100-C, ImageNet-C, KITTI, HARTH, and ExtraSensory.
  • NOTE supports single-instance, single-forward-pass inference, updates only normalization statistics and affine parameters, and adds negligible memory overhead for latency-sensitive applications.The updated parameters are approximately 0.02% of ResNet18’s total trainable parameters.

2 Background

TTA continually adapts a pretrained model using target samples available at test time, commonly under a covariate-shift assumption. Standard BN uses channel-wise feature statistics, while recent TTA methods recalculate them from recent test batches and further adapt model parameters.

  • TTA continually adapts a pretrained model using only target samples obtained at test time, unlike traditional domain adaptation using data collected beforehand.
  • Under covariate shift, source and target input distributions differ while the conditional label distributions remain equal.The paper states PS(x) ≠ PT(x) and PS(y|x) = PT(y|x).
  • Non-i.i.d. test-time adaptation models temporal changes by sampling each target pair from PT(x, y | t) rather than a time-invariant distribution.The paper treats this as a practical setting in many real-world applications.
  • Batch Normalization normalizes feature maps channel-wise using batch means and variances, with feature dimensions represented by batch size, channels, and feature-map size.For f ∈ R^(B×C×L), μ and σ² are channel-wise statistics in R^C.
  • Recent TTA methods replace source BN statistics with statistics computed from recent test batches to reduce distribution-shift bias.
  • Those methods may additionally adapt affine parameters through entropy minimization or update all parameters using self-training.The affine parameters are γ and β.

3 Method

NOTE addresses two failure modes of batch-normalization-based TTA under temporally correlated streams: instance-wise over-whitening and biased adaptation batches. It combines Instance-Aware Batch Normalization with Prediction-Balanced Reservoir Sampling to normalize and adapt using more robust statistics.

  • Motivation: Temporal correlation can make batch-based TTA remove useful instance-wise information and bias p(y), harming objectives such as entropy minimization.These correspond to normalization and class-imbalance failure modes under non-i.i.d. test streams.
  • Instance-Aware Batch Normalization: IABN combines batch and instance normalization by detecting out-of-distribution instances and correcting normalization on a per-sample basis.It avoids completely replacing learned statistics with statistics from a temporally correlated test batch.
  • Instance-Aware Batch Normalization: The IABN hyperparameter α controls the balance between learned batch statistics and current instance statistics.High α relies more on learned BN statistics, whereas low α favors statistics measured from the instance.
  • Instance-Aware Batch Normalization: IABN becomes instance normalization at α = 0 and batch normalization at α = ∞, while excessively small α may remove useful features and degrade performance.The authors chose α = 4 for all experiments and report that this choice was not too sensitive across tested scenarios.
  • Prediction-Balanced Reservoir Sampling: PBRS uses a small memory to mimic i.i.d. samples from temporally correlated streams through time-uniform and prediction-uniform sampling.The memory stores incoming samples and replaces majority-class entries when a new prediction is not from the majority class; otherwise it applies reservoir sampling.
  • Adaptation: PBRS supplies stored samples for updating IABN normalization statistics and affine parameters, with statistics updated through exponential moving averages.The affine parameters are optimized with entropy minimization, and the model adapts every N test samples using memory size N = 64.
  • Inference: NOTE performs batch-free inference with a single instance and one forward pass, unlike methods requiring inference batches or augmentation passes.Only IABN normalization statistics and affine parameters are updated during adaptation.

4 Experiments

Experiments evaluate NOTE against state-of-the-art TTA baselines on synthetic corruption benchmarks and real-world temporally correlated streams. NOTE remains robust as temporal correlation changes, outperforms baselines under non-i.i.d. data, and retains comparable performance under i.i.d. data.

  • Experimental setup: NOTE is evaluated on CIFAR10-C, CIFAR100-C, ImageNet-C, KITTI, HARTH, and ExtraSensory using three random seeds.The implementation uses PyTorch, with test batch and memory sizes set to 64 for adaptation.
  • Overall results: NOTE achieves an average 11.1% improvement over LAME, the best baseline, under the non-i.i.d. setting.Under the i.i.d. assumption, NOTE achieves comparable performance to the baselines; a batch-based variant improves by a further average 2.2%.
  • Synthetic non-i.i.d. streams: Lower δ produces stronger temporal correlation, and most baselines deteriorate as δ decreases while NOTE remains consistent across δ values.Figure 5a varies the Dirichlet concentration parameter to assess sensitivity to temporal correlation.
  • Batch-size analysis: NOTE is relatively insensitive to batch size, whereas most baselines recover as batch size increases; larger batches reduce temporal correlation but increase latency and slow adaptation.The batch-size experiment uses six batch sizes and identifies a trade-off between balanced samples and inference speed.
  • Real-world streams: On real-world KITTI, HARTH, and ExtraSensory streams, NOTE consistently reduces errors after adaptation.The reported trend is similar to the synthetic temporal-correlation experiments.
  • Ablation study: IABN alone and PBRS with BN each reduce errors, while their joint use performs better than either component alone.The ablation attributes the joint gain to PBRS providing better estimates for IABN’s normalization operation.

5 Related work

Related work frames TTA around batch-based normalization and test-time objectives, while noting approaches for non-i.i.d. streams and replay-based continual learning. Domain-specific self-supervision is complementary but difficult to design for real test streams.

  • Test-time adaptation: Many TTA methods adapt batch-normalization statistics or model parameters using incoming test batches and objectives such as entropy minimization or self-training.These approaches recalibrate normalization and may also update affine parameters or the full model.
  • Test-time adaptation: LAME restrains prediction changes to improve robustness in non-i.i.d. scenarios but offers no noticeable gains in class-balanced i.i.d. settings.The cited explanation is that LAME discourages deviations from the pretrained model’s predictions.
  • Test-time adaptation: Domain-specific self-supervision is complementary to the paper’s approach but is difficult to design for real test streams.The related methods are described as not applicable to the paper’s real-stream setups because suitable self-supervision is highly non-trivial.
  • Replay memory: Reservoir sampling stores time-uniform samples from unknown streams in one pass and serves as a strong replay-memory baseline in continual learning.Replay memory uses buffered past data to support future learning and prevent catastrophic forgetting.

6 Discussion and conclusion

NOTE addresses non-i.i.d. test streams and is reported to outperform baselines in non-i.i.d./real-distribution settings while remaining comparable under i.i.d. conditions. The paper also identifies architectural generality and adaptation-related societal risks as open concerns.

  • NOTE outperforms baselines under non-i.i.d. and real-distribution settings while showing comparable performance under the i.i.d. assumption.
  • NOTE and most state-of-the-art TTA algorithms assume backbone networks contain BN or IABN layers.Architectures such as LSTMs and Transformers may not embed BN layers.
  • Designing a TTA algorithm that generalizes to any architecture remains an open challenge.
  • TTA can expose adapted models to data-driven biases, fairness issues, adversarial attacks, and additional computation with environmental costs.The paper states that NOTE is computationally efficient and would not exacerbate the computational issue.

Checklist

The checklist reports that the paper addresses contributions, limitations, negative societal impacts, assets, participant considerations, and reproducibility requirements. It marks theoretical-result items as not applicable and experimental reporting items as completed.

  • The authors report that the main claims, limitations, and potential negative societal impacts are addressed in the paper.
  • Theoretical-assumption and proof checklist items are marked not applicable because the paper does not present theoretical results.
  • The authors report providing code, training details, error bars, compute information, asset citations, and asset licenses.
  • The paper reports including new assets by URL and discussing consent, personally identifiable information, and offensive content for curated data.
  • Participant instructions, participant risks, and compensation reporting are marked not applicable.

A Experimental details

Experiments use three random seeds, official baseline implementations, and reported or tuned hyperparameters. Baseline-specific settings are documented for PL, ONDA, TENT, LAME, and CoTTA.

  • All experiments use random seeds 0, 1, and 2, report average errors and standard deviations, and run on NVIDIA GeForce RTX 3090 GPUs.
  • Baseline implementations follow official code and reported best hyperparameters, with additional tuning when selection guidelines exist.
  • PL updates only BN layers with learning rate LR = 0.001.
  • ONDA uses update frequency N = 10 and moving-average decay m = 0.1 by default.
  • TENT uses learning rate LR = 0.001 except for ImageNet, while LAME uses a kNN affinity matrix with k = 5.
  • CoTTA uses restoration factor p = 0.01, EMA factor α = 0.999, and a confidence threshold selected by the authors’ quantile guideline.

A.2.1 Robustness to corruptions

The corruption experiments cover MNIST-C, CIFAR10-C, CIFAR100-C, and ImageNet-C, while temporally correlated streams are simulated with a Dirichlet-based partition and KITTI rain data.

  • MNIST-C applies 15 listed corruptions to MNIST, producing 150,000 corrupted test examples from 60,000 clean training examples.
  • CIFAR10-C and CIFAR100-C apply 15 corruption types at severity level 5 and provide 150,000 test examples for each dataset.
  • ImageNet-C applies the same 15 corruption types as CIFAR10-C and CIFAR100-C, using a pretrained ResNet18 fine-tuned with IABN layers.
  • Dirichlet-distribution partitions simulate non-i.i.d. streams from static vision datasets whose usual shuffled order produces i.i.d. streams.
  • KITTI test streams use original data with a rain intensity of 200mm/hr.

A.2.2 Real-distributions with domain shift

The paper evaluates domain shift using real-world or real-world-like datasets with temporally structured target streams. Source-to-target degradation is especially severe for HARTH and ExtraSensory.

  • Dataset construction: The evaluation covers KITTI, HARTH, and ExtraSensory as real-world or real-world-like datasets with domain-shifted target streams.KITTI uses object tracking frames, HARTH separates back and thigh accelerometer users, and ExtraSensory uses natural smartphone activity data.
  • Dataset construction: KITTI captures real-world distributions through driving across cities, rural areas, and highways, with object tracking selected to exploit temporal correlation.
  • Dataset construction: HARTH uses back-mounted accelerometer data from 15 users as the source domain and thigh-mounted data from seven users as the target domain.
  • Dataset construction: ExtraSensory contains natural behavior from smartphone users, retaining five mutually exclusive activities and four sensor modalities after preprocessing.The processed dataset uses five non-overlapping windows and leaves 23 users, with 16 source-domain and seven target-domain users.
  • Domain-shift results: HARTH and ExtraSensory show 30∼40%p higher error rates on target domains than on the source domain, indicating substantial domain gaps.

B.1 Robustness to corruptions

The robustness evaluation reports classification errors across temporally correlated and uniformly distributed corruption streams, real test streams, and ablation settings. The supplied passages primarily specify the evaluation layouts rather than individual numerical outcomes.

  • Corruption robustness: CIFAR10-C, CIFAR100-C, and ImageNet-C are evaluated on temporally correlated test streams, with average classification error reported per corruption.Results are averaged over three runs, and red markings identify performance degradation after adaptation.
  • Corruption robustness: MNIST-C, CIFAR10-C, and CIFAR100-C are also evaluated on uniformly distributed test streams, reporting average classification error per domain.The tables report corresponding standard deviations and average results over three runs.
  • Real-stream robustness: HARTH and ExtraSensory are evaluated using real test streams, with average classification error and standard deviations reported per domain.
  • Ablation analysis: Ablation settings are reported for CIFAR10-C and CIFAR100-C under temporally correlated and uniformly distributed streams.The ablation tables report average classification errors and standard deviations over three runs.

C Replacing BN with IABN during test time

The paper examines replacing BN with IABN directly at test time, without pre-training with IABN layers. This setting is evaluated alongside the stated datasets and implementation resources.

  • Test-time replacement: IABN* denotes replacing BN with IABN during test time without pre-training with IABN layers.
  • Test-time replacement: The experiment concerns CIFAR10-C and CIFAR100-C under temporally correlated and uniformly distributed streams, with results averaged over three runs.
  • Test-time replacement: Using IABN with pre-trained BN models ordinarily requires retraining after replacing the BN layers, adding computational cost and reducing convenience for off-the-shelf models.
  • Test-time replacement: Without retraining, IABN* achieves 27.1% error on CIFAR10-C, compared with 36.2% for LAME, and also improves when combined with PBRS.
Loading 2208.05117v3…