PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Stable Test-Time Adaptation in Dynamic Wild World

Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, Mingkui Tan

arXiv:2302.12400v1cs.LGcs.CV

TL;DR

Wild test-time adaptation can become unstable under mixed shifts, small batches, and imbalanced label streams, with normalization layers and noisy gradients contributing to collapse. The paper analyzes these failures, favors batch-agnostic normalization, and proposes SAR, which achieves more stable adaptation across wild settings while remaining efficient.

  • Problem

    Online TTA may fail to improve or may harm robustness when test streams contain mixed shifts, small batches, or imbalanced label distributions.

  • Method

    The paper analyzes normalization-layer effects and proposes SAR, which filters large-gradient noisy samples and minimizes entropy sharpness for reliable samples.

  • Results

    SAR performs best consistently on mixed distribution shifts and achieves the best average results across 15 corruption types on ResNet50-GN and VitBase-LN.

  • Takeaways & Limitations

    Batch-agnostic group and layer normalization provide a more stable basis for wild TTA, while SAR addresses remaining collapse cases.

  • Takeaways & Limitations

    Simple sample filtering or gradient clipping is difficult to generalize because gradient norms vary across models and distribution-shift types.

Abstract

from arXiv · show

Test-time adaptation (TTA) has shown to be effective at tackling distribution shifts between training and testing data by adapting a given model on test samples. However, the online model updating of TTA may be unstable and this is often a key obstacle preventing existing TTA methods from being deployed in the real world. Specifically, TTA may fail to improve or even harm the model performance when test data have: 1) mixed distribution shifts, 2) small batch sizes, and 3) online imbalanced label distribution shifts, which are quite common in practice. In this paper, we investigate the unstable reasons and find that the batch norm layer is a crucial factor hindering TTA stability. Conversely, TTA can perform more stably with batch-agnostic norm layers, \ie, group or layer norm. However, we observe that TTA with group and layer norms does not always succeed and still suffers many failure cases. By digging into the failure cases, we find that certain noisy test samples with large gradients may disturb the model adaption and result in collapsed trivial solutions, \ie, assigning the same class label for all samples. To address the above collapse issue, we propose a sharpness-aware and reliable entropy minimization method, called SAR, for further stabilizing TTA from two aspects: 1) remove partial noisy samples with large gradients, 2) encourage model weights to go to a flat minimum so that the model is robust to the remaining noisy samples. Promising results demonstrate that SAR performs more stably over prior methods and is computationally efficient under the above wild test scenarios.

1 INTRODUCTION

Test-time adaptation addresses distribution shifts by updating models on test data, but wild conditions can make online adaptation unstable. The paper identifies normalization and noisy-gradient issues behind these failures and proposes SAR to stabilize adaptation.

  • Distribution shifts from weather, sensor degradation, and other natural variations can severely reduce deep-network generalization.
  • Fully TTA adapts arbitrary pre-trained models without altering training, making it more generally applicable than Test-Time Training.
  • Wild test streams combine mixed shifts, small batches, and online imbalanced label distributions, conditions under which existing TTA may fail or harm robustness.
  • Batch normalization is identified as a key obstacle because wild settings bias its mean and variance estimates.
  • Group and layer normalization are more beneficial than batch normalization for stable TTA, but entropy minimization on them can still collapse to a single predicted class.
  • SAR stabilizes TTA by filtering samples with large noisy gradients and jointly minimizing entropy and entropy sharpness for reliable samples.

2 PRELIMINARIES

The preliminaries distinguish Test-time Training from Fully Test-time Adaptation. TTT uses a training-time self-supervised task before updating at test time, whereas Fully TTA adapts arbitrary pre-trained models through entropy minimization.

  • Test-time Training (TTT): Test-time Training modifies model training by combining supervised cross-entropy with a self-supervised rotation-prediction task.
  • Test-time Training (TTT): TTT partitions parameters into task-shared parameters and task-specific parameters for the supervised and self-supervised objectives.
  • Test-time Training (TTT): At testing time, TTT updates the model using the self-supervised task before applying the updated model.
  • Fully Test-time Adaptation (TTA): Fully Test-time Adaptation avoids changing the original training process and adapts arbitrary pre-trained models on test mini-batches.
  • Fully Test-time Adaptation (TTA): Tent performs Fully TTA by minimizing prediction entropy, using class probabilities produced by the model for each test input.

3 STABLE ADAPTATION BY TEST ENTROPY AND SHARPNESS MINIMIZATION

Wild test-time adaptation becomes unstable when its assumptions about shift homogeneity, batch size, and label balance are violated. The paper attributes this instability mainly to batch normalization and proposes SAR to prevent collapse on more stable GN/LN models by filtering unreliable samples and seeking flatter minima.

  • Failure sources: Wild TTA violates assumptions of uniform shifts, adequate batch sizes, and balanced online labels, causing prior methods to perform poorly or fail.The paper focuses on mixed distribution shifts, small batches, and online imbalanced label distributions as practical failure settings.
  • Normalization effects: BN hinders stable TTA because its test-time mean and variance estimates become biased under mixed shifts, small batches, and changing label distributions.BN statistics are estimated across test samples, so the three wild settings make shared estimates unreliable.
  • Normalization effects: GN and LN models are more stable than BN models under the three wild test settings and exhibit fewer failure cases.The empirical comparison covers TTA methods including TTT and Tent, with experiments summarized in the normalization-layer analysis.
  • Model collapse: Tent on GN models can still collapse under severe shifts by predicting the same class for inputs with different ground-truth classes.On shuffled ImageNet-C Gaussian noise with ResNet50-GN, collapse is associated with severe corruption, such as severity level 5.
  • SAR: Direct gradient-based filtering or clipping is difficult to generalize because gradient norms vary across models and distribution-shift types.SAR instead uses entropy-based reliability filtering and sharpness-aware optimization, with a model recovery scheme for extreme collapse cases.
  • SAR: SAR filters samples using entropy to remove some large-gradient and unreliable cases, then minimizes entropy sharpness to make adaptation less sensitive to remaining noisy gradients.Entropy provides a more transferable filtering scale than raw gradient norms; the sharpness objective seeks flat regions robust to noisy or large updates.

4 EMPIRICAL STUDIES OF NORMALIZATION LAYER EFFECTS IN TTA

The experiments compare TTT and Tent across batch, mixed-shift, and online imbalanced-label settings using BN, GN, and LN models. Batch-agnostic norms are generally more stable than BN, but method- and severity-dependent failures remain.

  • Small Test Batch Sizes: GN and LN models are generally less sensitive than BN models to small test batch sizes under Tent.BN performance degrades severely below batch size 8, whereas GN and LN remain stable in several tested settings.
  • Small Test Batch Sizes: TTT remains effective across batch sizes for all tested norm layers, but TTT with Vit-LN has large run-to-run variance from sample-order sensitivity.At batch size 1, TTT with BN benefits from multiple augmentations forming a mini-batch.
  • Mixed Distribution Shifts: Under mixed corruption shifts, GN and LN produce more comparable mixed-domain and average-domain accuracy than BN for both TTT and Tent.BN mixed-adaptation accuracy is consistently poorer than average adaptation, while GN and LN match it in several settings.
  • Online Imbalanced Label Shifts: Under online imbalanced label shifts, Tent with GN and LN is generally less sensitive than Tent with BN, although GN can fail under severe shifts.Tent with R-50-BN degrades severely as imbalance increases, while R-50-GN and Vit-LN can remain stable at severity level 3.
  • Overall Observations: Across wild test settings, R-50-GN and Vit-LN are more stable than R-50-BN, but suitability depends on the adaptation method.R-50-GN is more suitable for TTT, whereas Vit-LN is more suitable for Tent because Tent with GN is prone to collapse under severe shifts.

5 COMPARISON WITH STATE-OF-THE-ARTS

The experiments evaluate SAR against prior methods under online imbalanced labels, mixed corruption shifts, batch size one, component ablations, gradient clipping, and loss-surface sharpness. SAR generally achieves stronger and more stable adaptation, while remaining computationally efficient and avoiding several collapse cases.

  • Online Imbalanced Label Distribution Shifts: SAR achieves the best average accuracy across 15 corruption types on both ResNet50-GN and VitBase-LN under online imbalanced label shifts.On VitBase-LN, SAR reaches 58.0% versus EATA’s 49.9% average accuracy.
  • Mixed Distribution Shifts: SAR performs best consistently under mixed distribution shifts, whereas Tent collapses on several model-severity combinations.MEMO and DDA are more computationally expensive, while EATA requires 2,000 additional in-distribution samples; SAR requires neither.
  • Batch Size = 1: With batch size 1, SAR achieves the best results in many cases and generally outperforms EATA across the reported settings without pre-collected samples.MEMO and DDA remain stable by resetting or fixing parameters, but cannot exploit knowledge from previously seen images and incur higher computational complexity.
  • Comparison with Gradient Clipping: Gradient clipping is sensitive to threshold selection, with small thresholds limiting learning and large thresholds failing to prevent collapse; SAR avoids this tuning requirement.The reported thresholds were 0.001 for value clipping and 0.1 for norm clipping, while larger thresholds such as 0.005 and 1.0 degraded after collapse.
  • Effects of Components in SAR: Reliable entropy improves average accuracy from 47.6% to 53.1% on VitBase-LN and from 24.6% to 31.7% on ResNet50-GN.Adding sharpness-aware minimization further raises ResNet50-GN accuracy from 31.7% to 37.0%.
  • Sharpness of Loss Surface: SAR produces a flatter entropy-loss surface than Tent, with a larger low-loss region and greater robustness to noisy or large gradients.The comparison uses models learned on ImageNet-C Gaussian noise at level 5.

6 CONCLUSIONS

The paper concludes that SAR stabilizes online TTA under mixed shifts, small batches, and imbalanced label shifts. It attributes instability partly to normalization and noisy high-gradient samples, and reports SAR as stable and efficient.

  • SAR is proposed to stabilize online TTA under mixed shifts, small batches, and imbalanced label shifts.
  • The method addresses failures by suppressing the effect of certain noisy test samples with large gradients.
  • Extensive experiments demonstrate the stability and efficiency of SAR under wild test settings.

REPRODUCIBILITY STATEMENT

The reproducibility statement identifies datasets, models, and method protocols as the three dependencies for reproducing the reported results. It notes that the models and pretrained weights are publicly available.

  • Reproducing the results depends on dataset details, model availability, and the protocols of each method.
  • Experiments use ResNet50-BN, ResNet50-GN, and VitBase-LN on ImageNet-C/R and VisDA-2021.
  • The adopted models and pretrained weights are publicly available through torchvision and timm repositories.

A RELATED WORK

The paper situates SAR among adaptation methods and describes its reliable, sharpness-aware entropy-minimization pipeline for stabilizing online TTA.

  • Adaptation with Target Data: Target-data adaptation methods include offline unsupervised domain adaptation and online test-time adaptation.TTA is further divided into Test-Time Training and Fully TTA according to whether training is altered.
  • Test-time Adaptation: Prior TTA methods can become unstable with small batches, mixed domains, and online imbalanced label distributions.MEMO resets parameters after each adaptation, while DDA adapts inputs with frozen model weights, limiting cumulative exploitation of previous samples.
  • SAR: SAR filters unreliable test samples and minimizes the sharpness of the test-time entropy loss so updates are more robust to noisy, large gradients.The method combines reliable entropy minimization, sharpness-aware updating, and model recovery when the moving-average entropy indicates collapse.
  • SAR: SAR’s pseudocode processes each test sample by computing entropy and predictions, filtering by entropy, applying sharpness-aware updates, and tracking a recovery trigger.The algorithm specifies entropy computation, reliable-sample selection, gradient-based updating, and moving-average monitoring.
  • Evaluation Benchmarks: ImageNet-C evaluates adaptation across 15 corruption types, each with five severity levels, by corrupting the original ImageNet test set.The benchmark includes noise, blur, weather, brightness, contrast, transformation, pixelation, and JPEG compression corruptions.

C.2 MORE EXPERIMENTAL PROTOCOLS

The experiments use public BN, GN, and LN models and compare adaptation methods under controlled wild-test protocols, including computational efficiency and batch-size effects.

  • Models: The evaluated models are ResNet50-BN, ResNet-50-GN, VitBase-LN, and ConvNeXt-LN, using publicly available pretrained weights.The models come from torchvision and timm repositories.
  • SAR Configuration: SAR uses SGD with momentum 0.9, batch size 64 except for batch-size-one experiments, and model-specific learning rates.Its entropy threshold follows EATA, sharpness neighborhood size uses ρ=0.05, and recovery uses a moving-average entropy threshold.
  • Efficiency: Efficiency is measured on 50,000 ImageNet-C images with ResNet-50-GN using a single V100 GPU.The efficiency table records source-data needs, online updates, forward and backward passes, other computation, and GPU time.
  • Compared Methods: The comparison includes TTT, Tent, EATA, MEMO, and DDA, with implementations and hyperparameters specified for fair evaluation.TTT is modified to operate on arbitrary pretrained models, while DDA follows its official repository and default hyperparameters.

D ADDITIONAL RESULTS ON IMAGENET-C OF SEVERITY LEVEL 3

Additional ImageNet-C severity-level-3 results show SAR remains strongest under online label imbalance and single-sample adaptation across corruption types.

  • Online Imbalanced Label Shifts: SAR performs best on average across 15 corruption types under online imbalanced label shifts with imbalance ratio qmax/qmin = ∞.The results are consistent with the main paper at severity level 5; DDA is competitive on noise but weaker on other corruptions.
  • Single-sample Adaptation: SAR performs best on average across 15 corruption types when adapting with batch size = 1.These severity-level-3 results are consistent with the main paper’s findings.

E ADDITIONAL RESULTS ON IMAGENET-R AND VISDA-2021

Results on ImageNet-R and VisDA-2021 reproduce the paper’s normalization and SAR patterns under online imbalance and small-batch wild settings.

  • ImageNet-R: On ImageNet-R, ResNet50-GN and VitBase-LN are more stable than ResNet50-BN under online imbalance and batch-size-one adaptation.SAR achieves the best performance on the GN and LN models compared with Tent and EATA.
  • VisDA-2021: On VisDA-2021, Tent and EATA fail with ResNet50-BN but remain stable with ResNet50-GN and VitBase-LN under online label shifts.SAR further improves the adaptation performance of Tent and EATA on the GN and LN models.

F.1 EFFECTS OF COMPONENTS IN SAR

The ablations show that SAR’s reliable-sample filtering and sharpness-aware optimization jointly stabilize online TTA, while larger batches and batch-agnostic normalization improve stability but do not eliminate failures.

  • F.1 EFFECTS OF COMPONENTS IN SAR: Both reliable entropy and sharpness-aware optimization keep gradient norms in a normal range throughout online adaptation.The component ablation attributes this stabilization to their joint use.
  • F.1 EFFECTS OF COMPONENTS IN SAR: Under batch size 1 and mixed domain shifts, the reliable and sharpness-aware components generally work together to stabilize online TTA.Only VitBase-LN with batch size 1 activates model recovery, raising average accuracy from 55.7% to 56.4%.
  • F.1 EFFECTS OF COMPONENTS IN SAR: SAR’s loss surface avoids the collapse-associated optimization behavior observed for Tent on ResNet50-GN.Tent’s loss becomes hard to degrade and cannot find a proper minimum after the model begins collapsing.
  • F.1 EFFECTS OF COMPONENTS IN SAR: SAR works well with ρ = 0.05 across the tested architectures and datasets, with good performance over ρ in [0.03, 0.1].The tested architectures are ResNet50-BN, ResNet50-GN, and VitBase-LN; datasets include ImageNet-C and ImageNet-R.
  • F.1 EFFECTS OF COMPONENTS IN SAR: Uniform-output diversity regularization is infeasible for this setting because online labels may be imbalanced and the method requires a batch of samples.The paper instead addresses collapse from an optimization perspective.
  • F.1 EFFECTS OF COMPONENTS IN SAR: Larger batch sizes improve Tent’s adapted performance under mixtures of 15 corruption types, while ConvNeXt-LN is more stable than ResNet50-BN but still has failures.These results support batch-agnostic normalization and larger batches as stability factors, not complete solutions.

G.4 EFFECTIVENESS OF MODEL RECOVERY SCHEME WITH TENT AND EATA

Model recovery substantially helps Tent under wild test-time adaptation, but SAR provides a larger improvement than recovery-based Tent or EATA in the reported comparison.

  • G.4 EFFECTIVENESS OF MODEL RECOVERY SCHEME WITH TENT AND EATA: Model recovery raises Tent’s average accuracy from 22.0% to 26.1% on ResNet50-GN, while its gain for EATA is marginal.The comparison is conducted under online imbalanced label distribution shifts on ImageNet-C at severity level 5.
  • G.4 EFFECTIVENESS OF MODEL RECOVERY SCHEME WITH TENT AND EATA: 37.2% average accuracy for SAR exceeds 26.1% for Tent+recovery on ResNet50-GN.This comparison is reported as evidence of SAR’s effectiveness.
Loading 2302.12400v1…