PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Mean Teacher for Continual and Gradual Test-Time Adaptation

Mario Döbler, Robert A. Marsden, Bin Yang

arXiv:2211.13081v2cs.CV

TL;DR

Continual and gradual TTA must adapt to long sequences of changing domains without accumulating self-training errors. RMT combines mean-teacher self-training with symmetric cross-entropy, contrastive source-space alignment, and optional source replay, achieving state-of-the-art results across the evaluated benchmarks.

  • Problem

    Continual and gradual TTA must handle sequential or evolving domain shifts while limiting error accumulation in long test sequences.

  • Method

    RMT uses a mean teacher with symmetric cross-entropy, contrastive learning toward the initial source space, and optional source replay.

  • Results

    RMT achieves state-of-the-art results on common corruption benchmarks, ImageNet-R, and continual DomainNet-126.

  • Takeaways & Limitations

    Mean teachers with symmetric cross-entropy and contrastive learning provide an effective approach for continual and gradual TTA across varied domain shifts.

  • Takeaways & Limitations

    Source replay requires storing at least part of the source data in an on-device buffer, whose size can limit performance.

Abstract

from arXiv · show

Since experiencing domain shifts during test-time is inevitable in practice, test-time adaption (TTA) continues to adapt the model after deployment. Recently, the area of continual and gradual test-time adaptation (TTA) emerged. In contrast to standard TTA, continual TTA considers not only a single domain shift, but a sequence of shifts. Gradual TTA further exploits the property that some shifts evolve gradually over time. Since in both settings long test sequences are present, error accumulation needs to be addressed for methods relying on self-training. In this work, we propose and show that in the setting of TTA, the symmetric cross-entropy is better suited as a consistency loss for mean teachers compared to the commonly used cross-entropy. This is justified by our analysis with respect to the (symmetric) cross-entropy's gradient properties. To pull the test feature space closer to the source domain, where the pre-trained model is well posed, contrastive learning is leveraged. Since applications differ in their requirements, we address several settings, including having source data available and the more challenging source-free setting. We demonstrate the effectiveness of our proposed method 'robust mean teacher' (RMT) on the continual and gradual corruption benchmarks CIFAR10C, CIFAR100C, and Imagenet-C. We further consider ImageNet-R and propose a new continual DomainNet-126 benchmark. State-of-the-art results are achieved on all benchmarks.

1. Introduction

Test-time adaptation addresses performance degradation when deployed models encounter changing, unseen distributions. RMT targets continual and gradual shifts with robust self-training, contrastive feature alignment, and support for both source-free and source-available settings.

  • Continual TTA adapts online across sequential domain shifts, unlike standard TTA methods commonly demonstrated for one shift at a time.
  • A multi-view contrastive loss pulls test features toward the initial source space and promotes invariance to input changes.
  • RMT replaces cross-entropy with symmetric cross-entropy for mean-teacher self-training based on gradient-property analysis.
  • RMT achieves state-of-the-art results on corruption benchmarks, ImageNet-R, and the new continual DomainNet-126 benchmark.
  • The framework addresses both source-free adaptation and settings where source data remains available.

2. Related Work

Prior work spans domain generalization, unsupervised domain adaptation, standard test-time adaptation, and newer continual or gradual TTA settings. These approaches use augmentation, feature alignment, batch-statistics updates, self-training, and stabilization techniques to address changing domains and error accumulation.

  • Domain Generalization: Domain generalization seeks robustness on unseen test distributions through augmentation, domain-invariant features, and domain randomization.
  • Unsupervised Domain Adaptation: Unsupervised domain adaptation uses labeled source and unlabeled target data, with methods including adversarial learning, discrepancy losses, contrastive learning, and self-training.
  • Test-time Adaptation (TTA): Test-time adaptation updates deployed models using current test data, ranging from batch-normalization-statistics adaptation to methods that also perform backpropagation.
  • Continual and Gradual Test-time Adaptation: Continual TTA addresses sequentially changing domains, while gradual TTA exploits shifts that evolve incrementally over time.
  • Continual and Gradual Test-time Adaptation: Self-training methods in continual TTA can accumulate errors from miscalibrated predictions; CoTTA counters this with averaged predictions and stochastic restoration.
  • RMT framework: RMT combines source prototypes, contrastive learning, symmetric cross-entropy self-training, and teacher updates through an exponential moving average.

3. Methodology

RMT adapts deployed models to changing test distributions by combining mean-teacher self-training with symmetric cross-entropy, contrastive feature alignment, and optional source replay. Its design targets unstable pseudo-labels, error accumulation, and shifts that evolve over time.

  • 3.1. Robust Mean Teacher: RMT replaces cross-entropy with symmetric cross-entropy in mean-teacher self-training to obtain more desirable gradient properties.The method also optionally uses source replay, depending on source-data availability.
  • 3.1. Robust Mean Teacher: The mean-teacher framework updates the student using consistency losses and updates the teacher through an exponential moving average of student parameters.The moving-average teacher is described as more stable in changing environments, while ensembling student and teacher logits lets the student respond faster to shifts.
  • 3.1. Robust Mean Teacher: When student and teacher confidences agree above 0.5, cross-entropy has zero gradient, whereas symmetric cross-entropy increases the gradient with teacher confidence.This can increase student confidence and support self-training after small distribution shifts.
  • 3.1. Robust Mean Teacher: Symmetric cross-entropy balances gradients across high- and low-confidence predictions by combining cross-entropy with reverse cross-entropy.The reverse cross-entropy gradient is largest for one-hot teacher predictions and smallest for uniform predictions.
  • 3.2. Contrastive Learning: Contrastive learning uses augmented test views and source prototypes to learn input invariance and pull test features toward the initial source space.The framework also includes a mean-teacher warm-up before deployment and can use source replay to stabilize self-training against error accumulation.
  • 3.3. Source Replay: Source replay requires storing at least part of the labeled source data in an on-device buffer, making buffer size a practical constraint.The method investigates how buffer size affects performance.

4. Experiments

Experiments evaluate RMT across continual, gradual, single-sample, corruption, and natural domain-shift settings, including source-free and source-replay variants. RMT achieves strong results, with mean teachers particularly effective for gradual or easy-to-hard shifts.

  • Datasets and Settings: The evaluation covers CIFAR10C, CIFAR100C, ImageNet-C, ImageNet-R, and DomainNet-126 under continual and gradual settings.The corruption datasets contain 15 corruption types with five severity levels; DomainNet-126 contains four domains and 126 classes.
  • Results for Continual Test-Time Adaptation: RMT’s source-free variant outperforms GTTA-MIX on CIFAR10C and is only slightly worse on CIFAR100C, while source replay reduces error further.On ImageNet-R and DomainNet-126, the source-free approach sets new state-of-the-art results and source replay performs best with four updates.
  • Results for Continual Test-Time Adaptation: RMT achieves state-of-the-art results on corruption benchmarks, ImageNet-R, and continual DomainNet-126, with source replay further lowering error.The framework includes source-free operation and variants trading memory and computation for performance.
  • Results for Gradual Test-Time Adaptation: Mean-teacher approaches improve gradual adaptation by more than 18.3%, while easy-to-hard ordering yields 12% lower ImageNet-C error than hard-to-easy ordering.The gradual benchmark averages performance across severity levels and separately reports severity level 5.
  • Single-Sample Test-Time Adaptation: Using a sliding window enables single-sample TTA; at window size 16, RMT remains better or competitive with state-of-the-art batch-setting methods.Smaller batches slightly worsen performance because batch-statistics estimates become noisier.
  • Ablation Studies: Ablations show that symmetric cross-entropy, ensemble prediction, contrastive learning, warm-up, and source replay each contribute to lower error.The reported configuration analysis averages classification error over three runs.

5. Conclusion

The paper concludes that RMT combines a symmetric cross-entropy mean teacher with contrastive learning to improve continual and gradual TTA. It reports state-of-the-art benchmark performance and finds that limited source replay can provide additional gains.

  • 5. Conclusion: RMT sets a new standard in continual and gradual TTA using a mean teacher with symmetric cross-entropy and contrastive learning.The method is motivated by gradient-property analysis and evaluated on common benchmarks plus a new DomainNet-126 benchmark.
  • 5. Conclusion: State-of-the-art results are achieved on all common benchmarks, while a new DomainNet-126 benchmark broadens evaluation across domain shifts.The conclusion identifies DomainNet-126 as an additional demonstration of effectiveness.
  • 5. Conclusion: Replaying a small percentage of source data improves performance and permits multiple update steps that provide additional gains when privacy and accessibility allow it.This conclusion is conditional on source-data replay being acceptable for the application.

A. Adaptation with increasing difficulty

The study examines whether adaptation benefits from domain sequences ordered by the initial model’s difficulty. Mean teachers are especially effective when corruption difficulty increases from easy to hard.

  • A. Adaptation with increasing difficulty: Mean teachers are particularly well suited to easy-to-hard sequences, achieving 12% lower ImageNet-C error than hard-to-easy ordering.Corruptions are ordered using the initial source model’s error at severity level 5.

B. Ablation studies

Ablations assess update counts, source-data availability, loss weights, and hyperparameter sensitivity. They show benefits from additional updates and source replay, alongside stable performance across common contrastive temperatures.

  • Update Steps: Six updates achieve the best non-source-free results, while two or four updates provide a balance between performance and computational complexity.All datasets benefit from more update steps in the examined setting.
  • Source Replay: Using 1% of source data improves performance, whereas the source-free variant already sets new standards across all benchmarks.Source availability has only marginal effects on several datasets, but error increases slightly on CIFAR100C and DomainNet-126.
  • Sensitivity: RMT remains stable for contrastive temperatures 0.07, 0.1, 0.2, and 1.0, while very slow or fast teacher updates can reduce performance.The ablation varies both the contrastive temperature and mean-teacher momentum.
  • Loss Weights: Loss weights near λ ∈[0.5, 1.0] yield stable performance, while values λ ≤0.1 cause a performance drop.The analysis attributes the drop to reduced contributions from source replay and contrastive learning.

C. DomainNet-126

DomainNet-126 introduces a continual natural-shift benchmark with four domains and 126 classes, using sequential test-domain configurations to evaluate continual TTA.

  • DomainNet-126 contains four natural-shift domains—real, clipart, painting, and sketch—and 126 classes.
  • The benchmark addresses the lack of separated natural-shift evaluations by enabling continual TTA across sequentially changing domains.
  • Table 10 specifies four continual test sequences, including the pre-training domain and the ordered test domains.
  • Table 11 reports DomainNet-126 classification error rates for online continual TTA, with test domains arranged sequentially and results averaged over five runs.
Loading 2211.13081v2…