PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking the Teacher-Student Framework for Test-Time Adaptation

Damian Sójka, Marc Masana, Bartłomiej Twardowski, Sebastian Cygert

arXiv:2609.02507v1cs.LG

TL;DR

TTA must adapt without labels while remaining stable over changing and extended test sequences. This paper replaces the usual EMA teacher with a fixed Intransigent Teacher and finds improved long-sequence robustness, while noting slower adaptation and inherited method limitations in some settings.

  • Problem

    EMA teacher-student TTA does not strictly prevent long-term error accumulation and can lead to model collapse on extended test sequences.

  • Method

    The paper analyzes the stability-plasticity trade-off and replaces updating EMA teachers with fixed Intransigent Teachers in existing TTA methods.

  • Results

    Intransigent Teachers often prevent model collapse across diverse setups, improve hyperparameter robustness, and halt error accumulation while maintaining feature diversity.

  • Takeaways & Limitations

    TTA evaluation should include varied sequence lengths, and Intransigent Teachers provide a simple compatible alternative for reliable long-scenario adaptation.

  • Takeaways & Limitations

    Fixed teachers can slow adaptation in some cases, and adding them to existing methods inherits those methods’ limitations.

Abstract

from arXiv · show

Test-Time Adaptation (TTA) has recently emerged as a promising strategy that allows the adaptation of pre-trained models to changing data distributions at deployment time, without access to any labels. To mitigate error accumulation, researchers have widely adopted the teacher-student framework, though its long-term stability is often taken for granted. In this work, we challenge the common strategy of setting the teacher weights to an exponential moving average of the student by showing that error accumulation still occurs, although it is mostly apparent on longer sequences compared to those commonly utilized. We analyze the stability-plasticity trade-off within the teacher-student framework and propose to use an intransigent teacher that does not update its weights. Surprisingly, we show that this simple change allows TTA methods to significantly improve their performance on multiple datasets with longer scenarios and result in increased robustness to changes in hyperparameters. Finally, we show that those changes can be seamlessly and effectively applied to various architectures and experimental setups, including semantic segmentation. The code is available at https://github.com/dmn-sjk/intransigent_teacher.

1 INTRODUCTION

The paper challenges the assumed long-term stability of EMA teacher-student TTA and proposes a fixed Intransigent Teacher to reduce error accumulation on extended sequences.

  • TTA adapts pre-trained models online to changing test distributions without labels, but distribution shifts can cause inference performance issues.
  • EMA teachers are widely used in TTA because averaged predictions provide training signals that are intended to be accurate and consistent.
  • EMA teachers can fail to prevent model collapse on longer test sequences because they merely postpone error accumulation.
  • A fixed Intransigent Teacher prevents collapse while allowing the student to surpass the teacher.
  • Intransigent Teachers can replace EMA teachers in state-of-the-art methods, mitigating error accumulation and providing a comparative baseline for future TTA work.

2 RELATED WORK

Related work frames TTA as unsupervised online adaptation under shifting distributions and places the paper’s contribution within analyses of teacher-student stability, plasticity, and error accumulation.

  • Figure 1 compares average classification accuracy for baseline methods with and without Intransigent Teachers on original and 20×-repeated long benchmarks.
  • TTA adapts pre-trained models to shifting test distributions without labels, commonly using unsupervised objectives that may accumulate errors over iterations.
  • The stability-plasticity trade-off in EMA teachers is controlled by β, where β = 1 is fully stable and β = 0 is maximally plastic.
  • The paper applies stability-plasticity analyses from continual learning to teacher-student frameworks in TTA.
  • In knowledge distillation, a teacher guides a student through informative outputs; self-distillation uses matching architectures and can allow the student to outperform its teacher.

3 PRELIMINARIES

Continual TTA adapts a source model to sequential, unlabeled, non-stationary test batches, with a student updated by backpropagation and a teacher updated by EMA.

  • Continual TTA adapts a pre-trained source model to sequential unlabeled test batches arriving during inference.
  • Test distributions may change over time and differ from the source distribution, so adaptation uses current or past unlabeled test batches without source data or labels.
  • Student and teacher weights initially match the source weights, while the student is updated through backpropagation during TTA.
  • The teacher weights are updated as an EMA of the student weights, with β controlling update momentum, and the teacher is usually used for final predictions.

4 PRELIMINARY EXPERIMENTS

The experiments isolate teacher-student self-supervision and show that fixed teachers are more reliable on long adaptation sequences than EMA teachers. Varying teacher plasticity further reveals a short-term accuracy versus long-term stability trade-off.

  • Experimental setup: The experiments isolate self-supervised losses within the teacher-student framework by excluding additional regularization and stabilization components.They examine CoTTA’s consistency loss and AdaContrast’s contrastive objective.
  • Observations: An intransigent teacher combined with a self-supervised objective provides reliable adaptation on its own.This also holds for contrastive adaptation, which changes mainly the backbone through feature alignment.
  • Observations: EMA teachers do not prevent error accumulation: degradation becomes visible on long sequences and begins after the 2nd and 3rd loops for the two losses.Figure 2 compares per-batch teacher and student accuracy across 20 repeated ImageNet-C loops.
  • Observations: With consistency loss, students remain comparable to an intransigent teacher, whereas contrastive students can significantly outperform their teacher.Under EMA, the teacher-student accuracy gap remains small.
  • Plasticity-stability trade-off: Decreasing β improves short-term student accuracy but can cause collapse on longer sequences, making fixed teacher weights the most reliable long-term choice.An EMA with β = 0.9999 initially outperforms IT but falls below it when CoTTA on ImageNet-C is extended to 100 repetitions.

5 METHODOLOGY

The methodology replaces the updating EMA teacher with a fixed intransigent teacher while continuing to optimize the student. Its analysis attributes EMA failure to accumulated pseudo-label noise and loss of the source-model anchor.

  • 5.1 APPOINTING AN INTRANSIGENT TEACHER: The intransigent-teacher strategy sets β = 1, keeping teacher weights at their initial pre-trained values while the student adapts.The student supplies final predictions, and batch-normalization statistics follow the underlying TTA method.
  • 5.1 APPOINTING AN INTRANSIGENT TEACHER: The student is updated by gradient-based optimization, whereas the intransigent teacher remains fixed throughout adaptation.This is the central architectural change illustrated in Figure 4.
  • 5.2 ANALYTICAL FRAMEWORK FOR INTRANSIGENT TEACHERS: Without ground-truth labels, the student update contains both a constructive adaptation direction and harmful gradient noise from incorrect teacher pseudo-labels.The formalization separates the ideal oracle-guided gradient step from residual pseudo-label noise.
  • 5.2 ANALYTICAL FRAMEWORK FOR INTRANSIGENT TEACHERS: For β < 1, the EMA teacher becomes a geometrically weighted combination of the source model and past student states, so its source-model anchor decays over time.As n →∞, the β^nθ0 term tends to zero.
  • 5.2 ANALYTICAL FRAMEWORK FOR INTRANSIGENT TEACHERS: A degraded EMA teacher produces lower-quality pseudo-labels, increasing subsequent student noise and creating a destructive positive feedback loop.Setting β = 1 simplifies the recurrence to a fixed teacher, decoupling pseudo-label generation from noisy student updates while still enabling consistency or contrastive learning.

6 EXPERIMENTS

Experiments show that EMA-based teacher-student TTA methods degrade on long test sequences, while intransigent teachers substantially improve collapse prevention, reliability, and robustness across methods and settings.

  • Long-sequence adaptation: Teacher-student methods often underperform other baselines or collapse on extended sequences, especially on ImageNet-C and CCC.Performance is test-sequence-length dependent, and low batch sizes amplify the shortcomings.
  • Collapse prevention: Original methods decreased accuracy by at least 1% in 21 of 40 long-sequence cases, versus only three collapses with intransigent teachers at small batch size.In the three remaining cases, performance stayed near the source model; I-CoTTA improved DomainNet-126 (L) accuracy by 44.3%.
  • Reliability: With batch size 64, intransigent teachers improved average accuracy by 12.2, 2.4, and 8.0 percentage points for AdaContrast, CoTTA, and RoTTA.At smaller batch size, the respective improvements were 22.8, 25.2, and 7.4 percentage points, without hyperparameter tuning or additional parameters.
  • Comparison with stable adaptation: Intransigent teachers achieved competitive or superior performance to PeTTA, including a 3.8-point average I-AdaContrast improvement over PeTTA.I-AdaContrast led PeTTA by 1.7 points on CIFAR10-C (L), while I-RoTTA led by 2 points on ImageNet-C (L).
  • Robustness analyses: Intransigent teachers improved robustness across architectures and temporal-correlation settings, while learning-rate robustness improved in most but not all tested cases.On ImageNet-C, the IT extension was almost ineffective for RoTTA and did not fully prevent collapse under large learning-rate changes.
  • Student behavior: Students exceeded intransigent-teacher average accuracy in nearly all configurations, with exceptions for I-AdaContrast on ImageNet-R (L) and I-CoTTA on CIFAR10-C (L).Negative flip rates remained stable for IT-enhanced methods but increased for original methods in most experiments.

7 CONCLUSIONS

The paper concludes that EMA teachers can fail during long-term TTA, whereas fixed intransigent teachers often prevent collapse and improve hyperparameter robustness. The strategy remains limited by slower adaptation in some cases and inherits the limitations of the underlying methods.

  • Conclusions: EMA teachers can allow long-term error accumulation and model collapse, motivating evaluation across varied sequence lengths.The paper reports that EMA can work for common sequence lengths but fails to mitigate long-term accumulation.
  • Conclusions: The intransigent-teacher strategy often prevents collapse, improves hyperparameter robustness, and maintains feature diversity without manual tuning.The method is presented as a simple baseline compatible with current state-of-the-art TTA methods.
  • Limitations: Fixed teacher behavior can slow adaptation in some cases, and adding the intransigent teacher preserves the limitations of the existing method.The paper also reports experiments allowing teacher adaptation during an initial fixed number of steps, leaving further exploration for future work.

A.1 EXPERIMENTAL SETUP

The experiments evaluate TTA across corruption, natural-shift, and segmentation settings using established architectures, implementations, and hyperparameters. They also measure negative flips to quantify degradation from the source model.

  • Benchmarks: Experiments cover CIFAR10-C and ImageNet-C corruptions, plus natural domain shifts and semantic segmentation scenarios.Corruption benchmarks use 15 corruption types at five severity levels and the highest-severity standard sequence.
  • Architectures: Main classification experiments use WideResNet-28 on CIFAR10-C and ResNet50 on ImageNet-based benchmarks and DomainNet-126.Weights come from the RobustBench model zoo, with DomainNet-126 also using weights provided by Marsden et al.
  • Implementation: Long-scenario tests generally reuse common-sequence parameters, with image-classification batch sizes including 64 and segmentation batch size 1.The evaluation framework is adapted from prior TTA implementations and includes a preliminary ImageNet-C and CCC experiment.
  • Baselines: The study compares AdaContrast, CoTTA, RoTTA, and PETAL using their published adaptation components and reported parameter settings.These methods differ in losses, memory buffers, consistency objectives, resets, and normalization-statistics updates.
  • Metrics: Negative Flip Rate measures samples correctly classified by the source model but misclassified by the adapted model.The metric is intended to capture degradation of the source model’s initial knowledge and the direct effect of error accumulation.
  • Compute: The reported experiments run on single NVIDIA A100 or RTX 4080 GPUs, with runtimes ranging from about 30 minutes to 36 hours.A separate CCC result uses the first 623,000 of 7,500,000 images because full processing is estimated to require approximately 972 hours on one RTX 4080.

B.1 EVALUATION OF FEATURE REPRESENTATION STRENGTH.

The paper studies representation strength, segmentation transfer, source-model quality, hyperparameter robustness, and computational cost. Across these analyses, fixed-teacher variants preserve feature diversity and show robust long-sequence behavior.

  • Feature representation strength: Effective rank tracks the diversity of encoder features extracted from a domain-balanced image set during adaptation.The study uses 150 sampled images, constructs a feature matrix, computes singular values, and tracks effective rank every 50 batches.
  • Feature representation strength: A substantial effective-rank drop correlates with collapse, while the intransigent teacher helps maintain higher feature diversity.A two-fold decrease appears in four of six batch-size-64 collapse cases, including CoTTA and PETAL on C10-C (L), RoTTA on IN-C (L), and CoTTA on DN-126 (L).
  • Semantic segmentation: On CarlaTTA semantic segmentation, EMA-teacher CoTTA is sensitive to learning rate and degrades over time, whereas I-CoTTA is more robust and mitigates error accumulation.Results are reported as mean intersection-over-union across the long day2night scenario and separately for each loop.
  • Source model and hyperparameters: The source-model-quality study perturbs weights with Gaussian noise of varying σ to test how fixed-teacher adaptation depends on initialization quality.The corresponding benchmark comparison includes default, transfer-tuned, and Oracle learning-rate settings.
  • Source model and hyperparameters: IT consistently outperforms the compared methods under several learning-rate-selection conditions, while Oracle results underscore the difficulty of hyperparameter tuning.CoTTA performs better with its default learning rate than with ImageNet-C tuning, whereas AdaContrast and RoTTA improve under Transfer IN-C.
  • Computational efficiency: IT adds no computational overhead and processes 10,000 CIFAR10-C images at a speed comparable to baseline methods.Single-image MEMO adaptation is around 20× slower than AdaContrast and is generally outperformed except on ImageNet-R.

B.7 RESULTS ON COMMON-LENGTH BENCHMARKS

On common-length benchmarks, IT results are mixed because its fixed teacher adapts slowly, while its main advantage is robustness to longer sequences and hyperparameter choices.

  • Common-length performance: IT sometimes outperforms and sometimes trails the original methods on common-length benchmarks.The original methods were tuned for these benchmarks, whereas IT received no such tuning.
  • Adaptive momentum: A teacher that becomes fixed after the first loop outperforms standard IT in several cases, showing potential for adaptive momentum.The hybrid begins with β = 0.999 and switches to β = 1.0 after one full data cycle.
  • Adaptive momentum: The hybrid strategy is not uniformly superior: standard IT wins for AdaContrast on ImageNet-C (L) and CoTTA on DomainNet-126 (L).The results indicate that the fixed period length must be adjusted correctly.
  • Student-only adaptation: Student-only adaptation can perform reasonably with carefully tuned hyperparameters but underperforms IT and has greater performance variance across selection approaches.IT remains more consistent across the evaluated hyperparameter-selection methods.

B.10 EFFECTS OF INTRANSIGENCE AMOUNT EXTENDED EXPERIMENT

Extended-sequence experiments show that even slowly updating teachers and source-knowledge resets can fail over time. Fixed-teacher adaptation avoids this failure more consistently, though reset-based methods introduce tuning limitations.

  • Extended sequences: After 100 CIFAR10-C loops, CoTTA with EMA teacher β = 0.9999 degrades below IT.The result demonstrates that sufficiently long test sequences can expose degradation not visible in shorter evaluations.
  • Implication: The extended experiments motivate fixed teacher weights as a simple alternative to relying on reset schedules for preserving source knowledge.CoTTA’s reset mechanism is intended to preserve source knowledge, but the tuning results show practical limitations.
  • Reset mechanism: CoTTA’s stochastic source-weight restoration depends on a restoration-probability parameter whose optimal value varies across datasets.Realistic single-dataset tuning yields marginal average improvements, while gains appear only with Oracle tuning across all benchmarks.
  • Reset mechanism: RDumb’s periodic resets cause immediate accuracy drops, and its constant reset interval adds another parameter that may not suit every case.The paper contrasts this instability with IT’s long-sequence approach.

B.13 AUGMENTATION COUNT ABLATION

Reducing pseudo-label augmentation count preserves the long-sequence degradation pattern, while intransigent-teacher variants add improvements beyond batch-normalization-only adaptation. Teacher-statistics choices also affect performance.

  • Augmentation count: CoTTA and PETAL generate pseudo-labels by averaging predictions from either 1 or 32 randomly augmented images.Using 32 augmentations requires 31 additional forward passes per batch, creating a substantial computational burden.
  • Augmentation count: Accuracy degradation and eventual collapse persist across augmentation counts, despite some dataset-specific differences in absolute accuracy.On DomainNet-126 (L), one augmentation improves accuracy by 2% to 4% over 32 augmentations; on CIFAR10-C (L), 32 augmentations yield higher absolute accuracy, but both settings collapse.
  • Batch-normalization ablation: Across 32 benchmark, batch-size, and baseline combinations, full intransigent-teacher adaptation outperforms or matches batch-normalization-only variants in 26 cases.The comparison removes backpropagation so that only student BN-statistics adaptation remains in the ablation variant.
  • Batch-normalization ablation: Batch-normalization-only adaptation provides substantial gains but is limited to architectures containing batch-normalization layers.Intransigent-teacher distillation supplies further improvements across architectures in most evaluated scenarios.
  • Teacher statistics: Adapting teacher batch-normalization statistics at test time improves overall performance relative to fixed source-data statistics.The finding aligns with prior TTA results that adapting BN statistics can mitigate performance drops under distribution shift.

C REMAINING RESULTS

Additional results examine reused learning rates, EMA-β sensitivity, and long-sequence accuracy plots across benchmarks, methods, and architectures. The appendix also reports per-batch comparisons between EMA and intransigent teachers and negative-flip rates.

  • Architecture and hyperparameter results: Learning rates tuned for original methods on standard-length benchmarks are reused to evaluate intransigent-teacher variants on long scenarios and multiple architectures.The corresponding CIFAR10-C and ImageNet-C results are reported in Table A.10.
  • Architecture and hyperparameter results: EMA-β sensitivity is evaluated for AdaContrast and CoTTA, whose original default β is 0.999, with 20 repetitions defining long adaptation sequences.Gray table entries denote performance below the nonadapting source model.
  • Long-sequence accuracy: Figures A.7 and A.8 plot mean accuracy across loops for CoTTA, AdaContrast, and PETAL on long ImageNet-C and CIFAR10-C sequences.The plots use source-model accuracy as a reference line.
  • Teacher comparisons: Figures A.9 and A.10 compare per-batch accuracy for EMA and intransigent teachers across methods and architectures on ImageNet-C and CIFAR10-C.Teacher results use solid lines and student results use dashed lines; EMA and intransigent teachers are colored orange and blue, respectively.
  • Teacher comparisons: Figure A.11 reports the student's per-batch negative-flip rate for AdaContrast, CoTTA, RoTTA, PETAL, and their intransigent-teacher variants.The figure organizes results by method and benchmark.
Loading 2609.02507v1…