PaperlayerView exact revision on arXiv

Source-linked AI summary

Parameter-free Online Test-time Adaptation

Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, Luca Bertinetto

arXiv:2201.05718v2cs.CV

TL;DR

Online test-time adaptation addresses adapting pretrained vision models when training data and labeled target data are unavailable. The paper evaluates existing methods across realistic scenarios and proposes LAME, which adapts model outputs conservatively and achieves higher average accuracy with lower compute and memory.

  • Problem

    Online test-time adaptation must adapt pretrained models to target streams without access to source training data or labeled target samples.

  • Method

    LAME adapts the model's output by optimizing a manifold-regularized likelihood with Laplacian correction and an efficient concave-convex procedure.

  • Results

    Across the evaluated scenarios, existing test-time adaptation methods underperform a non-adaptive baseline, whereas LAME outperforms existing methods and the baseline while requiring less compute and memory.

  • Takeaways & Limitations

    Test-time adaptation should be evaluated without scenario-specific hyperparameter tuning, because existing methods are sensitive to the model and domain shift.

  • Takeaways & Limitations

    Scenario-specific hyperparameter tuning would require access to target labels, which are unavailable in the considered setting.

Abstract

from arXiv · show

Training state-of-the-art vision models has become prohibitively expensive for researchers and practitioners. For the sake of accessibility and resource reuse, it is important to focus on adapting these models to a variety of downstream scenarios. An interesting and practical paradigm is online test-time adaptation, according to which training data is inaccessible, no labelled data from the test distribution is available, and adaptation can only happen at test time and on a handful of samples. In this paper, we investigate how test-time adaptation methods fare for a number of pre-trained models on a variety of real-world scenarios, significantly extending the way they have been originally evaluated. We show that they perform well only in narrowly-defined experimental setups and sometimes fail catastrophically when their hyperparameters are not selected for the same scenario in which they are being tested. Motivated by the inherent uncertainty around the conditions that will ultimately be encountered at test time, we propose a particularly "conservative" approach, which addresses the problem with a Laplacian Adjusted Maximum-likelihood Estimation (LAME) objective. By adapting the model's output (not its parameters), and solving our objective with an efficient concave-convex procedure, our approach exhibits a much higher average accuracy across scenarios than existing methods, while being notably faster and have a much lower memory footprint. The code is available at https://github.com/fiveai/LAME.

1. Introduction

The paper targets online, unsupervised test-time adaptation of pretrained models when training data and test labels are unavailable and future conditions are uncertain. It finds existing methods are scenario-sensitive, motivating a conservative output-correction strategy.

  • Motivation: Adapting pretrained models supports resource reuse and privacy-focused applications by separating expensive pretraining from downstream adaptation.The paper frames pretrained models as trained once on broad data and adapted to newly encountered scenarios.
  • Problem setting: Online test-time adaptation must operate unsupervised on limited, potentially non-i.i.d. data without access to training data or training procedures.The intended system should also avoid tailoring its settings to a particular unknown test scenario.
  • Limitations of existing methods: Existing methods can perform well in selected scenarios but, with hyperparameters chosen across scenarios, do not outperform a non-adaptive baseline.Their effectiveness depends on the adapted model and domain shift, while scenario-specific tuning is unavailable when test conditions are unknown.
  • Proposed approach: LAME adapts model outputs rather than parameters and uses a conservative manifold-regularized likelihood optimized by an efficient concave-convex procedure.Across 7 datasets, 19 shifts, 3 training strategies, and 5 architectures, it improves average accuracy while reducing compute and memory.

2. Related work

The related work spans domain adaptation, domain generalization, source-free adaptation, and fully test-time adaptation. The paper distinguishes its setting by requiring online adaptability across architectures and training strategies, including non-i.i.d. streams.

  • Domain adaptation: Vanilla and unsupervised domain adaptation typically require access to target data during training, limiting their usability in the paper’s setting.Representative strategies learn invariant features or domain discriminators using source and target domains.
  • Domain generalization: Domain generalization learns from multiple source domains to generalize to unseen domains but lacks test-time adaptability.Its strategies include data augmentation, adversarial or generative learning, invariant representations, and domain-component separation.
  • Source-free adaptation: Source-free domain adaptation avoids training-data access during adaptation but may rely on source statistics, vendor-side preparation, generated samples, or other specialized procedures.These methods include shifted class centroids, augmented vendor training, and GAN-based sample generation.
  • Fully test-time adaptation: Fully test-time adaptation performs unsupervised adaptation without restricting or altering model training, as in TENT’s entropy-based batch-normalization updates.Adaptive batch-normalization methods also re-estimate statistics using test data.
  • Positioning: This paper additionally evaluates model-independence across training strategies and architectures and emphasizes online adaptation under non-i.i.d. conditions.The authors identify these dimensions as missing or insufficiently emphasized in prior work.

3. Problem Formulation

The paper formulates fully online test-time adaptation under covariate shift: a pretrained model must adapt and predict on unlabeled target streams whose input distribution differs from the source.

  • Setup: The setting provides a classifier trained on inaccessible labeled source data and an unlabeled target dataset sampled from an arbitrary target distribution.The target stream is used at test time for adaptation.
  • Assumption: Under covariate shift, source and target conditional label distributions are assumed equal while their input distributions differ.The resulting shift can arise through changes in class priors or class-conditional input likelihoods.
  • Objective: Online TTA addresses a model that may no longer approximate the target conditional distribution well after the source-to-target shift.The classifier must simultaneously adapt and predict on a potentially non-i.i.d. target stream.
  • Class structure: The practical setting relaxes the requirement that source and target classes coincide by allowing target classes to be superclasses in a predefined hierarchy.The paper reports using average-pooling of subclass predictions after finding it slightly preferable empirically to max-pooling.

4. On the Risks of Network Adaptation

Network adaptation methods can silently degrade models when online batches are narrow, hyperparameters are scenario-specific, or updates accumulate over time. The paper therefore motivates output-only adaptation with minimal tuning.

  • Mechanism of risk: NAMs modify selected network parameters using unsupervised losses, but consecutive updates on narrow target portions can overspecialize the model.The risk is amplified by limited batch diversity, class imbalance, and iterative optimization across batches.
  • Hyperparameter brittleness: Scenario-specific hyperparameter tuning requires labels and prior knowledge of the test scenario, defeating the intended TTA setting.The paper argues that hyperparameters should instead generalize across scenarios.
  • Failure mode: Entropy minimization can silently collapse predictions on non-i.i.d. streams, even while its objective appears to improve.In a low-diversity sequence, batches containing one class can move the decision boundary past a later class cluster.
  • Empirical evidence: Entropy minimization can degrade accuracy by up to an absolute 66% relative to the non-adaptive baseline in non-i.i.d. and class-imbalanced scenarios.The reported matrix uses scenario-specific hyperparameters and demonstrates over-specificity rather than a universal operating point.
  • Conservative alternative: The proposed alternative changes only model outputs, processes one batch at a time, and exposes only one hyperparameter for tuning.This design directly targets the risks associated with parameter updates and optimization-related brittleness.

5. The LAME method

LAME adapts classifier outputs rather than feature-extractor parameters, using manifold-regularized likelihood to produce conservative latent assignments for target batches. Its objective is optimized efficiently with CCCP under simplex constraints.

  • Method: LAME corrects classifier output probabilities while freezing the source model’s internal parameters.This avoids accumulating knowledge across batches, reduces compute and memory, and removes optimizer learning-rate and momentum tuning.
  • Latent assignments: Each target sample receives a latent assignment vector in the probability simplex, approximating its class distribution.The assignments are optimized against the source model’s predictions rather than directly changing model parameters.
  • Objective: The LAME objective combines KL divergence from source predictions with a Laplacian term and regularization against over-confident assignments.Negative-entropy regularization discourages one-hot assignments and implicitly enforces non-negativity.
  • Regularization: Laplacian regularization encourages nearby feature-space points to receive consistent latent assignments.Affinity is determined from the pretrained feature extractor’s representations, with larger affinities for closer points.
  • Optimization: CCCP replaces the concave Laplacian component with a tight linear upper bound while retaining the convex KL component.The resulting convex problem yields decoupled assignment updates from the KKT conditions, iterated until convergence.

6. Experimental design

The experiments evaluate test-time adaptation across diverse pretrained models, training procedures, domain shifts, class structures, and online sampling patterns. Validation selects one fixed hyperparameter set per method across 12 scenarios before testing on seven scenarios.

  • Experimental goals: The protocol tests both model independence and domain independence using multiple pretrained models and adaptation scenarios.A single fixed trained model is used across multiple scenarios, requiring its source classes to cover the test-time classes of interest.
  • Models: ImageNet-trained models provide variation across training procedures and architectures, including several ResNet-50 checkpoints and architectures from RN-18 to ViT-B.Figure 5 transfers hyperparameters selected with the original RN-50 across these model variants.
  • Validation data: Validation uses ImageNet, ImageNet-C-Val, and ImageNet-C16 to represent likelihood shifts and superclass mappings.ImageNet-C-Val applies nine realistic perturbations, while ImageNet-C16 maps ImageNet classes to 16 superclasses.
  • Evaluation protocol: Each method receives one hyperparameter set chosen by grid search for best average performance over 12 validation scenarios, then fixed for testing.The evaluated methods include the non-adaptive Baseline, TENT, SHOT-IM, PseudoLabel, AdaBN, and LAME.
  • Testing scenarios: The test set contains four i.i.d. and three non-i.i.d. scenarios, combining ImageNet-C-Test, ImageNet-V2, ImageNet-VID, and LaSOT.Class imbalance is introduced through Zipf-distributed class ratios, and non-i.i.d. evaluation presents sequences of tasks.

7. Experimental results

LAME remains effective across changes in training procedure and architecture, while avoiding the hyperparameter sensitivity of neural adaptation methods. It also offers substantial runtime and memory advantages over NAMs.

  • Domain-independent test-time adaptation: LAME’s hyperparameter-free bound optimization improves cross-shift average performance and worst-case degradation compared with TENT.Its only remaining tuning concerns the affinity function, which is less sensitive than NAM optimization hyperparameters.
  • Domain-independent test-time adaptation: 6.7% average absolute improvement, reaching 15% on ImageNet-v2, occurs for LAME in non-i.i.d. scenarios with prior shifts.LAME does not noticeably help standard i.i.d. likelihood shifts and is around 0.5% below the baseline in worst cases.
  • Robustness to training procedure: LAME retains its improvement across supervised and self-supervised RN-50 training procedures, whereas NAMs lose significant ground.On the SimCLR-trained RN-50, LAME retains a 4% improvement over the baseline and no other method beats it.
  • Generalization across architectures: Across five architectures, LAME is the only method retaining a consistently significant improvement over the baseline.The architectures include EfficientNet-B4, three ResNet variants, and ViT-B.
  • Domain-independent test-time adaptation: Adding hyperparameters to LAME may improve overall performance but worsen off-diagonal cross-shift results.This is presented as a speculation about the trade-off between aggregate performance and robustness across shifts.
  • Efficiency: LAME runs twice as fast and requires twice less memory than NAMs.Its runtime avoids the second forward pass and its memory use excludes gradient and intermediary buffers.

8. Conclusion

The paper evaluates online test-time adaptation under varied realistic conditions and proposes LAME as a conservative output-correction method. LAME outperforms existing methods and the non-adaptive baseline on average while using less compute and memory, but has limited gains in standard i.i.d. and class-balanced settings.

  • Conclusion: Existing online TTA methods underperform a non-adaptive baseline across the evaluated scenarios and can catastrophically degrade performance.The protocol covers several datasets, realistic shifts, and models without allowing test-time domain information to determine hyperparameters.
  • Conclusion: LAME finds latent assignments by discouraging deviations from pretrained predictions while encouraging label propagation under manifold smoothness.It adapts the classifier’s output rather than its internal parameters.
  • Conclusion: LAME outperforms existing methods and the non-adaptive baseline on average across the considered scenarios while requiring less compute and memory.The method uses an unsupervised objective with output correction rather than parameter adaptation.
  • Conclusion: LAME does not noticeably help in standard i.i.d. and class-balanced scenarios because it is restricted to correcting the classifier’s output.The paper identifies this restriction as an inherent limitation of the approach.
  • A. Mapping between source and target classes: The class mapping requires each source class to map to one corresponding target superclass or to the null variable.Source classes are denoted Y and target classes Z.
  • A. Mapping between source and target classes: Average pooling performed better overall than max-pooling in preliminary experiments.

B. Detailed derivation of LAME

LAME’s derivation solves a constrained optimization iteratively, while its practical configuration requires tuning fewer and less sensitive hyperparameters than NAMs. Its affinity choices produce roughly similar performance across the evaluated architectures.

  • Optimization derivation: The concave-convex update derives each next iterate by minimizing a constrained objective around the current solution.The derivation proceeds through a strictly convex objective with linear equality constraints, its Lagrangian, gradient conditions, and a recovered final solution.
  • NAM hyperparameters: NAM configuration requires choices such as learning rate, optimization momentum, and Batch Norm momentum.The shared grid searches learning rates {0.001, 0.01, 0.1}, momentum values 0.9 or 0, and alternatives for Batch Norm statistics.
  • LAME hyperparameters: LAME leaves the network fully frozen, so only hyperparameters of the affinity matrix require tuning.The method uses a k-NN affinity and selects k from {1, 3, 5}; other standard kernels were also evaluated.
  • Kernel comparison: LAME performs roughly similarly with k-NN, linear, and RBF kernels across the five evaluated model architectures.Figure 8 compares average accuracy over seven test scenarios for the three kernels.

D. Cross-shift validation matrices

Cross-shift validation shows that NAMs can degrade dramatically when hyperparameters selected for one scenario are transferred to another. LAME is more robust, while its loss-weight parameter presents a remaining trade-off.

  • Cross-shift robustness: NAMs suffer dramatic off-diagonal degradation when hyperparameters are tuned on one scenario and evaluated on another.The matrices evaluate absolute improvement or degradation relative to the baseline across scenario pairs.
  • Cross-shift robustness: LAME is much more robust across shifts, partly because it introduces fewer hyperparameters by design.The text reports improvements in average performance and worst-case degradation relative to TENT in the corresponding validation analysis.
  • Remaining trade-off: The LAME loss includes an additional scalar controlling the relative weight of its two terms.Tuning this scalar is expected to improve overall main-experiment performance but slightly worsen off-diagonal degradation.

E. Ablation study on the batch size

LAME maintains nearly the same average improvement across a broad range of batch sizes, supporting its use in low-batch online adaptation settings constrained by GPU memory.

  • Batch-size robustness: Close-to-4% average improvement over the baseline is preserved across a wide range of batch sizes.This result is reported for LAME’s batch-level output probability correction.
  • Memory constraints: Low batch sizes are desirable because NAMs exceed an 11 GB consumer GPU above batch size 128.For ViT-B, a batch size of 16 is already sufficient to max out memory.
  • Experimental data: The dataset summary organizes the experiments by whether datasets are used for validation or testing.The supplied table passages identify the dataset-summary table and its columns, including classes and samples.

F. Datasets

The experiments group datasets by validation or testing use and evaluate cross-shift behavior through scenario matrices. For ImageNet-to-ImageNet-Vid, an ancestral mapping connects source ImageNet classes to target classes.

  • Dataset organization: The datasets are grouped according to whether they serve validation or testing, with characteristics including class counts and sample counts.Table 2 summarizes the different datasets used in the experiments.
  • Cross-shift scenarios: Figure 10 reports absolute improvement or degradation relative to the baseline across source-scenario and evaluation-scenario pairs.Its four scenario categories are i.i.d., non-i.i.d., i.i.d. plus prior shift, and non-i.i.d. plus prior shift.
  • Class mapping: The ImageNet source outputs are mapped to valid target-class distributions using a deterministic mapping based on ImageNet concept ancestry.When a class has multiple target superclasses, the closest superclass is retained.
  • Class mapping: 279 of the 1000 original ImageNet classes were mapped to a superclass for the ImageNet-to-ImageNet-Vid scenario.The ancestral mapping scheme is qualitatively verified in Table 3.
Loading 2201.05718v2…