PaperlayerView exact revision on arXiv

Source-linked AI summary

Test-Time Adaptation via Self-Training with Nearest Neighbor Information

Minguk Jang, Sae-Young Chung, Hye Won Chung

arXiv:2207.10792v2cs.CVcs.AI

TL;DR

Test-time adaptation must work with online unlabeled test data and without training-procedure information, but classifier-generated pseudo labels may be inaccurate under domain shift. TAST trains adaptation modules using nearest-neighbor-based pseudo-label distributions and averages their predictions. It outperforms existing TTA methods across domain-generalization and image-corruption benchmarks, with a reported limitation for large-scale settings.

  • Problem

    TTA methods use online unlabeled test data without training-procedure information, but classifier-generated pseudo labels may be inaccurate under domain shift.

    Source: §1 · pp. 1–2

  • Method

    TAST trains randomly initialized adaptation modules to match nearest-neighbor-based pseudo-label distributions and averages their predicted class distributions.

    Source: §1 · p. 2

  • Results

    TAST outperforms current state-of-the-art TTA methods on domain-generalization and image-corruption benchmarks.

    Source: §1 · p. 2, §4.2.2 · p. 9

  • Takeaways & Limitations

    The experiments show that nearest-neighbor information and adaptation modules contribute to TAST’s performance increase.

    Source: §1 · p. 2

    TAST and TAST-BN may construct or update prototypes ineffectively on large-scale benchmarks, especially with many classes and small batch sizes.

    Source: §6 · p. 9

Abstract

from arXiv · show

Test-time adaptation (TTA) aims to adapt a trained classifier using online unlabeled test data only, without any information related to the training procedure. Most existing TTA methods adapt the trained classifier using the classifier's prediction on the test data as pseudo-label. However, under test-time domain shift, accuracy of the pseudo labels cannot be guaranteed, and thus the TTA methods often encounter performance degradation at the adapted classifier. To overcome this limitation, we propose a novel test-time adaptation method, called Test-time Adaptation via Self-Training with nearest neighbor information (TAST), which is composed of the following procedures: (1) adds trainable adaptation modules on top of the trained feature extractor; (2) newly defines a pseudo-label distribution for the test data by using the nearest neighbor information; (3) trains these modules only a few times during test time to match the nearest neighbor-based pseudo label distribution and a prototype-based class distribution for the test data; and (4) predicts the label of test data using the average predicted class distribution from these modules. The pseudo-label generation is based on the basic intuition that a test data and its nearest neighbor in the embedding space are likely to share the same label under the domain shift. By utilizing multiple randomly initialized adaptation modules, TAST extracts useful information for the classification of the test data under the domain shift, using the nearest neighbor information. TAST showed better performance than the state-of-the-art TTA methods on two standard benchmark tasks, domain generalization, namely VLCS, PACS, OfficeHome, and TerraIncognita, and image corruption, particularly CIFAR-10/100C.

1 INTRODUCTION

Test-time adaptation addresses domain shift when training data and labeled test-domain data are unavailable. TAST uses nearest-neighbor information to train adaptation modules, and outperforms existing TTA methods on domain-generalization and image-corruption benchmarks.

1 INTRODUCTION

  • Test-time domain shift degrades classifier performance, while accessing training data or labeled test-domain data during testing is often difficult.TTA uses online unlabeled test data without training-procedure information.Source: §1 · pp. 1–2
  • Self-training can overfit inaccurate classifier-generated pseudo labels, while T3A yields only marginal performance gains.The paper also describes normalization-based, entropy-minimization, and prototype-based TTA approaches.Source: §1 · p. 2
  • TAST trains randomly initialized adaptation modules using nearest-neighbor pseudo-label distributions and averages their predicted class distributions.The method assumes a test example and its embedding-space neighbors are likely to share a label.Source: §1 · p. 2
  • 1.01%: TAST surpasses the current state-of-the-art algorithm on average with ResNet-18 trained by ERM on domain-generalization benchmarks.The paper reports that nearest-neighbor information and adaptation modules both contribute to the increase.Source: §1 · p. 2

2 PRELIMINARIES

The section frames test-time domain shift as a mismatch between training and test distributions that can impair classification. It describes prototype-based TTA through T3A, which uses a support set of previous test examples and classifier predictions.

2 PRELIMINARIES

  • Under test-time domain shift, the training and test distributions differ, and trained classifiers often perform poorly on test data.The setup models a classifier as a feature extractor followed by a linear classifier.Source: §2 · p. 3
  • T3A forms class prototypes from a support set of previous test examples and their trained-classifier predictions, then classifies by nearest prototype.T3A does not modify classifier parameters during test time.Source: §2 · p. 3
  • TAST uses multiple randomly initialized adaptation modules trained with nearest-neighbor-based pseudo labels to extract useful classification information.The passage contrasts this approach with T3A, which does not update the trained model parameters.Source: §2 · p. 4

3 METHODOLOGY

TAST comprises adaptation modules and nearest-neighbor-based self-training. It freezes the feature extractor, learns module predictions from nearby support examples, and averages predictions across modules.

3 METHODOLOGY

  • The method has two components: adaptation-module utilization and pseudo-label generation using nearest-neighbor information.The modules are trained independently, and their predicted class distributions are averaged.Source: §3 · p. 4, §3.1 · p. 4

3.1 ADAPTATION MODULE

  • Randomly initialized adaptation modules are trained during test time on top of a frozen feature extractor, with an ensemble used to address initialization-related performance degradation.The approach avoids fine-tuning the whole network and does not require BN layers.Source: §3.1 · p. 4

3.2 SELF-TRAINING WITH NEAREST NEIGHBOR INFORMATION

  • TAST finds nearby support examples in the frozen feature space and uses their prototype-based class distributions to generate a test example’s pseudo-label distribution.The neighbor set contains the Ns nearest support examples; the support set is updated and unconfident examples are filtered.Source: §3.2 · pp. 4–5
  • The adaptation modules are fine-tuned to match the neighbor-based pseudo labels, and predictions are averaged across modules.The pseudo-labeling and fine-tuning steps are iterated T times per batch without propagating gradients into the pseudo labels.Source: §3.2 · p. 5
  • TAST-BN is a variant that fine-tunes BN layers instead of adaptation modules and stores test data rather than feature representations in its support set.The support representation changes because the feature-extractor embedding space changes during test time.Source: §3.2 · p. 5

4 EXPERIMENTS

Experiments compare TAST with test-time adaptation baselines on domain-generalization and image-corruption benchmarks. TAST leads on domain generalization, while TAST-BN leads on image corruption; ablations examine nearest-neighbor information and adaptation modules.

4 EXPERIMENTS

  • The experiments compare TAST with test-time adaptation baselines on domain-generalization and image-corruption benchmarks.Baselines include pseudo-labeling variants, Tent, SHOT/SHOTIM, and T3A; the image-corruption comparison also includes online TTT++.Source: §4 · pp. 5–6, §4.2 · p. 8

4.1.2 EXPERIMENTAL RESULTS

  • TAST outperforms all baselines across the domain-generalization benchmarks, exceeding T3A by 1.01% with ResNet-18 and 0.69% with ResNet-50 on average.TAST improves ERM classifiers by 2.17% and 1.21% on average for ResNet-18 and ResNet-50, respectively.Source: Table 1 · p. 6, §4.1.2 · p. 7

4.1.1 EXPERIMENTAL SETUP

  • The domain-generalization evaluation uses VLCS, PACS, OfficeHome, and TerraIncognita, with leave-one-domain-out testing and ResNet-18/50 classifiers.The setup follows T3A's dataset splits and hyperparameter-selection method; networks are trained with algorithms including ERM and CORAL.Source: §4.1.1 · pp. 6–7, §4.1 · p. 6
  • TAST uses Ns ∈ {1, 2, 4, 8}, T ∈ {1, 3}, M ∈ {1, 5, 20, 50, 100, −1}, and 20 adaptation modules.M = −1 stores all samples without filtering; Adam uses a 0.001 learning rate.Source: §4.1.1 · p. 7

4.1.2 EXPERIMENTAL RESULTS

  • Nearest-neighbor information improves performance by 0.43% on average, while multiple adaptation modules outperform TAST-N and T3A; a single module performs worse than TAST-N.The ablations use ResNet-18 trained by ERM; Figure 2 examines accuracy sensitivity to Ns and T on PACS when M = −1.Source: Table 2 · p. 7, §4.1.2 · p. 8, Figure 2 · p. 8

4.2.2 EXPERIMENTAL RESULTS

  • On CIFAR-10C and CIFAR-100C, TAST-BN gains 1.25% and 4.56% over Tent, respectively, and outperforms all tested TTA methods and TTT++.The comparison uses the highest corruption level; Table 3 reports average error rates.Source: Table 3 · p. 8, §4.2.2 · p. 9

4.2.1 EXPERIMENTAL SETUP

  • The image-corruption evaluation uses CIFAR-10C/100C with 15 corruption types at level 5, a ResNet-50 backbone, and an online TTT++ comparison.The setup uses Ns = 1, T = 1, 20 adaptation modules, M = 100, and test batches of 128.Source: §4.2.1 · p. 8, §4.2 · p. 8

5 RELATED WORKS

Related work distinguishes TTA from test-time training and source-free adaptation by the information available at test time. It also contrasts TAST’s module ensemble with prior ensemble approaches.

Test-time training methods

  • Test-time training fine-tunes classifiers with a self-supervised task used during training, whereas TTA lacks access to training-procedure information.The paper reports that TAST outperforms existing test-time training methods on image corruption without knowing the self-supervised task.Source: §5 · p. 9

Source-free domain adaptation methods

  • Source-free domain adaptation commonly uses the full unlabeled test set, while TTA adapts online; unlike NRC, TAST combines adaptation modules with prototype-based classification.NRC and TAST share the intuition that test examples and nearby examples are likely to share labels under domain shift.Source: §5 · p. 9

Ensemble scheme in test-time adaptation

  • BACS averages predictions from ensemble members adapted during training, whereas TAST ensembles adaptation modules to address their random initialization at test time.Source: §5 · p. 9

6 DISCUSSION

TAST adapts trained classifiers at test time using nearest-neighbor information, and the authors identify large-scale benchmarks as a limitation. They also describe four domain-generalization benchmarks and suggest the test-time ensemble could be combined with source-free domain adaptation or test-time training.

6 DISCUSSION

  • TAST uses nearest-neighbor information to adapt trained classifiers at test time, and the authors describe its ensemble as the first built at test time for TTA.They expect the ensemble scheme can be combined with other methods in source-free domain adaptation or test-time training.Source: §6 · p. 9
  • Prototype construction and updates can be ineffective on large-scale benchmarks, especially with many classes and small batches.TAST and TAST-BN rely on good prototypes and build or update them from encountered pseudo-labeled test data.Source: §6 · p. 9

A.1 DOMAIN GENERALIZATION BENCHMARKS

  • The domain-generalization evaluation covers VLCS, PACS, OfficeHome, and TerraIncognita, spanning photographic, artistic, office/home, and wild-animal images.The benchmarks contain 5, 7, 65, and 10 categories, respectively.Source: §A.1 · p. 14

A.2 IMPLEMENTATION DETAILS ON DOMAIN GENERALIZATION BENCHMARKS

The domain-generalization implementation follows T3A’s dataset-split and hyperparameter-selection approach. It evaluates backbones trained with several algorithms and uses specified adaptation settings, including a restricted support set for TAST-BN.

A.2 IMPLEMENTATION DETAILS ON DOMAIN GENERALIZATION BENCHMARKS

  • Training-domain validation uses an 80/20 split of each training-domain dataset, with the smaller set used to select hyperparameters by adapted-classifier accuracy.The split and selection procedure follows T3A.Source: §A.2 · p. 14
  • Backbones are trained using ERM, CORAL, MMD, or Mixup, and experiments use four random seeds: 0, 1, 2, and 3.CORAL aligns covariance matrices, MMD matches distributions, and Mixup uses interpolated examples and labels.Source: §A.2 · p. 14
  • Adaptation hyperparameters follow T3A and DomainBed; TAST-BN caps its support set at 150 examples for memory and runtime efficiency.Baseline adaptation learning rates are scaled by β values of 0.1, 1.0, or 10.0; PL and PLClf use a 0.9 confidence threshold.Source: §A.2 · p. 14

A.3 IMPLEMENTATION DETAILS ON IMAGE CORRUPTION BENCHMARKS

The image-corruption experiments specify test-time optimization and support-set settings. The method uses efficient adaptation modules, while the appendix describes runtime comparisons and the TAST-BN support-set procedure.

A.3 IMPLEMENTATION DETAILS ON IMAGE CORRUPTION BENCHMARKS

  • The image-corruption experiments use ResNet-50, batch size 128 at test time, and four random seeds; TAST-BN has no support-set cap on CIFAR-100C.The experiments use level-5 corruptions, and TAST-BN is capped at 200 support examples for other image-corruption experiments.Source: §A.3 · p. 14

A.4 RUNTIME COMPARISON

  • TAST requires 1/3 to 1/4 the runtime of methods updating the entire feature extractor, while TAST-BN takes about 2x their runtime.The comparison reports mean runtime in seconds for ResNet-18 under T = 1, Ns = 8, and M = −1.Source: Table 4 · p. 15, §A.4 · p. 15

A.5 DETAILS ABOUT ADAPTATION MODULES

  • TAST uses BatchEnsemble adaptation modules, which share weights and use rank-one factors to reduce parameter count and computation.Unlike deep ensembles, BatchEnsemble members share a weight matrix and can be vectorized and trained simultaneously.Source: §A.5 · p. 15
  • TAST trains its adaptation modules at test time and predicts from their averaged class distributions, whereas SimCLR uses feature-extractor outputs for downstream tasks.TAST trains modules using nearest-neighbor-based pseudo-label distributions.Source: §A.5 · p. 15

B PSEUDOCODE FOR TAST-BN

  • TAST-BN fine-tunes feature-extractor BN layers and stores test samples, rather than feature representations, because its embedding space changes.Its procedure updates the support set, retrieves nearby examples, and trains BN layers to match nearest-neighbor pseudo-label and prototype-based class distributions.Source: B PSEUDOCODE FOR TAST-BN · pp. 15–16

C.1 EXPERIMENTAL RESULTS USING CLASSIFIERS TRAINED BY DIFFERENT LEARNING ALGORITHMS

Across CORAL-, MMD-, and Mixup-trained classifiers, TAST improves performance and generally exceeds T3A. The section also reports worse performance when adaptation modules and BN layers are fine-tuned together, and describes CIFAR-10C hyperparameter patterns.

C.1 EXPERIMENTAL RESULTS USING CLASSIFIERS TRAINED BY DIFFERENT LEARNING ALGORITHMS

  • TAST improves trained-classifier performance by 1.73%, 1.81%, and 2.30% on average with CORAL, MMD, and Mixup, and exceeds T3A by 0.21%, 0.14%, and 0.40%, respectively.The results use ResNet-18 on domain-generalization benchmarks.Source: §C.1 · p. 16, Table 5 · p. 17

C.2 FINE-TUNING BOTH ADAPTATION MODULES AND BN LAYERS SIMULTANEOUSLY

  • TAST-both performs worse than TAST and TAST-BN when fine-tuning adaptation modules and BN layers simultaneously.The authors conjecture that random module initialization and BN-induced representation changes negatively affect learning in other layers.Source: §C.2 · p. 16, Table 6 · p. 17

C.3 EXPERIMENTAL RESULTS USING DIFFERENT HYPERPARAMETERS ON CIFAR-10C

  • On CIFAR-10C, T3A performs best when M is 100, while TAST and TAST-BN perform better with smaller Ns.The comparison evaluates different combinations of Ns and M.Source: §C.3 · p. 17

C.4 SENSITIVITY ANALYSIS ON HYPERPARAMETERS

The appendix examines sensitivity to hyperparameters and test batch size. TAST’s reported accuracy is robust to the tested changes in temperature, module dimension, and batch size.

C.4 SENSITIVITY ANALYSIS ON HYPERPARAMETERS

  • For domain generalization, hyperparameters are selected using validation accuracy on held-out training-domain data; image-corruption hyperparameters are manually set as in Tent.The appendix reports results for other hyperparameter combinations in Tables 8–11.Source: §C.4 · p. 17
  • TAST’s accuracy is robust to changes in softmax temperature τ and adaptation-module output dimension dφ.The experiments use ResNet-18 classifiers learned by ERM on PACS; the default settings are τ = 0.1 and dφ = dz/4.Source: §C.4 · p. 18, Table 8 · p. 18, Table 9 · p. 19
  • TAST remains robust to changes in test batch size, unlike Tent and PL, whose performance drops with smaller batches.The comparison uses ResNet-18 classifiers learned by ERM on PACS.Source: §C.4 · p. 18, Table 12 · p. 19

D TAST ON IMAGENET-C

On ImageNet-C, the paper describes challenges for prototype-based TAST and evaluates a fixed-prototype variant. It reports effective adaptation when that variant is combined with TAST-BN’s self-training approach.

D TAST ON IMAGENET-C

  • ImageNet-C is a larger-scale, more diverse image-corruption benchmark, and the paper notes that TAST may struggle because good prototypes require sufficient per-class data unavailable as labels under TTA.The authors say pseudo-labeling alleviates this issue on CIFAR-10/100C but not ImageNet-C.Source: D TAST ON IMAGENET-C · p. 19
  • Classifier bias under domain shift can concentrate estimated test labels in a subset of classes, leaving some prototypes unupdated even after many batches.For Gaussian noise, the authors report that all prototypes had been updated at least once only after 768 of 782 batches.Source: D TAST ON IMAGENET-C · p. 20
  • Because the number of classes (1000) greatly exceeds the test batch size (64), few method prototypes are updated per test batch; the rest remain unupdated, potentially affecting prototype-based classification.Addressing this might require a batch size larger than 1000, which the authors say is impossible due to hardware cost.Source: D TAST ON IMAGENET-C · p. 20
  • The fixed-prototype TAST-BN variant initializes prototypes from the last linear classifier’s weights and keeps them fixed during test time.It is introduced to address difficulty obtaining good prototypes early in test time when class count greatly exceeds batch size.Source: D TAST ON IMAGENET-C · p. 20
  • TAST-BN with fixed prototypes achieves effective ImageNet-C adaptation when combined with self-training (entropy minimization).The reported result is for ImageNet-C at severity level 5; Table 13 reports accuracy.Source: D TAST ON IMAGENET-C · p. 20, Table 13 · p. 20

E FULL RESULTS

The appendix provides full benchmark results across domain-generalization datasets, backbone networks, and training approaches, alongside CIFAR corruption error-rate tables. The supplied CORAL summary states that TAST outperforms baselines on most benchmarks.

E FULL RESULTS

  • Full ERM-trained ResNet-18 results are reported separately for VLCS, PACS, OfficeHome, and TerraIncognita.These are the full results associated with Table 1 of the manuscript.Source: Table 14 · p. 20, Table 15 · p. 21, Table 16 · p. 21
  • Full ERM-trained ResNet-50 results are reported separately for VLCS, PACS, OfficeHome, and TerraIncognita.These are the full results associated with Table 1 of the manuscript.Source: Table 18 · p. 21, Table 19 · p. 22, Table 20 · p. 22
  • With CORAL-trained classifiers, TAST outperforms all baselines on most domain-generalization benchmarks.Table 22 covers VLCS, PACS, OfficeHome, and TerraIncognita with ResNet-18 and ResNet-50 backbones.Source: Table 22 · p. 23
  • The appendix gives full CORAL-trained ResNet-18 results for each of the four domain-generalization benchmarks.The corresponding full-result tables are for VLCS, PACS, OfficeHome, and TerraIncognita.Source: Table 23 · p. 23, Table 24 · p. 23, Table 25 · p. 24
  • Full CORAL-trained ResNet-50 results are also provided for VLCS, PACS, OfficeHome, and TerraIncognita.These tables accompany the CORAL benchmark summary.Source: Table 27 · p. 24, Table 28 · p. 24, Table 29 · p. 25
  • For ResNet-18, average-accuracy tables cover MMD- and Mixup-trained classifiers across the domain-generalization benchmarks.Separate full-result tables are supplied for each benchmark under both training approaches.Source: Table 31 · p. 25, Table 36 · p. 26
  • Full results for ResNet-18 classifiers trained by MMD are reported for VLCS, PACS, OfficeHome, and TerraIncognita; Mixup-trained ResNet-18 results are reported for VLCS and PACS.The MMD results are in Tables 32–35, and the Mixup results are in Tables 37–38.Source: Table 32 · p. 25, Table 33 · p. 26, Table 34 · p. 26
  • The appendix also includes full ERM-trained ResNet-18 results for the four domain-generalization benchmarks and average error rates for severe CIFAR-10C and CIFAR-100C corruptions.The ERM tables correspond to Table 2 of the manuscript; the corruption tables report average error rate (%).Source: Table 41 · p. 28, Table 42 · p. 28, Table 43 · p. 28
Loading 2207.10792v2…