PaperlayerView exact revision on arXiv

Source-linked AI summary

Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors

Jonghyun Lee, Dahuin Jung, Saehyung Lee, Junsung Park, Juhyeon Shin, Uiwon Hwang, Sungroh Yoon

arXiv:2403.07366v1cs.CVcs.LG

TL;DR

Limited access to the full test set complicates TTA, and entropy alone can miss harmful samples under distribution shifts. DeYO combines entropy with PLPD, a measure of how object-shape distortion changes predictions. It consistently outperforms baselines across mild, wild, and biased scenarios in the reported experiments.

  • Problem

    Limited access to the entire test dataset during online TTA updates makes accurate test-distribution estimation difficult, and entropy reliability diminishes under severe spurious-correlation shifts.

    Source: §1 · pp. 1–2

  • Method

    DeYO combines entropy with PLPD, which measures the pseudo-label probability decrease after an object-destructive transformation, for sample selection and weighting.

    Source: §1 · p. 2

  • Results

    DeYO consistently outperforms baselines across ImageNet-C mild and wild scenarios and the reported biased benchmarks, with best performance in both mild and wild scenarios.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Entropy alone can miss harmful samples when predictions rely more on TRAP than CPR factors, motivating confidence measures that account for these disentangled influences.

    Source: §1 · p. 2, §2.3 · p. 5

    The theoretical analysis assumes a disentangled latent-factor representation and a linear classifier.

    Source: §2.2 · p. 3, §2.3 · p. 4

Abstract

from arXiv · show

Test-time adaptation (TTA) fine-tunes pre-trained deep neural networks for unseen test data. The primary challenge of TTA is limited access to the entire test dataset during online updates, causing error accumulation. To mitigate it, TTA methods have utilized the model output's entropy as a confidence metric that aims to determine which samples have a lower likelihood of causing error. Through experimental studies, however, we observed the unreliability of entropy as a confidence metric for TTA under biased scenarios and theoretically revealed that it stems from the neglect of the influence of latent disentangled factors of data on predictions. Building upon these findings, we introduce a novel TTA method named Destroy Your Object (DeYO), which leverages a newly proposed confidence metric named Pseudo-Label Probability Difference (PLPD). PLPD quantifies the influence of the shape of an object on prediction by measuring the difference between predictions before and after applying an object-destructive transformation. DeYO consists of sample selection and sample weighting, which employ entropy and PLPD concurrently. For robust adaptation, DeYO prioritizes samples that dominantly incorporate shape information when making predictions. Our extensive experiments demonstrate the consistent superiority of DeYO over baseline methods across various scenarios, including biased and wild. Project page is publicly available at https://whitesnowdrop.github.io/DeYO/.

1 INTRODUCTION

Limited access to test data makes reliable sample selection important for TTA, yet entropy can miss harmful samples under spurious-correlation shifts. DeYO combines entropy with PLPD, which measures how object-shape distortion changes predictions, and outperforms baselines across the reported scenarios.

1 INTRODUCTION

  • Limited access to the full test data makes distribution estimation difficult, while entropy reliability diminishes under severe spurious-correlation shifts.TTA methods use confidence metrics to select samples less prone to incorrect predictions and reduce error accumulation.Source: §1 · pp. 1–2
  • A sample can be harmful even at low entropy when its prediction relies more on TRAP factors, such as background, than CPR factors, such as shape.TRAP factors are positively correlated with labels only during training; CPR factors are commonly positively correlated with labels.Source: §1 · p. 2
  • DeYO uses PLPD to measure the pseudo-label probability decrease after object-shape distortion, selecting samples rooted in CPR factors and weighting those with low entropy and high PLPD more heavily.The method combines sample selection and sample weighting.Source: §1 · p. 2
  • DeYO consistently outperforms baselines across ImageNet-C mild and wild scenarios; on biased benchmarks, it surpasses pre-adaptation on Waterbirds and alone exceeds 50% worst-group accuracy on ColoredMNIST.The Waterbirds figure reports accuracy within the benchmark's worst group.Source: §1 · p. 2, Figure 1 · p. 2

2 REVISITING TTA: FROM THE PERSPECTIVE OF DISENTANGLED FACTORS

The section challenges entropy as a reliable confidence measure under spurious-correlation shifts and explains its limits through the influence of disentangled factors on predictions.

2.1 MOTIVATING OBSERVATIONS

  • On Waterbirds, the lowest-entropy quartile in the worst-performing group had lower accuracy than other entropy intervals, and entropy-based adaptation underperformed the pre-adaptation model.Grad-CAM visualizations showed correct predictions focusing on birds and incorrect predictions focusing relatively more on background.Source: §2.1 · p. 3

2.2 PRELIMINARIES

  • The analysis models an input with a disentangled latent-factor vector and, for its theoretical argument, assumes binary classification and a linear classifier.The classifier’s parameters are partitioned to correspond to the latent factors.Source: §2.2 · pp. 3–4, §2.3 · p. 4

2.3 ENTROPY IS NOT ENOUGH

  • A sample can be harmful for entropy-minimization adaptation when its prediction is more influenced by TRAP than CPR factors, even if its entropy is low.The paper identifies CPR factors as commonly label-correlated and TRAP factors as positively correlated with labels only during training; entropy cannot distinguish their influence from the compressed prediction.Source: §2.3 · pp. 4–5
  • When prediction relies more on TRAP than CPR contributions, adaptation can reduce class-logit separation and, in extreme cases, reverse the relative order of class logits.The proposed analysis motivates a confidence metric that avoids TRAP factors and incorporates CPR factors.Source: §2.3 · p. 5

3 METHODOLOGY

DeYO combines entropy and PLPD for sample selection and weighting, using changes in predictions after object-shape distortion to prioritize samples whose predictions rely on shape information.

3 METHODOLOGY

  • DeYO uses sample selection and sample weighting based on entropy and PLPD to account for CPR-factor influence, particularly object shape.The method is designed to address entropy’s unreliability when distribution shifts make predictions dependent on TRAP factors.Source: §3 · p. 5

3.1 SAMPLE SELECTION

  • PLPD measures the pseudo-label probability difference between an image and its patch-shuffled version, indicating how much shape information influences the prediction.Selection combines PLPD with entropy and requires one additional forward pass without additional back-propagation; patch-shuffling performed best among the tested transformations.Source: §3.1 · pp. 5–6

3.2 SAMPLE WEIGHTING

  • DeYO excludes samples in Areas 1–3 and varies the contribution of Area 4 samples to updates according to their reliability.Its weighting adds a PLPD-based term to entropy-based weighting, assigning larger weights when predictions are more grounded in object shape.Source: §3.2 · p. 6, Figure 4 · p. 6

3.3 OVERALL PROCEDURE OF DEYO

  • The overall procedure selects samples using Sθ(x), computes sample-wise weights αθ(x), and combines entropy-based and PLPD-based terms in selection and weighting.The weighted loss prioritizes samples whose predictions are particularly rooted in CPR factors.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments compare DeYO across mild, biased, and wild test scenarios, then examine PLPD’s role and DeYO’s hyperparameters and components. DeYO reports gains across these settings, while PLPD helps identify reliable samples beyond entropy alone.

4 EXPERIMENTS

  • The evaluation spans five benchmarks, mild and wild scenarios, new biased scenarios, and models with batch, group, or layer normalization.The benchmarks include ImageNet-C, ColoredMNIST, WaterBirds, ImageNet-R, and VisDA-2021.Source: §4 · pp. 6–7

4.1 MAIN RESULTS

  • DeYO leads under mixed corruption shifts at severity levels 3 and 5, with consistent results also reported on ImageNet-R and VisDA-2021.The mixed-shift performance improvement is described as less pronounced than in the other two wild scenarios.Source: Table 4 · p. 7, §4.1 · p. 8
  • DeYO exceeds EATA by 0.8% on mild ImageNet-C and the second-best methods by 17.43% on ColoredMNIST and 4.47% on WaterBirds.The comparisons cover accuracy; the two biased-scenario gains are reported respectively for ColoredMNIST and WaterBirds.Source: §4.1 · p. 7, Table 1 · p. 7, Table 2 · p. 7
  • With online imbalanced labels, DeYO beats baselines on ResNet50-GN and VitBase-LN; at batch size 1, it improves 7.8% over SAR on VitBase-LN.For VitBase-LN under imbalanced labels, DeYO leads on 14 corruption types, with Tent best on Zoom.Source: Table 5 · p. 8, §4.1 · p. 8

4.2 ROLE AND EFFECT OF PLPD

  • On Waterbirds’ worst group, entropy has larger AURC than PLPD variations; on ImageNet-C, high-PLPD areas are more accurate at matched entropy.The ImageNet-C comparison is for Gaussian noise at severity level 5 and uses pre-adaptation model accuracy.Source: §4.2 · pp. 8–9, Figure 5 · p. 8, Figure 6 · p. 8

4.3 HYPERPARAMETER AND ABLATION STUDIES ON DEYO

  • Patch-shuffling performs best; 4×4 patches peak, and τPLPD in [0.2, 0.3] yields excellent performance, while higher thresholds may limit adaptation knowledge.Patch-shuffling removes shape information while preserving other details; DeYO surpasses EATA from 3×3 patches onward.Source: §4.3 · p. 9, Figure 7 · p. 9
  • On ImageNet-C, combining PLPDθ and Entθ performs best; on WaterBirds, PLPDθ alone in sample selection outperforms using both.The WaterBirds ablation follows the same general trends, but entropy is less reliable under the severe shift.Source: Table 6 · p. 9, §4.3 · p. 9

5 CONCLUSION

The conclusion argues that low-entropy samples can still degrade adaptation when disentangled factors are ignored. It presents DeYO’s combination of shape-sensitive PLPD and entropy, reporting strong performance across mild and wild scenarios.

5 CONCLUSION

  • Even extremely low-entropy samples can degrade adaptation when the influence of disentangled factors is not considered.Source: §5 · p. 9
  • DeYO combines entropy with PLPD, which is designed to account for the influence of object shape on predictions.Source: §5 · p. 9
  • The paper reports DeYO as the first TTA method with the best performance in both mild and wild scenarios and significant gains across distribution shifts.Source: §5 · p. 9

A PROOF OF PROPOSITION 1

The proof relates a negative change in the mean-logit gap to reduced class discriminability, and DeYO combines entropy and PLPD for sample selection and weighting. The section also reviews TTA, disentangled factors, self-training, augmentation consistency, and adaptation efficiency.

A PROOF OF PROPOSITION 1

  • A negative change in the gap between class mean logits indicates that class discriminability decreases.The proof identifies conditions for this decrease, but those conditions are not fully stated in the supplied passages.Source: A PROOF OF PROPOSITION 1 · p. 16

B PSEUDO CODE OF DEYO

  • DeYO computes entropy, applies PLPD to eligible samples, selects and weights a reliable set, then minimizes a weighted entropy loss.The selection uses entropy and PLPD together, and the weighting accounts for both measures.Source: B PSEUDO CODE OF DEYO · p. 17

C EFFICIENCY

  • DeYO achieves the highest performance while requiring less adaptation time than SAR in the ImageNet-C Gaussian-noise severity-5 comparison.The runtime comparison uses ResNet-50-BN on 50,000 images and one A6000 GPU.Source: C EFFICIENCY · p. 18, Table 7 · p. 18

D.1 TEST-TIME ADAPTATION

  • TTA adapts at inference without ground-truth labels, making unsupervised losses such as entropy minimization central to its updates.Unlike continual learning, TTA adjusts a pre-trained feature space for a target benchmark at inference time.Source: §D.1 · p. 18

D.2 DISENTANGLED FACTORS

  • Disentanglement work seeks representations invariant to factors such as shape and color, while this paper focuses on weighted resampling with one augmentation step.The cited literature also uses data augmentation and representation learning to address shifts in factor distributions.Source: §D.2 · p. 19

D.3 SELF-TRAINING

  • Self-training can accumulate errors, and prior work also uses entropy to identify reliable target instances in unsupervised domain adaptation.The passage notes supervised-loss guidance as one response to error accumulation in semi-supervised learning.Source: §D.3 · p. 19

D.4 CONSISTENCY UNDER AUGMENTATIONS

  • Augmentation-consistency methods can select reliable instances by checking predictions across random transforms before optimizing on selected examples.The cited approach uses a committee of transforms to identify instances for alignment.Source: §D.4 · p. 19

E.1 BASELINE METHODS

  • The compared methods differ in their adaptation strategies: Tent minimizes entropy, EATA filters and weights samples, SAR uses sharpness awareness, and MEMO uses multiple augmentations.Except for MEMO, the listed methods adapt online during testing, with parameters reset after each dataset.Source: §E.1 · p. 20

E.2 MORE DETAILS ON BENCHMARK

The experiments span synthetic corruptions, natural distribution shifts, and benchmarks with spurious correlations. The protocol varies models and adaptation settings by benchmark and scenario.

E.2 MORE DETAILS ON BENCHMARK

  • ImageNet-C applies 15 corruptions, while ImageNet-R and Visda-2021 include more natural domain shifts such as artistic renditions and ObjectNet data.The evaluation covers five benchmarks, including ColoredMNIST and Waterbirds, which have strong training-time spurious correlations and weaker or absent test-time correlations.Source: §E.2 · p. 20

E.3 MORE DETAILS ON EXPERIMENTAL PROTOCOLS

  • The protocol adapts only normalization-layer parameters, typically uses batch size 64, and changes models, learning rates, and settings across benchmarks and scenarios.For label-shift experiments on ImageNet-R and Visda-2021, classes and samples within classes are randomized under an assumed infinite imbalance ratio.Source: §E.3 · p. 20
  • DeYO’s thresholds vary by scenario; ColoredMNIST and Waterbirds use a 0.5 label-flip threshold, and ColoredMNIST omits entropy filtering.The reported PLPD threshold is 0.3 for mild ImageNet-C, 0.5 for biased scenarios, and 0.2 for other experiments.Source: §E.3 · p. 21

F FURTHER EXPERIMENTS ON IMAGENET-C AT SEVERITY LEVEL 3

At severity level 3, the results are reported as consistent with severity level 5, with DeYO superior on average across corruption types. The tables separate mild and wild-scenario comparisons.

F FURTHER EXPERIMENTS ON IMAGENET-C AT SEVERITY LEVEL 3

F.1 COMPARISONS ON MILD SCENARIO

  • Table 8 compares baseline accuracy on ImageNet-C at severity level 3 under the mild scenario.Its caption says bold marks the top-performing result.Source: Table 8 · p. 21

F.2 COMPARISONS ON WILD SCENARIO

  • Table 9 compares baseline accuracy at severity level 3 under online imbalanced label shifts or batch-size-1 adaptation.The caption specifies an imbalance ratio of infinity for the label-shift setting.Source: Table 9 · p. 21

G FURTHER EXPERIMENTS ON IMAGENET-R AND VISDA-2021

DeYO is evaluated on ImageNet-R and Visda-2021 under mixed shifts, with additional label-shift and batch-size-1 scenarios. The reported comparisons favor DeYO across scenarios and architectures on both benchmarks.

G FURTHER EXPERIMENTS ON IMAGENET-R AND VISDA-2021

H FURTHER HYPERPARAMETER AND ABLATE RESULTS

The section reports DeYO hyperparameter choices and Waterbirds results on confidence reliability and component ablations.

H FURTHER HYPERPARAMETER AND ABLATE RESULTS

I FURTHER HYPERPARAMETER SENSITIVITY RESULTS

The appendix tests DeYO’s hyperparameter sensitivity and compares object-preserving AugMix with object-destructive transformations for PLPD and entropy filtering.

I FURTHER HYPERPARAMETER SENSITIVITY RESULTS

J EFFECT OF MULTIPLE OBJECT-PRESERVING AUGMENTATIONS

K EFFECT OF PLPD DURING PRE-TRAINING

The section reports PLPD-filtering results during pre-training, its role in wild scenarios, and a limitation of shape-based PLPD.

K EFFECT OF PLPD DURING PRE-TRAINING

L EFFECT OF PLPD IN WILD SCENARIOS

  • In wild label-shift and batch-size-1 settings, excluding TRAP factors gives the same robustness enhancement as in mild scenarios; CPR/TRAP information is less batch- or class-dependent than entropy.Source: L EFFECT OF PLPD IN WILD SCENARIOS · pp. 25–26
  • PLPD filtering favors CPR-dominant samples, selectively updating CPR weights while limiting TRAP-weight changes to preserve predictions for other ground-truth classes.Source: L EFFECT OF PLPD IN WILD SCENARIOS · p. 25

M LIMITATIONS

  • Identifying every CPR factor is practically impossible; shape-focused PLPD can be low when accurate local-feature predictions survive patch-shuffling.Source: M LIMITATIONS · p. 26
Loading 2403.07366v1…