PaperlayerView exact revision on arXiv

Source-linked AI summary

WISE-ATTA: When to Ask for Labels in Budgeted Active Test-Time Adaptation

Muhammad Huzaifa, Lea Schönherr, Thorsten Eisenhofer

arXiv:2609.37687v1cs.AI

TL;DR

Existing ATTA methods generally assume labels can be requested for each test batch, while annotation costs accumulate over long streams and the benefits of supervision vary over time. WISE-ATTA allocates a limited label budget across batches and selects drifting samples for supervision. Across synthetic corruptions and natural distribution shifts, it matches or improves on recent ATTA methods while using up to 50% fewer labels.

  • Problem

    Existing ATTA methods reduce labels within batches but leave open how to allocate limited supervision over time, despite annotation costs accumulating across long streams.

    Source: §1 · p. 1

  • Method

    WISE-ATTA allocates a global label budget across batches using online utility signals, then selects a single sample whose predictions continue to drift relative to an exponential-moving-average anchor.

    Source: §1 · p. 2

  • Results

    On ImageNet-C/R/K/A, WISE-ATTA matches or improves on state-of-the-art ATTA methods while using up to 50% fewer labels.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings underscore that temporal allocation of supervision matters for stable, label-efficient test-time adaptation.

    Source: §7 · p. 9

    WISE-ATTA still updates the model at every time step, so compute-constrained settings may also need to budget updates themselves.

    Source: §5 · p. 8

Abstract

from arXiv · show

Active test-time adaptation (ATTA) improves robustness under distribution shift by updating a deployed model during inference while selectively querying supervision. However, most existing ATTA methods implicitly assume that supervision can be requested for every incoming test batch, which can incur substantial annotation cost over long test streams. In this work, we introduce \emph{budgeted ATTA} in which labels are available for only a fraction of test batches. This formulation shifts the central challenge from deciding \emph{what} to label within a batch to deciding \emph{when} supervision should be applied over time. To address this challenge, we propose a budget-aware approach \emph{WISE-ATTA} that allocates supervision over the test stream based on lightweight signals computed online, prioritizing periods where supervision is likely to be most useful. When a batch is selected for supervision, we further employ a drift-based sample selection criterion that targets samples exhibiting ongoing, unconverged adaptation dynamics, enabling effective updates from a single labeled example. We evaluate this approach on synthetic corruptions (ImageNet-C) and natural distribution shifts (ImageNet-R/K/A). Across settings, WISE-ATTA achieves competitive or improved performance compared to recent ATTA methods while requiring substantially fewer labels. Overall, we find that the timing of supervision is a key, yet underexplored, aspect of active test-time adaptation. Code: https://github.com/Muhammad-Huzaifaa/WISE-ATTA

1. Introduction

Existing active test-time adaptation methods allocate labels within batches, leaving open how to use limited supervision over a costly, continual test stream. WISE-ATTA budgets labels across batches and time, matching or improving on recent methods with fewer labels.

1. Introduction

  • On ImageNet-K, WISE-ATTA achieves the lowest average error with fewer labels; across ImageNet-C/R/K/A, it matches or improves on recent ATTA baselines using substantially fewer labels.The contribution summary reports up to 50% fewer labels.Source: Table 1 · p. 1, §1 · p. 2
  • Sparse supervision matters because labels may require costly human experts or teacher models, and even low annotation rates can accumulate substantial deployment costs.Unsupervised adaptation can also become unstable over long streams as errors accumulate and representations drift.Source: §1 · p. 1
  • Prior ATTA methods generally annotate every incoming batch and reduce labels within batches, although supervision benefits can vary across time.Under a fixed global budget, uniform temporal allocation can be inefficient.Source: §1 · p. 1
  • WISE-ATTA estimates incoming-batch utility to allocate a global label budget over time, addressing when to supervise as well as what to label.The setting makes labels available for only a limited subset of batches over a long test stream.Source: §1 · pp. 1–2
  • When a batch is selected, WISE-ATTA chooses one sample by comparing current predictions with a temporally smoothed exponential-moving-average anchor.Persistent prediction drift relative to the anchor indicates that adaptation has not stabilized.Source: §1 · p. 2

2. Test-Time Adaptation

Test-time adaptation updates a pretrained model using incoming data during deployment, without requiring labels in its standard form. Online methods use entropy minimization, while active methods add queried labels; existing ATTA often selects samples within batches rather than allocating supervision over time.

2. Test-Time Adaptation

  • Test-time adaptation offers online deployment updates from incoming test data when continuous retraining is impractical because distributions evolve or supervision is limited.In standard TTA, updates use the incoming stream without ground-truth labels.Source: §2 · p. 2

2.1. Online Test-Time Adaptation

  • In streaming TTA, a pretrained model adapts to sequential batches under distribution drift, often using entropy minimization and favoring low-entropy samples to avoid noisy, unstable updates.High-entropy predictions can induce noisy gradients and unstable updates.Source: §2.1 · p. 2

2.2. Active Test-Time Adaptation

  • Active TTA adds queried ground-truth labels and a supervised objective, but prior methods mainly reduce annotations within each batch rather than allocate supervision over time.Examples include selecting representative or highly learnable samples; this batch-centric view leaves temporal allocation open.Source: §2.2 · pp. 2–3

3. Budgeted Active Test-Time Adaptation

Budgeted ATTA limits supervision to a subset of test batches, making the timing of labels a central decision. WISE-ATTA allocates labels online and selects a single sample using its adaptation drift.

3. Budgeted Active Test-Time Adaptation

  • Budgeted ATTA permits labels for only a limited subset of batches, coupling when to supervise with how to use each queried label.WISE-ATTA separates these decisions into budget-paced batch selection and drift-based single-sample selection.Source: §3 · p. 3

3.1. Batch Selection

  • The batch utility is the fraction of predictions with entropy below a fixed threshold, used as a proxy for adaptation reliability.The method uses this online confidence signal to identify batches where sparse corrective supervision may be reliably incorporated.Source: §3.1 · p. 3
  • A sliding history of recent utility scores sets an adaptive quantile threshold, while a pacing controller adjusts the effective annotation rate toward the target without knowing stream length.When labels have been used too slowly, the effective rate rises and the threshold becomes more permissive; when spent too quickly, it falls and the threshold tightens.Source: §3.1 · pp. 3–4
  • During warmup, the method samples batches at the target rate in expectation, then applies a rate-floor correction to limit persistent budget underuse.The correction forces annotation when usage falls below the target by more than a slack amount.Source: §3.1 · p. 4

3.2. Sample Selection

  • Drift-based sample selection chooses the batch sample whose predictions change most between the current model and an exponential-moving-average anchor.Prediction drift targets samples still undergoing adaptation but not yet stabilized, rather than relying on a static uncertainty snapshot.Source: §3.2 · p. 4

3.1. Batch Selection

  • At 0.5 labels/batch, budget-paced selection consistently improves over uniform and random allocation across ImageNet-C CTTA/FTTA and ImageNet-R/K/A FTTA.Figure 1 reports change in error relative to EATTA sample selection with uniform batch allocation; lower is better.Source: Figure 1 · p. 5

4. Evaluation

The evaluation isolates batch selection and sample selection, then tests WISE-ATTA across label budgets and examines when labels are spent and which samples are queried.

4.1. Analysis of Batch Selection Strategies

  • Budget-paced selection generally yields the largest ImageNet-C error reductions, while drift-based sample selection improves over EATTA under matched uniform or random allocation.Combining both selectors provides further gains; on ImageNet-R/K/A, average improvements are 0.4 points for RN50-BN and 1.1 points for ViT-B-16.Source: §4.1 · pp. 5–6, Table 3 · p. 6

4.2. Effectiveness of Drift-Based Sample Selection

  • WISE-ATTA achieves the lowest average ImageNet-C error in CTTA (53.3%) and FTTA (51.6%), improving over same-budget EATTA by 1.3 and 1.1 points.It also beats three-label HILTTA by 0.4 points in CTTA and 1.2 points in FTTA; on ImageNet-R/K/A, it records the best average error for RN50-BN (70.9%) and ViT-B-16 (56.6%).Source: §4.2 · pp. 6–7

4.3. Performance evaluation under varying label ratio

  • Across ImageNet-R and ImageNet-K, WISE-ATTA outperforms UNIFORM and RANDOM at all tested label ratios, with the largest gains at low budgets.Raising r from 0 to 0.2 improves performance by 4.12 points on ImageNet-R with RN50-BN (58.02 →53.90) and 7.66 points on ImageNet-K with ViT-B-16 (66.78 →59.12); at r = 0.6, it nearly matches r = 1.0.Source: §4.3 · p. 7, Figure 2 · p. 7

4.4. Label Utilization over time

  • WISE-ATTA allocates labels non-uniformly, concentrating them earlier while continuing to query near the end of evaluation.Allocation is especially front-loaded on ImageNet-K; on ImageNet-R it peaks in the early-to-mid stream (≈10 −40 %).Source: §4.4 · p. 7, Figure 3 · p. 7

4.5. Selected Sample Analysis

  • WISE-ATTA avoids the highest-entropy samples and favors samples with higher drift relative to its EMA anchor.Max-Entropy targets an ambiguous high-entropy tail, while EATTA tends toward lower-to-moderate entropy and comparatively small drift; WISE-ATTA instead selects samples still adapting and responsive to correction.Source: §4.5 · pp. 7–8, Figure 4 · p. 8

5. Discussion

WISE-ATTA still updates at every time step, and its evaluation assumes batches are dominated by a single shift. Delayed labels sharply degrade ATTA methods, motivating delay-aware extensions.

5. Discussion

  • WISE-ATTA updates at every time step, so compute-constrained deployments may also need to budget updates themselves.Skipping unsupervised updates on non-selected batches remains competitive, but changes the balance between unsupervised and supervised correction.Source: §5 · p. 8
  • The study assumes each batch is dominated by one shift, leaving heterogeneous batches with co-occurring shifts as an untested setting.In that setting, batch utility may need to reflect coverage across multiple modes rather than short-term adaptation potential alone.Source: §5 · p. 8
  • Delayed labels sharply degrade ATTA methods, motivating recency-weighted updates, latency-aware batch selection, or queries that anticipate future model state.The text presents delay-aware approaches as a necessary step toward deployment-ready active TTA.Source: §5 · p. 8

6. Related Work

Prior work uses test-time adaptation to update pretrained models on unlabeled streams and active supervision to stabilize those updates. Most ATTA methods focus on what to label within each batch, leaving global-budget timing underexplored.

6. Related Work

  • TTA updates pretrained models on unlabeled test streams using unsupervised objectives such as entropy minimization or self-training [23] [41] [45].Under long or severe shifts, unsupervised TTA can face error accumulation, catastrophic forgetting, and feature-statistic drift [2] [16] [28] [38] [45].Source: §6 · p. 8
  • ATTA adds sparse supervision to stabilize online adaptation, with prior methods querying by uncertainty, online model selection, or teacher-guided distillation [9] [22] [3].Recent methods reduce supervision to a single label per batch [42].Source: §6 · p. 8
  • Most ATTA methods assume labels for every batch, so deciding when to request supervision under a global budget remains largely unexplored.Earlier work instead emphasizes selecting what to label within each batch.Source: §6 · p. 8

7. Conclusion

The conclusion presents WISE-ATTA as budgeted active test-time adaptation that allocates labels across batches and selects a drift-based sample within queried batches. The appendix covers method analyses, deployment studies, and implementation details.

7. Conclusion

  • Across synthetic corruptions and natural distribution shifts, WISE-ATTA achieves competitive or improved robustness while using substantially fewer labels.The conclusion highlights temporal supervision allocation as important for stable, label-efficient test-time adaptation.Source: §7 · pp. 8–9
  • WISE-ATTA combines budget-paced batch selection to decide when to query with drift-based single-sample querying to decide what to label.The setting makes supervision available for only a fraction of test batches, requiring allocation over time.Source: §7 · p. 8

Appendix Overview

  • The appendix analyzes supervision roles, batch-utility alternatives, queried samples, selection hyperparameters, and complete batch-selection results.These method analyses are followed by studies of skipped updates, batch sizes, and delayed labels, then algorithm, hyperparameter, update-parameter, and reproducibility details.Source: Appendix Overview · p. 12

A. Batch-Level vs. Sample-Level Supervision

WISE-ATTA uses different signals for batch selection and sample selection: batch stability guides whether to supervise, while prediction drift guides which sample to label.

A. Batch-Level vs. Sample-Level Supervision

  • Batch and sample supervision decisions answer different questions and therefore rely on different signals.Batch selection estimates whether supervision is useful now; sample selection identifies a useful example within that batch.Source: A. Batch-Level vs. Sample-Level… · p. 12
  • Batch selection favors reasonably aligned batches with many low-entropy predictions, rather than maximally uncertain or already converged batches.Low-entropy predictions serve as a proxy for adaptation stability and the likelihood that supervision will reinforce adaptation rather than destabilize it.Source: A. Batch-Level vs. Sample-Level… · p. 12
  • Within a selected batch, prediction drift relative to an EMA anchor targets samples still adapting but not so unstably that one label is unlikely to help.The sample criterion seeks ongoing, unconverged adaptation dynamics rather than the most confident or most uncertain example.Source: A. Batch-Level vs. Sample-Level… · p. 12
  • WISE-ATTA tends to query moderate-entropy samples rather than the highest-entropy tail, consistent with prediction entropy and drift capturing complementary behavior.The cited analysis is shown in Figure 5.Source: A. Batch-Level vs. Sample-Level… · p. 12

B. Impact of Batch Utility Functions

COUNT(H < τ) performs best among the tested batch-utility functions, while WISE-ATTA queries moderately uncertain samples with larger prediction drift. These query patterns are consistent across RN50-BN and ViT-B-16.

B. Impact of Batch Utility Functions

  • COUNT(H < τ) achieves the best overall performance, reducing average error by 0.82 points versus MEAN ENTROPY across datasets and backbones.Entropy-based utilities perform worse across settings; top-k drift is stronger than mean drift and all entropy-based alternatives.Source: B. Impact of Batch Utility Functions · pp. 12–13, Table 4 · p. 13

C. Query Behavior

  • WISE-ATTA selects samples with moderate entropy and larger prediction drift, unlike the extreme-entropy, near-zero-drift samples favored by MAX-ENTROPY.This pattern is consistent across RN50-BN and ViT-B-16 and coincides with the best performance.Source: C. Query Behavior · pp. 13–14, Figure 5 · p. 13

D. Sensitivity to Batch-Selection Hyperparameters

WISE-ATTA is reasonably stable across tested batch-selection hyperparameters, with modest gains from slack, longer history windows, and shorter warmup.

D. Sensitivity to Batch-Selection Hyperparameters

  • The average error varies only modestly across tested settings, and the spread is at most 0.5 points across each hyperparameter sweep.The tested parameters are history window W, warmup length M, and slack δ.Source: Table 5 · p. 14, D. Sensitivity to Batch-Selection… · p. 14
  • Adding slack δ improves overall average error from 64.58 to 64.56 by avoiding overly aggressive catch-up that can force low-utility batches.With slack, supervision can be reserved for more informative batches.Source: D. Sensitivity to Batch-Selection… · p. 14
  • Very small history windows, especially W = 1, perform worse, while larger windows such as W = 250 and W = 300 yield the best averages; M = 1 gives the strongest tested warmup result.Longer warmup delays utility-based selection by spending more of the label budget on random allocation.Source: D. Sensitivity to Batch-Selection… · pp. 14–15

E. Detailed Batch-Selection Results

The detailed batch-selection results cover synthetic corruptions and natural distribution shifts, with evaluations spanning both CTTA and FTTA settings.

E. Detailed Batch-Selection Results

F. Skipping Updates on Non-selected Batches

The analyses test simplified updates, varying batch sizes and annotation delays, the selection algorithm, supervised update choices, and benchmark coverage. They report where WISE-ATTA retains advantages and describe the experimental setup.

F. Skipping Updates on Non-selected Batches

  • Skipping unsupervised updates on non-selected batches changes WISE-ATTA’s average error from 55.72 to 55.82 (+0.10), and it retains its advantage over Uniform and Random.At label ratio r = 0.2, removing these updates has negligible impact and permits updating only selected batches.Source: F. Skipping Updates on Non-selected… · p. 16, Table 8 · p. 16

G. Batch Selection Under Different Online Batch Sizes

  • WISE-ATTA outperforms Uniform and Random across batch sizes; at size 16, ImageNet-R errors are 55.08 vs. 56.45/57.51 for RN50-BN and 41.53 vs. 41.77/42.14 for ViT-B-16.The gap narrows at larger sizes but WISE-ATTA remains best; at size 128, its RN50-BN ImageNet-R error is 53.06 vs. 53.98/54.07.Source: G. Batch Selection Under Different… · pp. 16–17, Table 9 · p. 16

H. Effect of Label Annotation Delay

  • All three methods degrade as annotation delay grows, but full WISE-ATTA has the lowest error in 14 of 20 dataset-model-delay combinations.For ViT-B-16 on ImageNet-K, error rises from 58.0 to 93.0 as delay increases from 0 to 200 batches.Source: H. Effect of Label Annotation Delay · p. 17, Table 10 · p. 17

I. Algorithm and Hyperparameters

  • The online selector tracks recent utility scores, uses a quantile threshold after warmup, and adjusts selection when cumulative label use lags its target.The algorithm specifies a target ratio, history window W, warmup M, correction horizon Hc, and slack δ; default values include W = 250, M = 1, Hc = 50, and δ = 1.Source: §I · p. 18

J. Supervised Update: Which Parameters to Adapt?

  • Unlabeled batches update normalization parameters with entropy minimization, while labeled samples also update the classifier head; head-only supervision is most stable.Updating intermediate or deeper layers often hurts under single-label supervision, as summarized in Figure 6.Source: J. Supervised Update: Which Parameters… · p. 18, Figure 6 · p. 19

K.1. Datasets and Models

  • Evaluation uses ImageNet-C synthetic corruptions and ImageNet-R/K/A natural shifts, with RN50-BN and ViT-B-16 starting from pretrained ImageNet-1K checkpoints.ImageNet-C results use severity level 5, and adaptation has no access to source-domain data.Source: §K.1 · pp. 18–19

K.2. Reproducibility

  • The experiments average results over three random seeds, and code, configuration files, and evaluation scripts are available.The reported seeds are 0, 41, and 58.Source: §K.2 · p. 19

K.3. Dataset Details

  • The benchmark descriptions identify ImageNet-C’s 15 corruption types, ImageNet-R’s artistic renditions, ImageNet-K’s sketches, and ImageNet-A’s natural adversarial examples.ImageNet-C evaluations report severity level 5; the full sequential evaluation streams are used.Source: §K.3 · pp. 19–20

Related papers

Loading 2609.37687v1…