PaperlayerView exact revision on arXiv

Source-linked AI summary

Not Every Correction Helps: Gain-Guided Continual Test-Time Adaptation

Youjia Zhang, Huiling Liu, Soyun Choi, Jaehong Yoon, Sungeun Hong

arXiv:2609.36655v1cs.CV

TL;DR

Current-sample confidence does not capture how evolving target history supports a prediction, and history-derived corrections may become misaligned. GAIN evaluates each correction’s source-relative gain to control intervention strength; across varied continual shifts, it reports strong accuracy, calibration, and stability.

  • Problem

    Current-sample reliability measures primarily reflect model self-certainty, while accumulated target history may become misaligned as the distribution changes.

    Source: §1 · p. 1

  • Method

    GAIN uses compact target statistics to propose corrections and an uncertainty-aware evaluator to determine sample-specific intervention strength from source-relative gain.

    Source: §1 · p. 2

  • Results

    Across structured, dynamic, mixed-domain, and long-horizon shifts, GAIN achieves strong accuracy, calibration, and stability; on ImageNet-C CSC it reaches 61.9% accuracy with 6.0% ECE.

    Source: §5.2 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    Gain-controlled predictions update target statistics online, limiting propagation of unreliable corrections without backpropagation, sample storage, or replay.

    Source: §1 · p. 2, §4.2 · p. 6

    The study is limited to closed-set CTTA, where source and target domains share the same label space.

    Source: §6 · p. 9

Abstract

from arXiv · show

Continual test-time adaptation (CTTA) adapts a source model to an unlabeled test stream whose distribution may change over time. Existing TTA methods often assess prediction reliability using confidence or entropy, which primarily reflect the model's self-certainty for the current sample. In CTTA, accumulated target observations can provide complementary evidence for correcting the source prediction, but this history may become misaligned as the target distribution changes. The key question is therefore not how much the correction differs from the source prediction, but whether and how strongly it should be applied. This paper proposes Gain-Aware INtervention (GAIN), a backpropagation-free CTTA framework guided by a simple principle: history proposes, gain decides. GAIN maintains compact target statistics to form a correction proposal and a posterior-predictive evaluator that accounts for estimation uncertainty. The resulting source-relative gain estimates the proposal's benefit and determines a sample-specific intervention strength along a continuous path through efficient one-dimensional optimization. Gain-controlled predictions then update the target statistics online, limiting the propagation of unreliable corrections, all without backpropagation, sample storage, or replay. Across five benchmarks, our method achieves strong predictive performance, with favorable accuracy--calibration--efficiency trade-offs in continual adaptation. On ImageNet-C, for example, GAIN achieves 61.9% accuracy with near-source calibration. It remains stable under diverse and challenging continual shifts while running 15.9x faster than a representative optimization-based CTTA baseline.

1 INTRODUCTION

Current-sample confidence does not capture how evolving target context supports a prediction, and history-derived corrections may become misaligned. GAIN evaluates each correction’s source-relative gain and uses it to control intervention strength.

1 INTRODUCTION

  • Existing reliability measures based on entropy, sample selection, or region-level confidence primarily characterize the model’s self-certainty rather than support from evolving target context.Accumulated target observations can provide complementary predictive evidence.Source: §1 · p. 1
  • History-derived corrections may become misaligned as the target distribution evolves, so a correction is not necessarily beneficial just because it differs from the source prediction.Erroneous corrections can accumulate in the target state and affect subsequent adaptation.Source: §1 · p. 1
  • GAIN uses accumulated target statistics to propose a correction and evaluates its source-relative gain while accounting for uncertainty in the evolving statistics.The resulting gain determines whether and how strongly the correction influences the current prediction.Source: §1 · p. 2
  • A concave one-dimensional objective selects intervention strength along a continuous path from source retention to full correction, and gain-controlled predictions update target state causally.The source model stays frozen; adaptation requires no backpropagation, parameter updates, or replay of past target samples.Source: §1 · p. 2
  • GAIN is reported to achieve a strong accuracy–calibration–efficiency trade-off across structured, dynamic, mixed-domain, and long-horizon continual shifts.It combines sample-specific intervention with online maintenance of compact target statistics, without sample storage or replay.Source: §1 · p. 2

2 RELATED WORK

Prior CTTA work addresses reliability, knowledge preservation, and distributional modeling in different ways. GAIN distinguishes itself by evaluating history-derived corrections against source retention and continuously controlling their influence.

Reliability and Stability in Test-Time Adaptation

  • Reliability methods use entropy, sample filtering or reweighting, stabilized optimization, or representation and geometry adaptation; CAS (Jiang et al., 2026) instead makes a binary adapt-or-skip decision.GAIN evaluates whether a history-induced correction improves over source retention and continuously controls its strength.Source: §2 · p. 2

Knowledge Preservation in CTTA

  • Knowledge-preservation methods use source restoration or ensembling, sample storage, domain-specific modules, knowledge pools, or synthetic source-anchor replay.GAIN keeps the source model frozen and uses compact target statistics without sample storage or replay.Source: §2 · p. 3

Distributional Modeling in CTTA

  • Probabilistic and distributional CTTA methods model target evidence, but accumulated statistics may become inaccurate or stale as the distribution changes.GAIN treats the target estimate as a correction proposal and evaluates its source-relative gain before deciding its influence.Source: §2 · p. 3

3 PRELIMINARIES

CTTA adapts a frozen source model to an unlabeled target stream that may change over time, using past observations as context. Although history can add predictive evidence, finite and potentially stale estimates make it necessary to assess whether a proposed correction benefits the current prediction.

3.1 CONTINUAL TEST-TIME ADAPTATION

  • CTTA considers an unlabeled target stream whose distribution may change over time, while the source model remains frozen for single-pass adaptation without gradient-based updates.Preceding target observations form the accumulated context used during adaptation.Source: §3.1 · p. 3
  • Confidence and entropy reflect source-model self-certainty but ignore accumulated target context, whose relevance may diminish as the distribution changes.The section asks whether conditioning on target history provides useful evidence for the current sample.Source: §3.1 · p. 3

3.2 ACCUMULATED TARGET CONTEXT AS PREDICTIVE EVIDENCE

  • Historical context contributes class-dependent evidence beyond the current observation, with positive or negative conditional pointwise mutual information indicating evidence for or against a class.The history-conditioned probability conditions on the current observation and accumulated target context.Source: §3.2 · p. 3
  • GAIN uses a target proposal and an uncertainty-aware predictive evaluator to estimate source-relative gain along a continuous path from source prediction to correction.Oracle usefulness does not guarantee that an estimate from finite, unlabeled, potentially stale observations yields a beneficial correction; the target state is updated for future batches.Source: Figure 2 · p. 4, §3.2 · p. 4
  • Under logarithmic loss, conditioning on accumulated target context has non-negative predictive value for any predictive distribution in the oracle setting.Equality holds only under the condition stated in Proposition 3.1; the supplied passages do not specify it fully.Source: §3.2 · p. 4

4 GAIN: GAIN-AWARE INTERVENTION

GAIN uses accumulated target statistics to propose corrections, evaluates their source-relative gain under uncertainty, and lets that gain control both intervention strength and subsequent target-state updates.

4 GAIN: GAIN-AWARE INTERVENTION

  • Recursive sufficient statistics retain the aggregate effect of historical representations without storing or replaying them.These statistics support the history-derived target correction proposal.Source: §4 · p. 4, §4.1 · p. 5

4.1 RELIABLE TARGET-SIDE GAIN ESTIMATION

  • A source-centered probabilistic model uses accumulated target evidence to form class-specific correction proposals, with the source prototype anchoring estimates when evidence is limited or noisy.Target features are modeled with class-conditional Gaussians; recursively maintained sufficient statistics summarize preceding observations.Source: §4.1 · p. 5
  • The posterior-predictive evaluator accounts for class-center uncertainty: a correction must offer source-relative benefit and remain supported after uncertainty is considered.Its gain balances the potential benefit of correcting the source prediction against the mismatch between the proposed correction and its posterior-predictive evaluation.Source: §4.1 · p. 5

4.2 POSTERIOR-PREDICTIVE EVIDENCE INTERVENTION

  • The posterior-predictive objective is concave over the intervention path, yielding a globally optimal coefficient for scaling the correction relative to the source prediction.The evaluator assesses the correction rather than serving as an additional replacement prediction.Source: §4.2 · p. 6
  • Gain-controlled predictions provide reliability-weighted assignments for online updates to target statistics, limiting propagation of unsupported corrections into subsequent adaptation.The updated state is used from time t+1 onward, without storing or replaying previous target samples.Source: §4.2 · p. 6

5 EXPERIMENTS

Experiments evaluate GAIN across five benchmarks and four continual-stream settings. GAIN combines strong accuracy and calibration across ImageNet-C scenarios with a favorable accuracy–calibration–efficiency trade-off on ImageNet-3DCC and in intervention and efficiency analyses.

5.1 EXPERIMENTAL SETUP

  • Evaluation spans ImageNet-C, ImageNet-3DCC, ImageNet-R, ImageNet-V2, and ImageNet-Sketch, using accuracy and ECE across four continual-stream settings.The experiments use a frozen ImageNet-pretrained ViT-B/16; ImageNet-C and ImageNet-3DCC use severity 5 unless otherwise specified.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS ON IMAGENET-C

  • GAIN achieves 61.9% CSC accuracy with 6.0% ECE, 61.8% CDC mean accuracy, 71.5% MDS accuracy with 6.1% ECE, and stable 10-round LHA performance with ECE around 6.5%.LHA accuracy rises from 61.9% in R1 to 62.5–62.6% thereafter.Source: §5.2 · pp. 7–8, Table 1 · p. 7, Figure 3 · p. 8

5.3 EXPERIMENTS ON IMAGENET-3DCC

  • On ImageNet-3DCC, GAIN achieves the highest classification accuracy at 18.5× CoTTA inference speed, with low calibration error under CSC.The evaluation covers geometry- and imaging-related corruptions beyond conventional 2D corruptions.Source: Figure 4 · p. 8, §5.3 · p. 8

5.4 ABLATION STUDIES AND FURTHER ANALYSIS

  • Under CSC, GAIN’s source-relative posterior-predictive gain intervention achieves 61.9% accuracy, 6.0% ECE, and 1.9 NLL, outperforming fixed, entropy-based, and disagreement-based rules.Source retention forgoes target evidence, while full correction substantially improves accuracy but yields poor calibration.Source: §5.4 · p. 8, Table 2 · p. 9
  • Increasing κ0 from 0.5 to 2 raises accuracy from 61.6% to 61.8% and lowers ECE from 7.9% to 6.1%; accuracy peaks at 61.9% with κ0 = 3.Performance remains stable for κ0 from 2 to 4; larger values slightly improve ECE but gradually reduce accuracy.Source: §5.4 · p. 9, Figure 5 · p. 9
  • Batch-size performance largely saturates beyond 16 samples: accuracy remains within 61.8–61.9%, while ECE decreases to 6.0% at batch size 128.At batch size 1, accuracy is 60.2% and ECE is 11.4%; at 16, they are 61.9% and 6.3%.Source: §5.4 · p. 9, Figure 5 · p. 9
  • On ImageNet-C, GAIN reaches 15.9× CoTTA speed, 61.9% accuracy, and 6.0% ECE using a single forward pass and 0.81 GB of memory.Compared with BP-free DOTA, GAIN is faster, improves accuracy by 4.7 points, and reduces ECE from 38.2% to 6.0%.Source: §5.4 · p. 9, Table 3 · p. 9

6 CONCLUSION

GAIN evaluates history-derived corrections relative to retaining the source prediction, then controls their influence using gain-guided intervention. The appendix develops why source-only reliability is insufficient and how accumulated target context supplies predictive evidence.

6 CONCLUSION

  • GAIN uses a posterior-predictive evaluator and causal target-statistic updates to control history-derived corrections, limiting unreliable propagation without backpropagation or replay.Across structured, dynamic, mixed-domain, and long-horizon shifts, it achieves strong accuracy, calibration, and stability.Source: §6 · p. 9

A SOURCE-ONLY RELIABILITY IS INSUFFICIENT

  • The same source prediction can have different conditional correctness across target histories, so source-only confidence cannot recover adaptation reliability in all histories.The result does not imply confidence is uninformative; it is insufficient when correctness depends on evolving target context.Source: A SOURCE-ONLY RELIABILITY IS… · p. 13

B.1 CONDITIONAL POINTWISE MUTUAL INFORMATION REPRESENTATION

  • Conditioning on history reweights the current-sample posterior through class-dependent evidence, and the resulting log-posterior correction is conditional pointwise mutual information.The passage identifies C-PMI as the quantity associated with a particular realization, rather than an ad hoc calibration score.Source: §B.1 · p. 14

B.2 NON-NEGATIVE ORACLE PREDICTIVE VALUE

The section distinguishes the oracle value of conditioning on target history from the practical value of corrections estimated from finite unlabeled observations. It then describes a source-anchored probabilistic estimate of target class centers.

B.2 NON-NEGATIVE ORACLE PREDICTIVE VALUE

  • Conditioning on accumulated target context has non-negative oracle predictive value under logarithmic loss, but that result does not ensure a finite-sample estimate improves on the source prediction.Practical benefit depends on the estimation gap between the estimated and oracle history-conditioned posteriors.Source: §B.2 · p. 14

B.3 PRACTICAL CORRECTION GAIN

  • The practical gain from full correction is not guaranteed positive because its potential benefit is offset by the finite-sample estimation gap.Evolving target distributions can make accumulated statistics noisy, biased, or stale.Source: §B.3 · p. 15

C.1 SOURCE-ANCHORED PROBABILISTIC TARGET ESTIMATION

  • GAIN constructs a posterior over target class centers from a source-centered prior and reliability-weighted soft responsibilities derived from unlabeled target representations.The posterior mean estimates each target center, while its uncertainty reflects accumulated reliability-weighted support.Source: §C.1 · pp. 15–16

C RELIABLE TARGET-SIDE GAIN ESTIMATION

  • The posterior-mean target proposal uses estimated class centers, but leaves remaining center-estimation uncertainty for the posterior-predictive evaluator to account for.The method maintains the aggregate effect of soft responsibilities through recursive sufficient statistics without storing or replaying historical representations.Source: §C.1 · pp. 16–17

C.2 POSTERIOR-PREDICTIVE GAIN EVALUATION

GAIN evaluates a proposed target correction with a posterior-predictive distribution that accounts for class-center uncertainty. It selects intervention strength along a continuous source-to-target evidence path, with optimality restricted to that path and evaluator.

C.2 POSTERIOR-PREDICTIVE GAIN EVALUATION

  • The posterior-predictive evaluator accounts for class-center uncertainty, so source–target disagreement alone does not establish that a proposed correction is reliable.Its predictive covariance includes within-class representation variability and uncertainty in the estimated class center.Source: §C.2 · pp. 17–18

D.1 CONTINUOUS EVIDENCE PATH AND PREDICTIVE OBJECTIVE

  • GAIN scales source-relative evidence along a continuous path from source retention to the target proposal and evaluates risk reduction along that path.The intervention coefficient can be interpreted as a forward-KL projection of the evaluator onto the path.Source: §D.1 · p. 19

D.2 GLOBALLY OPTIMAL POSTERIOR-PREDICTIVE INTERVENTION

  • The path-wise objective is concave, giving a unique optimum except in the degenerate case; an interior solution is found by bisection in O(log(1/ϵλ)) iterations.The optimal partial intervention matches path-wise expected source-relative evidence to the gain supported by the posterior-predictive evaluator.Source: §D.2 · p. 20

D.3 SCOPE OF OPTIMALITY AND RELATION TO TARGET RISK

  • The selected prediction is optimal with respect to posterior-predictive logarithmic risk on the prescribed evidence path, not over all predictive distributions.The stated result fixes the current representation and pre-time-step statistical state.Source: §D.3 · pp. 20–21

D.4 WHY USE THE POSTERIOR-PREDICTIVE DISTRIBUTION AS AN EVALUATOR?

GAIN separates the target-side correction proposal from its uncertainty-aware evaluator, rather than directly predicting with the evaluator. Its online statistics use gain-controlled predictions and are updated causally after prediction.

D.4 WHY USE THE POSTERIOR-PREDICTIVE DISTRIBUTION AS AN EVALUATOR?

  • Directly predicting with the posterior-predictive evaluator is inferior to using it to assess the posterior-mean proposal; using one distribution to propose and evaluate instead yields full adoption.The proposal reflects estimated target geometry, while the evaluator marginalizes class-center uncertainty.Source: §D.4 · p. 21

E.1 CONTINUAL TARGET STATISTICS

  • GAIN maintains compact sufficient statistics for target centers, covariance, and class support, with a source prior stabilizing center estimates when target support is limited.With diagonal shared covariance, the statistics require O(KD) memory independent of stream length, without storing or replaying earlier samples.Source: §E.1 · pp. 22–23

E.2 OVERALL ALGORITHM

  • The algorithm predicts an entire mini-batch using the pre-batch state, then updates sufficient statistics, so current-batch observations affect only subsequent adaptation.This predict-then-update protocol avoids within-batch feedback and limits propagation of unreliable corrections.Source: §E.1 · p. 22, §E.2 · p. 23

F DETAILED EXPERIMENTAL SETUP

The experiments evaluate GAIN on ImageNet-C and additional image-domain shifts under several continual-stream protocols, with accuracy and calibration as primary metrics. The source network remains frozen, and adaptation maintains target statistics without accessing labels.

F DETAILED EXPERIMENTAL SETUP

  • ImageNet-C experiments use 15 corruption types at severity 5, with 5,000 images per corruption and 75,000 samples per complete cycle.Additional evaluations cover ImageNet-3DCC for continual adaptation and ImageNet-R, ImageNet-V2, and ImageNet-Sketch for standard TTA.Source: F DETAILED EXPERIMENTAL SETUP · pp. 23–24
  • Evaluation reports top-1 accuracy and expected calibration error, with negative log-likelihood added for ablations.Accuracy and ECE are reported in percentage points; higher accuracy and lower ECE are better.Source: F DETAILED EXPERIMENTAL SETUP · p. 24
  • Four stream settings test structured changes, irregular recurring domains, interleaved mixed domains, and ten repeated corruption cycles without resets or domain-boundary information.The settings are CSC, CDC, MDS, and LHA; the adaptation method receives no corruption identity or domain boundary.Source: F DETAILED EXPERIMENTAL SETUP · p. 25
  • Experiments use a frozen ImageNet-pretrained ViT-B/16 with mini-batches of 64 and recursive target statistics; the intervention coefficient is solved by one-dimensional bisection.Hyperparameters and tolerances remain fixed across the main continual settings.Source: F DETAILED EXPERIMENTAL SETUP · p. 25

G.1 DETAILED EXPERIMENTAL RESULTS ON IMAGENET-C

Across dynamic, mixed-domain, and long-horizon ImageNet-C streams, GAIN maintains strong accuracy and calibration. Its results show stable performance across irregular shifts, heterogeneous streams, and repeated cycles without resets.

Continual Dynamic Change

  • 61.8% accuracy with 6.2% ECE gives GAIN the highest CDC accuracy, beating REM by 2.5 accuracy points while reducing ECE by 2.6 points.The passage also reports degradation from CSC to CDC for DPCore, ReCAP, and PAID, and severe miscalibration for ROID and DOTA.Source: §G.1 · p. 26

Mixed-Domain Shift

  • 71.5% average accuracy and 6.1% average ECE make GAIN the top-accuracy method in MDS, ahead of REM (70.4%) and NEO (70.2%).Its average ECE is below NEO (11.0%) and DOTA (26.3%).Source: §G.1 · p. 26

Long-Horizon Adaptation

  • Across ten unrestarted corruption cycles, GAIN's accuracy rises from 61.9% to 62.5–62.6% while ECE remains around 6.5%.DPCore's accuracy declines, PAID deteriorates, and REM and ReCAP become increasingly miscalibrated despite competitive accuracy.Source: §G.1 · p. 27

G.2 FURTHER ABLATION STUDY AND ANALYSIS

The analyses examine GAIN's proposal–evaluator asymmetry, sample-wise intervention, and target-state updates, alongside further ImageNet-3DCC results. They report that gain-guided intervention tracks correction utility and that GAIN remains stable across diverse continual shifts.

G.2 FURTHER ABLATION STUDY AND ANALYSIS

  • Using the posterior-mean proposal directly yields 58.9% accuracy and 12.8% ECE, while reversing proposal and evaluator roles retains 61.7% accuracy but raises ECE from 6.0% to 8.0%.These comparisons support using the proposal to specify the correction and the posterior-predictive evaluator to assess source-relative utility.Source: §G.2 · p. 27
  • GAIN achieves 61.9% accuracy with 6.0% ECE, versus a best fixed-intervention accuracy of 61.0% and a lowest nonzero fixed ECE of 8.9%.Increasing fixed intervention strength initially improves accuracy, but aggressive correction eventually degrades both accuracy and calibration.Source: §G.2 · p. 27
  • Weighting the gain-controlled prediction by source support raises accuracy from 59.9% to 61.9% and reduces ECE from 11.9% to 6.0%.The reliability-weighted responsibility reduces the contribution of predictions weakly supported by the frozen source to future target statistics.Source: §G.2 · p. 27
  • The full reliability-balanced historical prior reaches 61.9% accuracy, 1.9 NLL, and 6.0% ECE, with reliability-normalized support and inverse-support balancing serving complementary roles.The former favors better-supported classes; the latter prevents frequently predicted classes from progressively dominating the prior.Source: §G.2 · p. 28
  • The estimated posterior-predictive gain is positively associated with realized correction gain across samples (Pearson r = 0.614; Spearman ρ = 0.532).The realized gain uses ground-truth labels only for retrospective evaluation; correction benefit also changes around distribution shifts, with some alternatives yielding limited or negative gain.Source: §G.2 · p. 28

G.3 MORE RESULTS ON IMAGENET-3DCC

  • On ImageNet-3DCC, GAIN achieves the highest average accuracy in CSC (60.5%), CDC (60.1%), and MDS (69.6%), and maintains 60.5–60.8% accuracy with 8.2% average ECE over ten LHA cycles.The evaluation covers varied corruption mechanisms, including depth of field, lighting and weather, video compression, and camera motion.Source: §G.3 · p. 29

G.4 EVALUATION UNDER TEST-TIME ADAPTATION

GAIN is also evaluated in standard TTA on ImageNet-R, ImageNet-V2, and ImageNet-Sketch. It achieves the highest mean accuracy across these target domains, with its best domain results on ImageNet-V2 and ImageNet-Sketch.

G.4 EVALUATION UNDER TEST-TIME ADAPTATION

  • GAIN achieves the highest mean top-1 accuracy of 63.4% across ImageNet-R, ImageNet-V2, and ImageNet-Sketch.The reported evaluation compares top-1 accuracy on each target domain and their average.Source: §G.4 · p. 29, Table 13 · p. 30
  • GAIN performs best on ImageNet-V2 and ImageNet-Sketch among the evaluated target domains.The evaluation examines whether gain-guided intervention generalizes to non-continual domain shifts.Source: §G.4 · p. 29
  • The standard TTA evaluation covers shifts from ImageNet to ImageNet-R, ImageNet-V2, and ImageNet-Sketch.The reported table gives top-1 accuracy for each target domain and their average.Source: §G.4 · p. 29, Table 13 · p. 30
Loading 2609.36655v1…