Source-linked AI summary
Continual Test-Time Adaptation via Entropy Sensitivity-Guidance in Strict Online Setting
Chandler Timm C. Doloriel, Yunbei Zhang, Muhammad Salman Siddiqui, Tor Kristian Stevik, Fadi Al Machot, Kristian Hovde Liland, Habib Ullah
TL;DR
Strict online CTTA with batch size one and no source or replay data is vulnerable to drift and collapse from noisy, sample-specific updates. SEGA probes predictive-entropy changes under structured erasing and reuses the resulting sensitivity to control recovery and sample selection. Across corruption benchmarks and aquaculture proxy streams, it reports more stable, lower-error adaptation with fewer backward passes.
Problem
Strict online CTTA with batch size one and no access to source or past samples is prone to drift, collapse, and fragile entropy-based adaptation over long horizons.
Method
SEGA uses progressive multi-view erasing to derive per-sample predictive-entropy sensitivity, which drives both trend-based recovery and quantile-gated update selection.
Results
SEGA yields more stable and lower-error adaptation across ImageNet-C, CIFAR10/100-C, and corruption-generated aquaculture streams while reducing backward passes through sensitivity-based gating.
Takeaways & Limitations
Per-sample sensitivity provides a practical feedback signal for strict online CTTA on corruption-style streams, coordinating recovery and update selection without replay or model reservoirs.
Abstract
from arXiv · showhide
Test-time adaptation (TTA) promises robustness under distribution shift by updating a pretrained model on unlabeled test data, but strict online TTA with batch size one and no access to source data is especially prone to drift or collapse. We introduce Sensitivity-Guided Erasing Adaptation (SEGA), a method for strict online continual TTA (CTTA) on corruption-style streams. SEGA uses a small number of structured erasures to probe how predictive entropy changes as information is removed, and uses the resulting per-sample sensitivity trajectories to coordinate recovery and sample selection rather than relying on raw entropy or batch statistics. This yields a practical feedback signal for long-horizon batch-size-one adaptation without periodic resets or model reservoirs. In experiments on ImageNet-C, CIFAR10/100-C, and corruption-generated aquaculture streams treated as controlled corruption-style proxies, SEGA yields consistent robustness and stability gains over strong CTTA baselines while reducing backward passes through sensitivity-based gating.
1. Introduction
Strict online CTTA with batch size one and no source or past samples is vulnerable to noisy per-sample updates, drift, and collapse. SEGA addresses this regime by using per-sample entropy sensitivity from structured erasing to guide recovery and update selection, with robustness and stability gains across corruption-style streams.
- Problem: Batch-size-one CTTA is especially fragile because each unlabeled update is noisy and sample-specific, allowing overfitting to accumulate into collapse over long streams.The setting excludes source data, past samples, and batch or memory-based regularization.
- Prior approaches: Prior stability methods commonly rely on raw entropy, batch statistics, periodic resets, or domain reservoirs rather than per-sample entropy sensitivity.
- Evidence: Under a uniform SAR-defined cutoff on ImageNet-C, SEGA has the fewest incorrect predictions and about 10.4% of samples in the unreliable zone versus 19.7% for SAR.
- SEGA: SEGA computes sensitivity from predictive-entropy trajectories under progressive multi-view erasing and uses it for both trend-based recovery and quantile-based sample gating.The recovery rule responds to sustained upward sensitivity drift, while the gate skips the lowest-sensitivity samples.
- Evaluation: SEGA is evaluated under strict batch-size-one CTTA on ImageNet-C, CIFAR10/100-C, and corruption-generated aquaculture streams.
2. Related Work
Related work spans source-free test-time adaptation, continual adaptation, and stability methods, but many approaches depend on batches, memory, auxiliary modules, entropy, or style-based routing. SEGA instead targets strict online batch-size-one corruption-style CTTA with a single backbone and a shared per-sample sensitivity control signal.
- Early Test-Time Adaptation: Early TTA methods use unlabeled test inputs for adaptation, but are mostly evaluated on short benchmarks with batches or augmentations that do not expose strict online failure modes.
- Continual Test-Time Adaptation: CTTA methods reduce drift through teacher averaging, source prototypes, domain modules, masked reconstruction, prompt coresets, structural priors, or gradient subspaces.
- Test-Time Stability: Stability-focused methods use reliable-entropy filtering, periodic resets, or domain-specialized reservoirs, while M2A uses progressive masking as a probe or adaptation component.
- SEGA’s Position: SEGA occupies a targeted design point: single-backbone strict online CTTA at B=1 on corruption-style streams, using sensitivity for recovery and update selection.
- Comparison: Unlike RDumb and ReservoirTTA, SEGA avoids explicit replay, routing, and domain-specialized model storage.
3. Sensitivity-Guided Erasing Adaptation
SEGA uses entropy sensitivity from progressively erased views as a shared signal for recovery and sample selection in strict online CTTA. It combines trend-based resets with sensitivity-gated updates and a multi-view adaptation objective.
- Sensitivity measurement: SEGA computes per-sample sensitivity from predictive-entropy changes under progressive spatial erasing, treating it as an operational proxy for collapse risk.Stable predictions change mildly under erasing, whereas brittle or overfit predictions produce sharper entropy changes.
- Erasing views: SEGA generates erased views using levels defined by erasing step size A and count N, with default ImageNet-C settings (A, N) = (0.1, 3).The same defaults are reused on CIFAR-C and aquaculture streams.
- Empirical validation: Figure 2 links the workflow to validation: sensitivity supports reset timing and sample filtering, with resets near sensitivity spikes and roughly 30% of samples filtered.The figure also presents the tradeoff between deterioration margin D, reset frequency, and error.
- Sensitivity-trend control: The sensitivity-trend rule monitors the stream-level sensitivity EMA and restores source-state parameters when sustained deterioration exceeds a margin after a minimum horizon.The default recovery settings are (T, D) = (50, 6.0).
- Sensitivity-gated sample selection: The sensitivity gate retains mid-to-high-sensitivity samples and skips low-sensitivity samples, focusing updates on examples with useful adaptation signals.For B = 1, the default quantiles (Qmin, Qmax) = (0.2, 1.0) skip roughly the lowest-sensitivity 20% of samples.
- Adaptation objective: For retained samples, SEGA minimizes a multi-view consistency and entropy objective using detached soft targets, without labels or teacher networks.The consistency term links predictions across erasing levels, while the entropy term encourages confident predictions; the default λ is 1.
4. Experiments
SEGA is evaluated under strict online CTTA across standard corruption benchmarks, dynamic corruption streams, an aquaculture proxy, efficiency tests, and component ablations. Across these settings, it delivers robust and stable adaptation while using selective updates and fewer backward passes.
- Datasets and setup: Experiments use severity-5 CIFAR10/100-C, ImageNet-C, and corruption-generated FreshFish-C streams, primarily with ViT-Base and batch size 1.FreshFish-C applies the same 15 synthetic corruptions used for the standard benchmarks.
- Main benchmarks: SEGA achieves the lowest mean error on ImageNet-C and CIFAR10-C, reaching 37.1% and 10.8%, respectively.It wins 8 of 15 corruptions on each dataset and keeps ImageNet-C below 56% on every corruption.
- Additional benchmarks: SEGA also attains the lowest mean error on CIFAR100-C and FreshFish-C, extending gains beyond the two primary benchmarks.The aquaculture stream is constructed through the same corruption protocol as the controlled benchmark streams.
- Continual dynamic change: Under continual dynamic change, SEGA is the only method with the best mean on both datasets, reaching 39.7% on ImageNet-C and 11.8% on CIFAR10-C.On ImageNet-C, M2A collapses to 89.6% mean, while SEGA remains balanced across corruption groups.
- Efficiency and buffering: SEGA updates only 0.03% of parameters and reduces backward passes from 1K to 0.69K on ImageNet-C and from 2K to 1.32K on CIFAR10-C.It uses the same number of forward passes as M2A and achieves lower error at similar wall-clock time.
- Transfer beyond batch size 1: With variable batch sizes between 1 and 128 on FreshFish-C, SEGA improves across five passes from 22.5% to 11.2%, averaging 15.2%.SAR and RDumb remain near 28.4% and 28.3%, while M2A averages 24.8%; GradCAM maps retain attention on fish regions.
- Control components: Removing both controls raises ImageNet-C error to 62.2%, while the trend rule alone reaches 37.7% versus 37.1% for full SEGA.On CIFAR10-C, removing the trend rule raises error to 67.0%, whereas trend-only reaches 11.1% versus 10.8% for full SEGA.
- Erasing mechanism and hyperparameters: Spatial block erasing reaches 37.1% on ImageNet-C, outperforming frequency erasing at 42.3% and geometric transforms ranging from 50.5% to 71.3%.Moderate settings perform best: N=3, a 20%–100% erasing band, and approximately U=2048 warm-up steps.
5. Conclusion
SEGA uses per-sample sensitivity from multi-view erasing to guide recovery and sample selection in strict online CTTA, improving stability and efficiency on corruption-style streams. Its strongest support is for corruption-style shifts and architectures—especially transformer-style backbones—where erasing yields a useful sensitivity signal.
- SEGA provides a practical feedback signal for single-backbone strict online CTTA through per-sample sensitivity under multi-view erasing.The signal guides both model recovery and sample selection without relying on periodic resets or model reservoirs.
- SEGA yields more stable adaptation and lower long-horizon error than entropy- and reset-based schemes across standard corruption benchmarks and corruption-generated aquaculture streams.
- Sensitivity-based gating reduces unnecessary backward updates while the trend rule prevents long-horizon collapse.
- The strongest evidence supports corruption-style streaming shifts and architectures for which erasing produces a useful sensitivity signal, especially transformer-style backbones.
- Reset-to-source discards cross-domain adaptations, motivating partial rollback or weight interpolation when knowledge accumulation across domains is essential.