PaperlayerView exact revision on arXiv

Source-linked AI summary

Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation

Hyun-Kurl Jang, Jihun Kim, Hyeokjun Kweon, Kuk-Jin Yoon

arXiv:2606.20196v2cs.CV

TL;DR

Continual test-time adaptation struggles to remain stable under long-term distribution shift when raw source data cannot be retained. DO-ALL distills source data into compact synthetic anchors that stabilize existing CTTA methods, consistently improving long-term robustness across CIFAR100-C, ImageNet-C, and CCC.

  • Problem

    Source-free CTTA lacks persistent source information, making stability under long-term, non-stationary target streams challenging because self-training becomes unreliable and forgetting accumulates.

  • Method

    DO-ALL distills the source dataset once into a tiny set of synthetic anchors and plugs them into existing CTTA methods as a stability buffer.

  • Results

    DO-ALL consistently improves long-term adaptation robustness across CIFAR100-C, ImageNet-C, and the CCC benchmark when integrated with diverse CTTA methods and DD strategies.

  • Takeaways & Limitations

    Distilled source data can serve as a compact, flexible memory for stabilizing CTTA without retaining the raw source dataset.

  • Takeaways & Limitations

    The paper identifies improving anchor selection and update schedules, and testing robustness to abrupt or adversarial target shifts, as open directions.

Abstract

from arXiv · show

Continual Test-Time Adaptation (CTTA) aims to maintain model performance under evolving target domains by adapting online without labeled data. However, practical deployments often cannot retain the source dataset due to privacy or licensing constraints, and purely source-free CTTA methods tend to become unstable under long-term distribution shift, suffering from compounding self-training errors and catastrophic forgetting. We introduce DO-ALL (Distill Once, Adapt Life-Long), a plug-and-play framework that revisits source information in a compact and privacy-conscious form via Dataset Distillation (DD). Before deployment, DO-ALL performs DD to produce a small set of synthetic distilled anchors that summarize the source distribution. During adaptation, each target sample is matched with its most semantically aligned anchor, which provides a stable reference for various CTTA via source replay, representation alignment, and manifold-smoothing regularization. DO-ALL can be seamlessly integrated into existing CTTA algorithms, consistently improving long-term robustness across CIFAR100-C, ImageNet-C, and the CCC benchmark. This demonstrates the potential of leveraging DD to enable stable and continuous adaptation without retaining raw source data. The code is available at https://github.com/blue-531/DOALL.

1 Introduction

DO-ALL addresses instability in source-free continual test-time adaptation by distilling source data into compact synthetic anchors before deployment. These anchors provide plug-and-play references that stabilize adaptation across evolving domains while avoiding raw-source retention.

  • Continual Test-Time Adaptation updates models online on unlabeled data from evolving domains while maintaining prediction quality.
  • Discarding source data limits long-term stability because unreliable self-training signals and repeated updates intensify catastrophic forgetting under distribution drift.
  • DO-ALL distills the source dataset once into a tiny set of synthetic samples that preserve class-discriminative structure while reducing storage and raw-data exposure.
  • DO-ALL integrates with existing CTTA algorithms without changing their objectives or architectures, acting as an optional stability buffer against drift and forgetting.Experiments report consistent improvements across CIFAR100-C, ImageNet-C, and the CCC benchmark, with stronger gains from more informative distilled anchors.
  • During adaptation, target samples are matched to semantically related anchors containing synthetic samples, source-model soft labels, and source-space latent features.The matched anchors guide source replay, MixUp-style regularization, feature-space alignment, and harm-adaptive restoration of unstable parameter groups.

2 Related work

Related work positions dataset distillation (DD) as a compact substitute for full datasets, including in continual learning and privacy-sensitive settings. Test-time adaptation (TTA) instead updates source-trained models online using unlabeled target data, commonly through entropy, consistency, or normalization-based strategies.

  • Dataset Distillation: Dataset distillation synthesizes compact image sets whose training effect approximates that of the full dataset, reducing storage and compute demands.Early DD methods incurred substantial computation and slow convergence, while later approaches improved scalability and stability.
  • Dataset Distillation: DD has been adopted for continual learning and privacy-sensitive regimes where raw data cannot be shared.DO-ALL uses distilled source data as a compact, privacy-aware anchor and explicit anti-forgetting prior during continual test-time adaptation.
  • Test-Time Adaptation: Test-time adaptation updates source-trained models on unlabeled target samples without accessing source data to address distribution-shift degradation.Common strategies include entropy minimization, consistency across stochastic augmentations, and normalization re-estimation for target activation statistics.

3 Method

DO-ALL distills the source dataset once into compact synthetic anchors that preserve source semantics and feature geometry, then matches each target sample to a semantically aligned anchor for continual adaptation. The method uses these correspondences for source replay, target–anchor manifold smoothing, and representation alignment without retaining the raw source dataset.

  • Problem formulation: CTTA adapts a source-trained model to a one-pass, unlabeled target stream drawn from time-varying distributions while avoiding catastrophic forgetting.The target stream is non-stationary, and target labels are unavailable at test time.
  • Source-distilled anchors: Before deployment, dataset distillation compresses the source dataset into a small synthetic set with Na ≪ |Ds|, providing a memory- and privacy-efficient source summary.The distilled set is constructed once and carried throughout CTTA instead of retaining the full source dataset.
  • Source-distilled anchors: Each source-distilled anchor packages a synthetic input, its source-model soft pseudo-label, and its latent feature representation.Soft labels preserve inter-class relations and confidence structure, while latent features capture source feature-space geometry.
  • Anchor retrieval: During adaptation, each target sample is matched to its nearest anchor by cosine similarity in feature space, forming the batch’s associated anchor set.This retrieves locally relevant source information and grounds updates in semantically consistent representation regions.
  • Adaptation objectives: A complementary representation-alignment objective keeps evolving features aligned with the anchor space and reduces representation drift across layers.The total adaptation loss combines the method’s replay and representation-preservation components.
  • Adaptation objectives: The replay objective combines temperature-scaled KL consistency on matched anchors with Mixup between each target sample and its anchor to preserve source predictions and smooth the target–anchor manifold.Direct replay regularizes the model toward source semantics, while anchor-based interpolation targets the region where adaptation occurs.

4 Experiments

Experiments evaluate DO-ALL for continual test-time adaptation on corruption and long-horizon distribution-shift benchmarks. The framework integrates with existing CTTA methods and consistently improves adaptation, including under severe ImageNet-C corruption and gradually shifting CCC streams.

  • Experimental setup: Experiments use CIFAR100-C and ImageNet-C, each with fifteen corruption types and five severity levels, reporting results at severity five.CTTA adapts online to fifteen corruptions, each presented as a stream of 5,000 unlabeled samples.
  • Baselines and integration: DO-ALL integrates with EATA, RMT, and ROID without modifying their adaptation objectives or architectures.The distilled anchor set is carried with the model and referenced during adaptation.
  • ImageNet-C results: At ImageNet-C severity level 5, EATA+DO-ALL reduces average error from 58.0% to 56.6%.The passage reports that DO-ALL improves each base method across nearly all corruption types.
  • ImageNet-C results: At ImageNet-C severity level 5, RMT+DOALL reduces average error from 59.8% to 57.4%.The improvements are attributed to anchoring representation updates, preventing drift under unstable adaptation dynamics.
  • CCC results: On CCC, DO-ALL improves ROID under long-horizon, continuously accumulating domain shifts that make models vulnerable to representation drift.Unlike ImageNet-C and CIFAR100-C, CCC gradually transitions the test stream across domains.

CCC Benchmark

On the CCC benchmark, DO-ALL uses distilled anchors as persistent references that prevent collapse over long adaptation periods. It improves short-term adaptation quality while maintaining robust long-term stability in continuously evolving environments.

  • Incorporating dataset-distillation information yields meaningful gains even on top of RMT, which already stores and uses source prototypes for CTTA.
  • Distilled anchors serve as a persistent reference that prevents collapse over long periods.
  • DO-ALL improves short-term adaptation quality and delivers robust long-term stability in continuously evolving environments.

4.3 Ablation Study · Components Analysis · Anchor Association Strategy

DO-ALL’s Replay, MMD alignment, and MixUp components provide complementary stabilization effects, while their combination and harm-adaptive Blending produce the strongest CTTA performance. Anchor assignment is also critical: nearest-feature anchors consistently outperform random or farthest choices, with ImageNet-to-ImageNet-C results evaluated online at corruption severity 5.

  • 4.3 Ablation Study: Replay, MMD alignment, and MixUp each independently reduce error through non-overlapping stabilization effects.Replay reduces forgetting, MMD mitigates representation drift, and MixUp regularizes local adaptation behavior.
  • Components Analysis: The full combination is necessary for DO-ALL’s benefit because Replay, MMD, MixUp, and harm-adaptive Blending address complementary stabilization needs.The components operate on different aspects of continual adaptation and jointly improve stability.
  • Components Analysis: 53.6% performance is achieved by harm-adaptive Blending, the best result among the evaluated component configurations.The result supports preventing harmful parameter drift accumulated over long adaptation trajectories.
  • Anchor Association Strategy: Nearest-anchor assignment in feature space consistently yields the best accuracy across all evaluated baselines.Anchor-use effectiveness depends critically on how distilled anchors are assigned to target samples.
  • Anchor Association Strategy: Random anchor selection provides negligible benefit, whereas farthest-anchor selection degrades performance, including RMT: 59.8% →61.9%.These comparisons show that semantically aligned anchors are important for stabilization.
  • Anchor Association Strategy: Table 7 reports classification error rate (%) for ImageNet-to-ImageNet-C using various dataset-distillation methods.All results use corruption severity level 5 and are evaluated online.

4.4 Efficiency

DO-ALL is designed as a lightweight CTTA stabilization component: anchors are distilled offline once, while stride-k scheduling amortizes anchor computation during deployment. It adds modest test-time overhead and small host-side storage while preserving robustness gains.

  • Offline deployment: DO-ALL generates distilled anchors once before deployment and carries them with the model, adding no distillation cost during testing.The framework plugs into existing CTTA algorithms without changing the original pipeline.
  • Computation scheduling: Stride-k computes the anchor branch only every k updates, optimizing solely the base CTTA objective on intervening steps.At anchor steps, DO-ALL additionally applies Lanchor and harm-adaptive blending.
  • Computation scheduling: DO-ALL consistently improves robustness over ROID even with large strides, including stride-7.Amortizing anchor computation does not alter the core CTTA optimization.
  • Deployment overhead: DO-ALL incurs only modest test-time overhead and requires small host-side anchor storage while delivering consistent robustness gains.These efficiency properties support its use as a lightweight CTTA stabilization component.

4.5 In-Depth Analysis on Source-Distilled Anchors · Experiments on Diverse DD Methods

DO-ALL remains effective across multiple dataset-distillation methods and can leverage distilled anchors regardless of the specific DD objective or formulation. Even a lightweight anchor set improves CTTA, while increasing the number of distilled samples generally provides further gains.

  • 4.5 In-Depth Analysis on Source-Distilled Anchors: Across all three CTTA baselines, SRe2L, DELT, and WMDD consistently improve performance over their respective baselines.This evaluates whether DO-ALL depends on a specific dataset-distillation algorithm.
  • 4.5 In-Depth Analysis on Source-Distilled Anchors: For ROID+DO-ALL, performance changes from 54.5% to 53.7%, 53.6%, and 53.6% with SRe2L, DELT, and WMDD, respectively.The reported values correspond to the three DD variants in that order.
  • Experiments on Diverse DD Methods: Higher-quality anchor sets correspond to stronger CTTA stabilization, as validation accuracy from training solely on the set positively trends with DO-ALL improvement.The figure uses anchor-only validation accuracy as the quality measure and the corresponding DO-ALL improvement as the stabilization measure.
  • Experiments on Diverse DD Methods: These results show that DO-ALL effectively utilizes distilled sets regardless of the specific DD objective or formulation.The method therefore does not rely on a particular synthetic-sample generation algorithm.
  • Experiments on Diverse DD Methods: IPC = 1 already yields noticeable improvements over the baseline across all CTTA baselines.Thus, DO-ALL does not require a large number of anchors to be effective.
  • Experiments on Diverse DD Methods: Increasing IPC provides generally consistent gains, indicating that richer distilled information can further improve performance.IPC denotes the number of distilled samples per class.

Analysis on IPC · Impact of DD Performance on CTTA Gain

DO-ALL’s CTTA gains increase with the standalone quality of distilled anchors, with dataset distillation outperforming coreset selection across CTTA methods. The framework also transfers across distillation and adaptation backbones, connecting advances in dataset distillation to robust continual adaptation.

  • Analysis on IPC: Standalone validation accuracy measures how well a small distilled or coreset set represents the source distribution.It is computed by training a classifier only on the small set and evaluating it on the original source validation data.
  • Impact of DD Performance on CTTA Gain: Higher standalone validation accuracy correlates positively with larger DO-ALL improvements across CTTA baselines.The relationship is plotted between each anchor set’s validation performance and resulting TTA improvement.
  • Impact of DD Performance on CTTA Gain: Dataset-distilled anchors consistently produce larger TTA gains than coreset-selected anchors across all evaluated CTTA methods.The comparison uses filled markers for distilled anchors and hollow markers for coreset-selected anchors.
  • Impact of DD Performance on CTTA Gain: DO-ALL’s gains reflect structured and compact source semantics rather than merely retaining a subset of source information.Distilled anchors act as high-density semantic cues for adaptation.
  • Impact of DD Performance on CTTA Gain: Experiments examine architecture variation in distilled anchors and ViT-based CTTA, including ResNet-18 distillation and ROID with IPC set to 1 for LGM.Table 9 covers ImageNet-to-ImageNet-C over 5 runs, while Table 10 reports ImageNet-C experiments with ViT.
  • Impact of DD Performance on CTTA Gain: Advances in dataset distillation can transfer directly into more stable and robust CTTA through DO-ALL’s distilled anchors.The framework leverages increasingly rich and compact source representations during continual adaptation.
  • Impact of DD Performance on CTTA Gain: Compatibility across diverse CTTA methods positions DO-ALL as a plug-and-play bridge between progress in dataset distillation and continual test-time adaptation.The stated role follows from the observed correlation and earlier compatibility results.
  • Impact of DD Performance on CTTA Gain: ResNet-18-distilled anchors applied to ResNet-50 CTTA yield virtually unchanged performance versus same-backbone anchors across ROID, RMT, and EATA.This indicates strong cross-backbone transfer of distilled anchors.

Cross-Architecture Experiments · 5 Conclusion

DO-ALL uses compact source-distilled anchors and lightweight integration to retain source-domain knowledge and stabilize continual test-time adaptation. The conclusion identifies anchor selection, update schedules, and robustness to challenging target shifts as directions for future work.

  • 5 Conclusion: DO-ALL uses dataset distillation to retain source-domain knowledge in a compact form for continual test-time adaptation.The framework is motivated as a compact mechanism for retaining source-domain knowledge.
  • 5 Conclusion: A compact set of source-distilled anchors provides DO-ALL’s stabilization mechanism for CTTA.The anchors are integrated as a lightweight stabilization layer.
  • 5 Conclusion: DO-ALL is a plug-and-play stabilization layer that works across different adaptation algorithms.It does so without modifying the underlying model.
  • 5 Conclusion: DO-ALL also works across different distillation methods without modifying the underlying model.The conclusion describes integration across adaptation and distillation choices.
  • 5 Conclusion: The experiments demonstrate that DO-ALL meaningfully improves long-term CTTA performance.The supplied passage truncates the specific endpoint of this improvement claim.
  • 5 Conclusion: Future work should improve anchor selection and update schedules under non-stationary streams.These are identified as promising directions for continued development.
  • 5 Conclusion: Future studies should examine robustness when target shifts are abrupt or adversarially corrupted.The conclusion explicitly highlights both conditions.
  • 5 Conclusion: Distilled data may serve as a general-purpose compact memory for stabilizing online learning beyond CTTA.This broader implication is presented as a suggestion from the findings.
Loading 2606.20196v2…