Source-linked AI summary
Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation
Yulu Gan, Yan Bai, Yihang Lou, Xianzheng Ma, Renrui Zhang, Nian Shi, Lin Luo
TL;DR
Continual test-time adaptation must handle unlabeled target domains that change over time without source data, while existing model-based approaches face error accumulation and forgetting. The paper learns image-level visual domain prompts with the source model frozen, using domain-specific and domain-agnostic prompts plus homeostasis-based adaptation. Across four benchmark datasets, the method reports significant gains over state-of-the-art methods and is presented as a lightweight alternative to model adaptation.
Problem
Continual test-time adaptation must handle changing target domains without source data, but existing approaches can neglect continuous shifts and encounter error accumulation or catastrophic forgetting.
Method
The method freezes the source model and adapts inputs with visual domain prompts, combining domain-specific and domain-agnostic prompts with homeostasis-based prompt adaptation.
Results
The proposed approach outperforms most state-of-the-art methods across extensive benchmark datasets covering synthetic and real-world domain gaps.
Takeaways & Limitations
Input-level visual prompts provide a lightweight CTTA approach that mitigates error accumulation and limits catastrophic forgetting without modifying the source model.
Takeaways & Limitations
The evaluation assumes a source model tested on multiple target datasets, and prior TTA methods discussed in the paper require stationary target distributions.
Abstract
from arXiv · showhide
Continual Test-Time Adaptation (CTTA) aims to adapt the source model to continually changing unlabeled target domains without access to the source data. Existing methods mainly focus on model-based adaptation in a self-training manner, such as predicting pseudo labels for new domain datasets. Since pseudo labels are noisy and unreliable, these methods suffer from catastrophic forgetting and error accumulation when dealing with dynamic data distributions. Motivated by the prompt learning in NLP, in this paper, we propose to learn an image-level visual domain prompt for target domains while having the source model parameters frozen. During testing, the changing target datasets can be adapted to the source model by reformulating the input data with the learned visual prompts. Specifically, we devise two types of prompts, i.e., domains-specific prompts and domains-agnostic prompts, to extract current domain knowledge and maintain the domain-shared knowledge in the continual adaptation. Furthermore, we design a homeostasis-based prompt adaptation strategy to suppress domain-sensitive parameters in domain-invariant prompts to learn domain-shared knowledge more effectively. This transition from the model-dependent paradigm to the model-free one enables us to bypass the catastrophic forgetting and error accumulation problems. Experiments show that our proposed method achieves significant performance gains over state-of-the-art methods on four widely-used benchmarks, including CIFAR-10C, CIFAR-100C, ImageNet-C, and VLCS datasets.
Introduction
Continual test-time adaptation addresses performance degradation when target domains change over time, but existing approaches often adapt the model itself and can face instability. The paper instead adapts inputs with lightweight visual prompts while keeping the source model frozen, aiming to mitigate error accumulation and catastrophic forgetting.
- Motivation: Changing target distributions can create substantial domain gaps, motivating adaptation across heterogeneous and dynamic deployment conditions.The paper illustrates this with weather and day-night changes in street signal recognition.
- Limitations of Existing Methods: Existing test-time adaptation methods modify network parameters or outputs, while many assume stationary target distributions and therefore neglect continuously changing domains.The cited limitation concerns adaptation methods designed for a single or stationary shift.
- Limitations of Existing Methods: Continual adaptation methods such as CoTTA use pseudo-label refinement and parameter restoration to address error accumulation and catastrophic forgetting, but remain model-based.CoTTA is described as addressing sequences of domain shifts through source-network-derived mechanisms.
- Proposed Approach: The proposed framework learns visual domain prompts for each domain, reformulates inputs during testing, and keeps the source model frozen.The approach shifts adaptation from changing model parameters to changing the input data.
- Proposed Approach: The method introduces domain-specific and domain-agnostic prompt components to extract current-domain knowledge while preserving domain-shared knowledge.A homeostasis-based strategy limits over-adaptation of domain-sensitive prompt parameters.
- Results: Experiments on extensive synthetic and real-world benchmark domain gaps report performance gains over most state-of-the-art methods at relatively low cost.The introduction names four benchmark settings in the broader paper context and emphasizes both performance and cost.
Related Work
Related work spans test-time adaptation, prompt learning, and continual learning, while this paper positions continual domain-shift adaptation as a distinct problem addressed at the input level.
- Test-time adaptation: Test-time adaptation uses only incoming test data without source data or supervision, through network-based or parameter-free approaches.Network-based methods update model components, whereas parameter-free methods adjust output distributions.
- Test-time adaptation: CoTTA addresses continual distribution shifts with a model-based approach designed to reduce error accumulation and catastrophic forgetting.It refines pseudo labels and periodically restores some neurons to source parameters.
- Prompt learning: Prompt learning originated in NLP and was later extended to visual and multimodal settings, but earlier approaches were not designed for domain shift.The paper distinguishes its text-modality-free approach from prior prompt-learning methods.
- Continual learning: Continual learning methods include replay, regularization, and parameter isolation for retaining knowledge across changing tasks or domains.The paper focuses specifically on catastrophic forgetting across different domains with the same task.
Method
The method adapts frozen source models by modifying incoming images with two visual prompts. A homeostatic update strategy separates domain-specific learning from domain-agnostic knowledge consolidation.
- Visual domain prompts: The method freezes the source model and continuously reformulates incoming target images to adapt them online.Only prompt parameters are updated; learned prompts from the previous batch are applied to the next batch.
- Visual domain prompts: Domain-specific prompts extract current-domain knowledge, while domain-agnostic prompts maintain knowledge shared across domains.Both are learnable parameter matrices added pointwise to input images before source-model prediction.
- Prompt updating: A teacher-student framework updates only the prompts, using cross-entropy for domain-specific prompts and regularization for domain-agnostic prompts.The regularization constrains domain-sensitive parameters while preserving domain-insensitive parameters.
- Homeostatic-based adaptation: Homeostatic regularization penalizes parameters sensitive to domain shifts and stably updates domain-insensitive parameters to consolidate shared knowledge.Parameter importance and changes between domains determine the homeostatic factor, while α controls regularization strength.
- Domain-shift detection: Domain changes are detected from batch-level prediction-confidence differences, triggering updates to the parameters and homeostatic factors.The method uses threshold S = 0.25 for the confidence change.
Experiments
The experiments evaluate continual test-time adaptation across corruption-based benchmarks and VLCS, including multiple target domains to assess anti-forgetting under real-world domain gaps.
- Datasets and tasks: The evaluation covers CIFAR10-to-CIFAR10C, CIFAR100-to-CIFAR100C, ImageNet-to-ImageNet-C, and VLCS benchmark tasks.CIFAR10-to-CIFAR10C includes both standard and gradual settings.
- Datasets and tasks: VLCS uses one domain as the source and the remaining domains as targets across multiple rounds.This setting evaluates anti-forgetting when the model encounters real-world domain changes.
- Implementation details: The corruption benchmarks use 32 × 32 images and batch size 100 for the CIFAR-based tasks.The implementation follows the CoTTA task setup and uses PyTorch.
C (Hendrycks et al. 2019) experiments
Across synthetic and real-domain experiments, the visual domain prompt approach improves continual adaptation while limiting forgetting. Results also show that prompt placement and the complementary DAP/DSP design affect performance, with random placement generally effective.
- CIFAR10-to-CIFAR10C: 43.5% source-model error fell to 16.2% with recent methods, and the proposed method reduced it by a further 2.3% on standard CIFAR10-to-CIFAR10C.The benchmark contains fifteen corruption types arriving sequentially during testing.
- CIFAR10-to-CIFAR10C: 6.0% error was achieved on gradually changing CIFAR10-to-CIFAR10C, a 4% improvement over CoTTA.The task varies severity levels and corruption types continuously.
- ImageNet-to-ImageNet-C: 11.5% higher performance than the state-of-the-art method was achieved on standard ImageNet-to-ImageNet-C experiments.The comparison uses average error over ten diverse corruption sequences at severity level 5.
- VLCS: On VLCS, mean error was 13% lower than CoTTA when VOC2007 was the source domain, while LabelMe error decreased from 67.4% to 66.1% across rounds.The experiments test three target domains sequentially for multiple rounds.
- Prompt analysis: Random prompt placement performed best, and prompt-size changes produced only 1∼2% performance variance.The reported size setting with best performance was 30, while the relative-position experiment used prompts of size 20 × 20.
- Prompt analysis: DAP and DSP each improve performance, with DSP’s improvement more obvious; their domain-agnostic and domain-specific knowledge are complementary.The analysis attributes DSP’s stronger standalone effect to its targeted domain-specific knowledge.
- Prompt analysis: The prompts are added to, rather than used as a mask over, the original image, and random placement can reduce coverage of important regions.The paper identifies learnable prompt positioning as a direction for further exploration.
Conclusion
CTAP addresses CTTA error accumulation by adapting changing target images with lightweight visual domain prompts instead of changing the model.
- CTAP introduces visual domain prompts as small image tokens dynamically added to input images.
- The prompts shift images from changing target domains toward the regular domain for adaptation.
- The framework combines a visual domain prompt updating module with a Homeostasis-based adapting strategy.
- Experiments on multiple benchmark datasets report state-of-the-art performance at relatively small cost.