PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Prompt Tuning for Test-time Domain Adaptation

Yunhe Gao, Xingjian Shi, Yi Zhu, Hao Wang, Zhiqiang Tang, Xiong Zhou, Mu Li, Dimitris N. Metaxas

arXiv:2210.04831v2cs.CV

TL;DR

Distribution shifts can reduce model performance, and test-time adaptation must work without source data during deployment. DePT tunes source-initialized visual prompts and combines memory-bank pseudo-labeling with hierarchical self-supervision; it reports state-of-the-art results on major benchmarks with few tunable parameters.

  • Problem

    Distribution shifts can cause significant performance drops, while test-time adaptation must adapt to unlabeled target data without access to source data.

    Source: §1 · p. 1

  • Method

    DePT tunes source-initialized visual prompts and combines memory-bank-refined pseudo-labeling with hierarchical self-supervised regularization for prompts.

    Source: §1 · p. 2

  • Results

    Across VisDA-C, ImageNet-C, and DomainNet-126, DePT reports state-of-the-art performance; with 0.19% tunable parameters, it achieves 1.2% higher performance than AdaContrast.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DePT reports data efficiency on VisDA-C: with 1% of unlabeled target data, it achieves 88.0% average accuracy, surpassing AdaContrast by 7.1%.

    Source: §1 · p. 2

    The method addresses the close-set TTA setting, with source data unavailable during adaptation.

    Source: §3 · p. 3

Abstract

from arXiv · show

Models should be able to adapt to unseen data during test-time to avoid performance drops caused by inevitable distribution shifts in real-world deployment scenarios. In this work, we tackle the practical yet challenging test-time adaptation (TTA) problem, where a model adapts to the target domain without accessing the source data. We propose a simple recipe called \textit{Data-efficient Prompt Tuning} (DePT) with two key ingredients. First, DePT plugs visual prompts into the vision Transformer and only tunes these source-initialized prompts during adaptation. We find such parameter-efficient finetuning can efficiently adapt the model representation to the target domain without overfitting to the noise in the learning objective. Second, DePT bootstraps the source representation to the target domain by memory bank-based online pseudo-labeling. A hierarchical self-supervised regularization specially designed for prompts is jointly optimized to alleviate error accumulation during self-training. With much fewer tunable parameters, DePT demonstrates not only state-of-the-art performance on major adaptation benchmarks VisDA-C, ImageNet-C, and DomainNet-126, but also superior data efficiency, i.e., adaptation with only 1\% or 10\% data without much performance degradation compared to 100\% data. In addition, DePT is also versatile to be extended to online or multi-source TTA settings.

1 INTRODUCTION

DePT addresses test-time adaptation without source data by tuning visual prompts and combining memory-bank pseudo-labeling with prompt-focused self-supervision. It reports state-of-the-art results across major benchmarks and stronger gains in low-data settings.

1 INTRODUCTION

  • Distribution shifts can cause significant test-performance drops, while covering infinitely many domains or collecting labeled data for each domain is costly or impractical.The introduction cites common corruptions and appearance variations as examples of shifts.Source: §1 · p. 1
  • TTA adapts a source-initialized model using unlabeled target data during testing without source-data access, in offline or online settings.Offline adaptation precedes inference; online adaptation updates while test data arrive in a stream.Source: §1 · p. 1
  • TTA must modulate models under noisy objectives: broad tuning risks overfitting, narrow tuning may lack capacity, and unsupervised objectives can be misaligned or noisy.The introduction frames model modulation and target-only learning objectives as two central challenges.Source: §1 · pp. 1–2
  • DePT source-pretrains visual prompts, then tunes prompts and the classification head while freezing the backbone; memory-bank-refined pseudo-labels and hierarchical prompt regularization are jointly optimized.The regularizer is designed to alleviate error accumulation during self-training.Source: §1 · p. 2
  • With only 0.19% tunable parameters, DePT achieves 1.2% higher performance than AdaContrast, and its advantage is larger at low VisDA data ratios.Figure 1 reports DePT-G outperforming AdaContrast on 100% target data and shows a more significant advantage in low-data settings.Source: Figure 1 · p. 2, §1 · p. 2

2 RELATED WORK

Related work contrasts source-access-dependent UDA with source-free TTA, reviews unsupervised objectives and model-modulation strategies, and situates DePT among self-supervised learning and prompt-tuning methods.

2 RELATED WORK

  • UDA methods use source and target data during training, an assumption that can fail after deployment, and often require framework retraining for new domains.Prior approaches include style transfer, feature alignment, and adversarial learning [18, 33, 40].Source: §2 · p. 2
  • TTA needs no source data during adaptation; prior methods vary in objectives and model modulation, whereas DePT freezes the backbone and combines memory-bank pseudo-labeling with prompt-specific self-supervision.The section distinguishes offline adaptation before inference from online batch-by-batch adaptation during inference.Source: §2 · p. 3
  • Self-supervised vision work spans pretext tasks, contrastive learning, and non-discriminative methods; DePT draws inspiration from BYOL and DINO for its hierarchical regularization.The passage also notes applications of self-supervised objectives to robustness and domain adaptation.Source: §2 · p. 3
  • Vision prompt methods include continual learning and transfer learning; unlike VPT’s randomly initialized prompts or DAPL’s reliance on vision-language models, DePT updates source-learned visual prompts without a text encoder.The passage describes soft prompts as continuous vectors optimized by gradient descent, with fewer tunable parameters than full fine-tuning.Source: §2 · p. 3

3 METHOD

DePT addresses close-set test-time adaptation, where source data is unavailable and source and target share a label space. It adapts a source-trained model by tuning visual prompts and the classification head, using memory-bank pseudo-labeling and hierarchical self-supervision to improve target representations.

3.1 VISUAL PROMPT TUNING

  • DePT inserts learnable prompts at selected Transformer stages and, during adaptation, tunes only the prompts and classification head while freezing the backbone.The number of stages and prompts controls the number of tunable parameters.Source: §3.1 · pp. 4–5

3.2 LEARNING OBJECTIVE

  • The learning objective combines memory-bank-refined online pseudo-labeling with hierarchical self-supervision on CLS and aggregated prompt representations.The two components bootstrap source-learned representations to the target domain and aim to reduce pseudo-label noise.Source: §3.2 · p. 5, §3.2.2 · p. 5

3.2.1 PSEUDO LABELING

  • An EMA teacher maintains a FIFO memory bank of target embeddings and predictions, whose top-k neighbors provide soft-voted pseudo-labels for strong-view student training.The teacher is updated by exponential moving average from the student rather than by gradients.Source: §3.2.1 · p. 5

3.2.2 HIERARCHICAL SELF-SUPERVISED REGULARIZATION

  • Hierarchical regularization matches student and teacher CLS outputs and aggregated prompts across views, while prompt diversity discourages all prompts from learning the same representation.The CLS token represents the image holistically; aggregated prompts provide fine-grained representations that can capture features at different stages.Source: §3.2.2 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate DePT on VisDA-C, ImageNet-C, and DomainNet-126, using ViT-B backbones and comparing against UDA and TTA methods. Across these settings, DePT combines strong benchmark performance with data efficiency and extensions to online and multi-source adaptation.

4.2 VISDA-C DATASET RESULTS

  • On VisDA-C, DePT-G and DePT-D outperform AdaContrast by +1.4% and +1.5% with 0.16M and 0.47M tunable parameters, respectively.DePT-S, with 0.048M tunable parameters, improves by +16.5% over no adaptation.Source: §4.2 · p. 7, Table 1 · p. 7
  • With 1% VisDA-C target data, DePT-G reaches 87.97% average accuracy, 2.59% below its 100% result; AdaContrast reaches 80.87%, 8.48% below its 100% result.DePT-S has the smallest decline, 1.17%; DePT-D has the greatest adaptation capacity with abundant data but declines more than DePT-G and DePT-S at 1%.Source: §4.2 · p. 7, Figure 3 · p. 8
  • In online VisDA-C adaptation, DePT achieves 85.9% average accuracy, surpassing AdaContrast by +4.6%.Each sequential target-data batch is used once, and the method disables the memory bank and uses model predictions as pseudo-labels.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 IMAGENET-C DATASET RESULTS

  • On ImageNet-C level-5 corruptions, DePT-G has the lowest error on 14 of 15 corruption types and beats online AdaContrast and CFA by 3.7% and 1.2%, respectively.The comparison reports top-1 error rates across 15 highest-severity corruption types.Source: §4.3 · p. 7, Table 2 · p. 8

4.4 DOMAINNET DATASET RESULTS

  • On DomainNet-126, all three DePT variants improve over no adaptation across seven domain shifts, and DePT-G achieves 81.7% average accuracy.Online DePT is 3.1% higher than AdaContrast; ViT-B models show stronger robustness to domain shifts than R50 models.Source: §4.4 · p. 8, Table 3 · p. 9
  • For multi-source DomainNet-126 adaptation, DePT uses domain-specific prompts with a shared domain-agnostic prompt, unlike SHOT’s separate per-domain adaptation and ensemble.The comparison is reported as multi-source classification accuracy; R denotes the remaining three domains except the single source.Source: §4.4 · p. 8, Table 4 · p. 9

4.5 ANALYSIS

  • On VisDA-C, embeddings become more class-distinguishable after adaptation despite a frozen backbone; increasing prompt stages or counts generally helps with abundant data, while one prompt remains effective.The t-SNE figure uses different colors for classes.Source: §4.5 · p. 8, Figure 3 · p. 8, Table 5 · p. 9
  • On VisDA-C, pseudo-labeling and memory-bank refinement improve accuracy by 12.4% with 100% data and 7.2% with 1%; adding prompt diversity lifts final accuracy to 90.6% and 88.0%, respectively.Prompt regularization alone collapses prompts, while tuning all parameters is prone to overfitting with limited data.Source: §4.5 · p. 9, Table 6 · p. 9

5 CONCLUSION

DePT adapts target representations by tuning visual prompts with a fixed backbone, while its prompt count and placement control adaptation capacity. The motivation experiments show the trade-off between overfitting with many tunable parameters and limited adaptation with few.

5 CONCLUSION

  • DePT uses source-trained visual prompts and adapts target-specific knowledge with prompts while keeping the backbone fixed.The method also places prompts at Transformer stages so their number and stage placement can control adaptation capacity and tunable parameters.Source: §5 · p. 9
  • The authors report state-of-the-art results across VisDA-C, ImageNet-C, and DomainNet-126, and verify extensions to online and multi-source adaptation.The conclusion also describes jointly learning online memory-bank pseudo-labeling and a hierarchical self-supervised objective for prompt tuning.Source: §5 · p. 9

A MOTIVATION OF USING PROMPT TUNING

  • Too many tunable parameters can overfit noisy unsupervised objectives, especially with limited target data, whereas too few restrict adaptation capacity.The motivation experiment compares AdaContrast full-model tuning and LayerNorm tuning using AdaContrast’s MoCo-based self-supervision and pseudo-labeling objective.Source: A MOTIVATION OF USING PROMPT TUNING · p. 13
  • At 1% VisDA-C data, AdaContrast overfits as training continues, while LayerNorm tuning avoids overfitting but remains capacity-limited.At 100% data, AdaContrast adapts well; the authors note there is no good target-label metric for deciding when to stop adaptation.Source: A MOTIVATION OF USING PROMPT TUNING · p. 13, Figure 4 · p. 13

B IMPLEMENTATION DETAILS

The pseudo-labeling procedure stabilizes student predictions with an EMA teacher and a memory bank, then trains on refined labels. A separate DINO-based objective regularizes the CLS token using sharpening and centering.

B.1 PSEUDO LABELING DETAILS

  • An EMA teacher tracks the rapidly updated student and acts as an ensemble, producing more stable, higher-accuracy pseudo-labels.Both models use the source-initialized prompt-augmented ViT architecture.Source: §B.1 · pp. 13–14
  • The teacher’s online memory bank stores feature embeddings and probabilities, starts from randomly selected samples, and updates as a first-in-first-out queue.The student retrieves top-k cosine-similar neighbors and averages their probabilities before taking an argmax to form a refined hard pseudo-label.Source: §B.1 · p. 14
  • The student is trained with cross-entropy against its output for a strongly augmented view, using the refined pseudo-label.The target image receives weak and strong random transformations.Source: §B.1 · p. 14

B.2 SELF-SUPERVISED OBJECTIVE DETAILS

  • The self-supervised objective reuses the student and teacher in a DINO framework and applies its loss to the CLS token, with sharpening and centering to avoid collapse.The teacher also receives a centering operation; τ and τ′ denote student and teacher temperatures, and c is an exponentially moving-averaged center.Source: §B.2 · p. 14
  • DePT omits DINO’s multi-crop and local-to-global regularization.The stated objective instead uses the described CLS-token loss and sharpening and centering operations.Source: §B.2 · p. 14

B.3 EXTENDING TO MULTI-SOURCE DOMAIN TTA SETTING

For multi-source test-time adaptation, DePT shares a domain-agnostic backbone while learning domain-specific prompts and shared general prompts. At test time, it averages source-domain prompts and initializes adaptation with them alongside the shared prompts.

B.3 EXTENDING TO MULTI-SOURCE DOMAIN TTA SETTING

  • The multi-source setting targets an unseen domain after training across multiple source domains, where diverse sources can make domain-invariant features difficult to learn and domain-specific knowledge hard to capture.Source: §B.3 · p. 14
  • DePT uses a shared domain-agnostic backbone, domain-specific prompts, and shared general prompts; source training optimizes the backbone and relevant prompts with supervised loss.Training iterations sample images from each source domain.Source: §B.3 · pp. 14–15
  • At test time, DePT averages prompts from all source domains, concatenates them with shared prompts for initialization, and then uses the ordinary single-source adaptation procedure.The experiment uses three DomainNet-126 domains as sources and the remaining domain as target.Source: §B.3 · p. 15

B.4 IMPLEMENTATION DETAILS

The implementation uses an ImageNet-pretrained ViT-B-16 and a 90/10 random source-data train/validation split. Training uses SGD with momentum and cosine learning-rate annealing.

B.4 IMPLEMENTATION DETAILS

  • Source training initializes ViT-B-16 from ImageNet-pretrained timm weights, with 90% of source data for training and 10% for validation.The cited model was pretrained on ImageNet21K with AugReg and then fine-tuned on ImageNet1K.Source: §B.4 · p. 15
  • Training runs for around 11k source iterations and around 6k target iterations for all datasets unless noted.Source: §B.4 · p. 15
  • All experiments use SGD with momentum 0.9, weight decay 1e-4, cosine annealing, and batch size 128.Source: §B.4 · p. 15

C.1 ABLATION ON THE HYPER-PARAMETER CHOICE

DePT remains state of the art despite modest performance variation across hyperparameter choices. The ablations identify useful settings for prompt learning rate, teacher temperature, and loss weights.

C.1 ABLATION ON THE HYPER-PARAMETER CHOICE

  • Learning rates of 1e-3 and 5e-3 yield decent performance, while further increases may destabilize training.The authors find that larger learning rates benefit visual prompt tuning.Source: §C.1 · p. 15
  • A warmup of the DINO teacher temperature balances the difficulty caused by large temperatures against the slightly worse performance of small temperatures.Large temperatures make the teacher output more uniform and initial optimization difficult.Source: §C.1 · p. 15
  • With α set to 1, β1 = β2 = β, and λ = β/20, smaller β values, such as below 0.1, perform better.Table 7 summarizes the hyperparameter ablation for DePT-G on VisDA-C.Source: §C.1 · p. 15, Table 7 · p. 16
  • Despite slight variation across hyperparameter choices, DePT consistently demonstrates state-of-the-art performance compared with other methods.The authors describe this as robustness to hyperparameter choice.Source: §C.1 · p. 16

C.2 COMPUTATION EFFICIENCY ANALYSIS

The computation analysis compares the time and memory costs of model-modulation approaches under a shared VisDA-C setup. DePT uses fewer resources than AdaContrast and SHOT with 50 prompts, but slightly more with 100 prompts.

C.2 COMPUTATION EFFICIENCY ANALYSIS

  • With 50 prompts, DePT consumes less time and memory than AdaContrast and SHOT, while 100 prompts consume slightly more.The comparison uses ViT-B; Table 8 also reports tunable parameter counts.Source: §C.2 · p. 16, Table 8 · p. 16
  • The comparison measures average training time and memory over 500 iterations on one NVIDIA A10G GPU, using batch size 16 and 224 × 224 images.All approaches use entropy minimization on VisDA-C.Source: §C.2 · p. 16
  • Because prompts are inserted at multiple ViT levels, test-time adaptation requires gradients to propagate back to the input.Source: §C.2 · p. 16

C.3 DETAILED EXPERIMENT RESULTS

The appendix provides detailed experiment results for VisDA-C and DomainNet-126. The supplied table captions specify result scope and backbone provenance, but do not provide cell values.

C.3 DETAILED EXPERIMENT RESULTS

  • The detailed experiments cover VisDA-C and DomainNet-126 results for methods with a ResNet backbone.Source: §C.3 · p. 16
  • Table 9 details classification accuracy for VisDA-C train-to-validation adaptation, with separate UDA, TTA, and online-setting results.ViT-B results are reproduced by the authors; R101 results are cited from the original paper.Source: Table 9 · p. 17
  • Table 10 reports classification accuracy across seven DomainNet-126 domain shifts.ViT-B results are reproduced by the authors, while R50 baseline results are cited from the original paper.Source: Table 10 · p. 17

Related papers

Loading 2210.04831v2…