PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe Self-Refinement for Transformer-based Domain Adaptation

Tao Sun, Cheng Lu, Tianshuo Zhang, Haibin Ling

arXiv:2204.07683v1cs.CVcs.LG

TL;DR

UDA seeks to transfer from labeled source data to unlabeled targets, a challenge intensified by large domain gaps. SSRT combines a vision transformer with perturbed-target self-refinement and adaptive safe training. It reports best performances across the cited benchmarks, including 45.2% on DomainNet.

  • Problem

    UDA must transfer from label-rich source data to unlabeled targets, while large domain gaps make adaptation challenging and labeled data costly to obtain.

    Source: §1 · p. 1

  • Method

    SSRT uses a vision-transformer backbone, refines it with predictions on perturbed target data, and adaptively adjusts training configuration using target-prediction diversity to detect collapse.

    Source: §1 · p. 2

  • Results

    SSRT reports best performances across the cited benchmarks: 85.43% on Office-Home, 88.76% on VisDA-2017, and 45.2% on DomainNet.

    Source: §1 · p. 2

  • Takeaways & Limitations

    On DomainNet, combining ViT-B/16 with adversarial adaptation reaches 38.5% average accuracy, above the reported current results using ResNet-101.

    Source: §1 · p. 1

    DomainNet accuracy reaches 45.2% but remains far from saturated; the authors identify multiple source domains and target-domain meta knowledge as future directions.

    Source: §5 · p. 8

Abstract

from arXiv · show

Unsupervised Domain Adaptation (UDA) aims to leverage a label-rich source domain to solve tasks on a related unlabeled target domain. It is a challenging problem especially when a large domain gap lies between the source and target domains. In this paper we propose a novel solution named SSRT (Safe Self-Refinement for Transformer-based domain adaptation), which brings improvement from two aspects. First, encouraged by the success of vision transformers in various vision tasks, we arm SSRT with a transformer backbone. We find that the combination of vision transformer with simple adversarial adaptation surpasses best reported Convolutional Neural Network (CNN)-based results on the challenging DomainNet benchmark, showing its strong transferable feature representation. Second, to reduce the risk of model collapse and improve the effectiveness of knowledge transfer between domains with large gaps, we propose a Safe Self-Refinement strategy. Specifically, SSRT utilizes predictions of perturbed target domain data to refine the model. Since the model capacity of vision transformer is large and predictions in such challenging tasks can be noisy, a safe training mechanism is designed to adaptively adjust learning configuration. Extensive evaluations are conducted on several widely tested UDA benchmarks and SSRT achieves consistently the best performances, including 85.43% on Office-Home, 88.76% on VisDA-2017 and 45.2% on DomainNet.

1. Introduction

The introduction frames large domain gaps and scarce target labels as challenges for UDA, then presents SSRT’s transformer backbone, perturbed-target refinement, and adaptive safe training. It also positions latent-token refinement against prior transformer adaptation and consistency-regularization approaches.

1. Introduction

  • UDA transfers knowledge from a labeled source domain to an unlabeled target domain, where large domain gaps make adaptation especially challenging.The introduction notes that obtaining large amounts of labeled training data can be costly or impractical.Source: §1 · p. 1
  • Vision transformers remain under-explored in UDA despite their use across vision tasks and their long-range self-attention and large-scale pre-training.The paper reports a prior DomainNet best average accuracy of 33.3% and identifies large-scale adaptation as unsatisfactory.Source: §1 · p. 1
  • SSRT uses a vision transformer and predictions on perturbed target data to refine the adapted model, with multi-layer perturbation and bi-directional supervision among its components.It adds random offsets to latent target-token sequences and minimizes the KL divergence between predicted probabilities for original and perturbed versions.Source: §1 · p. 2
  • Safe Training monitors target-prediction diversity and, when collapse occurs, restores a prior state and resets the learning configuration.The configuration is adjusted adaptively because a setting that works across most tasks may fail on a particular task, and perturbation magnitude can under- or over-shoot.Source: §1 · p. 2
  • 85.43% on Office-Home, 88.76% on VisDA-2017, and 45.2% on DomainNet are reported as SSRT’s best performances across widely tested benchmarks.The introduction also reports 38.5% average accuracy from ViT-B/16 with adversarial adaptation on DomainNet, above current ResNet-101-based results.Source: §1 · pp. 1–2

2. Related Work

  • Prior consistency-regularization methods mostly perturb raw images, whereas SSRT refines vision-transformer representations using latent token sequences and paired target examples.The related work contrasts SSRT with CDTrans’s source-target cross-attention and TVT’s transferable multi-head self-attention combined with adversarial adaptation.Source: §2 · p. 2

3. Proposed Method

SSRT adapts a source-trained classifier to unlabeled target data using a vision transformer and target-side self-refinement. Multi-layer token perturbations, bidirectional confidence-filtered supervision, and diversity-based Safe Training make up its method.

3.1. Problem Formulation

  • The adaptation objective combines source cross-entropy, adversarial domain loss, and a target loss instantiated as self-refinement.The adversarial term encourages domain-invariant features.Source: §3.1 · p. 3

3.2. Method Framework

  • For each target image, shared-parameter original and perturbed branches produce predictions for confidence-filtered refinement, with Safe Training guarding against collapse.The framework uses a vision transformer backbone and classifier head.Source: §3.2 · p. 3, Figure 1 · p. 3

3.3. Multi-layer Perturbation for Transformer

  • SSRT perturbs token sequences at a randomly selected transformer layer, sampling among multiple layers because the best layer varies across tasks.Randomly sampling layers is more robust than perturbing any single layer from the set; deeper layers perform better empirically but carry greater collapse risk.Source: §3.3 · p. 3

3.4. Bi-directional Self-Refinement

  • Bidirectional self-refinement combines KL divergence in both prediction directions, using confident target predictions to regularize predictions in latent feature space.The confidence filter retains examples whose maximum predicted probability exceeds a predefined threshold.Source: §3.4 · p. 4

3.5. Safe Training via Adaptive Adjustment

  • Safe Training monitors target prediction diversity and, when it detects a drop, restores a saved model and resets r to ramp perturbation and loss weights back up.Diversity is the number of unique predicted labels per target batch; drops are checked across multiple time scales.Source: §3.5 · pp. 4–5

4. Experiments

SSRT is evaluated across four UDA benchmarks, where its transformer-based results lead on three benchmarks and show gains over a strong ViT baseline. Experiments further examine multi-layer perturbation, bidirectional refinement, Safe Training, perturbation visualizations, and hyperparameter sensitivity.

4. Experiments

  • Evaluations span Office-31, Office-Home, VisDA-2017, and DomainNet, using ViT-small/base backbones, shared hyperparameters, and adversarial-adaptation and Mutual Information baselines.The benchmarks range from Office-31's three domains to DomainNet's six domains and 345 classes.Source: §4 · p. 6

4.1. Results on Benchmarks

  • 4.38% on Office-Home, 3.53% on VisDA-2017, and 6.7% on DomainNet over Baseline-B; SSRT-B reaches 45.2% average accuracy on DomainNet and leads on all three benchmarks.Transformer-based results are generally much better than CNN-based methods, and ViT-base outperforms ViT-small.Source: §4.1 · pp. 6–7

4.2. Effects of Multi-layer Perturbation

  • Latent-token perturbation outperforms raw-image perturbation on Office-Home and DomainNet; single-layer choices {0,4,8} reduce DomainNet average accuracy by -1.0%, -1.5%, and -1.5%, respectively.The best perturbation layer varies across tasks, so the experiments use a layer from {0,4,8}.Source: §4.2 · p. 7, Table 5 · p. 7

4.3. Effects of Bi-directional Self-Refinement

  • Bidirectional self-refinement is more robust than either one-way loss, although all losses fail on qdr-target tasks without Safe Training; blocking either tested gradient also degrades performance.The authors attribute robustness under blocked gradients to complementary losses; Figure 3 and Tables 6–7 describe the comparisons.Source: §4.3 · p. 7, Figure 3 · p. 7, Table 6 · p. 7

4.4. Effects of Safe Training

  • In clp→qdr, Safe Training resets r downward as diversity falls and continues past the collapse seen without it around 10k iterations; the unprotected run ends at 96.9% source and 0.3% target accuracy.For qdr→clp, r converges to 1.0 and training with or without Safe Training performs similarly.Source: §4.4 · p. 8

4.5. Visualization of Perturbation

  • Shallow-block perturbations visibly blend features with another image, an effect less noticeable in deep blocks because of the network's nonlinear transformations; this indicates complementary multi-layer perturbation.Figure 4 visualizes perturbations at different transformer blocks.Source: §4.5 · p. 8, Figure 4 · p. 8

4.6. Ablation Studies

  • Smaller T gives quicker responses, while larger L raises sensitivity and false-positive risk; Safe Training adapts even at α = 0.5, and α = 0 retains a gain from classifier-head dropout.Many combinations of T and L work well.Source: §4.6 · p. 8, Figure 5 · p. 8

5. Conclusion

SSRT combines a vision transformer with perturbed-target self-refinement and Safe Training. Benchmark experiments show strong performance, while DomainNet remains far from saturated.

5. Conclusion

  • SSRT uses a vision transformer backbone, perturbed target-domain data for self-refinement, and Safe Training to avoid model collapse.The paper presents these components as its UDA method.Source: §5 · p. 8
  • SSRT achieves its best reported DomainNet average accuracy of 45.2%, but the authors say the benchmark remains far from saturated.The authors propose combining multiple source domains or incorporating target-domain meta knowledge as future directions.Source: §5 · p. 8
  • The authors identify multiple source domains and target-domain meta knowledge as directions for extending the study.Source: §5 · p. 8

A. More Model and Training Details

The implementation uses pretrained ViT-base and ViT-small backbones, a bottleneck classifier head, and a domain discriminator. Training applies specified image preprocessing and SGD with momentum.

A. More Model and Training Details

  • The backbones are ViT-B/16 and ViT-S/16, initialized from Google JAX checkpoints pretrained on ImageNet-21k and fine-tuned on ImageNet-1k.Source: A. More Model and Training Details · p. 9
  • The classifier head combines a bottleneck module with a class predictor, while the domain discriminator uses the class predictor's structure with one output.The bottleneck includes Linear, BatchNorm1d, ReLU, and Dropout(0.5).Source: A. More Model and Training Details · p. 9
  • Training uses SGD with momentum 0.9 and random-flip/crop preprocessing; VisDA-2017 instead uses center-cropping, and the backbone learning rate is one-tenth of the main rate.Images are resized to 256 × 256 and cropped to 224 × 224; testing uses center-crops.Source: A. More Model and Training Details · p. 9

B. More Analysis on Bi-directional Self-Refinement

Additional gradient-blocking results extend the paper's analysis of bidirectional self-refinement. They report that allowing gradients for the teacher probabilities and b_l^x works better than other tested variants.

B. More Analysis on Bi-directional Self-Refinement

C. More Analysis on Safe Training

The Confidence Filter removes noisy supervisions, and Safe Training can prevent significant performance drops.

C. More Analysis on Safe Training

  • Without the Confidence Filter (ε = 0), performance may deteriorate; Safe Training can avoid significant performance drops.The filter removes noisy supervisions.Source: C. More Analysis on Safe Training · p. 9

D. More Analysis on Multi-layer Perturbation

Multi-layer perturbation performs best across adaptation tasks, while the best layer for perturbation varies by task. Blocking gradients during perturbation of relatively deep layers can lead to model collapse.

D. More Analysis on Multi-layer Perturbation

  • The best perturbation layer varies across tasks, and a layer that works for one task may fail on others.These results use the same perturbation amount at each layer without Safe Training.Source: D. More Analysis on Multi-layer… · p. 9
  • Perturbing relatively deep layers while blocking gradients of blx causes model collapse.The analysis also considers the importance of allowing gradient back-propagation for blx.Source: D. More Analysis on Multi-layer… · p. 9
  • Multi-layer perturbation obtains the best results, while perturbing any single layer can degrade performance on some adaptation tasks.Perturbing raw input performs similarly to perturbing the 0-th transformer block.Source: D. More Analysis on Multi-layer… · p. 9

E. Analysis on Model’s Robustness

SSRT is more robust than Baseline to perturbations during testing on the evaluated tasks. Its accuracy decreases less even when the perturbation magnitude exceeds what was seen during training.

E. Analysis on Model’s Robustness

  • On Pr → Ar and clp → pnt, SSRT is more robust than Baseline; at α = 0.4, it incurs less accuracy decrease despite exceeding training perturbations.Figure A.3 averages results over six random runs and shows accuracy decreases under perturbations at different layers.Source: E. Analysis on Model’s Robustness · p. 9, Figure A.3 · p. 10

F. Comparison with SSL methods

The paper compares Mixup and VAT as semi-supervised learning techniques for UDA. Their performance varies across benchmarks: both trail SSRT on Office-Home and VisDA-2017, while Mixup is inferior to Baseline-B on DomainNet.

F. Comparison with SSL methods

  • Mixup interpolates two target samples and their predictions, then trains the model to match the interpolated prediction.The Mixup loss uses the squared difference between the model prediction for the interpolated sample and the interpolated predictions.Source: F. Comparison with SSL methods · p. 11
  • VAT encourages consistent predictions within a norm-ball neighborhood of each target sample, and both VAT and Mixup losses are used as the target-domain objective.The comparison applies both techniques to raw input images.Source: F. Comparison with SSL methods · p. 11
  • On Office-Home and VisDA-2017, Mixup and VAT outperform Baseline-B but slightly trail SSRT; on DomainNet, VAT works while Mixup remains inferior to Baseline-B.The comparison covers three benchmarks using a ViT-base backbone.Source: F. Comparison with SSL methods · p. 11

G. Results with ViT-small Backbone

SSRT-S outperforms MDD+SCDA on DomainNet despite using fewer parameters than ResNet-101. ViT-small has fewer parameters than ResNet-101 but converges more slowly than ViT-base.

G. Results with ViT-small Backbone

Related papers

Loading 2204.07683v1…