Source-linked AI summary
TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation
Jinyu Yang, Jingjing Liu, Ning Xu, Junzhou Huang
TL;DR
UDA seeks to transfer knowledge from labeled source domains to unlabeled target domains, but ViT transferability across domains was previously unexplored. This paper investigates ViT, proposes TVT with transferability-aware attention and discriminative clustering, and reports improved performance across four UDA benchmarks.
Problem
ViT's transferability across domains and effective strategies for adapting it in UDA remained insufficiently investigated.
Method
TVT uses a Transferability Adaption Module to inject patch-level transferabilities into transformer attention and applies discriminative clustering to retain feature separation.
Results
TVT outperforms prior UDA methods across four benchmarks, including 83.56% versus 71.8% for the best prior method on Office-Home.
Takeaways & Limitations
ViT is more transferable than CNN counterparts in UDA, and exploiting transformer-specific features improves domain knowledge transfer.
Abstract
from arXiv · showhide
Unsupervised domain adaptation (UDA) aims to transfer the knowledge learnt from a labeled source domain to an unlabeled target domain. Previous work is mainly built upon convolutional neural networks (CNNs) to learn domain-invariant representations. With the recent exponential increase in applying Vision Transformer (ViT) to vision tasks, the capability of ViT in adapting cross-domain knowledge, however, remains unexplored in the literature. To fill this gap, this paper first comprehensively investigates the transferability of ViT on a variety of domain adaptation tasks. Surprisingly, ViT demonstrates superior transferability over its CNNs-based counterparts with a large margin, while the performance can be further improved by incorporating adversarial adaptation. Notwithstanding, directly using CNNs-based adaptation strategies fails to take the advantage of ViT's intrinsic merits (e.g., attention mechanism and sequential image representation) which play an important role in knowledge transfer. To remedy this, we propose an unified framework, namely Transferable Vision Transformer (TVT), to fully exploit the transferability of ViT for domain adaptation. Specifically, we delicately devise a novel and effective unit, which we term Transferability Adaption Module (TAM). By injecting learned transferabilities into attention blocks, TAM compels ViT focus on both transferable and discriminative features. Besides, we leverage discriminative clustering to enhance feature diversity and separation which are undermined during adversarial domain alignment. To verify its versatility, we perform extensive studies of TVT on four benchmarks and the experimental results demonstrate that TVT attains significant improvements compared to existing state-of-the-art UDA methods.
Introduction
The paper investigates ViT transferability for UDA and proposes TVT to exploit fine-grained transferable, discriminative features while preserving target-domain discrimination.
- Motivation: UDA transfers knowledge from labeled source domains to unlabeled target domains despite domain shift.Its goal is a shared feature space that is both discriminative and domain-invariant.
- Research gap: ViT transferability remains underexplored, while global-feature alignment assumes all image regions are equally transferable and discriminative.Background regions may align more easily, whereas foreground regions can be more discriminative.
- Findings: Experiments find ViT more transferable than strong CNN counterparts, with further gains from adversarial discrimination on class tokens.The paper attributes this partly to global context modeling and large-scale pre-training.
- Method: TVT introduces TAM, which measures patch-token transferability and injects it into self-attention to capture transferable and discriminative features.The framework uses ViT’s sequential patch tokens and self-attention rather than directly applying CNN-based adaptation strategies.
- Method: DCM uses discriminative clustering to preserve target-domain discriminative information that adversarial alignment may destroy.The module encourages separated individual predictions and globally diverse target predictions.
Related Work
Prior UDA methods primarily use CNN-based representations, whereas this paper studies ViT transferability and designs a ViT-specific adaptation framework.
- Unsupervised Domain Adaptation: UDA seeks transferable, domain-invariant features when target labels are unavailable, mainly addressing cross-domain distribution discrepancies.Earlier methods include domain-invariant representation learning and adversarial alignment.
- Unsupervised Domain Adaptation: Existing UDA methods largely use CNNs as their fundamental building blocks.The paper contrasts this CNN-centered literature with its exploration of ViT for UDA.
- Positioning: TVT is tailored to ViT and aims to exploit the transferability and discrimination of individual features for UDA.The related-work contrast positions TVT as a framework designed around ViT rather than inherited CNN adaptation strategies.
- Vision Transformer: ViT’s attention mechanism captures long-range dependencies, motivating its use beyond convolutional architectures.Transformers were originally developed in NLP and later applied to vision tasks.
- Research gap: Before this work, ViT transferability and ViT-specific UDA design had not been discussed in the literature to the authors’ knowledge.The paper therefore focuses on cross-domain knowledge transfer using ViT’s intrinsic merits.
Preliminaries
UDA jointly learns discriminative features, domain adaptation, and a classifier, commonly using adversarial losses; ViT’s core MSA captures dependencies through query-key attention.
- Adversarial Learning UDA: In UDA classification, labeled source data and unlabeled target data are used to learn features that are discriminative and domain-invariant.The objective jointly performs feature learning, domain adaptation, and classifier learning.
- Adversarial Learning UDA: The standard objective combines supervised classification loss with a transfer loss weighted by α.The transfer loss can have multiple implementations, including adversarial loss.
- Adversarial Learning UDA: Adversarial loss encourages domain-invariant features through a domain discriminator.The discriminator-based formulation is identified as one of the most commonly used transfer losses.
- Self-attention Mechanism: ViT’s multi-head self-attention captures long-range dependencies using queries, keys, and values.Scaled dot products between queries and keys produce softmax weights applied to values.
- Self-attention Mechanism: Multiple attention heads jointly attend to information from different representation subspaces and positions.MSA concatenates several scaled dot-product attention modules.
Methodology
TVT investigates ViT transferability for UDA and introduces adaptation modules that exploit patch-level transferability, attention, and target-domain clustering. Its framework combines adversarial alignment with mechanisms intended to preserve discriminative information while learning domain-invariant features.
- ViT’s Transferability: TVT first evaluates vanilla ViT for UDA and finds stronger transferability than CNN-based counterparts across multiple benchmarks.On Office-31, Source Only ViT reaches 89.45%, while on Office-Home it achieves 78.74% average accuracy and exceeds CNN-based UDA methods.
- ViT w/ Adversarial Adaptation: Baseline: Global adversarial adaptation improves vanilla ViT by 7.8%, 0.78%, 1.56%, and 3.21% across four benchmarks.The approach applies a domain discriminator to class-token outputs, but global alignment has limited benefits on more challenging object detection tasks.
- TVT Framework: TVT comprises Transferability Adaptation Module (TAM) and Discriminative Clustering Module (DCM), which complement adversarial domain adaptation.TAM targets transferable and semantically meaningful features, while DCM encourages large-margin clustering of aligned target features.
- Transferability Adaptation Module: TAM uses patch-level domain discrimination to estimate transferability and injects those estimates into class-token attention over sequential patch features.Patches that are difficult for the domain discriminator to classify receive higher transferability, while attention captures semantic importance.
- Transferability Adaptation Module: TAM is applied only to the last transformer layer, where patch features are spatially non-local and semantically richer.This design focuses attention on fine-grained features that are both transferable across domains and useful for classification.
- Discriminative Clustering Module: DCM maximizes mutual information between target inputs and predictions to preserve discriminative structure during cross-domain alignment.It encourages near one-hot individual predictions and a uniformly distributed global prediction, producing clustered and diverse target representations.
Experiments
TVT is evaluated across digit and object-recognition UDA benchmarks against established methods and ablations. It achieves strong cross-domain performance, with especially large gains on Office-Home and improvements attributable to TAM and DCM.
- Digit recognition: TVT obtains the best mean accuracy on each digit task and exceeds the Baseline by 2.7%.It also achieves performance comparable to the Target Only model on the digit benchmark.
- Object recognition: 83.56% vs 71.8% on Office-Home marks TVT’s improvement over the best prior UDA method.TVT also sets new benchmark results across Office-31, Office-Home, and VisDA-2017.
- Ablation study: TAM improves classification accuracy by an average 4.82% over Source Only, and DCM further improves performance.DCM produces its largest improvement on VisDA-2017, where the synthetic-to-real domain gap is large.
- Attention visualization: Attention visualizations show TVT selects local regions that are both transferable and discriminative, unlike attention focused mainly on less-transferable shoulder regions.The comparison uses target-domain images from VisDA-2017 against Source Only and Baseline models.
Conclusion
TVT investigates ViT transferability for domain adaptation and captures transferable, discriminative image features while preserving discriminative domain-invariant representations. Experiments on widely used benchmarks show that TVT outperforms prior UDA methods by a large margin.
- TVT investigates ViT’s transferability in unsupervised domain adaptation and finds ViT more transferable than CNN counterparts.
- TVT captures both transferable and discriminative features in each image.
- Figure 2 visualizes VisDA-2017 source synthetic renderings and target real images with red and blue points, respectively.
- Figure 3 visualizes attention maps for persons, trucks, and bicycles, with hotter colors indicating higher attention.
- TVT retains discriminative information in learned domain-invariant representations.
- TVT outperforms prior UDA methods by a large margin on widely used benchmarks.