PaperlayerView exact revision on arXiv

Source-linked AI summary

Patch-Mix Transformer for Unsupervised Domain Adaptation: A Game Perspective

Jinjing Zhu, Haotian Bai, Lin Wang

arXiv:2303.13434v2cs.CV

TL;DR

Domain shift limits vision models' generalization to new domains, motivating unsupervised adaptation from labeled source data to unlabeled target data. PMTrans builds an intermediate domain with PatchMix in a three-player min-max cross-entropy game. It surpasses state-of-the-art methods on Office-Home, Office-31, and DomainNet, with reported gains of +3.6%, +1.4%, and +7.2%, respectively.

  • Problem

    Domain shift limits vision models' generalization to new domains, motivating transfer from labeled source data to unlabeled target data.

    Source: §1 · p. 1

  • Method

    PMTrans uses PatchMix to form an intermediate domain from source and target patches in a three-player min-max cross-entropy game, with mixup losses and attention-based patch-label reweighting.

    Source: §1 · p. 2, §3.1.2 · p. 4, §3.2 · p. 4

  • Results

    +3.6% on Office-Home, +1.4% on Office-31, and +7.2% on DomainNet versus ViT-based and CNN-based state-of-the-art methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper treats source and target domains as aligned when mixing their patch representations is equivalent to mixing their corresponding labels.

    Source: §1 · p. 2

    The method assumes that source and target domains share the same label space.

    Source: §3 · p. 3

Abstract

from arXiv · show

Endeavors have been recently made to leverage the vision transformer (ViT) for the challenging unsupervised domain adaptation (UDA) task. They typically adopt the cross-attention in ViT for direct domain alignment. However, as the performance of cross-attention highly relies on the quality of pseudo labels for targeted samples, it becomes less effective when the domain gap becomes large. We solve this problem from a game theory's perspective with the proposed model dubbed as PMTrans, which bridges source and target domains with an intermediate domain. Specifically, we propose a novel ViT-based module called PatchMix that effectively builds up the intermediate domain, i.e., probability distribution, by learning to sample patches from both domains based on the game-theoretical models. This way, it learns to mix the patches from the source and target domains to maximize the cross entropy (CE), while exploiting two semi-supervised mixup losses in the feature and label spaces to minimize it. As such, we interpret the process of UDA as a min-max CE game with three players, including the feature extractor, classifier, and PatchMix, to find the Nash Equilibria. Moreover, we leverage attention maps from ViT to re-weight the label of each patch by its importance, making it possible to obtain more domain-discriminative feature representations. We conduct extensive experiments on four benchmark datasets, and the results show that PMTrans significantly surpasses the ViT-based and CNN-based SoTA methods by +3.6% on Office-Home, +1.4% on Office-31, and +17.7% on DomainNet, respectively.

1. Introduction

The introduction motivates bridging source and target domains through an intermediate domain and frames UDA as a three-player min-max cross-entropy game. It reports gains over state-of-the-art methods on Office-Home, Office-31, and DomainNet.

1. Introduction

  • +3.6% on Office-Home, +1.4% on Office-31, and +7.2% on DomainNet versus ViT-based and CNN-based state-of-the-art methods.Figure 1 reports the DomainNet comparison across tasks that use one domain as target and the others as source.Source: Figure 1 · p. 1, §1 · p. 2
  • Direct alignment can become less effective across large domain gaps because cross-attention depends on target pseudo-label quality.The introduction cites the gap between DomainNet's qdr domain and the other domains as an example.Source: §1 · p. 1
  • PMTrans uses PatchMix to sample source and target patches into an intermediate domain within a three-player min-max cross-entropy game.The players are the feature extractor, classifier, and PatchMix; PatchMix maximizes cross-entropy while the other players minimize it.Source: §1 · p. 2, Figure 2 · p. 2
  • Two semi-supervised mixup losses in feature and label spaces help minimize cross-entropy in the game.The paper interprets alignment as reached when mixing patch representations is equivalent to mixing their corresponding labels.Source: §1 · p. 2
  • ViT attention maps re-weight each patch's label by its significance to support more domain-discriminative features.Source: §1 · p. 2

2. Related Work

Prior UDA work aligns domains through metric learning, adversarial training, or optimal transport, while transformer methods have explored direct alignment and adaptation modules. PMTrans instead uses patch mixing to build an intermediate domain and frames alignment as a min-max cross-entropy game.

2. Related Work

  • Metric-learning methods use distribution discrepancy losses, adversarial methods seek domain-indiscriminative features, and optimal-transport methods minimize coupling cost between source and target distributions.The passage cites MMD, CMD [49], MDD, adversarial training [43] [44], and optimal transport.Source: §2 · p. 2
  • The paper notes that prevailing UDA approaches are not robust enough to noisy target pseudo labels and presents a min-max CE game with an intermediate domain as an alternative.The stated approach uses a pure ViT-based solution.Source: §2 · p. 2
  • PMTrans unifies global and local mixup by learning to combine source and target patches as ViT inputs.The paper describes learning patch-level mixup ratios through distribution estimation.Source: §2 · p. 2
  • Prior ViT-based UDA methods include transferable-feature adaptation [47], safe self-refinement [37], and cross-attention alignment with pre-generated pseudo labels [46].PMTrans differs by constructing an intermediate domain to bridge source and target domains.Source: §2 · p. 3

3. Methodology

PMTrans bridges source and target domains with a patch-mixed intermediate domain, framing adaptation as a three-player min-max cross-entropy game. Its framework uses feature- and label-space mixup losses and attention-weighted labels to align domains.

3.1.1 PatchMix

  • PatchMix samples source and target patches with weights drawn from a learnable Beta distribution, forming intermediate examples and interpolated labels.When mixing patches is equivalent to mixing labels, minimizing their cross-entropy facilitates domain alignment.Source: §3.1.1 · p. 3

3.1.2 A Min-Max CE Game

  • The feature extractor and classifier minimize cross-entropy while PatchMix maximizes it, with each player optimizing its own cost toward Nash equilibrium.The feature extractor and classifier share an objective combining source classification loss and cross-entropy; PatchMix’s cost uses the negative cross-entropy.Source: §3.1.2 · pp. 3–4

3.2. The Proposed Framework

  • Patch embedding, a ViT encoder, and a classifier form the pipeline; normalized attention scores reweight mixed labels according to patch importance.Classifier predictions select feature maps used to reweight patch sequences.Source: §3.2 · p. 4
  • Two semi-supervised mixup losses minimize discrepancies in feature and label spaces; feature matching uses label similarities without target supervision.The feature-space loss addresses the limited reliability of target pseudo-labels.Source: §3.2 · pp. 4–5
  • After optimization, the authors report that intermediate-to-source/target discrepancy is nearly zero, indicating source–target alignment.At this point, PatchMix no longer maximizes cross-entropy, and the feature extractor and classifier have no incentive to change their parameters.Source: §3.2 · p. 5

4. Experiments

Experiments evaluate PMTrans across four UDA benchmarks against CNN- and ViT-based methods, with results and ablations supporting its transfer performance and design choices.

4.1. Datasets and implementation

  • Experiments span four benchmarks and compare ResNet- and ViT-based baselines; PMTrans uses ImageNet-pretrained Swin, while Deit- and ViT-backbone comparisons train for 60 and 100 epochs.The main experiments train for 50 epochs, with a lower learning rate for VisDA-2017.Source: §4.1 · p. 6, §4.2 · p. 6

4.2. Results

  • PMTrans reports 89.0% accuracy on Office-Home, exceeding Swin by more than 5.4%, and 95.3% accuracy on Office-31, with gains over TVT, CDTrans, and SSRT.The Office-31 gains are +1.4%, +2.7%, and +1.8%, respectively.Source: §4.2 · p. 6
  • On VisDA-2017, PMTrans reaches 88.0% accuracy, 11.2% above the baseline, while the SoTA score is 88.7%; person accuracy rises from 29.0% to 70.3%.Table 3 presents the VisDA-2017 comparison.Source: §4.2 · p. 6, Table 3 · p. 7
  • On DomainNet, PMTrans beats SoTA by +7.2% accuracy and wins all 30 subtasks; on qdr targets, it averages 27.0% versus SSRT's 13.7% and CDTrans's 19.6%.Table 4 presents the DomainNet comparison.Source: §4.2 · p. 7, Table 4 · p. 7

4.3. Ablation Study

  • On Office-Home, semi-supervised mixup improves accuracy over source-only Swin by +1.0% in feature space and +4.3% in label space; label-space CE performs better.Table 5 reports the semi-supervised-loss ablation.Source: §4.3 · p. 7, Table 5 · p. 8
  • On Office-Home, learning the Beta-distribution parameters improves accuracy by +0.9% over Beta(1,1) and +0.8% over Beta(2,2).Table 6 reports the parameter-learning ablation.Source: §4.3 · p. 8, Table 6 · p. 8
  • On Office-Home, PMTrans exceeds Mixup by +0.7% and CutMix by +1.4% accuracy; the paper attributes this to capturing both global and local mixture information.Table 7 compares these mixing methods.Source: §4.3 · p. 8, Table 7 · p. 8
  • For Office-Home task A→C, t-SNE visualizations use blue for source and red for target; the paper reports better domain alignment with PMTrans than Swin across transformer backbones.The comparison includes Swin-based, PMTrans-Swin, PMTrans-ViT, and PMTrans-DeiT features.Source: §4.3 · p. 8, Figure 5 · p. 8

5. Conclusion and Future Work

The conclusion presents PMTrans as a game-theoretic UDA method and reports state-of-the-art results on four benchmark datasets, while identifying future applications.

5. Conclusion and Future Work

  • PMTrans formulates UDA as a game-theoretic optimization approach.The conclusion describes it as an optimization solution for UDA from a game perspective.Source: §5 · p. 8
  • PatchMix builds an intermediate domain, while two semi-supervised mixup losses and ViT attention-based patch-label reweighting support discriminative domain-invariant representations.The losses assist in finding Nash Equilibria; attention maps reweight patch labels by significance.Source: §5 · p. 8
  • PMTrans achieves state-of-the-art results on four benchmark UDA datasets; planned future work includes self-supervised and semi-supervised learning, semantic segmentation, and object detection.These are stated as future plans, not reported evaluations.Source: §5 · p. 8

Perspective –Appendix–

The supplied material identifies the paper's authors, affiliations, and arXiv version date.

Perspective –Appendix–

  • The listed authors are Jinjing Zhu, Haotian Bai, and Lin Wang.The author line marks Zhu and Bai with asterisks and Wang with a dagger.Source: Perspective –Appendix– · p. 12
  • The affiliations are AI Thrust at HKUST(GZ) and the Department of CSE at HKUST.The text associates the first affiliation with markers 1 and 2 with the second.Source: Perspective –Appendix– · p. 12
  • The supplied arXiv record is version 2, dated 6 June 2023.The record gives identifier arXiv:2303.13434v2.Source: Perspective –Appendix– · p. 12

Abstract

The supplementary material outlines a proof, method and algorithm details, experimental analyses, and discussion.

Abstract

  • The supplement begins by proving Theorem 1.The proof is assigned to Section 1 of the supplement.Source: Abstract · p. 12
  • It then presents details of the proposed method and the PMTrans algorithm.These topics are assigned to Sections 2 and 3, respectively.Source: Abstract · p. 12
  • Later sections provide results, analyses, ablations, and discussion of the proposed work.The overview assigns results and ablations to Sections 4 and 5, and discussion to Section 6.Source: Abstract · p. 12

1. Proof

The proof characterizes when patch-and-label mixing aligns source and target distributions, then describes the resulting constraints on class representations. It uses a linear classifier and requires representation dimension at least d−1.

1. Proof

  • The theorem’s condition is dim(H) ≥ d−1, where d is the number of classes; its proof uses a linear classifier c⋆(h)=A⊤h+b.The proof constructs A and b so that A⊤H+b⊤ can represent the d-dimensional identity matrix.Source: §1.1 · p. 12
  • When patch mixing is equivalent to mixing corresponding labels, the proof states that the source and target domains are aligned, with divergence D(PS, PT)=0.This result is established under the theorem’s dimensionality condition and the stated minimization assumptions.Source: §1.1 · p. 12
  • At the minimizer, each class representation lies on a subspace of dimension dim(H)−d+1, leaving most representation-space directions with zero class-conditional variance when dim(H) is larger.The representations retain some degrees of freedom to move independently.Source: §1.1 · p. 13
  • The paper uses Eq. 3 to measure gaps between the intermediate domain and the source and target domains in label and feature spaces.The divergence is defined through the minimum expected cross-entropy over the classifier and mixup functions.Source: §1.1 · pp. 12–13

2. Details

The details cover four UDA benchmarks, backbone-specific attention extraction, semi-supervised similarity losses, and a three-player optimization game. PMTrans also combines global and local mixup ideas through learned patch mixing.

2.1. Datasets

  • The experiments use Office-31, Office-Home, VisDA-2017, and DomainNet, spanning image classification domains and synthetic-to-real adaptation.Office-31 has 31 categories across three domains; Office-Home has 65 classes across four domains; DomainNet has 345 classes across six domains.Source: §2.1 · p. 13

2.2. Attention Map

  • Attention scores come from averaged self-attention and the CLS token for DeiT/ViT, while Swin uses class-specific activation maps derived from classifier units.For Swin, source ground-truth labels and target pseudo-labels select the class activation map, which is softmaxed for use as attention.Source: §2.2 · pp. 13–14

2.3. Semi-supervised mixup loss in the feature space

  • The feature-space loss compares normalized cosine similarities between intermediate samples and source or target samples against label similarities using cross-entropy.Intermediate-to-source label similarity uses shared labels; intermediate-to-target similarity is defined by patch overlap, without target supervision.Source: §2.3 · p. 14

2.4. Optimization

  • The feature extractor and classifier minimize source classification loss plus domain-gap cross-entropy, while PatchMix maximizes the domain-gap term in the three-player game.The game may reach a Nash equilibrium when no player can benefit by changing only its own strategy; training does not require gradient-reversal layers.Source: §2.4 · p. 14

2.5. Comparisons with Mixup variants

  • PatchMix unifies global and local mixup by learning to combine source and target patches and learn per-patch mixup ratios for intermediate samples.The comparison describes Mixup’s local ambiguity and CutMix’s possibility of producing images without valid objects despite proportional label mixing.Source: §2.5 · p. 15

3. Algorithm

The algorithm alternates between constructing an intermediate domain and computing feature- and label-space losses, then updates the feature extractor, classifier, and PatchMix parameters.

3. Algorithm

  • Each training iteration samples source and target batches, encodes their patches, computes normalized attention scores, samples mixup ratios from Beta(ωP), and constructs intermediate inputs.PatchMix mixes each patch element-wise using its ratio and combines source and target labels using the resulting scaled ratios.Source: §3 · p. 15
  • The algorithm computes semi-supervised feature- and label-space mixup losses and measures divergence between the intermediate domain and the source and target domains.It then updates the feature extractor, classifier, and PatchMix parameters using optimization (8) with AdamW.Source: §3 · p. 15

4. Results and Analyses

The supplied results discuss training behavior, efficiency, attention visualizations, and ablations, alongside benchmark comparison tables whose numerical contents are not included here. They also report qualitative effects of batch size, label handling, and learned mixup parameters.

4.3. Testing

  • PMTrans-ViT converges faster than PMTrans-Swin and PMTrans-Deit on Office-Home task A→C, while all three backbones reduce domain divergence.The reported comparison uses the same number of epochs; the cited accuracy and loss figures identify task A→C but do not provide cell values here.Source: §4.3 · p. 16, Figure 3 · p. 16, Figure 4 · p. 16

4.4. Complexity

  • PMTrans has stated complexity O(d+c+n), compared with CDTrans’s per-layer O(n^2d), excluding backbone choice.Here n is sequence length, d representation dimension, and c number of classes; PMTrans’s attention scores add no cost because they come from existing transformer and classifier parameters.Source: §4.4 · p. 16

4.5. Attention map visualization for target data

  • After domain alignment, attention maps tend to focus more on objects; in the pencil example, Swin distinguishes pencil from attached plasticine, unlike DeiT.The authors suggest this semantic focus may help explain Swin’s superior performance on many datasets.Source: §4.5 · p. 16, Figure 6 · p. 19

5.1. Batch size

  • Larger batch sizes generally improve UDA performance across the tested PMTrans backbones, but hardware limits training to batch size 32.The authors note performance may be lower than it could be under a batch-size-64 comparison.Source: §5.1 · p. 17, Table 4 · p. 18

5.2. Semi-supervised mixup loss with class information

  • Using class information in the semi-supervised mixup loss decreases domain gaps by reducing disparities between feature and label similarities.Target pseudo-labels are used to calculate the feature and label discrepancy.Source: §5.2 · p. 18, Table 5 · p. 18

6.3. Details of yit

  • The target label similarity yit is defined as an identity matrix without target pseudo-labels, avoiding the confirmation bias associated with noisy pseudo-labels.The paper also states that attention-weighted labels approximate label mixing and that attention helps convergence without sacrificing performance.Source: §6.3 · p. 18, §6.1 · p. 18

6.4. Learning hyper-parameters of mixup

  • Learning Beta-distribution parameters outperforms fixed mixup parameters within the same min-max CE game; fixed-ratio PatchMix still maximizes CE.The learned parameters provide flexible mixup ratios for constructing the intermediate domain.Source: §6.4 · p. 18

6.5. t-SNE visualization

  • The visualization comparison shows PMTrans forms a compact target-domain cluster closer to source-domain instances than Swin-Base; on Office-Home, PMTrans-ViT scores 88.9% with or without attention, and attention makes convergence easier.With the same batch size of 32, PMTrans and SSRT each score 88.0%.Source: §6.5 · p. 19, §6.6 · p. 19, §6.7 · p. 19

Related papers

Loading 2303.13434v2…