PaperlayerView exact revision on arXiv

Source-linked AI summary

Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation

Kendrick Shen, Robbie Jones, Ananya Kumar, Sang Michael Xie, Jeff Z. HaoChen, Tengyu Ma, Percy Liang

arXiv:2204.00570v4cs.LGcs.CV

TL;DR

UDA asks how to learn a target classifier from labeled source data and unlabeled target data. The paper uses contrastive pre-training on both domains followed by source-label fine-tuning, and reports competitive benchmark performance despite domain-separable features. Its theory explains transfer through disentangled class and domain information rather than domain invariance.

  • Problem

    UDA must generalize from labeled source data to an unlabeled target domain, while conventional methods often seek domain-invariant features.

    Source: §1 · p. 1

  • Method

    The paper pre-trains an encoder contrastively on unlabeled source and target data, then fine-tunes using labeled source data, and analyzes transfer through augmentation connectivity.

    Source: §1 · pp. 1–2

  • Results

    Across visual UDA benchmarks, contrastive pre-training is comparable to or better than strong methods; on DomainNet, SwAV averages 44.91% versus 45.81% for DANN+strong-augs.

    Source: §1 · p. 1, §5.1 · p. 10

  • Takeaways & Limitations

    The results suggest that target generalization in UDA does not require domain-invariant features when class and domain information are disentangled.

    Source: §1 · p. 2, §7 · p. 12

    The theoretical setup is highly stylized, and its stochastic block model requires a uniform marginal input distribution.

    Source: §7 · p. 12

Abstract

from arXiv · show

We consider unsupervised domain adaptation (UDA), where labeled data from a source domain (e.g., photographs) and unlabeled data from a target domain (e.g., sketches) are used to learn a classifier for the target domain. Conventional UDA methods (e.g., domain adversarial training) learn domain-invariant features to improve generalization to the target domain. In this paper, we show that contrastive pre-training, which learns features on unlabeled source and target data and then fine-tunes on labeled source data, is competitive with strong UDA methods. However, we find that contrastive pre-training does not learn domain-invariant features, diverging from conventional UDA intuitions. We show theoretically that contrastive pre-training can learn features that vary subtantially across domains but still generalize to the target domain, by disentangling domain and class information. Our results suggest that domain invariance is not necessary for UDA. We empirically validate our theory on benchmark vision datasets.

1 Introduction

The paper studies UDA, where labeled source data and unlabeled target data support learning a target classifier, and finds that contrastive pre-training is competitive with strong UDA methods. It explains theoretically and empirically how target transfer can coexist with features that remain distinguishable across domains.

1 Introduction

  • UDA addresses poor performance when training and test distributions differ, using labeled source examples and unlabeled target examples to learn a target classifier.Conventional methods often seek domain-invariant features, but the paper argues that invariance is not necessary for target generalization.Source: §1 · pp. 1–2
  • Contrastive pre-training on unlabeled source and target data followed by source-label fine-tuning matches or exceeds strong UDA methods across visual adaptation benchmarks.Benchmarks include DomainNet, BREEDS Living-17, BREEDS Entity-30, and STL-10→CIFAR-10.Source: §1 · p. 1
  • In DomainNet, domain prediction error is 8% for contrastive pre-training versus 14% for DANN, showing that the learned features remain domain-separable.The paper also reports that domain distinctions are as easy to make as class distinctions in this feature space.Source: §1 · p. 1
  • The theory attributes target transfer without domain invariance to disentangled features that predict both domain and class, even when the domains are far apart.Its connectivity condition requires changing only class or only domain through augmentation to be more likely than changing both.Source: §1 · p. 2
  • On Living-17 and DomainNet, estimated connectivities satisfy the theory's conditions, and connectivity ratios explain DomainNet target accuracy with R2 = 0.78.Pre-training on subsets chosen to alter connectivity underperforms random subsets of the same size by 4%-16% in target accuracy.Source: §1 · p. 2

2 Related work

Related work commonly seeks to align source and target features, while contrastive pre-training is competitive without an explicit domain-distance objective. The paper frames its connectivity model as another way to describe distribution shift.

2 Related work

  • Conventional domain adaptation methods often minimize source-target feature divergence, including through adversarial training and moment matching.Other approaches alter divergence measures or balance divergence and classification losses.Source: §2 · pp. 2–3
  • Most theoretical guarantees for contrastive learning concern downstream training and testing from the same distribution, unlike this paper's source-to-target setting.The cited contrastive methods learn by bringing augmentations of the same input together and separating random inputs.Source: §2 · p. 3
  • Prior self-supervised UDA methods seek to reduce domain distance in their objective or model selection, whereas contrastive pre-training has no explicit domain-distance objective.The paper reports that contrastive pre-training remains competitive and explains this through its connectivity theory.Source: §2 · p. 3
  • The paper's connectivity model offers an alternative description of distribution shift that permits adaptation under large H∆H-divergence.Other cited accounts assume gradual shift, target-classifier accuracy with expansion, or overlap after augmentation.Source: §2 · p. 3

3 Setup

The setup defines UDA as learning a target classifier from labeled source data and unlabeled data across domains. It contrasts ERM and domain-adversarial training with contrastive pre-training, which learns features from unlabeled data before source-label fine-tuning.

3.1 Unsupervised Domain Adaptation (UDA)

  • The setup treats each input as having a deterministic class label and domain, while noting that this simplifying assumption is not required.The more general formulation allows shared label and domain distributions conditioned on inputs, corresponding to covariate shift.Source: §3.1 · p. 3
  • UDA provides labeled source examples and unlabeled target examples, and evaluates a learned classifier by its error on the target distribution.The theoretical setup uses a mixture of source and target input distributions as the unlabeled data distribution.Source: §3.1 · pp. 3–4
  • Augmentations such as crops and color distortions are represented as a distribution over transformed versions of each input.These augmentations are used in the learning procedures defined in the methods section.Source: §3.1 · p. 4

3.2 Methods

  • ERM minimizes classification loss on augmented source examples with their original labels, and augmentations improve its target accuracy.The paper reports that its results also hold without augmentations.Source: §3.2 · p. 4
  • DANN combines source classification with a domain-confusion loss that makes the domain difficult to predict from encoded features.Its objective trades off the two losses using a positive parameter and a domain-classification head.Source: §3.2 · p. 4
  • Contrastive pre-training brings augmented views of the same input together and separates views of random inputs, then fine-tunes a classifier on labeled source data.The encoder is pre-trained using unlabeled source and target data; experiments fine-tune both the head and encoder.Source: §3.2 · p. 4
  • Table 1 uses augmented-image class-domain prediction error as a connectivity proxy; domains are harder to distinguish in DANN than SwAV features.Classes are very distinguishable in both feature spaces.Source: Table 1 · p. 5

4 Intuitions and Analysis

The analysis extends spectral contrastive learning to domain shift, using source-labeled linear classification after pre-training on unlabeled data. It explains how learned features can remain domain-distinguishable yet transfer, and identifies connectivity conditions under which this occurs.

4.1 Theoretical setup

  • The theoretical setup adapts spectral contrastive learning to distribution shift and trains a linear probe on labeled source-domain features.The spectral objective learns an encoder from unlabeled data; the probe minimizes squared loss.Source: §4.1 · p. 5

4.2 Simple example

  • The toy example uses four inputs—clock and butterfly sketches and photos—with labeled sketches, unlabeled data from both domains, and photos as the target.The input distribution for pre-training is uniform over all four images.Source: §4.2 · pp. 5–6
  • The toy example’s key augmentation condition favors preserving class and domain together over changing either, while changing both is less likely than changing just one.The paper motivates this condition with transformations such as aggressive crops and color jitter that can blur class or domain distinctions.Source: §4.2 · p. 6
  • With α and β greater than γ, the toy example disentangles class and domain features, allowing a source-trained classifier to achieve zero target error despite distinguishable domains.When α < γ, the target features flip and the source-trained classifier misclassifies them.Source: §4.2 · pp. 7–8, Figure 2 · p. 7

4.3 Theory for multi-class stochastic block model

  • The stochastic block model assumes independent edges, with probabilities ρ, α, β, or γ determined by whether a pair shares class and domain.The unlabeled distribution is uniform over the input set.Source: §4.3 · p. 8
  • Theorem 1 gives transferable features with feature dimension r + 1 when ρ > max{α, β} and min{α, β} > γ.The condition requires augmentations to be more likely to preserve both class and domain than to change either, and less likely to change both than just one.Source: §4.3 · p. 8
  • In the stochastic block model, error-reduction rate depends on the gap between α and β connectivities and γ; matrix concentration and perturbation analysis establish transfer with high probability.The analysis extends to more than two domains, and its bound can hold with probability arbitrarily close to 1.Source: §4.3 · p. 8

4.4 Setting where ERM/DANN underperform contrastive pre-training

  • Contrastive pre-training can achieve zero target error where ERM and DANN minimizers have target error 1/3.The construction concerns target examples unreachable by augmentations of labeled source inputs; ERM can predict arbitrarily on them, while DANN can swap their classes.Source: §4.4 · p. 9

5 Contrastive pre-training is a strong domain adaptation method

Contrastive pre-training on unlabeled source and target data, followed by source-label fine-tuning, is competitive with strong UDA methods across benchmark vision datasets.

5 Contrastive pre-training is a strong domain adaptation method

  • Experiments cover DomainNet, BREEDS Living-17 and Entity-30, and STL-10→CIFAR-10, using SwAV for the first three and pre-trained SimCLR for STL→CIFAR.The models are fine-tuned on labeled source data; baselines include ERM, DANN, SENTRY, and Dirt-T, with augmentation variants also compared.Source: §5 · pp. 9–10

5.1 Results

  • Contrastive pre-training is competitive across four benchmarks; SwAV is near the strongest baselines on DomainNet and Living-17, improves Entity-30, and SimCLR edges Dirt-T on STL→CIFAR.On DomainNet, SwAV averages 44.91% versus 45.81% for DANN+strong-augs; on Living-17, it reaches 75.12% versus SENTRY's 75.53%.Source: §5.1 · p. 10, Table 2 · p. 10
  • Using additional unlabeled data from related domains further improves SwAV by nearly 7% on DomainNet, 7% on Living-17, and 4% on Entity-30, but is not a fair comparison to other methods.The SwAV+extra setup pre-trains on a superset of domains before fine-tuning on one domain.Source: §5.1 · p. 10

6 Evaluating connectivity on real datasets

The empirical connectivity measures satisfy the theory's conditions, and connectivity ratios help explain contrastive pre-training's target accuracy. Ablations and classifier comparisons further probe the roles of connectivity and feature separation.

6 Evaluating connectivity on real datasets

  • Connectivity ratios explain SwAV target-accuracy gains across 12 DomainNet domain pairs, with R2 = 0.78 between predicted and observed accuracy.The prediction uses the product of across-domain and across-class ratios, normalized by each target's worst source/target pair.Source: Figure 3 · p. 11, §6 · p. 11
  • Estimated connectivities satisfy the theoretical conditions on Living-17 and DomainNet, and their ratios are predictive of contrastive pre-training target accuracy.The estimates use classifiers trained to identify class-domain pairs; input-space estimates may overstate true connectivity because they use much smaller training sets.Source: §6 · p. 11
  • The connectivity-ratio predictions also fit MoCo-V2 and MoCo-V3 target accuracies (R2 = 0.79 and 0.60), but fit baselines less well (average R2 = 0.21).Source: §6 · p. 11
  • Across five subset sizes, pre-training on examples farthest from a domain-classifier boundary reduces target accuracy more than random or least-confident subsets.The across-domain connectivity is described as bridging domains, and the regression assigns its ratio the largest weight.Source: Figure 3 · p. 11, §6 · p. 12
  • In SwAV features, domain-classifier weights are nearly orthogonal to source and target class weights, while source and target class weights are relatively aligned.Average cosine similarity between domain and class weights is below 0.02; same-class source/target similarities average 0.40 on Living-17 and 0.19 on DomainNet.Source: §6 · p. 12, Table 3 · p. 12

7 Conclusion

The paper concludes that contrastive pre-training can transfer across distribution shifts without learning domain-invariant features. Its theory has a stylized assumption, and the authors identify potential applications and directions for follow-up work.

7 Conclusion

  • Connectivity theory may guide pre-training data selection, augmentations, and fine-tuning, and may explain weak contrastive-pretraining performance on iWildCam2020-WILDS.The authors also suggest future UDA methods could focus on increasing connectivity rather than collapsing domains.Source: §7 · p. 12
  • The theoretical setup uses a stylized stochastic block model requiring a uniform marginal input distribution, which may not be realistic.HaoChen et al. (2022) prove linear transferability under a more general setting, with a different connectivity requirement.Source: §7 · pp. 12–13
  • The paper's exploratory classifier analysis uses target labels, although the authors state this cannot be done in practice.Source: §7 · p. 12

8 Ethics

The experiments use publicly available benchmark vision datasets, while large-scale unsupervised learning may involve scraped data and privacy concerns.

8 Ethics

  • Large-scale contrastive pre-training on internet-scraped data may raise privacy concerns, so dataset curation requires care.The reported experiments instead use public standard domain-adaptation benchmarks.Source: §8 · p. 13

A.1 Proof of Theorem 1

The proof models contrastive pre-training with a graph whose edges represent positive pairs and whose structure reflects class and domain connectivity. It analyzes predictions from source-labeled data on the target domain through the graph’s spectral structure.

A.1 Proof of Theorem 1

  • The graph model assigns edge probabilities according to whether two examples share a domain, class, both, or neither.The respective probabilities are ρ, α, β, and γ.Source: §A.1 · p. 17
  • The proof establishes accuracy for predictions in the expectation graph, then bounds the effect of replacing it with a sampled graph.It controls rank-k approximation error and adjacency-matrix concentration under stated conditions.Source: §A.1 · pp. 18, 20, 21
  • With k = r + m − 1, the expected adjacency matrix has a spectral gap of n min{r(β − γ), m(α − γ)} after its top k eigenvalues.The gap depends on class and domain connectivity parameters.Source: §A.1 · p. 18

A.2 Proof of Proposition 3

The proposition gives an example where contrastive pre-training achieves zero target error, while ERM and DANN each have minimizers with target error 1/3. Its proof contrasts augmentation reachability with connectivity across the target domain.

A.2 Proof of Proposition 3

  • ERM and DANN can leave target inputs outside the source augmentation-reachable set incorrectly classified, while target-wide connectivity lets contrastive learning reach multiple-hop inputs.For ERM, two of the six target inputs are outside the reachable set, permitting a minimizer with error 1/3.Source: Figure 4 · p. 24, §A.2 · p. 25
  • Contrastive pre-training achieves zero target error, whereas ERM and DANN each have minimizers with target error 1/3.The proposition asserts this contrast for all feature dimensions for the ERM and DANN minimizers.Source: §A.2 · p. 24
  • When α > γ + β, contrastive pre-training achieves zero target error in the example.This condition means same-class augmentations occur more often than different-class augmentations, as described in the passage.Source: §A.2 · p. 27
  • The contrastive proof derives representations from unlabeled positive pairs, then fits a linear probe using labeled source examples.The resulting classifier predicts every target example correctly in the constructed example.Source: §A.2 · pp. 27, 29

B.1 Additional Experiments

The additional experiments compare contrastive methods and UDA baselines across DomainNet and BREEDS, with further experiments varying stopping rules, methods, and compute. The appendix also documents datasets, tuning protocols, and model architectures.

B.1 Additional Experiments

  • On DomainNet, MoCo-V3 reports 41.59% target accuracy, ahead of DANN at 38.38%, but behind SwAV and DANN+strong-augs.Early stopping still leaves only DANN+strong-augs above SwAV, by 1.3% with source-accuracy stopping and 3.2% with target-accuracy oracle stopping; SwAV and MoCo (S+T) are within 1% on both BREEDS tasks.Source: §B.1 · p. 30

B.2 Datasets

  • The appendix evaluates BREEDS subpopulation shifts, filtered DomainNet domain pairs, and a nine-class STL→CIFAR task.The DomainNet experiments use four domains and 40 classes; STL images are resized to CIFAR resolution and two non-overlapping classes are removed.Source: §B.2 · p. 31

B.3 Algorithms and Hyperparameter Tuning

  • Joint-training baselines required extensive task-specific tuning, whereas SwAV hyperparameters were selected once using unlabeled Living-17 training curves and reused across datasets.For joint-training baselines, target test accuracy was used to select hyperparameters.Source: §B.3 · p. 31, §B.4 · p. 33

B.4 SwAV

  • SwAV was pretrained for 400 epochs; its fine-tuning protocol varies by dataset, including linear probing rather than fine-tuning for Entity-30.For full Living-17 and Entity-30, the queue is introduced at epoch 60; it is omitted on DomainNet and subsampled Living-17 to stabilize training.Source: §B.4 · pp. 32–33

B.5 SimCLR

  • SimCLR uses a 256-example batch, learning rate 0.2, and 1,000 pretraining epochs, followed by a 100-epoch linear probe.The probe uses batches of 512 and learning rate 0.1.Source: §B.5 · p. 33

B.6 MoCo-V2 and MoCo-V3

  • MoCo-V2 and MoCo-V3 use their respective default hyperparameters and the SwAV fine-tuning protocol on DomainNet and Living-17.The MoCo-V2 hyperparameters were tuned for ImageNet and were not further tuned for DomainNet, where its target accuracies were below several baselines.Source: §B.1 · p. 30, §B.6 · p. 33

B.7 Dino and Barlow Twins

  • Dino and Barlow Twins use downloaded ImageNet-pretrained weights, with fine-tuning on Living-17 and linear probing on Entity-30.Their pretraining durations were 800 and 1,000 epochs, respectively.Source: §B.7 · p. 33

B.8 Model architectures.

  • The experiments use ResNet-50 for BREEDS and DomainNet, and ResNet-18 for the listed STL→CIFAR baselines.The DomainNet ResNet-50 is modified for few-shot learning; DIRT-T uses a custom 18-layer CNN on STL→CIFAR.Source: §B.8 · p. 33

C Additional Experiments Verifying the Theory

This section tests whether connectivity terms from the theory explain target accuracy across DomainNet domain pairs. The fitted connectivity model has higher reported R2 for contrastive methods than for the DANN and SENTRY baselines.

C.1 Target accuracy as a function of connectivity

  • Connectivity-based predictions have R2 values of 0.78, 0.79, and 0.60 for SwAV, MoCo-V2, and MoCo-V3, versus 0.23–0.51 for DANN and −0.20–0.22 for SENTRY.The fitted model uses data from 12 DomainNet source/target pairs; Figures 5 and 6 plot observed against predicted target accuracy.Source: §C.1 · p. 34

C.2 Ablation of connectivity

The Living-17 connectivity ablation used a selected four-class subset; the following analysis found class and domain classifiers remain orthogonal in fine-tuned features while source- and target-classifier similarity increases.

C.2 Ablation of connectivity

  • The Living-17 ablation retained only classes 6, 11, 12, and 14, selected because their source-versus-target domain classifiers exceeded 70% accuracy.All considered data-selection methods chose subsets from this modified dataset.Source: §C.2 · p. 34

C.3 Disentanglement of class and domain information

  • In fine-tuned DomainNet features, class and domain classifiers remain orthogonal, while source- and target-classifier cosine similarity rises 41%, from 0.187 to 0.264.Source: §C.3 · p. 34

D Results and Protocol for Estimating Connectivity Parameters on Benchmark Datasets

The section estimates connectivity between class-domain pairs with binary classifiers and reports how connectivity patterns differ across input and learned feature spaces. Appendix results relate connectivity ratios to target accuracy and compare benchmark performance across methods and evaluation settings.

D Results and Protocol for Estimating Connectivity Parameters on Benchmark Datasets

  • Connectivity is estimated by training binary classifiers to distinguish two class-domain pairs, then treating strongly augmented test error as their connectivity.The average uses a weighted mean, with each pair weighted by the ratio of its less- to more-frequent label.Source: D Results and Protocol for Estimating… · p. 34
  • Input-space connectivity values are relative estimates; CLIP feature-space estimates are much lower than SwAV and DANN estimates, with across-class and across-domain magnitudes swapped relative to input space.The authors suggest this discrepancy may reflect CLIP training-data biases, such as captions being more informative about class than domain.Source: D Results and Protocol for Estimating… · pp. 34–35, Table 12 · p. 40
  • SwAV’s across-domain and across-class connectivities were similar (7.54 vs. 7.03), while DANN+strong-augs had higher across-domain connectivity (13.64 vs. 5.65).Table 12 reports classifier errors as empirical connectivity estimates for input and feature spaces.Source: D Results and Protocol for Estimating… · p. 35, Table 12 · p. 40

E Appendix Figures and Tables

  • On DomainNet, SwAV averages within 1% of DANN+strong-augs (44.91 vs. 45.81), and early stopping improves SENTRY more than DANN.SENTRY gains are 5.1% and 7.5% with source- and target-based early stopping, versus 0.2% and 2.7% for DANN; DANN+strong-augs exceeds SwAV by 1.3% and 3.2% under those settings.Source: Table 4 · p. 35, Table 5 · p. 36
  • On Living-17 and Entity-30, early stopping boosts SENTRY variants, while MoCo-V3 and ImageNet-pretrained Dino and Barlow Twins are comparable to SwAV.Under standardized compute, DomainNet DANN results are comparable to the main table while SENTRY results are lower; Living-17 and Entity-30 results are within 2.5% of the main-table results.Source: Table 6 · p. 36, Table 7 · p. 36, Table 9 · p. 37
  • Target-accuracy fits using connectivity ratios have R2 values of 0.78, 0.79, and 0.60 for SwAV, MoCo-V2, and MoCo-V3, while the ratios poorly predict DANN and SENTRY variants.For DANN and DANN+strong-augs, R2 ranges from 0.23 to 0.51; for SENTRY and SENTRY+strong-augs, it ranges from −0.20 to 0.22.Source: Figure 5 · p. 37, Table 10 · p. 38, Figure 6 · p. 38
Loading 2204.00570v4…