PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation

David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, Alex Kurakin

arXiv:2106.04732v2cs.LGcs.AIcs.CV

TL;DR

Unlabeled data can come from a shifted domain, while SSL, UDA, and SSDA have largely been addressed with separate techniques. AdaMatch unifies these settings and matches or exceeds state-of-the-art performance across them, including 28.7% versus 23.9% target accuracy on DomainNet224.

  • Problem

    SSL, UDA, and SSDA use different data configurations and domain-shift assumptions, but their techniques remain fragmented despite the practical importance of unlabeled data.

    Source: §1 · p. 1

  • Method

    AdaMatch unifies UDA, SSL, and SSDA with one hyperparameter set, accounting for source-target shifts through adaptive pseudo-labeling and distribution alignment.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    AdaMatch matches or exceeds state-of-the-art performance across UDA, SSL, and SSDA, achieving 28.7% average target accuracy versus 23.9% for NoisyStudent on DomainNet224.

    Source: §1 · p. 2, §5.1 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    A single method can perform well across SSL, UDA, and SSDA while improving accuracy when unlabeled or limited labeled data come from a shifted domain.

    Source: §7 · p. 9

    Confirmation bias in self-training may threaten fairness across major and minor classes, requiring further investigation of fairness and robustness.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

We extend semi-supervised learning to the problem of domain adaptation to learn significantly higher-accuracy models that train on one data distribution and test on a different one. With the goal of generality, we introduce AdaMatch, a method that unifies the tasks of unsupervised domain adaptation (UDA), semi-supervised learning (SSL), and semi-supervised domain adaptation (SSDA). In an extensive experimental study, we compare its behavior with respective state-of-the-art techniques from SSL, SSDA, and UDA on vision classification tasks. We find AdaMatch either matches or significantly exceeds the state-of-the-art in each case using the same hyper-parameters regardless of the dataset or task. For example, AdaMatch nearly doubles the accuracy compared to that of the prior state-of-the-art on the UDA task for DomainNet and even exceeds the accuracy of the prior state-of-the-art obtained with pre-training by 6.4% when AdaMatch is trained completely from scratch. Furthermore, by providing AdaMatch with just one labeled example per class from the target domain (i.e., the SSDA setting), we increase the target accuracy by an additional 6.1%, and with 5 labeled examples, by 13.6%.

1 INTRODUCTION

AdaMatch unifies SSL, UDA, and SSDA with shared hyperparameters while addressing domain shift and improving accuracy across these settings. It achieves state-of-the-art DomainNet results, strong SSDA gains with few target labels, and supports lower-resolution evaluation and ablation analysis.

  • SSL, UDA, and SSDA differ in their labeled and unlabeled data configurations, with UDA and SSDA additionally assuming domain shift unlike SSL.Table 1 summarizes the relations among these settings.Source: §1 · p. 1, Table 1 · p. 2
  • Unlabeled data is practically valuable because labeling is expensive, while shifted target data are common and exploiting them can substantially improve accuracy.The introduction argues that SSDA deserves equal attention because it reflects a realistic practical scenario despite receiving less attention than SSL and UDA.Source: §1 · p. 1
  • AdaMatch unifies semi-supervised learning, unsupervised domain adaptation, and semi-supervised domain adaptation using the same hyperparameters across datasets and tasks.It extends FixMatch with domain-shift-aware batch-normalization statistics, an adaptive pseudo-label confidence threshold, and modified distribution alignment.Source: §1 · p. 1
  • 28.7% UDA accuracy without pre-training and 33.4% with pre-training establish AdaMatch as state-of-the-art on DomainNet, with an 11.1% same-code-base gain.The introduction also reports that AdaMatch surpasses UDA by 6.1% with one target label per class and by 13.6% with five labels.Source: §1 · p. 2
  • AdaMatch reports minimal accuracy loss on a smaller 64 × 64 DomainNet and includes extensive ablations of its distinguishing modifications.The reduced-resolution evaluation avoids bias from selecting only easier or harder dataset pairs.Source: §1 · p. 2

2 RELATED WORK

Prior work studies UDA through domain-discrepancy reduction, SSL through unlabeled-data exploitation, and SSDA through limited labeled target-domain data. AdaMatch is positioned against these approaches while emphasizing evaluation without transfer learning, despite standard benchmarks often using ImageNet-pretrained models.

Unsupervised Domain Adaptation (UDA)

  • UDA methods reduce source-target representation discrepancy using distributional measures, correlation alignment, adversarial discriminators, or multiple classifiers, with MCD achieving state-of-the-art performance.Examples include maximum mean discrepancy, correlation alignment, domain-adversarial networks, and maximum classifier discrepancy (MCD).Source: §2 · p. 2

Semi-Supervised Learning (SSL)

  • SSL has advanced through consistency, augmentation, mixing, and pseudo-labeling methods, but relatively few SSL techniques have been applied to UDA.NoisyStudent uses ImageNet and JFT data but does not leverage distribution shift during training and evaluates only on ImageNet.Source: §2 · p. 2

Semi-Supervised Domain Adaptation (SSDA)

  • SSDA methods often use separate domain models with regularization, whereas MME works in both UDA and SSDA but requires a pre-trained network to perform well.Separate-model approaches are difficult to adapt when labeled data exists only in one domain.Source: §2 · p. 2

Transfer Learning

  • Transfer learning initializes models with weights from larger datasets and is standard on benchmarks such as Office-31, PACS, and DomainNet, but may be less general when pre-training data are unavailable.The paper therefore mainly evaluates methods without transfer learning, while also reporting transfer-learning results for historical comparison and compatibility.Source: §2 · pp. 2–3

3 ADAMATCH

AdaMatch unifies UDA, SSL, and SSDA by combining a FixMatch-style backbone with three techniques designed for source–target distribution shift. It uses random logit interpolation, distribution alignment, and a relative confidence threshold within a shared training procedure.

3 ADAMATCH

  • AdaMatch targets UDA, SSL, and SSDA using labeled source data and unlabeled target data, with the same classification task and class space across domains.For SSL, the source and target domains are the same; SSDA additionally supplies labeled target examples.Source: §3 · p. 3

3.1 METHOD DESCRIPTION

  • AdaMatch builds on FixMatch while introducing random logit interpolation, relative confidence thresholding, and modified distribution alignment to address source–target distribution differences.The algorithm applies weak and strong augmentations, uses source and mixed-domain batch-normalization statistics, and computes source and target loss terms.Source: §3.1 · pp. 3, 5
  • Distribution alignment rectifies weak target predictions toward the source output distribution, or a known target distribution, before selecting pseudo-labels for strongly augmented target images.When the target distribution is unknown, the source distribution is used as an approximation, though mismatch between domains can degrade this approximation.Source: §3.1 · pp. 4–5
  • Random logit interpolation trains on randomly selected points between source-only and source–target logits, encouraging alignment across domains without evaluating many interpolation points.The interpolation uses an independently sampled random factor for each logit and avoids the computational cost of dividing the interval into many segments.Source: §3.1 · p. 4
  • The relative confidence threshold scales a user-provided threshold by the mean source top-1 confidence, compensating for dataset- and domain-dependent calibration differences.Target predictions exceeding this relative threshold are retained through a binary mask for the target loss.Source: §3.1 · p. 5

3.2 HYPER-PARAMETERS

  • AdaMatch uses only two hyperparameters: τ = 0.9 and uratio = 3 in all experiments, where uratio sets the unlabeled target batch size relative to the labeled batch.The unlabeled batch therefore has size nTU = nSL · uratio.Source: §3.2 · p. 6

3.3 EXTENSION TO SSL AND SSDA

  • AdaMatch extends directly to SSL by substituting labeled and unlabeled same-domain data, and to SSDA by concatenating labeled target data with labeled source data.In SSDA, the effective uratio is implicitly halved because the labeled batch doubles in size.Source: §3.3 · p. 6

4 EXPERIMENTAL SETUP

AdaMatch is evaluated across SSL, UDA, and SSDA on DigitFive and DomainNet, using comparisons with established baselines and primarily training from scratch. The setup spans accessible DomainNet resolutions, task-specific architectures and training choices, and checkpoint-based accuracy aggregation.

Benchmarks

  • AdaMatch is evaluated on SSL, UDA, and SSDA using the DigitFive and DomainNet visual domain-adaptation benchmarks, alongside existing methods.DigitFive experiments and ablations used one V100 GPU; other experiments used one TPU.Source: §4 · p. 6

Datasets

  • DigitFive contains five image domains, while DomainNet contains six domains spanning 345 object categories; models are generally trained from scratch.Images are resized to 32×32 and converted to RGB; USPS and MNIST receive additional preprocessing.Source: §4 · p. 6

Democratic research

  • DomainNet experiments use both 64×64 and 224×224 images to improve accessibility for limited-compute researchers while testing performance across resolutions.The authors report that 64×64 AdaMatch can outperform prior DomainNet state of the art evaluated at 224×224.Source: §4 · p. 6

Network and training hyperparameters

  • Architectures vary by resolution: ResNetV2-101 for 224×224, WRN-34-2 for 64×64, and WRN-28-2 for 32×32, with shared core training settings.The setup specifies confidence-threshold selection, coarse hyperparameter tuning, resolution-scaled weight decay, and 32M-image DigitFive versus 8M-image DomainNet training.Source: §4 · p. 6

Baselines

Evaluation

  • Final accuracy is the median over the last ten checkpoints for each source-target pair, with summary tables reporting average final accuracy across target datasets.Most results omit ImageNet pre-training, but pre-trained AdaMatch, FixMatch+, and MCD are also evaluated for UDA in Section 5.1.1.Source: §4 · pp. 6–7

5 RESULTS

Across UDA, SSL, and SSDA, AdaMatch matches or surpasses competing methods, with strongest gains when labeled and unlabeled data come from different distributions. Performance generally improves with more target labels, while UDA results show benefits from accounting for distribution shifts and early stopping with pre-training.

5.1 UNSUPERVISED DOMAIN ADAPTATION (UDA)

  • 28.7% versus 23.9%: AdaMatch achieves the highest average UDA accuracy on DomainNet224, outperforming NoisyStudent and all other compared algorithms.Table 2 covers DigitFive, DomainNet64, and DomainNet224; AdaMatch outperforms all baselines across these benchmarks.Source: Table 2 · p. 7, §5.1 · p. 7

5.1.1 UDA WITH PRE-TRAINING

  • 35.6%: pre-trained AdaMatch reaches substantially higher DomainNet224 accuracy with oracle early stopping, versus 28.2% after 8 million images and 22.3% for pre-trained MCD.Early stopping is necessary for the best pre-trained AdaMatch results, which peak before the standard 8-million-image endpoint.Source: Table 3 · p. 7, §5.1.1 · pp. 7–8

5.2 SEMI-SUPERVISED LEARNING (SSL)

  • AdaMatch achieves state-of-the-art SSL performance on DomainNet224 and competitive performance with FixMatch+ on DigitFive and DomainNet64.Accuracy generally increases with more target labels, and AdaMatch’s performance gap over FixMatch+ widens on DomainNet224.Source: Figure 2 · p. 8, §5.2 · p. 8
  • AdaMatch improves more over FixMatch+ in SSDA and UDA than SSL, consistent with its random logit interpolation and relative confidence thresholding addressing distribution shifts.The additional components improve accuracy more when labeled and unlabeled data are drawn from different distributions.Source: §5.2 · p. 8

5.3 SEMI-SUPERVISED DOMAIN ADAPTATION (SSDA)

  • AdaMatch outperforms every compared method on SSDA for DomainNet64 and DomainNet224, with its lead increasing in the very low-label regime.Average target accuracy generally increases as the number of target labels grows for all methods.Source: Figure 3 · p. 8, §5.3 · p. 9

6 ABLATION STUDY

AdaMatch’s ablation study on DomainNet evaluates component importance and hyperparameter sensitivity across six domain-adaptation protocols. All proposed components contribute to performance, while high confidence thresholds, larger unlabeled-to-labeled ratios, and moderate L2 weight decay yield stronger or reliable accuracy.

  • The analysis uses DomainNet with 64×64 inputs and reports average accuracy across six domain-adaptation protocols.Source: §6 · p. 9

Exclude-One-Out Analysis

  • All three AdaMatch components—random logit interpolation, adaptive confidence thresholding, and distribution alignment—contribute to its success when excluded individually.Source: §6 · p. 9

Sensitivity Analysis

  • Higher confidence thresholds are required, larger uratio improves accuracy at greater computational cost, and L2 weight decay performs reliably between 0.0001 and 0.001.uratio is the ratio of unlabeled to labeled data within a mini-batch.Source: §6 · p. 9, Figure 4 · p. 9

7 CONCLUSION

AdaMatch unifies UDA, SSL, and SSDA, improving accuracy under domain shifts using unlabeled or limited labeled target-domain data. Despite outperforming prior work, substantial room remains for improving out-of-distribution performance.

7 CONCLUSION

  • AdaMatch unifies UDA, SSL, and SSDA while improving accuracy when unlabeled or limited labeled data from a shifted domain are available.The method is designed to boost accuracy on domain shifts using unlabeled data from the new domain.Source: §7 · p. 9
  • AdaMatch shows that SSL algorithms can be applied out-of-the-box to domain adaptation, while accounting for distribution shift improves upon SSL methods.The conclusion identifies adapting new SSL advances to domain adaptation as a promising direction.Source: §7 · p. 9
  • Although AdaMatch outperforms prior work by a large margin, substantial room remains to improve out-of-distribution shifts, including through more efficient unlabeled-data use or better shift modeling.The conclusion leaves open which direction matters more for future progress.Source: §7 · p. 10

ETHICS STATEMENT

AdaMatch targets robust domain-shift adaptation with limited labeled and abundant unlabeled data, while acknowledging fairness risks from self-training confirmation bias and calling for further investigation.

ETHICS STATEMENT

  • AdaMatch supports data-efficient machine learning for domain shifts, but self-training confirmation bias may undermine fairness across major and minor classes.Distribution alignment partially addresses this concern, yet deeper study of fairness and robustness in data-efficient ML remains necessary.Source: ETHICS STATEMENT · p. 10

A MOTIVATION AND ILLUSTRATING EXAMPLES · B BASELINEBN

This section motivates AdaMatch’s components for distribution shift through their effects on logits, class-prediction distributions, and pseudo-label selection, then defines the BaselineBN loss computation. Illustrative examples show how distribution alignment and relative confidence thresholding operate under differing prediction confidence and class frequencies.

A MOTIVATION AND ILLUSTRATING EXAMPLES

  • Random logit makes batch statistics more representative of both domains and implicitly aligns source and target domains in logit space.Source: A MOTIVATION AND ILLUSTRATING EXAMPLES · p. 14
  • Distribution alignment constrains class-prediction distributions toward the true distribution and otherwise provides no incentive to match target predictions to the source distribution.Source: A MOTIVATION AND ILLUSTRATING EXAMPLES · p. 14
  • Relative confidence thresholding compensates for poor out-of-distribution calibration by using target data as pseudo-labels as often as appropriate.Models can be poorly calibrated on out-of-distribution data, causing confidence thresholds to vary in effectiveness across datasets.Source: A MOTIVATION AND ILLUSTRATING EXAMPLES · p. 14
  • Distribution alignment changes the weakly augmented unlabeled target domain’s predicted class distribution from {0.6, 0.4} to {0.3, 0.7}, making the second class roughly twice as likely.The example source class-frequency distribution is {0.4, 0.6}, while weakly augmented source predictions have empirical distribution {0.3, 0.7}.Source: A MOTIVATION AND ILLUSTRATING EXAMPLES · p. 14
  • 0.9 × 0.7 = 0.63: relative thresholding lowers the default threshold using average labeled confidence, capturing more unlabeled examples when 0.9 would exclude almost all data.Source: A MOTIVATION AND ILLUSTRATING EXAMPLES · p. 14
  • When labeled confidence averages 1.0, relative and default thresholds coincide, making relative confidence thresholding a generalization of confidence thresholding.This is typical for CIFAR-10, where most or all labeled training examples reach softmax confidence 1.0.Source: A MOTIVATION AND ILLUSTRATING EXAMPLES · p. 14

B BASELINEBN

  • BaselineBN concatenates source and target into one batch but computes loss only on labeled source logits under weak and strong augmentations.It uses the same model and hyper-parameters as the referenced setup, changing only the loss computation.Source: B BASELINEBN · p. 14

C UDA MODEL EVAL CURVES WITH AND WITHOUT PRE-TRAINED WEIGHTS

On DomainNet224 Sketch→Clipart, Figure 5 compares AdaMatch and MCD with and without ImageNet-pretrained weights, showing that pretraining makes AdaMatch benefit significantly from early stopping. Across dataset pairs, a consistent 2-million-image early-stopping rule yields a 4.7% average advantage for pretrained AdaMatch over randomly initialized AdaMatch.

C UDA MODEL EVAL CURVES WITH AND WITHOUT PRE-TRAINED WEIGHTS

  • 4.7%: Across all dataset pairs, AdaMatch with pretraining and early stopping outperforms randomly initialized AdaMatch under a consistent 2-million-image stopping rule.The comparison averages target accuracy across dataset pairs; Figure 5 shows the early-stopping effect on a randomly selected Sketch→Clipart pair.Source: C UDA MODEL EVAL CURVES WITH AND… · p. 15, Figure 5 · p. 15
  • AdaMatch benefits significantly from early stopping, with a similar pattern observed across other dataset pairs.The plotted experiment uses a randomly selected DomainNet224 Sketch→Clipart pair.Source: C UDA MODEL EVAL CURVES WITH AND… · p. 15
  • Figure 5 compares target accuracy versus training images for AdaMatch and MCD with and without pretraining on DomainNet224 Sketch→Clipart.Pretrained runs use weight decay 0 and ImageNet-pretrained ResNet101 initialization.Source: Figure 5 · p. 15, C UDA MODEL EVAL CURVES WITH AND… · p. 15

D SUMMARY RESULTS · D.1 SSL · D.2 SSDA

The summary evaluates AdaMatch on SSL and SSDA across DomainNet224, DomainNet64, and DigitFive while varying available target labels. It reports state-of-the-art or competitive accuracy, with average target accuracy generally increasing as target labels increase.

D SUMMARY RESULTS

D.1 SSL

  • AdaMatch achieves state-of-the-art or competitive accuracy for SSL across DomainNet224, DomainNet64, and DigitFive.The evaluation varies available labels and reports average target accuracy across all source→target pairs.Source: Table 5 · p. 16
  • SSL average target accuracy generally increases as the number of target labels increases.The SSL summary varies the number of available labels and reports average accuracy on the target datasets.Source: Table 5 · p. 16

D.2 SSDA

  • AdaMatch achieves state-of-the-art or competitive accuracy for SSDA across DomainNet224, DomainNet64, and DigitFive.The evaluation varies target labels and reports average target accuracy across all source→target pairs.Source: Table 6 · p. 16
  • SSDA average target accuracy generally increases as the number of target labels increases.The SSDA summary evaluates the same three benchmarks while varying the number of target labels.Source: Table 6 · p. 16

E INDIVIDUAL DATASET RESULTS

This section presents individual dataset-pair results across the DigitFive, DomainNet64, and DomainNet224 benchmarks, covering UDA, SSL, and SSDA settings. It provides tables documenting AdaMatch alongside comparison methods, with particular coverage of the SSL and SSDA evaluations.

SSDA

SSL

SSL results

  • Individual SSL results are tabulated for DomainNet224 using MCD and NoisyStudent.The captions specify dataset-pair results for DomainNet224 in the SSL setting, but no numerical values are supplied.Source: Table 101 · p. 40, Table 103 · p. 41
  • Individual SSL results are tabulated for DomainNet64 using AdaMatch, BaselineBN, FixMatch+, MCD, and NoisyStudent.The supplied captions identify the methods and dataset, without exposing cell values or winners.Source: Table 107 · p. 42, Table 111 · p. 43, Table 115 · p. 44

UDA results

Related papers

Loading 2106.04732v2…