PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation

David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, Alex Kurakin

arXiv:2106.04732v2cs.LGcs.AIcs.CV

TL;DR

Machine learning models lose accuracy under domain shift, while existing SSL, UDA, and SSDA methods address different data configurations. AdaMatch unifies these settings in one method and, across vision classification tasks, matches or exceeds state-of-the-art performance with shared hyperparameters. The method achieves strong DomainNet UDA accuracy and further gains when target labels are available.

  • Problem

    Models suffer accuracy drops on out-of-distribution data, while SSL, UDA, and SSDA use fragmented approaches for different labeled and unlabeled data configurations.

  • Method

    AdaMatch extends FixMatch with domain-shift-aware batch normalization, adaptive pseudo-label confidence thresholding, and modified distribution alignment to unify UDA, SSL, and SSDA.

  • Results

    AdaMatch matches or exceeds state-of-the-art performance across UDA, SSL, and SSDA using the same hyperparameters, including 28.7% UDA accuracy without pre-training on DomainNet.

  • Takeaways & Limitations

    Access to unlabeled or limited labeled data from a shifted domain can greatly improve accuracy, and SSL algorithms can be adapted to domain adaptation.

  • Takeaways & Limitations

    Confirmation bias in self-training may threaten fairness across major and minor classes, and further investigation of fairness and robustness is needed.

Abstract

from arXiv · show

We extend semi-supervised learning to the problem of domain adaptation to learn significantly higher-accuracy models that train on one data distribution and test on a different one. With the goal of generality, we introduce AdaMatch, a method that unifies the tasks of unsupervised domain adaptation (UDA), semi-supervised learning (SSL), and semi-supervised domain adaptation (SSDA). In an extensive experimental study, we compare its behavior with respective state-of-the-art techniques from SSL, SSDA, and UDA on vision classification tasks. We find AdaMatch either matches or significantly exceeds the state-of-the-art in each case using the same hyper-parameters regardless of the dataset or task. For example, AdaMatch nearly doubles the accuracy compared to that of the prior state-of-the-art on the UDA task for DomainNet and even exceeds the accuracy of the prior state-of-the-art obtained with pre-training by 6.4% when AdaMatch is trained completely from scratch. Furthermore, by providing AdaMatch with just one labeled example per class from the target domain (i.e., the SSDA setting), we increase the target accuracy by an additional 6.1%, and with 5 labeled examples, by 13.6%.

1 INTRODUCTION

SSL, UDA, and SSDA use different labeled and unlabeled data configurations, with UDA and SSDA additionally assuming domain shift. AdaMatch unifies these settings with one hyperparameter set and reports strong DomainNet results.

  • UDA, SSL, and SSDA differ in their labeled and unlabeled data configurations, and UDA and SSDA assume domain shift unlike SSL.
  • Labeling data is expensive, while distribution shifts make models less robust when training and test distributions differ.
  • AdaMatch unifies UDA, SSL, and SSDA using the same hyperparameters regardless of dataset or task.It addresses shifted batch-normalization statistics, adapts the pseudo-label confidence threshold, and modifies distribution alignment.
  • 28.7% accuracy is reported for UDA without pre-training and 33.4% with pre-training on DomainNet, an 11.1% increase on the same code base.

2 RELATED WORK

Related work treats SSL, UDA, and SSDA as distinct settings with different assumptions and techniques. Transfer learning commonly boosts benchmark performance, but this paper emphasizes non-transfer learning as more general.

  • UDA trains on labeled source and unlabeled target data to achieve high target-domain accuracy while reducing representation discrepancy between domains.
  • SSL uses labeled and unlabeled portions of one dataset, but relatively few SSL methods have been applied to UDA problems.
  • SSDA assumes labeled data from multiple domains, and early methods used separate domain models with regularization constraints.
  • Transfer learning initializes models with weights learned on a larger dataset to compensate for limited labeled source data and improve performance.
  • The paper mainly evaluates without transfer learning because this setting is considered more general, while also reporting transfer-learning results for comparison.

3 ADAMATCH

AdaMatch unifies UDA, SSL, and SSDA by extending FixMatch with mechanisms that address source–target distribution differences. Its pipeline combines augmented inputs, shared and source-only batch statistics, pseudo-label correction, confidence masking, and source-plus-target loss.

  • 3 ADAMATCH: AdaMatch targets UDA, SSL, and SSDA with one hyperparameter set across datasets and tasks.The labeled data come from a source domain and unlabeled data from a target domain; for SSL, the domains coincide.
  • 3 ADAMATCH: Three additions address distribution shift: random logit interpolation, a relative confidence threshold, and modified distribution alignment.The algorithm retains FixMatch as its backbone.
  • 3 ADAMATCH: Two augmentations are applied to each image, and logits are computed using both source-only and combined source–target batches.The combined batch shares batch-normalization statistics, while the source-only pass provides source-specific statistics.
  • 3 ADAMATCH: Random logit interpolation samples points between source logits computed under different batch statistics, encouraging low loss across the connecting segment.Random sampling approximates evaluating many interpolation points without multiplying computation by the number of segments.
  • 3 ADAMATCH: Distribution alignment rectifies target pseudo-labels toward a desired class distribution, using the source output distribution when the target distribution is unknown.The method estimates the source distribution from model outputs rather than true labels and normalizes the rectified pseudo-labels.
  • 3 ADAMATCH: The target loss uses confidence-masked cross-entropy on strongly augmented images, while the total loss sums source and target terms.A relative threshold is based on mean top-1 confidence on weakly augmented source data; the unlabeled loss weight uses warmup.

4 EXPERIMENTAL SETUP

The experiments evaluate AdaMatch across SSL, UDA, and SSDA on DigitFive and DomainNet, using established baselines and both pretrained and from-scratch settings. DomainNet is also tested at 64×64 resolution to reduce computational barriers while preserving competitive performance.

  • 4 EXPERIMENTAL SETUP: AdaMatch is evaluated on SSL, UDA, and SSDA using the DigitFive and DomainNet visual domain-adaptation benchmarks.DigitFive experiments and ablations used one V100 GPU, while other experiments used one TPU.
  • 4 EXPERIMENTAL SETUP: DomainNet contains six domains and 345 object categories, and the study generally trains models from scratch rather than using prior pretraining protocols.Digit-Five contains five domains: USPS, MNIST, MNIST-M, SVHN, and synthetic numbers.
  • 4 EXPERIMENTAL SETUP: 64×64 DomainNet images make experiments more accessible to researchers with limited compute, while AdaMatch still outperforms prior 224×224 state of the art.The comparison uses two DomainNet resolutions: 64×64 and 224×224.
  • 4 EXPERIMENTAL SETUP: The study uses ResNetV2-101 at 224×224, WRN-34-2 at 64×64, and WRN-28-2 at 32×32, with limited coarse tuning of training settings.Confidence thresholds and the unlabeled-to-labeled batch ratio were selected under stated resource constraints.
  • 4 EXPERIMENTAL SETUP: Baselines include MCD, FixMatch+, NoisyStudent, and BaselineBN, covering established UDA and SSL methods plus supervised source training with target batch-normalization updates.MCD is included as the cited single-source DomainNet UDA state of the art, while FixMatch+ is included as an SSL state-of-the-art comparator.
  • 4 EXPERIMENTAL SETUP: Final accuracy is the median over the last ten checkpoints for each source–target pair, with summary tables averaging final accuracy across target datasets.The study also reports ImageNet-pretrained AdaMatch, FixMatch+, and MCD for comparison with prior UDA work.

5 RESULTS

Across UDA, SSL, and SSDA evaluations, AdaMatch achieves leading or competitive target accuracy, with benefits from pre-training, early stopping, and target labels. Its components also contribute individually, while performance generally improves as more labels are provided.

  • 5.1 UNSUPERVISED DOMAIN ADAPTATION (UDA): AdaMatch outperforms all compared algorithms on UDA across DigitFive, DomainNet64, and DomainNet224 benchmarks.The reported metric is average target accuracy across all source→target pairs.
  • 5.1 UNSUPERVISED DOMAIN ADAPTATION (UDA): 28.7% average target accuracy on DomainNet224 exceeds NoisyStudent’s 23.9% in the UDA comparison.This is the largest reported improvement among the evaluated dataset sizes.
  • 5.1.1 UDA WITH PRE-TRAINING: 35.6% target accuracy is achievable for pre-trained AdaMatch with oracle-selected early stopping, versus 28.2% for the standard 8M-image protocol.The pre-training comparison reports 28.2% for AdaMatch and 22.3% for MCD under the standard protocol; the 35.6% result uses oracle stopping.
  • 5.2 SEMI-SUPERVISED LEARNING (SSL): AdaMatch achieves state-of-the-art SSL performance on DomainNet224 and competitive performance on DigitFive and DomainNet64.Increasing the number of target labels generally improves accuracy, and the AdaMatch–FixMatch+ gap widens on DomainNet224.
  • 5.3 SEMI-SUPERVISED DOMAIN ADAPTATION (SSDA): Each AdaMatch component contributes to performance in the six-protocol DomainNet ablation study.The ablation excludes one component at a time.
  • 5.3 SEMI-SUPERVISED DOMAIN ADAPTATION (SSDA): AdaMatch outperforms all other SSDA algorithms on DomainNet64 and DomainNet224, with a larger lead in very low-label regimes.Average target accuracy generally increases as the number of target labels increases for all methods.

6 ABLATION STUDY

The ablation study evaluates AdaMatch components and hyperparameter sensitivity on DomainNet across six domain-adaptation protocols. Each proposed component contributes, while higher unlabeled-to-labeled ratios improve accuracy at greater computational cost.

  • AdaMatch is evaluated on 64×64 DomainNet images using average accuracy across 6 domain adaptation protocols.
  • Exclude-One-Out Analysis: Each AdaMatch component contributes to its overall success in exclude-one-out experiments.The analyzed components are random logit interpolation, adaptive confidence thresholding, and distribution alignment.
  • Sensitivity Analysis: Higher uratio improves accuracy but increases computation.uratio is the ratio of unlabeled to labeled data within a mini-batch, not across the full training course.
  • Sensitivity Analysis: AdaMatch requires a high confidence threshold, while L2 weight decay performs reliably between 0.0001 and 0.001.

7 CONCLUSION

AdaMatch addresses accuracy drops under distribution shift by applying one method across UDA, SSL, and SSDA. The authors report strong performance while noting that substantial room for improvement remains on out-of-distribution shifts.

  • AdaMatch is designed to improve accuracy on domain shifts using unlabeled data from the new domain and can perform across UDA, SSL, and SSDA.
  • Applying SSL algorithms out-of-the-box to domain adaptation is possible, and accounting for distribution shift can improve upon SSL methods.
  • Despite outperforming prior work by a large margin, AdaMatch leaves an even larger margin for improvement on out-of-distribution shifts.

ETHICS STATEMENT

The ethics statement highlights fairness and robustness concerns arising from self-training with limited labeled data and abundant unlabeled data.

  • Confirmation bias in self-training could threaten fairness across major and minor classes.The authors state that distribution alignment partially resolves this issue, while calling for deeper investigation of fairness and robustness.
  • Further investigation is needed into the fairness and robustness of data-efficient machine-learning algorithms.

A MOTIVATION AND ILLUSTRATING EXAMPLES

AdaMatch combines several mechanisms intended to address distribution shift and calibrate pseudo-label use. The examples illustrate how distribution alignment changes predicted class frequencies and how relative thresholds adapt confidence filtering.

  • Random logit interpolation makes batch statistics more representative of both domains and constrains source and target logits to align.
  • Distribution alignment constrains predicted class frequencies to better match the true distribution.Without it, the model has no incentive to match the target distribution to the source distribution.
  • Relative confidence thresholding addresses poor out-of-distribution calibration so target data is used as pseudo-labels as often as appropriate.
  • Distribution alignment changes target class frequencies from {0.6, 0.4} to {0.3, 0.7} in the illustrative example.
  • When labeled-data confidence averages 0.7, a default threshold τ = 0.9 would exclude almost all unlabeled data, motivating relative thresholding.
  • When labeled examples reach average confidence 1.0, the relative threshold equals the default threshold.

C UDA MODEL EVAL CURVES WITH AND WITHOUT PRE-TRAINED WEIGHTS

Figure 5 compares AdaMatch and MCD with and without pre-training on Sketch→Clipart from DomainNet224. AdaMatch benefits from early stopping, and pre-training with a consistent stopping rule improves average accuracy over random initialization.

  • Figure 5 compares target accuracy against training images for AdaMatch and MCD, with and without pre-training, on Sketch→Clipart.The experiment uses a randomly selected DomainNet224 dataset pair.
  • AdaMatch benefits significantly from early stopping in the pre-trained setting.The passage reports a similar pattern for other dataset pairs, although those results are not plotted.
  • 4.7%: AdaMatch with pre-training and early stopping outperforms randomly initialized AdaMatch on average across all dataset pairs.The comparison uses a consistent early stopping rule of 2 million images.

D SUMMARY RESULTS

The summary tables report AdaMatch results for SSL and SSDA across DomainNet224, DomainNet64, and DigitFive while varying the number of target labels. Average target accuracy generally increases as more target labels become available.

  • SSL summary: Table 5 reports SSL average target accuracy across source→target pairs on DomainNet224, DomainNet64, and DigitFive.The evaluation varies the number of available labels.
  • Summary pattern: Average target accuracy generally increases as the number of target labels increases in both SSL and SSDA.The corresponding accuracy-versus-label-count plots are Figure 2 for SSL and Figure 3 for SSDA.
  • SSDA summary: Table 6 reports SSDA average target accuracy across source→target pairs on DomainNet224, DomainNet64, and DigitFive.The evaluation varies the number of target labels.

E INDIVIDUAL DATASET RESULTS

The paper evaluates methods on individual dataset pairs from DigitFive, DomainNet64, and DomainNet224 across UDA, SSL, and SSDA. The supplied individual-result entries identify results by benchmark, task, and method.

  • Evaluation scope: Individual dataset-pair evaluations cover DigitFive, DomainNet64, and DomainNet224 across UDA, SSL, and SSDA.The evaluation is organized by individual source→target dataset pairs.
  • Compared methods: The individual-result entries include AdaMatch, BaselineBN, FixMatch+, MCD, and NoisyStudent.These methods appear in the supplied tables for DigitFive, DomainNet224, and DomainNet64.
  • SSDA results: The supplied entries separately report AdaMatch results for DigitFive, DomainNet224, and DomainNet64 in the SSDA setting.The entries are labeled as individual results for each dataset pair.
Loading 2106.04732v2…