PaperlayerView exact revision on arXiv

Source-linked AI summary

Extending the WILDS Benchmark for Unsupervised Adaptation

Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, Percy Liang

arXiv:2112.05090v2cs.LGcs.AIcs.CVstat.ML

TL;DR

Real-world distribution shifts are common, but existing benchmarks with unlabeled data do not cover the breadth of deployment scenarios. Wilds 2.0 adds curated unlabeled data to eight Wilds datasets and benchmarks adaptation methods; many do not outperform supervised training, with clear successes limited to two image-classification datasets.

  • Problem

    Existing unlabeled-data benchmarks do not reflect the breadth of realistic distribution shifts, despite unlabeled data often being more available than labels.

    Source: §1 · p. 3

  • Method

    Wilds 2.0 adds curated unlabeled data to 8 of 10 Wilds datasets and evaluates domain-invariant, self-training, and self-supervised methods.

    Source: §1 · p. 4

  • Results

    Many methods did not outperform supervised training, and clear successes were limited to Camelyon17-wilds and FMoW-wilds image classification.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The results underscore the importance of developing and evaluating unlabeled-data methods across a wider variety of real-world shifts.

    Source: §1 · p. 4

    Unsupervised adaptation adds hyperparameters that can significantly affect OOD performance, while labeled target data is unavailable for hyperparameter selection.

    Source: Hyperparameter tuning · p. 12

Abstract

from arXiv · show

Machine learning systems deployed in the wild are often trained on a source distribution but deployed on a different target distribution. Unlabeled data can be a powerful point of leverage for mitigating these distribution shifts, as it is frequently much more available than labeled data and can often be obtained from distributions beyond the source distribution as well. However, existing distribution shift benchmarks with unlabeled data do not reflect the breadth of scenarios that arise in real-world applications. In this work, we present the WILDS 2.0 update, which extends 8 of the 10 datasets in the WILDS benchmark of distribution shifts to include curated unlabeled data that would be realistically obtainable in deployment. These datasets span a wide range of applications (from histology to wildlife conservation), tasks (classification, regression, and detection), and modalities (photos, satellite images, microscope slides, text, molecular graphs). The update maintains consistency with the original WILDS benchmark by using identical labeled training, validation, and test sets, as well as the evaluation metrics. On these datasets, we systematically benchmark state-of-the-art methods that leverage unlabeled data, including domain-invariant, self-training, and self-supervised methods, and show that their success on WILDS is limited. To facilitate method development and evaluation, we provide an open-source package that automates data loading and contains all of the model architectures and methods used in this paper. Code and leaderboards are available at https://wilds.stanford.edu.

1. Introduction

Wilds 2.0 adds realistically obtainable unlabeled data to a broad real-world distribution-shift benchmark and evaluates methods that use it. Results are mixed: many methods do not beat supervised training, with clear successes limited to two image-classification datasets.

Motivation

  • The update addresses a gap: existing unlabeled-data evaluations do not represent the breadth of realistic distribution shifts encountered in the wild.Many prior evaluations focus on stylized or synthetic image shifts and digit-dataset variants, while methods successful on one shift need not generalize to others.Source: §1 · p. 3

Benchmark update

  • Wilds 2.0 adds curated, realistically obtainable unlabeled data to 8 of 10 Wilds datasets while retaining the original labeled data and evaluation metrics.The update adds 14.5 million unlabeled examples and expands each included dataset by 3–13×; Py150-wilds and RxRx1-wilds are omitted because unlabeled data is unavailable or inapplicable.Source: §1 · pp. 3–5, Figure 2 · p. 4

Evaluation

  • The benchmark evaluates domain-invariant representation learning, self-training, and self-supervised pre-training methods using unlabeled data.The standardized protocol assesses representative methods from these three categories.Source: §1 · p. 4

Results

  • Many methods did not outperform supervised training, and clear successes were limited to Camelyon17-wilds and FMoW-wilds image classification.Successful methods relied heavily on data augmentation; the same methods were unsuccessful on image regression and detection, while continued language-model pre-training did not help on text datasets.Source: §1 · p. 4

Resources

  • The open-source Wilds package includes unlabeled-data loaders, benchmarked methods, and experiment-replication scripts.The authors hope easier evaluation across the datasets will accelerate development of methods for real-world distribution shifts.Source: §1 · p. 5

Benchmark consistency

  • Wilds 2.0 keeps Wilds 1.0’s labeled data and metrics, so results remain part of the overall Wilds benchmark.Future results should specify what kind of unlabeled data, if any, was used.Source: §1 · p. 5

2. Comparison with existing unsupervised adaptation benchmarks

Existing unsupervised-adaptation benchmarks often center on particular shifts or modalities, whereas Wilds spans real-world distribution shifts across diverse tasks and modalities. Its text datasets also focus on subpopulation performance rather than adaptation to a wholly distinct domain.

Images

  • Image adaptation benchmarks commonly cover stylized, product, synthetic, or digit shifts, while Wilds covers real-world shifts across image types and tasks.Wilds includes satellite, microscope, agriculture, and camera-trap images, with classification, regression, and detection tasks.Source: §2 · p. 5

Text

  • Wilds text datasets target demographic or tail-population subgroups, unlike benchmarks centered on shifts between distinct textual sources.CivilComments-wilds focuses on demographic identities and Amazon-wilds on tail populations.Source: §2 · p. 5

Molecules

  • Although unlabeled molecules have been used for pre-training, standardized unsupervised-adaptation benchmarks for molecules have not been developed.Wilds 2.0 provides molecular-graph data as part of its broader benchmark.Source: §2 · p. 5

3. Problem setting

The problem setting draws data from domains, each associated with a distribution over inputs, predictions, and domain identifiers. Models train on labeled source data plus whichever types of unlabeled data are realistic for the application.

Domain-shift setting

  • Each domain corresponds to a distribution over input x, prediction y, and domain d.The setting follows the domain-shift formulation used in Wilds 1.0.Source: §3 · p. 5

Training data

  • Models use labeled source-domain data and unlabeled data from one or more domain types, depending on application realism.Labeled data is available from source, validation, and target domains; unlabeled data may come from one or more types.Source: §3 · p. 6, Table 1 · p. 6

Domain configurations

  • Domain relationships vary by application: all four domain types may be disjoint, or target domains may be subsets of source domains.The latter can represent evaluation on particular demographic subpopulations within a heterogeneous dataset.Source: §3 · p. 6

4. Datasets

Wilds 2.0 adds application-specific unlabeled data to eight datasets while preserving the labeled datasets and evaluation metrics from Wilds 1.0. The additions reflect realistic data availability across detection, molecular prediction, text, wildlife, histology, satellite classification, and poverty mapping.

iWildCam2020-wilds

  • iWildCam2020-wilds adds images from 3,215 extra camera traps, expanding camera-trap locations by 11× and examples by 5×.The added images capture diverse operating conditions for species classification across locations.Source: §4 · p. 6

Camelyon17-wilds

  • Camelyon17-wilds adds unlabeled patches from training and test hospitals, expanding tumor-classification data by 7.5×.Hospitals accumulate unlabeled slide images during normal operation, and differences such as staining protocols can affect color distributions.Source: §4 · p. 6

FMoW-wilds

  • FMoW-wilds adds satellite imagery across regions and train and test timeframes, expanding land-use classification data by 3.5×.The task seeks to generalize from pre-2013 training data to imagery taken after 2013.Source: §4 · pp. 6–7

PovertyMap-wilds

  • PovertyMap-wilds adds satellite imagery from train and test countries, expanding poverty-mapping data by 14×.Predicting real-valued asset wealth indices across countries uses labels that are expensive to collect because they require nationally representative field surveys.Source: §4 · p. 7

GlobalWheat-wilds

  • GlobalWheat-wilds adds unlabeled field images from training, test, and extra acquisition sessions, expanding wheat-head detection data by 10×.Sessions vary by location, time, and sensor, and densely packed overlapping wheat heads make labels costly and sensitive to annotators.Source: §4 · p. 7

OGB-MolPCBA

  • OGB-MolPCBA adds 5 million unlabeled molecules from source and target scaffolds, expanding molecular-property prediction data by 12.5×.Biological-activity labels require expensive laboratory experiments, while unlabeled molecular structures are available from large chemical databases.Source: §4 · p. 7

CivilComments-wilds

  • CivilComments-wilds adds unannotated online comments, expanding toxicity-classification data by 4.5× for demographic subpopulation shifts.The task is to classify whether comments are toxic.Source: §4 · p. 7

Amazon-wilds

  • Amazon-wilds adds unlabeled reviews from test and extra reviewers, expanding sentiment-classification data by 7.5×.The task predicts review star ratings and evaluates performance across new reviewers.Source: §4 · pp. 7–8

5. Algorithms

The paper benchmarks representative approaches from three families for using unlabeled data: domain-invariant representations, self-training, and self-supervision. Self-training uses model-generated pseudolabels, while self-supervision pre-trains on auxiliary tasks before fine-tuning on labeled source data.

Algorithms

  • The evaluation covers domain-invariant representation learning, self-training, and self-supervised pre-training as three approaches to leveraging unlabeled data.Self-supervised models are pre-trained on unlabeled data and then fine-tuned on labeled source data.Source: §5 · p. 8, Domain-invariant methods · p. 8

Domain-invariant methods

  • Domain-invariant methods penalize differences between learned source and target representations.DANN penalizes representations that an auxiliary classifier can use to distinguish source from target examples; CORAL penalizes differences in feature means and covariances.Source: Domain-invariant methods · p. 8

Self-training

  • Self-training assigns pseudolabels to unlabeled examples and trains on them, often with consistency regularization across augmented views.Pseudo-Label updates pseudolabels each batch; FixMatch uses weak and strong views, while Noisy Student alternates teacher and student phases.Source: Domain-invariant methods · p. 8

Self-supervision

  • Self-supervised methods train on auxiliary proxy tasks, including reconstruction and contrastive learning, to learn representations from unlabeled data.The paper evaluates vision and language approaches, including SwAV and masked language modeling.Source: Domain-invariant methods · p. 8

6. Experiments

The experiments apply Wilds models, splits, and metrics to benchmark unlabeled-data methods using a consistent data protocol and OOD validation for tuning. Results are mixed: most methods do not improve on ERM, with outcomes varying across datasets and tasks.

6.1 Setup

  • Experiments use Wilds default models, labeled training and test sets, and evaluation metrics, with one selected type of unlabeled data per dataset.Where possible, the experiments use target-domain unlabeled data; iWildCam2020-wilds and CivilComments-wilds instead use extra domains.Source: §6.1 · p. 9
  • Hyperparameters are tuned separately per dataset using labeled OOD validation data, and image classification and regression use RandAugment and Cutout.The OOD validation distribution differs from both training and OOD test distributions; no ID validation data or augmentation for the remaining datasets is used.Source: Hyperparameters · p. 9

6.2 Results

  • Most methods do not improve over ERM on Wilds despite access to unlabeled data and careful hyperparameter tuning.The authors report that their implementations, except CORAL, outperform ERM on DomainNet’s real-to-sketch shift, a standard unsupervised adaptation benchmark.Source: §6.2 · p. 9, Table 2 · p. 10
  • On Camelyon17-wilds and FMoW-wilds, Noisy Student and SwAV pre-training improve OOD performance and reduce replicate variability, while CORAL, DANN, Pseudo-Label, and FixMatch underperform ERM.On Camelyon17-wilds, ERM with augmentation achieves 82.0% accuracy versus 70.8% for vanilla ERM; the table reports ID and OOD performance across methods and datasets.Source: §6.2 · p. 9, Table 2 · p. 10
  • On iWildCam2020-wilds, no benchmarked method improves OOD performance over ERM despite 4× as many unlabeled images from extra domains and no target-domain images.Fully-labeled ERM models using ground-truth labels for the unlabeled data significantly outperform other methods on FMoW-wilds and iWildCam2020-wilds.Source: §6.2 · p. 11
  • All applicable methods underperform ERM on PovertyMap-wilds image regression, GlobalWheat-wilds image detection, and OGB-MolPCBA molecule classification.GlobalWheat-wilds methods relying on augmentation were not evaluated or were modified because standard augmentation can change detection labels; augmentation techniques are not well-developed for molecular graphs.Source: §6.2 · p. 11
  • On CivilComments-wilds and Amazon-wilds, benchmarked methods perform similarly to ERM, and continued masked-language-model pretraining does not improve target performance.These evaluations measure subpopulation performance; the authors suggest the pretraining corpus may be more similar to these datasets than to the text studied in Gururangan et al. (2020).Source: §6.2 · p. 11

7. Discussion

The discussion identifies constraints on adaptation methods and outlines directions for future work. It highlights augmentation, hyperparameter tuning, broader pre-training, and domain annotations as relevant considerations.

The role of data augmentation

  • Image-specific augmentation techniques restrict method generality across modalities and even across image types beyond photos.The authors suggest developing augmentation techniques for other applications and modalities.Source: §7 · pp. 11–12

Hyperparameter tuning

  • Unsupervised adaptation methods have more hyperparameters than standard supervised methods, and those choices can significantly affect OOD performance.Without labeled target data for tuning, the authors suggest improved methods might use unlabeled target data or labeled and unlabeled OOD validation data.Source: Hyperparameter tuning · p. 12

Pre-training on broader unlabeled data

  • The broad pre-training techniques evaluated appear insufficient for Wilds, although broad pre-training may help in conjunction with other techniques.Many models used ImageNet-pretrained weights or BERT derivatives; the paper proposes combining broad and curated unlabeled data as a direction for future work.Source: Hyperparameter tuning · p. 12

Leveraging domain annotations and task-specific structure

  • OOD robustness is ill-posed in general because models cannot be robust to arbitrary distribution shifts.The authors identify domain annotations and structured metadata available for labeled and unlabeled Wilds data as a potential resource for adaptation.Source: Hyperparameter tuning · p. 12

Ethics statement

The datasets address real-world discrimination and bias, while the authors caution that evaluation metrics may miss relevant aspects of these issues. They also note privacy considerations for satellite datasets.

Ethics statement

  • The datasets are anonymized and curated from public data sources whose licenses allow public release.Source: Ethics statement · p. 12
  • Several datasets concern real-world discrimination and bias, including demographic disparities in CivilComments and limited satellite-data availability in some countries and regions.The authors caution that standard models may perform poorly on these subpopulations and learned representations might be biased.Source: Ethics statement · p. 12
  • Evaluation metrics may miss relevant facets of discrimination and bias: toxicity annotations can themselves be biased, and FMoW region choices might obscure lower-performing sub-regions.Source: Ethics statement · p. 12
  • FMoW images have lower resolution than other public satellite data, while PovertyMap location metadata is noised to protect privacy.Source: Ethics statement · p. 12

Reproducibility statement

The authors provide public access to the datasets and experiment materials, including trained checkpoints and hyperparameters. Appendices document datasets, methods, augmentations, experimental protocols, and related details.

Reproducibility statement

  • Datasets, code, replication scripts, trained model checkpoints, results, and exact hyperparameters are publicly available.Source: Reproducibility statement · p. 13
  • Appendices A–D describe updated datasets and data processing, method implementations, augmentations, and experimental protocols including hyperparameter selection.Source: Reproducibility statement · p. 13
  • Appendices E–G cover DomainNet experiments, fully labeled ERM experiments, and a Wilds data-loader code example.Source: Reproducibility statement · p. 13

Appendix A. Additional dataset details

This appendix documents how unlabeled data are added and organized across WILDS datasets, including their tasks, data conditions, and scope. It also describes benchmark baselines and self-training methods, their implementations, and key method choices or limitations.

Appendix A. Additional dataset details

  • iWildCam2020-wilds adds 819,120 unlabeled images from 3,215 WCS camera traps disjoint from the labeled dataset, supporting species classification across camera-trap deployments.The task has 182 animal classes; the extra traps represent a newly deployed sensor network.Source: §A.1 · p. 26

A.2 Camelyon17-wilds

  • Camelyon17-wilds adds 2,999,307 unlabeled patches from 90 additional WSIs per hospital, with no WSI overlap between labeled and unlabeled data.The binary task classifies whether a 96×96 histology patch contains tumor in its central 32×32 region, with generalization to new hospitals.Source: §A.2 · pp. 27–28

A.3 FMoW-wilds

  • FMoW-wilds adds 340,469 unlabeled satellite images to 141,696 labeled images, using data from the original dataset across the same year-and-region domains.The 62-class task evaluates average and worst-region accuracy for images after 2013; the added data includes a sequestered split and images from training locations in target and validation timeframes.Source: §A.3 · pp. 28–29

A.4 PovertyMap-wilds

  • PovertyMap-wilds adds 261,396 unlabeled LandSat images from the same 23 countries to support cross-country prediction of real-valued asset wealth.Its evaluation averages Pearson correlation across five folds and considers the lower correlation for urban or rural subpopulations.Source: §A.4 · p. 30

A.5 GlobalWheat-wilds

  • GlobalWheat-wilds supplements bounding-box-labeled wheat-head images with unlabeled data from existing and extra acquisition sessions, increasing spatial coverage, temporal resolution, and protocol-matched diversity.Sessions vary by location, time, and sensor, and the task is to localize wheat heads in overhead field images.Source: §A.5 · p. 31

A.6 OGB-MolPCBA

  • OGB-MolPCBA adds PubChem molecules to a 128-assay binary classification task, where sparse assay labels contrast with abundant unlabeled molecules.The task tests generalization to new molecular scaffold structures and evaluates performance using Average Precision.Source: §A.6 · p. 32
  • For OGB-MolPCBA, each unlabeled molecule is assigned to the existing scaffold group of the labeled molecule with highest Jaccard similarity between 1024-dimensional Morgan fingerprints; no extra domains are added.The target split contains 43,793 scaffold groups; the largest scaffolds are in the source split and the smallest in the target split.Source: §A.6 · pp. 32, 35
  • Although OGB-MolPCBA’s labeled and unlabeled molecules all come from PubChem, possible selection bias in choosing labeled molecules could create an undocumented distribution shift between them.The authors identify unlabeled data as useful for domain adaptation, self-training, pre-training, and semi-supervised learning, and point to graph-agnostic and graph-specific approaches as future directions.Source: §A.6 · p. 35

A.7 CivilComments-wilds

  • CivilComments-wilds classifies comments as toxic or non-toxic and evaluates worst-group accuracy across 16 groups defined by demographic identity mentions.Of the unlabeled comments, 1,427,848 (92%) come from the same articles as labeled comments, including 140,082 from articles represented in the labeled test split.Source: §A.7 · pp. 35–36
  • CivilComments-wilds adds 1,551,515 unlabeled comments to 448,000 examples annotated for toxicity and demographic identities, while withholding the added comments’ toxicity labels and identity annotations.Because the added comments lack identity annotations, they cannot be grouped like labeled examples and are treated as coming from extra domains, though they may mention any number of identities.Source: §A.7 · p. 36

A.8 Amazon-wilds

  • Amazon-wilds adds 3,462,668 unlabeled reviews to 539,502 labeled reviews, including data from both labeled reviewers and extra reviewers; reviewers are distinct across splits and each has at least 75 reviews.The task is 1-to-5-star sentiment classification for new reviewers, evaluated by accuracy on the user at the 10th percentile.Source: §A.8 · p. 37

B.1 Empirical risk minimization (ERM)

  • ERM ignores unlabeled data and minimizes labeled loss, with strong data augmentation evaluated on applicable image datasets and class-balanced labeled batches used for CivilComments-wilds.For other datasets, labeled batches are sampled uniformly at random.Source: §B.1 · p. 39

B.2 Domain-invariant methods

  • Domain-invariant methods combine labeled loss with a weighted penalty that encourages similar labeled and unlabeled representations; CORAL penalizes feature-statistic differences, while DANN uses a domain discriminator.Both are omitted for GlobalWheat-wilds because they do not straightforwardly support detection, and domain-invariant methods are omitted for CivilComments-wilds because its labeled and unlabeled data come from the same distribution; the implementations distinguish labeled from unlabeled batches rather than using domain annotations.Source: §B.2 · pp. 39–42

B.3 Self-training methods

  • The methods use hard pseudolabels for classification, train on augmented unlabeled examples, and jointly optimize unlabeled and ERM labeled losses.The pseudolabel-generating function differs by algorithm, while a step-dependent λ(t) balances the two losses.Source: §B.3 · pp. 42–43
  • Pseudo-Label (Lee (2013)) regenerates pseudolabels each batch, applies strong augmentation and confidence thresholding, and ramps the unlabeled-loss weight to λ by 40% of training steps.Low-confidence examples contribute zero loss; evaluation excludes PovertyMap-wilds because hard pseudolabeling does not straightforwardly apply to regression.Source: §B.3 · pp. 43–44
  • For fairer comparisons, the implementations strongly augment labeled examples in Pseudo-Label and FixMatch, unlike the original FixMatch choice of weak augmentation.The paper also adds strong augmentation to Pseudo-Label, whereas Lee (2013) used no data augmentation.Source: §B.3 · pp. 43–44
  • FixMatch (Sohn et al. (2020)) generates pseudolabels from weakly augmented inputs, applies thresholded loss to strongly augmented views, and keeps λ constant.It is evaluated on iWildCam2020-wilds, Camelyon17-wilds, PovertyMap-wilds, and FMoW-wilds because the method requires image augmentations.Source: §B.3 · p. 44
  • Noisy Student (Xie et al. (2020)) fixes pseudolabels between training phases, cycles through two teacher-student iterations, and trains students with strong data augmentation and dropout (p = 0.5).It uses weakly augmented inputs for teacher pseudolabels, sets λ(t) = 1, and does not use confidence thresholding or stochastic depth.Source: §B.3 · p. 45
  • Noisy Student is evaluated on all datasets except CivilComments-wilds and Amazon-wilds, and omits data-augmentation noise on GlobalWheat-wilds and OGB-MolPCBA.Source: §B.3 · p. 45

B.4 Self-supervision methods

The paper pre-trains image and text models on unlabeled data using self-supervised objectives, then fine-tunes them on labeled data. Its augmentation choices vary by modality, and performance is assessed in both in-distribution and out-of-distribution settings where available.

B.4 Self-supervision methods

  • Self-supervised pre-training uses SwAV for images and masked language modeling for text, followed by ERM fine-tuning on labeled data.The procedure starts from each dataset’s standard model initialization, pre-trains on unlabeled data, then uses a new classifier head for fine-tuning.Source: §B.4 · p. 46
  • SwAV is evaluated on four image datasets because its training relies on image augmentations defined only for image datasets.The datasets are iWildCam2020-wilds, Camelyon17-wilds, PovertyMap-wilds, and FMoW-wilds.Source: §B.4 · p. 47
  • Masked language modeling masks text tokens and trains the model to predict the originals, encouraging syntactic, semantic, and world knowledge representations.The paper evaluates MLM on CivilComments-wilds and Amazon-wilds.Source: §B.4 · p. 47

Appendix C. Data augmentation

  • Image-classification augmentations use horizontal flips as weak augmentation and add RandAugment and Cutout for strong augmentation; SwAV uses its own crop-and-color transformation pipeline.PovertyMap-wilds instead uses affine transformations, RGB color jitter, and Cutout across channels; augmentation is omitted for GlobalWheat-wilds, OGB-MolPCBA, and the text datasets.Source: Appendix C · p. 48

D.1 In-distribution vs. out-of-distribution performance

  • The paper reports in-distribution and out-of-distribution metrics, except for OGB-MolPCBA, which has no separate in-distribution test set.The in-distribution comparisons are train-to-train for four image and satellite datasets, and average comparisons for CivilComments-wilds and Amazon-wilds.Source: §D.1 · p. 49

D.2 Model architectures

The appendix details the models, training and tuning protocols, and compute used for the benchmark, then checks the protocol on one DomainNet source-target pair. On DomainNet real-to-sketch, all benchmarked methods except CORAL improve over ERM, while the setup uses labeled target data for validation.

D.2 Model architectures

  • For all experiments, datasets use the same models as WILDS 1.0: iWildCam2020-wilds uses ResNet-50; Camelyon17-wilds and FMoW-wilds use DenseNet-121; PovertyMap-wilds uses multi-spectral ResNet-18; GlobalWheat-wilds uses Faster-RCNN.Source: §D.2 · p. 49

D.3 Batch sizes and batch normalization

  • Batch sizes are set to the maximum fitting in 12GB of GPU memory, with larger effective batches for most unlabeled-data methods.Batch-normalization handling differs by method: some jointly normalize labeled and unlabeled examples, while pre-training methods process them separately.Source: §D.3 · p. 49, Table 13 · p. 50

D.4 Hyperparameter tuning

  • The authors tune algorithms separately per dataset using ten sampled configurations and select hyperparameters by out-of-distribution validation performance.Pre-training hyperparameters are not tuned for computational reasons, though the resulting models’ fine-tuning is tuned as usual.Source: §D.4 · p. 50

D.5 Algorithm-specific hyperparameters

  • Algorithm-specific tuning includes CORAL and DANN penalty weights, self-training confidence thresholds, and default, untuned SwAV and masked-language-model settings.The number of epochs varies with the unlabeled-to-labeled batch ratio; the appendix lists dataset-level epoch counts.Source: §D.5 · pp. 50–51, Table 14 · p. 51

D.6 Compute infrastructure

  • SwAV pre-training takes about three days on four V100 GPUs per dataset, while masked-language-model pre-training takes about three days on one GPU.Other algorithms take less than a day on a V100.Source: §D.6 · p. 51

Appendix E. Experiments on DomainNet

  • All benchmarked methods except CORAL improve over ERM on DomainNet real-to-sketch.The table reports in-distribution and out-of-distribution test performance, including weak-augmentation variants for Pseudo-Label and FixMatch.Source: Appendix E · p. 52, Table 15 · p. 53
  • The DomainNet consistency check uses only the real-to-sketch pair, chosen as a common setting and to match ImageNet pretraining with realistic source photographs.The authors state that one pair was sufficient for their goal of checking consistency with prior results.Source: §E.1 · p. 52

E.1 Setup

  • DomainNet training uses labeled source examples and unlabeled target examples, but validation uses labeled target training examples, which overestimates performance in true adaptation.Source test examples serve as the in-distribution test set and target test examples as the out-of-distribution test set.Source: §E.1 · p. 52

E.2 Results

The real-to-sketch experiments found gains from strong augmentation and most unsupervised adaptation methods, while comparisons with prior implementations varied. The appendix also estimates a self-training upper bound using hidden labels and describes its training setup.

E.2 Results

  • On real →sketch, all unsupervised adaptation methods except CORAL improved over ERM, and strong augmentation raised ERM performance from 34.9% to 35.9%.Strong augmentation for labeled examples also improved Pseudo-Label and FixMatch.Source: §E.2 · p. 53
  • FixMatch reached 50.2% versus 45.3% in Zhang et al. (2021), while DANN was similar at 39.4% versus 40.0% and Pseudo-Label was lower at 36.1% versus 40.6%.The authors attribute part of FixMatch’s difference to strong labeled-data augmentation, which increased performance by 0.9%, and believe Pseudo-Label’s difference reflects hyperparameter-tuning variation.Source: §E.2 · p. 53
  • Noisy Student reached 39.7% versus 32.6% in Berthelot et al. (2021), a difference the authors expected because their models used ImageNet pretraining while the prior work trained from scratch.Source: §E.2 · p. 53
  • On DomainNet, SwAV pre-training improved over ERM but CORAL did not; the paper found no comparable prior results for either method on real →sketch.Source: §E.2 · p. 53

Appendix F. Fully-labeled ERM experimental details

  • Fully labeled ERM experiments use ground-truth labels hidden from the main experiments to estimate an informal upper bound for self-training with perfect pseudolabels.For Amazon-wilds, the experiments combine labeled source examples with labeled versions of unlabeled target examples, while evaluation remains on labeled target splits.Source: Appendix F · p. 54

F.1 Hyperparameters

  • The fully labeled experiments pool labeled and unlabeled minibatches at dataset-size ratios, normalize them separately for batch normalization, and tune learning rate with 10 random hyperparameter samples.They early-stop on OOD validation performance; epoch counts were adjusted to account for batch-size ratios, and other experimental details followed the paper’s standard setup.Source: §F.1 · p. 54

F.2 Dataset-specific details

The dataset-specific appendix reports which unlabeled splits were used and, for Amazon-wilds, compares target-domain data with a larger extra split. The additional examples did not improve the reported Amazon accuracies.

Amazon-wilds

  • Amazon-wilds extra data did not improve performance: target data yielded 73.6 (± 0.1) average accuracy and 56.4 (± 0.8) 10th percentile accuracy, versus 73.1 (± 0.1) and 54.7 (± 0.0) with extra data.The extra split contained 2,927,841 examples, about 10× the 268,761 target examples.Source: §F.2 · p. 54

CivilComments-wilds

  • CivilComments-wilds used 1,551,515 unlabeled extra examples and class-balanced labeled and unlabeled batches to account for label imbalance.Source: §F.2 · p. 55

iWildCam2020-wilds

  • iWildCam2020-wilds excluded 108,452 extra examples whose species labels were absent from the labeled training and test sets, leaving 710,668 for training.Fully labeled ERM required twice as many epochs as the other unlabeled methods, so its compute allocation was doubled.Source: §F.2 · p. 55

FMoW-wilds

  • FMoW-wilds used 173,208 examples from the unlabeled target split.Source: §F.2 · p. 55

Appendix G. Using the Wilds library with unlabeled data

The Wilds library adds unlabeled-data loaders for eight datasets while retaining compatibility with the Wilds 1.0 APIs. It also provides implementations and scripts for standardized evaluation of several adaptation and pretraining methods.

Appendix G. Using the Wilds library with unlabeled data

  • The library adds data loaders for each of the eight datasets with unlabeled data, compatible with Wilds 1.0 APIs so unlabeled examples can be accessed similarly to labeled examples.Source: Appendix G · p. 56
  • Figure 3 gives an example of loading both labeled and unlabeled data.Source: Figure 3 · p. 56
  • The package implements CORAL, DANN, Pseudo-Label, FixMatch, and Noisy Student, and includes scripts for masked language model and SwAV pretraining.Automated downloading and shared Wilds interfaces support standardized evaluation and extension of these algorithms.Source: Appendix G · p. 56
Loading 2112.05090v2…