WILDS: A Benchmark of in-the-Wild Distribution Shifts

• Wilds targets domain generalization and subpopulation shift, where training and test domains differ or test populations change in proportion, degrading deployed ML accuracy.Examples include hospitals, camera traps, and under-represented demographic groups (Zech et al., 2018; Beery et al., 2018; Buolamwini and Gebru, 2018; Koenecke et al., 2020).Source: §1 · p. 4, Figure 1 · p. 4
• Existing ML datasets underrepresent real-world shifts, and robustness to synthetic corruptions or dataset shifts does not reliably transfer across deployment settings.Some methods can even harm robustness on real-world satellite imagery (Taori et al., 2020; Djolonga et al., 2020; Xie et al., 2020).Source: §1 · p. 5
• Wilds contributes a curated benchmark of 10 datasets with realistic train/test splits and metrics covering applications from animal recognition and tumor identification to code completion.The datasets include natural shifts across cameras, hospitals, molecular scaffolds, experiments, demographics, countries, time periods, users, and codebases (Beery et al., 2020a; Bandi et al., 2018; Raychev et al., 2016; Lu et al., 2021).Source: §1 · p. 5
• Dataset selection required substantial performance drops under shift, real-world relevance established with domain experts, and non-trivial but solvable tasks with domain metadata.Training data span multiple domains, enabling invariant representation learning or uniformly strong performance across subpopulations (Sun and Saenko, 2016; Ganin et al., 2016; Hu et al., 2018; Sagawa et al., 2020a).Source: §1 · pp. 5–6
• The benchmark collectively varies tasks, modalities, dataset sizes, and numbers of domains, while excluding considered datasets whose shifts caused no appreciable performance drop.This design supports evaluation across a broad range of real-world distribution shifts.Source: §1 · p. 6
• Wilds standardizes datasets and provides automated loading, evaluation, default models, reproducible baselines, and a public leaderboard to accelerate research on robust ML.Code, leaderboards, and updates are available at https://wilds.stanford.edu.Source: §1 · p. 6

Earlier datasets

• Earlier datasets covered shifts in tasks including tagging, sentiment analysis, land-cover classification, object recognition, and flow cytometry but are less widely used partly because they are smaller.Source: §2 · p. 6

Recent benchmarks

• Recent benchmarks commonly induce shifts through synthetic image corruptions, altered backgrounds or colors, cross-dataset generalization, scale changes, or disjoint subclasses.Examples include ImageNet-C, Backgrounds Challenge, Waterbirds, Colored MNIST, PACS, DeepFashion Remixed, and BREEDS.Source: §2 · pp. 6–7

Benchmark utility

• Existing benchmarks enable clean analysis of isolated shifts and robustness evaluation for widely used models.They support controlled experiments, such as testing background effects or whether Gaussian blur improves performance on blurry images.Source: §2 · p. 7

Limitations and motivation

• Robustness to synthetic shifts need not transfer to real-world deployment shifts, making these datasets alone insufficient for developing and evaluating robust training methods.WILDS therefore complements existing benchmarks by curating natural distribution shifts across diverse data modalities and applications.Source: §2 · p. 7

3. Problem settings

• WILDS models data as mixtures of domain-specific distributions, with training and test distributions potentially differing in their domain mixture weights.Training exposes domain annotations, while testing provides either x or (x, d), depending on the application.Source: §3 · p. 7

3.1 Domain generalization (Figure 1-Top)

• Domain generalization evaluates average test error on domains disjoint from training domains, which are typically similar across settings such as hospitals or camera traps.Formally, Dtrain ∩ Dtest = ∅.Source: §3.1 · p. 7

3.2 Subpopulation shift (Figure 1-Bottom)

• Subpopulation shift evaluates worst-domain error when all test domains were seen during training but their proportions change, often without test-time domain identities.Formally, Dtest ⊆ Dtrain while qtest ≠ qtrain.Source: §3.2 · pp. 7–8

3.3 Hybrid settings

• Hybrid settings combine unseen-domain generalization with reweighted known domains, as in FMoW across time and geographical regions.FMoW uses temporal train/test separation while retaining regions across splits and seeking performance across all regions.Source: §3.3 · p. 8

4. Wilds datasets

• WILDS frames each dataset as a domain generalization, subpopulation-shift, or hybrid problem reflecting real-world challenges, using “-wilds” names for modified datasets.Additional distribution-shift context is provided in Appendix E.Source: §4 · p. 8

4.1.1–4.1.2

• iWildCam2020-wilds classifies 182 animal species across disjoint camera traps using macro F1, while Camelyon17-wilds predicts tumor patches across five hospitals using average accuracy.The datasets exploit substantial training-domain variation while evaluating generalization to unseen cameras or hospitals.Source: §4.1.1 · p. 8, Figure 4 · p. 9, §4.1.2 · p. 10

4.1.3 RxRx1-wilds: Genetic perturbation classification across experimental batches

• RxRx1-wilds classifies 1,139 genetic treatments across disjoint experimental batches, with 33 training batches used to assess normalization of batch effects while preserving biological signal.Batch-specific technical artifacts confound high-throughput biological measurements.Source: §4.1.3 · pp. 10–11, Figure 5 · p. 10

4.1.4 OGB-MolPCBA: Molecular property prediction across different scaffolds

• OGB-MolPCBA predicts 128 biochemical assay labels for molecules with disjoint scaffolds, averaging Average Precision across assays and training on molecules from over 40,000 scaffolds.The split targets generalization to structurally different molecules.Source: §4.1.4 · p. 11, Figure 6 · p. 11

4.1.5 GlobalWheat-wilds: Wheat head detection across regions of the world

• GlobalWheat-wilds detects wheat heads across countries unseen during training, evaluating accuracy at a fixed IoU threshold averaged across acquisition sessions.The dataset contains 12 countries, with training images from 18 sessions across five countries.Source: §4.1.5 · p. 12, Figure 7 · p. 12

4.2.1 CivilComments-wilds: Toxicity classification across demographic identities

• CivilComments-wilds classifies toxic comments from disjoint articles using the lowest true positive/negative rate across eight overlapping demographic groups.The design targets spurious demographic correlations and permits models to use domain annotations.Source: §4.2.1 · p. 13, Figure 8 · p. 13

4.3.1 FMoW-wilds: Land use classification across different regions and years

• FMoW-wilds evaluates 62-category satellite land-use classification on future years and the worst geographic region, combining temporal domain generalization with regional subpopulation shift.Training uses data before 2013, validation uses 2013–2015, and testing uses 2016 and later.Source: §4.3.1 · pp. 13–14, Figure 9 · p. 14

4.3.2–4.3.4

• PovertyMap-wilds predicts asset wealth across countries absent from training using the lower urban/rural correlation, while Amazon-wilds and Py150-wilds test consistency on unseen users and repositories.Amazon-wilds uses 10th-percentile per-user accuracy with 5,008 training users; Py150-wilds evaluates class and method-token accuracy across over 5,300 repositories.Source: §4.3.2 · p. 14, §4.3.3 · pp. 15–16, §4.3.4 · p. 16

5. Performance drops from distribution shifts

• Wilds measures distribution-shift drops as the difference between OOD and ID performance of standard models trained by empirical risk minimization.OOD performance uses official train/test splits, while ID settings are constructed case by case because practical constraints limit available test-domain data.Source: §5 · p. 17

5.1 In-distribution performance should be measured on P test, not P train

• ID performance should be evaluated on P test when feasible because fixing the evaluation distribution isolates the distribution-shift effect; otherwise, Wilds uses standard evaluation on P train.Evaluating ID models on P train can confound distribution shift with differences in task difficulty between P train and P test.Source: §5.1 · p. 17

5.2 Types of in-distribution settings

• Wilds considers test-to-test, mixed-to-test, train-to-train, average, and random-split ID settings, selecting feasible comparisons that balance distributional control against data requirements.Mixed-to-test controls the evaluation distribution while reducing test-domain data needs, whereas train-to-train and random-split comparisons are more practical but do not control evaluation distribution.Source: §5.2 · pp. 18–19

5.3 Model selection

• Standard architectures use an OOD validation set for model selection, and reusing its hyperparameters for ID evaluation slightly underestimates the reported ID-OOD gaps.The protocols use ResNet or DenseNet for images, DistilBERT for text, GIN for graphs, and Faster-RCNN for detection.Source: §5.3 · p. 19

5.4 Results

• OOD performance is consistently and substantially lower than corresponding ID performance across every Wilds dataset, while mixed-to-test results show the gap is not caused by intrinsically harder OOD test sets.Models trained on mixtures of ID and OOD data can simultaneously achieve high performance on both distributions where mixed-to-test comparisons are feasible.Source: §5.4 · p. 19

6.4 Results

• 70.0% OOD accuracy for Group DRO on CivilComments-wilds versus 56.0% for ERM, but below ERM’s 92.2% ID accuracy; other baselines generally fail to improve.CORAL and IRM also perform well on CivilComments-wilds because the implementation heuristically upsamples the minority domain; all other datasets involve domain generalization, where the baselines fail.Source: Table 2 · p. 20, §6.4 · p. 22

6.1 Domain generalization baselines

• CORAL penalizes domain differences in feature means and covariances, whereas IRM penalizes domains having different optimal linear classifiers.Both methods add invariance-oriented penalties to the ERM objective; CORAL is associated with domain adaptation, while IRM builds on invariant-predictor work.Source: §6.1 · p. 21

6.2 Subpopulation shift baselines

• Group DRO explicitly minimizes worst-case domain loss to perform well across relevant subpopulations.It uses distributionally robust optimization and builds on the maximin approach developed in Meinshausen and Bühlmann (2015).Source: §6.2 · p. 21

6.3 Setup

• The study trains CORAL, IRM, and Group DRO on every dataset, also applying each method beyond its originally targeted shift setting.Group DRO is used for domain generalization, while CORAL and IRM are tested on subpopulation shifts; OOD validation data tune early stopping and CORAL/IRM penalty weights.Source: §6.3 · p. 21

6.4 Results

• The results show that training models robust to distribution shifts in the wild remains a significant open challenge.The authors suggest future progress may come from methods designed for domain generalization and from exploiting WILDS’s diverse domains and domain metadata.Source: §6.4 · p. 22

7.1 Underspecification

• Excluding Amazon-wilds and CivilComments-wilds, OOD performance variance is comparable to ID variance: 2.6% versus 2.0% average standard deviation.The two exceptions use smaller OOD subpopulations, making OOD performance naturally more variable.Source: §7.1 · p. 22

7.2 Model selection with in-distribution versus out-of-distribution validation sets

• OOD validation generally provides comparable or higher OOD performance than ID validation for selecting baseline models.Wilds provides multiple training domains and OOD validation sets, whereas ID validation sets are not used for model selection or official evaluation; differences from DomainBed may reflect dataset domains and comparison procedures.Source: §7.2 · pp. 22–23

7.3 The compounding effects of multiple distribution shifts

• Combined shifts can exacerbate performance drops: unseen domains widen subpopulation gaps in PovertyMap-wilds and FMoW-wilds, although Amazon-wilds shows no such compounding.In FMoW-wilds, regional subpopulation differences emerge only when the time shift is also considered, motivating evaluation of practical shift combinations rather than isolated shifts.Source: §7.3 · p. 23

8.1 Algorithmic fairness

• Algorithmic fairness benchmarks are often small, feature-poor, weakly disparate, unrealistic, or difficult to release, motivating datasets with sensitive attributes, consequential tasks, and sufficient scale.COMPAS has only a few thousand rows, while logistic regression performs comparably to a commercial black-box algorithm; high-stakes domains also restrict sharing.Source: §8.1 · pp. 23–24
• Fairness experiments found large racial and geographic performance disparities that persisted after training on affected groups, suggesting noisier data from biased policing rather than distribution shift.Black and Hispanic pedestrians comprised over 90% of stops, contrasting with the usual minority-underperformance setting.Source: §8.1 · p. 24

8.2 Medicine and healthcare

• Healthcare benchmarks must distinguish distribution-shift effects from disparities caused by access, comorbidities, biased outcomes, concept drift, and privacy restrictions.Medical shifts include demographic, hospital, and temporal variation, but concept drift and limited data sharing complicate benchmark construction.Source: §8.2 · pp. 24–25

8.3 Genomics

• Genomics models showed a significant in-distribution versus cross-cell-type performance gap because cell-type-specific protein complexes can change sequence-binding relationships.The authors excluded this dataset from WILDS because models failed to generalize across all cell types, and split difficulty varied substantially.Source: §8.3 · p. 26

8.4 Natural language and speech processing

• Speech and language systems exhibit subgroup and cross-language shifts, but available datasets often lack diverse accents, natural speech, or adequate low-resource-language coverage.Reviewed Amazon and Yelp user, category, and time shifts produced only modest performance drops.Source: §8.4 · pp. 26–27

8.5 Education

• Education systems face shifts from rater bias, essay length, prompts, and subgroup differences, yet standardized datasets remain scarce because of privacy and proprietary testing constraints.The authors identify future dataset construction as an active direction, including possible use of massive open online course data.Source: §8.5 · p. 27

8.6 Robotics

• BDD100K multi-label classification showed no substantial performance drops across user, weather, time, or location shifts, unlike prior detection and segmentation results.The authors speculate that detection and segmentation may be more vulnerable because they are harder than classification and regression tasks.Source: §8.6 · p. 28

8.7 Feedback loops

• WILDS excludes adaptive settings where models alter future data through feedback loops, including recommendations, dialogue, molecular optimization, decision systems, and fraud detection.Handling these model-dependent shifts remains an important area of ongoing work.Source: §8.7 · p. 28

9.1 General-purpose and specialized training algorithms

• Use Wilds’ standardized datasets without external data and default architectures to isolate algorithmic contributions; evaluate on all applicable datasets, while supporting both general-purpose and shift-specific methods.The benchmark’s primary leaderboard focuses on submissions following these guidelines, but significant gains on one shift type remain valuable even without broad improvements.Source: §9.1 · pp. 28–29

9.2 Methods beyond training algorithms

• Evaluate new model architectures and external pre-training on Wilds, with such submissions tracked separately from training-algorithm methods.These approaches are identified as additional promising directions for improving distributional robustness.Source: §9.2 · p. 29

9.3 Avoiding overfitting to the test distribution

• Use OOD validation sets for development and model selection, reserving OOD test sets for final evaluation to avoid overfitting, especially with few test domains.Camelyon17-wilds is highlighted as having only one OOD test hospital, and performance can vary substantially across domains.Source: §9.3 · p. 29

9.4 Reporting both ID and OOD performance

• Report both ID and OOD performance to distinguish gains from improved in-distribution accuracy from gains in robustness to distribution shift.Prior work found ID and OOD performance can be highly correlated across architectures and hyperparameters, and ID improvements may also improve OOD performance.Source: §9.4 · p. 29

9.5 Extensions to other problem settings

• Wilds can support other realistic settings that leverage additional information, such as unlabeled test data in unsupervised domain adaptation, inviting community contributions.The paper focuses on domain generalization and subpopulation shift, while identifying additional leverage as a promising route to improving OOD performance.Source: §9.5 · p. 29

10. Using the Wilds package

• The open-source PyTorch package provides a unified interface for downloading datasets, accessing splits, using flexible data loaders, and running standardized evaluations.It also includes utilities for domain annotations and other metadata needed by training algorithms.Source: §10 · pp. 29–30, Figure 13 · p. 30, Figure 15 · p. 31
• Grouper objects flexibly extract domain annotations and other groupings from dataset metadata for algorithms that use them.The grouping scheme can be specified through metadata-derived group annotations.Source: §10 · p. 30, Figure 14 · p. 30
• The package’s standardized evaluation method returns all metrics reported in the paper and on the leaderboard.Evaluation is automated for each dataset.Source: §10 · p. 30, Figure 15 · p. 31

Reproducibility

• The executable CodaLab release provides the exact commands, code, environment, data, and trained model weights used for the experiments.The WILDS package is also released as open-source software.Source: Reproducibility · p. 31

• Dataset realism comprises three independently variable components: realistic data, a realistic task and evaluation metric, and a realistic train/test distribution shift.Assessing realism is subtle, contextual, and often requires domain expertise.Source: Appendix A · p. 61
• Data are realistic when inputs and metadata plausibly reflect what would be collected and available in the application context.The appropriate standard can differ between well-equipped hospitals and clinics using older imaging technology.Source: Appendix A · p. 61
• Tasks and metrics are realistic when they match application goals, such as estimating wildlife populations and prioritizing rare-species classification over average accuracy.Species classification remains relevant but is less realistic than population estimation for the conservation application described.Source: Appendix A · p. 61
• Distribution shifts are realistic when their train/test splits could arise in deployment, such as training on some hospitals and testing on unseen hospitals rather than separating pedestrian images only by day and night.The latter split is less realistic because practical pedestrian-detection datasets would include both daytime and nighttime images.Source: Appendix A · p. 61
• WILDS fills benchmarks’ gap in realistic distribution shifts by selecting diverse tasks with realistic data and metrics, plus deployment-relevant shifts; it maps settings to dataset-specific challenges such as hospital generalization and underrepresented demographic groups.Domain generalization also fits settings with costly or continuously expanding domains, including hospitals, countries, users, and camera traps.Source: Appendix A · pp. 61–62

Appendix B. Prior work on ML benchmarks for distribution shifts

• Existing benchmarks induce shifts through synthetic transformations, constrained splits, or dataset combinations, whereas WILDS focuses on shifts occurring in the wild between real data distributions.The review restricts attention to publicly available datasets and notes that some realistic proprietary datasets cannot be used because of privacy or commercial constraints.Source: Appendix B · pp. 62–63
• Synthetic-to-real benchmarks can model realistic transfers when real data are difficult to acquire, but WILDS does not study these synthetic distribution shifts.Examples include SYNTHIA, StreetHazards, and synthetic-to-real transfer datasets used for detection, adaptation, and generalization.Source: Appendix B · p. 62

Appendix C. Potential extensions to other problem settings

• WILDS currently targets domain generalization and subpopulation shifts, while other domain-shift problem settings are identified as important directions for future benchmarking and community contributions.The broader framework varies by whether test domains are seen, domain annotations are available, and unlabeled test data can be accessed.Source: Appendix C · p. 63, §C.1 · pp. 63–64

C.1 Problem settings in domain shifts

• The four factors defining domain-shift settings determine which methods apply, and WILDS uses the configuration of unseen or seen domains and domain-information availability for its core settings.Domain annotations may be observed during training or testing, while varying amounts of unlabeled test data affect adaptation possibilities.Source: §C.1 · p. 64

C.2 Unsupervised domain adaptation

• Unsupervised domain adaptation assumes abundant unlabeled data from each test distribution and resources to train a separate model for each, unlike domain generalization.Some domain-generalization methods also apply to adaptation, while other methods require knowledge of the test distribution.Source: §C.2 · p. 64

C.3 Test-time adaptation

• Test-time adaptation instead updates a learned model using a small amount of unlabeled test data, making it more feasible when many potential test domains exist.This setting is computationally lighter because it typically needs only the model and limited test data rather than the training set and a large test pool.Source: §C.3 · p. 65

C.4 Selective prediction

• Selective prediction allows models to abstain below a confidence threshold, handing uncertain cases to human experts through probabilities, ensembles, or jointly learned abstention.Potential experts include pathologists, content moderators, and wildlife experts.Source: §C.4 · p. 65
• Selective prediction can improve accuracy under distribution shifts, but shifts make desired abstention rates difficult to maintain and cause confidence estimates to drift over time.This limits the reliability of abstention-based safeguards even when models remain accurate on points close to the training distribution.Source: §C.4 · p. 65

D.1 Model hyperparameters

• OOD-validation early stopping produced similar or slightly better OOD test performance than ID-validation hyperparameter selection, while potentially using domain metadata implicitly.OOD validation can incorporate metadata through metrics or domain-based split selection, yet large OOD–ID performance gaps remain.Source: §D.1 · pp. 65–66, Table 3 · p. 66
• Hyperparameters selected for ERM were generally reused for other baselines, with grid searches over penalty weights for CORAL and IRM and a fixed 0.01 Group DRO step size.The tested CORAL penalty weights were {0.1, 1, 10}, and the IRM weights were {1, 10, 100, 1000}.Source: §D.1 · p. 66

D.2 Replicates

• Results typically average three random-seed replicates with unbiased standard deviations, except for five-fold PovertyMap cross-validation, ten Camelyon17 replicates, and five CivilComments replicates.Source: §D.2 · p. 66

D.3 Baseline algorithms

• Classification models use cross-entropy, whereas PovertyMap-wilds regression uses mean-squared error; CORAL, IRM, and Group DRO implementations follow prior work.Source: §D.3 · p. 66
• CORAL and IRM use domain-balanced sampling and domain-specific minibatches, while Group DRO samples uniformly across domains and retains its worst-loss objective.The default minibatch allocation for CORAL and IRM is eight examples per domain, and uniform sampling improves Group DRO optimization stability.Source: §D.3 · pp. 66–67

Appendix E. Additional dataset details and results

• Appendix E expands each WILDS dataset’s motivation, task, shift, evaluation, baseline results, context, and modifications, using Appendix D’s protocol unless otherwise specified.Source: Appendix E · p. 67

E.1 iWildCam2020-wilds

• iWildCam2020-wilds studies species classification across new camera-trap deployments, where variation in illumination, viewpoint, background, vegetation, and animal frequencies causes poor generalization.The dataset is a variant of iWildCam 2020 (Beery et al., 2020a).Source: §E.1 · p. 67, §E.1.1 · p. 67

E.1.1 Setup

• The dataset comprises 203,029 images from 323 camera traps, partitioned by camera trap into training (129,809 images; 243 traps), OOD validation (14,961; 32), and OOD test (42,791; 48).Each image is associated with camera trap ID, sequence ID, and datetime; sequences group images from the same burst, but baseline models do not exploit this information and evaluation treats each image individually.Source: §E.1.1 · pp. 67–68

E.1.2 Baseline results

• iWildCam evaluation uses camera-trap location splits, with OOD differences in class balance and empty-image proportions causing macro F1 and average accuracy to vary across validation and test sets.The reported ID–OOD gap was assessed in a train-to-train comparison because many camera traps were split independently between training and test; test-to-test and mixed-to-test comparisons remain future work.Source: §E.1.2 · pp. 69–70
• 47.0 macro F1 on the ID test set fell to 31.0 on the OOD test set for ERM on iWildCam2020-wilds, while CORAL, IRM, Group DRO, and label reweighting did not improve upon ERM.The ID–OOD gap was large, and IRM performed especially poorly; the authors attribute this partly to possible negative bias in its penalty estimator for small domains.Source: §E.1.2 · pp. 69–70, Table 4 · p. 70

E.1.3 Broader context

• Across iWildCam camera traps, changes in illumination, camera angle, background, vegetation, color, and animal distributions likely encourage location-specific overfitting and performance drops.Related wildlife-monitoring settings face analogous sensor-specific backgrounds, species distributions, and rare-species data scarcity.Source: §E.1.2 · p. 70, §E.1.3 · p. 71

E.2.1 Setup

• Camelyon17-wilds tests binary tumor-region classification across hospitals using 450,000 patches from 50 whole-slide images spanning five Dutch hospitals.Inputs are 96x96 histopathology patches, with labels determined by tumor presence in the central 32x32 region; baseline algorithms do not use slide metadata.Source: §E.2 · p. 72, §E.2.1 · p. 72
• Camelyon17-wilds trains on three hospitals, validates OOD on a fourth, tests OOD on a visually distinctive fifth, and evaluates average patch accuracy on balanced splits.The dataset omits an ID test set because uniformly sampled labeled patches without labels for the remaining patches from the same slide are impractical; the test hospital choice can substantially affect performance.Source: §E.2.1 · p. 72, §E.2.4 · p. 75

E.2.2 Baseline results

• Camelyon17-wilds ERM accuracy showed a large ID–OOD gap and substantial test variability, including a 6.4% standard deviation across 10 random seeds, while CORAL, IRM, and Group DRO performed comparably or worse.OOD validation selection outperformed ID validation selection, but test accuracy remained much more variable than validation accuracy.Source: Table 5 · p. 73, §E.2.2 · pp. 73–74, Figure 18 · p. 74
• Adding one test-hospital slide to Camelyon17-wilds training produced significantly higher accuracy on the remaining test slides, indicating that the observed drop reflects distribution shift rather than only intrinsic example difficulty.The comparison used nine test slides and mixed in only one of the original ten slides, so it likely underestimates fully in-distribution performance.Source: Table 6 · p. 73, §E.2.2 · p. 74

E.2.3 Broader context

• Stain variation is a primary Camelyon17 source of hospital-level variation, but patient demographics, cancer subtypes, scanners, technicians, and other deployment factors can also shift medical data distributions.The benchmark is intended as a controlled testbed for general-purpose robustness across hospitals, while specialized stain augmentation and normalization remain effective in some contexts.Source: §E.2.1 · p. 73, §E.2.3 · p. 75

E.2.4 Additional details

• Camelyon17-wilds provides standardized lesion-level patches and explicitly models hospital distribution shifts, unlike the original dataset’s splits containing slides from all five hospitals.Related datasets used different hospital configurations or fewer hospitals.Source: §E.2.4 · p. 76

E.3 RxRx1-wilds

• RxRx1-wilds studies whether models can recognize 1,139 genetic treatments from three-channel fluorescent microscopy images across unseen experimental batches with visually significant batch differences.The dataset derives from a six-channel Cell Painting assay but includes only the first three channels.Source: §E.3 · p. 76, §E.3.1 · pp. 76–77

E.3.1 Setup

• The dataset contains 51 experiments across four cell lines, using four 384-well plates in which siRNA perturbations alter cell morphology, count, and distribution, often subtly.The experiments include 24 HUVEC, 11 RPE, 11 HepG2, and 5 U2OS experiments.Source: §E.3.1 · pp. 77–78
• Training uses 33 experiments and 40,612 site-1 images, while OOD validation and test use 4 and 14 unseen experiments, respectively, and ID test uses site 2 of the training experiments.The OOD validation and test sets contain 9,854 and 34,432 images, respectively; ID test contains 40,612 images.Source: §E.3.1 · p. 78
• Models are evaluated by average accuracy across test images because cell types are imbalanced, with higher performance on over-represented HUVEC and lower performance on under-represented U2OS.The authors avoid worst-cell-type accuracy because limited U2OS data makes a subpopulation-shift formulation difficult, and they caution against post-prediction methods that exploit dataset structure without improving OOD generalization.Source: §E.3.1 · p. 79

E.3.2 Baseline results

• 35.9% ID accuracy fell to 29.9% OOD accuracy for ERM on RxRx1-wilds, while CORAL, IRM, and Group DRO performed worse than ERM.In the mixed-to-test setting, accuracy was 39.8%, exceeding the 35.9% ID result because of cell-type composition differences; this suggests Table 7 understates the distribution-shift gap.Source: §E.3.2 · pp. 79–80, Table 7 · p. 80, Table 8 · p. 80

E.3.3 Broader context

• Batch effects in biological experiments can correlate with biological signal, so correction methods must balance nuisance removal against signal preservation.The section notes that high-throughput screening batch effects recur across genomics, transcriptomics, proteomics, and metabolomics, and contrasts RxRx1-wilds with BBBC021.Source: §E.3.2 · p. 80, §E.3.3 · pp. 80–81

E.3.4 Additional details

• RxRx1-wilds contains 125,510 quality-filtered 256×256 images derived from downsampled, cropped, three-channel versions of the original six-channel dataset.Its splits and evaluation differ from the Kaggle competition, including site-level accuracy for treatment and control classes rather than aggregated well-level treatment accuracy.Source: §E.3.4 · p. 81

E.4 OGB-MolPCBA

• The OGB-MolPCBA benchmark studies 128-task molecular-activity prediction under scaffold-based domain generalization to structurally distinct molecules.It is adopted from the Open Graph Benchmark and originally curated by MoleculeNet.Source: §E.4 · p. 82, §E.4.1 · p. 82

E.4.1 Setup

• Scaffold groups are highly skewed, but label and missingness statistics remain similar across splits, indicating that molecular graph structure is the primary distribution shift.The training split contains 44,930 scaffolds averaging 7.8 molecules, while the test split contains 43,793 singleton scaffolds.Source: §E.4.1 · pp. 82–83, Figure 22 · p. 83
• 7.2 AP points separate ERM performance on the conventional random split from its performance on the harder official scaffold split.ERM also outperformed CORAL, IRM, and Group DRO; CORAL and IRM underfit training data, with training AP of 20.0% and 15.9% versus 36.1% for ERM with 0.5 dropout.Source: §E.4.1 · pp. 83–84, Table 9 · p. 83, Table 10 · p. 84

E.4.4 Additional details

• OGB-MolPCBA contains 437,929 molecules and 128 bioassay labels, with additional dataset details provided through MoleculeNet (Wu et al., 2018) and the Open Graph Benchmark (Hu et al., 2020b).The dataset is represented as molecular graphs and uses scaffold-based splits; its labels are curated from PubChem.Source: §E.4.1 · p. 82, §E.4.4 · p. 85

E.5 GlobalWheat-wilds

• GlobalWheat-wilds contains 6,515 images and 275,187 wheat heads from 47 acquisition sessions across 12 countries, testing generalization to new countries and sessions.Training uses European sessions, validation uses Asian and African sessions, and testing uses Australian and North American sessions, creating location and acquisition variation.Source: §E.5 · pp. 85–86
• The in-distribution test split contains images from the same 18 European training acquisition sessions, comprising 16,093 wheat heads across 357 images.Table 11 defines the acquisition-session domains and split relationships, while Table 12 reports baseline results with standard deviations across 3 replicates.Source: §E.5 · p. 86, Table 11 · p. 86, Table 12 · p. 87
• Detection accuracy is computed at IoU 0.5, averaged first within images, then across acquisition sessions to prevent uneven image counts from dominating.The metric is used instead of average precision because it was adopted by prior Global Wheat Challenges (David et al., 2020, 2021), and instead of worst-case accuracy because session difficulty varies.Source: §E.5 · p. 87
• Training spans 5 countries and 18 acquisition sessions, covering all growth stages and variation in sensors, illumination, genotypes, growing conditions, and planting strategies.Test domains include unseen conditions such as sensors and genotypes, motivating evaluation of robustness to these shifts.Source: §E.5 · p. 87

E.5.2 Baseline results

• 63.3 (1.7) mixed-to-test ID accuracy exceeded 49.6 (1.9) official-split OOD accuracy on the controlled subsampled test set (Table 13).The mixed-to-test design splits each test domain between training and evaluation, controlling for differences in test-distribution difficulty.Source: §E.5.2 · p. 88, Table 13 · p. 88
• Accuracy drops varied across acquisition sessions, with KSU and Terraref showing larger declines; Terraref remained difficult because of image contrast and hidden wheat heads.KSU retained comparatively high mixed-to-test accuracy but still suffered a large official OOD drop, while development stage did not appear to explain the gap.Source: §E.5.2 · p. 89
• Group DRO performed poorly relative to ERM, while data augmentation and domain-specific variation remain potential directions for reducing performance gaps.CORAL and IRM were left for future work because applying them to detection tasks was not straightforward; the baselines used no augmentation.Source: §E.5.2 · p. 89

E.5.3 Broader context

• GlobalWheat-wilds motivates generalizable plant-phenotyping models because related tasks also face variation in illumination, sensors, deployments, and plant species.The authors aim for the dataset to encourage collaboration and new multi-domain plant datasets covering crop-growing regions worldwide.Source: §E.5.3 · p. 89

E.5.4 Additional details

• GlobalWheat-wilds uses 2021 Global Wheat Challenge data but changes the splits to create ID sets and continent-disjoint OOD validation and test sets.North American images were training data in the 2020 challenge but evaluation data in 2021 and are assigned to the GlobalWheat-wilds test set.Source: §E.5.4 · p. 90

E.6 CivilComments-wilds

• CivilComments-wilds studies toxicity classification under subpopulation shift, targeting comments that mention demographic identities rather than comments written by demographic-group members.The task uses 450,000 crowdsourced comments with toxicity labels and eight binary demographic-mention indicators.Source: §E.6 · p. 90, §E.6.1 · p. 90

E.6.1 Setup

• Comments are split by disjoint articles into 269,038 training, 45,180 validation, and 133,782 test examples, with results averaged over five seeds because small groups increase variability.Demographic annotations are available during training, providing data for each target group, and are relatively straightforward to predict at test time despite being unavailable there.Source: §E.6.1 · pp. 90–91
• CivilComments-wilds evaluates worst-group accuracy over 16 overlapping groups formed by eight demographic identities crossed with toxic versus non-toxic labels.This label-wise construction measures both sensitivity and specificity disparities, preventing equal aggregate group accuracies from hiding opposing bias patterns.Source: §E.6.1 · pp. 90–91

E.6.2 Baseline results

• CivilComments-wilds ERM reaches 92.2% average accuracy but only 57.4% on toxic comments mentioning other religions, while reweighting reaches 69.2% worst-group accuracy at 89.8% average accuracy.A substantial average-versus-worst-group gap remains despite reweighting.Source: §E.6.2 · pp. 91–92, Table 15 · p. 92
• CORAL, IRM, and group DRO with Label × Black partitions do not significantly improve performance, with slight Black-group gains offset by degradation on groups such as non-toxic LGBTQ comments.CORAL and IRM perform similarly to reweighting, and their penalty searches select the lowest penalty values.Source: §E.6.2 · p. 93

E.6.3 Broader context

• CivilComments-wilds targets disparities associated with demographic identities mentioned in comments rather than requiring authors’ demographics, while broader NLP and commercial systems also show demographic performance disparities.This setting is relevant when user demographics are unavailable, such as anonymous online comments.Source: §E.6.3 · pp. 93–94

E.6.4 Additional details

• CivilComments-wilds evaluates worst-group accuracy over demographic identities because subgroup AUCs can assume separate subgroup thresholds, while acknowledging intrinsic difficulty, intersectional sparsity, and annotation subjectivity.The benchmark uses majority-voted labels and recognizes that balanced TPR/FPR need not imply equitable performance.Source: §E.6.4 · p. 94

E.7.1 Setup

• FMoW-wilds tests 62-category satellite-image classification across three time domains and five geographic regions, measuring average and worst-region OOD accuracy under temporal shifts.Training uses 76,863 images from 2002–2013, with OOD validation and test data from later periods; splits use disjoint coordinates, and Africa and Oceania have the fewest examples.Source: §E.7.1 · p. 96

E.7.2 Baseline results

• 6.7% average accuracy and 26.0% worst-region accuracy drops occur under FMoW-wilds time shifts, with an 11.6% average-accuracy decline on 2017 images.The ERM model performs much worse in Africa than in other regions under the shift.Source: Table 20 · p. 98, §E.7.2 · p. 99
• A 26.0% ID–OOD worst-region accuracy gap shrinks to 16.3% when training mixes OOD-period data, but remains significant, implicating time and region shifts.The region disparity appears mainly under the combined time-and-region shift; Africa also exhibits a large label shift, whereas the <2013 versus ≥2013 split does not.Source: §E.7.2 · p. 99

E.7.4 Additional details

• FMoW-wilds uses 523,846 RGB satellite images from over 200 countries and separates post-2013 data into OOD validation and testing while treating each image independently.Images are resized to 224 × 224 pixels, and the split removes post-2013 examples from training while keeping locations disjoint across splits.Source: §E.7.4 · p. 101

E.8 PovertyMap-wilds

• PovertyMap-wilds targets global poverty estimation despite scarce survey labels, leveraging satellite imagery to generalize across countries where ground-truth measurements are unavailable.The benchmark is intended to support more robust and cheaper measurements for broader computational-sustainability challenges.Source: §E.8 · pp. 101–102, §E.8.3 · p. 106

E.8.1 Setup

• PovertyMap-wilds evaluates asset-wealth prediction across 23 countries and urban/rural domains using five folds with fold-specific OOD countries, including validation and test countries distinct from training countries.Examples include location and urban/rural metadata; evaluation reports Pearson correlation, average correlation, and the lower urban/rural correlation.Source: §E.8.1 · pp. 102–103
• PovertyMap-wilds exhibits country-level wealth variation from -0.4 to +0.8, with urban wealth positive in every country and rural wealth mostly negative.The benchmark’s design reflects substantial socioeconomic, infrastructure, agricultural, and cultural variation across countries and urban/rural areas.Source: §E.8.1 · pp. 102–103, Figure 25 · p. 103

E.8.2 Baseline results

• A 0.04 drop in average r under unseen-country testing expands to a 0.12 drop for the worst urban/rural subpopulation, with rural correlation consistently lower.CORAL, IRM, and Group DRO are generally comparable to ERM and remain susceptible to country and urban/rural shifts.Source: §E.8.2 · pp. 104–105, Table 21 · p. 104
• Mixed training across all countries produces a much smaller ID–OOD drop, indicating that the baseline degradation largely reflects shifts to unseen countries rather than intrinsic OOD difficulty.The mixed split preserves training-set size while sampling from more countries, and models trained on the official split degrade especially on rural subpopulations.Source: §E.8.2 · p. 104, Table 22 · p. 105

E.8.4 Additional details

• Removing nighttime-light information lowers OOD r by 0.04 on average and 0.06 for rural subpopulations, demonstrating its usefulness for asset-wealth estimation.The dataset also perturbs location metadata for privacy, and its random ID splits may permit some spatial memorization that improves in-domain performance.Source: §E.8.4 · p. 106

E.9.1 Setup

• Amazon-wilds defines reviewer domains for 1-to-5-star sentiment classification and evaluates accuracy at the reviewer 10th percentile to measure tail performance across subpopulations.The data contain 539,502 reviews, with disjoint reviewer sets for training, OOD validation, and OOD test, plus ID splits containing seen reviewers.Source: §E.9.1 · p. 107

E.9.2 Baseline results

• Accuracy varies from 100.0% to 12.0% across Amazon-wilds reviewers, with 53.8% at the 10th percentile versus 65.4% for a matched random baseline.ERM achieves 71.9% average accuracy, but tail performance is low for both seen and unseen reviewers, with additional drops on unseen reviewers.Source: §E.9.2 · p. 108
• Reweighting, CORAL, IRM, and Group DRO fail to improve Amazon-wilds tail performance, while CORAL, IRM, and Group DRO also fail to improve average accuracy on ID and OOD sets.Reweighting for uniform class balance does not solve reviewer variation, and the selected CORAL and IRM penalties are both λ = 1.0.Source: §E.9.2 · p. 109

E.9.4 Additional details

• Amazon-wilds preprocessing enforces reviewer and product coverage, removes overly long, duplicate, HTML-containing, and generic reviews, and reserves 75 reviews per reviewer for evaluation.Reviewer IDs and additional metadata are provided, enabling methods to model variation across users and potentially mitigate disparities among seen reviewers.Source: §E.9.1 · p. 107, §E.9.4 · p. 110

E.10.1 Setup

• Py150-wilds measures code-completion accuracy across repositories, focusing on class and method tokens because these are more relevant to real developer usage than common tokens such as punctuators.The dataset contains 150,000 Python files from 8,421 GitHub repositories, split by repository into ID and OOD sets.Source: §E.10.1 · pp. 110–111, §E.10.4 · p. 113

E.10.2 Baseline results

• 75.4% accuracy on class and method tokens in Test (ID) falls to 67.9% on Test (OOD), while all-token accuracy drops from 74.5% to 69.6%.The smaller all-token gap reflects many universally used tokens, including punctuators and keywords.Source: §E.10.2 · p. 112, Table 26 · p. 112
• CORAL and IRM reduce the ID–OOD performance gap on Py150-wilds but neither improves final OOD performance over ERM.Both methods use λ = 1 penalties, and repositories are treated as domains for robust-training baselines.Source: §E.10.2 · p. 112

E.10.3 Broader context

• Py150-wilds addresses repository, programming-language, and synthetic-to-real usage shifts, while its repository-based split prevents CodeSearchNet pretraining overlap in validation and test repositories.The dataset re-splits the original files by repositories and emphasizes class-name and method-name prediction to better reflect deployment usage.Source: §E.10.3 · p. 112, §E.10.4 · p. 113

F.1.1 Setup

• Weapon-classification evaluation measures precision for each race and borough at 60% global weapon recall, reflecting concerns about low-precision stops and Fourth Amendment violations.The results are similar when using different recall thresholds.Source: §F.1.1 · p. 114

F.1.2 Baseline results

• The baseline uses logistic regression with Adam, early stopping, reweighted cross-entropy, and separately tuned hyperparameters across four training subsets.Training subsets include Black pedestrians, Bronx pedestrians, and size-matched samples of all pedestrians.Source: §F.1.2 · p. 114, §F.1.1 · p. 114
• Precision was lowest for Black pedestrians and Bronx pedestrians, and training specifically on either group improved precision by less than 0.005 relative to the all-group model.These results suggest the disparities are not caused by disadvantaged groups being small minorities; performance may instead be intrinsically noisier and require additional features.Source: §F.1.2 · pp. 114–115, Table 27 · p. 115, Table 28 · p. 115

F.2.1 Setup

• ENCODE predicts each transcription factor’s binding status from local DNA sequence and cell-type-specific DNase-seq accessibility, using 12,800-bp windows with 128 central 200-bp bins.Ambiguous bins are ignored, and models predict each transcription factor separately across cell types.Source: §F.2.1 · pp. 116–117, §F.2 · p. 116, Figure 28 · p. 117
• ENCODE domain-generalization splits hold out cell types and chromosomes, using either the original liver-only challenge split or round-robin cell-type assignments.The liver-only evaluation may not generalize because liver is the sole primary tissue, whereas training uses immortalized cell lines.Source: §F.2.1 · p. 118
• ENCODE performance is measured by average precision over non-ambiguous binary bin-level binding predictions, motivated by more than 100-fold class imbalance between unbound and bound bins.Out-of-domain validation and test sets use held-out cell types, while in-domain sets use training cell types.Source: §F.2.1 · p. 118

F.2.2 Baseline results

• ERM baselines on ENCODE used a modified U-Net to predict 128 genomic bins with average cross-entropy, but optimization instability, cell-type variability, and class imbalance complicated interpretation.Performance varied substantially across domains, and test-to-test versus train-to-train comparisons could yield different conclusions; the instability may partly reflect class imbalance, although this remains speculative.Source: §F.2.2 · pp. 118, 119, 121
• For MAX, round-robin Test (ID) and Test (OOD) AP averaged 64.9 (2.1) and 59.6 (2.0), respectively, without a significant difference; JUND models could not fit binding patterns across multiple cell types.Source: §F.2.2 · pp. 120, 122

F.2.3 Additional details

• ENCODE ERM OOD test performance was competitive with the original challenge but lower than Li and Guan (2019), whose domain-adaptation strategy contributed substantially to stronger performance.The difference also reflects changes in data processing, splits, architecture, and cross-domain training.Source: §F.2.3 · p. 124

F.3.1 Setup

• BDD100K evaluates nine-category multi-task binary image classification under shifts defined by time of day and geographic location, using average accuracy across validation and test sets.The time shift uses mixed-daylight training and validation data with non-daytime test images, while the location shift trains and validates on New York and tests on California.Source: §F.3.1 · pp. 124–125, §F.3.2 · p. 125, §F.3.3 · p. 126

F.3 BDD100K: Object recognition in autonomous driving across locations

• BDD100K showed no substantial time-of-day accuracy drop and only a small location-shift drop, with test accuracy remaining comparable to validation accuracy despite disjoint training and test locations.The authors argue that object recognition may be too easy for autonomous-driving shift evaluation compared with detection or segmentation.Source: Table 36 · p. 124, §F.3 · p. 124, §F.3.2 · p. 125

F.4.1 Setup

• The Amazon time-shift experiment fine-tuned BERT-base-uncased models on 1,000,000 earlier reviews and evaluated future-year generalization using separate 20,000-review OOD validation and test sets.Models were evaluated by average accuracy on the OOD test set.Source: §F.4.1 · p. 126, §F.4.2 · p. 126

F.4.2 Time shifts

• On Amazon Reviews, ERM achieved 76.0% average OOD accuracy and 75.4% accuracy in the worst year, with modest time-shift degradation and a largest 1.1% gap in 2018.The task is multi-class sentiment classification with review-writing years as domains and star ratings from 1 to 5.Source: §F.4.2 · pp. 126–127

F.4.3 Category shifts

• BERT-base-uncased ERM reaches 75.4% on four in-distribution categories, while unseen-category accuracies vary widely and can exceed in-distribution accuracy.This variation shows that apparent gaps may reflect intrinsic category difficulty as well as distribution shift.Source: §F.4.3 · p. 128
• Category-shift performance drops on some targets, including Clothing, Shoes, and Jewelry from 83.0% to 75.2% and Pet Supplies from 78.8% to 76.8%, but gaps are modest elsewhere.Test-to-test comparisons control for intrinsic category difficulty, though smaller oracle training sets may underestimate in-distribution performance.Source: §F.4.3 · pp. 128–129
• Training on four product categories modestly but consistently outperforms training on Books alone across many unseen categories.The comparison keeps training-set size constant and uses ERM on the Amazon Reviews dataset.Source: Table 39 · p. 128, §F.4.3 · p. 129

F.5 Yelp: Sentiment classification across different users and time

• The Yelp experiments model star-rating sentiment, using BERT-base-uncased with shared hyperparameters for time and user shifts.Time domains are review years, while user-shift domains are reviewers; evaluation targets future years or unseen reviewers.Source: §F.5 · p. 129, §F.5.1 · p. 129, §F.5.2 · p. 129

F.5.2 Time shifts

• 76.0% average accuracy and 73.9% in the worst year show strong future-year performance, with the largest out-of-distribution drop reaching 3.1% in 2018.These results use ERM on Yelp reviews from 2006–2013 to predict reviews from 2014–2019.Source: §F.5.2 · p. 129, Table 40 · p. 130, Table 41 · p. 130

F.5.3 User shift

• 71.5% average accuracy versus 56.0% at the 10th-percentile reviewer indicates modest reviewer variation, exceeding the 60.1% random-binomial baseline.Reviewer-level evaluation uses unseen reviewers for testing and reports the 10th-percentile accuracy.Source: §F.5.3 · p. 130, Table 42 · p. 131