PaperlayerView exact revision on arXiv

Source-linked AI summary

WILDS: A Benchmark of in-the-Wild Distribution Shifts

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, Percy Liang

arXiv:2012.07421v3cs.LG

TL;DR

Real-world distribution shifts are common but underrepresented in standard ML datasets, limiting evaluation of deployed-model robustness. WILDS curates 10 datasets with natural shifts, standardizes their evaluation, and finds substantially lower OOD than ID performance, including under existing shift-mitigation methods.

  • Problem

    Real-world distribution shifts are underrepresented in widely used ML datasets despite their prevalence and ability to degrade deployed-system accuracy.

  • Method

    WILDS curates 10 datasets with realistic train/test splits and metrics, then provides standardized data loading, evaluation, and default models.

  • Results

    OOD performance is consistently and substantially lower than corresponding ID performance, while CORAL, IRM, and Group DRO generally fail to improve over ERM.

  • Takeaways & Limitations

    WILDS provides a standardized benchmark for developing and comparing models intended to be robust to real-world distribution shifts.

  • Takeaways & Limitations

    The benchmark excludes some real-world datasets when models cannot generalize across settings or when split difficulty varies substantially.

Abstract

from arXiv · show

Distribution shifts -- where the training distribution differs from the test distribution -- can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild. Despite their ubiquity in the real-world deployments, these distribution shifts are under-represented in the datasets widely used in the ML community today. To address this gap, we present WILDS, a curated benchmark of 10 datasets reflecting a diverse range of distribution shifts that naturally arise in real-world applications, such as shifts across hospitals for tumor identification; across camera traps for wildlife monitoring; and across time and location in satellite imaging and poverty mapping. On each dataset, we show that standard training yields substantially lower out-of-distribution than in-distribution performance. This gap remains even with models trained by existing methods for tackling distribution shifts, underscoring the need for new methods for training models that are more robust to the types of distribution shifts that arise in practice. To facilitate method development, we provide an open-source package that automates dataset loading, contains default model architectures and hyperparameters, and standardizes evaluations. Code and leaderboards are available at https://wilds.stanford.edu.

1. Introduction

WILDS addresses the underrepresentation of natural, real-world distribution shifts by curating 10 datasets spanning diverse applications, modalities, and domains. It standardizes these datasets and evaluations to support robust ML research and deployment.

  • Motivation: Distribution shifts can significantly degrade ML accuracy, yet widely used datasets underrepresent the shifts encountered in real-world deployments.Existing datasets often assume i.i.d. training and testing or use synthetic and artificial shifts.
  • Shift types: The benchmark includes domain generalization, where training and test domains are disjoint, and subpopulation shift, where domains overlap but proportions differ.Models are evaluated on unseen domains or on performance across test subpopulations, respectively.
  • Benchmark contribution: WILDS curates 10 datasets covering natural shifts across hospitals, cameras, molecular scaffolds, experiments, demographics, countries, time periods, users, and codebases.The benchmark spans animal categorization, tumor identification, bioassay prediction, toxicity classification, poverty mapping, sentiment analysis, and code completion.
  • Dataset design: Wilds datasets were selected for substantial performance drops, real-world relevance, tractability, domain annotations, and broad coverage of tasks, modalities, sizes, and domains.The selection process involved domain experts and retained settings where robust-model development remained possible.
  • Infrastructure: An open-source Python package automates data loading and evaluation and provides default models so the benchmark’s baseline results can be replicated.The package standardizes datasets for use in common ML frameworks.
  • Research goal: Wilds aims to accelerate systematic comparisons and the development of models robust to real-world distribution shifts.Its standardized datasets and evaluations are intended to support reliable deployment in the wild.

2. Existing ML benchmarks for distribution shifts

Existing distribution-shift benchmarks enable controlled method development, but many focus on synthetic or artificially constructed shifts. WILDS complements them with natural shifts arising in real-world deployments.

  • Earlier benchmarks: Earlier distribution-shift datasets covered tasks including tagging, sentiment analysis, land-cover classification, object recognition, and flow cytometry but were often smaller than modern datasets.Their limited scale contributed to their reduced use in contemporary ML research.
  • Synthetic and constructed shifts: Many recent benchmarks induce shifts through synthetic corruptions, altered backgrounds or colors, disparate datasets, changed scales, or disjoint subclasses.Examples include ImageNet-C, Backgrounds Challenge, Waterbirds, Colored MNIST, PACS, DeepFashion Remixed, and BREEDS.
  • Strengths: These benchmarks support clean analysis, controlled experimentation, and robustness evaluations of widely used models.Their isolated shifts can reveal effects such as background changes or the value of Gaussian blur training.
  • WILDS motivation: Robustness to synthetic shifts need not transfer to real-world deployment shifts, making those datasets alone insufficient for developing and evaluating real-world robustness.WILDS therefore complements existing benchmarks with diverse natural distribution shifts.

3. Problem settings

WILDS represents overall data as a mixture of domain-specific distributions and distinguishes shifts by how training and test domains and their proportions differ. Domain generalization uses unseen domains, whereas subpopulation shift changes proportions among seen domains.

  • Formalization: Each domain d corresponds to a fixed distribution P_d over inputs, targets, and domain labels, while the overall training and test distributions mix these domains.The mixture weights q^train_d and q^test_d encode the domain composition in training and testing.
  • Formalization: Training domains are those with positive training mixture weights, and test domains are those with positive test mixture weights; domain annotations are available during training.At test time, applications may provide either x or (x, d).
  • Domain generalization: In domain generalization, test domains are disjoint from training domains, but the domains are typically related enough to make generalization tractable.Camelyon17-wilds and iWildCam2020-wilds use hospitals and camera traps as distinct training and test domains.
  • Subpopulation shift: In subpopulation shift, test domains are seen during training but their proportions change, so models are evaluated for performance across the test domains.CivilComments-wilds seeks high accuracy across demographic subpopulations without demographic identity at test time.
  • Hybrid shifts: Some datasets combine domain generalization across time with subpopulation shift across regions, preventing a clean assignment to only one category.The training and test sets can use different time periods while sharing regions.

4. Wilds datasets

Wilds organizes real-world distribution-shift problems into dataset-specific settings, including domain generalization across cameras, hospitals, experimental batches, molecular scaffolds, and acquisition sessions. Each dataset defines application-relevant domains, prediction tasks, and evaluation setups for testing generalization beyond training data.

  • Benchmark design: Wilds frames each dataset as domain generalization, subpopulation shift, or a hybrid reflecting real-world application challenges.The benchmark appends “-wilds” to modified source datasets and provides additional distribution-shift context for each.
  • Camera traps: iWildCam2020-wilds tests 182-species classification across disjoint camera-trap deployments, emphasizing macro F1 to prioritize rare species.The training set includes over 200 camera traps, while test photos come from disjoint camera traps.
  • Hospitals: Camelyon17-wilds evaluates tumor detection across five hospitals, using class-balanced patches from separate hospitals and average accuracy.Hospital variation can arise from patient populations, staining, and image acquisition; staining differences are suggested as the main source in similar datasets.
  • Experimental batches: RxRx1-wilds evaluates classification of 1,139 genetic treatments across disjoint experimental batches to assess robustness to batch effects.The training data span 33 batches, each containing one sample for every class, and evaluation uses out-of-distribution test batches.
  • Scaffolds and acquisition sessions: OGB-MolPCBA predicts 128 biochemical assay labels for molecules with disjoint training and test scaffolds, while GlobalWheat tests wheat-head detection across acquisition sessions and unseen locations.OGB-MolPCBA uses over 40,000 training scaffolds and averages Average Precision across assays; GlobalWheat images come from 12 countries and its split primarily captures location shift.

5. Performance drops from distribution shifts

Wilds measures distribution-shift effects by comparing out-of-distribution performance with carefully constructed in-distribution settings. Across datasets, OOD performance is consistently and substantially lower, while mixed-distribution training can often achieve high performance on both distributions.

  • Dataset selection: Wilds selects datasets whose official train/test shifts cause significant performance drops in standard models.The benchmark excludes real-world shifts that do not produce appreciable drops.
  • Measurement strategy: The primary comparison measures OOD performance on official splits against ID performance under a feasible alternative setting.Wilds uses test-distribution evaluation when practical and otherwise standard training-distribution evaluations, while tracking held-out training-domain performance.
  • Measurement strategy: Evaluating ID and OOD models on the same test distribution isolates performance loss from the distribution shift rather than differences in evaluation difficulty.The conventional train-on-P_train, evaluate-on-P_train comparison does not control the evaluation distribution.
  • ID settings: Test-to-test comparisons are directly comparable but infeasible because held-out P_test data would need to match the size of D_train.Wilds therefore uses mixed-to-test comparisons where feasible, which retain fixed test-distribution evaluation with less test-domain training data.
  • ID settings: Mixed-to-test comparisons may underestimate ID performance because training jointly fits P_train and P_test, but they test whether one model can fit both distributions.Failure to learn such a model would suggest the shift is intractable for the model family.
  • Results: OOD performance is consistently and substantially lower than corresponding ID performance across every Wilds dataset.Where mixed-to-test comparisons are available, models can simultaneously achieve high ID and OOD performance, indicating the OOD sets are not intrinsically harder.

6. Baseline algorithms for distribution shifts

Wilds evaluates representative algorithms designed for domain generalization and subpopulation shifts against ERM. These methods generally do not eliminate OOD performance drops, with a notable improvement only for CivilComments-wilds under Group DRO.

  • Domain generalization: CORAL penalizes differences in feature means and covariances across domains, while IRM penalizes domain-specific optimal linear classifiers.Both methods add invariance-oriented penalties to the ERM objective.
  • Subpopulation shift: Group DRO explicitly minimizes loss on the worst-case domain for subpopulation-shift robustness.The experiments also apply Group DRO to domain generalization and CORAL and IRM to subpopulation shifts.
  • Results: Table 2 shows that CORAL, IRM, and Group DRO generally fail to improve over ERM.The benchmark evaluates these algorithms using the same dataset-specific OOD metrics as the ERM comparisons.
  • Results: 70.0% OOD accuracy for Group DRO on CivilComments-wilds exceeded ERM’s 56.0%, but remained below ERM’s 92.2% ID accuracy.The improvement occurs on the worst-performing minority subpopulation and is associated with upweighting that domain.
  • Implications: Robustness to real-world distribution shifts remains an open challenge because existing methods were often designed for different settings.Wilds also offers more diverse domains and metadata that future methods might leverage.

7. Empirical trends

Wilds provides multiple domains and OOD validation sets to support model selection under shift. Its empirical trends include dataset-dependent validation effects and compounding gaps when multiple shifts occur together.

  • Model selection: Wilds provides multiple training domains and OOD validation sets because similar ID performance can correspond to substantially different OOD generalization.Most datasets also provide ID validation or test sets for comparing ID and OOD performance, without using them for official evaluation.
  • Model selection: On Wilds baselines, OOD-validation model selection gives comparable or higher OOD performance than ID-validation selection.This contrasts with findings on DomainBed datasets, where ID validation produced higher OOD performance.
  • Model selection: The differing validation outcomes may reflect Wilds’ larger number of domains and procedural differences from DomainBed.The paper identifies further study of validation choices as useful future work.
  • Compounding shifts: Hybrid shifts can exacerbate performance gaps: in PovertyMap-wilds and FMoW-wilds, unseen-domain shifts worsen subpopulation gaps and vice versa.FMoW-wilds shows a regional subpopulation difference only when the time shift is also considered, whereas Amazon-wilds does not show this compounding effect.
  • Compounding shifts: These observations support evaluating combinations of distribution shifts that occur in practice rather than each shift in isolation.The strength of the compounding effect is not universal across datasets.

8. Distribution shifts in other application areas

Wilds surveys distribution shifts beyond its benchmark, including fairness, healthcare, genomics, education, and autonomous driving. These areas offer important applications but often lack datasets meeting Wilds’ requirements for realistic, measurable performance drops.

  • Future benchmarks: Constructing realistic distribution-shift benchmarks in additional application areas remains an important avenue for future work.The paper welcomes community contributions of datasets in these areas.
  • Algorithmic fairness: Future fairness benchmarks need sensitive attributes, immediately relevant prediction tasks, sufficient data and features, and disparities large enough to motivate improved methods.Existing fairness datasets are often small, feature-limited, or show modest subgroup performance disparities.
  • Algorithmic fairness: The stop-and-frisk dataset exhibits large performance disparities across race groups and locations, but targeted training did not reduce them.Black and Hispanic pedestrians comprised over 90% of stops, unlike the typical minority-underrepresentation setting.
  • Medicine and healthcare: Medical distribution-shift benchmarks must distinguish shift-related disparities from differences caused by access, comorbidities, or biased outcome definitions.The paper identifies these non-shift factors as challenges when interpreting healthcare disparities.
  • Genomics: Genomics evaluations found a gap between cross-cell-type and within-cell-type performance, attributed to cell-type-specific regulatory complexes.The investigated subset was not included in Wilds because a model could not generalize across all cell types simultaneously.
  • Education: Education lacks standardized public datasets partly because of student privacy and proprietary standardized tests.The paper identifies large-scale online-course data as a potential source for future benchmark construction.
  • Autonomous driving: BDD100K showed no substantial performance drops across user, weather, time, or location shifts, so it was not an appropriate Wilds benchmark.This illustrates that realistic shifts alone are insufficient without measurable degradation.

9. Guidelines for method developers

Wilds is intended to support systematic evaluation of both general-purpose and shift-specific algorithms, while keeping algorithm comparisons standardized. The authors emphasize unresolved questions about cross-dataset robustness and caution against overinterpreting limited test-domain results.

  • Wilds benchmarks training algorithms for robustness to distribution shifts and encourages standardized datasets and default architectures to isolate algorithmic contributions.
  • Developers are encouraged to evaluate methods across all applicable Wilds datasets because performance may depend on the type and structure of the shift.
  • A single general-purpose algorithm that performs well across all datasets without modeling each shift’s structure remains an open question.
  • Wilds also welcomes improvements from new architectures and external pre-training, tracked separately from standardized training-algorithm submissions.
  • Some datasets contain few domains, so developers should avoid overfitting to specific test sets and use the provided out-of-distribution validation sets.
  • The benchmark focuses on domain generalization and subpopulation shift, while other settings such as unsupervised domain adaptation remain possible extensions.

10. Using the Wilds package

The Wilds package provides a standardized interface for loading datasets, accessing metadata, and evaluating models. Its utilities support varied algorithm requirements while automating the metrics used in the paper and leaderboard.

  • Datasets and data loading: The PyTorch-based package exposes a simple interface, handles downloads, and provides data loaders and utilities for domain annotations and metadata.
  • Datasets and data loading: Dataset initialization, split access, and data-loader setup are standardized across the benchmark.
  • Datasets and data loading: Multiple data loaders accommodate algorithms that require different data-loading schemes.
  • Domain information: Grouper objects extract flexible domain or group annotations from dataset metadata for algorithms that use this information.
  • Evaluation: Calling each dataset’s eval method produces the metrics reported in the paper and on the leaderboard.
  • Reproducibility: The executable paper provides the exact commands, code, environment, data, and trained weights used for the reported experiments.

Appendix A. Dataset realism

Wilds evaluates benchmark realism across data, task and metric, and train/test distribution shift. It selects datasets intended to reflect varied real applications and deployment-relevant shifts rather than only realistic inputs.

  • Benchmark realism depends independently on the realism of the data, task and evaluation metric, and train/test split.
  • Data: Realistic data should reflect what could plausibly be collected and made available to a deployed model, including relevant metadata.
  • Task and evaluation metric: A realistic task and metric should match application goals; wildlife monitoring may prioritize rare-species accuracy over average classification accuracy.
  • Distribution shift: A realistic distribution shift reflects deployment conditions, such as training on a few hospitals and testing on unseen hospitals.
  • Wilds design: Wilds addresses existing benchmark gaps by combining varied tasks, realistic data and metrics, and deployment-relevant train/test splits.
  • Shift selection: The benchmark assigns each dataset a shift setting based on application context, including domain generalization for cross-hospital deployment.

Appendix B. Prior work on ML benchmarks for distribution shifts

Prior distribution-shift benchmarks use transformations, synthetic-to-real transfers, constrained splits, or combinations of datasets, but may not represent shifts arising in real deployments. Wilds focuses on domain generalization and subpopulation shift while identifying other settings as future extensions.

  • Transformations: Existing benchmarks induce shifts through synthetic transformations such as rotations, translations, texture changes, color changes, and corruptions.
  • Synthetic-to-real transfers: Synthetic-to-real benchmarks transfer from simulated data to real or varied conditions such as seasons, weather, time, or architectural style.
  • Constrained splits: Other benchmarks use realistic data but constrained splits that do not necessarily reflect distribution shifts arising in the wild.
  • Across-dataset shifts: Combining disparate datasets creates shifts across datasets or renditions, including examples built from Flickr and Google image queries.
  • Problem settings: Wilds currently focuses on domain generalization and subpopulation shifts, while other domain-shift settings are proposed as future work.
  • Problem-setting axes: Domain-shift settings differ by whether test domains are seen, whether domain annotations are available, and whether unlabeled test data can be used.
  • Adaptation settings: Unsupervised domain adaptation assumes abundant unlabeled test-domain data and separate models for each test distribution, whereas test-time adaptation adapts a model to test examples.
  • Selective prediction: Selective prediction is orthogonal to these settings because models may abstain on low-confidence examples for human experts to handle.

Appendix D. Additional experimental details

Appendix D standardizes model selection, replication, losses, and implementations across Wilds experiments. OOD validation guides early stopping and generally matches or slightly improves OOD test performance relative to ID validation.

  • Model selection: OOD validation performance determines early stopping and hyperparameter selection, yielding similar or slightly better OOD test performance than ID validation.This protocol is applied to standard ERM training and compared directly with ID-based selection.
  • Baseline algorithms: CORAL, IRM, and Group DRO are adapted from prior implementations, with specified penalty-weight searches for CORAL and IRM and a fixed Group DRO step size.ERM hyperparameters are generally reused for the algorithm baselines.
  • Replication: The benchmark reports results across three random-seed replicates by default, with five for CivilComments and ten for Camelyon17 because of dataset-specific variability.PovertyMap instead uses five-fold cross-validation.
  • Training objectives: Classification models use cross-entropy loss, whereas PovertyMap regression uses mean-squared error.
  • Baseline algorithms: Uniform domain sampling reweights CORAL and IRM toward equal domain contributions, while Group DRO still optimizes the worst-domain loss and gains optimization stability.Group DRO reweighting methods require a non-vanishing training loss, achieved through early stopping.
  • Implementation constraints: CORAL and IRM require enough examples from each domain in a minibatch, creating practical difficulties when datasets contain many domains.

Appendix E. Additional dataset details and results

Appendix E details Wilds dataset construction, evaluation, and empirical behavior, emphasizing iWildCam’s camera-trap shift. iWildCam exhibits a substantial ID-to-OOD performance drop, domain variability, and limited gains from standard shift algorithms.

  • iWildCam2020-wilds: iWildCam2020-wilds studies species classification across camera traps, using 203,029 images from 323 traps and 182 animal-species labels.Each image includes camera-trap, sequence, and datetime metadata.
  • iWildCam2020-wilds: The dataset uses camera-trap-disjoint OOD validation and test splits, alongside ID splits from training traps on different days.Training uses 243 traps, OOD validation 32, OOD test 48, and the ID splits reuse training traps.
  • Evaluation: Macro F1 is the primary metric because average accuracy can obscure performance on rare species in heavily imbalanced camera-trap data.
  • Results: 47.0 macro F1 on ID test falls to 31.0 on OOD test for ERM, demonstrating a substantial and consistent camera-trap distribution-shift gap.
  • Results: OOD split metrics vary because camera traps differ in class balance, including the presence of rare species and the fraction of empty images.
  • Limitations: The reported ID-OOD gap relies on a train-to-train comparison, and its magnitude under test-to-test or mixed-to-test comparisons remains for future work.
  • Results: CORAL, IRM, Group DRO, and label reweighting did not improve upon ERM; illumination, backgrounds, and animal distributions likely encourage location-specific overfitting.

E.2 Camelyon17-wilds

Camelyon17-wilds tests tumor detection across hospitals using standardized histopathology patches. Performance varies sharply across held-out hospitals, making OOD evaluation and model selection unreliable from a single split or seed.

  • Dataset and task: Camelyon17-wilds predicts tumor presence in 96x96 patches across hospitals, using 450,000 patches from 50 slides in five Dutch hospitals.The task is binary lesion-level classification based on the central 32x32 region.
  • Dataset and task: Training uses three hospitals, OOD validation uses a fourth, and OOD test uses a visually distinctive fifth hospital; ID validation reuses training hospitals.
  • Evaluation: The dataset is preprocessed into balanced patches so average patch accuracy is a practical evaluation metric.Class balancing addresses severe class imbalance and the complexity of slide-level evaluation.
  • Results: 6.4% standard deviation across 10 random seeds occurs on the held-out test hospital, alongside large ID-to-OOD validation and OOD-validation-to-test gaps.OOD validation early stopping contributes to the latter comparison.
  • Discussion: The results expose a model-selection failure mode: training and validation data cannot reliably identify models that will perform well on unseen hospitals and seeds.
  • Results: The choice of held-out OOD hospital substantially affects performance because the test hospital’s patches were visually more distinct.
  • Dataset construction: The original dataset’s splits included all five hospitals and did not standardize patches or explicitly evaluate hospital distribution shifts.

E.3 RxRx1-wilds

RxRx1-wilds evaluates whether models can recognize genetic treatments across unseen experimental batches while preserving biological signal. Its batch-based splits expose substantial OOD degradation, and existing robustness methods perform poorly.

  • Problem setting: RxRx1-wilds uses fluorescent microscopy images of four human cell lines to classify which of 1,139 genetic treatments the cells received across experimental batches.The domain is the experimental batch, and the images use a 3-channel variant of the Cell Painting assay.
  • Data construction: The dataset repeats one experimental design across 51 batches, using four 384-well plates with treatment and control siRNAs distributed across wells.Each experiment uses one cell type; the four cell lines are HUVEC, RPE, HepG2, and U2OS.
  • Data splits: Training contains 40,612 images from 33 experiments and site 1, while OOD validation and test use disjoint experiments and both imaging sites.The ID test set uses site 2 from the same 33 training experiments, enabling a train-to-train comparison.
  • Results: 35.9% ID accuracy fell to 29.9% OOD accuracy for ERM, demonstrating a substantial performance drop across experimental batches.Models are evaluated by average accuracy across test images, with the two images per test well evaluated independently.
  • Results: 39.8% accuracy in the mixed-to-test setting exceeded the 35.9% train-to-train ID result, indicating that cell-type composition affects the comparison and that the standard ID result understates the shift.The train-to-train test set contains relatively more U2OS and less HUVEC, while performance is lower on U2OS and higher on HUVEC.
  • Results: CORAL, IRM, and group DRO all performed poorly relative to ERM, potentially because treating heterogeneous cell-type experiments as equivalent domains is challenging.The experiments in the training set comprise different cell types, complicating domain-based methods.

E.4 OGB-MolPCBA

OGB-MolPCBA evaluates molecular-property prediction under scaffold-based distribution shift, where test molecules come from structurally disjoint scaffolds. The scaffold split is substantially harder than a conventional random split, and standard ERM outperforms the tested scaffold-aware baselines.

  • Dataset and shift: OGB-MolPCBA uses molecular graphs as inputs, 128-dimensional biochemical-assay labels, and scaffold identity as the domain, with disjoint scaffolds across training and test sets.The benchmark provides scaffold groupings so models can learn representations that extrapolate across molecular structure groups.
  • Dataset and shift: The scaffold split is designed to test generalization across differences in molecular graph structure.The paper identifies the difference in input molecular graph structure as the main distribution shift.
  • Results: 7.2 points AP is the ERM performance drop when using the scaffold split instead of the conventional random split.Both splits use an 80/10/10 train/validation/test ratio, but the scaffold split is harder.
  • Results: ERM performs better than CORAL, IRM, and Group DRO on this scaffold-based evaluation.For CORAL and IRM, smaller penalties performed better because larger penalties made training insufficient; the selected penalties were 0.1 and λ = 1, respectively.

F.2 ENCODE: Transcription factor binding across different cell types

The ENCODE analysis uses DNA sequence and chromatin accessibility to predict transcription-factor binding across cell types, but substantial cell-type variability limits reliable conclusions. These limitations led the authors to exclude the ENCODE splits from Wilds.

  • Task and setup: DNA sequence and chromatin accessibility are combined to predict binding patterns for multiple transcription factors in new cell types.Each input is a 12,800-bp sequence window with chromatin-accessibility features, and outputs binding status across 128 genomic bins.
  • Distribution shifts and limitations: Large ID-OOD performance gaps appeared for both MAX and JUND on the ENCODE-DREAM splits, but variability across cell types discouraged interpreting results from a single liver test cell type.The ENCODE-DREAM setup trains on immortalized cell lines and evaluates on primary liver tissue, creating a realistic but narrow generalization setting.
  • Round-robin results: 64.9 (2.1) vs. 59.6 (2.0) average Test (ID) and Test (OOD) AP for MAX showed no significant difference across round-robin splits.The comparison uses the train-to-train setting across round-robin splits.
  • Round-robin results: 54.1 (4.2) vs. 42.9 (3.2) average Test (ID) and Test (OOD) AP for JUND showed a larger gap, but training variability made the result less reliable.Test-to-test ID performance was also significantly higher than train-to-train ID performance, suggesting difficulty fitting variation across cell types.
  • Why ENCODE was excluded: Optimization instability and insufficient model capacity or feature information were major obstacles to using the ENCODE data as a Wilds benchmark.The authors note that robust distribution-shift training algorithms are unlikely to substantially improve OOD performance when the main issue lies in the model family or data distribution.
Loading 2012.07421v3…