Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time

• Temporal shifts arise over time and can degrade deployed models; examples include worsening predictions of future language and flu incidence.Source: §1 · pp. 1–2
• Natural temporal shifts remain understudied, while prior benchmarks emphasize domain or subpopulation shifts or artificial task sequences.Source: §1 · p. 2
• Wild-Time benchmarks natural temporal shifts across real-world tasks and offers fixed-time-split and data-stream evaluation strategies.Source: §1 · p. 2
• The paper adapts invariant-learning methods to unsegmented temporal streams by treating windows of time as domains.Source: §1 · p. 2
• Invariant, self-supervised, and continual-learning approaches do not show substantial improvements over standard ERM training.Source: §1 · p. 2

2 Problem and Evaluation Settings

• The temporal setting represents data as distributions indexed by timestamps and allows consecutive timestamps to have identical distributions.Source: §2 · p. 3

2.1 Criteria for Dataset Selection

• Dataset selection requires naturally collected timestamped data, substantial train-test performance drops, and gradual rather than sudden shifts.Source: §2.1 · p. 3

2.2 Evaluation Strategies

• Eval-Fix uses one fixed past-to-future split and reports average and worst-time performance; Eval-Stream evaluates each timestamp on the next K timestamps.Source: §2.2 · pp. 3–4

• Yearbook contains 33,431 grayscale high-school portraits from 1930–2013, with gender labels and a pre-1970 training split.Source: §3 · p. 4
• FMoW-Time predicts one of 62 land-use categories from satellite images, using 2002–2015 for training and 2016–2017 for testing.Source: §3 · p. 4
• MIMIC-IV contains 216,487 records grouped into three-year timestamps and supports readmission and in-hospital mortality prediction.Source: §3 · p. 4
• HuffPost classification predicts news categories from headlines, retaining 11 categories across 2012–2022 and splitting at 2015.Source: §3 · pp. 4–5
• The arXiv task predicts preprint primary category from titles across 172 categories and the 2007–2022 dataset period.Source: §3 · p. 5

Classical Supervised Learning and Continual Learning

• The benchmark evaluates ERM in both fixed-split and stream settings, with stream-mode ERM applied at every timestamp.The method suite also includes four continual-learning algorithms: fine-tuning, EWC, SI, and A-GEM.Source: §4 · p. 5

Temporally Invariant Learning

• Invariant learning methods seek cross-domain invariance through feature alignment, selective augmentation, stronger representation-label correlations, or worst-group optimization.The benchmark selects CORAL, IRM, LISA, and GroupDRO, adapting them to train incrementally at each timestamp.Source: §4 · p. 5
• Invariant methods are adapted by treating sliding windows of timestamps as domains, yielding T − L + 1 substreams for a stream with T timestamps and window length L.CORAL, GroupDRO, and IRM become CORAL-T, GroupDRO-T, and IRM-T; the illustrated case T = 7 and L = 3 produces four substreams.Source: §4 · p. 5, Figure 3 · p. 5
• The temporal adaptations address missing domain labels because benchmark streams are unsegmented and timestamps need not mark distribution changes.LISA is not adapted because intra-label LISA works without domain information; the adaptation study also varies window length and overlap.Source: §4 · pp. 5–6

Self-Supervised Learning and Ensemble Learning

• Self-supervised methods SimCLR and SwaV are evaluated on Yearbook and FMoW-Time, while SWA represents ensemble learning.SWA averages parameter values along a stochastic-gradient-descent trajectory.Source: §4 · p. 6

5 Experiments

• The experiments benchmark the methods from Section 4 on each dataset using both Eval-Fix and Eval-Stream, with accuracy for most classification tasks and ROC-AUC for MIMIC-Mortality.The Eval-Stream results are reported in Appendix D.Source: §5 · p. 6, §5.1 · p. 6

5.1 Experimental Setup

• Both evaluations use training timestamps earlier than test timestamps, and temporal OOD robustness is measured on the test set against held-out training-time ID performance.The ID reference averages per-timestep performance on held-out examples from training timestamps; the holdout is 10%, or 20% for the two MIMIC tasks.Source: §5.1 · p. 6, §5 · p. 6

5.2 Performance Drops from Temporal Distribution Shifts

• OOD performance is substantially lower than ID performance, and experiments training ERM on both ID and OOD examples indicate the gap reflects distribution shift rather than training-time difficulty.Table 1 compares ID and OOD test performance under Eval-Fix; Table 2 reports each method’s OOD performance in that setting.Source: §5.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7

5.3 Baseline Comparison

• In FMoW-Time and both MIMIC tasks, performance tends to be higher on test timestamps closer to training, whereas Yearbook fluctuates and HuffPost and arXiv perform best earliest.Across the datasets, the gap from best ID performance to OOD performance changes over time, indicating that shift characteristics are correlated with timestamps.Source: §5.3 · pp. 7–8
• Most invariant methods show no clear improvement over ERM, while self-supervised and ensemble approaches show no consistent benefit.The baseline comparison also finds ERM to be a strong baseline when the number of training examples is reduced.Source: §5.3 · p. 8
• Incremental training improves OOD performance on arXiv and MIMIC-Readmission but yields the worst OOD performance on HuffPost.The authors relate this pattern to gradual shifts and incremental training’s bias toward recent timestamps; outside Yearbook, incremental methods perform worse than invariant approaches.Source: §5.3 · p. 8
• Most Eval-Stream findings agree with Eval-Fix, with invariant methods outperforming continual methods in more cases except FMoW-Time, HuffPost, and arXiv.The authors caution that invariant methods are not restricted in buffer size in this comparison.Source: §5.3 · p. 8

Relation to Distribution Shift Benchmarks

• Many domain-generalization benchmarks use transformed images or image collections with different styles and backgrounds, rather than natural shifts arising in real-world applications.Examples include ImageNet-A, ImageNet-C [33], PACS, and DomainNet.Source: §6 · p. 9
• Wild-Time targets temporal distribution shifts in real-world applications, unlike WILDS and SHIFTS [72], which address broader distribution-shift settings.WILDS emphasizes domain and subpopulation shifts, while SHIFTS covers weather prediction, machine translation, and self-driving motion prediction.Source: §6 · p. 9

Relation to Continual Learning Benchmarks

• Many continual-learning benchmarks create task sequences through image transformations, class splits, or separate datasets; Wild-Time instead includes natural temporal shifts and Eval-Stream for incremental learning evaluation.Examples of prior benchmarks include permuted MNIST [50], Split CIFAR100, and Visual Domain Decathlon [64].Source: §6 · p. 9

Conclusion and Discussion

• Wild-Time covers six tasks from five datasets across facial recognition, news, and healthcare, and benchmarks 13 approaches across several method families.The benchmark includes continual, invariant, self-supervised, and ensemble learning approaches.Source: §7 · p. 9
• Experiments show large ID–OOD performance gaps across all tasks, and no evaluated invariant, continual, self-supervised, or ensemble approach is consistently more robust than ERM.Source: §7 · p. 9
• The authors aim for Wild-Time to facilitate further research on temporally robust methods for deployment in the wild.Source: §7 · p. 9

A.1.1 Setup

• Yearbook classifies grayscale portraits by gender; Eval-Fix trains on 1930–1970 and tests on 1971–2013, while Eval-Stream treats each year as a timestamp.The dataset reflects changes in fashion and social norms over the decades.Source: §A.1.1 · pp. 16–17

A.1.2 Broader Context

• Across the applications, the section connects temporal robustness to changing human appearance, satellite imagery, healthcare, and news or research content.For clinical decision support, changes in healthcare over time are described as an obstacle to deployment.Source: §A.1.2 · p. 17, §A.2.2 · p. 18, §A.3.2 · p. 19

A.2.1 Setup

• FMoW-Time predicts a region’s functional purpose from a satellite image, using the first 12 years for Eval-Fix training and the last four for testing.Its visual features change over time with human and environmental activity.Source: §A.2.1 · pp. 17–18, §A.2.2 · p. 18

A.3.1 Setup

• MIMIC-Readmission and MIMIC-Mortality use concatenated diagnosis and treatment ICD9 codes to predict readmission within 15 days and in-hospital death, respectively.The records span 2008–2019 in three-year timestamps; Eval-Fix trains on 2008–2013 and tests on 2014–2019.Source: §A.3.1 · pp. 18–19

A.4.1 Setup

• Huffpost classifies headlines into 11 news categories; Eval-Fix trains on 2012–2015 and tests on 2016–2018, while Eval-Stream treats each year as a timestamp.The motivating context is that news style and content change over time.Source: §A.4.1 · pp. 19–20, §A.4.2 · p. 20

A.5.1 Setup

• The arXiv task predicts one of 172 primary paper categories from a title, with Eval-Fix training on 2007–2016 and testing on 2017–2022.The paper notes that research-field evolution is reflected in changing preprint categories.Source: §A.5.1 · p. 20, §A.5.2 · p. 21

B Algorithm Description

• ERM ignores timestamp information and minimizes training loss; continual-learning baselines include fine-tuning, EWC, SI, and A-GEM.EWC constrains adapted parameters near those learned previously, while A-GEM targets average performance across examples in episodic memory.Source: §B.1 · p. 21, §B.2 · pp. 21–22

B.3 Temporal Invariant Learning

• The benchmark also considers invariant-learning methods, mixup, contrastive and clustering-based self-supervision, and stochastic weight averaging.The invariant-learning methods include CORAL, IRM, LISA, and GroupDRO.Source: §B.3 · p. 22, §B.4 · p. 22

C Experimental Details

• Results are averaged over three random seeds; hyperparameters are tuned by cross-validation with grid search, and training samples are drawn uniformly from the ID set or each timestamp.The number of training iterations is reported per dataset and setting, with Eval-Stream updates counted at each timestamp.Source: C Experimental Details · p. 23

C.1 Eval-Fix Split Determination

• Eval-Fix split times are selected from candidate ID/OOD proportions between 40%/60% and 80%/20%, choosing the split with the largest ERM ID–OOD average-accuracy discrepancy..Source: §C.1 · p. 23

C.2 Detailed Set Split Strategy

• Eval-Fix trains and validates on timestamps before the split, then tests on all examples at later timestamps; Eval-Stream evaluates at each time over the next K timestamps.Eval-Fix can be viewed as one Eval-Stream evaluation at the split time.Source: §C.2 · p. 23, Figure 5 · p. 23, Figure 6 · p. 24

C.3.1 General Settings

• Dataset-specific models include a convolutional network for Yearbook, DenseNet-121 for FMoW-Time, a Transformer for MIMIC-IV, and DistilBERT for Huffpost and arXiv.The described settings also specify optimizer, learning rate, batch size, and iteration counts by dataset.Source: §C.3.1 · pp. 23–24

C.3.2 Algorithm-Specific Hyperparameters

• For temporal invariant learning, GroupDRO, CORAL, and IRM sample uniformly across substreams, while CORAL penalties compare features across substream pairs.The stated penalties are λc = 0.1 for MIMIC-Mortality, 1.0 for MIMIC-Readmission, and 0.9 for other datasets; IRM uses λi = 1.0.Source: §C.3.2 · pp. 24–25, Table 8 · p. 25

Continual Learning Methods

• Continual-learning hyperparameters use an A-GEM buffer of 1000, EWC regularization of 0.5 for most datasets and 1.0 for MIMIC-Mortality, and SI regularization of 0.1.Source: Continual Learning Methods · p. 25

Self-Supervised Methods

• SimCLR and SwAV learn representations before fine-tuning with the same labeled training data, while SWA averages parameter values along the SGD trajectory.SimCLR uses image augmentations for Yearbook and FMoW-Time; SwAV uses two views.Source: Self-Supervised Methods · pp. 25–26, Bayesian Methods · p. 26, Table 9 · p. 26

D Results Under Eval-Stream Setting

• Continual-learning methods often emphasize backward transfer and catastrophic forgetting, whereas Wild-Time evaluates forward transfer to future timestamps.The authors note that gains on manually delineated artificial task sequences do not establish gains on natural temporal shifts.Source: D Results Under Eval-Stream Setting · p. 26
• Eval-Stream findings closely match Eval-Fix, and invariant-learning approaches perform slightly better than continual-learning approaches in most tasks.The reported stream results average performance over each evaluated time window; MIMIC tasks report average timestamp performance because testing covers the next three-year block.Source: D Results Under Eval-Stream Setting · p. 26, Figure 7 · p. 27

E.1 Standard Split vs. Mixed Split

• Large standard-versus-mixed-split performance gaps indicate that the ID–OOD gap is not caused by greater difficulty of OOD examples.The mixed split combines training data from all timestamps and evaluates on the original OOD examples.Source: §E.1 · p. 27, Table 12 · p. 27
• Label distributions change over time in all Wild-Time classification datasets.Source: §E.1 · p. 27, Figure 8 · p. 28

E.2.1 Temporal Adaptation Improves Performance

• Temporal adaptation improves invariant-learning performance over vanilla invariant-learning approaches on Wild-Time.Source: §E.2.1 · p. 28, Table 13 · p. 29

E.2.2 Effect of Time Window Size

• Reducing the time-window length marginally worsens invariant-learning performance overall, though the table caption reports improved performance for FMoW-Time.The caption reports that Yearbook performance worsens when L is reduced, but FMoW-Time improves.Source: §E.2.2 · p. 28, Table 14 · p. 29

E.2.3 Non-Overlapping Time Windows

• Overlapping time windows generally yield better invariant-learning performance than non-overlapping windows on Yearbook, Huffpost, arXiv, and both MIMIC tasks.The authors retain non-overlapping-window results in the main-paper table despite the aggregate advantage of overlap.Source: §E.2.3 · p. 28

E.3 Effect of Model Backbones

• Neither invariant-learning nor continual-learning methods make models more robust to temporal shift even when the tested image and text backbones change.The backbone comparison uses FMoW-Time and arXiv under Eval-Fix.Source: §E.3 · pp. 28–29

E.4 Reducing the Number of Training Examples

• With 30% rather than 90% of data allocated for training at each timestamp, ERM still outperforms invariant- and continual-learning approaches.Results are reported under Eval-Fix.Source: §E.4 · p. 29, Table 17 · p. 31

F Datasets without Gradual Temporal Distribution Shifts

• The appendix discusses two additional datasets that do not satisfy Wild-Time’s criteria for gradual temporal distribution shifts.Source: F Datasets without Gradual Temporal… · p. 30

F.1.1 Dataset Setup

• BindingDB predicts drug–target binding affinity from molecular information; Eval-Fix trains on 2013–2016 and tests on 2017–2020, while Eval-Stream evaluates the next three years using PCC.The dataset excludes 2021 because it contains only one month of data.Source: §F.1.1 · pp. 30–31

F.1.2 Baseline Results and Analysis

• OOD average performance drops from 0.724 (mixed split) to 0.357 (standard split), and a sudden 2017 drop leads the authors to exclude Drug-BA.The figure reports performance per test timestamp and results under Eval-Stream.Source: §F.1.2 · pp. 31–32, Figure 9 · p. 32

F.1.3 Broader Context

• Drug discovery is costly and time-consuming, motivating virtual screening that predicts drug–target binding activity as new proteins and compound classes emerge.The passage describes machine learning as a way to reduce screening costs and widen the search space.Source: §F.1.3 · p. 32

F.2.1 Dataset Setup

• Precipitation uses 123 meteorological features to classify nine levels; Eval-Fix trains on October 2018–April 2019 and tests May–August 2019.Eval-Stream treats each month as a timestamp; average and worst-time OOD accuracy measure temporal generalization and robustness to seasonal trends.Source: §F.2.1 · pp. 32–33

F.3 Results

• Precipitation shows no clear performance gap between standard and mixed splits, so it fails the dataset-selection criterion and is excluded from Wild-Time.Figure 10 presents standard and mixed splits alongside Eval-Stream and Eval-Fix results.Source: §F.3 · pp. 33–34, Figure 10 · p. 34

F.3.1 Broader Context

• Precipitation patterns change seasonally, while measurement locations are geographically uneven and polar regions are under-represented [72].Source: §F.3.1 · p. 34

G.1 User Guide of Wild-Time

• Wild-Time provides dataset preprocessing and preprocessed data except for MIMIC-IV; dataset and baseline access is for research, with dataset-specific licenses.The code is released under the MIT license.Source: §G.1 · p. 34

G.2 Using the Wild-Time Package

• The Python package supports dataset loading and custom dataset or baseline construction; standardized method evaluation is planned for the future.Figure 11 illustrates APIs for loading datasets and training a baseline.Source: §G.2 · p. 35, Figure 11 · p. 35

G.3 Maintenance Plan

• Authors maintain Wild-Time and welcome pull requests; others may create documented derivatives that reference the benchmark.Source: §G.3 · p. 35

H.1 Limitations

• The paper does not categorize covariate shift versus concept drift, or precisely identify the causes of sudden shifts.Source: §H.1 · p. 35

H.2 Ethics Discussion

• Yearbook's binary gender prediction excludes nonbinary individuals, may misgender transgender individuals, and may reinforce problematic gender norms.Source: §H.2 · p. 35
• FMoW raises remote-sensing privacy and surveillance concerns, and WILDS reports poor model performance on satellite images from Africa.Poor performance in some geographic regions can harm populations when remote sensing is used for development or humanitarian purposes.Source: §H.2 · p. 36
• MIMIC data are not released; users must obtain PhysioNet credentials and use the provided script to generate the dataset.Source: §H.2 · p. 36

H.3 Comments on Designing Temporally Robust Models

• Most existing approaches cannot effectively mitigate natural temporal distribution shifts.Source: §H.3 · p. 36
• Temporal invariance can change over time, making it important to determine when and how to update an invariant model.Combining temporal invariance with supervised and unsupervised adaptation remains an open problem.Source: §H.3 · p. 36