Source-linked AI summary
A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out
Mahdi Naser Moghadasi, Faezeh Ghaderi
TL;DR
Public benchmarks for time-series foundation models predate the models, so strong scores cannot distinguish generalisation from memorisation. The paper builds a post-release hold-out across five domains and tests whether performance differences follow intrinsic series properties or corpus familiarity. Pretrained models win 5 of 7 groups, but the strongest advantage aligns with a domain represented in one model’s pretraining corpus, while several scope and evidence limitations remain.
Problem
Public benchmark archives predate evaluated models, leaving reported performance unable to distinguish generalisation from memorisation.
Method
The paper evaluates thirteen forecasters on seven groups from five domains using observations published after every model’s release, then compares intrinsic-series explanations with corpus familiarity.
Results
Pretrained models win 5 of 7 groups, and their largest advantage occurs on Wikipedia pageviews, a domain represented in one model’s pretraining corpus.
Takeaways & Limitations
A temporal hold-out removes memorisation of a test window but not familiarity with a domain, so evaluation should relate domain hold-outs to disclosed corpora.
Takeaways & Limitations
The corpus-overlap result is an association rather than a demonstrated mechanism, and broader testing requires corpus disclosure across several models.
Abstract
from arXiv · showhide
Time-series foundation models are evaluated almost exclusively on public archives that predate them, so a strong score cannot be separated from having seen the test set during pretraining. The obvious remedy is a hold-out that postdates the models. We build one: thirteen forecasters -- four classical, three trained per dataset, six pretrained -- on seven groups drawn from five domains, every observation published after the last model was released, and every dataset rebuildable without an API key. Under this protocol pretrained models win 5 of 7 groups, lose one to a Theta baseline, and on daily exchange rates are indistinguishable from a seasonal naive forecast, along with every other method tested. We then ask what separates the wins from the losses, and report a negative result: the two intrinsic properties one would reach for -- seasonal strength and spectral entropy, measured on the input window -- do not account for the pattern, and seasonal strength is if anything negatively associated with the advantage. What does track it is corpus familiarity. Our largest gain (28% lower MASE than the best classical method, on weekly Wikipedia pageviews) falls on Wikipedia pageviews, the domain TimesFM's authors describe as the bulk of its pretraining corpus, at the same granularities and differing only in time window. Within the pretrained family, where every model forecasts identical series so that series difficulty cancels, the TimesFM family outranks the Chronos family by -0.53 ranks on Wikipedia against -0.09 everywhere else (1,500 vs. 754 series, Mann-Whitney p < 1e-5). We conclude that a temporal hold-out removes memorisation of a window but not familiarity with a domain, that benchmarks therefore need domain hold-outs stated relative to disclosed corpora, and that the practitioner's question is less which model is better than whether their domain is one the model was raised on.
1 Introduction
Public time-series benchmarks predate the models evaluated on them, so their scores cannot distinguish generalisation from memorisation. The paper proposes a post-release, contamination-free hold-out and finds that pretrained-model advantages vary by domain and track corpus familiarity rather than the tested intrinsic series properties.
- Motivation: Existing public benchmarks cannot separate genuine generalisation from memorisation because they predate the models and were publicly available for corpus construction.This ambiguity applies to widely used archives such as M-competition data, ETT, and Monash.
- Protocol: The evaluation uses seven groups from five domains whose test windows postdate every compared model release.The sources are continuously published by institutions and are intended to be rebuildable without API keys.
- Motivation and headline finding: Post-release hold-outs prevent test-window memorisation but do not remove familiarity with domains represented in pretraining corpora.The strongest gains occur on a domain matching one model’s corpus, with the same source and granularities but a later time window.
- Findings: Pretrained models are not uniformly superior: they win some domains and fail to outperform alternatives on others.The paper’s contribution list highlights a Theta win in one domain and no advantage over seasonal naive in another.
- Findings: Neither seasonal strength nor spectral entropy explains where pretrained-model advantages appear.The paper instead reports evidence consistent with corpus familiarity as the relevant pattern.
- Evaluation principle: The study prioritizes significance-aware comparisons, reporting rank intervals and corrected paired tests rather than relying on decimal-place differences.This reporting choice is intended to distinguish robust gaps from apparent leaderboard differences.
2 Related work
Prior forecasting benchmarks and evaluations largely use public data that predate foundation models, leaving contamination difficult to distinguish from generalisation. The paper positions its temporal hold-out as complementary to corpus audits and membership detectors because it reveals residual domain familiarity even when every test observation is clean.
- Compared methods: The paper compares pretrained forecasters with supervised target-dataset models and classical baselines using forecasting-specific rank-based evaluation.The related-work framing includes N-BEATS, automated ARIMA and ETS, and Theta among the relevant comparison families.
- Benchmarks and contamination: Published forecasting benchmarks cannot separate generalisation from recall because their public data predates model training.This limitation concerns benchmark lineage rather than a criticism of the archives’ construction.
- Contamination audits: Corpus audits detect whether datasets entered training or fine-tuning corpora, but do not quantify inflation of reported zero-shot performance.The cited audit work motivates evaluation on continuously sourced, post-release test data.
- Contamination detectors: Adaptation-dynamics detectors answer dataset-membership questions without requiring a clean test set, whereas this paper evaluates observations that did not yet exist during training.The approaches address related but distinct contamination questions.
- Complementarity: A temporal hold-out and a corpus audit are complementary: the hold-out exposes domain familiarity that a membership audit correctly labels clean.No test observation can be present in a corpus when the observation postdates the model checkpoint.
3 Protocol
The protocol fixes a common forecast origin, uses continuously published keyless sources, and applies shared selection, horizon, context, metric, and significance procedures. It emphasizes rank-based comparisons because means can be dominated by degenerate series and because forecast rankings depend on the chosen error measure.
- 3.1 Contamination-free hold-out: Every model is evaluated from 1 January 2026, with test observations unavailable when any compared checkpoint was trained.The protocol uses model release dates to guarantee that the evaluated observations postdate the models.
- 3.2 Data sources: All five domains come from continuously published institutional sources that are retrievable without registration or API keys.The resulting groups are listed in Table 1 and are not redistributions of existing benchmarks.
- 3.2 Data sources: The dataset includes Wikipedia pageviews, weather, air quality, electricity, and exchange rates to vary periodicity, smoothness, spikiness, and expected forecast difficulty.Air quality is paired with weather to separate hourly sampling from smoothness, while exchange rates test whether any method beats naive forecasting.
- 3.2 Data sources: Series selection is fixed before results are seen, with constant or near-constant series excluded when MASE is undefined.Exactly one Danish hydro-generation series is removed because its in-sample seasonal-naive error is zero.
- 3.3 Horizons, seasonality and context: Horizons and seasonal periods mostly follow M4 conventions, with explicit departures for weekly seasonality and monthly Wikipedia’s shorter available hold-out.Weekly uses m = 1, while monthly Wikipedia uses an eight-step horizon because only eight complete hold-out months exist.
- 3.3 Horizons, seasonality and context: All models receive identical input contexts, including a 2,016-point cap for hourly groups, to keep comparisons consistent across architectures.The loader truncates inputs uniformly rather than tailoring context lengths per model.
- 3.4 Metrics and significance: The study reports four measures and uses average ranks with Nemenyi intervals plus Holm-corrected Wilcoxon tests for model comparisons.Signed-rank testing is chosen because panel MASE is strongly right-skewed, while weighted quantile loss is proper for probabilistic evaluation.
- 3.4 Metrics and significance: A single degenerate series can shift mean MASE by six orders of magnitude without changing the rank-based conclusion.This motivates the paper’s preference for ranks over means when summarizing group performance.
4 Results
Across seven groups, pretrained models win five, lose one to Theta, and are indistinguishable from all methods on exchange rates. Their advantage is not explained by seasonal strength or spectral entropy, but aligns with corpus familiarity, especially for Wikipedia pageviews.
- 4.1 Accuracy: Pretrained models win 5 of 7 groups, lose one to Theta, and are indistinguishable from every method on exchange rates.The exchange-rate null is not treated as proof of no effect, but every model's point estimate is essentially seasonal naive.
- 4.2 The seven groups, one at a time: TimesFM-3 ranks first on daily and weekly Wikipedia pageviews, while Moirai-2 ranks first monthly and every classical method is clearly lower.Wikipedia pageviews produce the strongest pretrained result in the study.
- 4.3 Series properties: Neither seasonal strength nor spectral entropy explains where pretrained models gain, and seasonal strength is if anything negatively associated with the advantage.Wikipedia has little measurable seasonality yet the largest gains, whereas weather is strongly seasonal and shows a smaller advantage.
- 4.4 Corpus familiarity survives the hold-out: The TimesFM family is 0.53 ranks ahead of the Chronos family on Wikipedia versus 0.09 ranks elsewhere, across identical series, with Mann–Whitney p < 10^-5.The comparison uses 1,500 Wikipedia series and 754 series from the other domains; the reported rank-biserial effect is -0.13.
- 4.4 Corpus familiarity survives the hold-out: The study interprets the pattern as corpus overlap rather than capability, while acknowledging that the design supports this hypothesis rather than demonstrating its mechanism.A decisive next experiment would stratify benchmark domains by membership in each model's disclosed corpus.
- 4.6 Probabilistic forecasts, where the ordering changes: Pretrained models are systematically overconfident: every 80% interval covers less than 80% of outcomes, while automatic classical methods cover above nominal.Moirai-2 misses nominal coverage by 9 percentage points and Theta exceeds it by 9 points.
- 4.7 Cost: Automatic ARIMA costs a median 54× more per series than a pretrained forward pass and is beaten in accuracy by the best pretrained model on all seven groups where it completes.The cost ratio ranges from 5× on weekly series to 2,689× on hourly weather.
5 Failure modes we hit
The paper documents failures that materially affect benchmark interpretation, including pathological metrics, divergent baselines, and hardware-dependent timing. It also verifies that most supposedly deterministic models produce identical forecasts across seeds.
- A single near-zero hydro series moved the group’s mean MASE by six orders of magnitude, although the rank-based conclusion was unchanged.The series was excluded because its seasonal-naive denominator was 10^-9, making small positive predictions score 4 × 10^8 and 6 × 10^7.
- AutoETS produced weighted quantile losses of order 10^14 on two Wikipedia groups by exploding on spiky, heavy-tailed traffic.Its negative quantiles for a strictly non-negative quantity exposed a failure invisible in the finite point-forecast MASE table.
- Accelerator fallbacks changed timing for TimesFM and Moirai-2 without changing forecasts, so the reported cost comparison uses CPU fallbacks where required.
- Nine models rerun under different seeds agreed exactly to the last bit, with maximum |∆| = 0.This check supports treating their single reported results as deterministic rather than sampled draws.
6 Limitations
The corpus-familiarity interpretation is supported but not demonstrated mechanistically, and the evaluation has limited temporal, domain, and sample coverage. These constraints make the headline association provisional rather than universal.
- The corpus-overlap finding is an association rather than a demonstrated mechanism, based on one publicly described corpus and one matching domain.The within-family comparison narrows alternative explanations but does not close them; the reported rank-biserial correlation is -0.13.
- The evaluation uses one forecast origin per group, relies on partly unverifiable pretraining cutoffs, and covers only five domains.Rolling-origin evaluation would estimate temporal variance, while undisclosed cutoffs are replaced by model release dates.
- The openly published datasets are a biased sample toward data likely to appear in pretraining corpora, which cuts against generalizing the headline effect.
7 Conclusion
Temporal hold-outs show that pretrained forecasters’ advantage depends more on corpus familiarity than on intrinsic series properties. The benchmark therefore needs domain hold-outs and disclosed pretraining corpora, while the released tooling enables further tests.
- Pretrained models are strong on some domains, unnecessary on others, and useless where nothing works; periodicity and entropy do not explain that spread, but pretraining-domain familiarity does.
- Benchmarks must hold out domains as well as dates and disclose pretraining corpora so users can assess whether their data resembles what a model was raised on.
- The authors release fetchers, a harness, and per-series losses so the negative result and corpus effect can be tested on additional domains.
8 Dataset documentation
The benchmark is rebuilt directly from institutional open-data publishers through credential-free fetchers, with manifests exposing panel changes. Its scope is constrained by small groups, reanalysis data, one-country coverage, and selection toward continuously published open data.
- Every group is rebuilt from its publisher by a no-credentials script, with manifests recording request parameters, retrieval time, series count, and output checksum.
- The sources are institutional open-data services covering Wikimedia pageviews, weather, air quality, electricity, and exchange rates, each governed by the publisher’s reuse terms.
- The panel is defined by publisher rules rather than a frozen file, so series additions or retirements appear through changed counts or checksums.
- The repository generates machine-readable Croissant metadata from fetcher manifests, including each source’s series count, date range, and SHA-256.
- Two groups are small, weather and air quality use smoother reanalysis products, electricity covers one country, and open continuous publishers bias the domain sample.
Reproducibility
The project makes its fetchers, reproduction scripts, tables, tests, and per-series losses available so the benchmark and alternative analyses can be rerun.
- The repository provides code, data fetchers, and per-series results for reproducing the benchmark.
- Dedicated scripts reproduce the holdout runs, significance tests, tables, and quoted numbers, while released per-series losses support alternative tests without rerunning models.