Source-linked AI summary
When, Not How Much: Evaluating Time-Series Foundation Models on Sparse Events
Daniel Schoess, Florian von Wangenheim
TL;DR
Many sparse-series decisions depend on when activity occurs, but standard TSFM benchmarks do not assess event ranking. Across five sparse datasets, the study compares point forecasts with event-supervised probes of frozen backbones and finds that probes generally rank events better than the forecasts.
Problem
Sparse-series decisions often depend on whether activity occurs, not its size, yet standard benchmarks do not assess which future periods contain events.
Source: §1 · p. 1Method
Across five sparse datasets, the study evaluates 12 TSFMs’ point forecasts against training-free references and trains event-supervised linear probes on six frozen backbones.
Source: §1 · p. 2Results
At most 0.031 AP skill separated TSFM point forecasts from training-free references, while probes improved on point forecasts in 29 of 30 backbone–dataset pairs.
Source: §1 · p. 2Takeaways & Limitations
Sparse-event evaluations should report point forecasts, supervised probes, raw-context and randomized controls side by side, with absolute scores and paired differences.
Source: §5 · p. 9The conclusions cover five public datasets, target-only inputs, a 64-step horizon, and mixed event-and-zero windows, not fully inactive horizons.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Pretrained time-series foundation models (TSFMs) are evaluated as forecasters of future values, yet for sparse series many decisions depend only on which future periods contain activity. Standard benchmarks do not assess this. On five sparse datasets, we rank positions within forecast windows that contain both events and zeros. The released point forecasts of 12 TSFMs improve chance-corrected average precision over training-free references by at most 0.031, and in chance-corrected AUC the median TSFM falls below them on every dataset. With event supervision, linear probes of six frozen backbones improve on their backbone's point forecast in 29 of 30 backbone--dataset pairs. Averaging the predicted quantiles instead of taking their median improves the ranking of most TSFMs that forecast the median, and on two datasets the strongest such outputs rival the probes. The probes' advantage over raw-context learners depends on the dataset, and under the same probe, pretrained features outperform randomly initialized ones for five of six backbones. For sparse-event ranking, released point forecasts thus add little over simple references, whereas lightweight event heads on frozen TSFMs rank events better than these forecasts, and the best of them exceed gradient-boosted trees trained on the raw context on three of the five datasets. More broadly, assessing pretrained forecasters on tasks beyond value forecasting requires reporting their outputs, supervised probes of their representations, and raw-context and randomized controls side by side, since each supports a different conclusion.
1 INTRODUCTION
Sparse-event decisions depend on which future periods contain activity, but standard TSFM benchmarks assess forecast values rather than within-window event ranking. The paper evaluates TSFM outputs and supervised probes on sparse datasets, finding little gain from released point forecasts and stronger event ranking from probes.
1 INTRODUCTION
- When decisions depend on whether activity occurs, the relevant task is ranking future periods by event probability rather than by event size.Examples include allocating inspections, staffed shifts, or service visits that cost the same regardless of event size.Source: §1 · p. 1
- Standard benchmarks do not assess sparse-event ranking because zero-heavy windows are a minority and their metrics score forecast values, not within-window event ordering.The paper notes that benchmarks such as GIFT-Eval and BOOM contain relatively few zero-heavy targets.Source: §1 · p. 1
- For nonnegative targets, the mean combines event probability with event size, while a median forecast is zero whenever event probability is below 50%.Thus, a zero median cannot distinguish event probabilities such as 1% and 49%.Source: §1 · p. 2
- The study compares 12 TSFM point forecasts with a training-free reference and trains event-supervised linear probes on six frozen backbones alongside raw-context and randomized controls.It evaluates five sparse datasets and uses chance-corrected average precision and AUC for within-window rankings.Source: §1 · p. 2
- Probes improve on their backbone’s point forecasts in 29 of 30 backbone–dataset pairs, while point forecasts gain at most 0.031 AP skill over the training-free reference.The median TSFM falls below the reference in AUC skill on every dataset.Source: §1 · p. 2
- The authors recommend reporting outputs, frozen-representation probes, raw-context learners, and randomized controls side by side because each supports a different conclusion.Probes’ advantage over direct event learning and pretrained weights depends on the dataset and backbone, respectively.Source: §1 · p. 3
2 RELATED WORK
Prior work models sparse-event occurrence in several ways, but commonly evaluates forecasts of amounts rather than within-window event rankings. Existing TSFM representation studies and probe controls motivate comparing forecast outputs, supervised probes, and matched controls.
2 RELATED WORK
- Intermittent-demand and count models may represent occurrence separately, yet their forecasts still target an amount per period rather than ranking event occurrences within a window.Examples include hurdle and zero-inflated models, Croston’s method, TSB, and deep renewal processes.Source: §2 · p. 3
- TSFM benchmarks score point or probabilistic forecast errors, while prior frozen-representation probes target labels other than the occurrence of the forecast target.This study probes forecast-target occurrence and scores the probe and released forecast with the same ranking metric.Source: §2 · p. 3
- Prior work cautions that probes can decode information from random features, motivating comparisons with raw-context learners and randomized backbones.The paper reports these controls alongside the absolute scores of both routes.Source: §2 · p. 3
- Table 1 organizes the evaluation around three paired comparisons: forecast versus reference, probe versus forecast and raw context, and pretrained versus randomized backbone.The linear probe is primary; a LightGBM probe is a sensitivity analysis.Source: Table 1 · p. 4
3 THREE QUESTIONS ON FORECASTS, REPRESENTATIONS, AND PRETRAINING
The evaluation pairs within-window event rankings across forecast, representation, and pretraining comparisons. It uses five sparse datasets, frozen TSFM features and matched raw-context and randomized controls, with chance-corrected metrics and paired uncertainty estimates.
3 THREE QUESTIONS ON FORECASTS, REPRESENTATIONS, AND PRETRAINING
- The study asks whether TSFM point forecasts beat a training-free reference, whether supervised probes beat forecasts and raw-context learners, and whether pretraining improves over random initialization.These are paired comparisons on the same windows.Source: §3 · p. 4
3.1 TASK, DATASETS, AND WINDOWS
- Five public datasets cover retail, e-commerce, transport, and web traffic; an event is a positive observed value, and evaluated target windows contain both events and zeros.The task uses 64 target positions following 364 daily or 336 hourly context observations; standard benchmarks have few zero-heavy windows.Source: §3.1 · p. 4
3.2 POINT FORECASTS, FROZEN REPRESENTATIONS, AND RAW CONTEXTS
- The study compares 12 TSFM point forecasts with validation-selected training-free references and probes six frozen backbones using representations aligned to the forecast horizon.Raw-context logistic regression is the matched learner-family control, LightGBM is a strong direct learner, and randomized backbones test the role of pretrained weights.Source: §3.2 · pp. 4–5
3.3 SCORING AND UNCERTAINTY
- Each method ranks target positions within each window, and the primary estimates average window-level skill with equal weight rather than pooling positions.Table 3 compares point forecasts against a validation-selected reference under equal-window weighting.Source: §3.3 · pp. 5–6, Table 3 · p. 6
- The evaluation reports chance-corrected AP and AUC skills, with paired bootstrap intervals that are pointwise and uncorrected for multiplicity.AP emphasizes the ranking top; AUC weights all positive–negative pairs, and estimates are conditional on fitted predictors.Source: §3.3 · p. 6
4 RESULTS
TSFM point forecasts offer little advantage over training-free references for sparse-event ranking, while event-supervised probes of frozen backbones usually improve on those forecasts. Probe comparisons with raw-context learners vary by dataset, as does the benefit of pretrained features over randomized ones.
4.1 POINT FORECASTS GIVE LIMITED GAINS OVER TRAINING-FREE REFERENCES
- Point forecasts gain at most 0.031 AP skill over the training-free reference, while median TSFM AUC skill falls below it on all five datasets.Averaging quantiles improves on most median forecasts and raises the largest reference gain to 0.057 AP skill.Source: §4.1 · p. 6
4.2 LINEAR PROBES OUTPERFORM POINT FORECASTS
- Linear probes improve on their backbone’s point forecast in 29 of 30 backbone–dataset pairs under each metric, with TimeMoE on Amazon the only exception.The exception’s paired intervals span zero.Source: §4.2 · p. 7
- The best linear probes exceed raw-context LightGBM on Amazon, M5, and NYC taxi by up to 0.012 AP skill, but all six trail it on web traffic.On web traffic, the median gap is 0.115 AP skill.Source: §4.2 · p. 8
4.3 THE ADVANTAGE OF PRETRAINED FEATURES DEPENDS ON THE BACKBONE
- Pretrained probes outperform randomized ones in 26 of 30 AP and 25 of 30 AUC comparisons, but Timer’s randomized features lead on three datasets under both metrics.On Favorita, Timer scores 0.172 AP skill with randomized features versus 0.135 with pretrained features.Source: §4.3 · p. 8
- Randomized probes exceed their backbone’s point forecast in 15 of 30 AP and 16 of 30 AUC comparisons, including all six backbones on web traffic.They also do so for Sundial, Timer, and Toto-2.0 on every daily dataset.Source: §4.3 · p. 8
- Outside Amazon, no randomized probe reaches raw LightGBM under either metric; on M5 and NYC taxi, probes’ advantage over it appears only with pretrained weights.On M5, no randomized probe reaches even raw logistic regression.Source: §4.3 · p. 8
5 DISCUSSION AND LIMITATIONS
The discussion finds that frozen probes usually beat point forecasts, but comparisons with raw-context learners and randomized features vary by dataset and backbone. It recommends reporting matched controls and stresses that the conclusions are bounded by the datasets, windows, and representations studied.
5 DISCUSSION AND LIMITATIONS
- Probes beat their backbone’s point forecast in 29 of 30 pairs and raw-context logistic regression in 22 of 30, while results against LightGBM depend on the dataset.On M5, three probes exceeded raw LightGBM by 0.007–0.012 AP skill; all five seeded splits had the same sign.Source: §5 · p. 9
- Quantile averages improve most median-forecast rankings, and the strongest rival most probes on M5 and NYC taxi, so rankings depend on point-forecast reduction.The discussion identifies forecasting event probability and positive amount as a next step.Source: §5 · p. 9
- Timer’s pretrained features encode recent event rate better but event pattern worse than randomized features; that pattern loss tracks probe gaps without explaining dataset reversals.Web traffic is the one dataset where all probes trail raw LightGBM by a wide margin.Source: §5 · p. 9
- The authors recommend reporting point-forecast outputs, raw-context learners, and randomized-backbone controls alongside frozen-representation probes, with absolute scores and paired differences.They also call for stating the ranking metric and averaging unit.Source: §5 · p. 9
- The evaluation covers five public datasets, target-only inputs, 64-step horizons, and mixed-label windows, and does not assess fully inactive horizons.The authors also note that zeros can represent missing records or unavailability, intervals are uncorrected for multiplicity, and ranking metrics do not assess probability calibration or downstream value.Source: §5 · pp. 9–10
A.1 SOURCES AND SERIES CONSTRUCTION
- Retained series need at least two active positions and 40% inactive positions, with the inactive share computed over dataset-specific calendar ranges or series lengths.Amazon, Favorita, and M5 use the common calendar range; NYC taxi uses each calendar year, and web traffic uses series length.Source: §A.1 · p. 13
A.2 WHAT A ZERO MEANS
- A zero marks no recorded activity, not necessarily true inactivity; web-traffic targets include missing page views stored as zeros.The passage reports missing values in 68% of web-traffic zero labels, and methods cannot distinguish them from inactivity.Source: §A.2 · pp. 13–14
A.3 SPLITS, WINDOWS, AND ELIGIBILITY
- Evaluation windows require at least two positive context observations and both target classes, so results exclude wholly inactive target windows.Daily datasets use series-disjoint splits; NYC taxi is split by year because it has too few locations for series-disjoint splits.Source: §A.3 · p. 14
A.4 ZERO-HEAVY TARGETS IN FORECASTING BENCHMARKS
The benchmark audit finds that zero-heavy windows are uncommon in established forecasting benchmarks, while the pretraining-overlap analysis is limited by incomplete disclosures and uneven dataset coverage.
A.4 ZERO-HEAVY TARGETS IN FORECASTING BENCHMARKS
- Windows with at least 80% zeros contain 3.96% of GIFT-Eval target points but only 0.28% of its absolute target mass.In the five study datasets, 46–85% of candidate windows meet the 80% threshold.Source: §A.4 · pp. 14–15
A.5 PRETRAINING-SOURCE OVERLAP
- TSFM documentation classifies source overlap, exclusion, or insufficient disclosure, but web-traffic pretraining may include the evaluated pages and dates.The GIFT-Eval Pretrain web-traffic source contains all 145,063 study pages over the same dates.Source: §A.5 · p. 15
- Disclosed-exclusion comparisons cannot isolate an overlap effect: eight of 12 exclusion pairs are Amazon or M5, while NYC taxi and web traffic contribute only Toto-2.0.Disclosed exclusion also does not verify that evaluated series were absent from pretraining.Source: §A.5 · p. 15
B WITHIN-WINDOW RANKING METRICS
The appendix defines chance-corrected ranking metrics within windows and explains how ties, transformations, and pooling affect their interpretation. It also records a construction showing that positive AP skill need not imply unequal marginal event probabilities.
B.1 THRESHOLD AP AND ITS PERMUTATION EXPECTATION
- AP skill subtracts the conditional permutation baseline AP0 and scales by 1 − AP0; the permutation law normalizes scores rather than asserting label exchangeability.The skill is identically zero for constant scores and is retained when negative.Source: §B.1 · p. 17
- For tie-free scores with n = 64 and k = 63, AP skill can reach about −3.0, and its normalization up-weights windows with few zeros.Excluding such windows leaves all reported signs unchanged.Source: §B.1 · p. 17
B.2 AUC SKILL AND INVARIANCE TO PER-WINDOW TRANSFORMATIONS
- A strictly increasing per-window transformation that preserves exact ties leaves within-window order, AP, and AUC pair credits unchanged.Pooling positions across differently transformed windows need not preserve that invariance.Source: §B.2 · p. 17
B.3 POOLING CAN REWARD BETWEEN-WINDOW ACTIVITY
- Pooling two constant-score windows yields AP skill 1/15 and AUC skill 4/15, despite zero skill within each window.The pooled ranking rewards the higher score of the more active window, although neither window’s score discriminates among its positions.Source: §B.3 · p. 17
B.4 POSITIVE AP SKILL UNDER EQUAL MARGINAL EVENT PROBABILITIES
- With equal marginal event probabilities, mean AP skill is 1/46 while mean AUC skill is zero, so positive AP skill need not indicate unequal marginal event probabilities.This construction limits that interpretation; it does not invalidate AP or establish that AUC is generally preferable.Source: §B.4 · p. 17, Table 10 · p. 18
C.1 POINT FORECASTS AND REPRESENTATIONS
- The probe study covers six of ten TSFMs meeting the extraction criterion, omitting four strong point forecasters on Favorita, M5, and web traffic.Whether probes of those omitted backbones gain similarly over their forecasts or randomized copies remains open.Source: §C.1 · p. 18
C.2 TRAINING-FREE REFERENCES
Training-free references are selected from seasonal profiles and ramps using validation AP skill. Supervised probes and raw-context learners then use matched data splits and validation-based model selection.
C.2 TRAINING-FREE REFERENCES
- The training-free references are event-rate and mean seasonal profiles plus increasing and decreasing ramps, with candidates chosen by validation AP skill.Daily datasets use a weekly period; NYC taxi uses daily and weekly periods.Source: §C.2 · p. 18
- The weekly mean profile is selected for every Favorita and M5 split, while selected references vary across web traffic, Amazon, and NYC taxi.Web traffic and Amazon use decreasing ramps on some splits; NYC taxi uses the hour-of-day mean profile.Source: §C.2 · p. 19
C.3 PROBE AND RAW-CONTEXT TRAINING
- Supervised procedures train on at most 100,000 windows, and raw-context and representation learners share training, validation, and test windows.The linear probe maps frozen representations to position scores; raw logistic regression uses standardized context, and both select L2 penalty by validation AP.Source: §C.3 · p. 19
D COMPLETE RESULTS BY DATASET
This section presents complete dataset-level comparisons of TSFM point forecasts, raw-context learners, and probes, followed by views of AP/AUC skill and event counts among the highest-ranked positions.
D COMPLETE RESULTS BY DATASET
- The dataset tables compare TSFM point forecasts and raw-context learners with a training-free reference, then report probe contrasts against backbone forecasts and raw-context learners.The tables pool test windows across splits with equal-window weighting; equal-series values are reported in Appendix E.1.Source: D COMPLETE RESULTS BY DATASET · p. 19, Table 14 · p. 20, Table 15 · p. 21
D.1 AP AND AUC SKILL OF THE PROBED BACKBONES
- Figures 5 and 6 show AP skill and AUC skill, respectively, for six probed backbones across five datasets, with dataset-specific axes and a broken web-traffic axis.Figure 5's top row repeats Figure 2; Figure 6's top row repeats Figure 3, and per-backbone intervals are in Tables 14–18.Source: §D.1 · p. 26, Figure 5 · p. 27, Figure 6 · p. 27
D.2 EVENTS AMONG THE EIGHT HIGHEST-RANKED POSITIONS
- Table 20 reports the expected number of events among each window's eight highest-ranked positions, equivalent to 8× precision@8, against random-order and oracle benchmarks.Windows are scored only when they contain both events and zeros; these conditional values are not expected deployment yields.Source: §D.2 · p. 26, Table 20 · p. 28
E.1 EQUAL-WINDOW AND EQUAL-SERIES AVERAGING
This section tests how averaging units and evaluation subsets affect point-forecast and probe comparisons, and reports split stability and sensitivity to sparse-window composition and web-traffic missingness.
E.1 EQUAL-WINDOW AND EQUAL-SERIES AVERAGING
- Equal-window averaging weights series by their number of evaluated windows, whereas equal-series averaging weights every series equally; the averages diverge when skill covaries with window count.On the other four datasets, median and maximum TSFM differences retain their signs under both weightings, except Amazon's largest AUC difference shifts from −0.006 to +0.002.Source: §E.1 · p. 29
- On web traffic, equal-series averaging makes even the best TSFM trail the reference by 0.039 AP skill and 0.064 AUC skill, reversing the equal-window comparison.For TimesFM-2.5, the AP difference changes from +0.0064 [0.0032, 0.0095] to −0.0395 [−0.0500, −0.0291]; excluding the identified event-run windows gives −0.0021 [−0.0085, 0.0041].Source: §E.1 · p. 29, Table 21 · p. 30
E.2 SPLIT STABILITY
- All five seeded splits agree in 169 of 192 contrasts, but overlapping training series mean these counts measure robustness to re-splitting and refitting, not independent replication.Of 174 contrasts whose pooled interval excludes zero, 168 agree across all five splits; the two exceptions on Favorita each have one opposite-sign split.Source: §E.2 · pp. 29–30
E.3 WINDOWS WITH FEW ZEROS
- Windows with at least 58 events among 64 positions are at most 14% of evaluated windows but carry up to 63% of TimesFM-2.5's forecast weight; excluding sparse-zero windows changes no reported AP-skill signs.The same sign and interval results hold when excluding windows with at least 52 events.Source: §E.3 · p. 30
E.4 WEB-TRAFFIC WINDOWS WITHOUT MISSING VALUES
- On fully observed web-traffic windows, raw LightGBM remains ahead of all probes, while the filtered subsets retain at most 15% of windows and differ in composition as well as missingness.The authors caution that smaller gaps on these subsets do not measure how missing values change full-dataset estimates; all models were trained with missing values stored as zeros.Source: §E.4 · pp. 30–31
E.5 PROBE TRAINING-SIZE SENSITIVITY
The section examines probe training-size sensitivity, compares linear and GBM probes, and describes near-flat and near-zero TSFM point forecasts.
E.5 PROBE TRAINING-SIZE SENSITIVITY
- Raising the training cap from 100,000 to 150,000 windows, or to all 105,189 Amazon windows, changes AP skill by at most 0.0054 and preserves all eight pretrained-minus-randomized difference signs.This check covers Chronos-2 and Timer only and does not establish that skill has saturated.Source: §E.5 · pp. 31–32, Figure 7 · p. 32
E.6 GBM PROBE
- Linear probes exceed GBM probes for all six backbones on Amazon, Favorita, M5, and NYC taxi, while GBM probes are higher for all six on web traffic; all 60 paired intervals exclude zero.On web traffic, GBM narrows but does not close the gap to raw LightGBM. For GBM randomized controls, Chronos-2's eight differences are positive, while Toto-2.0's comparisons vary by dataset and are negative on web traffic.Source: §E.6 · pp. 32–33, Figure 8 · p. 33
F.1 NEAR-FLAT AND NEAR-ZERO POINT FORECASTS
- Outside Amazon, median forecasts are near-zero in 10.7–54.3% and near-flat in 14.4–65.3% of scored windows, whereas five TSFMs using means or model point outputs have rates no higher than 1.7% and 3.9%.These shares describe forecast shape, not ranking skill; exactly constant forecasts occur in at most 5.2% of any TSFM–dataset pair's scored windows, and exactly zero forecasts do not occur.Source: §F.1 · p. 33
F.2 RANKINGS FROM QUANTILES
The section tests alternative rankings from TSFM quantiles and supervised readouts of forecast outputs. Quantile averages improve most median-forecasting TSFMs, while comparisons with probes and raw-context learners depend on the dataset and readout setup.
F.2 RANKINGS FROM QUANTILES
- The approximate event-probability score 1 − F̂(0) is lower than the point forecast in 48 of 50 AP-skill and 46 of 50 AUC-skill comparisons, with no interval above zero.It is computed from saved quantiles or samples and is an approximation, not a native TSFM event probability; outside the saved quantile range, its value depends on extrapolation.Source: §F.2 · pp. 33–34, Table 26 · p. 35
- The strongest quantile average exceeds most probes on M5 and NYC taxi under both metrics, but exceeds the best probe only on NYC taxi in AP skill; at least five probes remain ahead on Favorita and web traffic.Within-backbone probes exceed their own backbone's quantile average in 28 of 30 point-estimate comparisons; these gains do not establish that probes beat the strongest outputs across TSFMs.Source: §F.2 · p. 34
- Quantile averages improve on median forecasts in 27 of 35 AP-skill and 23 AUC-skill comparisons; the largest TSFM advantage over the reference rises from 0.031 to 0.057 AP skill.The largest quantile-average gain is by Chronos-2 on NYC taxi; the increase in maximum AP skill is a descriptive test-set maximum.Source: §F.2 · p. 34, Table 27 · p. 36
F.3 SUPERVISED READOUT OF THE OUTPUTS
- On Favorita, M5, and web traffic, the supervised readout is below every probe of its own backbone under both metrics, while on Amazon and NYC taxi median AP-skill gaps are at most 0.009.The readout itself improves on the point forecast most on Favorita, by 0.038–0.056 AP skill.Source: §F.3 · p. 36, Table 28 · p. 37
- Readout-versus-probe gaps are not matched comparisons because their training populations, sample sizes, and penalty selection differ.Cross-fitting also changes raw-context logistic regression AP skill by −0.014 to −0.004 on daily datasets and by +0.086 on NYC taxi, where training years differ.Source: §F.3 · p. 36
G REPRESENTATION DIAGNOSTICS
Representation diagnostics show that Timer’s pretrained features reverse the usual probe advantage on Favorita, while effective-rank changes alone do not explain that reversal. Recent-pattern readouts track probe differences more closely than effective rank or event-rate readouts, though these correlations are descriptive and based on few points.
G REPRESENTATION DIAGNOSTICS
- The diagnostics use effective rank and ridge readouts of recent event rate and binary event patterns, fitted and scored on disjoint 60/20/20% partitions.Table 29 reports randomized-to-pretrained effective-rank ratios and pretrained-minus-randomized held-out R2 for the rate and pattern readouts.Source: G REPRESENTATION DIAGNOSTICS · p. 37, Table 29 · p. 37
- −0.038 [−0.039, −0.036] AP skill is the pretrained-minus-randomized difference for Timer on Favorita.On Amazon, Timer’s AUC interval lies below zero while its AP interval spans zero.Source: G REPRESENTATION DIAGNOSTICS · p. 37
- Timer has the largest pretraining-related effective-rank reduction, but Chronos-Bolt shows a similar reduction while its pretrained probe leads on all five datasets.Thus, reduced effective rank alone does not produce Timer’s reversal.Source: G REPRESENTATION DIAGNOSTICS · p. 37
- Pattern-readout differences track probe differences more closely than effective-rank or event-rate differences, but the relationship reverses within Timer.Correlations are descriptive and uncorrected; within-dataset and within-Timer estimates rest on five or six points.Source: G REPRESENTATION DIAGNOSTICS · p. 38