Evaluation Choices Decide the Forecasting Leaderboard: Evidence from a Production Marketplace Panel

1 Why this paper exists

• Forecasting benchmarks are meant to answer which method to use, yet the paper argues that the winner can be a property of the benchmark rather than the methods.Prior work has examined evaluation errors, including error measures, data partitioning, and statistical testing.Source: §1 · p. 1
• The paper measures how far the conclusion moves when evaluator decisions change while the production panel, forecast horizon, period, and methods stay fixed.It also runs the released protocol on a public retail panel to test whether the result is specific to the firm.Source: §1 · p. 2
• The released protocol changes one evaluation choice at a time and measures winner movement, then replays a deployed selection rule against a floor and a ceiling.Its results are reported on two panels.Source: §1 · p. 2
• A deployed system must choose a method each month, so the paper measures the value of its selection rule against a floor and a ceiling.The authors position this forecasting study alongside earlier work in another field that also found evaluation choices could reverse or move results with models fixed [17].Source: §1 · p. 2

2 The setting and the data

• Market totals are smooth because buyer fluctuations largely cancel, while individual buyer series are often spiky, short, and dominated by a few large accounts.The business earns each dollar again monthly from project work, with no subscription floor.Source: §2 · p. 2
• The panel contains 5,779 buyers from January 2021 to July 2026, but only 1,887 are eligible at one or more evaluation origins after the 13-month history threshold.The remaining buyers form an abstention cohort and are not scored.Source: §2 · p. 2
• Buyers are grouped as Smooth, Erratic, Intermittent, or Lumpy using demand frequency and amount variability, with classes recomputed from each origin’s training window.The standard cut-offs are an average demand interval of 1.32 months and a squared coefficient of variation of 0.49; using the full series would leak future information.Source: §2 · pp. 2–3

3 What we compare, and how

• The roster includes 24 forecasting methods plus a textbook reference, spanning classical, intermittent-demand, panel-trained machine-learning, foundation-model, and baseline methods.Six time series foundation models are used without training on the production data; the production method is seasonal naive with drift, alongside plain seasonal naive as the scale-free reference.Source: §3 · p. 3
• Every method uses the production system’s own implementation rather than a study-specific reimplementation.This avoids discrepancies between a re-coded method and the deployed version.Source: §3 · p. 3
• Eight rolling cutoffs spaced three months apart forecast the following six months, with panel-trained methods retrained at each cutoff using only earlier data.Six months is the deployed selection rule’s cross-validation block length, so it is the horizon at which methods are chosen.Source: §3 · p. 3
• At twelve months, the ranking correlates 0.98 with the six-month ranking, no method moves more than three places, and the same six methods remain at the top.Chronos T5-Large is excluded on compute grounds; the production baseline ranks twenty-third of twenty-four, with sixteen methods significantly beating it.Source: §3 · p. 3
• MASE and RMSSE divide errors by plain seasonal naive error, while Table 1 contrasts WAPE on the market total with RMSSE across buyers.Diebold–Mariano tests compare methods with the production baseline; Friedman–Nemenyi tests compare all methods, with false discovery rate control within demand classes.Source: §3 · pp. 3–4, Table 1 · p. 4
• Friedman blocks are correlated because a buyer can contribute up to eight origins, so the reported critical difference is narrower than it should be.Using one origin per buyer widens the critical difference from 0.37 to 0.92, while mean rank correlation with the full ranking is 0.98 across twenty draws; four methods score 10,341 of 11,516 blocks.Source: §3 · p. 4

4 Result one: the unit of analysis reverses the conclusion

• The production baseline ranks second of nineteen on the market total, with no statistically distinguishable winner, but twenty-third of twenty-five per buyer, where nineteen methods beat it.On a shared roster and origins, it ranks second on the total and seventeenth per buyer, confirming that the reversal is driven by the unit of analysis.Source: §4 · p. 5
• Aggregation cancels buyer swings: the market total moves 7% of its average level in a typical month, versus 32% for the median scored buyer.The authors distinguish their measured between-level rank reversal from prior work on ranking instability within a level [14].Source: §4 · p. 5
• Weighting buyers by average monthly volume moves the baseline from twenty-third to twentieth of twenty-five, with nineteen methods still ahead.Pooling absolute errors over dollars rather than buyers gives the same picture, so equal buyer weighting does not account for the reversal.Source: §4 · p. 5

4.1 A second choice hiding inside the first

• The Diebold–Mariano test finds no significant winner after pooling 126 market-total errors, but almost everything is significant at p < 0.0001 when tens of thousands of per-buyer errors are pooled.A monthly-average check retains significance for all nineteen methods that beat the baseline; the authors note cross-buyer dependence could matter when margins are thinner.Source: §4.1 · pp. 6–7

4.2 From a contrast to a curve

• As the panel is split from one total into 625 groups, the baseline’s Friedman mean-rank placement worsens monotonically from fifth to seventeenth of nineteen.Most of the decline occurs by five groups; at 625 groups, only 27% of aggregates are Smooth, linking disaggregation to a changed demand mix.Source: §4.2 · p. 7, Table 3 · p. 7
• Random grouping preserves a monotone baseline-rank decline, while the public M5 hierarchy places it first at the total and last across 30,490 item–store series.On M5, rank correlation with level size is 0.93, or 0.94 on the Friedman convention.Source: §4.2 · p. 8

4.1 A second choice hiding inside the first

• The public M5 hierarchy also shows no significant challenger at aggregate levels, but all six challengers separate at the item–store level.The authors caution that significance can lose information on large panels and report Nemenyi mean-rank separation alongside tests.Source: §4.1 · p. 7, §4.2 · p. 8

5 Result two: the winner depends on the demand pattern

• Three methods lead the four demand classes, and the pooled winner leads only two.Exhibit 4 splits the comparison by demand class.Source: §5 · p. 9
• The split’s size is measured on one production panel; intermittent-demand methods win on intermittent demand.The authors describe that result as the taxonomy doing what it was built to do.Source: §5 · p. 9
• Because pooling is the default, a single reported winner can falsely imply it leads everywhere; the authors recommend routing series by behavior.Section 7 measures the value of that routing against the alternative.Source: §5 · p. 9

6 Result three: scoring intervals reorders the field

• Table 5 reports deployed conformal-band coverage and compares method rankings on point accuracy with rankings on interval quality.The interval metric is mean scaled interval score; coverage is calibrated at each origin using that buyer’s earlier origins.Source: Table 5 · p. 10
• The bands are calibrated where demand is regular and under-cover where it is not.The deployed system’s nominal 95% bands deliver 92% coverage on Smooth demand and about 74% on Lumpy and Intermittent buyers.Source: Table 5 · p. 10, §6 · p. 10
• The comparison isolates the point forecaster behind each band, not a model’s built-in uncertainty output.Every method’s band is built by split conformal prediction on that method’s point errors.Source: §6 · p. 10

7 Result four: what the deployed selection rule is worth

• The rule closes 55% of the floor-to-ceiling distance, improves on the floor by 23.5%, and wins in 77% of blocks.The ceiling chooses the single best method at each origin with hindsight; the pattern holds in every demand class.Source: §7 · p. 11
• Always serving DeepAR closes 52% of the same gap, so the rule’s edge over one champion is real but modest.The best method with full support closes 54%; the rule wins in every demand class except Lumpy.Source: §7 · p. 11
• The authors present this as measured value from a reasonable, simple, deployed rule, not as a claim that the rule is optimal.The replay requires validation history and is measurable only at origins with at least three earlier origins.Source: §7 · p. 11
• With disjoint folds, the rule still closes 55% of the gap and improves on the floor by 26% on 2,022 blocks.The authors conclude that overlapping folds were not responsible for the replay result.Source: §7 · p. 11
• Table 7 applies the released protocol to monthly M5 data and reports unchanged placements under non-overlapping folds and MASE reranking.Its verdict uses Diebold–Mariano tests on pooled squared-error differentials with Benjamini–Hochberg adjustment.Source: Table 7 · p. 12

M5 replication

• At the M5 market total the baseline ranks first of seven, but per item–store it ranks last, with all six challengers beating it at p < 0.001.The ten store totals give the same answer; the baseline is also last across all four demand classes, and the pooled Diebold–Mariano test finds nothing at the total but significance per series.Source: §8 · p. 12

Selection-rule replay

• 64.7% of the floor-to-ceiling distance is closed by the replayed selection rule per item–store, and it wins in 77% of 101,729 blocks.With non-overlapping folds, it still closes 60.5% of the distance on 56,772 blocks.Source: §8 · p. 12

Extended M5 roster

• With five zero-shot foundation models added, FlowState alone beats the baseline by a distinguishable margin at the M5 total, while the baseline remains second there and last per item–store.The extended roster excludes Chronos T5-Large on compute grounds; damped simple smoothing leads per item, ahead of TimesFM and FlowState by a nose.Source: Table 8 · p. 13, §8 · p. 13

Interval calibration

• On M5, the baseline’s nominal 95% conformal band covers 86% of Smooth items, 79% of Intermittent items, and 68% of Lumpy items.The passages attribute the calibration failure to spiky data and report that it follows the data across panels.Source: §8 · pp. 13–14

Interval ranking

• Point and interval rankings correlate at 0.89 overall across seven reference methods, versus 0.39 across twenty-five production methods; extending the roster to twelve leaves M5 at 0.89.On Intermittent items, the M5 correlation is 0.68; the passages do not locate the cause of the production-panel reordering.Source: §8 · pp. 13–14

Tourism boundary

• On the 304-series Australian tourism panel, the baseline falls from first at the national total to last per series, but only one of six challengers separates significantly.The fade persists against an M5 sample thinned to 304 series, supporting a data-shape boundary rather than a sample-size explanation.Source: §8 · p. 14

Unit of analysis

• State the unit of analysis before reporting a winner; on this panel, it changes the answer more than any difference between methods.The recommendation is grounded in the paper’s unit-of-analysis reversal.Source: §9 · p. 14

Statistical reporting

• Report rank separation alongside p-values when the panel is large, because significance across tens of thousands of pooled errors can stop discriminating.The practical concern is that large pooled samples can make significance nearly automatic.Source: §9 · p. 14

Demand classes

• Split results by demand class before naming a winner: three methods lead the four classes, while the pooled winner leads only two.The class-specific winners differ from the pooled outcome.Source: §9 · p. 14

Interval scoring

• Score intervals separately when intervals are deployed; point-accuracy rankings fail to transfer on the production roster and carry no information about interval quality on the hardest series.The public-panel rankings agree more closely, so the transfer failure is a risk to check rather than a law; the check costs one rolling calibration run.Source: §9 · p. 14

Selection-rule value

• Measure the deployed selection rule’s value, not only which individual method has the lowest error.The business question is what the system is worth, which the paper distinguishes from model ranking.Source: §9 · p. 14

Replication scope

• The full twenty-five-method leaderboard remains evidence from one firm, panel, and industry, while public-panel replications use reference methods plus five zero-shot foundation models rather than the full production roster.The replicated design effects do not establish that the production leaderboard itself generalizes; the interval ranking reversal also remains production-only evidence with its cause unlocated.Source: §10 · pp. 14–15

Predicting reversal strength

• No single panel statistic found in the tested designs predicts how strongly the unit-of-analysis effect will move a leaderboard.Two candidate designs failed their pre-registered tests.Source: §10 · p. 15

Forecast horizon

• The headline tables use a six-month horizon; the twelve-month check reproduces the ranking on only five origins and omits one of the twenty-four methods.The authors characterize the twelve-month result as a robustness check, not a second full benchmark.Source: §10 · p. 15

Run-to-run variability

• Across two full runs, deterministic methods reproduce to three decimal places, but one neural method moves in the third run.The authors caution that the two neural-method rows without fixed random seeds carry this much noise.Source: §10 · p. 15

The error

• The initial analysis made the deployed rule appear substantially worse than doing nothing because its blend used seasonal naive mean absolute error as the RMSSE denominator.The seasonal naive’s mean absolute error is always smaller than its root mean squared error, inflating the blend’s score by a constant factor while comparison methods were not similarly scaled.Source: §11 · p. 15

Lesson

• A scale-free metric is comparable only when every quantity under comparison is scaled the same way.The mismatch arose when a combination—the deployed rule’s blend—was scored alongside individual methods.Source: §11 · p. 15

Detection

• The incorrect result was not obviously implausible, so a plausibility check would not have exposed the scaling error.The authors report this as the reason the paper presents two metrics rather than one.Source: §11 · p. 15

• The authors are Field Nation employees, and the company approved publication and consented to publication of aggregate statistics from its proprietary panel.Source: Disclosures · pp. 15–16
• The research received no external funding.Source: Disclosures · p. 16
• The underlying marketplace data are proprietary, but the released protocol includes runnable evaluation and scoring code, experiment scripts, and replication packages.A synthetic panel supports reproducing the direction of the unit-of-analysis result and the spikiness result.Source: Disclosures · p. 16
• Md Rezwanul Islam and Wael Mohammed report distinct contributions, including shared conceptualization and writing contributions.Islam also lists methodology, software, formal analysis, investigation, data curation, and visualization; Mohammed lists project administration, resources, supervision, and validation.Source: Disclosures · p. 16
• The authors used Claude to help organize the manuscript and improve writing clarity, then reviewed and edited the content and accepted responsibility for it.Source: Disclosures · p. 16