PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond a Universal Forecasting Selector: Demand-Conditioned Model Selection across Demand Patterns and Horizons

Adolfo González

arXiv:2609.04425v1cs.LG

TL;DR

Forecasting-model selection is difficult because no model or selector is established as universally best across heterogeneous demand contexts. The study evaluates selector performance across demand classes and conditions, finding that the leading selectors vary rather than one dominating throughout.

  • Problem

    No forecasting model consistently outperforms others across datasets, demand structures, horizons, or evaluation criteria, and less is known about whether selector suitability also varies by context.

    Source: §1 · p. 2, §2 · p. 5

  • Method

    The study examines selection mechanisms as a distinct decision layer, assessing them with pre-evaluation model metrics and ex post Global Relative Accuracy across demand classes and configurations.

    Source: §1 · p. 3, §3.5.7 · p. 14, §4.3 · p. 24

  • Results

    CCG-AHSC or CCG-AHSCD ranked highest in all Smooth and two of three Erratic configurations, while OWA led in two Intermittent and two Lumpy configurations.

    Source: §4.3 · p. 25

  • Takeaways & Limitations

    Selector effectiveness varies with demand structure, historical data availability, and forecasting horizon, supporting context-dependent selection rather than a single invariant criterion.

    Source: §5 · p. 25

    GRA excludes holding and shortage costs, service-level penalties, replenishment constraints, and production-capacity restrictions, so it is not a comprehensive operational objective.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Forecasting-model selection remains difficult in heterogeneous demand because the most suitable decision rule may vary with demand structure, data availability, and forecasting horizon. This study examines whether the selector itself should be treated as a context-dependent component of the forecasting process. Five selection mechanisms - RMSSE, ERA, OWA, CCG-AHSC, and CCG-AHSCD - are compared across 24 optimized forecasting models, nine datasets, three training-testing partitions, and horizons from 1 to 12 cycles. Selector performance is evaluated ex post using Global Relative Accuracy (GRA), statistical tests, and a best-attainable-model reference. No selector dominates across all conditions. CCG-AHSC and CCG-AHSCD are more competitive for Smooth demand and several Erratic configurations, whereas OWA and ERA perform better in Intermittent and Lumpy settings. Selector suitability also changes with historical data availability and horizon, supporting a context-dependent rather than universal approach to forecasting-model selection.

1. Introduction

No forecasting model consistently leads across datasets, demand structures, horizons, or evaluation metrics, making model choice context dependent. The study examines whether selection mechanisms also vary in suitability across demand classes, data availability, and horizons.

1. Introduction

  • No single forecasting model consistently outperforms alternatives across datasets, demand structures, frequencies, horizons, or evaluation metrics.Different error measures can also produce different model rankings.Source: §1 · p. 2
  • The study frames selector effectiveness as context dependent across Smooth, Intermittent, Erratic, and Lumpy demand, historical data availability, and forecasting horizon.It distinguishes selecting a model from selecting the mechanism that chooses among models.Source: §1 · p. 2
  • Five selection mechanisms are compared across 24 optimized forecasting models, nine datasets, three training–testing partitions, and horizons from 1 to 12 cycles.The selectors are RMSSE, ERA, OWA, CCG-AHSC, and CCG-AHSCD.Source: §1 · pp. 2–3
  • The study asks whether any selector consistently outperforms others and how suitability varies by demand class, historical data availability, and horizon.It evaluates selector performance using Global Relative Accuracy as an ex post measure of volumetric coherence, supplemented by statistical comparisons.Source: §1 · p. 3
  • An ex post best-model reference estimates attainable performance within the candidate set and the gap from the model assigned before future demand is observed.The study shifts attention from which forecasting model performs best to which selection mechanism identifies an appropriate model under specific conditions.Source: §1 · p. 3

2. Literature Review

Prior work addresses forecasting heterogeneity through automated model selection, metric choice, feature-based recommendation, and adaptive forecasting approaches. This review identifies a distinct open question: whether the selector itself should vary with demand structure, data availability, and horizon.

2. Literature Review

  • Forecasting performance varies with series characteristics, and automatic approaches address the impracticality of manual selection across many products and methods.Prior approaches include out-of-sample validation, information criteria, feature-based methods, expert systems, and multicriteria procedures; Salvador et al. (2019) extended CASH to multicomponent predictive systems.Source: §2 · pp. 3–4
  • Different error measures can yield different model rankings, so the appropriate criterion depends on the forecasting objective and operational application.This is especially relevant to inventory and purchasing, where pointwise accuracy and aggregate forecast volume can both affect decisions.Source: §2 · p. 4
  • Feature-based and meta-learning methods recommend forecasting algorithms from series characteristics, while the selector that chooses among models is a distinct decision level.Reina-Jiménez et al. (2026) use statistical and temporal meta-features with an interpretable CART meta-learner.Source: §2 · p. 4
  • Smooth, Intermittent, Erratic, and Lumpy patterns may respond differently to both forecasting models and the criteria used to distinguish among them.Demand characteristics may therefore condition selector suitability as well as model suitability.Source: §2 · p. 4
  • Adaptive forecasting studies address model structure, forecast combination, or direct recommendation, but comparatively less attention has gone to context-dependent selection mechanisms.Prior work also reports that predictive performance can evolve with horizon and that adaptation may depend on data availability.Source: §2 · pp. 4–5
  • The review motivates comparing selection procedures across demand scenarios and horizons to identify their strengths, limitations, and suitable application contexts.It rejects assuming a universally optimal rule.Source: §2 · p. 5

3. Materials and Methods

The methods compare reference selectors with class-conditioned adaptive selectors, using evaluation metrics chosen for distinct predictive and volumetric roles. The adaptive procedures screen candidate models by demand class before applying further selection rules.

3.1. Predictive Performance Evaluation Metrics

  • RMSSE is the primary scaled-error comparator, MAE refines absolute-error assessment, sMAPE evaluates relative error, and BIAS diagnoses directional error or breaks ties.Percentage-based metrics are interpreted cautiously near zero demand; R² is descriptive, while BIAS relates to inventory and service exposure.Source: §3.1 · p. 6

3.2. Global Relative Accuracy (GRA)

  • GRA measures agreement between aggregate forecasted and observed demand volumes, with values nearer 1 indicating greater agreement.Negative values can occur when aggregate forecast error exceeds realized demand; zero-total-realized-demand cases are excluded.Source: §3.2 · p. 6
  • GRA is an intermediate volumetric-coherence indicator, not a substitute for pointwise accuracy or a comprehensive inventory-cost or production-planning objective.It does not explicitly account for holding or shortage costs, service-level penalties, replenishment constraints, or production-capacity restrictions.Source: §3.2 · p. 7

3.3. Model Selectors

  • The reference selectors use minimum RMSSE, aggregated ordinal ranks, or benchmark-relative errors, while AHSC and AHSCD apply class-specific multicriteria rules after CCG screening.RMSSE selects the lowest-error candidate; ERA aggregates MAE, RMSE, and R² ranks; OWA compares MASE and sMAPE with benchmark errors.Source: §3.3 · p. 7, §3.3.1 · p. 8, §3.3.2 · p. 8

3.3.4. Class-Conditioned GRA (CCG)

  • CCG reduces the candidate set before final selection, using demand-class-specific scoring and retention rules with parameters fixed before final evaluation.Its class-conditioned criteria differ across Smooth, Intermittent, Erratic, and Lumpy demand; retained candidates proceed to AHSC or AHSCD.Source: §3.3.4 · pp. 9–10

3.3.5. Class-Conditioned Adaptive Hybrid Selector (AHSC)

  • AHSC applies Pareto filtering, then a Minimax criterion and, if needed, a refinement metric to select among CCG-retained models.The metrics used at each stage depend on the demand class.Source: §3.3.5 · pp. 10–11

3.3.6. Adaptive Hybrid Selector with Class-based Discrimination (AHSCD)

  • AHSCD applies Pareto filtering, a class-specific discriminating metric, and then a refinement criterion to select among CCG-retained models.Its Pareto, discrimination, and refinement metrics are specified according to demand class.Source: §3.3.6 · pp. 11–12

3.4. Backtesting Strategy

The backtesting strategy reserves each series’ final observations for out-of-sample evaluation and uses earlier observations for model estimation. Fixed-origin forecasts preserve temporal order and prevent information leakage.

3.4. Backtesting Strategy

  • Final observations form the out-of-sample segment, while all earlier observations are used for model estimation.Source: §3.4 · p. 12
  • Fixed-origin multi-step forecasts use only pre-evaluation observations, preserving temporal order and preventing information leakage.Source: §3.4 · p. 12

3.5. Experimental Design

The design compares five selectors under a shared evaluation protocol spanning heterogeneous datasets, demand classes, partitions, and forecast horizons. It also benchmarks selectors against an ex post best-model reference built from each SKU’s optimized candidates.

3.5. Experimental Design

  • Five selectors—RMSSE, ERA, OWA, CCG-AHSC, and CCG-AHSCD—are compared using standardized models, validation, metrics, and ex post GRA assessment.The framework groups 10 stages into data preparation, model estimation, selector-based assignment, and ex post forecast assessment.Source: §3.5 · p. 12, Figure 1 · p. 13

3.5.1. Stage 1: Dataset.

  • Nine datasets span demand frequencies, sample sizes, and demand patterns classified as Smooth, Intermittent, Erratic, or Lumpy.Table 1 presents dataset characteristics and the distribution of series across demand classes.Source: §3.5.1 · p. 13, Table 1 · p. 13

3.5.2. Stage 2: Backtesting.

  • The evaluation uses 12-cycle horizons and three train-test partitions—91:9, 80:20, and 70:30—to examine behavior under differing historical-data availability.Source: §3.5.2 · p. 13, §3.5.3 · p. 14

3.5.4. Stage 4: Demand forecasting models.

  • Twenty-four statistical, machine-learning, ensemble, and deep-learning forecasting models are optimized using HEF as a common criterion.Grid Search handles exhaustive-search models, while Optuna is used for continuous-space optimization.Source: §3.5.4 · p. 14, §3.5.6 · p. 14, §3.5.5 · p. 14

3.5.7. Stage 7: Metrics and predictions.

  • Forecasts for horizons 1–12 yield multiple evaluation metrics; only training-derived GRATest informs CCG, while ex post GRA uses reserved future observations.Source: §3.5.7 · p. 14

3.5.8. Stage 8: Model selectors.

  • RMSSE selects by scaled error, ERA aggregates metric ranks, and OWA compares MASE and sMAPE with a benchmark; CCG preselects by demand class before AHSC or AHSCD makes the final choice.AHSC applies Pareto filtering and Minimax selection; AHSCD uses Pareto filtering and class-specific discrimination, with class-specific metrics fixed across datasets and partitions.Source: §3.5.8 · pp. 15, 19

3.5.9. Stage 9: GRA measurement.

  • Selector comparisons use paired GRA outcomes, Friedman tests, and Holm-adjusted Wilcoxon tests after significant global differences; GRA is assessed across horizons and partitions.When the global test is not significant, rankings are interpreted descriptively rather than as pairwise superiority.Source: §3.5.9 · p. 19, §3.5.10 · p. 19

3.5.10. Stage 10: Results.

  • An ex post reference identifies, for each SKU, the best-performing applied model across the 12-cycle horizon, serving as an empirical upper bound for selector performance.Source: §3.5.10 · pp. 19–20

3.6. Experimental Protocol

A standardized protocol applies the same forecasting and selector workflow across datasets and partitions, then evaluates selected forecasts against future observations. The experiments use a documented Python-based computing environment.

3.6. Experimental Protocol

  • The protocol covers nine datasets, three partitions, five selectors, and horizons 1–12, with ex post GRA and an optimized-model reference for each SKU.Source: §3.6 · p. 21

3.7. Hardware and Software

  • Models and analyses use Python scientific-computing libraries, with Grid Search or Optuna selected according to model structure.Source: §3.7 · p. 21
  • Experiments ran in Python 3.13.9 on Linux using a 12-core AMD Ryzen AI 9 HX 370 system with 32 GB RAM and an NVIDIA RTX 5070 Laptop GPU.Source: §3.7 · p. 21

4. Results

No selector ranked first across all demand classes and experimental partitions. The leading selector varied by demand class and, in several cases, by training–testing partition.

4. Results

  • Across 2,203 Smooth, 1,880 Intermittent, 229 Erratic, and 1,021 Lumpy series, five selectors were evaluated under three partitions and horizons h = 1 to h = 12.The partitions were 91:9, 80:20, and 70:30.Source: §4 · p. 21

4.1. Distribution of selected forecasting models

  • Forecasting-model selections were distributed across several models and varied by selector and experimental partition, with a similarly distributed pattern for the BEST MODEL.Figure 2 presents the global selection frequencies.Source: Figure 2 · p. 22, §4.1 · p. 22

4.2.1. Smooth demand

  • For Smooth demand, CCG-AHSC led at 91:9 with median GRA 0.9463, while CCG-AHSCD led at 80:20 and 70:30 with 0.9408 and 0.9355.Global selector differences were significant in all three partitions (p < 0.001); at 91:9, the CCG-AHSC versus CCG-AHSCD post-hoc comparison was not significant after Holm correction.Source: Figure 3 · p. 22, §4.2.1 · p. 23

4.2.2. Intermittent demand

  • For Intermittent demand, OWA led at 91:9 (median GRA 0.6975) and 80:20 (0.6720), while CCG-AHSCD led at 70:30 (0.6404), without significant global differences.GRA profiles declined with horizon, particularly under 80:20 and 70:30.Source: Figure 4 · p. 23

4.2.3. Erratic demand

  • For Erratic demand, ERA led at 91:9 (median GRA 0.5765), while CCG-AHSCD led at 80:20 (0.5658) and 70:30 (0.5500); global differences were significant in all partitions.Figure 5 presents median GRA by horizon and partition.Source: Figure 5 · p. 23, §4.2.3 · p. 24

4.2.4. Lumpy demand

  • For Lumpy demand, ERA led at 91:9 (median GRA 0.4625), while OWA led at 80:20 (0.4706) and 70:30 (0.4557); global differences were significant in all partitions.GRA profiles declined markedly as the forecasting horizon increased.Source: §4.2.4 · p. 24, Figure 6 · p. 24

4.3. Summary of global results

  • CCG-AHSC or CCG-AHSCD led all Smooth configurations and two Erratic configurations; OWA led two Intermittent and two Lumpy configurations, while ERA led Erratic and Lumpy at 91:9.The top-performing selector therefore depended on demand class and experimental partition.Source: Table 3 · p. 24, §4.3 · p. 25

5. Discussion

No selector is universally superior: performance varies with demand structure, training–testing partition, and forecasting horizon. The results also show that strong candidate models are available, but identifying the best one in advance remains difficult.

5. Discussion

  • No selector consistently outperforms the others across demand classes, training–testing partitions, and forecasting horizons.Selector performance depends on demand structure, historical data availability, and horizon.Source: §5 · p. 25
  • For Smooth demand, CCG-AHSC and CCG-AHSCD performed most strongly and stably across partitions, though their 91:9 difference was not statistically significant.The discussion attributes their competitiveness to class-specific candidate reduction followed by Pareto-based and hierarchical decision rules.Source: §5 · p. 25
  • For Intermittent demand, OWA led under 91:9 and 80:20, while CCG-AHSCD led under 70:30; Friedman tests found no significant selector differences.The ranking differences indicate contextual variation, not proof of selector superiority.Source: §5 · p. 25
  • For Erratic demand, ERA led at 91:9 while CCG-AHSCD led at 80:20 and 70:30; for Lumpy demand, ERA led at 91:9 and OWA at the other partitions.Unlike Intermittent demand, these differences were statistically significant, and the preferred selector changed as training information decreased.Source: §5 · pp. 25–26
  • GRA generally declined as the horizon increased, especially for Intermittent and Lumpy demand, while relative selector performance also varied across horizons.The discussion recommends assessing both overall performance and its evolution with horizon length.Source: §5 · p. 26
  • The ex post best model varied across forecasting methods, leaving the key challenge of identifying the most appropriate available model before future demand is observed.The gap between ex ante selection and the best ex post model indicates remaining opportunity to improve automatic selection.Source: §5 · p. 26
  • CCG-AHSC and CCG-AHSCD are not universal replacements, but their demand-class conditioning was particularly effective for Smooth demand and several Erratic configurations.Simpler selectors remained competitive or preferable under other conditions.Source: §5 · p. 26
  • The selector is a distinct decision layer from the forecasting model itself, and its suitability varies with demand pattern, historical data, and horizon.This conditioned perspective provides a basis for reducing the gap between ex ante assignment and the best model identified ex post.Source: §5 · p. 27

6. Conclusion

The study finds that selector effectiveness depends on demand class, historical data, and forecasting horizon, rather than one mechanism dominating across conditions. It frames the remaining challenge as identifying high-performing models in advance and calls for broader robustness and operational evaluation.

6. Conclusion

  • None of the five mechanisms consistently outperformed the others across all demand classes, partitions, and horizons.CCG-AHSC and CCG-AHSCD were more competitive for Smooth demand and several Erratic configurations, while OWA and ERA had advantages in some Intermittent and Lumpy conditions.Source: §6 · p. 27
  • Selector effectiveness depends on demand structure, historical data availability, and forecasting horizon, making the selector part of the forecasting decision.CCG-AHSC and CCG-AHSCD narrow the candidate set by demand class before applying class-specific multicriteria rules.Source: §6 · p. 27
  • GRA adds a measure of volumetric coherence relevant to inventory decisions, while the ex post best-model reference provides an upper bound within the optimized candidate set.Source: §6 · p. 27
  • Future work should test CCG parameter sensitivity, use operational criteria such as inventory costs and service levels, and validate generalizability across datasets and demand frequencies.The remaining challenge is consistently identifying high-performing available models before future demand is known.Source: §6 · p. 27

Author contributions statement

The authors report their respective contributions and confirm that both reviewed and approved the final manuscript.

Author contributions statement

  • Adolfo González contributed to conceptualization, methodology, software, validation, analysis, investigation, data curation, drafting, and visualization; Víctor Parada supervised and reviewed the manuscript.Both authors reviewed and approved the final version.Source: Author contributions statement · p. 27

Funding

The study received no external funding.

Funding

  • The authors report that the research received no external funding.Source: Funding · p. 28

Data availability statement

The study uses publicly available data from retail, forecasting competitions, and industrial spare-parts repositories. Retail sales are aggregated across stores to form one weekly demand series per SKU.

Data sources

  • The Walmart, M3, M4, M5, BRAF, MAN, Beer, Analgesics, and Cheeses datasets are publicly available through their respective repositories or data sources.The sources include Kaggle, the International Institute of Forecasters, Spare-Part-Demand-Forecasting repositories, and the University of Chicago Booth Kilts Center.Source: Data availability statement · p. 28

Retail data preparation

  • Beer, Analgesics, and Cheeses sales were aggregated across stores for each UPC, producing one weekly demand time series per SKU.This removes the store dimension while preserving temporal demand structure.Source: Data availability statement · p. 28

Dataset provenance

  • The industrial BRAF and MAN series come from public spare-parts forecasting repositories, while the three retail datasets come from the Dominick's Dataset.The retail files are wber, wana, and wche, respectively.Source: Data availability statement · p. 28

Appendix A

Appendix A documents the candidate forecasting models, their architectures, and explored parameter ranges. The candidates span regression, smoothing, boosting, recurrent, attention-based, and hybrid approaches.

Candidate models

  • Table A1 covers candidate forecasting models and optimization groups, including statistical regressors, smoothing, boosting, and neural forecasting approaches.The appendix describes models such as Linear Regression, Simple Exponential Smoothing, XGBoost, LSTM, and Transformer.Source: Table A1 · p. 28, Appendix A · pp. 29–30

Neural architectures

  • The LSTM, GRU, BiLSTM, BRNN, and Transformer candidates use horizon-adjusted output layers for direct multi-step forecasting.The appendix also describes a DNN-LSTM hybrid combining recurrent layers with dense layers.Source: Appendix A · pp. 29–30

Parameter ranges

  • The appendix specifies explored parameter ranges for candidate models, including XGBoost, Huber and Bayesian ridge regression, and neural-network training epochs.For example, XGBoost n_estimators ranges from 10 to 2000, while the listed neural models use epoch ranges of 1 to 80.Source: Table A2 · p. 30, Appendix A · p. 31
Loading 2609.04425v1…