Source-linked AI summary
Machine Learning and ARIMA Model Averaging for Adaptive Public Health Forecasting: Comparative Evaluation and an Ontario COVID-19 Case Study
Yushu Zou, Ye Li, Johra Moosa, Martin Grunnill, Samir N. Patel, Venkata R. Duvvuri
TL;DR
Public health forecasting must adapt to abrupt surveillance changes without over-extrapolating noise or temporary trends. The study compares ARIMA and machine-learning models across operating conditions and develops MLAMA, which achieved lower retrospective normalized error across plotted horizons. The findings support condition-specific model selection and averaging, while the analysis remains bounded by its retrospective Ontario case study and private software repository.
Problem
Public health forecasts need to balance responsiveness to changing surveillance trends with avoidance of noise and transient-trend extrapolation.
Method
The study compares ARIMA, random forest, and XGBoost using rolling-origin evaluation across responsiveness, one- to six-week horizons, and training-history depth, then combines them with condition-specific MLAMA weights.
Results
MLAMA produced lower retrospective normalized error across the plotted forecast horizons, while ARIMA responded faster and random forest and XGBoost were more stable at longer horizons.
Takeaways & Limitations
Forecasting methods should be selected according to operating conditions rather than treating one algorithm as universally preferred; MLAMA offers a framework for combining complementary forecasts.
Takeaways & Limitations
The evidence is retrospective, based on a single aggregated Ontario surveillance series, and the package and repository remain private during validation and reproducibility review.
Abstract
from arXiv · showhide
Public health forecasts must respond to abrupt changes in surveillance data without over-extrapolating noise, reporting artifacts, or temporary trends. We evaluated autoregressive integrated moving average (ARIMA), random forest, and extreme gradient boosting (XGBoost) models using 190 weekly observations of publicly available Ontario COVID-19 case counts from January 2020 to October 2023. Rolling-origin time-series cross-validation preserved temporal order during model tuning and evaluation. Performance was assessed across three operating dimensions: responsiveness following selected turning points, forecast horizons of one to six weeks, and the amount of historical training data. We also developed Machine Learning and ARIMA Model Averaging (MLAMA), a non-negative performance-weighted ensemble with weights that vary by forecast horizon and responsiveness setting. Retrospective comparisons showed that ARIMA adapted rapidly after turning points but its normalized error increased at longer horizons. Random forest and XGBoost were less responsive initially but maintained more stable normalized error over longer horizons. For two-week forecasts at the end of the study period, training on the most recent data outperformed using longer historical periods, particularly for XGBoost. MLAMA achieved the lowest normalized mean absolute percentage error across most forecast horizons and ranked among the best-performing methods across responsiveness settings. These findings support selecting forecasting models according to operating conditions rather than relying on a single universally preferred approach. MLAMA provides a practical framework for combining complementary statistical and machine-learning forecasts. The accompanying Python package is currently maintained in a private repository while software validation and reproducibility testing are completed.
1 Introduction
Public health forecasting must balance rapid response to changing surveillance patterns with protection against noise and transient trends. This study therefore evaluates complementary models and condition-specific model averaging across operational dimensions beyond predictive accuracy alone.
- Timely infectious-disease forecasts support preparedness, resource planning, situational awareness, and public health response.
- Abrupt changes in reporting, testing, interventions, behaviour, vaccination, and variants can make surveillance series difficult to forecast reliably.
- ARIMA is interpretable and often effective for short-horizon prediction, whereas machine-learning methods can model nonlinear relationships and interactions.
- Operational evaluation should consider responsiveness, robustness, forecast horizon, training-history relevance, and performance when recent information is unavailable.
- The study compares ARIMA, random forest, and XGBoost across operating conditions and develops MLAMA with condition-specific weights.
2 Methods
The study evaluates ARIMA, random forest, and XGBoost using temporally ordered retrospective procedures spanning responsiveness, forecast horizon, and historical training depth. MLAMA combines candidate forecasts with condition-specific weights estimated from rolling out-of-sample predictions, alongside an operational rolling workflow.
- Data source and analytical periods: The Ontario case study used 190 weekly aggregated COVID-19 case counts from January 2020 through October 2023.The data were publicly available and contained no individual-level records.
- Data source and analytical periods: Retrospective evaluation divided the series into three broad analytical periods to create temporally distinct training and evaluation settings.The periods were not intended as formal variant-defined epidemic waves.
- Forecasting models: The candidate methods were ARIMA, random forest, and XGBoost, with machine-learning models fitted as autoregressive regressors using only pre-origin observations.MLAMA can include additional models that generate forecasts under the same prospective evaluation protocol.
- Rolling-origin cross-validation and model tuning: Rolling-origin cross-validation used expanding training windows and later validation blocks, with preprocessing, predictor construction, tuning, and forecasting restricted to data available before each interval.The illustrated configuration used 12 expanding folds and fixed 12-week validation blocks.
- Evaluation dimensions: Performance was assessed over responsiveness indices 1–4, forecast horizons of one to six weeks, and three historical training depths.Responsiveness transferred post-turning-point observations sequentially into training, while training-depth comparisons focused on two-week forecasts after the third period.
- Evaluation metrics: MAPE was min–max normalized across configurations within each period and then averaged across periods, while MSRE scaled squared errors by observed counts.The authors note that MAPE and nMAPE can be unstable when observed counts are zero or close to zero.
- MLAMA ensemble: MLAMA combines candidate forecasts with non-negative weights summing to one, estimated separately for each forecast-horizon and responsiveness condition by minimizing aggregate MSRE.This permits ensemble composition to differ between short and longer horizons and between responsive and stable operating conditions.
- Python package and operational workflow: The Python package implements candidate configuration, rolling-window construction, hyperparameter optimization, performance-based weight estimation, and final ensemble prediction.Its operational workflow supports continuous historical data, configurable overlap, temporal separation of evaluation intervals, and refitting on the most recent admissible data.
3 Results
The retrospective evaluation found a trade-off between ARIMA’s rapid adaptation after turning points and machine-learning models’ greater stability at longer horizons. MLAMA generally performed best across forecast horizons and remained among the strongest methods across responsiveness settings.
- Base-model responsiveness and forecast horizon: ARIMA error declined rapidly as post-turning-point observations entered training, while random forest and XGBoost changed more gradually.The pattern indicates faster immediate adaptation for ARIMA and greater stability but slower response for the machine-learning models.
- Base-model responsiveness and forecast horizon: At longer forecast horizons, normalized ARIMA error increased more steeply, whereas random forest and XGBoost showed flatter error profiles.This demonstrates a practical trade-off between rapid local adaptation and longer-horizon stability.
- Amount of historical training data: Both ARIMA and XGBoost had lower MAPE for a two-week forecast when trained only on the most recent period rather than longer historical periods.The comparison followed the third analytical period and used three historical training depths.
- Amount of historical training data: XGBoost showed especially pronounced deterioration when the second and third periods were combined with the most recent period.The result suggests that older observations from substantially different conditions can make additional historical data harmful.
- MLAMA performance: MLAMA had the lowest nMAPE across the plotted forecast horizons and increased more gradually than the individual candidate models.Across responsiveness settings, it was among the lowest-error methods and avoided XGBoost’s large intermediate excursion.
- MLAMA performance: MLAMA was among the lowest-error methods across responsiveness settings while retaining lower error than the more slowly adapting candidates at several settings.These retrospective results support condition-specific model averaging.
4 Discussion
The study argues that forecasting performance depends on operating conditions: models differ by responsiveness, horizon, and training-history relevance, while MLAMA combines these strengths in a continuous workflow. However, the retrospective Ontario analysis has important scope and validation limitations.
- Model performance: ARIMA adapted rapidly after changed local directions, whereas random forest and XGBoost were less immediately responsive but more stable at longer horizons.These differences support evaluating models across operating conditions rather than selecting one universally preferred algorithm.
- Model averaging: MLAMA produced lower retrospective normalized error across plotted forecast horizons and competitive responsiveness by combining condition-specific model strengths.Its weights are estimated for operating conditions, translating model differences into an explicit averaging problem.
- Training history: Including older observations from materially different surveillance and transmission regimes could weaken current-signal relevance, motivating training windows based on recency and distributional change.More historical data did not necessarily improve prediction.
- Operational workflow: The package uses rolling windows and sequential forecast-origin updates to support prospective forecast-revision evaluation without advance knowledge of future peaks or troughs.This extends the retrospective analysis into a continuous operational workflow.
- Limitations: Findings may not generalize beyond one aggregated provincial COVID-19 series, other pathogens, spatial scales, or surveillance systems.The analytical periods and turning points were also defined retrospectively.
- Limitations: The normalized metric does not retain the original error scale, MAPE is problematic near zero, uncertainty was not quantified, and software validation remains incomplete.The Python package and executable analysis repository remain private while testing and reproducibility review are completed.
5 Conclusion
The Ontario case study found that ARIMA and machine-learning models have different strengths across forecasting conditions. MLAMA combines those strengths through operating-condition-specific weights, while the package remains subject to validation before public release or operational use.
- ARIMA and machine-learning models exhibited different strengths across responsiveness, forecast horizon, and historical training depth.
- MLAMA achieved lower normalized error across plotted forecast horizons and competitive responsiveness in the retrospective Ontario case study.
- Public release and operational use should follow comprehensive software and statistical validation.
Data and software availability
The case study used aggregated Ontario COVID-19 surveillance data, and MLAMA was implemented as a Python package with documentation for forecasting and maintenance. The source repository remains private while validation testing is completed.
- The case study used publicly available aggregated Ontario COVID-19 surveillance data from the Ontario Respiratory Virus Tool.
- MLAMA was implemented as a Python package with a user manual covering configuration, rolling windows, tuning, weighting, prediction, and maintenance planning.
- The source repository remains private while unit, integration, leakage, reproducibility, and numerical-validation testing are completed.