Source-linked AI summary
Generalized Gibbs Ensemble Weighting for Forecast Combination
Prasen R. Nuthanakaluva, Nava K. Gaddam
TL;DR
Forecast combination lacks a universally best aggregation rule because performance varies with datasets, horizons, deployment settings, and forecast disagreement. The paper develops GGEW, which uses stabilized Gibbs weighting with diversity corrections and online hyperparameter adaptation. Experiments suggest it is useful and competitive, but its relative performance depends on the evaluation setting rather than showing universal dominance.
Problem
Forecast-combination methods are often strong but typically apply fixed rules despite substantial variation in cross-model disagreement and forecasting situations.
Method
GGEW converts normalized predictive-loss scores into Gibbs ensemble weights, adds stabilization and diversity-aware corrections, and uses Local-UCB to adapt hyperparameter states online.
Results
Gibbs-style adaptive weighting is a useful and competitive tool across M4, Traffic, Electricity, and Solar experiments, without universal dominance.
Takeaways & Limitations
The experiments motivate a regime-aware view of forecast combination in which usefulness depends on dataset, horizon, forecast pool, deployment protocol, and disagreement structure.
Takeaways & Limitations
GGEW requires a base-forecast pool whose quality, diversity, and stability constrain achievable performance.
Abstract
from arXiv · showhide
Forecast combination is a reliable way to improve predictive performance when several forecasting models are available. Simple aggregation rules such as the mean, median, trimmed mean, inverse-loss weighting, and exponential weighting are often strong baselines, but their relative performance can vary across datasets, forecast horizons, deployment settings, and levels of disagreement among base forecasters. We develop Generalized Gibbs Ensemble Weighting (GGEW), a probabilistic framework that treats forecasting models as experts and assigns ensemble weights using a Gibbs-style exponential transformation of normalized predictive loss. The framework extends this basic weighting rule through numerical stabilization, diversity-aware score corrections, and online hyperparameter adaptation. GGEW produces a family of related methods, including Stable Gibbs weighting, Directional Gibbs-NCL, and Symmetric Gibbs-NCL. These variants share one core algorithm and differ only in the score used inside the exponential weighting rule. For sequential deployment, we adopt a UCB-style bandit mechanism, called online Local-UCB, to adapt the learning rate, diversity strength, and Gibbs variant without evaluating the full hyperparameter grid at every prediction step. We evaluate GGEW on official M4 competition forecast submissions and external rolling-origin deployment experiments using Monash Traffic Hourly, Electricity Hourly, and Solar Weekly datasets. Results suggest that Gibbs-style adaptive weighting is a useful and competitive tool across several benchmark settings, although its relative performance varies across datasets, forecast horizons, deployment protocols, and forecast disagreement groups. The contribution is not a universal dominance claim, but a framework and empirical study motivating further investigation of when adaptive Gibbs-style forecast combination is useful.
1 Introduction
Forecast combination addresses the limits of relying on one model by exploiting complementary predictive behavior. GGEW frames adaptive combination as Gibbs-style expert weighting, while studying how weighting behaves across disagreement regimes and deployment settings.
- Different forecasting models respond differently to trend, seasonality, intermittency, noise, structural breaks, and local scale changes.
- Classical aggregation methods are strong, but fixed rules may be restrictive when cross-model disagreement varies across forecasting situations.
- GGEW treats predictive loss as an expert score, assigning higher probability mass to lower-loss models and producing a convex ensemble.
- The framework combines stabilized simplex-preserving updates, diversity-aware score corrections, and online Local-UCB hyperparameter adaptation.
- The empirical question is whether adaptive Gibbs weighting helps across disagreement levels and sequential deployment, without assuming universal dominance.
4 Generalized Gibbs Ensemble Weighting
GGEW converts historical predictive-loss scores into Gibbs weights, with normalization for stability and optional corrections for forecast diversity. Stable Gibbs, Directional Gibbs-NCL, and Symmetric Gibbs-NCL share this allocation procedure but use different score corrections.
- The concentration parameter η controls allocation sharpness: larger values emphasize lower-score models, while smaller values produce smoother weights.
- The Gibbs allocation assigns larger weights to experts with lower aggregate historical scores.The aggregate score is based on historical normalized predictive loss and may include variant-specific terms.
- Row-wise normalization reduces the dominance of very large squared errors and improves stability across heterogeneous time series.
- Directional and symmetric diversity corrections modify the score before Gibbs allocation, with the symmetric term favoring forecasts distinct from ensemble consensus.
- Stable Gibbs, Directional Gibbs-NCL, and Symmetric Gibbs-NCL are covered by one generalized allocation equation.
5 Weight Optimization and Gradient Structure
The optimization maintains ensemble weights on the probability simplex and updates them using relative performance against the remaining ensemble mass. Centering and clipping are applied before the update.
- The algorithm maintains a persistent vector of ensemble weights on the simplex and updates it iteratively using the score matrix.
- The leave-one-out gradient compares each expert with the remaining ensemble mass rather than only its own normalized allocation.
- A higher score relative to the rest of the ensemble produces a larger relative performance update signal.
- The gradient is centered and clipped before weights are updated.
6 Stable Exponentiated-Gradient Update
Stable exponentiated-gradient optimization performs repeated inner updates in log-weight space, then normalizes the weights and may apply a floor before renormalization. The experiments use 30 inner Gibbs iterations.
- Each Gibbs weighting call performs K inner optimization iterations, with K = 30 in the experiments.
- Weights are updated in log space to support the exponentiated-gradient procedure.
- After updating, the weights are normalized to maintain their ensemble interpretation.
- A small weight floor may be applied after normalization, followed by renormalization.
7 Algorithmic Recipe
GGEW uses one weighting algorithm for several Gibbs variants, differing only in their score corrections. The recipe stabilizes and updates ensemble weights through exponentiated-gradient normalization, with optional flooring.
- Stable Gibbs, Directional Gibbs-NCL, and Symmetric Gibbs-NCL are variants of one weighting mechanism with different score corrections.
- The algorithm requires historical forecasts and outcomes, future forecasts, a variant, learning rate, correction strength, iterations, stabilizer, clipping threshold, and optional weight floor.
- The recipe centers and clips the gradient before updating weights with exponentiated-gradient normalization.
- Optional weight flooring is followed by renormalization.
8 Online Local-UCB Hyperparameter Adaptation
Online Local-UCB adapts the active Gibbs hyperparameter state sequentially rather than selecting one fixed state using test-set performance. It evaluates a local candidate set and favors low-loss states while encouraging exploration.
- Online adaptation changes the active parameter state sequentially as new outcomes are observed instead of fixing it from test-set performance.
- Each state uses an exponentially smoothed loss estimate and an evaluation count, with β controlling memory.
- Local-UCB prefers smaller uncertainty-adjusted loss scores while temporarily favoring states evaluated fewer times.
- Candidate losses are median-scaled before updates to keep loss magnitudes comparable across cases.
- The local candidate set contains the current state, neighboring states, and states sufficiently close to the best uncertainty-adjusted score.
- The procedure initializes states, constructs and budgets candidates, evaluates selected states, produces forecasts, observes outcomes, updates statistics, and selects the next state.
- After candidate evaluation, the method produces forecasts with the active state, computes and scales realized losses, updates counts, selects the next state, and returns diagnostics.
9 Computational Complexity
Online Local-UCB evaluates only selected candidate parameter states rather than the full state space. This makes online adaptation substantially cheaper while retaining the ability to change states when recent data support it.
- The online cost depends on the number of models, historical observations, Gibbs iterations, and candidate states evaluated at time t.
- A full-grid online method scales with the total number of parameter states.
- Local-UCB is substantially cheaper than full-grid online adaptation while still allowing state changes supported by recent data.
10 Disagreement-Regime Stratification
The empirical protocol stratifies forecasting series by relative cross-model disagreement. Scale-adjusted disagreement is summarized at the series level and used to form low, medium, and high groups.
- The protocol stratifies series by relative cross-model forecast disagreement because diversity is important for forecast-combination performance.
- Cross-model variance is defined across forecasts for each series, horizon, and model.
- Relative variance is used because raw variance depends on scale.
- A series-level disagreement score summarizes the relative cross-model variation.
- Series are partitioned into high, medium, and low disagreement groups.
11 Experimental Evaluation
The evaluation treats ensemble methods as online forecasters, comparing Gibbs-family adaptation with practical baselines on M4 forecast submissions and external Monash rolling-origin experiments. The designs vary in forecast-pool source, disagreement structure, online memory, and deployment construction.
- Online evaluation: Each method is treated as an online forecaster, updating ensemble weights from past observed cases while Local-UCB adapts Gibbs-family hyperparameters.The controller selects among candidate states rather than evaluating the full hyperparameter grid at every step.
- Datasets and forecast pools: The study combines official M4 submitted forecast matrices with external rolling-origin experiments on Traffic Hourly, Electricity Hourly, and Solar Weekly.For Monash, deterministic rolling baseline forecasts were generated because ready-made forecast matrices were unavailable.
- Compared methods: The baseline suite spans equal weighting, robust aggregation, loss-based weighting, exponential expert weighting, and historical expert selection.It includes mean, median, trimmed mean, inverse-loss, exponential-weighted, best-historical-model, and top-3 average methods.
- Gibbs-family configuration: Stable Gibbs, Stable Gibbs-NCL, and Symmetric Gibbs use candidate learning rates η∈{0.01, 0.05, 0.1, 0.2}, with NCL variants additionally varying λ∈{0.01, 0.05, 0.1, 0.5}.The variants differ by whether they use no NCL correction, directional NCL, or symmetric diversity correction.
- M4 online experiments: M4 uses 41 valid submitted forecasting systems, stratifies series by relative cross-model forecast variance, and balances streams across frequencies and disagreement regimes.Each frequency/regime result uses 3000 evaluated cases, with 500 warmup cases per block and a Gibbs historical memory of T=200 past cases.
- Monash rolling-origin experiments: Monash deployment tests use deterministic pools tailored to hourly daily and weekly structure, with a local historical window spanning the previous five rolling forecast windows after five-window warmup.These experiments test online deployment behavior rather than comparisons against official Monash submission matrices.
12 Results and Discussion
Across M4 and Monash experiments, Gibbs-family methods are often competitive and sometimes leading, but their performance is not universal. Results depend on disagreement structure, deployment protocol, and the forecast pool.
- M4 Online Results: Gibbs-family methods win several M4 frequency–disagreement regimes, while robust and performance-based baselines remain strongest in others.Symmetric Gibbs wins M4 Yearly medium, M4 Quarterly medium, and M4 Monthly high; Stable Gibbs-NCL wins M4 Quarterly low and M4 Monthly low, whereas median and trimmed mean lead M4 Quarterly high.
- M4 Online Results: High disagreement does not uniformly favor robust aggregation: median and trimmed mean lead M4 Quarterly high, while Symmetric Gibbs leads M4 Monthly high.The results associate performance with the structure of disagreement rather than disagreement magnitude alone.
- Monash Rolling-Origin Results: Local-UCB Gibbs methods achieve the lowest aggregate losses across the three Monash deployment tests, with Stable Gibbs-NCL leading Traffic and Stable Gibbs leading Electricity and Solar.Solar Weekly is described as a smaller robustness experiment, but Gibbs-family methods occupy its leading ranks.
- Overall Empirical Pattern: Overall, online Gibbs-family methods are frequently competitive and sometimes leading, while classical methods remain important comparison points.The study presents adaptive Gibbs weighting as an addition to, rather than a universal replacement for, existing aggregation rules.
- Overall Empirical Pattern: The empirical conclusion is to evaluate adaptive forecast combination alongside the forecast pool, online deployment protocol, and disagreement among base forecasters.These factors define the settings in which different aggregation rules are compared.
13 Limitations
The framework has scope and evaluation boundaries: performance depends on the forecast pool, disagreement measure, hyperparameter state space, deployment protocol, and forecast type. Its empirical results therefore motivate extensions rather than universal claims.
- Forecast-pool dependence: Performance depends on the quality, diversity, and stability of the base-forecast pool, so poor or redundant candidates constrain adaptive weighting.Adaptive weighting cannot recover information absent from the candidate forecasts.
- Disagreement measurement: The disagreement-regime stratification is empirical rather than uniquely defined, leaving alternative diagnostics such as entropy, correlation, rank disagreement, and residual covariance open.Relative cross-model forecast variance is described as transparent and scale-adjusted, but not as the only disagreement measure.
- Online adaptation: Online Local-UCB currently selects from a discrete hyperparameter state space and does not solve continuous adaptation of learning rate, diversity strength, or exploration.Continuous adaptation remains a proposed extension.
- Evaluation protocol: Monash rolling-origin experiments use deterministic forecasts from raw public datasets, so they are not directly comparable to M4 experiments based on external submitted forecasting systems.The protocols test different forecast-pool construction settings.
- Forecast type and loss: The framework evaluates point forecasts with squared loss, leaving probabilistic, conformal, quantile, and interval forecast combination for future work.These extensions are identified as important directions rather than evaluated components.