PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking Silos: Adaptive Model Fusion Unlocks Better Time Series Forecasting

Zhining Liu, Ze Yang, Xiao Lin, Ruizhong Qiu, Tianxin Wei, Yada Zhu, Hendrik Hamann, Jingrui He, Hanghang Tong

arXiv:2505.18442v1cs.LGcs.AI

TL;DR

Sample-level inspection finds that no forecasting model consistently leads across test samples, despite strong benchmark results. TIMEFUSE uses input meta-features to adaptively fuse heterogeneous models, achieving near-universal improvement over individual models across benchmark long- and short-term tasks.

  • Problem

    Sample-level inspection finds no universal winner: models that perform well on benchmarks lead on only up to 23.2% of test samples, while different models have distinct strengths.

    Source: §1 · p. 2

  • Method

    TIMEFUSE extracts temporal meta-features and trains a learnable fusor to predict input-specific weights for combining heterogeneous forecasting models.

    Source: §1 · p. 2

  • Results

    TIMEFUSE achieves near-universal improvement across benchmark long- and short-term forecasting tasks, outperforming the best individual model on up to 95.1% of samples.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments support TIMEFUSE as a versatile tool for tackling complex time-series forecasting challenges.

    Source: §6 · p. 9

    Some base models perform strongly on meta-training data but behave differently on test data, which may affect fusion decisions.

    Source: D. Discussion on Limitations and… · p. 17

Abstract

from arXiv · show

Time-series forecasting plays a critical role in many real-world applications. Although increasingly powerful models have been developed and achieved superior results on benchmark datasets, through a fine-grained sample-level inspection, we find that (i) no single model consistently outperforms others across different test samples, but instead (ii) each model excels in specific cases. These findings prompt us to explore how to adaptively leverage the distinct strengths of various forecasting models for different samples. We introduce TimeFuse, a framework for collective time-series forecasting with sample-level adaptive fusion of heterogeneous models. TimeFuse utilizes meta-features to characterize input time series and trains a learnable fusor to predict optimal model fusion weights for any given input. The fusor can leverage samples from diverse datasets for joint training, allowing it to adapt to a wide variety of temporal patterns and thus generalize to new inputs, even from unseen datasets. Extensive experiments demonstrate the effectiveness of TimeFuse in various long-/short-term forecasting tasks, achieving near-universal improvement over the state-of-the-art individual models. Code is available at https://github.com/ZhiningLiu1998/TimeFuse.

1. Introduction

Sample-level inspection finds that forecasting models have distinct strengths rather than a universal winner, motivating TIMEFUSE’s adaptive fusion of heterogeneous models. Across benchmark long- and short-term tasks, TIMEFUSE improves on individual models and can generalize to unseen datasets.

1. Introduction

  • TIMEFUSE outperforms the best individual model on up to 95.1% of samples and achieves near-universal improvement across benchmark long- and short-term tasks.The figure illustrates the sample-level comparison; the introduction reports the broader benchmark-task result.Source: Figure 1 · p. 1, §1 · p. 2
  • Across seven datasets, even state-of-the-art models ranked first on only up to 23.2% of test samples, while lower-ranked models also led on a non-negligible fraction.The analysis points to distinct model strengths, including TimeMixer for high spectral complexity and Non-stationary Transformer for low stationarity.Source: §1 · p. 2
  • TIMEFUSE uses temporal meta-features to predict input-specific fusion weights, combining heterogeneous model outputs instead of applying a static ensemble.Its meta-features include statistical, temporal, and spectral descriptors, and the fusor is trained to minimize fused forecasting error across diverse samples.Source: §1 · p. 2
  • The fusor can be jointly trained on samples from different datasets, supporting zero-shot performance on datasets unseen during meta-training.The paper also describes the method as interpretable because users can inspect model contributions and learned weights.Source: §1 · p. 2

2. Preliminaries

The forecasting task maps recent multivariate observations to values over a future horizon. The paper frames adaptive fusion as using a collection of independently trained models to produce an input-specific forecast.

2. Preliminaries

  • The task predicts each variable’s values over the next T_out steps from the most recent T_in observations; the input has d variables.The input is denoted X_in ∈ R^(T_in×d), and the forecast output X_out ∈ R^(T_out×d).Source: §2 · p. 2
  • A base forecasting model maps an input series to a predicted output series, and models may use distinct architectures to capture different temporal patterns.Source: §2 · p. 2
  • The model zoo is a set of independently trained forecasting models whose diverse predictions are to be adaptively combined for each input.The formal objective is to use input characteristics to leverage the models’ strengths and obtain an optimal fused prediction against the ground truth.Source: §2 · pp. 2–3

3. Methodology

TIMEFUSE extracts task-agnostic meta-features from each input time series and uses a learnable fusor to assign weights to base-model forecasts. Its meta-training can be decoupled from base-model training and jointly mix data from multiple forecasting tasks.

3.1. Time Series Meta-feature Extraction

  • Task-agnostic meta-features standardize inputs across forecasting tasks, while reducing dimensionality to filter noise and redundancy and retaining interpretable semantics.The described semantics include periodicity and spectral density.Source: §3.1 · p. 3
  • TIMEFUSE uses 24 statistical, temporal, spectral, and multivariate meta-features, matching the performance of TSFEL’s 165-variable feature set.The authors report contributions from each feature domain to forecasting performance.Source: §3.1 · pp. 3–4

3.2. Adaptive Model Fusion with Learnable Fusor

  • A single-layer fusor linearly maps meta-features to model weights, then combines base-model forecasts using those weights.A softmax normalizes the weights; more complex fusor structures do not improve performance and compromise runtime efficiency and interpretability.Source: §3.2 · p. 4
  • The fusor is trained to choose weights that minimize the forecasting error of the fused output against ground truth.The authors use Huber loss and train on a held-out validation set not used for base-model training.Source: §3.2 · p. 4
  • Meta-training is decoupled from base-model training and uses meta-features, base-model predictions, and ground truth, allowing new models to be added by including their predictions.The meta-training data can be collected and stored independently.Source: §3.2 · p. 4
  • For cross-task training, TIMEFUSE oversamples task datasets to the largest dataset’s size and alternates batches across tasks.The strategy addresses inconsistent prediction dimensions and imbalanced sample counts across tasks.Source: §3.2 · p. 5

4. Experiments

TIMEFUSE was evaluated on 16 real-world long- and short-term benchmarks with 13 base models; fusors were jointly trained across tasks for each prediction length. Experiments compare forecasting results with individual models and static ensembles and probe adaptability, generalization, and learned features.

4.1. Main Results

  • TIMEFUSE reduced average MSE by 3.61%/6.88%/11.77%/8.39% across seven datasets versus TXer/Tmixer/PAttn/iTF, respectively.Source: §4.1 · p. 6
  • On PEMS, TIMEFUSE reduced average MAPE by 20.46%/9.89%/15.37% versus TimeXer/iTransformer/LightTS; on EPF, it was more accurate across all five tasks.Source: §4.1 · p. 6, Table 3 · p. 6, Table 4 · p. 6
  • TIMEFUSE outperformed validation-filtered static mean/median ensembles across all tasks, while the best ensemble size varied from 10 models on ETTh1 to two on traffic.Source: §4.1 · p. 6, Figure 3 · p. 7

4.2. Further Analysis

  • TIMEFUSE uses broad model ensembles on ETTh1/m1 but selective integration on Electricity/Traffic; learned weights also associate temporal properties with models’ relative strengths.Source: §4.2 · pp. 7–8, Figure 4 · p. 7, Figure 6 · p. 8
  • Forecasting performance improved as more diverse base models entered the model zoo, with the evaluation adding newer models sequentially to an initial Informer/Autoformer set.Source: §4.2 · p. 7, Figure 5 · p. 7
  • When trained on all other datasets and tested on a held-out target, zero-shot TIMEFUSE outperformed the best individual model in most cases.Source: §4.2 · p. 7, Table 5 · p. 7
  • The 24-variable meta-feature set matched the more complex 165-variable TSFEL set, and ablations showed significant contributions from each feature domain.Source: §4.2 · p. 8, Table 6 · p. 8
  • Additional appendix analyses report effectiveness, efficiency, and adaptability across forecasting scenarios.Source: §4.2 · p. 8

5. Related Works

Related work covers forecasting methods developed to capture temporal patterns and long-range dependencies, alongside earlier time-series ensemble approaches. The paper contrasts those approaches with adaptive fusion across heterogeneous models.

Time-series Forecasting

  • Forecasting research spans statistical, recurrent, convolutional, TimesNet, Transformer, multiscale-mixing, exogenous-variable, and multimodal approaches.Source: §5 · p. 8

Ensemble Learning

  • Earlier time-series ensemble studies used mean/median, bagging, or multiple neural models, but the cited neural approaches used homogeneous architectures and static ensembles.Source: §5 · p. 9
  • TIMEFUSE differs by dynamically fusing heterogeneous model architectures at the sample level.Source: §5 · p. 9

6. Conclusion

TIMEFUSE adaptively fuses forecasting models for each input time series, learning model combinations from input characteristics. Across the reported experiments, it consistently achieved new state-of-the-art performance by leveraging different models’ strengths at test time.

  • TIMEFUSE consistently achieved new state-of-the-art performance across the reported experiments by dynamically leveraging different models’ strengths at test time.Source: §6 · p. 9
  • TIMEFUSE adaptively learns and selects model combinations based on each input time series’ characteristics.Source: §6 · p. 9
  • The framework makes its model fusion interpretable as well as adaptive.Source: §6 · p. 9

Impact Statement

The paper presents a learning-based ensemble framework for time-series forecasting. It aims to improve forecasting by adaptively fusing different models for each input at test time.

  • The paper presents a learning-based ensemble time-series forecasting framework.Source: Impact Statement · p. 9
  • Its goal is more accurate forecasting through adaptive fusion of models for each input time series at test time.Source: Impact Statement · p. 9
  • The authors note potential social impacts but say none need to be highlighted.Source: Impact Statement · p. 9

A.1. Dataset Descriptions

The appendix describes the long- and short-term forecasting datasets and specifies the evaluation metrics used for each task. Long-term experiments span seven real-world datasets, while short-term experiments use PEMS traffic and EPF electricity-price datasets.

A.1. Dataset Descriptions

  • Long-term experiments cover seven real-world datasets, including ETT, Weather, Electricity, and Traffic; Table 7 gives dataset statistics.ETT comprises four hourly or 15-minute subsets; Electricity covers 321 clients, and Traffic uses 862 sensors.Source: §A.1 · p. 14
  • Short-term experiments use four PEMS traffic-flow datasets and five EPF day-ahead electricity markets, each spanning six years.Source: §A.1 · p. 14

A.2. Evaluation Metric Details

  • Long-term tasks and EPF use MSE and MAE, while PEMS reports MAE, RMSE, and MAPE on the unnormalized test set.Source: §A.2 · p. 14

A.3. Implementation Details

The appendix details the base-model suite and TIMEFUSE training setup, then reports inference efficiency and performance under longer lookback lengths. TIMEFUSE maintains or improves strong performance across input lengths and adds negligible fusor overhead.

A.3. Implementation Details

  • The base-model suite includes 13 forecasting models, spanning recent architectures such as TimeXer, TimeMixer, PAttn, PatchTST, and iTransformer.The models were implemented using TSLib and official optimal hyperparameters when available; otherwise, training used 10 epochs, ADAM, L2 loss, and validation-based early stopping with patience 3.Source: §A.3 · pp. 14–15
  • For long-term tasks, validation samples from the datasets are jointly used to train one fusor; short-term PEMS and EPF datasets likewise use joint fusor training and testing.The fusor is a single-layer neural network mapping meta-features to model weights, optimized with ADAM and Huber loss, batch size 32, and learning rate 1e-3.Source: §A.3 · p. 15

B.1. Inference Efficiency

  • The single-layer fusor adds negligible inference overhead, runs significantly faster than transformer-based models, and is on par with DLinear.The benchmark uses batch size 32 and prediction length 96.Source: §B.1 · p. 15, Table 8 · p. 16

B.2. Forecasting with Extended Input Horizons

  • TIMEFUSE consistently improves or maintains strong forecasting performance across seven long-term datasets at lookback lengths of 336 and 512.PatchTST and PAttn benefit from longer inputs, while TimeXer, iTransformer, and TimesNet show mixed or degraded performance.Source: §B.2 · pp. 15–16

B.3. Comparison with AutoML and Advanced Ensemble Baselines

TIMEFUSE is compared with advanced and AutoML ensembles and Chronos across forecasting settings. It achieves the best average performance across three task categories, while addressing input-dependent variation that static ensemble weights do not capture.

  • Across 16 datasets, TIMEFUSE achieves the best average performance in long-term multivariate, short-term multivariate, and short-term univariate forecasting.Source: §B.3 · p. 16
  • TIMEFUSE adjusts fusion weights for each input using meta-features, unlike forward-selection and portfolio ensembles that set static weights from aggregate validation performance.Source: §B.3 · p. 16
  • Zero-shot ensembles condition on dataset-level similarity rather than individual samples, and their effectiveness depends on the diversity and coverage of training tasks.Source: §B.3 · p. 16
  • Chronos-Bolt is optimized for short-horizon univariate tasks, and its predictions were unstable or degenerate in evaluated long-term or multivariate settings.Source: §B.3 · p. 16
  • AutoGluon-TimeSeries is limited to univariate forecasting, and its per-variate inference can take hours on large datasets without GPU acceleration for many constituent models.Source: §B.3 · p. 16
  • With longer input contexts, PatchTST and PAttn benefit on specific datasets, while the other three base models generally do not; TIMEFUSE retains an advantage across context lengths.Source: Table 9 · p. 17

C. Full Results

The full results report robust performance across prediction lengths and datasets. TIMEFUSE consistently outperforms the task-specific best individual models across different datasets and metrics.

  • TIMEFUSE consistently outperforms the task-specific best individual models across datasets and metrics.Source: C. Full Results · p. 17
  • The detailed long- and short-term forecasting results show robust TIMEFUSE performance at various prediction lengths.Source: C. Full Results · p. 17
  • The main text reports average performance scores across prediction lengths for long-term datasets and short-term PEMS datasets because of space constraints.Source: C. Full Results · p. 17

D. Discussion on Limitations and Future Directions

The discussion identifies distribution shift, fusor expressiveness, task imbalance, and spatial structure as limitations or directions for future work. It also notes practical constraints in some evaluated baseline settings.

  • Distribution shifts between meta-training and meta-test sets can affect fusion decisions when base models behave differently on test data than on meta-training data.In some cases, excluding such models from the model zoo has been observed to improve results.Source: D. Discussion on Limitations and… · p. 17
  • As task diversity increases, a more expressive fusor may capture complex meta-level patterns, but its complexity must be balanced against generalization.Source: D. Discussion on Limitations and… · p. 17
  • The current oversampling strategy may not provide the optimal contribution balance across tasks with different sample sizes.Source: D. Discussion on Limitations and… · p. 18
  • TIMEFUSE does not explicitly incorporate spatial or graph information, which could support more adaptive fusion where spatial relationships matter.Source: D. Discussion on Limitations and… · p. 18
  • AutoGluon inference took over three hours in the reported out-of-time setting, while Chronos was unable to handle long-term forecasts extending beyond its 64-step pretraining prediction length.Source: D. Discussion on Limitations and… · p. 19
Loading 2505.18442v1…