Source-linked AI summary
Improving the Accuracy of Global Forecasting Models using Time Series Data Augmentation
Kasun Bandara, Hansika Hewamalage, Yuan-Hao Liu, Yanfei Kang, Christoph Bergmeir
TL;DR
The paper addresses limited-data settings in which global forecasting models have too few related series to learn effectively. It generates synthetic series with three augmentation methods and transfers their information through pooling or transfer learning, improving baseline GFM accuracy and outperforming univariate benchmarks across evaluated datasets.
Problem
Deep-learning GFMs require many related time series, but many databases contain too few series for reliable model estimation.
Method
The framework generates synthetic series using GRATIS, MBB, and DBA, then transfers their learned information through pooled training or adaptation of a pre-trained GFM.
Results
Across the benchmark suite, DBA.TL.Dense.Freeze achieves the best average Rank sMAPE and Rank MASE, while most proposed variants outperform the baseline and statistical benchmarks.
Takeaways & Limitations
Data augmentation combined with pooling or transfer learning can improve GFM forecasting accuracy in data-sparse environments.
Takeaways & Limitations
The authors identify replacing the stacking architecture with encoder-decoder feature extraction as future work for reusing latent features on target datasets.
Abstract
from arXiv · showhide
Forecasting models that are trained across sets of many time series, known as Global Forecasting Models (GFM), have shown recently promising results in forecasting competitions and real-world applications, outperforming many state-of-the-art univariate forecasting techniques. In most cases, GFMs are implemented using deep neural networks, and in particular Recurrent Neural Networks (RNN), which require a sufficient amount of time series to estimate their numerous model parameters. However, many time series databases have only a limited number of time series. In this study, we propose a novel, data augmentation based forecasting framework that is capable of improving the baseline accuracy of the GFM models in less data-abundant settings. We use three time series augmentation techniques: GRATIS, moving block bootstrap (MBB), and dynamic time warping barycentric averaging (DBA) to synthetically generate a collection of time series. The knowledge acquired from these augmented time series is then transferred to the original dataset using two different approaches: the pooled approach and the transfer learning approach. When building GFMs, in the pooled approach, we train a model on the augmented time series alongside the original time series dataset, whereas in the transfer learning approach, we adapt a pre-trained model to the new dataset. In our evaluation on competition and real-world time series datasets, our proposed variants can significantly improve the baseline accuracy of GFM models and outperform state-of-the-art univariate forecasting methods.
1. Introduction
Global forecasting models can outperform univariate methods when many related series are available, but deep-learning GFMs struggle with data-sparse databases. This study augments limited datasets synthetically and transfers the resulting knowledge through pooled and transfer-learning strategies.
- GFMs are competitive when many related time series are available, but deep-learning versions require large datasets to estimate their numerous parameters.
- Data augmentation addresses sparse time-series databases by generating synthetic observations when expert knowledge is unavailable.
- The study uses GRATIS, moving block bootstrap, and dynamic time warping barycentric averaging to generate augmented series with different relationships to the original data-generating process.
- Augmented knowledge is transferred either by pooling synthetic and original series or by pre-training a GFM and adapting it to the original dataset.
- The framework is evaluated against a baseline GFM and state-of-the-art statistical forecasters using five competition and real-world time-series databases.
2. Related Work
Related work covers data augmentation and transfer learning for time-series analysis, emphasizing synthetic-series generation and knowledge transfer across datasets or tasks. It identifies feature-transfer and parameter-transfer learning as common deep-learning paradigms.
- Time-series augmentation research includes bootstrapping, averaging, statistical generative models, and neural-network approaches for producing synthetic series.
- Transfer-learning methods differ in the knowledge representation transferred and the mechanism used to transfer it between source and target tasks.
- Transfer learning is categorized as inductive, transductive, or unsupervised according to task, domain, and label relationships.
- Recent forecasting studies apply transfer learning to settings such as cross-building energy prediction and limited-history new-building estimation.
- Feature-transfer and parameter-transfer learning are identified as the most common paradigms in deep-learning applications because neural networks can extract latent data representations.
3. Forecasting Framework
The forecasting framework preprocesses time series, trains a residual recurrent network on moving windows, and reverses the transformations after prediction. Its architecture supports direct multi-step forecasting and alternative recurrent cells.
- The framework comprises preprocessing, RNN training, and post-processing layers, with residual recurrent units forming the primary prediction engine.
- Time series are mean-normalised, log-transformed to stabilise variance, and decomposed with STL into seasonal, trend, and residual components.
- Moving-window training uses MIMO outputs matching the forecast horizon, allowing the network to predict all future steps at once and avoid stepwise error accumulation.
- The network uses LSTM cells with stacking and dense output mapping, while L1 loss and L2 regularisation support training and control overfitting.
- Final forecasts are reseasonalised when required and denormalised by reversing local scaling, the log transform, and the series-level scaling.
4. Transfer Learning Architectures
The proposed transfer-learning architectures adapt a pre-trained residual recurrent base model to a target dataset by adding dense or LSTM layers. Variants either freeze the base layers or retrain them with the new layers.
- TL.Dense, TL.AddDense, and TL.LSTM add respectively one dense layer, multiple dense layers, or residual LSTM layers plus a dense layer to the pre-trained base model.
- The schemes vary the number and type of trainable parameters introduced when adapting the base model to the target dataset.
- Freeze variants train only newly added layers, whereas Retrain variants update both the pre-trained layers and newly added layers.
- The transfer setup uses the same forecasting task for source and target while changing the datasets, corresponding to transductive transfer learning.
5. Time Series Augmentation
The study augments time-series datasets using GRATIS, MBB, and DBA, then transfers knowledge to GFMs through approaches suited to the generated data. These techniques differ in whether they generate diverse series, preserve original distributions, or combine characteristics across series.
- The study uses three augmentation techniques: GRATIS, moving block bootstrap, and dynamic time warping barycentric averaging.MBB and DBA are expected to resemble the original data-generation process, whereas GRATIS generates more diverse characteristics.
- 5.1. GRATIS: GRATIS uses mixture autoregressive models to generate time series with diverse features.Its mixture weights combine multiple autoregressive processes whose coefficients are selected from Gaussian distributions.
- 5.1. GRATIS: GRATIS-generated series are used with transfer learning because combining substantially different datasets can reduce GFM accuracy.The model is pretrained on GRATIS-generated series and then transferred to the target dataset.
- 5.2. Moving Block Bootstrapping: MBB removes seasonal and trend components, bootstraps the remainder, and adds those components back to create new series.Because original observations are used, the generated data closely resemble the original training distribution, including its seasonality and trend.
- 5.3. Dynamic Time Warping Barycentric Averaging: DBA averages multiple time series using dynamic time warping, allowing synthetic samples to mix characteristics across series.This contrasts with MBB, which applies bootstrapping separately to each series, and better represents global characteristics within a group.
6. Experimental study
Across benchmark datasets, the proposed augmentation-based GFM variants generally improve on the LSTM baseline and often outperform statistical forecasting benchmarks, with performance depending on the augmentation method and dataset.
- Overall results: DBA.TL.Dense.Freeze achieves the best overall average ranks for both sMAPE and MASE across the benchmark suite.Its improvements over the LSTM baseline and statistical benchmarks are highly significant for sMAPE.
- Statistical testing: 2.66 × 10−10 is the Friedman-test p-value for sMAPE, indicating highly significant overall differences across methods.DBA.TL.Dense.Freeze is selected as the control method in the post-hoc analysis.
- Statistical testing: 2.58 × 10−10 is the Friedman-test p-value for MASE, although improvements over LSTM.Baseline are not statistically significant in this comparison.Improvements over ES, Prophet, and ARIMA are highly significant.
- Dataset-level results: Except for NN3 in the Mean sMAPE analysis, many proposed variants outperform ES, Prophet, and ARIMA across the benchmark datasets.The ES method achieves the best Mean sMAPE for NN3, while proposed variants are strongest on the other listed datasets.
- Dataset-level results: For Mean MASE, proposed variants obtain the best accuracies apart from NN3, with MBB strongest on AusEnergy-Demand and DBA strongest on AusGrid-Energy.The GRATIS.TL.AddDense.Retrain variant achieves the best Mean MASE for NN5.
- Interpretation: DBA and MBB generally outperform GRATIS because methods generating series closer to the original distribution better improve baseline accuracy.MBB performs better than DBA on the small AusEnergy-Demand dataset, where DBA has limited source-series information.
7. Conclusions
The study introduces data augmentation and knowledge-transfer strategies to improve recurrent neural network global forecasting models when only limited time series are available. Across five benchmark datasets, the variants outperform the baseline global model and many statistical forecasting methods under small to medium training-set conditions.
- The framework synthetically generates time series with three augmentation techniques and transfers their learned representations using pooled or transfer learning strategies.Pooling combines augmented and original series during training, while transfer learning pre-trains on augmented series before adapting to the target dataset.
- The proposed variants outperform the baseline global model and many state-of-the-art univariate forecasting methods with statistical significance under small to medium training-set conditions.The evaluation covers five benchmark datasets, including two competition and three real-world datasets.
- A future direction is replacing the model’s stacking architecture with more sophisticated feature extraction, such as Encoder-Decoder architectures.The extracted latent features could then be repurposed for forecasting on a target dataset.