PaperlayerView exact revision on arXiv

Source-linked AI summary

LSTM-MSNet: Leveraging Forecasts on Sets of Related Time Series with Multiple Seasonal Patterns

Kasun Bandara, Christoph Bergmeir, Hansika Hewamalage

arXiv:1909.04293v2stat.APcs.LGcs.NEstat.ML

TL;DR

Forecasting multiple seasonal cycles is difficult because existing methods are largely univariate and cannot use shared structure across related series. LSTM-MSNet combines global LSTM learning with seasonal decomposition, and its training choices are adapted to whether series are homogeneous or heterogeneous.

  • Problem

    Existing multi-seasonal forecasting methods are mostly univariate, so they cannot exploit shared patterns across related time series.

  • Method

    LSTM-MSNet globally trains an LSTM across related series and supplements learning with statistical decomposition, normalisation, and alternative seasonal-training paradigms.

  • Results

    LSTM-MSNet achieves competitive forecasting results, with decomposition helping heterogeneous datasets while exogenous seasonal variables or no preprocessing suit homogeneous series.

  • Takeaways & Limitations

    The framework can be adapted to the seasonal characteristics of a time-series group by selecting deseasonalised or seasonal-exogenous training.

Abstract

from arXiv · show

Generating forecasts for time series with multiple seasonal cycles is an important use-case for many industries nowadays. Accounting for the multi-seasonal patterns becomes necessary to generate more accurate and meaningful forecasts in these contexts. In this paper, we propose Long Short-Term Memory Multi-Seasonal Net (LSTM-MSNet), a decomposition based, unified prediction framework to forecast time series with multiple seasonal patterns. The current state of the art in this space are typically univariate methods, in which the model parameters of each time series are estimated independently. Consequently, these models are unable to include key patterns and structures that may be shared by a collection of time series. In contrast, LSTM-MSNet is a globally trained Long Short-Term Memory network (LSTM), where a single prediction model is built across all the available time series to exploit the cross series knowledge in a group of related time series. Furthermore, our methodology combines a series of state-of-the-art multiseasonal decomposition techniques to supplement the LSTM learning procedure. In our experiments, we are able to show that on datasets from disparate data sources, like e.g. the popular M4 forecasting competition, a decomposition step is beneficial, whereas in the common real-world situation of homogeneous series from a single application, exogenous seasonal variables or no seasonal preprocessing at all are better choices. All options are readily included in the framework and allow us to achieve competitive results for both cases, outperforming many state-of-the-art multi-seasonal forecasting methods.

I. INTRODUCTION

Multiple seasonal cycles create forecasting challenges in high-frequency series, while common univariate methods cannot exploit shared structure across related series. LSTM-MSNet addresses this gap with a unified LSTM framework supplemented by decomposition techniques and alternative training paradigms.

  • Motivation: High-frequency series can contain daily, weekly, yearly, and other seasonal effects that complicate accurate forecasting.An Australian household example has daily period 48 and weekly period 336; longer observations may also show yearly seasonality with period 17532.
  • Research gap: Univariate methods forecast each series independently, preventing them from using shared information in collections of correlated time series.Related series occur in retail demand, computer-centre performance, and household smart-meter data.
  • Research gap: Although unified models and neural sequence methods have advanced, handling multiple seasonal patterns across related series remains insufficiently studied.The paper identifies RNNs and LSTMs as recent approaches suited to sequence data, while noting limited explicit treatment of multiple seasonality.
  • Contribution: LSTM-MSNet globally trains an LSTM across related series to exploit common seasonal structures and behaviours.The framework is designed to improve forecasting by borrowing strength across the collection rather than estimating each series separately.
  • Contribution: The framework supplements LSTM learning with multiple decomposition techniques and provides training paradigms for homogeneous and inhomogeneous series groups.The paper motivates these choices because global-model competitiveness depends on the characteristics of the series collection.

III. LSTM-MSNET FRAMEWORK

LSTM-MSNet is a global forecasting framework that combines decomposition with an LSTM architecture for series containing multiple seasonal patterns. Its pipeline normalises and transforms inputs, processes them recurrently, then reconstructs forecasts.

  • Problem statement: The problem formulation uses observations from all series and their seasonal periods to forecast M future points for a target series.The global model f uses the collection X, seasonal information S, and parameters θ to produce the forecast.
  • Architecture: LSTM-MSNet has preprocessing, recurrent, and post-processing layers that transform inputs, learn with an LSTM, and restore final forecasts.Post-processing denormalises and reseasonalises the time series.
  • Normalisation and variance stabilisation: The global model uses each series’ mean as a scale factor because series in the collection may have different value ranges.The normalised observation and the number of observations are explicitly defined in the framework’s scaling description.
  • Normalisation and variance stabilisation: After normalisation, a logarithmic transformation stabilises variance and converts seasonality to an additive form.The additive form is required by the decomposition techniques used later in the pipeline.

C. Seasonal Decomposition

The seasonal decomposition layer separates multiple recurring patterns before LSTM learning. MSTL extends STL by iteratively extracting multiple seasonal components alongside trend and remainder terms.

  • Decomposition strategy: LSTM-MSNet uses statistical decomposition techniques to detach slowly changing, repeating seasonal components from a time series.The resulting deseasonalised series is intended to reduce the complexity of the subsequent neural-network learning process.
  • STL: STL additively decomposes an observation into seasonal, trend, and remainder components that can reconstruct the original series.The decomposition represents the original observation as the sum of its component estimates.
  • MSTL: MSTL extends STL by iteratively estimating multiple seasonal components in addition to trend and remainder.Its formulation sums distinct seasonal components with the trend and remainder terms.
  • MSTL: The MSTL implementation allows control over seasonal smoothness through the s.window parameter.Setting s.window to “periodic” limits changes in extracted seasonal components to zero, separating deterministic seasonality.

2) Seasonal-Trend decomposition by Regression (STR):

STR is a regression-based additive decomposition technique, while Prophet and TBATS are used to extract multiple seasonal components through decomposition procedures. Fourier terms represent periodic effects with configurable seasonal periodicities and flexibility.

  • Seasonal-Trend decomposition by Regression (STR): STR is a regression-based additive decomposition technique that can incorporate external regressors, although this study uses its default mode without them.STR is implemented through the AutoSTR function from the stR package in R.
  • TBATS: TBATS is used as a deseasonalisation technique to extract relevant seasonal components after fitting the model with the tbats function.TBATS is described as handling complex seasonal patterns through trigonometric terms.
  • Prophet: Prophet uses an additive decomposition layer with seasonal, trend, residual, and holiday components, and is applied here to obtain multiple seasonal components.The authors apply the Prophet algorithm through the prophet package in R.
  • Fourier Transformation: Fourier terms approximate periodic effects using separate seasonal periodicities, with k controlling the number of sine-cosine pairs and the speed of seasonal variation represented.Higher k values represent seasonal patterns that change more quickly than lower k values.
  • Summary: Table I summarizes the decomposition techniques, their software packages, and the types of seasonalities extracted.The table provides an overview of the methods used for multi-seasonal decomposition.

D. Recurrent Layer

The recurrent layer is the primary prediction module of LSTM-MSNet and uses stacked LSTMs to model sequence data. Its peephole-connected gating mechanism combines short- and long-term temporal information through the hidden state and memory cell.

  • Recurrent Layer: The recurrent layer is LSTM-MSNet’s primary prediction module and uses LSTMs within a stacked architecture for sequence modelling.LSTMs are used because their recurrent structure is suited to sequence data and time series research.
  • Peephole-connected LSTM: LSTM gating and its self-contained memory cell enable the network to capture nonlinear long-term temporal dependencies.The architecture uses input and forget gates that include the previous memory-cell state.
  • Peephole-connected LSTM: Peephole connections allow the hidden and memory states to participate in gate computations through dedicated weight matrices and peephole matrices.The input, forget, output, and memory-cell gates use corresponding weights, input weights, peephole weights, and biases.
  • State updates: The candidate cell state updates the memory cell, while element-wise multiplication, sigmoid, and hyperbolic tangent functions implement the recurrent equations.The hidden update also uses tanh as its activation function.

1) Moving Window Transformation:

LSTM-MSNet converts past observations into moving input-output windows for multi-step forecasting, then supports deseasonalised or seasonal-exogenous training paradigms. DS reduces training complexity but depends on the base learner, whereas SE supplies seasonal components as exogenous inputs.

  • Moving Window Transformation: The Moving Window strategy transforms each time series into multiple input-output frames that serve as LSTM-MSNet’s primary training data.The transformation is applied before network training.
  • Moving Window Transformation: A series of length K yields K − n − m records, each containing n input observations and m output observations under the MIMO principle.The output window directly predicts the intended multi-step forecast horizon.
  • Moving Window Transformation: The output window length m is set equal to the forecast horizon M, avoiding potential error accumulation from step-by-step forecasting.The final output window is reserved for validation and hyper-parameter tuning.
  • Training Paradigms: The DS paradigm trains on seasonally adjusted windows and later requires reseasonalisation, whereas SE uses seasonal components as exogenous variables and needs no reseasonalisation.SE retains the original time-series components in training, while DS excludes seasonal factors.
  • Moving Window Transformation: Figure 3 illustrates an input window of x1 through xn and an output window of y1 through ym after preprocessing and seasonal adjustment.The displayed windows are formed after normalisation, variance stabilisation, and decomposition.
  • Training Paradigms: DS reduces LSTM training complexity by excluding seasonal factors, while SE uses exogenous variables to assist modelling seasonal trajectories.The two approaches differ in which components enter the moving-window training process.
  • Training Paradigms: DS performance depends on the balance between the deseasonalisation base learner and the LSTM, motivating tests of different base learners.The paper notes that base-learner complexity can affect subsequent LSTM training and final performance.
  • Training Paradigms: SE may require the LSTM to learn relationships that vary substantially with seasonal period and within-period position, which can be difficult from limited input windows.The paper empirically examines this learning capacity in practical situations.

3) LSTM Learning Scheme:

LSTM-MSNet trains with an L1 objective augmented by L2 regularisation, then post-processes forecasts through reseasonalisation and renormalisation. The DS paradigm requires reseasonalisation, whereas SE does not.

  • Loss Function: The L1 objective minimises absolute differences between target and estimated output-window values and is intended to be robust to anomalies.The loss is defined over the training time steps.
  • Loss Function: The learning objective includes an L2-regularisation term to minimise possible network overfitting.The regularisation term is weighted by ψ and uses the network weights wi.
  • Post-processing: In DS, post-processing adds the relevant seasonal components to LSTM forecasts by repeating the last seasonal components across the forecast horizon.SE does not require this reseasonalisation phase.
  • Training Paradigms: Figure 4 contrasts DS, which trains on deseasonalised targets and requires reseasonalisation, with SE, which uses seasonal exogenous variables and original-distribution targets.The two paradigms differ in whether seasonal information is removed or supplied as input.
  • LSTM Architecture: Figure 5 represents the peephole LSTM over time, with hidden state ht for short-term dependencies and memory cell Ct for long-term dependencies.Wt, Dt, and Ŷt denote the input window, dense layer, and projected LSTM output at time t.
  • Post-processing: Renormalisation back-transforms forecasts to the original scale by restoring the local factor, exponentiating values, and applying the normalisation scaling factor.This produces the final forecasts after post-processing.

IV. EXPERIMENTS

The experiments evaluate LSTM-MSNet variants and benchmarks on three datasets containing multiple seasonal cycles, using sMAPE and MASE to assess forecast accuracy. Dataset heterogeneity and seasonal distributions are also examined.

  • Datasets: The evaluation covers M4-Hourly, AusGrid-Energy, and Traffic datasets with multiple seasonal cycles.AusGrid-Energy includes half-hourly and aggregated hourly series, while the Traffic data contain hourly freeway occupancy series.
  • Seasonal characteristics: The datasets differ in seasonal-distribution homogeneity, with substantial variation among M4 series.The seasonal components are extracted with MSTL from the initial 200 observations of each series.
  • Evaluation metrics: The experiments compare forecasting methods using sMAPE and MASE.sMAPE is symmetric mean absolute percentage error, while MASE is mean absolute scaled error.

C. Statistical tests of the results

The study uses statistical significance testing alongside systematic hyperparameter selection and compares LSTM-MSNet variants with established multi-seasonal forecasting benchmarks.

  • Statistical testing: Friedman rank-sum tests and Hochberg post-hoc procedures assess whether forecasting-method differences are statistically significant.The tests use sMAPE and a significance level of α = 0.05.
  • Hyperparameter selection: Bayesian global optimisation selects hyperparameters more systematically than Random Search and Grid Search.The optimisation targets the unknown validation function over candidate hyperparameter settings.
  • Benchmarks: The comparison includes TBATS, Prophet, FFORMA, and two Dynamic-Harmonic-Regression variants.The variants differ in whether tslm or auto.arima is used through the forecast package.
  • LSTM-MSNet variants: LSTM-MSNet variants include decomposition-based training paradigms and an LSTM baseline trained directly on original observations.DS excludes seasonal factors, whereas SE supplies seasonal variables as exogenous inputs.

F. Computational Performance

Across the benchmark datasets, different LSTM-MSNet variants lead on different data characteristics, while most proposed variants outperform major statistical benchmarks. Computational costs are generally below TBATS and FFORMA but above Prophet and Dynamic-Harmonic-Regression (T).

  • M4 dataset: LSTM-MSTL-DS achieves the best Mean sMAPE on the 414 hourly M4 series, while FFORMA leads Median sMAPE and Median MASE.Dynamic-Harmonic-Regression with auto.arima achieves the best Mean MASE; LSTM-MSNet variants with DS generally outperform SE.
  • AusGrid-Energy half-hourly: LSTM-MSTL-SE leads Mean and Median sMAPE on the 300 half-hourly AusGrid-Energy series, while LSTM-Prophet-DS leads Mean and Median MASE.Most SE variants outperform DS variants on this dataset, and both named variants surpass the state of the art across performance metrics.
  • AusGrid-Energy hourly: LSTM-Fourier-SE (k = 1) achieves the best results on every metric for aggregated hourly AusGrid-Energy data.SE variants outperform their DS counterparts, while the proposed variants outperform FFORMA, TBATS, and Prophet.
  • Traffic dataset: LSTM-Fourier-SE (k = 1) outperforms the other benchmarks on the Traffic dataset, and most LSTM-MSNet variants beat FFORMA, TBATS, and Prophet.The Friedman test reports p < 10^-10, with LSTM-Fourier-SE (k = 1) as the control method.
  • Computational cost: Most proposed LSTM-MSNet variants have lower execution times than TBATS and FFORMA, but Prophet and Dynamic-Harmonic-Regression (T) are more computationally efficient.The latter methods do not display competitive results according to the reported comparison.

I. Discussion

LSTM-MSNet’s performance depends on the seasonal characteristics and homogeneity of the dataset. Deseasonalisation helps heterogeneous series, while seasonal exogenous inputs and selected decompositions perform better for homogeneous series and can improve computational efficiency.

  • The LSTM-Baseline performs inconsistently across datasets, outperforming the DS paradigm on AusGrid-Energy and Traffic but not on M4.M4’s less homogeneous seasonal components make direct seasonal learning difficult, whereas longer series may provide enough data to learn seasonality directly.
  • MSTL, STR, and Prophet perform best on M4, while MSTL and Prophet variants perform better on AusGrid-Energy and Traffic.STR-based variants are unstable on AusGrid-Energy, which the paper attributes to the longer series lengths.
  • MSTL and Prophet are computationally more efficient decomposition techniques than TBATS.
  • DS is more suitable for inhomogeneous series with unknown origins, whereas SE is better for homogeneous series sharing similar seasonal-component shapes.
  • For longer time series, most LSTM-MSNet variants are computationally more efficient than state-of-the-art univariate forecasting techniques.

V. CONCLUSION

The paper presents LSTM-MSNet as a unified framework for forecasting related time series with multiple seasonal cycles. It combines decomposition with globally trained LSTMs and finds competitive accuracy and computational time, while identifying hybrid seasonal handling as future work.

  • LSTM-MSNet is a three-layered framework that forecasts related time series with multiple seasonal cycles using decomposition and LSTM networks.Its unified model exploits structures, behaviours, and patterns shared within a group of time series.
  • The framework uses decomposition techniques and two training paradigms—Deseasonalised and Seasonal Exogenous—to supplement LSTM learning.
  • LSTM-MSNet is competitive with current state-of-the-art methods in accuracy and computational time, and can be more efficient than many univariate methods.
  • A hybrid approach is proposed as future work, combining deseasonalisation for shorter seasonalities with seasonal exogenous inputs for longer seasonalities.
Loading 1909.04293v2…