Source-linked AI summary
Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case
Neo Wu, Bradley Green, Xue Ben, Shawn O'Banion
TL;DR
ILI reports are delayed, motivating forecasting methods for timely disease monitoring. The paper develops a Transformer-based framework that uses self-attention for time-series dependencies and applies it to ILI forecasting, where it achieves state-of-the-art results. The framework also supports observed series, embeddings, and univariate or multivariate data.
Problem
Weekly ILI reports typically have at least a one-week delay, making forecasting important for real-time disease monitoring and public-health resource planning.
Method
The paper develops a general Transformer-based forecasting model using self-attention to learn dependencies in time-series data.
Results
The Transformer-based model achieves state-of-the-art forecasting results for ILI.
Takeaways & Limitations
The framework can model observed time series and phase spaces through time-delay embeddings, and can be adapted to univariate and multivariate data.
Takeaways & Limitations
The paper's current case study focuses on time-series data; extending the approach to spatio-temporal data remains future work.
Abstract
from arXiv · showhide
In this paper, we present a new approach to time series forecasting. Time series data are prevalent in many scientific and engineering disciplines. Time series forecasting is a crucial task in modeling time series data, and is an important area of machine learning. In this work we developed a novel method that employs Transformer-based machine learning models to forecast time series data. This approach works by leveraging self-attention mechanisms to learn complex patterns and dynamics from time series data. Moreover, it is a generic framework and can be applied to univariate and multivariate time series data, as well as time series embeddings. Using influenza-like illness (ILI) forecasting as a case study, we show that the forecasting results produced by our approach are favorably comparable to the state-of-the-art.
1. Introduction
The paper introduces a general Transformer-based approach for time series forecasting, using self-attention to learn complex dependencies. An influenza-like illness case study demonstrates accurate forecasting and state-of-the-art results.
- Motivation: ILI forecasting supports real-time disease monitoring and helps public health agencies allocate resources for potential pandemics.Weekly ILI reports typically have at least a one-week delay because of data collection and aggregation.
- Approach: The proposed approach processes entire sequences and uses self-attention to learn dependencies rather than processing observations in order.The authors position this design as suitable for complex time-series dynamics.
- Contributions: The authors developed a general Transformer-based model for time series forecasting.
- Contributions: The approach is complementary to state space models and can model observed data, state variables, and system phase spaces through embeddings.
- Evaluation: Using ILI forecasting as a case study, the model accurately forecasts ILI prevalence with a variety of features.
- Evaluation: The Transformer-based model achieves state-of-the-art forecasting results in the ILI case.
2. Related Work
Prior ILI forecasting work spans mechanistic, statistical, machine learning, Internet-data, and deep learning approaches. Internet-based models evolved from Google Flu Trends toward autoregressive, adaptive, ensemble, and attention-based methods.
- Internet-based forecasting: Google Flu Trends used predefined Google search terms in a linear model to estimate current ILI ratios, but later overestimated peak magnitudes.
- Internet-based forecasting: Researchers extended or improved Google Flu Trends with autoregression, query selection, model updates, and additional signals.
- Deep learning: Deep learning ILI models have incorporated Google Trends, climate, air pollution, virological surveillance, and spatio-temporal adjustment factors.
- Deep learning: Attention-based ILI forecasting has included multi-channel LSTM networks that learn from different input types.
3. Background
The background introduces influenza-like illness surveillance, state space and sequence-modeling approaches, and their respective limitations. It also motivates time-delay embeddings as a way to connect observed data with underlying dynamical systems.
- 3.1. Influenza and ILI: Influenza-like illness (ILI) ratios summarize weekly patients with ILI symptoms relative to total patient visits.The CDC publishes USA and state-level ILI ratios, with state values normalized by population.
- 3.2. State Space Models: State space models represent observable variables and the non-observable state variables that control dynamical-system evolution.The observation equation relates observations to states, while the state equation governs state evolution over time.
- 3.2. State Space Models: Compartmental models such as SIR describe infectious-disease dynamics through time-dependent state variables, with ILI as an observable variable.The SIR example uses susceptible, infected, and recovered states, and defines ILI(t) as the infected proportion.
- 3.2. State Space Models: Compartmental models require prior knowledge of differential-equation parameters and lack flexibility when updating parameters after new observations.This limits their adaptability for settings where new surveillance data arrive over time.
- 3.2. State Space Models: Time-delay embeddings encode past scalar observations into delay coordinates that can approximate underlying phase-space dynamics.Prior work used TDEs for short-range disease forecasting, while the paper notes that their use in machine-learning models has been limited.
- 3.3. Sequence Models: RNNs process inputs recurrently, whereas LSTMs use input, forget, and output gates to address gradient problems on long sequences.Seq2Seq models use encoder, context-vector, and decoder components, but long sequences can lose long-term dependencies in the intermediate vector.
4. Model
The model formulates ILI forecasting as supervised learning from fixed-length historical windows and uses a Transformer encoder–decoder to predict future values. Training and evaluation construct input–output pairs with sliding windows, while benchmarking compares the Transformer with ARIMA, LSTM, and Seq2Seq models.
- Problem Description: ILI forecasting is formulated as supervised learning, with historical observations serving as inputs and future observations as prediction targets.Each data point may be scalar or contain multiple features.
- Problem Description: Fixed-length sliding windows construct training and test pairs from scaled ILI data, using previous N observations as features and next M observations as labels.The dataset uses a 2:1 train–test split and min-max scaling based on training-set extrema.
- Model Architecture: The Transformer model follows an encoder–decoder architecture with input projection, positional encoding, four encoder layers, four decoder layers, and an output layer.The encoder maps inputs to dmodel-dimensional vectors, while the decoder maps its final representation to the target time sequence.
- Model Architecture: Look-ahead masking and a one-position decoder offset ensure each predicted time-series point depends only on preceding data points.The decoder begins with the encoder input’s last data point and applies attention over encoder outputs.
- Training Data and Batching: The training setup predicts four future weekly ILI ratios from ten trailing weekly data points.The encoder receives (x1, ..., x10), the decoder receives (x10, ..., x13), and the target is (x11, ..., x14).
- Evaluation and Benchmarks: The evaluation compares the Transformer with ARIMA, LSTM, and Seq2Seq benchmarks using Pearson correlation and root-mean-square error.ARIMA is used as a univariate baseline, while LSTM and Seq2Seq provide recurrent neural-network comparisons.
5. Experiment
The experiments evaluate Transformer forecasting on ILI data alone, multivariate feature vectors, and time-delay embeddings. Results show competitive performance with state-of-the-art ARGONet and limited gains from additional features, while embedding dimensionality mainly affects RMSE.
- 5.1. One-step-ahead Forecasting Using ILI Data Alone: The Transformer was evaluated against ARIMA, LSTM, and Seq2Seq with attention for one-week-ahead ILI forecasting.Evaluation used Pearson correlation and RMSE across states.
- 5.1. One-step-ahead Forecasting Using ILI Data Alone: 27% and 8.4% relative RMSE decreases were achieved by Transformer compared with LSTM and Seq2Seq with attention, respectively.Correlation coefficients were similar across the deep learning models, with Transformer slightly higher.
- 5.2. One-step-ahead Forecasting Using Feature Vectors: The multivariate model incorporated week number and first- and second-order differences as additional features for ILI forecasting.These features were introduced to test whether the model could learn from multiple signals.
- 5.2. One-step-ahead Forecasting Using Feature Vectors: Mean Pearson correlation of 0.931 and mean RMSE of 0.585 were obtained with the additional features, but the improvement over ILI alone was not significant.The authors suggest the added difference features encode little new information.
- 5.2. One-step-ahead Forecasting Using Feature Vectors: Against ARGONet, Transformer achieved slightly higher mean correlation but slightly worse mean RMSE.ARGONet had correlation 0.912 and RMSE 0.550; Transformer had correlation 0.931 and RMSE 0.593.
- 5.3. Forecasting Using Time Delay Embedding: TDE dimensionality did not significantly alter Pearson correlation, while RMSE reached its minimum at dimensionality 8.The experiments varied embedding dimensions from 2 to 32 with τ = 1.
6. Conclusions
The paper presents a Transformer-based forecasting framework that uses self-attention to model dependencies across varied lengths. It is described as generic and extensible across observed series, phase-space embeddings, and univariate or multivariate data, while spatio-temporal extension remains future work.
- 6. Conclusions: The proposed Transformer approach uses self-attention to learn complex dependencies of various lengths in time series data.This distinguishes it from sequence-aligned deep learning methods.
- 6. Conclusions: The framework can model observed time series and phase space through time-delay embeddings.The paper demonstrates both capabilities in the ILI case study.
- 6. Conclusions: The approach is adaptable to univariate and multivariate time series with minimum modifications to model implementations.The authors characterize it as a generic framework for various non-linear dynamical systems.
- 6. Conclusions: Extending the approach to spatio-temporal data indexed by time and location is identified as future work.The paper presents this extension as a hypothesis rather than a demonstrated capability.