PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting

Bryan Lim, Sercan O. Arik, Nicolas Loeff, Tomas Pfister

arXiv:1912.09363v3stat.MLcs.LG

TL;DR

Existing multi-horizon forecasting models often treat heterogeneous inputs as black boxes, limiting both input coverage and interpretability. TFT addresses this with specialized temporal, feature-selection, gating, and attention components, achieving 7% lower P50 and 9% lower P90 losses on average than the next-best model.

  • Problem

    Existing multi-horizon forecasting architectures often overlook heterogeneous inputs and provide black-box predictions without interpretable temporal insights.

  • Method

    TFT combines sequence-to-sequence and attention-based temporal processing with static covariate encoding, variable selection, gating, and quantile prediction.

  • Results

    7% lower P50 and 9% lower P90 losses on average than the next-best model across the evaluated datasets.

  • Takeaways & Limitations

    TFT demonstrates the benefits of aligning forecasting architecture with heterogeneous inputs and temporal relationships while enabling interpretable multi-horizon forecasts.

Abstract

from arXiv · show

Multi-horizon forecasting problems often contain a complex mix of inputs -- including static (i.e. time-invariant) covariates, known future inputs, and other exogenous time series that are only observed historically -- without any prior information on how they interact with the target. While several deep learning models have been proposed for multi-step prediction, they typically comprise black-box models which do not account for the full range of inputs present in common scenarios. In this paper, we introduce the Temporal Fusion Transformer (TFT) -- a novel attention-based architecture which combines high-performance multi-horizon forecasting with interpretable insights into temporal dynamics. To learn temporal relationships at different scales, the TFT utilizes recurrent layers for local processing and interpretable self-attention layers for learning long-term dependencies. The TFT also uses specialized components for the judicious selection of relevant features and a series of gating layers to suppress unnecessary components, enabling high performance in a wide range of regimes. On a variety of real-world datasets, we demonstrate significant performance improvements over existing benchmarks, and showcase three practical interpretability use-cases of TFT.

1. Introduction

Multi-horizon forecasting provides predictions across multiple future time steps for important applications, but practical settings combine heterogeneous inputs with unknown interactions and often rely on difficult-to-explain black-box models. The paper introduces the Temporal Fusion Transformer (TFT), an attention-based architecture designed to improve forecasting performance while enabling interpretability.

  • Introduction: Multi-horizon forecasting predicts variables of interest across multiple future time steps, helping users optimize actions over an entire future path.Applications include inventory optimization, treatment planning, retail, healthcare, and economics.
  • Introduction: Practical forecasting combines known future information, historically observed exogenous series, and static metadata without prior knowledge of their interactions.Examples include holiday dates, customer foot traffic, and store location.
  • Introduction: Many existing architectures fail to accommodate the heterogeneity of common multi-horizon forecasting inputs.The introduction identifies this limitation alongside the broader development of recurrent and attention-based forecasting models.
  • Introduction: Black-box nonlinear interactions make forecasts difficult to explain, hindering user trust and model debugging.Conventional post-hoc methods such as LIME and SHAP are also described as poorly suited to time series because they ignore input-feature ordering.
  • Introduction: The Temporal Fusion Transformer is an attention-based DNN architecture for multi-horizon forecasting that aims to deliver high performance with new forms of interpretability.Its design aligns the architecture with the full range of potential inputs and temporal relationships in multi-horizon forecasting.

2. Related Work

Prior deep-learning approaches to multi-horizon forecasting divide into iterated autoregressive and direct sequence-to-sequence methods. Related interpretability work uses attention or variable-importance techniques, but limitations remain for static covariates and time-ordered inputs.

  • DNNs for Multi-horizon Forecasting: Deep-learning multi-horizon forecasting methods comprise iterated autoregressive approaches and direct sequence-to-sequence approaches.
  • DNNs for Multi-horizon Forecasting: Iterated methods recursively feed predictions into future inputs, with Deep AR and DSSM using LSTMs for one-step-ahead forecasting.
  • DNNs for Multi-horizon Forecasting: Direct methods generate forecasts for multiple predefined horizons, commonly using sequence-to-sequence architectures with encoders and horizon-specific prediction components.
  • Time Series Interpretability with Attention: Time-series attention methods use attention weights to identify salient inputs, but prior approaches did not consider the importance of static covariates.
  • Instance-wise Variable Importance with DNNs: Post-hoc explanation methods such as LIME, SHAP, and RL-LIM are not designed for input time ordering, limiting their use for complex time-series data.

3. Multi-horizon Forecasting

The multi-horizon forecasting setup models multiple entities with static covariates, time-dependent inputs, and scalar targets, while producing simultaneous quantile forecasts across future time steps. Historical target and known-input information is restricted to a finite look-back window and the forecast start time.

  • Problem formulation: Each entity i has static covariates, time-dependent inputs, and scalar targets observed at each time step.The setup represents static covariates as s_i, time-dependent inputs as χ_i,t, and targets as y_i,t.
  • Problem formulation: Time-dependent inputs include features measurable only at each step and known inputs that can be predetermined, such as the day of the week.The passage distinguishes inputs that are unknown beforehand from predetermined known inputs x_i,t.
  • Forecasting objective: Quantile regression produces prediction intervals by forecasting the 10th, 50th, and 90th percentiles at each future time step.Prediction intervals support indications of likely best- and worst-case target values for decision optimization and risk management.
  • Forecasting objective: The model simultaneously outputs forecasts for τ_max future steps, using past targets and known inputs within a finite look-back window through the forecast start time.The direct forecasting setup predicts τ ∈ {1, . . . , τ_max} and incorporates past information within window k.

4. Model Architecture

TFT combines input-specific representations, variable selection, gated residual processing, static-context enrichment, and interpretable temporal attention. Its architecture supports adaptive complexity while modeling both local processing and long-term temporal relationships.

  • Input representations: TFT processes static metadata, known future inputs, and historically observed inputs through canonical components designed for broad forecasting scenarios.The architecture includes separate representations for static, known, and observed inputs.
  • Temporal fusion: TFT uses recurrent layers for local processing and modified multi-head self-attention for long-term dependencies, sharing values across heads so attention weights remain interpretable.Different heads can learn distinct temporal patterns while attending to a common set of input features through additive aggregation.
  • Gated residual processing: Gating mechanisms let TFT skip unused components, while GLU-based GRNs suppress nonlinear contributions when simpler processing is sufficient.GLU outputs can be close to zero, effectively skipping a GRN layer and enabling adaptive depth and network complexity.
  • Prediction intervals: Quantile forecasts provide prediction intervals that estimate the range of likely target values at each prediction horizon.This is a dedicated TFT component for uncertainty-aware multi-horizon prediction.
  • Variable selection: Variable selection networks choose relevant variables instance-wise for static and time-dependent covariates, helping exclude noisy inputs.Separate selection networks are used for static, past, and future inputs; temporal selection weights are generated using GRNs and Softmax.
  • Static enrichment: Separate static GRN encoders produce context vectors for temporal variable selection, local temporal processing, and static enrichment of temporal features.The contexts are cs, ce, cc, and ch, with roles spanning selection, local processing, and enrichment.

5. Loss Functions

TFT is trained by jointly minimizing quantile loss across multiple output quantiles. At test time, performance is assessed using normalized quantile losses over the full forecasting horizon, with emphasis on P50 and P90 risk.

  • TFT is trained by jointly minimizing quantile loss summed across all quantile outputs.
  • The experiments use output quantiles Q = {0.1, 0.5, 0.9}.
  • 0.5 and 0.9: normalized quantile losses are evaluated across the entire forecasting horizon, focusing on P50 and P90 risk for consistency with previous work.

6. Performance Evaluation

The evaluation spans simple and complex real-world multi-horizon forecasting datasets and compares TFT with direct and iterative benchmarks. TFT outperforms competing methods, while ablations show that temporal processing, feature selection, covariate encoders, and gating all contribute to performance.

  • Datasets: The study evaluates TFT on Electricity and Traffic datasets with known inputs, plus Retail and Volatility datasets containing richer static and observed inputs.Retail forecasts log product sales 30 days ahead using 90 days of history; Volatility contains daily realized volatility for 31 stock indices and daily returns.
  • Baselines: TFT is compared with direct models including Seq2Seq and MQRNN, and with iterative models including DeepAR, DSSM, and ConvTrans.For complex datasets, comparisons focus on ConvTrans because of its prior strong performance against other iterative models.
  • Forecasting performance: 7% lower P50 and 9% lower P90 losses on average versus the next best model demonstrate TFT’s performance advantage across the evaluated datasets.Table 2 reports TFT outperforming competing methods across all experiments, improving on the next best alternative by between 3% and 26%.
  • Forecasting performance: Accounting for observed inputs matters on complex datasets, where ConvTrans performs more poorly when observed-input imputation is required, while direct quantile regression helps when targets are poorly modeled by Gaussian distributions.The Traffic dataset provides an example of the benefits of direct models under non-Gaussian target distributions.
  • Ablation analysis: Temporal local processing and self-attention have the largest ablation impacts, increasing P90 loss by > 6% on average and > 20% on select datasets.Local processing is especially critical in Traffic, Retail, and Volatility, with temporal-component impacts varying across datasets.
  • Ablation analysis: Static covariate encoders, instance-wise variable selection, and gating increase average P90 loss by more than 2.6%, 4.1%, and 1.9%, respectively.Gating is most important for Volatility, where its ablation increases P90 loss by 4.1%; covariate encoders and variable selection provide their largest benefits on Electricity.

7. Interpretability Use Cases

The TFT supports three interpretability use cases: identifying important input variables, revealing persistent temporal patterns, and detecting regimes or events associated with shifts in temporal dynamics. Across datasets, it extracts intuitive input subsets, learns seasonality from raw data, and changes attention behavior across volatility regimes.

  • Overview: The paper frames interpretability around input importance, persistent temporal patterns, and regime or event detection as three complementary analyses of learned model relationships.These use cases examine variable selection weights, attention patterns, and temporal-dynamics shifts respectively.
  • Variable importance: The TFT’s variable-selection analysis identifies a subset of key inputs that intuitively play significant roles in predictions.For Retail, entity-identifying static covariates and past target values receive particularly important weights.
  • Persistent temporal patterns: Attention weights reveal persistent temporal patterns learned directly from raw training data rather than human-specified seasonality or lag rules.Electricity and Traffic show daily attention spikes, Retail shows weaker weekly patterns and a decaying recent-history trend, while Volatility lacks strong persistent patterns.
  • Persistent temporal patterns: The learned temporal patterns can support expert sanity-checking and guide feature engineering or additional data collection.These uses follow from the model learning persistent patterns without hard-coding them.
  • Regime identification: The TFT’s attention behavior changes across volatility regimes, providing evidence that it learns different temporal dynamics under low- and high-volatility conditions.For S&P 500 realized volatility, attention is distributed across past inputs during low volatility but focuses more on sharp trend changes during high volatility, including around the 2008 financial crisis.

8. Conclusions

The paper introduces TFT as an attention-based deep learning model for interpretable, high-performance multi-horizon forecasting. Its specialized components address static covariates, known future inputs, and historically observed inputs across diverse forecasting datasets.

  • Conclusions: TFT is a novel attention-based deep learning model for interpretable, high-performance multi-horizon forecasting.The conclusion presents TFT as the paper’s central contribution.
  • Conclusions: TFT is designed to handle static covariates, a priori known inputs, and observed inputs across a wide range of multi-horizon forecasting datasets.The architecture explicitly targets these different input types and broad dataset settings.
  • Conclusions: Its temporal processing components use sequence-to-sequence and attention-based mechanisms to capture time-varying relationships at different timescales.The passage identifies these mechanisms as specialized components of TFT.
  • Conclusions: TFT also includes static covariate encoders as part of its specialized architecture.The supplied passage names static covariate encoders but does not provide further detail about their operation.

APPENDIX · Appendix A. Dataset and Training Details

Appendix A provides feature pre-processing and train/test-split details for reproducibility across Electricity, Traffic, Retail, and Volatility datasets. The datasets use distinct sampling periods, normalization procedures, inputs, entities, and forecast horizons.

  • Appendix A. Dataset and Training Details: The appendix documents feature pre-processing and train/test splits to support reproducibility.The stated goal is to provide sufficient implementation information.
  • Appendix A. Dataset and Training Details: Electricity uses 500k samples from 2014-01-01 to 2014-09-01, with the first 90% for training and the last 10% for validation.Testing covers the 7 days immediately following the training set, with entity-wise z-score normalization for real-valued inputs.
  • Appendix A. Dataset and Training Details: Across the datasets, the appendix specifies dataset-specific temporal windows, validation and test horizons, normalization choices, and covariates.Electricity and Traffic use 7-day tests, Retail uses a 30-day horizon after validation, and Volatility uses calendar-period splits.
  • Appendix A. Dataset and Training Details: Traffic uses 500k training samples before 2008-06-15 and follows Electricity’s split and 7-day post-training test procedure.Z-score normalization is applied across entities; inputs include occupancy, calendar variables, a time index, and categorical entity identifiers.
  • Appendix A. Dataset and Training Details: Retail treats each product number-store number pair as a separate entity, totaling over 135k entities.It uses 450k training samples, 50k validation samples from the subsequent 30 days, and testing over all entities for the following 30-day horizon; inputs come from the Kaggle competition and daily resampling imputes missing days using the last available observation.
  • Appendix A. Dataset and Training Details: Volatility spans 2000-01-03 to 2019-06-28, with training before 2016, validation during 2016-2017, and testing from 2018 onward.The target is 5-min sub-sampled realized volatility, augmented with daily open-to-close returns and calendar and region variables.

Appendix B. Interpretability Results

Appendix B presents the remaining interpretability results, complementing the prominent findings highlighted in Section 7. Its purpose is to provide the additional results for completeness.

  • Appendix B reports interpretability results apart from the prominent findings highlighted in Section 7.
  • The appendix contains the remaining results not presented in Section 7.
  • These additional results are included for completeness.

Appendix B.1. Variable Importance

Variable importance analysis shows that TFT assigns full static-input weight to dataset identifiers or region, while emphasizing past target values and calendar effects among time-dependent inputs. Past target values rank among the top inputs across datasets, and hour-of-day is especially important for Electricity and Traffic forecasts.

  • Static inputs: The network allocates full weight to the entity identifier for Electricity and Traffic, and to the region input for Volatility.These are the only static inputs available in the respective datasets.
  • Time-dependent inputs: Calendar effects also matter, with hour-of-day playing a significant role in forecasts.In Electricity and Traffic, both past and future hour-of-day values are important, highlighting the role of seasonality.
  • Time-dependent inputs: Past values of the target are among the top 1 or 2 most significant past inputs across Electricity, Traffic, and Volatility.The table identifies past target values as consistently important time-dependent inputs.
Loading 1912.09363v3…