PaperlayerView exact revision on arXiv

Source-linked AI summary

TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables

Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Guo Qin, Haoran Zhang, Yong Liu, Yunzhong Qiu, Jianmin Wang, Mingsheng Long

arXiv:2402.19072v4cs.LGcs.AI

TL;DR

Forecasting only endogenous variables can miss useful external information, while existing approaches do not fully address the asymmetric and irregular nature of exogenous inputs. TimeXer equips a canonical Transformer with patch-wise endogenous attention, variate-wise exogenous interaction, and global endogenous tokens. It achieves consistent state-of-the-art performance across twelve real-world benchmarks and supports both univariate and multivariate forecasting.

  • Problem

    Real-world forecasting often requires external variables because target series are influenced by factors not contained in their own histories, yet incorporating exogenous information remains underexplored.

  • Method

    TimeXer uses endogenous patch-level representations, exogenous variate-level representations, and global endogenous tokens to reconcile temporal and cross-series information without architectural modifications.

  • Results

    TimeXer achieves consistent state-of-the-art performance on twelve real-world forecasting benchmarks and performs effectively in both univariate and multivariate settings.

  • Takeaways & Limitations

    TimeXer provides a general Transformer-based approach for ingesting exogenous information while jointly capturing endogenous temporal dependencies and external-to-endogenous correlations.

  • Takeaways & Limitations

    The discussion suggests that excessive numbers of patches may emphasize overall trend changes and reduce precision at changing points.

Abstract

from arXiv · show

Deep models have demonstrated remarkable performance in time series forecasting. However, due to the partially-observed nature of real-world applications, solely focusing on the target of interest, so-called endogenous variables, is usually insufficient to guarantee accurate forecasting. Notably, a system is often recorded into multiple variables, where the exogenous variables can provide valuable external information for endogenous variables. Thus, unlike well-established multivariate or univariate forecasting paradigms that either treat all the variables equally or ignore exogenous information, this paper focuses on a more practical setting: time series forecasting with exogenous variables. We propose a novel approach, TimeXer, to ingest external information to enhance the forecasting of endogenous variables. With deftly designed embedding layers, TimeXer empowers the canonical Transformer with the ability to reconcile endogenous and exogenous information, where patch-wise self-attention and variate-wise cross-attention are used simultaneously. Moreover, global endogenous tokens are learned to effectively bridge the causal information underlying exogenous series into endogenous temporal patches. Experimentally, TimeXer achieves consistent state-of-the-art performance on twelve real-world forecasting benchmarks and exhibits notable generality and scalability. Code is available at this repository: https://github.com/thuml/TimeXer.

1 Introduction

Forecasting with exogenous variables uses external factors to improve endogenous-series prediction, but requires models to handle asymmetric roles, dependencies, and irregular data. TimeXer addresses this with patch- and variate-level representations plus global endogenous tokens, achieving strong results across twelve datasets.

  • Motivation: External factors such as economic indicators, demographic changes, and societal events influence time-series variation, making exogenous information valuable for forecasting.Electricity prices, for example, depend strongly on market supply and demand.
  • Problem setting: Exogenous variables provide auxiliary information without needing to be forecast, unlike endogenous variables that are the prediction targets.This distinction creates a forecasting setting different from standard multivariate modeling.
  • Challenges: Models must reconcile multiple external factors with endogenous series while reasoning about causal time lags and avoiding unnecessary interactions, complexity, and memory use.Real-world exogenous inputs may also contain missing values, temporal misalignment, frequency mismatch, and length discrepancies.
  • TimeXer: TimeXer uses endogenous patch-level tokens, exogenous variate-level representations, and learnable global endogenous tokens to jointly capture temporal dependencies and cross-series correlations.The global tokens bridge macroscopic endogenous information with temporal patches and external variables.
  • TimeXer: TimeXer empowers the canonical Transformer to model endogenous and exogenous variables simultaneously without architectural modifications.Its attention design operates across patch and variate levels.
  • Results: Experiments on twelve datasets show that TimeXer better utilizes exogenous information for endogenous forecasting in both univariate and multivariate settings.The reported benchmarks cover real-world forecasting tasks.

2 Related Work

Prior work includes Transformer forecasters organized around point-wise, patch-wise, or variate-wise representations, while exogenous forecasting methods commonly concatenate external and endogenous features. TimeXer instead uses embeddings that accommodate irregular or misaligned external series.

  • Transformer-based forecasters: Transformer-based time-series models are categorized by attention granularity into point-wise, patch-wise, and variate-wise representations.Patch-based methods preserve local temporal semantics, whereas variate-oriented methods target cross-series dependencies.
  • Transformer-based forecasters: PatchTST captures temporal dependencies through subseries-level patches but is weak at modeling multivariate correlations under channel independence.Other patch-based models extend this approach with multi-scale representations or broader temporal-pattern modeling.
  • Transformer-based forecasters: Table 1 compares related methods by forecasting capability and marks models applicable to multivariate forecasting without explicitly modeling cross-variate dependency.The ✧ marker identifies that capability distinction.
  • Forecasting with exogenous variables: Classical forecasting methods such as ARIMAX and SARIMAX extend ARIMA to incorporate exogenous variables, while prior neural methods often concatenate exogenous and endogenous features at each time point.These approaches require alignment between the two series types.
  • Forecasting with exogenous variables: TimeXer introduces external information through an embedding strategy that injects it into endogenous patch-wise representations, accommodating time lags and missing records.This design addresses missing values and uneven sampling that challenge aligned feature concatenation.

3 TimeXer

TimeXer adapts the canonical Transformer for forecasting endogenous series with exogenous information through distinct embeddings and hierarchical attention. It combines patch-wise temporal modeling, variate-wise external-information integration, and a global endogenous token, while also generalizing to parallel multivariate forecasting.

  • Problem Settings: TimeXer predicts the endogenous series using historical endogenous observations and multiple exogenous series, which may have different look-back lengths.The exogenous series are covariates used for prediction rather than targets, and Tex need not equal T.
  • Structure Overview: The model repurposes the canonical Transformer without modifying its components, using different embedding strategies for endogenous and exogenous variables.Self-attention captures temporal-wise dependencies, while cross-attention captures variate-wise dependencies.
  • Endogenous Embedding: TimeXer splits the endogenous series into non-overlapping patches, projects them into temporal tokens, and adds one learnable global token for the whole series.The patch tokens and global token are fed into the Transformer encoder.
  • Exogenous Embedding: Each exogenous series is represented by a series-wise variate token, accommodating missing values, misaligned timestamps, different frequencies, and discrepant look-back lengths.Patch-level exogenous representations would add computational complexity and unnecessary noise.
  • Attention Mechanisms: Endogenous self-attention jointly processes temporal patches and the global token, capturing patch dependencies and each patch’s relationship to the entire series.The global token also bridges endogenous and exogenous variables through the cross-attention mechanism.
  • Parallel Multivariate Forecasting: TimeXer extends to parallel multivariate forecasting by treating each variable as endogenous and the remaining variables as exogenous under channel independence.This frames multivariate forecasting as a unified instance of forecasting with exogenous variables.

4 Experiments

TimeXer is evaluated across short-term exogenous forecasting, long-term multivariate forecasting, and practical scenarios including longer histories, missing values, scalability, interpretability, and efficiency. It achieves strong benchmark performance while retaining robustness and favorable efficiency in the reported analyses.

  • Main Results: Experiments cover short-term electricity-price forecasting, long-term multivariate forecasting, and long-term forecasting with exogenous variables across diverse real-world datasets.The evaluation uses electricity price datasets, seven public multivariate benchmarks, and additional exogenous-variable experiments.
  • Main Results: TimeXer achieves consistent state-of-the-art performance on all five short-term electricity-price datasets, outperforming various baseline models.The baselines include Transformer-, CNN-, and linear-based forecasters, including TiDE for exogenous forecasting.
  • Main Results: TimeXer achieves consistent state-of-the-art performance on most conventional multivariate long-term forecasting datasets, demonstrating effectiveness and generality.Results are averaged across prediction lengths {96, 192, 336, 720}.
  • Practical Situations: Increasing endogenous look-back length provides greater benefits than enlarging historical exogenous series, while increasing both improves performance further.The look-back analysis varies lengths from {96, 192, 336, 512, 720}.
  • Practical Situations: Replacing exogenous histories with zeros or random values does not significantly reduce performance, reflecting robustness to uninformative exogenous data.TimeXer’s separate attention layers allow endogenous temporal representations to dominate predictions.
  • Scalability: On a large weather dataset with 3,850 stations and 36 exogenous variables, TimeXer is evaluated for seven-day histories and three-day forecasts across mismatched sampling frequencies.The dataset uses hourly endogenous temperatures and exogenous meteorological indicators sampled every three hours.
  • Model Analysis: Cross-attention between endogenous global and exogenous variate tokens yields attention patterns consistent with known relationships, including CO2 concentration with Air Density.Maximum Wind Velocity receives relatively low attention in the reported Weather case study.
  • Model Analysis: TimeXer has a lower memory footprint than iTransformer with many variables by avoiding interactions among learned exogenous variate tokens across layers.Exogenous tokens are embedded initially, shared across layers, and interact with the endogenous global token through cross-attention.

5 Conclusion

TimeXer equips the canonical Transformer with exogenous-variable modeling without architectural modifications. Its per-patch-per-variate attention and global token capture endogenous temporal dependencies and endogenous–exogenous correlations.

  • Conclusion: TimeXer incorporates exogenous information into the canonical Transformer without modifying its architecture.The design revisits attention in a per-patch-per-variate manner.
  • Conclusion: A global token helps TimeXer capture endogenous temporal dependencies and multivariate correlations between endogenous and exogenous variables.The approach combines patch-level and variate-level representations through the redesigned attention mechanism.

A.1 Dataset Descriptions

The experiments use public multivariate, electricity-price, and large-scale weather datasets with distinct endogenous and exogenous variables. Training follows a fixed PyTorch configuration with specified horizons, patches, and representation dimensions.

  • Dataset Descriptions: Long-term experiments include ECL, Weather, and other public multivariate datasets, while short-term experiments use five electricity-price markets spanning six years each.ECL uses one client’s consumption as endogenous and other clients as exogenous; electricity-price datasets include prices with grid-load and wind-power forecasts.
  • Dataset Descriptions: Dataset descriptions organize dataset sizes as Train, Validation, and Test and abbreviate endogenous and exogenous variables as En. and Ex.These conventions are specified in the dataset-description table.
  • Implementation Details: Experiments run on a single NVIDIA 4090 24GB GPU using Adam, L2 loss, ten epochs with early stopping, and searched model dimensions from {128, 256, 512}.Patch length is 16 for long-term forecasts and 24 for short-term forecasts.

B Ablation Study

The ablations examine patching, patch length, and embedding choices, showing that TimeXer remains efficient and effective while series-wise exogenous representations are generally advantageous.

  • Patching: Non-overlapping patches give TimeXer the lowest complexity while achieving optimal performance compared with overlapping patches.The comparison uses patch length 24 and stride 12 for overlapping patches.
  • Patch Length: Average prediction performance changes little across patch lengths P ∈{2, 4, 6, 8, 12, 24}, indicating robustness to this hyper-parameter.The experiments fix the look-back window at 96 across five short-term electricity price datasets.
  • Embedding Design: TimeXer uses patch-level temporal tokens and a learnable global endogenous token, while representing exogenous variables at the variate level.This design targets temporal dependencies and cross-attention between endogenous and exogenous variables.
  • Embedding Design: Replacing variate embedding with patch embedding generally reduces prediction accuracy and would significantly increase computational complexity for all exogenous variables.On Traffic, however, concatenating variate-level exogenous and endogenous tokens with self-attention performs better than the proposed design.

C TimeXer Generality under Missing Exogenous Values

TimeXer is evaluated under progressively degraded exogenous histories and under extended historical windows, addressing missing-data conditions and look-back generality.

  • Missing Exogenous Values: With endogenous histories kept complete, exogenous masking is increased from 0% to 99% to simulate progressively lower-quality external data.The experiments use three EPF datasets and report forecasting performance under the masked exogenous series setting.
  • Extended Look-back: TimeXer benefits from an extended endogenous or exogenous look-back length under multivariate forecasting on Weather.Figure 8 evaluates historical lengths T ∈{96, 192, 336, 512, 720}.

E Model Efficiency

TimeXer is designed to balance modeling fineness and efficiency, especially when many exogenous variables are present, while retaining competitive multivariate efficiency.

  • Complexity Analysis: TimeXer has O(C) variate-dimensional complexity from cross-attention, whereas iTransformer increases to O(C^2) as the number of exogenous variables C grows.The analysis identifies this as the main source of TimeXer’s efficiency advantage when C is large.
  • Empirical Efficiency: On Weather with input-96-predict-96, TimeXer has similar performance to Crossformer but significantly lower training time and memory usage.Its efficiency is close to PatchTST under the same multivariate forecasting comparison.
  • Complexity Trade-offs: Because TimeXer retains full attention, its quadratic complexity can be reduced by adapting patch length or replacing full attention with linear attention.The cited complexity expression is associated with the unchanged Transformer attention component.

F Representation Analysis

CKA analysis compares first- and last-block representations, showing that TimeXer learns highly similar representations across layers and distinguishes endogenous representation from iTransformer’s global representation.

  • CKA Analysis: TimeXer’s first- and last-layer representations have high CKA similarity, which the analysis associates with appropriate representations for forecasting.The comparison includes other baselines and evaluates representation similarity between the first and last blocks.
  • Representation Scope: Unlike iTransformer’s all-variable representation, the analysis separately examines the endogenous representation because TimeXer distinguishes endogenous and exogenous variables.iTrm-All denotes all-variable representation, while iTrm-En denotes the endogenous-variable representation.

G Discussion

TimeXer performs well on broad future trends but can miss sharp numerical spikes, especially in the Traffic multivariate setting. The authors associate this weakness with patch-level temporal representations and propose balancing temporal patches with global tokens.

  • TimeXer’s MAE on Traffic is close to iTransformer’s, but its MSE remains substantially worse.The squared error amplifies TimeXer’s inaccurate forecasts of future spikes.
  • TimeXer and PatchTST capture overall future trends effectively but struggle to predict the precise values of changing points.Both models use patch-level representations for temporal dependencies.
  • The authors attribute the issue to many temporal patches competing with one global variate-level token during prediction.They suggest increasing patch length or adding learnable tokens to rebalance these representations.
  • Removing either endogenous or exogenous history produces poorer predictions, indicating that both information sources contribute to forecasting performance.The visualization also considers settings with predicted exogenous values and without historical endogenous information.

H.2 Visualization of Prediction Results

Visualizations compare TimeXer with five forecasting baselines across BE, DE, and PJM datasets. Across the showcased injection points, TimeXer is reported to predict endogenous targets more precisely, particularly near inflection points, while maintaining stable EPF performance.

  • Dataset showcases: The BE, DE, and PJM showcases compare endogenous forecasts against ground truth while displaying selected exogenous variables.Inputs use a look-back length of 168 and a prediction horizon of 24.
  • Prediction behavior: TimeXer forecasts endogenous targets more precisely at marked inflection points, whereas other models tend to oscillate around or exceed the ground truth.The authors interpret this behavior as greater robustness than the alternatives.
  • Dataset showcases: TimeXer is reported to outperform all challengers when predicting four injections over 24 future time points in BE, DE, and PJM examples.The comparisons include Crossformer, iTransformer, PatchTST, TiDE, and DLinear.
  • Prediction behavior: The visual analysis reports stable TimeXer performance on the EPF dataset across five runs with different random seeds.The stability is reported alongside additional baseline comparisons.

I.4 Full Results of Long-term Multivariate Forecasting

The paper evaluates TimeXer on long-term multivariate forecasting benchmarks using a 96-step look-back and horizons of 96, 192, 336, or 720. The supplied material also documents reproducibility, statistical-reporting, compute-resource, and societal-impact checklist information.

  • Experimental setting: Long-term multivariate forecasting uses a look-back length of 96 and prediction lengths of 96, 192, 336, and 720.Results are listed in Table 12.
  • Resources and scope: The supplied checklist material states that the paper has no theoretical results and discusses societal-impact considerations for foundational research.It also records a discussion-section reference for limitations.
  • Reproducibility: The paper states that the architecture is fully described in the main text and refers to Appendix A.2 for implementation details.Code is available through the supplementary materials and the TimeXer repository.
  • Statistical reporting: The checklist notes that error bars should identify their variability source and whether they represent standard deviation or standard error.It also recommends explaining how reported error bars were calculated.
  • Resources and scope: The paper reports sufficient computer-resource information for reproducing experiments and directs readers to Appendix A.2.The checklist concerns compute-worker type, memory, and execution time.
Loading 2402.19072v4…