Timer-XL: Long-Context Transformers for Unified Time Series Forecasting

1 INTRODUCTION

• Time-series Transformers typically use contexts of up to hundreds of tokens, limiting global-tendency learning for univariate series and modeling of inter-channel dependencies.The paper contrasts this with natural-language and vision Transformers that learn dependencies across thousands to millions of tokens.Source: §1 · p. 1
• Decoder-only Transformers accommodate varying lookback and prediction lengths, and the authors report that they mitigate long-context degradation observed in encoder-only forecasters.The comparison is reported in Figure 3.Source: §1 · pp. 1–2
• The paper generalizes language-model training to multivariate next-token prediction, treating forecasting tasks as long-context prediction over multidimensional series.Timer-XL extends context to thousands of patch tokens.Source: §1 · p. 2, Figure 1 · p. 2
• TimeAttention models intra- and inter-series dependencies with positional awareness while maintaining causality and scalability.Specialized position embeddings encode temporal order and permutation-equivalence across variables.Source: §1 · p. 2
• Timer-XL achieves state-of-the-art performance on univariate, multivariate, and covariate-informed benchmarks, with notable zero-shot performance after large-scale pre-training.The model is presented as a versatile, one-for-all forecasting Transformer.Source: §1 · p. 2

2 RELATED WORK

• Time-series Transformers have used point-wise, patch-wise, and variable-wise tokenizations, but few works uniformly address multidimensional series with long-context Transformers without architectural modification.The paper uses causal Transformers to formulate these tasks as multivariate next-token prediction.Source: §2 · pp. 2–3
• Decoder-only models support multi-length forecasting, but channel-independent pre-training leaves them inaccessible to inter-series dependencies.The paper contrasts this with encoder-only work that captures multivariate dependencies but can be incompatible with causal forecasting.Source: §2 · p. 3
• Timer-XL renovates attention to disentangle token dependencies into variable dependencies and temporal causal masks; Table 1 compares representative time-series Transformers and highlights their differences.The attention design targets intra- and inter-series dependencies while maintaining causality and scalability.Source: §2 · p. 3, Table 1 · p. 3

3.1 TIMER

• Timer treats consecutive groups of P time points as patch tokens and trains a decoder-only Transformer to predict each next patch from prior tokens.The model embeds and projects tokens and supervises predictions with MSE loss.Source: §3.1 · p. 3

3.2 GENERALIZE 1D SEQUENCES TO 2D TIME SERIES

• Timer-XL generalizes next-token prediction to multivariate patches, letting each prediction use prior-time tokens from all variables and expanding context length from T to NT.This captures within-series causal dependencies while incorporating correlations from other series.Source: §3.2 · p. 4
• RoPE encodes temporal order, while two learnable scalars per head distinguish endogenous from exogenous series and preserve variable permutation-equivalence.The design requires chronological ordering over time while shuffling variable order changes only output order.Source: §3.2 · p. 4
• TimeAttention flattens multivariate patches in temporal-first order and uses variable dependencies with a temporal causal mask to capture causal patch-wise dependencies within and across variables.For multivariate forecasting, the variable-dependency matrix is all ones; the Kronecker product expands the temporal mask into a block mask.Source: §3.2 · pp. 4–5, Figure 2 · p. 5
• Timer-XL supports univariate, multivariate, and covariate-informed forecasting by flattening 2D time series into one context and supervising target variables.Covariate settings specify variable dependencies and use position embeddings for temporal and variable dimensions.Source: §3 · p. 3, §3.2 · p. 5

4.1 UNIVARIATE TIME SERIES FORECASTING

• Timer-XL generally improves univariate accuracy when daily context is extended to monthly, outperforms encoder-only and linear forecasters in excessively long contexts, and beats PatchTST at every ERA5-S site.The monthly and yearly context gains slow and can deteriorate; the authors suggest increased noise and training difficulty may contribute.Source: §4.1 · pp. 5–6, Table 2 · p. 6

4.2 MULTIVARIATE TIME SERIES FORECASTING

• Timer-XL achieves the best results on prior and new multivariate benchmarks, improves over Timer by 13.2% on ECL and 6.3% on Traffic, and generalizes across prediction lengths better than UniTST.The evaluation includes established and long-context benchmarks; the reported gains are especially on high-dimensional series with thousands of context tokens.Source: §4.2 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7
• Timer-XL combines explicit inter-series modeling with a decoder-only architecture, a combination the ablation identifies as suitable when multivariate training samples are sufficient.The comparison varies channel independence versus explicit inter-series modeling and encoder-only versus decoder-only architecture.Source: §4.2 · p. 7, Table 5 · p. 8

4.3 COVARIATE-INFORMED TIME SERIES FORECASTING

• Timer-XL outperforms state-of-the-art models on covariate-informed EPF forecasting, and the causal version performs better than its noncausal covariate variant.Timer-XL models fine-grained patch-wise covariate dependencies, unlike TimeXer’s whole-covariate token.Source: §4.3 · p. 8, Table 6 · p. 8

4.4 PRE-TRAINED TIME-SERIES TRANSFORMERS

• Timer-XL beats PatchTST in all three ERA5-Large generalization settings, and Timer-XL with longer-context pre-training outperforms Timer on all benchmark datasets in zero-shot evaluation.The generalization tests hold out stations, time spans, or both; pre-training context grows from 1440 to 2880.Source: §4.4 · pp. 8–9, Figure 5 · p. 9, Table 7 · p. 9

4.5 MODEL ANALYSIS

• Measured Timer-XL overhead is much less than N times Timer’s despite theoretical attention complexity O(N^2T^2) versus O(NT^2), and FlashAttention reduces its memory footprint to O(NT) without affecting performance.Feed-forward components have O(NT) complexity and dominate overhead in existing benchmarks where contexts are not large enough.Source: §4.5 · pp. 9–10, Figure 6 · p. 10
• TimeAttention visualizations show stronger diagonal submatrices, token dependencies consistent with autocorrelation lags, and averaged submatrix values that illustrate raw-data Pearson correlations.The showcase uses a Traffic sample with length 672 and visualizes the last 10 variables, each with seven tokens.Source: §4.5 · p. 10, Figure 7 · p. 10

5 CONCLUSION AND FUTURE WORK

• Timer-XL targets unified forecasting with causal Transformers, enhanced self-attention for temporal dynamics and variable correlations, and large-scale heterogeneous pre-training.The conclusion also identifies challenging long-context benchmarks and future work on computational efficiency and domain-specific large models.Source: §5 · p. 10

A.1 SETUPS

• L denotes Transformer blocks, D the embedding dimension, H the head count, and dk the query, key, and value dimension per head.The FFN hidden dimension is αD; the analysis considers N variables and TP time points, with patch length P and context length T.Source: §A.1 · p. 14

A.2 FLOPS

• For typical short contexts where T ≪ D, linear transformations dominate time-series Transformer computation rather than self-attention.FlashAttention can increase FLOPs through recomputation yet improve runtime by reducing memory reads and writes.Source: §A.2 · p. 15
• Channel-independent models treat N variables as a batch multiplier, while fine-grained intra- and inter-series models expand the token count to NT.The FLOPs expressions for these approaches differ in their attention terms, with the latter scaling with N^2T^2.Source: §A.2 · p. 15

A.3 PARAMETER COUNT

• The parameter-count expressions distinguish flatten-head projection from token-wise projection, with the latter using fewer projection parameters.Both approaches include attention, normalization, and feed-forward parameters.Source: §A.3 · p. 15

A.4 MEMORY FOOTPRINT

• Training memory is predominantly determined by activation storage, because model and optimizer parameter memory is negligible in practice.The activation-memory expression excludes the quadratic attention-map storage when FlashAttention is used.Source: §A.4 · p. 16

B.1 DATASETS

• ERA5-S spans 40 years across seven subsets; ERA5-MS uses those subsets as a multivariate benchmark, and ERA5-Large covers 4920 stations for pre-training.ERA5-Large contains around half a billion time points.Source: §B.1 · pp. 16–17

B.2 BASELINE MODELS

• The comparisons cover univariate, multivariate, covariate-informed, and zero-shot forecasting, with more than 20 baselines included.The baseline set includes PatchTST for univariate forecasting and TimeXer for covariate-informed forecasting.Source: §B.2 · p. 17

B.3 IMPLEMENTATION DETAILS

• Experiments use PyTorch on NVIDIA A100 GPUs, Adam optimization, and MSE loss; univariate forecasting adopts channel independence.The reported standard deviations across three random-seed runs indicate stable Timer-XL performance.Source: §B.3 · p. 17, Table 10 · p. 18
• MSE and MAE evaluate most experiments, while SMAPE measures one-for-all generalization on ERA5-Large.For multivariate series, metrics are averaged over variables.Source: §B.3 · p. 19
• The configuration table lists Timer-XL and baseline Transformer settings, and the dataset table specifies variables and chronological split lengths.The reported configurations use Adam with default β1 = 0.9 and β2 = 0.999.Source: Table 9 · p. 18, Table 11 · p. 19, §B.3 · p. 19

C HYPERPARAMETER SENSITIVITY

• Timer-XL performance generally improves with more layers, while its optimal patch size is generally close to the prediction length.The authors suggest different input and output patch sizes as a future direction for better long-term forecasting; the optimal inference lookback need not match training length.Source: C HYPERPARAMETER SENSITIVITY · pp. 19–20, Figure 8 · p. 20

D SHOWCASES

• In randomly selected forecasting showcases, Timer-XL presents the most accurate predictions among the compared Transformers.The comparisons include PatchTST, Timer, and UniTST across the visualized datasets and prediction settings.Source: D SHOWCASES · p. 20, Figure 9 · p. 20

• Each model is trained with 672 input and 96 output steps, then rolled forward to forecast lengths of 96, 192, 336, or 720.Source: §E.1 · p. 20
• Time-series Transformers significantly outperform other baseline models on the one-for-all multivariate forecasting benchmark, where Timer-XL achieves state-of-the-art performance.The authors describe Timer-XL as a potential backbone for a one-for-all forecaster.Source: §E.1 · p. 20
• The benchmark evaluates deep forecasters’ versatility for real-world forecasting requirements without extensive model training and storage.Source: §E.1 · p. 20

E.2 FULL RESULT OF ZERO-SHOT FORECASTING

• Zero-shot evaluation uses Timer-XL pre-trained on aggregated UTSD and LOTSA datasets, with rolling forecasts extended to lengths of 96, 192, 336, or 720.Source: §E.2 · pp. 20, 22
• Evaluation uses datasets excluded from each model’s pre-training, each model’s maximum inference input length, and MSE/MAE averaged over predicted test windows.Source: §E.2 · p. 22

E.3 ABLATION STUDY OF TIMEATTENTION

• Variable embeddings distinguishing endogenous from exogenous variables improve performance, while RoPE outperforms other temporal position embeddings.The authors attribute RoPE’s advantage to its affine transformation, compared with the additive transformations of other embeddings.Source: §E.3 · p. 22

E.4 SUPPLEMENTARY RESULTS OF LONG-CONTEXT FORECASTING

• Timer-XL and DLinear generally improve as ERA5 context grows, whereas PatchTST degrades after thousands of time points.The ERA5 evaluation covers lookbacks from daily to yearly contexts.Source: §E.4 · p. 22, Table 15 · p. 24
• In learned-attention case studies, the decoder-only model focuses selectively on prior context, while PatchTST focuses on noisy parts.The authors connect causal masking with coherent token embeddings and say unmasked attention may break causality.Source: §E.4 · p. 22, Figure 11 · p. 24
• Timer-XL achieves competitive benchmark performance without ReVIN, whereas PatchTST may rely heavily on this normalization.The text notes that instance normalization can cause unmatched statistics during decoder-only multi-step autoregression.Source: §E.4 · p. 24, Table 16 · p. 24

E.5 ILLUSTRATION OF TIMEATTENTION

• TimeAttention uses grouped causality over flattened 2D sequences, with a Kronecker-based mask separating temporal and variable dependencies.Source: §E.5 · p. 24, Figure 12 · p. 25
• The dependency formulation can extend to covariates or predefined variable dependencies.Source: §E.5 · p. 25

• Long-term autoregressive forecasting may accumulate errors and limit output-length flexibility.Source: F LIMITATIONS · p. 25
• The authors plan to incorporate multiresolution patches for input and output series.Source: F LIMITATIONS · p. 25
• TimeAttention complexity remains a reduction target, particularly for high-dimensional and lengthy series.Source: F LIMITATIONS · p. 25