Source-linked AI summary
Multi-resolution Time-Series Transformer for Long-term Forecasting
Yitian Zhang, Liheng Ma, Soumyasundar Pal, Yingxue Zhang, Mark Coates
TL;DR
Patch-based transformers can model temporal patterns within and across patches, but existing patch-based methods do not explicitly incorporate multi-scale analysis. MTST combines multiple patch sizes in a multi-branch transformer with relative positional encoding, and reports the lowest MSE in all 28 test settings.
Problem
Patch-based time-series transformers do not explicitly incorporate multi-scale analysis, despite patch tokenization helping attention capture temporal patterns.
Source: §1 · p. 1Method
MTST uses multiple patch-size tokenizations in multi-branch transformer layers and relative positional encoding to model temporal patterns at different scales.
Source: §1 · pp. 1–2Results
MTST achieves the lowest mean squared error on 28 out of 28 test settings across multiple forecasting benchmarks.
Source: §1 · p. 2Takeaways & Limitations
MTST’s multi-resolution branches let it model temporal patterns of different frequencies simultaneously, with larger patches capturing coarser patterns and smaller patches finer ones.
Source: §1 · p. 1The forecasting setup assumes a training dataset is available for learning model parameters.
Source: §2 · p. 2
Abstract
from arXiv · showhide
The performance of transformers for time-series forecasting has improved significantly. Recent architectures learn complex temporal patterns by segmenting a time-series into patches and using the patches as tokens. The patch size controls the ability of transformers to learn the temporal patterns at different frequencies: shorter patches are effective for learning localized, high-frequency patterns, whereas mining long-term seasonalities and trends requires longer patches. Inspired by this observation, we propose a novel framework, Multi-resolution Time-Series Transformer (MTST), which consists of a multi-branch architecture for simultaneous modeling of diverse temporal patterns at different resolutions. In contrast to many existing time-series transformers, we employ relative positional encoding, which is better suited for extracting periodic components at different scales. Extensive experiments on several real-world datasets demonstrate the effectiveness of MTST in comparison to state-of-the-art forecasting techniques.
1 INTRODUCTION
Timestamp-level tokenization can limit transformers’ ability to capture temporal patterns, while patch-based methods have not explicitly incorporated multi-scale analysis. MTST uses multiple patch sizes and relative positional encoding to model patterns at different scales, and achieves the lowest MSE in all 28 test settings.
1 INTRODUCTION
- Patch-based transformers can model patterns within a patch and relationships between patches, whereas timestamp-level tokenization can prevent effective pattern capture.The introduction notes that some time-series transformers are insensitive to token order and can be outperformed by simple linear models.Source: §1 · p. 1
- MTST forms multiple token sets with different patch sizes and uses multi-branch self-attention to model temporal patterns at different frequencies simultaneously.Larger patches mainly capture lower-frequency, coarser patterns, while smaller patches contribute to higher-frequency, finer patterns.Source: §1 · p. 1
- MTST uses relative positional encoding, which the authors describe as naturally aligned with periodic temporal patterns at different scales.This replaces learned or fixed absolute positional encoding in the design described in the introduction.Source: §1 · pp. 1–2
- MTST reaches the lowest mean squared error on 28 out of 28 test settings across multiple forecasting benchmarks.The experiments also include ablation studies and analysis of the proposed designs.Source: §1 · p. 2
2 PROBLEM STATEMENT
The paper frames long-term multivariate forecasting as predicting future timestamps from a recent historical window. It assumes a training dataset and assesses test-set forecasts using MSE and MAE.
2 PROBLEM STATEMENT
- The forecasting task predicts the next T timestamps from the preceding L timestamps of a multivariate time series.L is the look-back window and T is the prediction horizon.Source: §2 · p. 2
- The setup assumes a training dataset is available to learn the model parameters.The training examples contain historical inputs and targets, while future values in test examples must be estimated.Source: §2 · p. 2
- Forecasting performance is assessed on the test set with mean squared error and mean absolute error between predictions and ground truth.The passage introduces the equations defining both metrics.Source: §2 · p. 2
3 METHODOLOGY
MTST models time-series at multiple resolutions through parallel branches that tokenize representations into patches, apply self-attention with relative positional encoding, and fuse branch outputs. It processes each variate independently by default and combines the resulting forecasts.
3 METHODOLOGY
- MTST processes each variate independently by default, then combines the separate forecasts into a multivariate forecast.The architecture is not limited to channel-independence and can be extended to patch-based transformers that model dependencies among variates.Source: §3 · p. 2
3.1 Branch specific tokenization
- Branch-specific patch sizes let shorter patches model short-term features at higher resolution, while longer patches support learning longer-term seasonalities and trends.Each branch tokenizes the layer representation into overlapping patches using its patch size and stride; padding is added at the end when needed to make the token count an integer.Source: §3.1 · p. 3
3.2 Self-attention
- Self-attention operates independently on each branch’s patch-level tokens to capture relationships between tokens, using relative-position information and multi-head weights.The attention output is further processed with residual connections, normalization, and a feed-forward network.Source: §3.2 · pp. 3–4
3.2.1 Relative positional encoding
- Relative positional encoding represents the distances between token pairs; on a synthetic dataset with strong periodic patterns, it helped tokens identify similar tokens more accurately and improved forecasts.The encoding uses the sign of the token-index difference and a positional function of its absolute value.Source: §3.2.1 · p. 4
3.3 Fusing representations from all branches
- At each layer, branch representations are fused into one embedding, allowing information to be shared across scales before the next layer.The described fusion begins by flattening each branch output and concatenating the flattened vectors.Source: §3.3 · p. 4
4 RELATED WORK
Long-horizon forecasting has moved from timestamp-level transformers toward patch-based models, while research has explored multi-scale features and positional encoding. MTST is presented as a multi-resolution approach designed to extract multiple periodicities.
4.1 Long-horizon Time-Series Forecasting
- Convolutional and recurrent models are described as failing to capture long-range dependencies, while patch-based transformers address limits of timestamp-level tokenization by using multi-timestep windows as tokens.Earlier transformer approaches treated timestamps as tokens; several models also incorporated inductive biases for trends and seasonalities.Source: §4.1 · p. 4
4.2 Multi-scale Feature Learning
- Prior multi-scale approaches use methods such as multi-rate sampling, varied convolution kernels, 2D sequence representations, or pyramidal attention, while MTST is designed to extract multiple periodicities through multi-resolution decomposition.The passage notes that Pyraformer and Scaleformer forecasts are dramatically inferior to PatchTST, which does not form explicit multi-scale decompositions.Source: §4.1 · p. 4, §4.2 · pp. 4–5
4.3 Positional Encoding
- MTST uses relative positional encoding, which represents distances between tokens and is described as better suited than absolute encoding for modeling periodic patterns in time-series.The passage attributes the relative encoding design to a useful inductive bias for forecasting.Source: §4.3 · p. 5
5 EXPERIMENTS
Across seven benchmark datasets, MTST achieves state-of-the-art performance in every dataset, horizon, and metric case. Ablations and analyses generally support its multi-resolution design and relative positional encoding.
5.1.1 Datasets; 5.1.2 Baselines and Experimental Setup
- The evaluation uses seven public benchmarks, compares MTST with transformer, multiscale, and linear baselines, and follows the PatchTST setup with L=336 and T∈{96, 192, 336, 720}.Training and test splits are 6:2:2 for the ETT datasets and 7:1:2 for the others.Source: §5.1.1 · p. 5, §5.1.2 · p. 5
5.1.4 Experimental Results
- MTST has the best average algorithm rank and lowest rank variability across seven datasets and four prediction horizons.Ranks are based on MSE; Figure 4 shows their distributions.Source: Figure 4 · p. 5, §5.1.4 · p. 6
- MTST achieves state-of-the-art performance in all cases and significantly outperforms PatchTST on 27 of 28 MSE cases.The main comparison uses seven datasets, four prediction horizons, and MSE and MAE.Source: Table 2 · p. 6, §5.1.4 · p. 6
5.1.3 Hyperparameters
- MTST uses instance normalization and denormalization, scaled dot-product attention, batch normalization, relative positional encoding, and Adam optimization with MSE loss.Architecture and training hyperparameters vary by dataset and horizon and are detailed in the appendix.Source: §5.1.3 · p. 6
5.2.1 Ablation: Multi-Resolution
- Removing either the lowest- or highest-resolution branch generally degrades performance, with a few small-dataset exceptions for removing the highest-resolution branch.The largest-patch branch mainly targets low-frequency patterns, while the smallest-patch branch mainly targets high-frequency patterns; the exceptions may reflect overfitting.Source: §5.2.1 · p. 7, Table 3 · p. 7
5.2.2 Ablation: Positional Encoding
- Relative positional encoding benefits forecasting in most cases compared with fixed or learned absolute positional encoding.The comparison uses MTST with relative positional encoding as the base model.Source: §5.2.2 · pp. 7–8, Table 4 · p. 7
5.2.3 Analysis: Look-back Window
- MTST’s advantage over PatchTST and DLinear persists across nearly all tested look-back windows for prediction horizons T=96 and 720.The analysis tests L∈{48, 96, 192, 336, 720} on ETTh1 and Traffic.Source: Figure 5 · p. 7, §5.2.3 · p. 8
5.2.4 Analysis: Qualitative Comparisons
- In two qualitative examples, MTST better predicts the overall trend after a deep trough and the amplitude of a high peak than PatchTST.The authors attribute these examples potentially to the low-resolution and high-resolution branches, respectively.Source: §5.2.4 · p. 8, Figure 6 · p. 8
6 CONCLUSION
MTST brings multi-scale analysis to patch-based time-series transformers using multiple patch sizes and a multi-branch architecture. Experiments on seven multivariate datasets show performance beyond previous state-of-the-art approaches.
6 CONCLUSION
- MTST uses multiple patch-based tokenizations and multi-branch transformer layers to model temporal patterns at different scales.The design avoids the up/down-sampling techniques used in previous work.Source: §6 · p. 9
- Relative positional encoding is incorporated to match the seasonality characteristics found in many real-world datasets.Source: §6 · p. 9
- Experiments on seven multivariate datasets show MTST outperforming previous state-of-the-art approaches; ablations support the design choices.Visualized predictions also support the authors’ multi-scale learning hypothesis.Source: §6 · p. 9
A.1 Detailed Architecture of Transformer Layers in MTST
The transformer layer applies residual connections, batch normalization, and a feed-forward network after self-attention, then fuses branch outputs. MTST has attention complexity based on the finest-resolution token count and reports training times similar to PatchTST.
A.1 Detailed Architecture of Transformer Layers in MTST
- After self-attention, the layer applies residual connections, batch normalization, and a two-layer feed-forward network before fusing outputs from different branches.Batch normalization and the feed-forward network’s weights and biases are specified in the architectural description.Source: §A.1 · p. 12
A.2 Computational Complexity
- MTST’s attention complexity is O(J_b′n^2), where J_b′n is the finest-branch token count; its training time is reported as similar to PatchTST.The authors attribute similar training time to MTST’s empirically shallower optimal architectures despite its multiple branches.Source: §A.2 · p. 12
B EXPERIMENTAL DETAILS
The experiments align baseline look-back windows for comparison, tune MTST over a specified set of architectural choices, and report variability across random seeds.
B.1 Baseline settings
- Transformer baselines use L=96, but TimesNet and MICN are also rerun at L=336 because 96 timestamps may be inadequate for horizons over 300.MTST is tested at L=336 as well for a fair comparison.Source: §B.1 · p. 12
B.2 Hyperparameters
- MTST grid search varies layers across {1, 2, 3}, branches across {2, 3, 4}, and branch patch sizes across {4, 8, 16, 24, 36, 48, 64, 96}.Stride defaults to half the patch size; the model is trained by minimizing forecast MSE with Adam.Source: §B.2 · p. 13, Table 6 · p. 13
C.1 Forecasting Results with Standard Deviations
- Three runs with seeds 2021, 2022, and 2023 show small MSE and MAE variances, indicating minimal impact from changing the random seed.The main-paper Table 2 results use seed 2021.Source: §C.1 · p. 13
C.2 Results for a Shorter Look-back Window (L=96)
With L=96, MTST ranks first in most reported MSE and MAE cases, while PatchTST leads on ETTm1; the section also specifies MTST's shorter-window setup.
C.2 Results for a Shorter Look-back Window (L=96)
- For L=96, MTST uses two layers and two branches with patch lengths [12, 16] across datasets and horizons.Other hyperparameters follow the L=336 setting.Source: §C.2 · p. 13
- The authors associate PatchTST's ETTm1 lead with the possibility that one-branch patch models can perform slightly better on small datasets because of high-resolution overfitting.Source: §C.2 · p. 13
- At L=96, MTST ranks first in 19 of 28 MSE cases and 25 MAE cases, while PatchTST leads on ETTm1.The table reports prediction lengths of 96, 192, 336, and 720.Source: §C.2 · p. 13, Table 8 · p. 14
C.3 Results on ILI Dataset
MTST outperforms baselines on the ILI dataset across its tested horizons and uses adjusted patch sizes to represent multiple scales. The related-work discussion frames patch-level multi-scale modeling as an open design question, and forecasting visualizations contrast the model with other methods.
C.3 Results on ILI Dataset
- MTST surpasses all baselines on ILI across horizons 24, 36, 48, and 60 in MSE.The experiments use L=104; the table marks statistically significant differences between the top two results with an asterisk.Source: §C.3 · p. 15, Table 9 · p. 15
D.1 Long-term Time-Series Forecasting
- PatchTST (Nie et al., 2023) uses patch-level tokens, but incorporating time-series inductive biases after patching remains an open question; MTST is described as among the first to add multi-scale features.Source: §D.1 · pp. 15–16
D.2 Multi-scale Feature Learning
- Rather than pooling patch-level tokens, which can yield sub-optimal representations, the paper constructs multi-resolution representations by adjusting patch sizes.Source: §D.2 · p. 16
E FORECAST VISUALIZATION
- On 336-step forecasts, MTST captures Electricity weekend patterns better than PatchTST and DLinear, and predicts Traffic peaks more accurately than the compared baselines.The text attributes weekend-pattern capture to the low-resolution component and finer Traffic predictions to the high-resolution branch; removing that branch noticeably lowers performance.Source: E FORECAST VISUALIZATION · pp. 16–17, Figure 7 · p. 17, Figure 8 · p. 18