Source-linked AI summary
Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting
Peng Chen, Yingying Zhang, Yunyao Cheng, Yang Shu, Yihang Wang, Qingsong Wen, Bin Yang, Chenjuan Guo
TL;DR
Transformers for time series forecasting often model limited or fixed scales, making characteristics across multiple scales difficult to capture. Pathformer combines multi-scale patch division, dual attention, and adaptive pathways, achieving strong forecasting performance and generalization across transfer scenarios.
Problem
Transformers for time series forecasting mainly model time series from limited or fixed scales, making characteristics spanning various scales challenging to capture.
Method
Pathformer uses multiple patch sizes, inter-patch and intra-patch attention, and adaptive pathways that select and aggregate scale-specific characteristics from input dynamics.
Results
Pathformer is best in 81 of 88 multivariable forecasting cases, with 8.1% lower MSE and 6.4% lower MAE than PatchTST, and surpasses baselines in transfer evaluation.
Takeaways & Limitations
Pathformer demonstrates strong prediction performance and generalization capability across forecasting tasks and transfer scenarios.
Abstract
from arXiv · showhide
Transformers for time series forecasting mainly model time series from limited or fixed scales, making it challenging to capture different characteristics spanning various scales. We propose Pathformer, a multi-scale Transformer with adaptive pathways. It integrates both temporal resolution and temporal distance for multi-scale modeling. Multi-scale division divides the time series into different temporal resolutions using patches of various sizes. Based on the division of each scale, dual attention is performed over these patches to capture global correlations and local details as temporal dependencies. We further enrich the multi-scale Transformer with adaptive pathways, which adaptively adjust the multi-scale modeling process based on the varying temporal dynamics of the input, improving the accuracy and generalization of Pathformer. Extensive experiments on eleven real-world datasets demonstrate that Pathformer not only achieves state-of-the-art performance by surpassing all current models but also exhibits stronger generalization abilities under various transfer scenarios. The code is made available at https://github.com/decisionintelligence/pathformer.
1 INTRODUCTION
Time series contain patterns across diverse temporal resolutions and distances, exposing limitations in fixed or incomplete Transformer designs. Pathformer addresses these challenges with integrated multi-scale attention and adaptive pathways.
- Time series forecasting supports applications across energy, finance, traffic, logistics, and cloud computing, while also supporting other analytics.
- Real-world series exhibit variations across temporal scales, requiring models to capture dependencies from multiple temporal intervals.Temporal resolution determines patch or unit length, while temporal distance determines which time steps are modeled together.
- Existing multi-scale Transformer designs can be incomplete because changing resolution does not explicitly and efficiently model dependencies across different ranges.Temporal-distance modeling is also affected by the data division, so a single resolution view remains incomplete.
- Pathformer combines multiple patch sizes with dual attention to model global correlations across patches and local details within patches.Its multi-scale Transformer block unifies temporal resolution and temporal distance.
- Adaptive pathways use a multi-scale router to select patch sizes and subsequent attention based on input dynamics.The router works with temporal decomposition and an aggregator to adaptively extract and combine multi-scale characteristics.
- Experiments report state-of-the-art prediction accuracy and strong generalization in transfer learning across datasets.
2 RELATED WORK
Related work covers Transformer architectures for time series forecasting and growing use of multi-scale modeling in other domains. Prior time-series methods address attention efficiency, temporal dynamics, and multi-scale representations through varied designs.
- Transformer Models for Time Series: Transformer models have gained attention in time series forecasting through designs including sparse attention, triangular architectures, auto-correlation, and Fourier transformations.
- Multi-scale Modeling for Time Series: Multi-scale modeling has proven effective for correlation learning and feature extraction in computer vision and multi-modal learning.
- Multi-scale Modeling for Time Series: Multi-scale modeling remains relatively less explored in time series forecasting than in computer vision and multi-modal learning.
3 METHODOLOGY
Pathformer combines multi-scale temporal resolutions and distances through patch division, dual attention, and adaptive pathways. Its router selects patch scales from input dynamics, while aggregation combines the resulting representations.
- Overall Architecture: The Pathformer network consists of Instance Norm, stacked Adaptive Multi-Scale Blocks, and a Predictor.The AMS Block contains the multi-scale Transformer block and adaptive pathways.
- Multi-scale Transformer Block: Pathformer’s Multi-scale Transformer Block uses multiple patch sizes and dual attention to model temporal characteristics across scales.Patch sizes provide different temporal resolutions, while inter-patch and intra-patch attention model dependencies across and within patches.
- Multi-scale Division: Multi-scale division splits the input X into patches of size S, producing P = H/S patches with different temporal-resolution views.Each patch contains S time steps and d features.
- Dual Attention: Intra-patch attention captures relationships within each patch, whereas inter-patch attention captures global correlations across patches.The two attention operations are applied to patch-divided series and their outputs are combined into the dual-attention result.
- Adaptive Pathways: Adaptive pathways address scale redundancy by selecting patch sizes according to input-dependent routing weights.The multi-scale router uses temporal decomposition involving seasonality and trend patterns to control scale selection.
4 EXPERIMENTS
Experiments evaluate Pathformer across real-world datasets, forecasting baselines, transfer settings, ablations, parameter choices, and pathway visualizations. Pathformer achieves strong forecasting and transfer results, while adaptive scale selection varies with temporal dynamics.
- Experimental Setup: Pathformer is evaluated on nine real-world datasets spanning electricity, transportation, weather, cloud computing, and illness forecasting.
- Experimental Setup: The comparison includes PatchTST, NLinear, Scaleformer, TIDE, FEDformer, Pyraformer, and Autoformer under shared input and prediction-length settings.
- Main Results: 81 of 88 forecasting cases rank Pathformer first, while 5 rank it second; versus PatchTST, MSE decreases 8.1% and MAE decreases 6.4%.Against Pyraformer and Scaleformer, MSE decreases 36.4% and MAE decreases 19.1%.
- Transfer Learning: Transfer experiments pre-train on ETTh1 and ETTm1 before fine-tuning on ETTh2 and ETTm2, and also assess transfer toward future data.
- Transfer Learning: Across direct prediction and full-tuning, Pathformer surpasses baseline models; part-tuning reduces training time by 52% on average while maintaining nearly comparable accuracy.
- Ablation Studies: Removing adaptive pathways decreases prediction accuracy, while K = 2 and K = 3 outperform K = 1 and K = 4.Pathways use decomposition to help identify patch sizes, and pathway weights favor larger patches for longer seasonality but smaller patches for shorter seasonality.
5 CONCLUSION
Pathformer combines multi-scale temporal resolutions and distances through multiple patch sizes and dual attention, then adaptively selects and aggregates scale-specific characteristics. The resulting model achieves outstanding prediction performance and strong generalization across forecasting tasks.
- Pathformer uses multiple patch sizes and dual attention to model multi-scale temporal resolutions and temporal distances comprehensively.
- Adaptive pathways dynamically select and aggregate scale-specific characteristics according to different temporal dynamics.
- Pathformer demonstrates outstanding prediction performance and strong generalization capability across several forecasting tasks.
A.1.1 DATASETS
The experiments use datasets covering energy, weather, electricity consumption, and related forecasting settings. The dataset statistics are presented in Table 5.
- ETT datasets contain seven variables from two electric transformers, covering January 2016 to January 2018 at 15-minute and one-hour granularities.
- The Weather dataset contains 21 meteorological indicators from Germany collected every 10 minutes.
- The Electricity dataset contains power-consumption measurements from 321 users.
- Table 5 reports the statistics of the datasets used in the experiments.
A.1.2 BASELINES
The study selects recent state-of-the-art forecasting models as baselines, including Autoformer, FEDformer, PatchTST, and NLinear. Their code repositories are documented for reproducibility.
- The baseline set includes the 2021 SOTA Autoformer, 2022 SOTA FEDformer, and 2023 SOTA PatchTST and NLinear.
- The Autoformer code repository is provided at github.com/thuml/Autoformer.
A.2 UNIVARIATE TIME SERIES FORECASTING
Pathformer is evaluated for univariate forecasting on ETT and Cloud cluster datasets across the prediction settings specified for Table 6. It achieves the strongest reported performance across nearly all evaluated instances.
- Results: Pathformer achieves the best performance in 50 of 56 univariate forecasting cases.It is also second-best in 5 of the remaining instances.
- Results: Pathformer outperforms the second-best PatchTST baseline, especially on the Cloud cluster datasets.The reported results indicate strong predictive performance across both ETT and Cloud cluster datasets.
- Experimental setup: Table 6 evaluates univariate forecasting with input length H = 96 across multiple prediction lengths.Prediction lengths are F ∈{96, 192, 336, 720} for ETT and F ∈{24, 48, 96, 192} for cloud clusters.
A.3 VARYING THE INPUT LENGTH WITH TRANSFORMER MODELS
The experiments test Pathformer across input lengths and broader dataset settings, including long sequences, large datasets, and qualitative predictions. Across these settings, Pathformer consistently outperforms baselines while modeling longer and more complex temporal patterns.
- Varying input length: Pathformer consistently outperforms baselines on ETTh1, ETTh2, Weather, and Electricity across varying input lengths.For H = 48 and H = 192, it achieves the best performance in 46 and 44 of 48 cases, respectively.
- Varying input length: As input length increases, Pathformer’s prediction metrics continue to decrease, indicating effective modeling of longer sequences.The experiments visualize prediction results for input lengths 48 and 192.
- Long-input baselines: With input length H = 336, Pathformer outperforms DLinear, NLinear, and N-HiTS.The comparison uses long input sequences and reports results in Table 9.
- Large datasets: On PEMS07 and Wind Power, Pathformer demonstrates superior predictive performance compared with PatchTST, DLinear, and Scaleformer.Wind Power contains 7397147 timestamps, while PEMS07 includes 883 variables.
- Visualization: For Electricity prediction lengths F = 96, 192, 336, 720, Pathformer’s curves closely align with ground truth and capture multi-period and complex trends.The visualization uses input length H = 96.