PaperlayerView exact revision on arXiv

Source-linked AI summary

TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysis

Shiyu Wang, Jiawei Li, Xiaoming Shi, Zhou Ye, Baichuan Mo, Wenze Lin, Shengtong Ju, Zhixuan Chu, Ming Jin

arXiv:2410.16032v5cs.LGcs.AI

TL;DR

Time series models need to capture patterns across diverse tasks, scales, and periodicities, but existing approaches have limited universal pattern extraction. TIMEMIXER++ addresses this with multi-resolution imaging, decomposition, and mixing, and reports state-of-the-art performance across eight analytical tasks. The paper positions the framework as a general-purpose time series pattern machine for predictive analysis.

  • Problem

    Existing time series approaches have limited ability to capture universal patterns across diverse tasks and multi-scale, multi-periodicity dynamics.

  • Method

    TIMEMIXER++ uses multi-resolution time imaging, time image decomposition, multi-scale mixing, and multi-resolution mixing to extract task-adaptive patterns.

  • Results

    TIMEMIXER++ achieves state-of-the-art performance across 8 analytical tasks, outperforming both general-purpose and task-specific models.

  • Takeaways & Limitations

    TIMEMIXER++ demonstrates a general-purpose pattern-machine framework with strong representational capabilities across forecasting, classification, anomaly detection, and few-shot and zero-shot learning.

  • Takeaways & Limitations

    Current advanced deep time series models remain relatively modest in parameter size because available time series data have limited quality and scale.

Abstract

from arXiv · show

Time series analysis plays a critical role in numerous applications, supporting tasks such as forecasting, classification, anomaly detection, and imputation. In this work, we present the time series pattern machine (TSPM), a model designed to excel in a broad range of time series tasks through powerful representation and pattern extraction capabilities. Traditional time series models often struggle to capture universal patterns, limiting their effectiveness across diverse tasks. To address this, we define multiple scales in the time domain and various resolutions in the frequency domain, employing various mixing strategies to extract intricate, task-adaptive time series patterns. Specifically, we introduce a general-purpose TSPM that processes multi-scale time series using (1) multi-resolution time imaging (MRTI), (2) time image decomposition (TID), (3) multi-scale mixing (MCM), and (4) multi-resolution mixing (MRM) to extract comprehensive temporal patterns. MRTI transforms multi-scale time series into multi-resolution time images, capturing patterns across both temporal and frequency domains. TID leverages dual-axis attention to extract seasonal and trend patterns, while MCM hierarchically aggregates these patterns across scales. MRM adaptively integrates all representations across resolutions. This method achieves state-of-the-art performance across 8 time series analytical tasks, consistently surpassing both general-purpose and task-specific models. Our work marks a promising step toward the next generation of TSPMs, paving the way for further advancements in time series analysis.

1 INTRODUCTION

TIMEMIXER++ is proposed as a general-purpose time series pattern machine for multi-scale, multi-periodicity data. It combines multi-resolution imaging, decomposition, and mixing to capture task-adaptive patterns, achieving state-of-the-art performance across eight analytical tasks.

  • Time series analysis supports forecasting, classification, anomaly detection, and imputation across diverse application domains.
  • Transformer architectures model long-range dependencies through token-wise self-attention, but overlapping temporal contexts make time series distinct from language tasks.
  • Multi-scale and multi-periodicity dynamics challenge models because different temporal scales emphasize different patterns.Daily data can capture hourly fluctuations, whereas yearly data reflect long-term trends and seasonal cycles.
  • TIMEMIXER++ transforms multi-scale time series into multi-resolution time images for pattern extraction across temporal and frequency domains.
  • Time image decomposition disentangles seasonal and trend patterns, while multi-scale and multi-resolution mixing aggregate and integrate representations.TID uses dual-axis attention; MCM aggregates across scales and MRM integrates representations across resolutions.
  • TIMEMIXER++ sets a new state-of-the-art across 8 time series analytical tasks, outperforming both general-purpose and task-specific models.

2 RELATED WORK

Prior time series work spans traditional decomposition, recurrent and convolutional architectures, 2D signal representations, MLPs, and transformers. Related methods differ in how they capture periodic, local, sequential, and long-range patterns.

  • Time Series Analysis: ARIMA and STL model periodic and trend patterns effectively but struggle with non-linear dynamics.
  • Time Series Analysis: Recurrent models capture sequential dependencies but face long-term dependency limitations, while TCNs improve local extraction but remain limited for long-range dependencies.
  • Time Series Analysis: TimesNet treats time series as 2D signals to enhance long-range pattern extraction, whereas MLP-based methods emphasize simplicity and effectiveness.
  • Time Series Analysis: Transformer models such as PatchTST and iTransformer use self-attention to model long-range dependencies and demonstrate good forecasting performance.
  • Hierarchical Time Series Modeling: Hierarchical time series methods commonly decompose temporal patterns into seasonal and trend components before modeling them with attention mechanisms.

3 TIMEMIXER++

TIMEMIXER++ is an encoder-only, multi-scale time series pattern machine that combines frequency-based time imaging, seasonal-trend decomposition, and adaptive mixing. Its architecture projects inputs, processes them through residual MixerBlocks, and ensembles scale-specific prediction heads.

  • Architecture: TIMEMIXER++ uses input projection, L stacked MixerBlocks, and output projection to process multi-scale time series.The MixerBlocks transform inputs into multi-resolution time images, decompose seasonality and trend, and mix patterns across scales and resolutions.
  • Input Projection: Channel mixing applies variate-wise self-attention at the coarsest scale before embedding all scales into deep pattern representations.This design captures cross-variable interactions rather than projecting each channel independently.
  • Output Projection: Scale-specific prediction heads focus on distinct temporal patterns, while ensembling aggregates complementary outputs for task-adaptive prediction.Each head specializes in a scale, and their outputs are combined after the MixerBlocks.
  • Multi-resolution Time Imaging: Multi-resolution time imaging applies FFT-based top-K frequency selection to convert each scale into images indexed by periods and frequencies.The images preserve the original data while representing temporal segments as columns and consistent time points across periods as rows.
  • Time Image Decomposition: Dual-axis attention separates each time image into seasonal and trend representations, using columns for within-period seasonality and rows for trend structure.Two-dimensional convolutions operate on the images to capture long-range temporal patterns.
  • Multi-scale Mixing: Multi-scale mixing combines seasonal patterns from fine to coarse scales and trend patterns from coarse to fine scales using residual convolutions.Seasonal patterns are treated as compositions of shorter patterns, whereas coarser scales emphasize overall trends.
  • Multi-resolution Mixing: Multi-resolution mixing weights representations from K periods by their amplitudes, then aggregates them adaptively at each scale.The resulting seasonal and trend patterns are summed and reshaped back into one-dimensional representations.

4 EXPERIMENTS

TIMEMIXER++ is evaluated across eight time series analytical tasks, including forecasting, imputation, classification, anomaly detection, and transfer settings. It consistently outperforms contemporary baselines across these benchmarks, with reported gains in both standard and data-scarce scenarios.

  • Overall evaluation: The evaluation covers eight tasks: long-term, univariate short-term, multivariate short-term, imputation, classification, anomaly detection, few-shot forecasting, and zero-shot forecasting.Experiments use established benchmarks including ETT, M4, PeMS, electricity, weather, UEA, and anomaly-detection datasets.
  • Long-term forecasting: TIMEMIXER++ outperforms competing models in long-term forecasting, including 7.3% lower MSE and 6.3% lower MAE than iTransformer on Electricity.It also reports 4.9% lower MSE than TimeMixer on ETT (Avg) and improvements on Solar-Energy.
  • Univariate short-term forecasting: TIMEMIXER++ reduces SMAPE by 9.7% and MASE by 15.7% versus iTransformer on univariate short-term forecasting.It achieves up to a 23.3% reduction in MASE versus TiDE and the lowest OWA, improving over TimesNet by 3.5%.
  • Multivariate short-term forecasting: On multivariate short-term forecasting, TIMEMIXER++ improves MAE by 30.8%, MAPE by 32.5%, and RMSE by 24.9% over PatchTST.The PeMS datasets contain 170 to 883 variables, providing a high-dimensional evaluation setting.
  • Imputation: For imputation, TIMEMIXER++ reduces MSE by an average of 25.7% and MAE by 17.4% compared with TimesNet.It consistently outperforms baselines across six datasets and handles input lengths up to 1024.
  • Few-shot and zero-shot forecasting: TIMEMIXER++ achieves 13.2% lower MSE than PatchTST in few-shot learning and 13.1% lower MSE plus 5.9% lower MAE than iTransformer in zero-shot learning.It also reports improvements over TimeMixer in few-shot learning and over PatchTST in zero-shot transfer.
  • Classification and anomaly detection: TIMEMIXER++ reaches 75.9% classification accuracy and an 87.47% anomaly-detection F1-score, exceeding TimesNet by 2.3% and 2.59%, respectively.The results are attributed to its multi-scale and multi-resolution architecture for pattern learning.

5 CONCLUSION

TIMEMIXER++ is presented as a universal time series pattern machine that combines multi-resolution imaging, dual-axis decomposition, and hierarchical mixing. Extensive evaluations report consistent outperformance of existing general-purpose and task-specific models.

  • TIMEMIXER++ constructs time images at multiple resolutions to represent temporal dynamics more effectively.
  • Dual-axis attention decomposes time images into seasonal and trend components within deep representations.
  • Multi-scale and multi-resolution mixing fuse information across hierarchical levels, supporting broad time series predictive analysis.
  • TIMEMIXER++ consistently outperforms existing general-purpose and task-specific time series models in extensive evaluations.

A IMPLEMENTATION DETAILS

The implementation evaluates TIMEMIXER++ across diverse datasets, tasks, metrics, and baselines, using a pipeline that mixes variables, scales, periods, and resolutions. Its core design transforms multi-scale series into time images, separates seasonal and trend patterns, and integrates them hierarchically.

  • Datasets and evaluation: Experiments cover long- and short-term forecasting, imputation, classification, anomaly detection, and UEA datasets, with dataset details organized in tables.
  • Datasets and evaluation: The evaluation uses 27 advanced baselines spanning CNN-, MLP-, RMLP/RLinear-, and Transformer-based architectures.
  • Datasets and evaluation: Long-term forecasting uses MSE and MAE, while short-term forecasting and M4 evaluation use task-specific metrics including MAE, MAPE, RMSE, SMAPE, MASE, and OWA.
  • Model components: Channel mixing applies variate-wise self-attention at the coarsest scale before projecting the multivariate series into an embedding space.
  • Model components: FFT-selected dominant periods generate multi-resolution time images that capture temporal and frequency-domain patterns across scales.
  • Model components: Dual-axis attention extracts seasonality within periods and trends across periods, after which multi-scale and multi-resolution mixing integrate the representations.

C ADDITIONAL ABLATION STUDIES

Ablation studies show that removing individual TIMEMIXER++ components degrades performance across short-term forecasting and anomaly detection tasks. Channel mixing is especially influential on high-dimensional PeMS data, while time image decomposition contributes most to anomaly detection.

  • Short-term forecasting: TIMEMIXER++ achieves the lowest average SMAPE and MAPE scores in short-term forecasting across the ablation configurations.
  • Short-term forecasting: 14.95% improvement is associated with channel mixing on PeMS datasets with large variable dimensions.
  • Anomaly detection: TIMEMIXER++ achieves the highest average F1 score for anomaly detection, with time image decomposition contributing a 9.8% improvement.
  • Anomaly detection: Channel mixing, multi-scale mixing, and multi-resolution mixing also enhance anomaly-detection performance.

D ADDITIONAL REPRESENTATION ANALYSIS

Representation analyses examine TIMEMIXER++ across datasets, scales, resolutions, and tasks. The visualizations show separation of seasonal and trend structure, while CKA analyses indicate task-dependent representation transformations and state-of-the-art performance across four tasks.

  • Representation visualizations: Traffic representations are visualized across three scales and three resolutions, revealing disentangled seasonal and trend patterns.
  • Representation visualizations: ETTm1 representations show stronger high-frequency components at fine scales and more visible low-frequency components at coarse scales.
  • Representation visualizations: Electricity representations capture a time-varying trend that first declines and then rises, including gradient features in the trend image.
  • Representation visualizations: Multi-scale and multi-resolution analysis represents high- and low-frequency, seasonal, and trend components across time series data.
  • Task-wise representation analysis: CKA comparisons between first- and last-layer representations show state-of-the-art performance across four tasks and task-adaptive representation transformations.

E EFFICIENCY ANALYSIS

TIMEMIXER++ is evaluated for forecasting and imputation alongside efficiency, sensitivity, and statistical reliability analyses. It balances memory footprint, training time, and performance while showing generally improved error with more scales, layers, and selected periods.

  • Efficiency comparison: TIMEMIXER++ is compared with eight models across forecasting and imputation using performance, training speed, and memory footprint.The comparison includes iTransformer, PatchTST, TimeMixer, TIDE, Fedformer, TimesNet, MICN, and SCINet.
  • Efficiency comparison: TIMEMIXER++ achieves a comparable balance among memory footprint, training time, and performance on Weather imputation and ETTm1 long-term forecasting.Figure 14 reports the corresponding MSE scores for these tasks.
  • Hyperparameter sensitivity: As the number of scales increases, MSE generally decreases, but gains diminish substantially beyond three scales.The analysis therefore sets the number of scales M from 1 to 3.
  • Hyperparameter sensitivity: Increasing Top-K generally reduces MSE, with a larger impact for longer prediction lengths.Top-K selects periods with the highest amplitudes in the spectrogram.
  • Hyperparameter sensitivity: Adding layers typically reduces MSE, with the largest change between one and two layers and stronger gains for shorter prediction lengths.The sensitivity analysis examines the number of layers L across prediction settings.
  • Statistical reliability: All experiments are repeated three times, with standard deviations and statistical significance tests reported for TIMEMIXER++ and comparison methods.The tests cover ETT, Weather, Solar-Energy, Electricity, Traffic, PEMS, and M4 datasets.

H FULL RESULTS

The appendix provides complete benchmark results for forecasting, classification, and anomaly detection, alongside qualitative prediction visualizations. These visualizations compare ground-truth and predicted series across multiple datasets and settings.

  • Full benchmark results: Full results are provided for long-term forecasting, univariate and multivariate short-term forecasting, classification, and anomaly detection.The corresponding tables cover different datasets, prediction lengths, and evaluation metrics.
  • Qualitative comparisons: Qualitative comparisons visualize test-set forecasting outputs across datasets, and TIMEMIXER++ exhibits superior performance among the compared models.The visualizations include ETTh1, Electricity, Traffic, Weather, Solar-Energy, M4, and PEMS datasets.

J BROADER IMPACT

TIMEMIXER++ is positioned as a general-purpose model for broad time series applications and research. Its evaluation spans diverse tasks and benchmarks, while its time-image and hierarchical-mixing design is presented as a source of future inspiration.

  • Real-world applications: TIMEMIXER++ targets forecasting, classification, anomaly detection, few-shot learning, and zero-shot learning in real-world applications.Suggested application areas include energy, weather, financial markets, supply chains, and anomaly detection.
  • Benchmark scope: The paper reports full long-term forecasting results averaged across prediction lengths 96, 192, 336, and 720.Table 16 compares extensive competitive models under these prediction lengths.
  • Benchmark scope: Short-term forecasting is evaluated on univariate M4 data and multivariate PEMS data using their respective task metrics.M4 reports SMAPE, MASE, and OWA, while PEMS reports MAE, MAPE, and RMSE.
  • Benchmark scope: Anomaly detection and classification results are reported with task-specific evaluation conventions, including precision, recall, F1-score, and classification accuracy.Higher values indicate better performance for the reported metrics.
  • Academic research: The paper presents time-image conversion and hierarchical mixing across scales and resolutions as inspiration for future time series research.This framing is described as part of the method’s academic research impact.
  • Model robustness: The robustness evaluation covers 8 task types, more than 30 benchmarks, and 27 advanced baseline models.The reported results emphasize performance across variability and complexity in time series data.

K LIMITATIONS AND FUTURE WORK

The paper identifies limited data quality and scale as barriers to applying scaling laws to time series pattern machines. It therefore treats TIMEMIXER++ as an initial backbone and points to large-scale datasets as future work.

  • Limitations: Current deep time series models remain relatively modest in parameter size because available time series data are limited in quality and scale.The paper contrasts this constraint with the field’s growing focus on large time series language and foundation models.
  • Future work: Applying scaling laws to time series pattern machines remains an essential open challenge.The authors identify this as a direction for future investigation.
  • Future work: The study presents TIMEMIXER++ as an effective backbone and plans to construct large-scale time series datasets for further exploration.The stated future work is intended to support investigation of scaling laws for TIMEMIXER++.
Loading 2410.16032v5…