Source-linked AI summary
TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting
Md Atik Ahamed, Qiang Cheng
TL;DR
Long-term forecasting requires models that capture dependencies while remaining scalable and computationally efficient. TimeMachine uses multi-scale contexts and four integrated Mamba modules to handle both channel-mixing and channel-independence settings. Experiments report superior accuracy, scalability, and memory efficiency across benchmark datasets.
Problem
Long-term forecasting still requires simultaneous long-term dependency capture, linear scalability, and computational efficiency.
Method
TimeMachine uses an integrated quadruple-Mamba architecture with multi-scale global and local contextual cues for multivariate forecasting.
Results
TimeMachine reports superior prediction accuracy, scalability, and memory efficiency on benchmark datasets.
Takeaways & Limitations
TimeMachine provides one architecture for channel-mixing and channel-independence forecasting while retaining linear scalability and small memory footprints.
Takeaways & Limitations
Performance ranks second on the Weather dataset with small T, and the authors identify potential for improved ground-truth alignment.
Abstract
from arXiv · showhide
Long-term time-series forecasting remains challenging due to the difficulty in capturing long-term dependencies, achieving linear scalability, and maintaining computational efficiency. We introduce TimeMachine, an innovative model that leverages Mamba, a state-space model, to capture long-term dependencies in multivariate time series data while maintaining linear scalability and small memory footprints. TimeMachine exploits the unique properties of time series data to produce salient contextual cues at multi-scales and leverage an innovative integrated quadruple-Mamba architecture to unify the handling of channel-mixing and channel-independence situations, thus enabling effective selection of contents for prediction against global and local contexts at different scales. Experimentally, TimeMachine achieves superior performance in prediction accuracy, scalability, and memory efficiency, as extensively validated using benchmark datasets. Code availability: https://github.com/Atik-Ahamed/TimeMachine
1 Introduction
Long-term forecasting must simultaneously capture dependencies, scale efficiently, and remain computationally practical, but existing approaches often achieve only some of these goals. TimeMachine addresses this gap with multi-scale contexts and an integrated quadruple-Mamba architecture covering both channel-handling regimes.
- Long-term time-series forecasting is important across weather, anomaly detection, and resource-planning applications.
- Existing approaches typically achieve only one or two of long-term dependency capture, linear scalability, and computational efficiency.
- SSMs can capture long-range correlations with linear complexity and context-aware selectivity, but remained underexplored for long-term forecasting.
- Channel mixing captures between-channel dependencies, whereas channel independence treats each channel separately; a unified architecture can adapt to both regimes.
- Multi-scale contexts address redundancy in individual time points and provide contextual information across different temporal resolutions for long-range dependency capture.
- TimeMachine produces global and local contextual cues at two resolutions using four Mamba modules for channel-mixing and channel-independence cases.The architecture uses consecutive resolution reduction or downsampling to create high- and low-resolution contexts.
2 Related Works
Long-term forecasting methods span classical, deep non-Transformer, Transformer-based, and self-supervised approaches. Non-Transformer models emphasize simplicity and scalability, while convolutional and Transformer models model temporal dynamics or dependencies with competitive accuracy.
- Long-term forecasting methods are grouped into non-Transformer supervised, Transformer supervised, and self-supervised representation-learning categories.
- Non-Transformer-based Supervised Approaches: Classical non-Transformer methods include ARIMA, VARMAX, GARCH, and RNN, while deep models use MLPs and CNNs.
- Non-Transformer-based Supervised Approaches: MLP-based models such as DLinear, TiDE, and RLinear use simple linear structures to achieve favorable complexity and scalability.
- Non-Transformer-based Supervised Approaches: CNN-based models extract temporal features and complex dynamics, often matching or occasionally exceeding Transformer-based models in performance.
3 Proposed Method
TimeMachine normalizes multivariate inputs, builds two-stage embeddings, and processes them with four integrated Mambas. The architecture supports both channel-independent and channel-mixing processing while using selective state-space dynamics and multi-scale representations.
- Input normalization and embedding: TimeMachine normalizes each multivariate input sequence before constructing two-stage embedded representations with E1 and E2.The model supports RevIN and Z-score normalization; RevIN is empirically more helpful than Z-score in the described experiments.
- Channel mixing and independence: Channel mixing is favored when the channel count is comparable to the look-back length, while channel independence is useful for datasets with fewer channels.The channel-mixing configuration uses all four Mambas to learn channel correlations and global context at different scales.
- Channel mixing and independence: TimeMachine handles channel independence by reshaping BML to (B × M)1L, whereas channel mixing keeps channels together throughout the network.The two modes allow independent processing or exploitation of inter-channel correlations without additional manual preprocessing.
- Residuals and projections: The model uses residual additions around the projection stages, and its MLP embeddings are not heavily relied upon for performance.The stated residual connections add outputs element-wise, while the ablation reports stable or improved results across tested MLP sizes.
- Integrated Quadruple Mambas: The architecture uses four Mambas in pairs at two embedding levels to capture long-term dependencies and provide local contexts.Outer Mambas process the BMn1 representation, while inner Mambas process BMn2.
- Selective state-space dynamics: Mamba adapts its state-space dynamics to each input by making B, C, and ∆ input-dependent, enabling selective propagation or forgetting of sequence information.The SSM maps inputs through latent states using discrete updates hk = ¯A hk−1 + ¯B uk and outputs vk = C hk.
4 Result Analysis
TimeMachine is evaluated across seven benchmark datasets against 11 strong baselines using accuracy, varying look-back windows, memory footprint, scalability, and qualitative forecasts. It generally achieves strong forecasting performance while maintaining small memory use and scalable behavior, although Weather performance with small prediction lengths and ground-truth alignment remain improvement areas.
- Benchmark forecasting accuracy: TimeMachine demonstrates superior or comparable performance to strong baselines across benchmark datasets, including large-channel Traffic and Electricity.Experiments use MSE and MAE for prediction lengths T = {96, 192, 336, 720} with L = 96.
- Look-back sensitivity: Longer look-back windows generally improve TimeMachine’s performance while preserving consistent behavior across datasets.The evaluated look-back windows are L = {192, 336, 720}.
- Quantitative comparison: Figure 2 reports normalized percentage gains over three SOTA methods, indicating improvement over the strong baselines using average MSE.The figure compares TimeMachine and SOTA baselines with L = 96; closer to the circle boundary indicates better performance.
- Memory and scalability: TimeMachine uses very small memory footprints and scales with a relatively linear increase in learnable parameters as look-back length grows.On Traffic, its memory use is very similar to DLinear; E1 depends on L while the remaining networks are relatively independent of L.
- Qualitative comparison: Qualitative forecasts follow the actual future trend, with closer ground-truth alignment than iTransformer for part of the Traffic example.The cited Traffic interval is approximately 75-90, and the visualization shows 100 predicted time points.
- Limitations: TimeMachine ranks second on Weather with small T, and its predicted trajectories still have potential for improved ground-truth alignment.These observations identify specific areas for future improvement despite strong overall results.
5 Hyperparameter Sensitivity Analysis and Ablation Study
The ablations examine MLP dimensions, dropout rates, residual connections, convolution width, state and dimension expansion factors, and model levels while holding other parameters fixed. The selected defaults generally balance forecasting performance, memory, and training efficiency.
- Effect of MLPs’ Parameters (n1, n2): MLP dimensions have limited influence on performance, with smaller MLPs improving ETTh1 at T = 720 and ETTh2 remaining mostly stable.The study evaluates 10 combinations of n1 and n2 from {512, 256, 128, 64, 32}.
- Sensitivity of Dropouts: Dropout rates that are too low or too high are unhelpful, so the model uses 0.7 after both MLP stages.This setting is chosen for balance on the ETTh1 and ETTh2 datasets.
- Ablation of Residual Connections: Residual connections improve both datasets and are retained in all reported results.The connections stabilize training and reduce overfitting, especially for smaller datasets with channel independence.
- Mamba Parameter Sensitivity: A local convolutional kernel width of 2 performs more promisingly than width 4 and becomes the default for all datasets and Mambas.
- State Expansion Factor of Mambas: Higher state expansion factors can improve performance for varying prediction lengths, motivating the default N = 256.The ablation varies N from 8 to 256 while keeping other parameters fixed.
- Mamba Dimension Expansion Factor: Increasing the Mamba dimension expansion factor does not consistently improve performance and increases memory and training-time costs, so E = 1 is used.
- Model Levels: Two model levels are selected because they balance performance and memory footprint.
6 Strengths and Limitations
TimeMachine outperforms numerous baselines across benchmark datasets while providing memory efficiency and stable performance across look-back and prediction lengths. Its remaining weakness is lower ranking on Weather with small T and imperfect alignment with ground truth.
- TimeMachine outperforms numerous baselines, including transformer-based methods, across benchmark datasets while maintaining memory efficiency and stable performance across varying look-back and prediction lengths.
- TimeMachine ranks second on the Weather dataset with small T, and Figure 3 indicates potential to improve alignment with ground truth.
7 Conclusion
The paper introduces TimeMachine for long-term multivariate time-series forecasting with long-range dependency modeling, linear scalability, and small memory footprints. Its quadruple-Mamba architecture uses multi-scale global and local contextual cues across channel-mixing and channel-independence settings, with experiments reporting superior accuracy, scalability, and memory efficiency.
- TimeMachine captures long-term dependencies in multivariate time series while maintaining linear scalability and small memory footprints.
- An integrated quadruple-Mamba architecture provides multi-scale global and local contextual cues while unifying channel-mixing and channel-independence settings.
- Extensive experiments report superior accuracy, scalability, and memory efficiency compared with state-of-the-art methods.Future work will explore TimeMachine in a self-supervised learning setting.
Supplementary Materials
The supplementary materials provide ablation and qualitative analyses covering residual connections, dropout, forecasting results, expanding factors, model levels, and parameter scalability. These materials examine design choices across specified datasets, sequence lengths, prediction horizons, and visualization settings.
- Residual Connections: Residual-connection ablations use L = 96 and T = {96, 192, 336, 720}.
- Dropout Sensitivity: Dropout sensitivity compares MSE across diverse dropout rates with L = 96.
- Forecasting Results: Long-term forecasting results vary input lengths L = {192, 336, 720} and prediction horizons T = {96, 192, 336, 720}.
- Qualitative Comparison: Qualitative comparisons use the test set with L = 96, T = 720, one randomly selected channel, and a 100-point window.
- Expanding Factor: The expanding-factor analysis compares model behavior across expansion settings.
- Model Levels: Model-level ablations use L = 96 and T = {96, 192, 336, 720}.
- Scalability: Parameter scalability is evaluated against the look-back window.