PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Mamba Effective for Time Series Forecasting?

Zihan Wang, Fanheng Kong, Shi Feng, Ming Wang, Xiaocui Yang, Han Zhao, Daling Wang, Yifei Zhang

arXiv:2403.11144v3cs.LG

TL;DR

The paper asks whether Mamba can provide effective TSF while avoiding the Transformer’s quadratic computational burden. It proposes S-Mamba, which combines bidirectional Mamba for inter-variate correlations with an FFN for temporal dependencies, and reports leading performance with low computational overhead across thirteen public datasets. The broader experiments examine Mamba’s potential in TSF.

  • Problem

    Transformer-based TSF models can capture relevant patterns but incur quadratic complexity, creating computational and deployment costs.

  • Method

    S-Mamba tokenizes variates, uses bidirectional Mamba for inter-variate correlations, and uses an FFN for temporal dependencies.

  • Results

    Across thirteen public datasets, S-Mamba achieves leading TSF performance while requiring low computational overhead.

  • Takeaways & Limitations

    The experiments support further investigation of Mamba as a TSF architecture with strong performance and low resource requirements.

  • Takeaways & Limitations

    Transformer and MLP comparisons retain scope boundaries: Transformers face long-sequence and quadratic-cost issues, while MLPs can struggle with volatile, non-stationary patterns and global dependencies.

Abstract

from arXiv · show

In the realm of time series forecasting (TSF), it is imperative for models to adeptly discern and distill hidden patterns within historical time series data to forecast future states. Transformer-based models exhibit formidable efficacy in TSF, primarily attributed to their advantage in apprehending these patterns. However, the quadratic complexity of the Transformer leads to low computational efficiency and high costs, which somewhat hinders the deployment of the TSF model in real-world scenarios. Recently, Mamba, a selective state space model, has gained traction due to its ability to process dependencies in sequences while maintaining near-linear complexity. For TSF tasks, these characteristics enable Mamba to comprehend hidden patterns as the Transformer and reduce computational overhead compared to the Transformer. Therefore, we propose a Mamba-based model named Simple-Mamba (S-Mamba) for TSF. Specifically, we tokenize the time points of each variate autonomously via a linear layer. A bidirectional Mamba layer is utilized to extract inter-variate correlations and a Feed-Forward Network is set to learn temporal dependencies. Finally, the generation of forecast outcomes through a linear mapping layer. Experiments on thirteen public datasets prove that S-Mamba maintains low computational overhead and achieves leading performance. Furthermore, we conduct extensive experiments to explore Mamba's potential in TSF tasks. Our code is available at https://github.com/wzhwzhwzh0921/S-D-Mamba.

1. Introduction

Time series forecasting requires models to capture temporal dependencies and inter-variate correlations, but Transformer quadratic complexity raises computational costs. The paper introduces S-Mamba, using Mamba and an FFN to pursue effective, lower-overhead forecasting.

  • TSF models use historical information to forecast future states by learning temporal dependencies and inter-variate correlations.
  • Transformer-based TSF models face quadratic computational complexity, increasing overhead with more variates or longer lookback sequences.
  • Mamba offers near-linear sequence processing and motivates further investigation of its performance and efficiency for TSF.
  • S-Mamba tokenizes each variate, uses bidirectional Mamba for inter-variate correlations, and applies an FFN for temporal dependencies.
  • Experiments compare S-Mamba with representative and state-of-the-art TSF models, reporting superior forecast performance with fewer computational resources.
  • The paper also conducts extensive experiments to characterize Mamba’s potential in TSF tasks.

2. Related Work

Related work covers Transformer and linear TSF models alongside Mamba applications in sequence, vision, language, and multimodal tasks. These studies motivate testing whether Mamba can combine efficiency with strong TSF performance.

  • TSF research includes Transformer-based and linear-model architectures.
  • Transformer-based Models: Transformer models perform well but face challenges with sequential patterns, long inputs, position encoding, computational cost, and GPU memory use.
  • Linear Models: MLP-based models are simpler and more efficient, but can struggle with volatile or non-stationary patterns and global dependencies.
  • Mamba Applications: Mamba has attracted research across NLP, computer vision, and other artificial intelligence domains.
  • Mamba Applications: Prior studies apply Mamba to mixture-of-experts systems, clinical note generation, speech separation, biomedical segmentation, and image fusion.
  • Mamba Applications: Mamba has also supported efficient long-term video modeling and multimodal inference with performance comparable to existing systems.
  • Mamba Applications: Across these applications, Mamba is reported to reduce parameter size and inference cost while maintaining similar or better performance.

3. Preliminaries

The preliminaries define TSF inputs and outputs, introduce continuous and discretized state space models, and describe Mamba’s selective processing. They also outline the Mamba block and S-Mamba forecasting procedure.

  • Time Series Forecasting: TSF maps a historical sequence of length L across V variates to a future sequence of length T.
  • State Space Models: State Space Models map input sequences to output sequences through latent states representing internal system evolution.
  • State Space Models: Continuous SSMs are discretized using step size Δ, enabling an efficient linear recursive calculation.
  • Mamba: Mamba adds data-dependent selection to S4 and uses hardware-aware parallel algorithms to capture contextual information in long sequences efficiently.
  • Mamba Block: The Mamba block expands hidden representations, applies convolution and SiLU activation, generates state representations, and combines them with a residual pathway.
  • Forecasting Procedure: S-Mamba receives a historical batch with shape (B, L, V) and outputs forecasts with shape (B, T, V).
  • Forecasting Procedure: The forecasting algorithm includes linear tokenization, Mamba-layer processing, bidirectional-information fusion, residual connection, FFN encoding, and projection.
  • Mamba Block: The Mamba block outputs a representation Y in R^B×V×D after its processing stages.

4. Methodology

S-Mamba uses four sequential layers to tokenize multivariate time series, encode inter-variate correlations and temporal dependencies, and project the result into forecasts. Its bidirectional Mamba and FFN components separately target cross-variate and temporal information.

  • 4. Methodology: S-Mamba comprises Linear Tokenization, Mamba VC Encoding, FFN TD Encoding, and Projection layers.The first layer tokenizes the series, the middle layers learn inter-variate and temporal information, and the final layer maps processed representations to forecasts.
  • 4.1. Linear Tokenization Layer: A linear layer tokenizes the input time series and produces the representation used by subsequent S-Mamba layers.The Linear Tokenization Layer receives U_in and outputs U.
  • 4.2. Mamba VC Encoding Layer: A bidirectional Mamba block encodes inter-variate correlations by capturing mutual information among variates.The bidirectional outputs are aggregated and connected through a residual network.
  • 4.3. FFN TD Encoding Layer: The FFN TD Encoding Layer applies normalization and a feed-forward network to learn sequential temporal relationships for future-series representations.A final normalization layer adjusts the resulting future representations.
  • 4.4. Projection Layer: A mapping layer reconstructs the tokenized temporal information into the predicted time series and transposes it into the final output.The Projection Layer is responsible for mapping processed information to the model forecast.

5. Experiments

The experiments evaluate S-Mamba on thirteen public real-world datasets spanning traffic, electricity-transformer temperature, and other domains, against nine representative or state-of-the-art forecasting models. The baselines include Transformer-based, linear-based, and temporal-convolutional approaches.

  • 5.1. Datasets and Baselines: Experiments use thirteen real-world datasets grouped into traffic-related, ETT, and other dataset types.The groups differ in variate count, periodicity, regularity, and sampling characteristics.
  • 5.1. Datasets and Baselines: Traffic-related datasets include Traffic and four PEMS subsets, and are characterized by many mostly periodic variates.Traffic contains hourly road-occupancy measurements from 862 sensors; PEMS contains public California traffic-network subsets.
  • 5.1. Datasets and Baselines: The ETT group contains ETTm1, ETTm2, ETTh1, and ETTh2, with few variates and weak regularity.ETT records load and oil temperature from electricity transformers.
  • 5.1. Datasets and Baselines: Other datasets are Electricity, Exchange, Weather, and Solar-Energy, covering electricity consumption, exchange rates, meteorological indicators, and photovoltaic power.These datasets include hourly, daily, and ten-minute sampling settings across multiple domains.
  • 5.1. Datasets and Baselines: S-Mamba is compared with nine representative or state-of-the-art models spanning Transformer-based, linear-based, and temporal-convolutional methods.The baselines include iTransformer, PatchTST, Crossformer, FEDformer, Autoformer, RLinear, TiDE, DLinear, and TimesNet.
  • 5.1. Datasets and Baselines: The baseline set includes models using attention, decomposition, frequency enhancement, linear structures, MLPs, and two-dimensional temporal kernels.These design choices are represented by the individual descriptions of the comparison models.

5.2. Overall Performance

S-Mamba performs well across traffic-related, Electricity, and Solar-Energy datasets, while its advantage is less pronounced on ETT and Exchange. Forecast visualizations compare S-Mamba with iTransformer under matched input and forecast lengths.

  • S-Mamba attains commendable outcomes on traffic-related, Electricity, and Solar-Energy datasets.These datasets contain numerous variates, most of which are periodic.
  • S-Mamba does not demonstrate pronounced superiority on ETT and Exchange, where it can exhibit suboptimal performance.These datasets contain few, predominantly aperiodic variates with weak inter-variate correlations.
  • Figure 4 compares S-Mamba and iTransformer forecasts on five datasets with input length 96 and forecast length 96.Blue denotes ground truth and red denotes forecasts.
  • The two models exhibit similar performance on Exchange and ETTh1.The passage attributes this to the datasets’ few variates and the absence of an evident fusion gap.

5.3. Model Efficiency

S-Mamba combines favorable forecast accuracy with a low computational resource footprint in comparisons across multiple datasets. Ablation experiments indicate complementary roles for Mamba in inter-variate encoding and FFN-based temporal encoding.

  • S-Mamba attains the most favorable MSE on PEMS07, Electricity, and Traffic.The comparison evaluates MSE, training duration, and allocated GPU memory.
  • Compared with Transformer-based models, S-Mamba typically requires shorter training time and less allocated GPU memory.The experiments use PEMS07, Electricity, Traffic, and ETTm1 with independent runs on one NVIDIA RTX3090 GPU and batch size 16.
  • RLinear uses minimal GPU memory and reduces training time but does not provide competitive forecast precision.
  • Figure 5 compares S-Mamba and six baselines on MSE, training time, and GPU memory.The lookback length is 96; the forecast length is 12 for PEMS07 and 96 for other datasets.
  • 5.4. Ablation Study: Ablation findings show that Mamba performs better for VC encoding, whereas FFN remains dominant for TD encoding.The experiments cover Electricity, Traffic, Weather, Solar Energy, and ETTh2.

5.5. Can Variate Order Affect the Performance of S-Mamba?

The paper examines whether variate ordering affects S-Mamba because its Mamba VC Encoding Layer processes variates as an implicitly ordered sequence. On Electricity, performance remains largely unaffected by perturbing variate order.

  • Mamba may prioritize closer variates at initialization, creating a bias against more distant variates.This motivates investigating whether variate order affects S-Mamba performance.
  • The study categorizes Electricity variates as periodic or aperiodic using Fourier transforms before altering their order.Periodic variates are assumed more likely to contain learnable patterns, while aperiodic variates may contain unreliable information.
  • S-Mamba’s performance remains largely unaffected by perturbing variate order on Electricity.The experiment uses Electricity because it contains many periodic and aperiodic variates.
  • Figure 7 reports allocated GPU memory, training time, and MSE for Transformer and Mamba variants across four datasets.Purple solid and dotted lines represent average Transformer-model and Mamba-model performance, respectively.
  • Figure 8 evaluates forecasting performance across lookback lengths 96, 192, 336, and 720.The forecast length is 12 for PEMS04 and 96 for the other datasets.
  • 5.6. Can Mamba Outperform Advanced Transformers?: Replacing advanced Transformer encoders with Mamba reduces GPU usage and training time while slightly improving overall performance.The comparison covers Autoformer, Flashformer, and Flowformer variants on Electricity, Traffic, PEMS07, and ETTm1.

5.7. Can Mamba Help Benefit from Increasing Lookback Length?

The study examines whether increasing lookback length benefits Mamba-based and Transformer-based forecasters. Across experiments with varying lookback lengths, encoding temporal sequence information beforehand did not resolve the Transformer-related issue.

  • Observed pattern: Transformer-based models do not consistently improve when the lookback sequence becomes longer.The paper attributes this possibility to self-attention overlooking or even inverting sequential order.
  • Experimental setup: The experiments compare Reformer, Informer, Transformer, S-Mamba, and iTransformer as lookback length increases.Tests use Electricity, Traffic, and PEMS04 among four datasets.
  • Observed pattern: Encoding temporal sequence information into the model beforehand does not resolve the issue.

6. Conclusion

The conclusion presents S-Mamba as a low-overhead Mamba-based forecaster with leading performance across broad TSF benchmarks. It attributes the advantage primarily to bidirectional Mamba for inter-variate correlation encoding and reports robust Mamba capabilities in TSF.

  • Conclusion: S-Mamba requires low computational overhead and achieves leading performance across thirteen public datasets.The comparison includes nine representative and state-of-the-art models across traffic, weather, electricity, and energy forecasting tasks.
  • Conclusion: S-Mamba transfers inter-variate correlation encoding to a bidirectional Mamba block and uses an FFN to extract temporal dependencies.
  • Conclusion: The bidirectional Mamba block is primarily credited with improving inter-variate-correlation understanding at lower overhead than the Transformer.
  • Conclusion: Extensive experiments indicate that Mamba possesses robust capabilities in time series forecasting tasks.The supplied passage continues into a comparison of Mamba’s stability with the Transformer.

7. Future Work

The paper identifies future opportunities around Mamba’s scalability and transferability in time series forecasting. It highlights processing more variates under limited resources and developing pretrained Mamba models.

  • Future Work: Mamba is positioned as suitable for extracting increasingly valuable global inter-variate correlations as the number of variates grows.The paper links this setting to Mamba’s ability to detect long-range dependencies while controlling computational-demand escalation.
  • Future Work: Compared with Transformer, Mamba may process information from more variates simultaneously in resource-limited real-world scenarios.
  • Future Work: Mamba matches the Transformer in generalization and stability according to the experimental results.
  • Future Work: Developing a Mamba-based pretraining model for time series forecasting is suggested as a fruitful direction.
Loading 2403.11144v3…