PaperlayerView exact revision on arXiv

Source-linked AI summary

SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion

Lu Han, Xu-Yang Chen, Han-Jia Ye, De-Chuan Zhan

arXiv:2404.14197v3cs.LG

TL;DR

Multivariate forecasting must balance the robustness of channel independence with the value of channel correlations, while existing correlation mechanisms can be complex or unreliable under distribution drift. SOFTS uses a centralized STAR module to aggregate channels into a global core and fuse it with individual series. It achieves state-of-the-art performance with linear complexity, including strong results across six datasets, while its core representation and aggregation strategy remain areas for further study.

  • Problem

    Channel-independent forecasting improves robustness to distribution drift but overlooks channel correlations, while correlation-based methods can be complex or struggle under distribution drift.

  • Method

    SOFTS is an MLP-based forecaster whose STAR module centrally aggregates channel representations into a global core and fuses that core with each series.

  • Results

    SOFTS achieves state-of-the-art performance with linear complexity and delivers the best or second-best average predictive outcomes across all six evaluated datasets.

  • Takeaways & Limitations

    Centralized core aggregation offers a scalable way to capture channel correlations while retaining the robustness associated with channel-independent forecasting.

  • Takeaways & Limitations

    STAR’s effectiveness depends heavily on the quality of its global core representation, whose robustness across diverse datasets remains challenging.

Abstract

from arXiv · show

Multivariate time series forecasting plays a crucial role in various fields such as finance, traffic management, energy, and healthcare. Recent studies have highlighted the advantages of channel independence to resist distribution drift but neglect channel correlations, limiting further enhancements. Several methods utilize mechanisms like attention or mixer to address this by capturing channel correlations, but they either introduce excessive complexity or rely too heavily on the correlation to achieve satisfactory results under distribution drifts, particularly with a large number of channels. Addressing this gap, this paper presents an efficient MLP-based model, the Series-cOre Fused Time Series forecaster (SOFTS), which incorporates a novel STar Aggregate-Redistribute (STAR) module. Unlike traditional approaches that manage channel interactions through distributed structures, \textit{e.g.}, attention, STAR employs a centralized strategy to improve efficiency and reduce reliance on the quality of each channel. It aggregates all series to form a global core representation, which is then dispatched and fused with individual series representations to facilitate channel interactions effectively.SOFTS achieves superior performance over existing state-of-the-art methods with only linear complexity. The broad applicability of the STAR module across different forecasting models is also demonstrated empirically. For further research and development, we have made our code publicly available at https://github.com/Secilia-Cxy/SOFTS.

1 Introduction

Multivariate forecasting methods face a tension between robust channel independence and useful cross-channel information. SOFTS addresses this with a centralized STAR module that captures channel interactions efficiently while retaining lower complexity.

  • Channel-independent methods improve robustness to distribution drift but overlook channel correlations, limiting further performance improvements.
  • Attention-based approaches capture channel correlations but can incur high complexity or fail to achieve state-of-the-art performance.
  • SOFTS is an MLP-based forecaster that uses STAR to aggregate information across channels into a global core representation.
  • STAR fuses the global core with each local series, enabling indirect channel interaction through a centralized rather than distributed structure.
  • SOFTS achieves state-of-the-art performance with lower complexity, while STAR’s applicability is also validated across attention-based forecasters.

2 Related Work

Prior work reduces forecasting complexity through efficient temporal or attention mechanisms, but channel mixing can remain vulnerable to distribution drift. SOFTS instead targets the performance–efficiency trade-off with linear complexity in both channels and lookback length.

  • Time series forecasting: RNNs and CNNs model temporal variation using recurrent dependencies and temporal convolutions, respectively.
  • Efficient long-term multivariate forecasting and channel independence: Transformer-based forecasters reduce attention costs through probabilistic subsampling, autocorrelation, or frequency-domain processing.
  • Efficient long-term multivariate forecasting and channel independence: Channel-mixing models can be less robust to distribution drift, motivating channel-independent approaches that simplify architectures and improve robustness.
  • Efficient long-term multivariate forecasting and channel independence: Channel independence can ignore interactions among variates, limiting further advances in multivariate forecasting.
  • Efficient long-term multivariate forecasting and channel independence: SOFTS addresses the performance–efficiency trade-off with state-of-the-art performance and linear complexity in channels and lookback-window length.

3 SOFTS

SOFTS is an MLP-based multivariate forecaster that embeds each series, exchanges channel information through the centralized STAR module, and predicts future values with linear complexity in channels, lookback length, and horizon.

  • 3 SOFTS: SOFTS forecasts multivariate series by mapping historical channel values X ∈ R^C×L to future values Y ∈ R^C×H.L is the lookback-window length, C the number of channels, and H the forecast horizon.
  • 3 SOFTS: Reversible instance normalization centers and scales input series before forecasting, then restores their original statistics to the predictions.For PEMS, normalization is selectively applied according to performance.
  • 3 SOFTS: Series embedding applies a linear projection to each channel’s lookback window without introducing the extra dimension produced by patch embedding.The resulting representation is S0 ∈ R^C×d, where d is the hidden dimension.
  • 3 SOFTS: STAR aggregates channel representations into a global core, redistributes that core to every channel, and fuses the resulting representations through MLP layers with a residual connection.The core encodes global information across channels; stochastic pooling aggregates the series, while Repeat_Concat appends the core to each series representation.
  • 3 SOFTS: SOFTS has encoding complexity O(CLd + Cd^2 + CdH), which is linear in lookback length L, channel count C, and forecast horizon H when d is treated as constant.The predictor then maps the final STAR representation to the forecast horizon.

4 Experiments

SOFTS is evaluated on six real-world dataset groups against Linear-, MLP-, Transformer-, and CNN-based forecasters using standardized long-term forecasting settings. It achieves strong accuracy while maintaining linear channel-scaling efficiency, and STAR improves performance across pooling methods, models, and lookback lengths.

  • Experimental setup: The experiments use six widely used real-world dataset groups, fixed lookback L = 96, dataset-specific horizons, and MSE and MAE evaluation.
  • Main results: SOFTS achieves the best or second-best average predictive outcome on all 6 datasets.
  • Main results: 13.9% relative average MSE reduction on PEMS07 improves the comparison from 0.101 to 0.087.
  • Model efficiency: SOFTS maintains complexity scaling linearly with channel count while PatchTST and iTransformer require substantially more memory as channels increase.
  • Pooling ablation: STAR consistently improves performance across pooling methods, while stochastic pooling outperforms mean and max pooling in nearly all datasets.
  • Universality of STAR: Replacing attention with STAR in PatchTST, iTransformer, and modified Crossformer uses fewer computational resources while maintaining or improving performance on several datasets.
  • Lookback sensitivity: SOFTS performs consistently better across lookback lengths, especially for shorter windows, while improving as longer windows provide more data.
  • Embedding adaptation: After STAR adjustment, adapted Traffic embeddings improve forecasting from 0.414 MSE to 0.376, a 9% improvement.

5 Conclusion

The paper addresses the tension between robust channel independence and useful channel correlation in multivariate forecasting. SOFTS and STAR target this tension with low-complexity correlation modeling and are presented as a scalable direction for resource-constrained forecasting.

  • Channel independence improves robustness, but ignoring channel correlation can limit further forecasting improvements.
  • SOFTS uses the STAR module to capture channel correlation efficiently while achieving state-of-the-art performance with low complexity.
  • The paper explores scalable forecasting that maintains equal or better performance than state-of-the-art methods under resource constraints.

A Datasets Description

The evaluation uses six real-world multivariate time-series dataset groups spanning electricity, traffic, weather, and solar-energy applications. They differ in channel counts, sampling intervals, and collection periods.

  • Electricity and transformer data: ETT contains four transformer datasets with seven oil and load features, sampled hourly or every 15 minutes from July 2016 to July 2018.
  • Traffic data: Traffic contains hourly road-occupancy rates from San Francisco freeway sensors recorded during 2015–2016.
  • Electricity and transformer data: Electricity records hourly electricity consumption from 321 clients during 2012–2014.
  • Weather and solar data: Weather contains 21 German weather indicators sampled every 10 minutes during 2020.
  • Weather and solar data: Solar-Energy records 10-minute solar-power production from 137 photovoltaic plants in 2006.
  • Traffic data: PEMS contains California public traffic-network data collected in five-minute windows.
  • Dataset metadata: Dataset descriptions include channel counts, train-validation-test splits, prediction lengths, and sampling granularity.

B.1 Overall architecture of SOFTS

SOFTS embeds each series, repeatedly aggregates the series into a core representation, fuses that core back with each series, and projects the resulting embeddings into forecasts. Its core uses stochastic pooling, while the model is evaluated with MSE and MAE.

  • Architecture: SOFTS first embeds each channel series and applies encoder layers that aggregate, fuse, and update series representations.Each encoder derives a core from the series embeddings, concatenates it with each series, and applies another MLP.
  • Core representation: The STAR core is obtained by applying an MLP to series representations and stochastic pooling across the channels.The core representation has dimension d′ and is formed by aggregating the representations of C series.
  • Forecasting output: After the encoder stack, a linear layer projects the final series embeddings into predicted series with H future time points.The prediction output has shape C×H.
  • Core representation: Stochastic pooling computes dimension-wise probabilities from normalized softmax activations and samples a channel during training.The selected activation forms the pooled result for that dimension.
  • Evaluation: The model is optimized with MSE and compared using MSE and MAE on six real-world datasets against ten prior state-of-the-art models.The benchmark description reports strong performance across these tests.

C.2 Full Results of Pooling Method Ablation

The pooling ablation evaluates STAR against channel-independent MLPs and compares alternative pooling choices. Incorporating STAR consistently improves performance, while stochastic pooling performs better than mean and max pooling.

  • STAR ablation: STAR consistently improves performance across the evaluated pooling methods compared with an MLP using channel independence without STAR.The w/o STAR condition removes STAR while retaining an MLP with the Channel Independent strategy.
  • Evaluation setting: The benchmark table reports multivariate forecasting results across multiple prediction lengths with a fixed lookback window of L = 96.PEMS uses H ∈{12, 24, 48, 96}, while other datasets use H ∈{96, 192, 336, 720}.
  • Pooling methods: Stochastic pooling performs better than mean and max pooling in the pooling-method comparison.Mean pooling averages series representations, whereas max pooling selects the maximum hidden feature across channels.

C.4 More Results of Lookback Ablation

The extended lookback ablation evaluates SOFTS across window lengths from 48 to 720. SOFTS is almost consistently better than other models, with a warning about potential overfitting at the largest windows.

  • Lookback sensitivity: SOFTS performs almost consistently better than other models across lookback window lengths from 48 to 720.The evaluated lengths are L ∈{48, 96, 192, 336, 512, 720}.
  • Lookback sensitivity: Potential overfitting is reported when the lookback length is very large, specifically L = 512 or L = 720.This caveat accompanies the extended lookback comparison.

C.5 Full Results of Hyperparameter Sensitivity Experiments

Hyperparameter sensitivity experiments vary the series dimension d, core dimension d′, and encoder-layer count N. Complex traffic datasets require larger series dimensions and more layers, whereas changes in d′ have minimal overall influence.

  • Series dimension and depth: Complex traffic datasets such as Traffic and PEMS require larger hidden dimensions d and more encoding layers N.The sensitivity study attributes these requirements to the intricacy of the traffic datasets.
  • Core dimension: Variations in the core hidden dimension d′ have minimal influence on overall model performance.The finding is reported in the corresponding sensitivity experiment.
  • STAR universality: STAR can replace attention in several transformer-based forecasters while maintaining or improving performance with fewer computational resources.The universality experiment covers PatchTST, iTransformer, and a modified Crossformer.

D Error Bar

The paper reports robustness experiments across datasets and forecasting horizons, alongside visualizations of the learned core and predictions. Results are averaged over five random seeds.

  • Five-seed averages are used to evaluate SOFTS robustness across datasets and forecasting horizons.The corresponding results are reported in Table 9.
  • The learned core is visualized by decoding final-layer series embeddings with a two-layer MLP autoencoder.The red line represents the core alongside the original input channels.
  • Prediction showcases compare SOFTS with iTransformer and PatchTST on ECL, ETTh2, Traffic, and PEMS03 using a 96-step lookback and horizon.The listed datasets correspond to Figures 12–15.

E.3 More Results on Adaptation of Series Embedding

Additional experiments examine STAR’s adaptation of series embeddings and its handling of abnormal channels. The visualizations use high-channel-count datasets and show abnormal-channel behavior in prediction examples.

  • STAR adaptation is visualized on ECL, PEMS03, and Traffic, which contain 321, 358, and 862 channels, respectively.These datasets were selected because their channel counts are large enough to show relationships in embedding space.
  • In PEMS03, SOFTS captures the slowly increasing trend after channel 160 sharply decreases and then enters a smooth period.The example illustrates the model’s handling of an abnormal channel.

F Limitations and Future Works

The paper identifies limitations involving the quality of STAR’s global core representation and the limited exploration of alternative aggregation and redistribution strategies. It also documents checklist-level disclosures about scope, societal impacts, reproducibility, and limitations.

  • Limitations and Future Works: SOFTS performance may degrade when the global core fails to capture essential features of individual series.The authors identify robustness and accuracy of the core across diverse datasets as an open challenge.
  • Limitations and Future Works: The paper explores alternative aggregate-redistribute strategies only to a limited extent.Future work could investigate other aggregation and redistribution methods for improved performance and robustness.
  • Additional Results: The paper includes visualizations of prediction results, abnormal-channel behavior, and series-embedding adaptation in supplementary experiments.These materials include prediction figures for PEMS03 and ECL and t-SNE visualizations of channel adaptation.
  • Societal Impacts: The societal-impact discussion mentions privacy, bias, over-reliance on automated forecasts, and the need for data protection, bias monitoring, ethical policies, and human oversight.These concerns are framed as potential risks of deployment in fields including healthcare and finance.
  • Checklist Disclosures: The paper states that its contributions and scope are reflected in the abstract and introduction and that limitations are discussed in Appendix F.The checklist responses mark both disclosures as present.
  • Theory Assumptions and Proofs: The checklist notes that the paper has no theoretical results and therefore receives an NA assessment for theory assumptions and proofs.The justification explicitly states that theoretical results are not included.
  • Reproducibility: The paper reports open datasets, anonymous review code, reproduction scripts, and reproducibility information in Appendix B.The code was stated to become public after acceptance.
  • Checklist Guidance: The checklist guidance emphasizes stating assumptions and reflecting on claim scope, empirical coverage, and robustness to assumption violations.These considerations are presented as expectations for limitations disclosure.
Loading 2404.14197v3…