PaperlayerView exact revision on arXiv

Source-linked AI summary

SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameters

Shengsheng Lin, Weiwei Lin, Wentai Wu, Haojun Chen, Junjie Yang

arXiv:2405.00946v2cs.LG

TL;DR

Long-term time-series forecasting must model complex dependencies over extended horizons despite substantial computational demands. SparseTSF uses Cross-Period Sparse Forecasting to separate periodicity from trend through downsampling and cross-period trend prediction. It achieves competitive or superior performance with fewer than 1k parameters, while showing generalization capabilities for small-sample and low-quality-data scenarios; its effectiveness is limited for ultra-long periods.

  • Problem

    Long-term forecasting requires modeling extensive temporal dependencies, but complex models can require millions or tens of millions of parameters, limiting use under restricted computational resources.

  • Method

    SparseTSF applies Cross-Period Sparse Forecasting, downsampling sequences into subsequences to decouple periodicity from trend and predict cross-period trends with a lightweight model.

  • Results

    SparseTSF achieves competitive or superior predictive accuracy with fewer than 1k trainable parameters and demonstrates robust generalization capabilities.

  • Takeaways & Limitations

    SparseTSF is positioned for deployment in computation resource-constrained environments and for transfer to small-sample and low-quality-data scenarios.

  • Takeaways & Limitations

    SparseTSF does not perform optimally for ultra-long periods exceeding 100 because the Sparse technique creates overly sparse parameter connections.

Abstract

from arXiv · show

This paper introduces SparseTSF, a novel, extremely lightweight model for Long-term Time Series Forecasting (LTSF), designed to address the challenges of modeling complex temporal dependencies over extended horizons with minimal computational resources. At the heart of SparseTSF lies the Cross-Period Sparse Forecasting technique, which simplifies the forecasting task by decoupling the periodicity and trend in time series data. This technique involves downsampling the original sequences to focus on cross-period trend prediction, effectively extracting periodic features while minimizing the model's complexity and parameter count. Based on this technique, the SparseTSF model uses fewer than *1k* parameters to achieve competitive or superior performance compared to state-of-the-art models. Furthermore, SparseTSF showcases remarkable generalization capabilities, making it well-suited for scenarios with limited computational resources, small samples, or low-quality data. The code is publicly available at this repository: https://github.com/lss-1138/SparseTSF.

1. Introduction

Long-term forecasting requires modeling complex dependencies over extended horizons, often with resource-intensive models. SparseTSF addresses this by separating periodicity from trend through cross-period sparse forecasting, using fewer than 1k parameters while retaining competitive accuracy and generalization.

  • Motivation: Longer forecasting horizons increase uncertainty and require models to capture more extensive temporal dependencies, raising computational complexity.Transformer-based models may use millions or tens of millions of parameters, limiting deployment under restricted computational resources.
  • Motivation: Periodic patterns and trends can be decomposed by resampling sequences with stable cycles into subsequences, reframing periodicity as inter-subsequence dynamics.This perspective supports lightweight forecasting-model design.
  • Method: Cross-Period Sparse Forecasting downsamples periodic sequences and predicts each subsequence, simplifying forecasting into cross-period trend prediction.The technique decouples periodicity and trend while focusing prediction on trend changes.
  • Efficiency: Fewer than 1k trainable parameters make SparseTSF 1∼4 orders of magnitude smaller than its counterparts.The model is designed to reduce forecasting models’ computational resource demand.
  • Results: SparseTSF achieves competitive or superior predictive accuracy relative to state-of-the-art models and demonstrates robust generalization capabilities.The introduction presents this as a central model outcome.

2. Related Work

LTSF research has developed Transformer, CNN, and MLP approaches to model long-term dependencies. Lightweight work has progressively reduced model size while maintaining predictive performance, motivating further efficiency improvements.

  • Development of Long-term Time Series Forecasting: LTSF models evolved from Transformer adaptations such as Informer, Autoformer, and FEDformer to newer architectures including PatchTST and PETformer.CNN and MLP approaches, including SCINet, TimesNet, MICN, TiDE, and HDMixer, also became mainstream.
  • Progress in Lightweight Forecasting Models: Lightweight forecasting research includes LightTS, TiDE, TSMixer, and HDformer after DLinear showed simple models could extract strong temporal periodic dependencies.These studies pushed LTSF toward smaller designs.
  • Progress in Lightweight Forecasting Models: FITS was the first reported lightweight LTSF model to reach the 10k parameter level while maintaining excellent predictive performance.The passage presents FITS as a milestone in lightweight LTSF development.

3. Methodology

SparseTSF reformulates long-term forecasting around periodicity-aware downsampling, shared sparse prediction, and trend-focused modeling. Its lightweight architecture uses a parameter-efficient linear layer, with normalization and aggregation addressing distribution shifts, information loss, and outliers.

  • SparseTSF addresses long-term forecasting by modeling individual channels and using a shared function to reduce inter-channel complexity.
  • Cross-Period Sparse Forecasting: Cross-Period Sparse Forecasting downsamples sequences into periodic subsequences, predicts each with shared parameters, and upsamples them into the complete forecast.The process is equivalent to applying a linear layer of size n×m to a w×n representation.
  • Cross-Period Sparse Forecasting: The basic sparse approach can lose information and amplify outliers because it uses only one point per period for prediction.
  • SparseTSF: Sliding aggregation incorporates neighboring points and mitigates outlier effects before sparse prediction, using a 1D convolution with zero-padding.
  • SparseTSF: SparseTSF subtracts each input sequence mean before modeling and adds it back afterward to address distributional shifts between training and testing.
  • Parameter Efficiency: L × H parameter growth is reduced to a much lighter scale when the look-back window and forecast horizon are large relative to periodicity.The paper illustrates this setting with L and H up to 720 and periodicity w such as 24.
  • Cross-Period Sparse Forecasting: The sparse formulation uses periodic components as references while predicting future trend components, so periodic features need not be explicitly modeled.This reformulation focuses the forecasting task on trend variations after separating periodic structure.

4. Experiments

Experiments evaluate SparseTSF on mainstream long-term forecasting benchmarks, efficiency metrics, ablations, representation learning, hyperparameter sensitivity, and cross-domain generalization. The results show competitive forecasting with very small computational cost, robust generalization, and sensitivity to the assumed main period.

  • 4.2. Main Results: SparseTSF ranks within the top two across all evaluated scenarios while using a significantly smaller parameter scale.Its five-run standard deviation is within 0.001 in most cases.
  • 4.3. Efficiency Advantages of SparseTSF: SparseTSF is over ten times smaller than the next-best model in parameters and MACs, supporting deployment on devices with very limited computational resources.DLinear has the lowest runtime overhead because it uses a shorter lookback, while SparseTSF incurs resampling overhead.
  • 4.4. Ablation Studies and Analysis: Incorporating the Sparse technique improves average performance by 4.7% for Linear, 21.4% for Transformer, and 12.4% for GRU models.The ablation evaluates the technique across Linear, Transformer, and GRU base models.
  • 4.4. Ablation Studies and Analysis: SparseTSF learns more distinct evenly spaced weight stripes than Linear, indicating stronger extraction of periodic features.The equivalent L × H weight matrix is obtained by feeding H one-hot vectors into SparseTSF and transposing the outputs.
  • 4.4. Ablation Studies and Analysis: SparseTSF generalizes better than other models from ETTh2 to ETTh1 and from Electricity to ETTh1, with performance nearly matching direct SparseTSF modeling.The authors connect this result to stable periodic-feature extraction and note its relevance for small-sample and low-quality data.

5. Discussion

SparseTSF is designed for data with a stable main period, but its sparse decomposition has boundaries for ultra-long or multiple intertwined periods. The paper also distinguishes its preprocessing and parameter-sharing design from N-HiTS and OneShotSTL.

  • Limitations: SparseTSF may be less effective for ultra-long periods because sparse parameter connections become overly sparse.The paper identifies periods exceeding 100 as an example and reports that performance is not optimal in such scenarios.
  • Limitations: SparseTSF may struggle with multiple periods because its sparse technique downsampled and decomposed only one main period.The paper reports that some multi-period cases, including daily patterns superimposed with weekly patterns, can nevertheless remain strong.
  • Future work: For ultra-long periods, the paper concludes that a denser connected model is a better choice, while some multi-period scenarios remain favorable for SparseTSF.These conclusions are based on the paper’s additional investigations in Appendix C.
  • Compared with N-HiTS: SparseTSF downsamples and upsamples before and after prediction, whereas N-HiTS performs these operations within stacked modules.SparseTSF’s downsampling is equivalent to reshaping and transposing, while N-HiTS uses max-pooling and interpolation.
  • Compared with OneShotSTL: SparseTSF is a neural network with parameter-sharing linear layers, whereas OneShotSTL is a non-neural online forecasting method.OneShotSTL separately computes trend and seasonal subseries, while SparseTSF resamples the sequence into w subseries.

6. Conclusion

The paper concludes that Cross-Period Sparse Forecasting and SparseTSF provide lightweight long-term forecasting with periodic-feature extraction and competitive predictive performance. The workflow uses a historical window to produce forecasts and supports multivariate data through shared parameters.

  • Conclusion: SparseTSF combines the Cross-Period Sparse Forecasting technique with an extremely lightweight forecasting model.The technique is presented as the paper’s central contribution.
  • Conclusion: SparseTSF achieves competitive or superior performance to current state-of-the-art models with a minimal parameter scale.The conclusion also reports potent generalization capabilities for small-sample and low-quality-data scenarios.
  • Conclusion: SparseTSF is presented as suitable for deployment in computation-resource-constrained environments.The conclusion frames lightweighting and periodic-feature extraction as central properties of the model.
  • Workflow: The workflow takes a univariate historical look-back window as input and outputs the corresponding forecast sequence.Shared parameters across channels enable effective multivariate time-series forecasting under the CI strategy.
  • Workflow: Figure 5 provides a schematic illustration of SparseTSF.The supplied passage identifies the figure but does not describe its visual elements further.

A.2. Experimental Details

The experiments use standard PyTorch training procedures, period-aware hyperparameter settings, and dataset splits aligned with prior lightweight forecasting studies.

  • Training: SparseTSF is trained in PyTorch with Adam for 30 epochs, learning-rate decay, and early stopping.The learning rate decays by 0.8 after the initial three epochs, with patience of five for early stopping.
  • Data splitting: ETT datasets are split into training, validation, and test proportions of 6:2:2.Other dataset splits follow the procedures used by FITS and Autoformer.
  • Hyperparameters: The period w is set to an inherent data cycle or reduced for datasets with extremely long cycles.Examples include w = 24 for ETTh1 and w = 4 for ETTm1.
  • Hyperparameters: Batch size is selected according to dataset size, including the number of channels.The supplied passage gives thresholds for datasets with fewer than 100 or 300 channels.
  • Comparison protocol: SparseTSF and FITS use an input length of 720 for fair comparison, while other baselines retain their official input lengths.This setting controls the comparison protocol across baseline models.

B. Theoretical Proofs

The theoretical analysis formalizes SparseTSF as forecasting across periodically downsampled subsequences. Under a periodic-plus-trend decomposition, the sparse task predicts future trends while using a constant periodic component as reference.

  • Parameter analysis: SparseTSF contains a 1D convolutional layer for sliding aggregation and a linear layer for sparse sliding prediction.The parameter proof derives the convolutional count from kernel size and the linear-layer count from its input and output sizes.
  • Sparse forecasting formulation: Downsampling divides a length-L historical series into w subsequences, each containing every w-th data point.Each subsequence has length n determined by collecting one point every w time steps.
  • Sparse forecasting formulation: A shared forecasting function predicts each downsampled future subsequence from its corresponding past subsequence.The downsampled forecast horizon is denoted m, while H is the original forecast horizon.
  • Sparse forecasting formulation: The original forecasting problem is reformulated as a series of smaller forecasting tasks on downsampled subsequences.The predicted subsequences are subsequently combined into the full forecast sequence.
  • Theoretical interpretation: Assuming X(t) = P(t)+T(t), periodic downsampling makes the periodic component constant within each downsampled sequence.The resulting task predicts future trend components while using the constant periodic component as a reference.

C.1. Multi-Period Scenarios

SparseTSF handles multiple periodicities by decoupling the primary daily cycle, allowing its linear layer to focus on remaining cross-period patterns. On Traffic, this produces stronger daily and weekly periodic patterns than the original approach.

  • Traffic data contains distinct daily and weekly cycles, represented by rush-hour patterns and weekday–weekend differences.
  • Daily-period downsampling preserves a clear weekly cycle with w′ = 7 in the Traffic data.
  • SparseTSF decouples the daily cycle before predicting across downsampled subsequences, simplifying the remaining forecasting task.
  • SparseTSF captures stronger daily and weekly periodic patterns than the original approach, shown by more pronounced equidistant stripes.

C.2. Ultra-Long Period Scenarios

For datasets with ultra-long periods, SparseTSF’s performance depends on the sparsity interval w. Large w can make connections excessively sparse, while w = 4 retains strong performance and keeps the model lightweight.

  • The ultra-long-period evaluation covers ETTm1&ETTm27 and Weather8, whose primary periods extend up to 96 and 144, respectively.
  • Large w can deteriorate performance by creating excessively sparse connections that limit the information available for prediction.
  • Increasing w can also improve performance, suggesting denser sparse connections may suit datasets with longer periods.
  • w = 1, equivalent to a fully connected linear layer, performs worse than sparse connection-based predictions.
  • With w = 4, SparseTSF ranks within the top 3 in most cases while remaining significantly lighter than other mainstream models.

D.1. Comparison Results after Fixing the Code Bug

After correcting a test-metric code bug, SparseTSF still achieves impressive performance with minimal overhead. Its results depend on historical-window length, especially when the window cannot cover important periodicities.

  • A codebase bug discarded incomplete test batches, making batch size affect reported test metrics and potentially producing incorrect results.
  • After fixing the bug, SparseTSF retains impressive performance with minimal overhead and remains tens of times smaller than FITS under the stated comparison.
  • Longer look-back windows generally improve SparseTSF, indicating its ability to model long-term dependencies.
  • Traffic and Electricity vary more across look-back lengths than ETTh1 and ETTh2, with accuracy notably decreasing at a look-back of 96.
  • Traffic’s daily period w = 24 and weekly period w = 168 explain why a look-back of 96 cannot cover the full weekly pattern.
  • Instance Normalization is necessary for the smaller ETTh1 and ETTh2 datasets but has limited effect on the larger Traffic and Electricity datasets.

D.4. Comparison Results with N-HiTS and OneShotSTL

SparseTSF is compared with N-HiTS and OneShotSTL using the evaluation settings and reported results from their respective sources. It outperforms these methods in most cases.

  • The comparison uses multivariate MSE for SparseTSF versus N-HiTS and univariate MAE for SparseTSF versus OneShotSTL.
  • SparseTSF outperforms N-HiTS and OneShotSTL in most cases.
Loading 2405.00946v2…