PaperlayerView exact revision on arXiv

Source-linked AI summary

Frequency-domain MLPs are More Effective Learners in Time Series Forecasting

Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Defu Lian, Ning An, Longbing Cao, Zhendong Niu

arXiv:2311.06184v1cs.LGcs.AI

TL;DR

MLP-based forecasters are efficient but limited by point-wise mappings and information bottlenecks for global dependency learning. The paper introduces FreTS, which converts signals to complex frequency representations and applies redesigned MLPs across channel and temporal dimensions; experiments report consistent superiority across short- and long-term benchmarks.

  • Problem

    MLP-based forecasting methods use point-wise mappings and face information bottlenecks that limit global dependency modeling.

  • Method

    FreTS converts time-domain signals into complex frequency components and separately learns their real and imaginary parts with frequency-domain MLPs across channel and temporal dimensions.

  • Results

    FreTS outperforms baselines across seven short-term and six long-term benchmarks, including more than 20% average MAE and RMSE reductions versus Transformer-based models.

  • Takeaways & Limitations

    Frequency-domain MLPs offer global view and energy compaction, supporting simple, efficient forecasting with clearer learned patterns.

Abstract

from arXiv · show

Time series forecasting has played the key role in different industrial, including finance, traffic, energy, and healthcare domains. While existing literatures have designed many sophisticated architectures based on RNNs, GNNs, or Transformers, another kind of approaches based on multi-layer perceptrons (MLPs) are proposed with simple structure, low complexity, and {superior performance}. However, most MLP-based forecasting methods suffer from the point-wise mappings and information bottleneck, which largely hinders the forecasting performance. To overcome this problem, we explore a novel direction of applying MLPs in the frequency domain for time series forecasting. We investigate the learned patterns of frequency-domain MLPs and discover their two inherent characteristic benefiting forecasting, (i) global view: frequency spectrum makes MLPs own a complete view for signals and learn global dependencies more easily, and (ii) energy compaction: frequency-domain MLPs concentrate on smaller key part of frequency components with compact signal energy. Then, we propose FreTS, a simple yet effective architecture built upon Frequency-domain MLPs for Time Series forecasting. FreTS mainly involves two stages, (i) Domain Conversion, that transforms time-domain signals into complex numbers of frequency domain; (ii) Frequency Learning, that performs our redesigned MLPs for the learning of real and imaginary part of frequency components. The above stages operated on both inter-series and intra-series scales further contribute to channel-wise and time-wise dependency learning. Extensive experiments on 13 real-world benchmarks (including 7 benchmarks for short-term forecasting and 6 benchmarks for long-term forecasting) demonstrate our consistent superiority over state-of-the-art methods.

1 Introduction

Existing MLP forecasters offer simple, low-complexity alternatives but rely on point-wise mappings that limit global dependency modeling. The paper therefore explores frequency-domain MLPs, whose learned patterns exhibit global views and energy compaction, and introduces FreTS to learn frequency representations.

  • Motivation: Deep forecasting architectures can impose expensive computation and parameter-related robustness concerns, motivating simpler MLP-based alternatives.The passage contrasts sophisticated deep models with MLP methods that offer simple structure, low complexity, and strong forecasting performance.
  • Limitations of Existing MLPs: MLP-based forecasting methods rely on point-wise mappings that cannot adequately handle global time-series dependencies.
  • Frequency-Domain MLPs: Frequency-domain MLPs provide a more complete signal view for learning global spatial and temporal dependencies.The frequency spectrum exposes global periodic patterns more clearly than time-domain MLPs.
  • Frequency-Domain MLPs: Frequency-domain MLPs concentrate learning on a smaller set of key frequency components, preserving clearer patterns while filtering noise influence.Their learned weights show clearer diagonal dependencies than the more scattered time-domain weights.
  • FreTS: FreTS converts time-domain signals into complex frequency components and separately learns their real and imaginary mappings with redesigned MLPs.The architecture applies these stages to support channel-wise and time-wise dependency learning.

2 Related Work

Related work spans statistical, deep neural, frequency-domain, and MLP-based approaches to time-series forecasting. These methods differ in how they model temporal, series-wise, and frequency information.

  • Time-Domain Forecasting: Traditional forecasting includes exponential smoothing, ARMA, VAR, and ARIMA methods operating on time-domain series.
  • Time-Domain Forecasting: RNN-, CNN-, and GNN-based models learn nonlinear temporal or series-wise dependencies directly in the time domain.Examples include DeepAR, LSTNet, TCN, SCINet, AGCRN, MTGNN, and GraphWaveNet.
  • Frequency-Domain Forecasting: Frequency-domain forecasting methods use Fourier transforms for hidden-state decomposition, graph convolution, autocorrelation, or frequency enhancement.The cited approaches include SFM, StemGNN, Autoformer, and FEDformer.
  • MLP-Based Forecasting: MLP-based forecasting models use architectures such as stacked residual MLPs, Fourier-assisted periodicity modeling, and lightweight sampling-oriented MLPs.The cited examples are N-BEATS, DEPTS, and LightTS.

3 FreTS

FreTS learns multivariate forecasts by converting signals into frequency representations and applying complex-valued frequency-domain MLPs along channel and temporal dimensions. Its design uses separate channel and temporal learners, then projects the learned representation to future timestamps.

  • Problem Definition: FreTS takes a length-L multivariate lookback window and predicts a horizon of τ future timestamps.The formulation represents inputs as X_t ∈ R^N×L and forecasts as Ŷ_t = f_θ(X_t).
  • Frequency Learning Architecture: A learnable dimension-extension vector maps the input lookback window into a more expressive hidden representation before frequency learning.The representation is formed as H_t = X_t × ϕ_d.
  • Domain Conversion/Inversion: Fourier domain conversion decomposes signals into complex frequency components whose real and imaginary parts encode cosine- and sine-based contributions.FreTS performs conversion and inversion along both channel and time dimensions.
  • Frequency Channel Learner: The Frequency Channel Learner models inter-series dependencies by applying frequency-domain MLPs along channels with shared weights across timestamps.Its output is converted back to the time domain before temporal processing.
  • Frequency Temporal Learner: The Frequency Temporal Learner models temporal dependencies by applying frequency-domain MLPs along time with shared weights across channels.Its frequency-domain output is inverted and assembled across channels.
  • Frequency-Domain MLPs: FreMLP applies complex-valued weights and biases while separately computing real and imaginary mappings before recombining them.This redesign targets global view and energy compaction in frequency components.
  • Energy Compaction: Equal time- and frequency-domain signal energy implies that concentrated frequency components can represent the signal while discarding others with limited energy impact.The paper connects this property to clearer pattern preservation in frequency-domain learning.
  • Frequency-Domain MLPs: Frequency-domain MLP operations correspond to global convolutions in the time domain.

4 Experiments

Experiments evaluate FreTS across short- and long-term forecasting benchmarks, component ablations, and efficiency settings. FreTS consistently outperforms baselines while retaining favorable parameter and training-time efficiency.

  • Main Results: FreTS is evaluated on thirteen real-world benchmarks spanning short-term and long-term forecasting, against representative and state-of-the-art baselines.The experiments use six short-term and six long-term benchmarks in the reported main tables, with additional benchmark coverage described in the evaluation setup.
  • Main Results: 9.4% lower MAE and 11.6% lower RMSE on average distinguish FreTS from all baselines across the short-term datasets.Table 1 uses input and prediction lengths of 12 and reports improvements across six datasets.
  • Main Results: More than 20% average reductions in both MAE and RMSE versus the best Transformer-based models occur across the long-term benchmarks.FreTS also generally outperforms LTSF-Linear and PatchTST under the reported long-term settings.
  • Model Analysis: Frequency channel learning is more important for short-term forecasting, whereas frequency temporal learning is more effective for long-term forecasting.The comparison removes one learner at a time through the FreCL and FreTL variants.
  • Model Analysis: 6.4% MAE and 11.4% RMSE improvements on Exchange follow from replacing DLinear's original MLP with FreMLP.On Weather, the corresponding improvements are 4.9% in MAE and 3.5% in RMSE; similar gains are reported for NLinear.
  • Efficiency Analysis: O(N log N + L log L) complexity accompanies parameter and training-time savings as the number of variables or prediction length increases.Compared with AGCRN, FreTS reduces parameters by 30% and training time by 20% on Wiki; on Exchange, it is at least 3 times smaller than Transformer baselines and trains faster.
  • Model Analysis: Both real and imaginary FreMLP weight parts exhibit energy aggregation in the Traffic visualization.The visualization uses a lookback window of 48 and prediction length of 192.

5 Conclusion Remarks

The paper applies redesigned frequency-domain MLPs to time series forecasting and validates the approach through FreTS. Experiments across short- and long-term benchmarks support its forecasting performance and efficiency potential.

  • Conclusion Remarks: FreTS redesigns MLPs in the frequency domain to capture time-series patterns through global view and energy compaction.The paper presents FreTS as a simple architecture built on these frequency-domain MLPs.
  • Conclusion Remarks: Experiments on seven short-term and six long-term benchmarks validate the superiority of the proposed methods.The conclusion also highlights simple MLPs' potential for high-efficiency forecasting.

B.1 Datasets

The experiments use thirteen real-world benchmarks for short- and long-term forecasting, comparing FreTS with eighteen representative state-of-the-art baselines.

  • Thirteen real-world benchmarks evaluate forecasting accuracy across short-term and long-term settings.
  • The benchmarks vary substantially in dimensionality, including approximately 145k Wikipedia series, 963 San Francisco freeway lanes, and 370 electricity clients.
  • The datasets cover meteorological, solar-power, Wikipedia-view, traffic, ECG, electricity-consumption, COVID-19, and foreign-exchange time series.
  • Eighteen representative baselines span LSTM-based, GNN-based, and Transformer-based models, alongside statistical and convolutional approaches.
  • Baseline implementations generally follow recommended configurations, with dataset-specific adjustments for graph structures and static features where needed.

B.5 Ablation Experimental Settings

The ablation settings replace DLinear’s component-wise linear layers with frequency-domain MLP variants while retaining the recommended LTSF-Linear experimental setup.

  • DLinear decomposes inputs into trend and seasonal components, applying one linear layer to each component.
  • The ablation replaces DLinear’s two linear layers with two different frequency-domain MLPs, forming DLinear (FreMLP).
  • The DLinear (FreMLP) comparison uses the same experimental settings recommended in LTSF-Linear.
  • Complex multiplication combines real and imaginary components as (ac − bd) + j(ad + bc).

D.1 Proof of Theorem 1

Theorem 1 establishes energy preservation between a time series and its frequency-domain representation, using conjugation, inverse transformation, and Parseval-style equality.

  • Theorem 1 states that a time series’ energy in the time domain equals the energy of its frequency-domain spectrum.
  • The proof represents the raw series and its frequency components, then integrates over either the channel or temporal dimension.
  • The derivation uses the conjugate of the frequency representation and the inverse discrete Fourier transform.

D.2 Proof of Theorem 2

Theorem 2 shows that frequency-domain MLP operations correspond to global convolutions in the time domain, connecting spectral multiplication with convolution.

  • Theorem 2 represents frequency-domain MLP operations on the transformed series as global convolutions in the time domain.
  • The correspondence follows from multiplying the frequency representation by complex weights and biases before transforming back.
  • The proof applies the convolution theorem separately along the channel or temporal dimension.
  • In this equivalence, circular convolution in time corresponds to multiplication in the frequency domain.

E.1 Ablation Study

Ablations show that FreTS’s learner contributions depend on forecasting horizon, while both real and imaginary frequency components matter and hyperparameters exhibit non-monotonic sensitivity.

  • Frequency Channel and Temporal Learners: Longer prediction lengths favor the frequency temporal learner, whereas the channel learner can worsen performance at lengths 336 and 720.The channel learner is more prone to overfitting as prediction horizons increase.
  • Impacts of Real/Imaginary Parts: Both real and imaginary input components are indispensable, while the real part contributes more than the imaginary part to model performance.The real part of the learned weight W also has a more significant role than its imaginary part.
  • Parameter Sensitivity: Performance first improves and then declines as input length increases on ECG, reflecting added pattern information followed by redundancy or overfitting.The tested input lengths were {6, 12, 18, 24, 30, 36, 42, 50, 60}.
  • Parameter Sensitivity: Performance first increases and then decreases as embedding size grows, because larger embeddings improve fitting ability but can induce overfitting.The embedding sizes tested were {32, 64, 128, 256, 512}.

F Additional Results

Additional experiments report strong performance across multi-step forecasting and long-term Exchange settings, with FreTS outperforming the compared baselines throughout the stated configurations.

  • F.1 Multi-Step Forecasting: FreTS outperforms other baselines at every forecasting step on METR-LA and COVID-19 multi-step experiments.The experiments use input length 12 and prediction lengths {3, 6, 9, 12}.
  • F.1 Multi-Step Forecasting: FreTS captures channel-wise and time-wise dependencies effectively in the multi-step forecasting experiments.The comparison includes models with pre-defined graph structures.
  • F.2 Long-Term Forecasting under Varying Lookback Window: FreTS outperforms all listed baselines on the Exchange dataset across all lookback-window and prediction-length settings.Lookback lengths are L ∈ {96, 192, 336}; prediction lengths are {96, 192, 336, 720}.
  • F.2 Long-Term Forecasting under Varying Lookback Window: FreTS achieves significant improvements over FEDformer, Autoformer, Informer, and Reformer in the reported Exchange long-term forecasting settings.PatchTST and LTSF-linear are also included among the baselines.

G.1 Weight Visualizations for Energy Compaction

Weight visualizations show energy aggregation in both real and imaginary frequency-domain weights and connect frequency learning with global periodic-pattern fitting.

  • Energy Compaction: Real and imaginary weight coefficients exhibit energy aggregation through clear diagonal patterns across Traffic and Electricity settings.The visualizations vary lookback-window sizes and prediction lengths, with Wr and Wi denoting the real and imaginary parts.
  • Global View: The visualized weights compare time-domain inputs on the left with real-part frequency-spectrum inputs on the right.These comparisons are presented for Traffic and Electricity datasets under the stated lookback and prediction settings.
  • Global Periodic Patterns: Forecast visualizations on Traffic and Electricity show that FreTS can fit cyclic patterns.The figures present forecast-versus-actual visualizations with lookback-window and prediction-length settings denoted by I/O.
  • Global Periodic Patterns: The reported visualizations associate frequency-domain global-view characteristics with FreTS’s capability to capture global periodic patterns.The conclusion is stated for the Traffic and Electricity visualization experiments.
Loading 2311.06184v1…