PaperlayerView exact revision on arXiv

Source-linked AI summary

FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting

Tian Zhou, Ziqing Ma, Xue wang, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin

arXiv:2205.08897v4cs.LGstat.ML

TL;DR

Long-term forecasting requires preserving useful history without overfitting its noise. FiLM combines Legendre projections, Fourier-based noise reduction, low-rank approximation, and multiscale modeling, achieving state-of-the-art results across benchmark datasets while serving as a general forecasting component.

  • Problem

    Long-term forecasting faces a trade-off between preserving critical historical information and reducing noisy signals that can derail predictions.

  • Method

    FiLM combines a redesigned Legendre Projection Unit, Frequency Enhanced Layers using Fourier analysis and low-rank approximation, and multiscale mixture-of-experts modeling.

  • Results

    20.3%: FiLM reduces overall relative MSE versus FEDformer in multivariate forecasting across six benchmark datasets and achieves the best performance at all horizons.

  • Takeaways & Limitations

    FiLM’s representation module is designed as a general plug-in that can improve long-term prediction performance when combined with other deep learning modules.

  • Takeaways & Limitations

    FiLM is less fast to train than some MLP-based models such as N-HiTS, although it has shorter per-step training time and smaller memory usage.

Abstract

from arXiv · show

Recent studies have shown that deep learning models such as RNNs and Transformers have brought significant performance gains for long-term forecasting of time series because they effectively utilize historical information. We found, however, that there is still great room for improvement in how to preserve historical information in neural networks while avoiding overfitting to noise presented in the history. Addressing this allows better utilization of the capabilities of deep learning models. To this end, we design a \textbf{F}requency \textbf{i}mproved \textbf{L}egendre \textbf{M}emory model, or {\bf FiLM}: it applies Legendre Polynomials projections to approximate historical information, uses Fourier projection to remove noise, and adds a low-rank approximation to speed up computation. Our empirical studies show that the proposed FiLM significantly improves the accuracy of state-of-the-art models in multivariate and univariate long-term forecasting by (\textbf{20.3\%}, \textbf{22.6\%}), respectively. We also demonstrate that the representation module developed in this work can be used as a general plug-in to improve the long-term prediction performance of other deep learning modules. Code is available at https://github.com/tianzhou2011/FiLM/

1 Introduction

Long-term forecasting must preserve critical historical information while reducing noise, because long horizons involve dependencies, complex patterns, and error propagation. FiLM addresses this trade-off with Legendre-based representation, Fourier filtering, low-rank reduction, and multiscale modeling.

  • Motivation: Long-term forecasting is challenging because predictions must handle long-term dependencies, error propagation, complex patterns, and nonlinear dynamics.These challenges occur across applications including energy, weather, economics, and transportation.
  • Motivation: Deep models can miscapture noise while preserving history, producing forecasts that diverge from the ground-truth distribution.Figure 1 compares vanilla Transformer and LSTM predictions with real-world time-series ground truth.
  • Approach: FiLM uses a mixture of experts for robust multiscale time-series feature extraction.This architecture is presented as one of the work’s key contributions.
  • Approach: FiLM redesigns the Legendre Projection Unit as a general data-representation tool for preserving historical information in forecasting models.The representation can be combined with different deep learning modules.
  • Approach: FiLM introduces Frequency Enhanced Layers that combine Fourier analysis and low-rank matrix approximation to reduce noisy signals, dimensionality, and overfitting.The method is intended to retain long-term dependencies while improving computational efficiency.
  • Results: 19.2% and 26.1%: FiLM improves state-of-the-art methods in multivariate and univariate forecasting, respectively, across six benchmark datasets.The experiments span energy, traffic, economics, weather, and disease domains, and also report improved computational efficiency through dimensionality reduction.

2 Time Series Representation in Legendre-Fourier Domain

FiLM represents long histories with fixed-size Legendre-polynomial memories, then uses Fourier-domain processing to select informative frequencies and reduce noise. The design also supports multiscale history processing and low-rank computation.

  • Legendre Projection: Legendre projection compresses a signal history into a fixed-size representation using polynomial basis functions of degree at most N −1.The compressed memory dynamically represents the evolving history over a window of size θ.
  • Legendre Projection: Theorem 1 bounds approximation error for Lipschitz and smoothly differentiable signals, with accuracy improving as the number of Legendre basis functions increases.The bounds are O(θL/N) for Lipschitz functions and O(θ^kN^(−k+1/2)) for functions with k-th order bounded derivatives.
  • Frequency Enhancement: Increasing the Legendre basis can overfit noisy history, motivating Fourier-based feature selection before prediction.The paper reports that directly feeding these features into MLP, RNN, or vanilla Attention does not achieve state-of-the-art performance because of noisy historical signals.
  • Multiscale Representation: The representation is processed across multiple history lengths {T, 2T, ... nT} to capture information at different temporal scales.The overall architecture uses this multiscale structure alongside Legendre projection and Fourier processing.
  • Frequency Enhancement: Fourier mode selection retains a lower-dimensional frequency subspace; when amin is sufficiently small, the selected space is almost the original one.The default policy keeps the lowest M modes, while random high-frequency additions can improve some datasets.

3 Model Structure

FiLM maps input sequences to forecasts through Legendre Projection and Fourier Enhanced layers, with optional normalization and multiscale mixture-of-experts processing. Low-rank factorization reduces the learnable parameter burden, while RevIN provides an optional robustness aid with a training-speed trade-off.

  • Overall Architecture: FiLM maps X ∈ R^(T×D) to Y ∈ R^(T×D) using one LPU layer and one FEL layer.An optional RevIN block is added for data normalization and robustness.
  • Legendre Projection Unit: The LPU state-space layer updates memory with fixed matrices A and B, then projects signals into memory and reconstructs them through LPU_R.Its memory dimension N corresponds to the number of Legendre polynomials.
  • Frequency Enhanced Layer: Low-rank factorization replaces FEL weight matrix W with W1 · W2 · W3 using N′ << N.With N = 256 and N′ = 4, the learnable weight is reduced to 0.4% with minor accuracy deterioration.
  • Frequency Enhanced Layer: FEL selects a subset of Fourier modes to reduce noise and increase training speed, defaulting to the lowest M modes.Random high-frequency modes can provide additional improvement on some datasets.
  • Mixture of Multiscale Experts: A multiscale mixture of experts processes input histories with lengths {T, 2T, ... nT}, merges expert predictions, and improves performance consistently across datasets.Each expert forecasts horizon T from a different history scale.
  • Data Normalization: RevIN normalizes each instance before forecasting and reverses the normalization on the model output.It is optional because it slows training by 2-5 times and does not improve all datasets consistently.

4 Experiments

FiLM is evaluated across six datasets and multiple forecasting settings, showing strong multivariate and univariate results. Ablations examine LPU, FEL, multiscale processing, normalization, compression, and computational efficiency.

  • Main Results: FiLM achieves the best multivariate performance on all six benchmark datasets across all evaluated horizons.The experiments use prediction lengths of 96, 192, 336, and 720, with different lengths for ILI.
  • Main Results: 20.3% overall relative MSE reduction is achieved against FEDformer in multivariate forecasting.The improvement exceeds 30% on the Exchange dataset.
  • Main Results: 22.6% overall relative MSE reduction is achieved against FEDformer in univariate forecasting.Improvements exceed 40% on datasets including Weather and Electricity.
  • LPU Boosting: LPU improves average long-term forecasting performance for MLP, LSTM, CNN, and Attention backbones.The reported improvements are 119.4% for MLP, 97.0% for LSTM, 13.8% for CNN, and 8.2% for Attention.
  • Efficiency: Low-rank approximation reduces FEL parameter size to 0.1%–0.4% with minor accuracy deterioration.Compared with Transformer-based baselines, FiLM reduces learnable parameters by 80% and memory usage by 50%.
  • Ablation Study: Ablations identify consistent gains from multiscale processing, while normalization produces mixed dataset-dependent effects.The multiscale module improves performance across all datasets; RevIN improves Traffic and Illness but only slightly improves the remaining datasets.

5 Discussions and Conclusion

The paper frames long-term forecasting as a trade-off between preserving historical information and reducing noise. FiLM addresses this trade-off with a general representation framework that combines Legendre and Fourier projections.

  • Accurate long-term forecasting requires both comprehensive historical-information capture and effective noise removal.
  • FiLM is designed to preserve historical information accurately while removing noisy signals.
  • The framework is intended as a general building block that can be modified for different forecasting scenarios.
  • Traditional ARIMA-based models are limited by their linear autoregressive assumption in complex forecasting tasks.
  • Orthogonal bases, including polynomial and Fourier families, can serve as plug-in operations for neural-network representation learning.

B Algorithm Implementation

The implementation describes Fourier-enhanced layers, Legendre projection components, benchmark datasets, and training procedures. Experiments compare FiLM with FEDformer and evaluate univariate forecasting performance.

  • Frequency Enhanced Layer: The Frequency Enhanced Layer multiplies selected Fourier modes and returns the transformed representation to physical space.
  • Frequency Enhanced Layer: The layer supports configurable input and output channels, Fourier mode counts, and optional low-rank approximation.
  • Legendre projection: The Legendre projection uses N as the polynomial order and dt as a step size roughly inverse to sequence length.
  • Datasets: The benchmark covers ETT, Electricity, Exchange, Traffic, and Weather time-series datasets with multiple forecasting settings.
  • Training: FiLM is trained with ADAM, typically for 15 epochs, and the lowest-validation-loss model is used for final testing.
  • Results: FiLM outperforms FEDformer overall in repeated multivariate forecasting experiments with error bars.

C.5 ETT Full Benchmark

On the full ETT benchmark, FiLM reduces mean squared error relative to FEDformer in both multivariate and univariate forecasting. Additional analyses examine low-rank approximation and sensitivity to Legendre-polynomial and frequency-mode counts.

  • ETT full benchmark: 14.0% relative MSE reduction is achieved for multivariate forecasting over FEDformer on the full ETT benchmark.
  • ETT full benchmark: 16.8% relative MSE reduction is achieved for univariate forecasting over FEDformer on the full ETT benchmark.
  • Low-rank approximation: Low-rank approximation is studied for its effect on model accuracy in the Frequency Enhanced Layer.
  • Parameter sensitivity: Reconstruction MSE decreases sharply before saturating at an optimal Legendre-polynomial count proportional to input length.
  • Parameter sensitivity: For Electricity forecasting, the best performance occurs at N > 64 for input length 192 and N larger than 128 for input length 720.

G Noise Injection Experiment

Noise-injection experiments evaluate FiLM’s robustness in long-term forecasting, while the KS test assesses whether input and output sequences share a distribution. FiLM’s performance deteriorates by less than 1.5% in the worst reported noise condition.

  • Noise injection: Less than 1.5% deterioration occurs in the worst case when Gaussian noise is added during training or testing.
  • Noise injection: FiLM’s robustness remains consistent across different forecasting horizons under noise injection.
  • KS test: The KS test checks whether model input and output sequences come from the same probability distribution.
  • KS test: The KS statistic is the supremum difference between the empirical distribution functions of two compared samples.

H.2 Distribution Analysis

The analysis compares whether forecasting outputs remain distributionally similar to input sequences using the Kolmogorov-Smirnov test on ETTm1 and ETTm2.

  • The KS test evaluates whether models’ output sequences are consistent with their input sequences on ETTm1 and ETTm2.Larger p-values indicate that rejecting the same-distribution hypothesis is less likely.
  • At a standard P-value of 0.01, most existing baselines produce outputs likely sampled from distributions different from their input signals.
  • Autoformer and FEDformer have larger P-values, mainly attributed to their seasonal-trend decomposition mechanism.

I Learnable Parameter Size

FiLM substantially reduces learnable parameters relative to Transformer-based baselines through its lightweight design and low-rank approximations.

  • 80% learnable parameter reduction distinguishes FiLM from Transformer-based baseline models.
  • Table 16 compares baseline and FiLM parameter sizes across different low-rank approximation settings on the ETT dataset.The subscript identifies the low-rank approximation parameter k.

J Training Speed and Memory Usage

The experiments examine FiLM’s memory usage, training speed, benchmark performance, and plug-in behavior across long-term forecasting settings. FiLM is reported as lightweight, memory-efficient, fast, and broadly effective across backbones and datasets.

  • Memory Usage: FiLM maintains quasi-constant memory usage as output length increases, with memory usage only linear in input length.Its smaller memory footprint is associated with its simple architecture and low-rank parameter compression.
  • Training Speed: FiLM trains faster than other models as output length increases under the reported comparison settings.The toy experiment fixes input length at 96; the reported setup uses one NVIDIA V100 32GB GPU.
  • Training Speed: FiLM-256 shows advantages in both speed and accuracy by using batch size = 256 versus batch size = 32 for baselines on one GPU.The larger batch size is enabled by FiLM’s low memory usage.
  • Training Speed: FiLM has smaller performance variation and degradation with large batch sizes, consistent with its shallow structure and fewer trainable parameters.
  • Ablation and Benchmark Results: FiLM remains the most consistent and effective variant across six datasets, although LPU+MLP is slightly better on ETTm1.FEL is reported as a better backbone than MLP, LSTM, CNN, and vanilla attention modules.
  • Ablation and Benchmark Results: LPU consistently boosts most tested deep learning backbones across selected datasets, while pure LPU can reduce performance without data normalization.
  • Ablation and Benchmark Results: Low-rank approximation substantially reduces the Fourier Enhanced Layer’s parameter size and can improve performance on some datasets.

K.5 Ablation univariate forecasting experiments for frequency mode selection with all six datasets

The ablation studies evaluate frequency-mode selection, backbone variants, LPU boosting, and low-rank approximation across forecasting experiments.

  • Frequency Mode Selection: The frequency enhanced layer compares lowest-frequency, random, and mixed low/high-frequency mode-selection policies with m = 64.
  • Frequency Mode Selection: Lowest-mode selection is reported as the most stable frequency selection policy.
  • FEL Ablation: Replacing the FEL layer with MLP, LSTM, CNN, or Transformer provides the backbone variants used in the ablation study.Experiments are performed on ETTm1 and Electricity with relative-value metrics.
  • LPU Ablation: LPU’s boosting effect is evaluated across MLP, LSTM, CNN, and attention backbones, with degraded performance marked by ‘+’.
  • Low-Rank Approximation: Low-rank approximation experiments compare the default Fourier layer with compressed versions, where Comp. K=0 is default and K=1 is the largest compression.
Loading 2205.08897v4…