PaperlayerView exact revision on arXiv

Source-linked AI summary

FreDF: Learning to Forecast in the Frequency Domain

Hao Wang, Licheng Pan, Zhichao Chen, Degui Yang, Sen Zhang, Yifei Yang, Xinggao Liu, Haoxuan Li, Dacheng Tao

arXiv:2402.02399v2cs.LGcs.AIstat.APstat.ML

TL;DR

Direct Forecast methods overlook dependencies among future labels, although label autocorrelation can bias their learning objective. FreDF forecasts in the frequency domain to mitigate that autocorrelation, and experiments report improved performance across diverse datasets and prevalent forecasting models.

  • Problem

    Direct Forecast methods generate multistep predictions while overlooking autocorrelation among future labels, despite dependencies between future steps.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    FreDF refines the Direct Forecast objective by transforming labels into the frequency domain, where frequency components become decorrelated.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Experiments show that FreDF enhances state-of-the-art forecasting models across diverse datasets and improves the performance of prevalent forecast models.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    FreDF is presented as a model-agnostic learning objective for reducing bias associated with label autocorrelation in time-series forecasting.

    Source: §5 · p. 10

    The Fourier transform uses predefined exponential bases that cannot adapt to specific data properties, and performance benefits beyond time series remain unexplored.

    Source: §5 · p. 10

Abstract

from arXiv · show

Time series modeling presents unique challenges due to autocorrelation in both historical data and future sequences. While current research predominantly addresses autocorrelation within historical data, the correlations among future labels are often overlooked. Specifically, modern forecasting models primarily adhere to the Direct Forecast (DF) paradigm, generating multi-step forecasts independently and disregarding label autocorrelation over time. In this work, we demonstrate that the learning objective of DF is biased in the presence of label autocorrelation. To address this issue, we propose the Frequency-enhanced Direct Forecast (FreDF), which mitigates label autocorrelation by learning to forecast in the frequency domain, thereby reducing estimation bias. Our experiments show that FreDF significantly outperforms existing state-of-the-art methods and is compatible with a variety of forecast models. Code is available at https://github.com/Master-PLC/FreDF.

1 INTRODUCTION

Time-series forecasting must contend with autocorrelation in both historical inputs and future labels, but modern Direct Forecast methods focus primarily on the inputs. FreDF addresses overlooked label autocorrelation by forecasting in the frequency domain, and experiments report improved performance across diverse datasets.

1 INTRODUCTION

  • Time-series autocorrelation occurs in both input and label sequences, while prior architectures and frequency-based models primarily address input-sequence autocorrelation.Source: §1 · p. 1
  • Direct Forecast generates multistep predictions simultaneously but overlooks dependencies among future labels, and this oversight biases its learning objective.Future steps are autoregressively dependent, while DF implicitly assumes step-wise independence.Source: §1 · pp. 1–2
  • FreDF aligns forecasts and labels in the frequency domain, where label correlation diminishes, while retaining Direct Forecast’s sample efficiency and implementation simplicity.Source: §1 · p. 2
  • Experiments show that FreDF enhances state-of-the-art forecasting models across a diverse range of datasets.Source: §1 · p. 2

2 PRELIMINARIES AND RELATED WORK

The section distinguishes modeling autocorrelation in historical inputs from respecting dependencies among future labels. It reviews model architectures and forecast paradigms before positioning this work as a frequency-domain refinement of the Direct Forecast loss.

2.1 PROBLEM DEFINITION

  • A multivariate forecast maps the historical input sequence L of length H to a predicted label sequence Y of length T.The model g maps from R^(H×D) to R^(T×D).Source: §2.1 · p. 2
  • Time-series modeling addresses autocorrelation through both architectures that encode input dependencies and forecast paradigms that respect label dependencies.Source: §2.1 · p. 2

2.2 MODEL ARCHITECTURES

  • Architectures reviewed range from statistical methods and neural networks to Transformers and MLPs, with Transformers scaling better but costing more computationally and MLPs generally more efficient but less scalable.Source: §2.2 · p. 2
  • Frequency-domain sequence representations are used across Transformers, MLPs, and GNNs to manage autocorrelation with limited cost.Source: §2.2 · p. 2

2.3 ITERATIVE FORECAST V.S. DIRECT FORECAST

  • Iterative Forecast models one step at a time but can suffer error propagation, whereas Direct Forecast generates multiple steps simultaneously and is associated with fast inference, easier implementation, and superior accuracy.Source: §2.3 · p. 3
  • This work refines Direct Forecast through the loss function to address bias from label autocorrelation, unlike frequency methods focused on architecture or computation.Source: §2.3 · p. 3

3 PROPOSED METHOD

The section diagnoses bias in Direct Forecast’s independent-step objective when future labels are autocorrelated, then develops FreDF, which aligns forecasts and labels in the frequency domain.

3.1 MOTIVATION

  • DF computes temporal loss independently at each forecast step despite autoregressive label dependence, biasing its objective against the practical NLL; the bias vanishes when labels are uncorrelated.Source: §3.1 · pp. 3–4

3.2 REDUCE LABEL AUTOCORRELATION WITH FOURIER TRANSFORM

  • On Weather, approximately 37.5% of time-domain label correlations exceed 0.3, while about 3.6% of frequency-domain correlations exceed 0.1.Source: Figure 1 · p. 4, §3.2 · pp. 4–5
  • The DFT projects labels onto orthogonal Fourier bases, where frequency components become asymptotically uncorrelated for a zero-mean wide-sense stationary process as T → ∞.The transformation can be computed by FFT with O(T log T) complexity.Source: §3.2 · p. 4

3.3 MODEL IMPLEMENTATION

  • FreDF aligns forecast and label sequences in the frequency domain to mitigate bias while retaining efficient inference and supporting multiple forecasting models.It is compatible with models such as Transformers and MLPs.Source: §3.3 · p. 5
  • FreDF combines temporal and frequency forecast errors using α, and uses frequency-domain ℓ1 loss to balance components with widely varying magnitudes.The authors state that this choice seeks a more balanced and stable optimization process.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments evaluate FreDF’s long-term forecasting performance and examine its components, generality, transformation choices, hyperparameter sensitivity, and sample efficiency.

4.1 SETUP

  • Experiments use public long-term forecasting and imputation datasets, established model baselines, and benchmark settings, with input length fixed at 96.Datasets are split chronologically; the baseline set includes Transformer, MLP-based, and other forecasting models.Source: §4.1 · p. 6, §4 · p. 6

4.2 OVERALL PERFORMANCE

  • 0.019: On ETTm1, FreDF reduces iTransformer’s MSE by 0.019, with similar gains across other datasets and some results surpassing models that originally outperformed iTransformer.The results are reported in Table 1.Source: §4.2 · pp. 6–7, Table 1 · p. 6

4.3 ABLATION STUDIES

  • Frequency loss consistently improves performance over temporal loss, while combining both domains generally helps; its gain over frequency-only loss is marginal.The ablation uses iTransformer and reports results in Table 2.Source: Table 2 · p. 7, §4.3 · p. 8

4.4 GENERALIZATION STUDIES

  • FreDF improves iTransformer, DLinear, Autoformer, and Transformer, with substantial benefits for Autoformer and Transformer.On ECL, FreDF-enhanced Autoformer outperforms DLinear; Figure 4 summarizes means across forecast lengths with confidence intervals.Source: §4.4 · p. 8, Figure 4 · p. 8
  • Time- and variable-axis FFT yield similar gains, while two-dimensional FFT further improves performance by accommodating correlations across both axes.Table 3 reports the FFT implementation results.Source: Table 3 · p. 7, §4.4 · p. 8
  • Legendre and Fourier projections perform better than other tested bases, supporting orthogonality as an important basis-selection property.Figure 5 summarizes results across forecast lengths.Source: §4.4 · pp. 8–9, Figure 5 · p. 9

4.5 HYPERPARAMETER SENSITIVITY

  • Increasing α from 0 toward 1 generally reduces forecast error, though error rises slightly near the range’s end and optima can occur below 1.For ECL at T=192, MAE and MSE decrease from approximately 0.258 and 0.167 to 0.247 and 0.158, respectively.Source: §4.5 · p. 9, Figure 6 · p. 9

4.6 LEARNING-CURVE ANALYSIS

  • With only 30% of training data, frequency-domain learning matches time-domain learning trained on the full dataset.The learning curve is shown for ETTm1 in Figure 7; the passage attributes the sample efficiency to more consistent frequency-domain representations.Source: §4.6 · pp. 9–10, Figure 7 · p. 9

5 CONCLUSION

FreDF addresses bias in the Direct Forecast objective caused by label autocorrelation. The appendix explains how double machine learning estimates partial correlations while accounting for input sequences.

5 CONCLUSION

  • FreDF is a model-agnostic objective that transforms labels into the frequency domain to mitigate label autocorrelation and reduce bias.Experiments show improved performance for prevalent forecast models.Source: §5 · p. 10

A.1 MOTIVATION

  • Input sequences can create spurious correlations between future labels, making traditional correlation measures ineffective for quantifying their direct dependence.The appendix uses double machine learning (DML) to distinguish direct relationships from these spurious ones.Source: §A.1 · p. 13

A.2 METHOD

  • DML first removes control-variable effects from the outcome and treatment, then regresses the residuals to estimate their partial correlation.This isolates the treatment–outcome relationship from correlation induced by the controls.Source: §A.2 · p. 13

A.3 EXPERIMENTAL SETTINGS

The appendix estimates time- and frequency-domain partial correlations with DML, controlling for the input sequence. Its settings simplify estimation by focusing on selected input and feature components.

A.3 EXPERIMENTAL SETTINGS

  • The DML experiment uses linear-regression base learners, adjusts for the input sequence, and treats its last step as representative.It focuses on correlations within each dataset’s last feature, making the outcome scalar.Source: §A.3 · pp. 13–14
  • For time-domain estimates, DML treats one label step as the treatment and a later step as the outcome, with the fitted coefficient measuring their partial correlation.The input sequence is included as the control.Source: §A.3 · p. 14
  • For frequency-domain estimates, DML treats one frequency component as the treatment and another as the outcome, estimating their partial correlation separately for real and imaginary parts.The separate treatment is needed because frequency components are complex-valued.Source: §A.3 · p. 14

A.4 MORE EXPERIMENTAL RESULTS

Across datasets and forecast lengths, label correlations are pronounced in the time domain but negligible among frequency components given the input. The theoretical analysis connects label autocorrelation to bias in DF’s learning objective and establishes asymptotic decorrelation under stated conditions.

A.4 MORE EXPERIMENTAL RESULTS

  • Non-diagonal label correlations are often large in the time domain but negligible in the frequency domain across three datasets and four ECL forecast lengths.The figures show partial correlations given the input sequence.Source: §A.4 · p. 14, Figure 9 · p. 15, Figure 10 · p. 16

B THEORETICAL JUSTIFICATION

  • With label autocorrelation, DF’s time-domain MSE is biased relative to the practical NLL and can mislead forecast-model updates; the bias approaches zero as autocorrelation diminishes.The two-step theorem gives the bias expression, and the result extends to multivariate label sequences.Source: B THEORETICAL JUSTIFICATION · pp. 17–18
  • For a zero-mean, wide-sense stationary label process, normalized DFT coefficients at different frequencies become asymptotically uncorrelated as sequence length tends to infinity.The proof attributes cancellation at distinct frequencies to oscillatory complex exponentials.Source: B THEORETICAL JUSTIFICATION · pp. 18–19

C GENERALIZED TRANSFORMATION ONTO DIFFERENT BASES

FreDF can use different transform bases, whose suitability depends on data properties and analysis goals. The appendix describes these bases and illustrates how non-Fourier bases can capture trends and periodicity that DF struggles with.

C GENERALIZED TRANSFORMATION ONTO DIFFERENT BASES

  • The appendix describes Fourier, Legendre, Chebyshev, and Laguerre transforms, each using its corresponding basis for projection.Fourier bases are suited to periodic or strongly frequency-driven signals; the polynomial bases have distinct interval and approximation properties.Source: C GENERALIZED TRANSFORMATION ONTO… · pp. 19–20
  • In selected trend and periodicity snapshots, DF struggles to capture increasing trends or high-frequency periods, while FreDF forecasts capture these dominant patterns.The case study uses iTransformer forecasts.Source: C GENERALIZED TRANSFORMATION ONTO… · p. 20, Figure 11 · p. 20
  • FreDF is not limited to Fourier bases; the basis can be selected according to data characteristics and analysis objectives.The appendix presents alternative polynomial bases as options for adapting the transform.Source: C GENERALIZED TRANSFORMATION ONTO… · pp. 19–20

D.1 DATASET DESCRIPTIONS

The study uses several public time-series datasets spanning electricity, weather, traffic, and transformer measurements. It follows established chronological splits and benchmark training settings, with α tuned when adding FreDF to existing models.

D.1 DATASET DESCRIPTIONS

  • The datasets cover transformer measurements, weather, electricity consumption, road occupancy, and California traffic-network data.ETT includes hourly and 15-minute subsets; the other datasets include 10-minute weather, hourly electricity and road-occupancy records, and 5-minute traffic windows.Source: §D.1 · pp. 20–21
  • Data are split chronologically, with input length 96 and forecast lengths of 96, 192, 336, or 720 for ETT, Weather, ECL, and Traffic.The test phase disables the dropping-last trick.Source: §D.1 · p. 21

D.2 IMPLEMENTATION DETAILS

  • Baseline models use Adam, select learning rates from 10^-3, 5 × 10^-4, and 10^-4 to minimize MSE loss, and train for at most 10 epochs with early stopping after three epochs without validation improvement.Source: §D.2 · p. 21
  • When integrating FreDF into existing models, the study retains benchmark hyperparameters, tunes α within [0, 1], and conservatively adjusts the learning rate.Validation tuning minimizes MSE averaged across forecast lengths; learning-rate adjustment accommodates the different magnitudes of temporal and frequency losses.Source: §D.2 · p. 21

E.1 OVERALL PERFORMANCE

Across forecasting and imputation evaluations, FreDF improves existing models, with broad gains across long-term cases and several model architectures. The studies also examine its behavior across loss weights, DTW-based objectives, and input lengths.

E.1 OVERALL PERFORMANCE

  • FreDF with iTransformer achieves the lowest MSE in 31 of 45 long-term cases and the lowest MAE in 40 of 45, while improving FreTS in most short-term cases.The long-term comparison uses Table 5, and the short-term comparison uses Table 6.Source: §E.1 · p. 21, Table 5 · p. 22, Table 6 · p. 22
  • In short-term forecasting, three cases favor FreTS because α is tuned for validation error averaged across forecast lengths rather than separately for each length.The authors note that per-length tuning is feasible but was not used.Source: §E.1 · p. 24
  • For missing-data imputation, FreDF improves iTransformer and outperforms most competitive methods, despite irregular labels disrupting the Fourier transform’s physical semantics.The authors attribute the effectiveness to aligning time-series properties with DF’s implicit conditional-independence assumption, not to Fourier-transform semantics.Source: Table 7 · p. 23, §E.1 · p. 24
  • The forecast-sequence showcases show FreDF mitigating blur and capturing high-frequency components.The examples are from two ETTm2 snapshots at forecast length 336.Source: §E.1 · p. 24, Figure 12 · p. 24, Figure 14 · p. 25

E.2 GENERALIZATION STUDIES

  • FreDF improves iTransformer, DLinear, Autoformer, and Transformer across five datasets, with particularly strong benefits for Transformer-based architectures.The reported errors are averaged over forecast lengths, and the figure shows 95% confidence intervals.Source: §E.2 · p. 24, Figure 16 · p. 26

E.3 HYPERPARAMETER SENSITIVITY

  • Increasing α from 0 to 1 generally reduces forecast error across datasets and forecast lengths, though minima often occur below 1, such as α = 0.8 on ETTh1.The authors suggest that combining temporal and frequency supervisory signals helps; gains over α = 1 may be incremental.Source: §E.3 · p. 27, Figure 17 · p. 28, Figure 18 · p. 29

E.4 COMPARISON WITH DTW-BASED LEARNING OBJECTIVES

  • FreDF significantly outperforms DTW-based Dilate and DPP on both evaluated datasets in the iTransformer comparison.The authors distinguish FreDF by its ability to debias the learning objective, which they say Dilate and DPP do not provide.Source: §E.4 · p. 27, Table 8 · p. 27

A CASE STUDY WITH PATCHTST AND VARYING INPUT LENGTH.

B RUNNING COST ANALYSIS

The appendix examines FreDF’s added running cost, sensitivity to random seeds, and the contributions of amplitude and phase alignment. It reports a small training-time overhead, limited seed sensitivity, and the importance of phase alignment.

B RUNNING COST ANALYSIS

  • For label sequences with T < 720, FreDF adds approximately 1 ms to training, and frequency-loss computation is not required at inference.The timing analysis separates forward-pass FFT calculation from backward-pass frequency-loss and gradient computation.Source: B RUNNING COST ANALYSIS · pp. 31–32, Figure 1 · p. 31

C RANDOM SEED SENSITIVITY

D AMPLITUDE V.S. PHASE ALIGNMENT

  • Both amplitude and phase alignment contribute to FreDF’s performance, with phase alignment particularly important; amplitude-only alignment yields subpar performance.The authors relate phase sensitivity to the substantial time-domain discrepancies that even minor phase deviations can produce.Source: D AMPLITUDE V.S. PHASE ALIGNMENT · p. 32, Table 3 · p. 32

E COMPARISON WITH ADDITIONAL FORECAST ARCHITECTURES

The study applies FreDF to TimeMixer and ScaleFormer by replacing their temporal MSE loss with the proposed loss. It reports performance improvements for both architectures.

E COMPARISON WITH ADDITIONAL FORECAST ARCHITECTURES

Loading 2402.02399v2…