PaperlayerView exact revision on arXiv

Source-linked AI summary

Fredformer: Frequency Debiased Transformer for Time Series Forecasting

Xihao Piao, Zheng Chen, Taichi Murayama, Yasuko Matsubara, Yasushi Sakurai

arXiv:2406.09009v4cs.LGcs.AI

TL;DR

Transformers may overlook useful high-frequency time-series features when dominant low-frequency components receive disproportionate attention. Fredformer analyzes and targets this frequency bias with band-based designs; experiments report strong results across eight datasets and a lightweight variant for computational efficiency.

  • Problem

    Time-series frequencies can be entangled, while Transformers may prioritize high-energy low-frequency components over useful high-frequency variations.

    Source: §1 · p. 1

  • Method

    Fredformer analyzes frequency bias and combines frequency-band patching, sub-frequency-independent normalization, and channel-wise attention to support learning across bands.

    Source: §1 · p. 2

  • Results

    Across eight datasets, Fredformer achieves 60 top-1 and 20 top-2 cases out of 80.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that Fredformer mitigates frequency bias, and its lightweight variant addresses computational efficiency for practical application.

    Source: §7 · p. 8

    Channel-wise attention increases computational costs with channel count, potentially limiting applicability to many-channel settings.

    Source: §5.4 · p. 7

Abstract

from arXiv · show

The Transformer model has shown leading performance in time series forecasting. Nevertheless, in some complex scenarios, it tends to learn low-frequency features in the data and overlook high-frequency features, showing a frequency bias. This bias prevents the model from accurately capturing important high-frequency data features. In this paper, we undertook empirical analyses to understand this bias and discovered that frequency bias results from the model disproportionately focusing on frequency features with higher energy. Based on our analysis, we formulate this bias and propose Fredformer, a Transformer-based framework designed to mitigate frequency bias by learning features equally across different frequency bands. This approach prevents the model from overlooking lower amplitude features important for accurate forecasting. Extensive experiments show the effectiveness of our proposed approach, which can outperform other baselines in different real-world time-series datasets. Furthermore, we introduce a lightweight variant of the Fredformer with an attention matrix approximation, which achieves comparable performance but with much fewer parameters and lower computation costs. The code is available at: https://github.com/chenzRG/Fredformer

1 INTRODUCTION

The introduction identifies frequency bias as a reason Transformers may overlook useful high-frequency variations and presents Fredformer as a frequency-debiased approach. It outlines the analysis behind the bias and the model designs and experiments intended to address it.

1 INTRODUCTION

  • Time-series variations across frequencies are entangled, and heuristic top-K or random-K selection may introduce suboptimal frequency correlations.The introduction identifies frequency decomposition and frequency-domain feature selection as existing approaches.Source: §1 · p. 1
  • Transformers may prioritize high-energy, low-frequency components and neglect high-frequency features that can indicate short-term variations.The introduction links low-frequency dominance to larger amplitudes and more observations for the model.Source: §1 · p. 2
  • Fredformer combines frequency-band patching, sub-frequency-independent normalization, and channel-wise attention within each band to support fairer frequency learning.The authors report that their empirical analysis identifies proportional differences between key frequency components as the main cause of bias.Source: §1 · p. 2
  • Nyström approximation reduces attention-map computation, yielding a lightweight variant with competitive performance.The introduction describes this as an applicability contribution.Source: §1 · p. 2
  • Fredformer records 60 top-1 and 20 top-2 cases out of 80 in experiments across eight datasets.The authors describe these results as superior performance.Source: §1 · p. 2

2 PRELIMINARY ANALYSIS

The preliminary analyses examine how Transformers learn frequency components and test whether normalization or direct frequency-domain modeling can reduce frequency bias. Across synthetic cases, learning favors higher-amplitude components, while frequency-domain modeling with proportion mitigation improves learning of less-dominant components.

2.1 Preliminary

  • The preliminary analysis uses DFT coefficients to represent frequency amplitudes and IDFT to reconstruct time-series data.The section frames the case studies as analyses of frequency-driven bias and potential debiasing strategies.Source: §2.1 · pp. 2–3, §2 · p. 2

2.2 Case Studies

  • After 50 epochs, relative error falls to around 0.01 for k1 but remains almost 0.95 for k3; reversing amplitude order shifts focus to k3 (Δk1 > Δk2 > Δk3).Figure 2(a) presents these learning dynamics and errors for the two synthetic datasets.Source: Figure 2 · p. 3, §2.2.1 · p. 3

2.1 Preliminary

  • The frequency-bias metric uses relative error Δk for each frequency component, building on its component-wise mean-square error.The metric is used in the case studies and later experiments.Source: §2.1 · p. 3

2.2 Case Studies

  • The case studies use synthetic series with three key components of reversed amplitude ordering and a four-component dataset for debiasing analyses.The three-component datasets keep the component proportions consistent between observations and ground truth.Source: §2.2 · p. 3
  • Frequency normalization equalizes input-component amplitudes and improves forecasting for the latter three frequencies, but their relative errors remain high.The procedure applies DFT, normalizes amplitudes, and uses IDFT before the Transformer input.Source: §2.2.1 · p. 3
  • Direct frequency-domain modeling markedly improves the latter three components and effectively eliminates bias in the 60–75 Hz component.The analysis pairs DFT-based frequency modeling with IDFT to return the forecast to the time domain.Source: §2.2.1 · pp. 3–4

3 FREQUENCY BIAS FORMULATION

The formulation defines key frequency components and characterizes frequency bias through their relative errors and amplitude proportions. It frames debiasing as making the Transformer respond evenly to these components.

3.1 Frequency Bias Definitions

  • Key components are defined as relatively high-amplitude frequencies that remain consistent between historical observations and future series and robust to time shifts.These criteria describe which components count as key for forecasting.Source: §3.1 · p. 4
  • The spectrum is divided into sliding sub-frequency windows, and each window’s maximum amplitude is used in defining key components.Each window contains s amplitudes.Source: §3.1 · p. 4
  • The collection of key components, ˜A, should be present in both the historical spectrum and the ground truth.The formulation identifies ˜A as the collection of all key components.Source: §3.1 · p. 4
  • Frequency bias is characterized by larger amplitude proportions aligning with smaller relative errors and higher component rankings; the Transformer attends more to high-ranked components during training.The formulation links component proportion to relative error, while the empirical analysis notes the attention pattern.Source: §3.1 · p. 4

3.2 Problem Statement

  • The proposed debiasing premise is that assigning equal attention to all key frequency components could mitigate frequency bias.Source: §3.2 · p. 4
  • The stated objective is to make relative error independent of each key component’s amplitude proportion, thereby balancing the Transformer’s response across components.Source: §3.2 · p. 4

4 FREDFORMER

Fredformer transforms time series into frequency components, processes sub-frequency bands independently, and summarizes them to produce forecasts. Its design combines local normalization with channel-wise attention to reduce disproportionate focus on frequency components.

4 FREDFORMER

  • Fredformer decomposes inputs with DFT, learns debiased frequency representations through refinement, local frequency-independent learning, and global summarization, then reconstructs forecasts with IDFT.Patches from different channels within the same sub-frequency band are embedded as tokens and encoded independently.Source: §4 · p. 4

4.1 Backbone

  • The DFT-to-IDFT backbone decomposes inputs into frequency coefficients, extracts debiased features with a Transformer encoder, and reconstructs the time-domain signal.Source: §4.1 · p. 4

4.2 Frequency Refinement and Normalization

  • Patching splits frequency coefficients into local sub-frequencies, and normalizing each patch projects values to 0–1 to reduce amplitude differences between key components.Patch length is adaptable to the requirements of real-world scenarios.Source: §4.2 · pp. 4–5

4.3 Frequency Local Independent Modeling

  • Frequency-local Transformer encoders apply channel-wise attention independently within each sub-frequency band, weighting channel correlations without allowing one band to influence another.The design aims to avoid disproportionate attention to frequency components.Source: §4.3 · p. 5

4.4 Frequency-wise Summarization

  • The frequency-wise summarizing operation uses linear projections and IDFT to turn the learned frequency features into forecasts.Source: §4.4 · p. 5

5 EXPERIMENTS

Experiments evaluate Fredformer across eight real-world datasets, analyzing forecasting performance, frequency-bias behavior, component ablations, and computational applicability. Results show strong benchmark rankings and a lightweight variant that reduces computational requirements without compromising accuracy in the reported comparison.

5.1 PROTOCOLS

  • The protocol tests eight datasets against 11 state-of-the-art baselines using prediction lengths 96, 192, 336, and 720 and a look-back window of 96.Forecasting is assessed with MSE and MAE.Source: §5.1 · pp. 5–7

5.2 Results

  • Fredformer earns 14 top-1 and 2 top-2 positions across eight datasets, plus 34 top-1 and 6 top-2 rankings across 40 outcomes on four selected datasets.Results use a look-back length of 96; the full four-dataset results cover four prediction lengths.Source: Table 1 · p. 6, Table 2 · p. 6, §5.2 · p. 7
  • On ETTh1, Fredformer learns selected mid-to-high frequency components with decreasing relative errors, while baselines show less effective or uneven learning.PatchTST improves on k2 and k3 late in training; FEDformer may miss unselected components, and iTransformer overlooks some mid-to-high frequencies.Source: §5.2 · p. 7

5.3 Ablation Study

  • Removing either channel-wise attention or frequency refinement reduces performance, while frequency local normalization alone outperforms channel-wise attention alone.Increasing patch length coarsens extracted frequency features and decreases forecasting accuracy on ETTh1.Source: §5.3 · p. 7, Table 5 · p. 7

5.4 Discussion of Applicability

  • Nyström-Fredformer reduces attention complexity from O(L/PC^2) to O(L/PC) without changing feature extraction or the Transformer data stream.The plain model has high accuracy and low computational costs on 7-channel ETTh1, while costs rise on 321-channel ECL; the variant reduces requirements without compromising accuracy there.Source: §5.4 · pp. 7–8, Figure 5 · p. 8

6 RELATED WORKS

Prior work applies Transformers to temporal dependencies and combines them with frequency decomposition or spectrum analysis. Other approaches target short-term variation and cross-channel relationships.

Transformer for Time Series Forecasting

  • Transformer forecasting research includes methods for temporal dependencies and models that integrate frequency decomposition or spectrum analysis.Some methods place attention layers directly in the frequency domain.Source: §6 · p. 8

Modeling Short-Term Variation in Time Series

  • Short-term variation methods include Pyraformer, FEDformer, and TimesNet, which use pyramidal attention, Fourier-spectrum attention, and Inception blocks, respectively.These approaches target temporal dependencies, pivotal frequency components, or intra-period and inter-period variation.Source: §6 · p. 8

Channel-wise Correlation

  • Crossformer and iTransformer use channel-wise Transformer frameworks, while other studies model inter-channel correlations with graph neural networks.The passage notes reported experimental effectiveness for the channel-wise frameworks.Source: §6 · p. 8

7 CONCLUSION

The paper concludes that Fredformer mitigates frequency bias and performs well across eight datasets, while a lightweight variant addresses computational efficiency. Its case-study data-generation procedure moves prominent low-frequency components into the middle of the spectrum for analysis.

7 CONCLUSION

  • Experiments across eight datasets support Fredformer’s performance, and visual analysis indicates that it mitigates frequency bias.The authors also report model analyses of how the designs aid debiasing and offer preliminary guidelines for future model design.Source: §7 · p. 8
  • The lightweight variant addresses computational efficiency to facilitate practical application.The conclusion presents it as an efficiency-focused extension of the Fredformer framework.Source: §7 · p. 8

A DETAILS OF THE CASE STUDIES

  • For the second case study, four prominent low-frequency components are moved to the mid-frequency range before the spectrum is rearranged and transformed back to time data.The procedure preserves inherent noise and instability from real data while constructing data with prominent middle-frequency components.Source: A DETAILS OF THE CASE STUDIES · p. 12

B MORE DETAILS OF THE DATASETS

The appendix describes the benchmark datasets and outlines both a procedure for generating modified frequency data and the Fredformer processing pipeline.

B MORE DETAILS OF THE DATASETS

  • The benchmarks include Weather, four ETT datasets, Electricity, Solar-Energy, and Traffic, spanning weather, energy, and road-occupancy observations.The supplied dataset descriptions include 321 electricity clients and 862 traffic sensors.Source: B MORE DETAILS OF THE DATASETS · p. 12
  • The case-study data-generation algorithm applies DFT, selects four prominent low-frequency components, moves them to mid-frequency, rearranges spectrum parts, and applies IDFT.It preserves and reinserts the date column after processing.Source: B MORE DETAILS OF THE DATASETS · p. 12

C FREDFORMER ALGORITHM

  • Fredformer transforms inputs with DFT, segments frequency bands, applies normalization and a Transformer for cross-channel learning, then summarizes frequencies and returns forecasts through IDFT.The pipeline is described as a DFT-to-IDFT backbone with frequency-local independent learning and frequency-wise summarization.Source: C FREDFORMER ALGORITHM · p. 12

D LOOK-BACK WINDOW ANALYSIS

Longer look-back windows improved forecasting accuracy on ETTh1 and Weather, while the appendix also examines hyperparameter robustness and visualizes predictions across channels. The sensitivity analysis favors stable hyperparameter selection, and the time-domain examples highlight channel-wise and frequency-domain modeling.

D LOOK-BACK WINDOW ANALYSIS

E HYPERPARAMETER SENSITIVITY

F VISUALIZATIONS OF THE FORECASTING RESULTS

  • The appendix groups added forecasting visualizations into time-domain prediction accuracy and frequency-domain component capture.The time-domain examples include ETTh1 channels #5 and #2.Source: F VISUALIZATIONS OF THE FORECASTING… · p. 14

F.1 Time domain

  • Across channels with differing data characteristics, Fredformer captures similarities and differences better than FEDformer and more detailed global and local features than iTransformer.The examples are ETTh1 channels #5 and #2; the authors also point to the importance of channel-wise attention and frequency-domain modeling.Source: §F.1 · p. 14

F.2 Frequency domain

The frequency-domain visualization examines whether Fredformer learns low-amplitude mid-to-high-frequency components. It reports successful learning of three components and diminishing relative error for a fourth, contrasting with the baselines.

F.2 Frequency domain

  • Fredformer accurately identifies mid-to-high-frequency components k1, k2, and k3, with uniformly decreasing relative errors during training.These components have amplitudes significantly lower than the low-frequency components.Source: §F.2 · p. 15
  • For k4, relative error consistently diminishes despite a larger learning error.The passage does not report that k4 is accurately identified.Source: §F.2 · p. 15
  • The baselines are less effective at capturing these frequency components and show unequal reductions in relative error.The comparison concerns the low-amplitude mid-to-high-frequency components discussed in the visualization.Source: §F.2 · p. 15

G DETAILS OF THE LEMMA 1

The appendix explains why normalizing across a whole time-domain patch or frequency spectrum does not remove amplitude disparities among key components. It also outlines a Nyström approximation that uses landmarks to approximate channel-wise attention and reduce its computational burden.

G DETAILS OF THE LEMMA 1

  • Time-domain patch normalization and global frequency-domain normalization leave amplitude bias among key frequency components unresolved.The passage relates the two operations through Parseval’s energy equivalence.Source: G DETAILS OF THE LEMMA 1 · pp. 15–16
  • Normalizing a time-domain patch corresponds to normalizing the weights of all frequency components across its S points, but does not equalize component maxima across patches.The authors use this distinction to explain why whole-patch or whole-spectrum normalization does not remove key-frequency amplitude bias.Source: G DETAILS OF THE LEMMA 1 · p. 16

H NYSTRÖM APPROXIMATION IN TRANSFORMER SELF-ATTENTION MECHANISM

  • The Nyström method averages rows or columns to select m landmarks, simplifies Q_n and K_n, and uses them to approximate channel-wise attention.The method approximates the softmax matrix and is presented as offering computational benefits.Source: H NYSTRÖM APPROXIMATION IN TRANSFORMER… · pp. 16–17

I DETAILED RESULTS OF ALL DATASETS

With a default look-back window of 96, Fredformer achieves leading performance across most datasets and prediction lengths. The full-dataset results report 60 top-1 cases out of 80 possible outcomes.

I DETAILED RESULTS OF ALL DATASETS

Loading 2406.09009v4…