Source-linked AI summary
FECAM: Frequency Enhanced Channel Attention Mechanism for Time Series Forecasting
Maowei Jiang, Pengyu Zeng, Kai Wang, Huan Liu, Wenbo Chen, Haoran Liu
TL;DR
Time-series forecasting models may fail to capture frequency information that is abundant in real-world data, while Fourier-based extraction can introduce Gibbs-related high-frequency noise. FECAM uses Discrete Cosine Transform-based channel attention to model frequency interdependencies, achieving state-of-the-art results across six datasets and improving several mainstream architectures.
Problem
Mainstream time-series forecasting models may inadequately model frequency information, while Fourier-based extraction can introduce high-frequency noise through the Gibbs phenomenon.
Method
FECAM uses Discrete Cosine Transform to build frequency-domain channel attention that learns the importance of different frequencies and channels.
Results
FECAM achieves state-of-the-art performance on six real-world datasets and consistently improves LSTM, Reformer, Informer, Autoformer, and Transformer models.
Takeaways & Limitations
FECAM is a general frequency-enhanced module that can be flexibly added to different forecasting architectures with few lines of code.
Abstract
from arXiv · showhide
Time series forecasting is a long-standing challenge due to the real-world information is in various scenario (e.g., energy, weather, traffic, economics, earthquake warning). However some mainstream forecasting model forecasting result is derailed dramatically from ground truth. We believe it's the reason that model's lacking ability of capturing frequency information which richly contains in real world datasets. At present, the mainstream frequency information extraction methods are Fourier transform(FT) based. However, use of FT is problematic due to Gibbs phenomenon. If the values on both sides of sequences differ significantly, oscillatory approximations are observed around both sides and high frequency noise will be introduced. Therefore We propose a novel frequency enhanced channel attention that adaptively modelling frequency interdependencies between channels based on Discrete Cosine Transform which would intrinsically avoid high frequency noise caused by problematic periodity during Fourier Transform, which is defined as Gibbs Phenomenon. We show that this network generalize extremely effectively across six real-world datasets and achieve state-of-the-art performance, we further demonstrate that frequency enhanced channel attention mechanism module can be flexibly applied to different networks. This module can improve the prediction ability of existing mainstream networks, which reduces 35.99% MSE on LSTM, 10.01% on Reformer, 8.71% on Informer, 8.29% on Autoformer, 8.06% on Transformer, etc., at a slight computational cost ,with just a few line of code. Our codes and data are available at https://github.com/Zero-coder/FECAM.
1 INTRODUCTION
Time-series forecasting models can deviate substantially from real-world ground truth, motivating frequency-aware modeling. FECAM uses DCT-based frequency-enhanced channel attention and achieves strong performance while generalizing across models.
- Time-series forecasting supports decision-making across weather, health, energy, traffic, and finance applications.
- Transformers and other deep-learning methods capture temporal dependencies effectively but can still produce forecasts derailed from ground-truth distributions.
- Real-world datasets contain rich frequency information, which mainstream forecasting methods may fail to exploit during modeling.
- Fourier-transform-based frequency extraction can introduce boundary errors and high-frequency noise through the Gibbs phenomenon.
- FECAM achieves state-of-the-art performance on six real-world benchmarks and can generalize to other network architectures with few lines of code.
- FECAM uses frequency-enhanced channel attention based on Discrete Cosine Transform to avoid Gibbs-related noise and model frequency information.
2 RELATED WORK AND PRELIMINARY
Prior forecasting architectures model temporal dependencies but do not explicitly capture frequency-domain patterns. Frequency is therefore motivated as an auxiliary representation, while Fourier-based methods face Gibbs-related artifacts.
- RNNs can suffer from long-term dependency modeling, while Transformer variants develop specialized mechanisms for efficient temporal forecasting.
- Transformers capture long-range temporal dependencies but do not explicitly model frequency-domain pattern occurrences across time cycles.
- FECAM analyzes forecasting from frequency properties and is designed as a general block applicable to multiple models with few operations.
- Real-world time series contain rich frequency information, with most dataset energy concentrated in low-frequency ranges.
- Fourier sums can overshoot at jump discontinuities, and the resulting Gibbs phenomenon introduces high-frequency noise.
3 FECAM: FREQUENCY ENHANCED CHANNEL ATTENTION MECHANISM
FECAM enriches channel attention by modeling time-series information in the frequency domain with DCT, avoiding Gibbs-related high-frequency noise and incorporating multiple frequency components. The mechanism can be integrated into forecasting architectures to learn channel–frequency interactions and enhance extracted features.
- Motivation: GAP captures only the lowest DCT frequency component, so relying solely on temporal average pooling can inadequately extract time-series information.The paper identifies this limitation as potential information loss and motivates adding further frequency components.
- Frequency representation: DCT uses symmetric periodic extension, avoiding the discontinuities associated with DFT and thereby intrinsically mitigating Gibbs phenomenon and high-frequency components.The paper also states that DCT has more efficient energy compaction than Fourier Transform.
- Frequency representation: FECAM splits feature maps into channel subgroups, applies corresponding DCT frequency components from low to high frequency, and stacks the resulting frequency vectors.Each channel in a subgroup is processed by the same frequency component, producing a frequency channel vector.
- Channel attention: The resulting frequency vector is passed through an SE-like neural structure to learn attention weights that model channel interdependencies.The learned frequency-domain attention represents V ∈ R^(C×L), and the mechanism can be added to existing forecasting models or used with a projection layer.
- Channel attention: By interacting channel features with frequency components, FECAM learns channel and frequency-pair importance and encourages more diverse extracted features.The paper presents this interaction as comprehensive temporal-information acquisition from the frequency domain.
4 EXPERIMENTS
Experiments evaluate FECAM across six real-world forecasting benchmarks and as a plug-in for Transformer and recurrent architectures. FECAM achieves strong forecasting results, improves existing models with few added parameters, and yields larger gains on datasets with more low-frequency energy.
- Experimental Setup: Experiments evaluate FECAM on six real-world time-series forecasting benchmarks using chronological train, validation, and test splits.The split ratio is 3:1:1 for ETT and 7:2:2 for the other datasets.
- Datasets: The benchmark suite includes Electricity, ETT, Exchange, ILI, Traffic, and Weather datasets covering consumption, transformer loads, exchange rates, illness, road occupancy, and meteorology.The datasets differ in sampling frequency, dimensionality, and application domain.
- Main Results: FECAM achieves state-of-the-art multivariate forecasting performance across all benchmarks and prediction lengths, while using fewer parameters than Autoformer.Compared with Autoformer, FECAM reports a 21.52% relative MSE reduction and 10.78% relative MAE reduction.
- Main Results: 36.40% relative MSE reduction is achieved on Exchange for prediction lengths 96, 192, 336, and 720 compared with previous state-of-the-art results.The authors also report a 24.9% reduction on ILI for the corresponding comparison.
- Module Generality: FECAM improves LSTM, Reformer, Informer, Autoformer, and Transformer by averaged promotions of 35.99%, 10.01%, 8.71%, 8.29%, and 8.06%, respectively.The module adds only a few parameters and is inserted into Transformer and LSTM architectures.
- Analysis: Module gains are large on Exchange, ETTm2, and Weather but small on Traffic, which the authors associate with differences in low-frequency energy.The authors report more low-frequency energy in the datasets with larger gains.
- Interpretability and Qualitative Results: Visualizations show FECAM extracting distinct channel and frequency importance patterns, while FECAM-equipped Transformer and LSTM models improve qualitative predictions over vanilla versions.The DCT experiment also reports more efficient energy compaction than the Discrete Fourier Transform for signals concentrated at low frequencies.
5 CONCLUSION
The paper frames time-series forecasting as a frequency-domain modeling problem and proposes FECAM, a DCT-based channel-attention mechanism. It reports state-of-the-art results on six benchmarks and flexible integration into Transformer and RNN models.
- 5 CONCLUSION: FECAM models time series in the frequency domain using Discrete Cosine Transform-based channel attention to avoid Gibbs phenomenon from Fourier-based extraction.The method assigns channel weights and learns the importance of different frequencies for each channel.
- 5 CONCLUSION: FECAM achieves state-of-the-art performance on six real-world benchmarks and can be added to Transformer and RNN models with a few lines of code.The conclusion emphasizes both generality and forecasting performance.