Source-linked AI summary
Frequency Enhanced Hybrid Attention Network for Sequential Recommendation
Xinyu Du, Huanhuan Yuan, Pengpeng Zhao, Jianfeng Qu, Fuzhen Zhuang, Guanfeng Liu, Victor S. Sheng
TL;DR
Self-attention-based sequential recommenders often lose high-frequency information and miss periodic behavior patterns. FEARec combines frequency-ramped time attention, frequency-domain autocorrelation attention, contrastive learning, and frequency regularization; experiments on four benchmark datasets report superiority over state-of-the-art models.
Problem
Self-attention-based sequential recommenders capture low-frequency global interactions but inadequately model high-frequency behavior and hidden periodic patterns.
Method
FEARec combines frequency-ramped time-domain attention with autocorrelation-based frequency-domain attention, supplemented by contrastive learning and frequency regularization.
Results
FEARec shows superiority over state-of-the-art models on four public benchmark datasets.
Takeaways & Limitations
Modeling multiple frequency ranges and periodic characteristics provides the basis for FEARec’s hybrid time- and frequency-domain recommendation approach.
Takeaways & Limitations
After embedding, distinct item IDs may map to the same feature value, so item identity is not guaranteed to remain unique in the representation.
Abstract
from arXiv · showhide
The self-attention mechanism, which equips with a strong capability of modeling long-range dependencies, is one of the extensively used techniques in the sequential recommendation field. However, many recent studies represent that current self-attention based models are low-pass filters and are inadequate to capture high-frequency information. Furthermore, since the items in the user behaviors are intertwined with each other, these models are incomplete to distinguish the inherent periodicity obscured in the time domain. In this work, we shift the perspective to the frequency domain, and propose a novel Frequency Enhanced Hybrid Attention Network for Sequential Recommendation, namely FEARec. In this model, we firstly improve the original time domain self-attention in the frequency domain with a ramp structure to make both low-frequency and high-frequency information could be explicitly learned in our approach. Moreover, we additionally design a similar attention mechanism via auto-correlation in the frequency domain to capture the periodic characteristics and fuse the time and frequency level attention in a union model. Finally, both contrastive learning and frequency regularization are utilized to ensure that multiple views are aligned in both the time domain and frequency domain. Extensive experiments conducted on four widely used benchmark datasets demonstrate that the proposed model performs significantly better than the state-of-the-art approaches.
1 INTRODUCTION
FEARec addresses self-attention’s loss of high-frequency information and difficulty distinguishing periodic behavior by combining frequency-enhanced time attention with frequency-domain autocorrelation.
- Motivation: Self-attention captures low-frequency global preference signals but inadequately models high-frequency interactions occurring within short periods.The paper characterizes Transformer-based self-attention as a low-pass filter.
- Motivation: Users’ online behaviors exhibit periodic trends that are difficult to identify from overall time-domain item attention.The paper motivates frequency-domain analysis as a way to uncover periodic patterns hidden in historical sequences.
- Proposed Approach: FEARec uses an adaptive frequency ramp so different time-attention layers learn distinct frequency ranges, including both low- and high-frequency information.After the discrete Fourier transform, each layer selects a frequency component as its input feature.
- Proposed Approach: Its frequency-domain attention computes autocorrelation across time-delay sequences to discover period-based dependencies and combines this mechanism with time-domain attention.The hybrid design unifies item-level and sequence-level modeling.
- Contributions: The model also introduces contrastive learning and frequency-domain regularization, while experiments on four public datasets report superiority over state-of-the-art baselines.These auxiliary objectives are intended to align representations across views and domains.
2 RELATED WORK
Sequential recommendation developed from Markov and recurrent models toward self-attention architectures, with auxiliary contrastive objectives becoming another training strategy. Fourier-based modeling is established elsewhere but remains relatively limited in sequential recommendation.
- Sequential Recommendation: Early sequential recommenders commonly used Markov-chain assumptions, followed by deep models such as GRU4Rec and Caser.These approaches model future items through item-transition correlations.
- Sequential Recommendation: Self-attention models including SASRec, BERT4Rec, and S3Rec became prominent for learning user-preference representations.Many subsequent methods retained next-item supervised training, while some added auxiliary tasks.
- Sequential Recommendation: Contrastive approaches such as CL4SRec, CoSeRec, and DuoRec use multiple views or augmentations to improve sequence representations.Their training strategies supplement recommendation objectives with contrastive learning.
- Fourier-based Modeling: Fourier transforms have been widely applied in signal processing, vision, language, forecasting, and differential-equation solving, but relatively few works use them for sequential recommendation.FMLP-Rec is identified as an early Fourier-related sequential recommendation method.
3 PROPOSED METHOD
FEARec transforms sequential representations into the frequency domain and combines frequency-aware time attention with autocorrelation-based frequency attention. Its frequency ramp assigns layers different spectral ranges, while auxiliary regularization supports representations across both domains.
- Overall Architecture: FEARec applies FFT to item-sequence embeddings before hybrid attention captures attention scores and periodic behavior across frequency bands.The model then applies contrastive learning and frequency-domain loss to improve representations in both time and frequency domains.
- 3.2.1 Frequency Ramp Structure: The frequency ramp samples different frequency ranges across layers, retaining all components when α=1 and using average sampling when α≤1.Layer positions determine sampled indexes, with sampling directed from high frequencies toward low frequencies.
- 3.2.2 Time Domain Self-Attention Layer: Ramp-based time attention learns low-frequency information in top layers and high-frequency information in bottom layers, improving local-behavior modeling.Sampled frequency components are zero-padded before time-domain attention computes attention weights.
- 3.2.3 Frequency Domain Attention Layer: Frequency-domain attention uses autocorrelation to identify related time-delay sequences and aggregate periodic dependencies at the subsequence level.The method selects the most related time lags using Topk autocorrelation before time-delay aggregation.
- 3.2.3 Frequency Domain Attention Layer: The hybrid layer combines time-domain and frequency-domain attention, while a two-layer GELU MLP supplies nonlinearity after the attention operations.The two attention modules remain linear and therefore do not by themselves model complex nonlinear relations.
3.3 Prediction Layer
The prediction layer converts the final sequence representation into item-relevance probabilities and jointly trains recommendation with dual-domain auxiliary regularization. Contrastive learning aligns augmented user views, while frequency regularization aligns their spectra.
- 3.3 Prediction Layer: FEARec multiplies the final behavior-sequence representation by the item embedding matrix and applies softmax to predict candidate-item relevance probabilities.The recommendation objective uses cross-entropy so the user-adopted item receives a higher predicted score.
- Auxiliary Training: Multi-task training jointly optimizes the main recommendation loss with contrastive and frequency-domain regularization.The scale weight λ controls the strength of contrastive regularization.
- 3.4.1 Contrastive Learning: Dropout-based augmentations produce two encoder views, while frequency features are mixed to construct harder positive samples.This replaces typical item cropping, masking, and reordering augmentations because data-level changes may not guarantee semantic similarity.
- Frequency Regularization: Frequency-domain L1 regularization encourages augmented views with similar time-domain semantics to have similar frequency spectra.It complements time-domain contrastive learning by enriching spectrum regularization.
4 EXPERIMENT
FEARec is evaluated against representative sequential-recommendation baselines on four benchmark datasets using full-item ranking and HR/NDCG metrics. Experiments report state-of-the-art overall performance and show that frequency-ramp sampling, hybrid attention, contrastive learning, frequency regularization, and dataset-dependent module weighting affect results.
- Experimental Settings: Experiments use Beauty, Clothing, Sports2, and MovieLens-1M, spanning sparse short sequences and dense long sequences.Users and items with fewer than five interactions are removed during preprocessing.
- Experimental Settings: Evaluation uses leave-one-out prediction, ranks over the whole item set without negative sampling, and reports HR@5, HR@10, NDCG@5, and NDCG@10.HR denotes Hit Rate and NDCG denotes Normalized Discounted Cumulative Gain.
- Overall Performance: FEARec achieves the state-of-the-art result among all baseline models and outperforms competing methods across sparse and dense datasets on all reported metrics.The comparison includes non-sequential, standard sequential, and contrastive-learning sequential recommenders.
- Ablation Study: Removing frequency-ramp sampling from either time-domain or frequency-domain attention significantly reduces performance.The reported layerwise pattern captures high-frequency information in lower layers and progressively captures low-frequency information.
- Ablation Study: Removing either time-domain or frequency-domain attention decreases performance, reflecting item-level and sub-sequence-level preference modeling roles.Time-domain attention models item-level scores, while frequency-domain attention uses autocorrelation to identify periodic characteristics.
- Ablation Study: Combining contrastive learning with frequency-domain regularization improves performance, whereas frequency regularization alone produces poor results.The regularizer reduces spectral distance between two augmented perspectives.
- Hyper-parameter Sensitivity: Frequency-domain attention receives greater weight for the best results on Beauty, Clothing, and Sports, while time-domain attention is more important on dense ML-1M.The hybrid ratio controls the mixture of time-domain and frequency-domain attention.
- Hyper-parameter Sensitivity: Increasing the autocorrelation aggregation parameter improves performance up to a point, after which including irrelevant delayed sequences degrades it.The parameter determines how many highly autocorrelated time-delay sequences are aggregated.
5 CONCLUSION
FEARec combines frequency-enhanced time-domain attention with frequency-domain autocorrelation to model both frequency information and periodic dependencies. The model also aligns multiple views and outperforms state-of-the-art models in experiments across four datasets.
- Model overview: FEARec builds a hybrid attention framework spanning time and frequency domains.Its improved time-domain attention learns low- and high-frequency information, while frequency-domain attention models dependencies through autocorrelation.
- Model overview: The frequency ramp structure enables time-domain attention to learn both low- and high-frequency information.
- Model overview: Frequency-domain attention calculates autocorrelation across time delays to explore inherent dependencies in sequential behaviors.
- Training objectives: Contrastive learning and frequency-domain regularization align multiple views in the model.
- Empirical evaluation: FEARec shows superiority over all state-of-the-art models in experiments on four public datasets.
A.1.1 Discrete Fourier Transform.
The one-dimensional DFT converts finite sequential data into complex frequency-domain components and provides a one-to-one mapping with the original time-domain representation.
- Definition: The one-dimensional DFT converts a finite sequence into complex numbers in the frequency domain.Each complex value represents a signal component at a corresponding frequency.
- Interpretation: DFT decomposes a sequence of values into components with different frequencies.
- Interpretation: The DFT is a one-to-one unique mapping between time-domain and frequency-domain representations.
- Inverse mapping: The original feature-domain sequence can be recovered from frequency representations using the inverse DFT.
- Properties: For real inputs, the DFT has conjugate symmetry.
A.1.2 Fast Fourier Transform.
FFT provides an efficient way to compute the DFT for sequential data, reducing its computational complexity from quadratic to quasi-linear time.
- Efficiency: IFFT efficiently computes time-domain features corresponding to the original sequence.The paper denotes FFT and IFFT by F and F^-1, respectively.
A.2 Wiener-Khinchin Theorem
The Wiener-Khinchin theorem enables autocorrelation to be computed in the frequency domain, making all time-delay similarities available efficiently for sequential recommendation.
- Autocorrelation: Autocorrelation measures time-delay similarity between a sequence and its lagged version.This property is useful for uncovering trends and patterns in user behavior, which can exhibit periodic tendencies.
- Frequency-domain computation: The Wiener-Khinchin theorem computes time-domain autocorrelation from frequency-domain quantities using FFT.
- Efficiency: FFT computes autocorrelation for all lags simultaneously.This gives autocorrelation an O(N log N) complexity.