An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention

Yehjin Shin; Jeongwhan Choi; Hyowon Wi; Noseong Park

Source-linked AI summary

An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention

Yehjin Shin, Jeongwhan Choi, Hyowon Wi, Noseong Park

arXiv:2312.10325v2Dec 16, 2023cs.LGcs.AIcs.IR

TL;DR

Transformer-based sequential recommendation can suffer from insufficient inductive bias and self-attention-induced oversmoothing. BSARec combines Fourier-based attentive inductive bias with self-attention, and outperforms seven baselines across six benchmark datasets.

Problem
Transformer-based sequential recommendation is limited by insufficient inductive bias and self-attention’s low-pass filtering, which causes oversmoothing and loss of detailed preferences.
Method
BSARec combines Fourier-based attentive inductive bias with self-attention and integrates low- and high-frequency information to mitigate oversmoothing.
Results
BSARec outperforms seven baseline methods across six benchmark datasets in recommendation performance.
Takeaways & Limitations
The findings identify oversmoothing in sequential recommendation self-attention and support combining attentive inductive bias with self-attention.
Takeaways & Limitations
The attentive inductive bias assumes a ring graph in which an item is influenced by its previous item.

Abstract

from arXiv · show

Sequential recommendation (SR) models based on Transformers have achieved remarkable successes. The self-attention mechanism of Transformers for computer vision and natural language processing suffers from the oversmoothing problem, i.e., hidden representations becoming similar to tokens. In the SR domain, we, for the first time, show that the same problem occurs. We present pioneering investigations that reveal the low-pass filtering nature of self-attention in the SR, which causes oversmoothing. To this end, we propose a novel method called $\textbf{B}$eyond $\textbf{S}$elf-$\textbf{A}$ttention for Sequential $\textbf{Rec}$ommendation (BSARec), which leverages the Fourier transform to i) inject an inductive bias by considering fine-grained sequential patterns and ii) integrate low and high-frequency information to mitigate oversmoothing. Our discovery shows significant advancements in the SR domain and is expected to bridge the gap for existing Transformer-based SR models. We test our proposed approach through extensive experiments on 6 benchmark datasets. The experimental results demonstrate that our model outperforms 7 baseline methods in terms of recommendation performance. Our code is available at https://github.com/yehjin-shin/BSARec.

Introduction

The introduction identifies insufficient inductive bias and self-attention’s low-pass filtering, which causes oversmoothing and obscures important sequential patterns in Transformer-based sequential recommendation. It presents BSARec, which uses Fourier-based frequency information and a high-pass frequency rescaler, and reports evaluation on 6 datasets against 7 baselines.

Motivation: Transformer-based sequential recommendation models face insufficient inductive bias and self-attention’s low-pass filtering, which causes oversmoothing.Oversmoothing can smooth important embedding patterns and hinder modeling temporal dynamics and accurate prediction.
Method: BSARec injects frequency-based inductive bias through the Fourier transform to capture patterns and periodicity overlooked by self-attention.The method combines Fourier-transform frequency information with self-attention.
Method: BSARec’s frequency rescaler applies high-pass filters while preserving access to both short-term high-frequency and long-term low-frequency behavioral patterns.High-frequency signals represent abrupt short-term interest changes, whereas low-frequency signals represent persistent long-term interests.
Evaluation: 6 benchmark datasets and 7 baseline methods are used to evaluate BSARec, which consistently outperforms the baselines in recommendation performance.Additional experiments examine the approach’s necessity, oversmoothing mitigation, recommendation accuracy, and generalization capabilities.

Preliminaries

Sequential recommendation predicts a user’s next interaction from chronologically ordered item sequences and recommends a Top-k candidate list. The preliminaries also define self-attention and Fourier-domain representations, including low- and high-frequency components associated with sequential signals.

Sequential Recommendation: Sequential recommendation sorts each user’s interacted items chronologically and predicts potential next items as a Top-k recommendation list.The sequence contains the user’s historical item interactions, with each position denoting an interacted item.
Self-Attention: Self-attention estimates correlations among sequence items through scaled dot products, while SASRec also uses feed-forward, residual, and layer-normalization components.The proposed method uses the self-attention matrix and adds an inductive bias to balance the two methods.
Frequency Domain: The Discrete Fourier Transform projects sequence values into the frequency domain, with the spectrum represented as s_x = F x and reconstructed using the inverse transform.The DFT is represented as a mapping from real-valued sequences to complex-valued frequency representations.
Frequency Domain: Low-frequency components contain the c lowest spectral elements, whereas high-frequency components contain the remaining N−c elements.Both components are mapped back to real-valued sequence signals using Fourier bases and the inverse transform.
Graph Fourier Transform: For sequential recommendation, low-frequency information corresponds to slowly changing signals across item embeddings and represents a user’s long-standing interest.The DFT is a special case of the Graph Fourier Transform on a ring graph with N nodes.

Motivation

The paper shows that self-attention in Transformer-based sequential recommendation acts as a low-pass filter that erases high-frequency information and causes oversmoothing. It motivates using high-pass filtering and inductive bias to preserve detailed preferences and capture short-term user behavior patterns.

Spectral analysis: Self-attention’s spectrum concentrates in the low-frequency region, revealing that it continuously erases high-frequency information as a low-pass filter.The paper also provides theoretical justification for this filtering behavior.
Theoretical justification: Theorem 1 states that self-attention inherently acts as a low-pass filter for all inputs.The result is independent of the input key and query matrices and follows from the Perron-Frobenius theorem.
Theoretical justification: Repeatedly applying self-attention causes outputs to lose all feature expressiveness as the number of layers approaches infinity.This follows from the attention matrix’s low-pass filtering behavior under successive application.
Oversmoothing: As Transformer-based sequential recommendation models deepen, cosine similarity increases and singular values decay rapidly, producing oversmoothing and performance degradation.The resulting loss of feature representation prevents models from capturing users’ detailed preferences.
Method motivation: The proposed motivation is to use a high-pass filter and inductive bias to alleviate oversmoothing and capture short-term user behavior patterns.This targets the detailed preferences lost through self-attention’s low-pass behavior.

Proposed Method

BSARec augments self-attention with a frequency-rescaled attentive inductive bias that combines low- and high-frequency information to mitigate oversmoothing. Its BSA encoder stacks this mechanism with self-attention and feed-forward components, then predicts the next item using a similarity score optimized with cross-entropy loss.

BSA Encoder: BSARec’s encoder contains BSA layers with three modules: a BSA layer, an attentive inductive bias with frequency rescaler, and a feed-forward network.The architecture uses stacked BSA blocks built on the embedding layer.
Attentive Inductive Bias: The discrete Fourier transform supplies a universal sequential inductive bias, while α controls the trade-off between this bias and trainable self-attention.The bias corresponds to influence from an item’s previous item under a ring-graph interpretation.
Attentive Inductive Bias: The frequency rescaler separates low- and high-frequency components, with trainable β scaling the high-pass filter.β may be either a D-dimensional vector or a scalar parameter.
Attentive Inductive Bias: Selective use of high-pass information prevents oversmoothing, whereas relying only on low-frequency components cannot prevent it.Low-frequency components capture common signals, while high-frequency components capture locally fluctuating signals.
BSA Layer: BSARec combines the learned self-attention matrix with the attentive inductive bias so self-attention can capture non-obvious attentions.Separating the two components lets the self-attention mechanism focus on attentions not represented by the inductive bias.
Prediction and Optimization: The model computes item preference by a dot product between the final sequence representation and an item embedding, then optimizes next-item prediction with cross-entropy loss.The task treats next-item prediction as classification over the whole item set.

Experiments

Across six sequential-recommendation datasets, BSARec achieves the best accuracy across all reported datasets and metrics, outperforming established sequential and contrastive-learning baselines. Ablations, sensitivity analyses, case studies, and runtime comparisons support the contributions of its attentive inductive bias and high-frequency modeling with only marginal parameter overhead.

Recommendation performance: BSARec records the best performance across all datasets and metrics, including a 27.49% HR@10 improvement on LastFM, without contrastive learning.The baselines include RNN/CNN, Transformer, and contrastive-learning sequential recommenders.
Ablation studies: On Beauty, the AIB-only ablation reaches HR@20 of 0.1338 versus 0.1265 for self-attention-only, while full BSARec performs best.The result indicates that both self-attention and the attentive inductive bias are required for best accuracy.
Sensitivity analysis: The preferred α and c values vary by dataset: Beauty favors larger α and c=5, whereas ML-1M achieves best accuracy at α=0.3 and improves as c increases.These results show that the trade-off between self-attention and inductive bias is dataset-dependent.
Learned β visualization: Learned β assigns more weight to high-frequency information in the first layer than the second, with especially high weights on LastFM and Beauty.This supports emphasizing high-frequency information earlier in the model.
Case study: For LastFM user u322, only BSARec recommends a pop artist after a rock-heavy history, capturing an abrupt change in preference.The case study illustrates BSARec’s ability to model high-frequency signals corresponding to sudden interaction changes.

Related Work

Related work in sequential recommendation spans Markov-chain, convolutional, recurrent, self-attention, contrastive-learning, and filter-enhanced MLP approaches. Oversmoothing was first studied in graph research and has also been observed in Transformers.

Sequential Recommendation: FPMC models item-item transitions with Markov Chains, while Fossil extends this approach to higher-order item transitions.These methods target next-item recommendation from sequential patterns in users’ historical interactions.
Sequential Recommendation: Caser represents the sequence-item embedding matrix as an image and applies convolutional operators for sequence modeling.
Neural Sequential Recommendation: GRU4Rec introduces GRUs, while SASRec and BERT4Rec demonstrate the efficacy of self-attention in sequential recommendation.Transformer-based successes motivated further exploration of self-attention for sequential recommendation.
Recent Sequential Recommendation: Recent work uses contrastive learning and filter-enhanced MLPs, including DuoRec’s augmentations and semantic positives and FMLPRec’s global filter.
Oversmoothing: Oversmoothing originated in graph research and was later observed in Transformers, where early work attributed it empirically to attention collapse.In graph neural networks, repeated neighbor message exchange can make representations converge to a constant as depth increases.

Conclusion

The paper identifies insufficient inductive bias and low-pass filtering in Transformer self-attention as limitations for sequential recommendation, including oversmoothing. It introduces BSARec to address these issues using attentive inductive bias alongside vanilla self-attention.

Conclusion: Transformer self-attention in sequential recommendation suffers from insufficient inductive bias and low-pass filtering properties.The paper frames these as limitations of Transformer-based sequential recommendation.
Conclusion: The paper reveals that low-pass filtering causes oversmoothing in sequential recommendation.This extends the oversmoothing problem to the sequential recommendation setting.
Conclusion: BSARec addresses these limitations by combining attentive inductive bias with vanilla self-attention.The proposed method is presented as a response to the identified self-attention limitations.

More Preliminaries on Fourier Transform

This section introduces the discrete Fourier transform (DFT) used in the paper, including its frequency-domain representation and matrix form. It also explains the inverse DFT’s conjugate-transpose relationship and unitary normalization.

DFT background: The paper uses the discrete Fourier transform to convert real-valued discrete time-domain signals into complex frequency-domain representations.The DFT is defined on real-valued inputs and is motivated by applications in signal processing.
DFT background: The DFT can be represented as a matrix whose rows are Fourier basis vectors and applied to signals through matrix multiplication.The basis vectors and their matrix arrangement provide the transformation’s explicit representation.
Inverse DFT: The inverse DFT uses the DFT matrix’s conjugate transpose, scaled by 1/√N, making the transformation unitary and norm-preserving.This normalization preserves signal energy across the time and frequency domains and links the DFT and IDFT without a separate transformation matrix.

Definition of a Low-Pass Filter

The paper defines a low-pass filter as a transformation that preserves low-frequency components while reducing high-frequency components, with recurrent frequency attenuation providing the formal criterion.

Definition of a Low-Pass Filter: A low-pass filter preserves low-frequency components while reducing high-frequency components.This definition characterizes the filter by its selective frequency behavior.
Definition of a Low-Pass Filter: A transformation is a low-pass filter if repeated application causes DFT high-frequency components to diminish relative to low-frequency components over time.The criterion evaluates frequency behavior after recurrently applying the transformation to any input.

Proof of Theorem 1

Theorem 1 establishes that the softmax-defined matrix A inherently acts as a low-pass filter. Its proof begins from A’s stochastic structure and analyzes repeated application through its Jordan canonical form.

Theorem statement: A inherently acts as a low-pass filter for all x ∈ R^N.This is the central claim of Theorem 1.
Proof setup: The proof uses that A has non-negative entries and each row sums to unity.These properties follow from A being defined by the softmax function.
Proof setup: The argument analyzes how x evolves under repeated application of A.This repeated-application perspective is introduced as the proof’s analytical objective.
Jordan-form analysis: A’s Jordan canonical form J is introduced through a similarity transformation represented by P.The proof denotes the Jordan canonical form of A as J and the similarity transformation matrix as P.

A “ PJP´1, (16) … Performance under Different Setting

The paper establishes that repeated application of the transformation is low-pass because the dominant eigenvalue suppresses high-frequency components, then evaluates BSARec across datasets, baselines, settings, and analyses. Results show that combining self-attention with the attentive inductive bias improves accuracy, while sensitivity, ablation, complexity, and alternative evaluation results clarify its behavior and efficiency.

A “ PJP´1, (16): The dominant eigenvalue λ1 governs repeated applications of A, causing high-frequency components to attenuate faster than the primary low-frequency component.This behavior makes A a low-pass filter regardless of the specific input matrices Q and K.
Details of Datasets: Experiments use 6 benchmark datasets, including sparse, short-sequence Amazon subsets, Yelp3 transactions after January 1st, 2019, and ML-1M movie recommendations.The datasets comprise Beauty, Sports, Toys, Yelp, LastFM, and ML-1M.
Details of Baselines: Baselines span RNN/CNN-based sequential models and Transformer-based sequential models, including GRU4Rec, Caser, SASRec, and BERT4Rec.SASRec is described as the first sequential recommender based on self-attention and a popular baseline.
Experimental Settings & Hyperparameters: Experiments use Ubuntu 18.04 LTS, Python 3.9.7, PyTorch 1.8.1, NumPy 1.24.3, SciPy 1.11.1, CUDA 11.1, an NVIDIA RTX 3090, and tuned α, c, head-count, and Adam learning-rate ranges.The tested ranges are α in {0.1, 0.3, 0.5, 0.7, 0.9}, c in {1, 3, 5, 7, 9}, heads in {1, 2, 4}, and learning rates in {5 × 10^-4, 1 × 10^-3}.
Ablation Studies: BSARec using both A and AIB outperforms using either component alone, while a learnable vector β for rescaling high frequencies performs better than a scalar β.The ablation study reports these findings across all datasets.
Sensitivity Analyses: Sensitivity to α and c varies by dataset: larger α is preferred for Beauty, Toys, Yelp, and LastFM, while optimal c is 5 for Beauty and Sports and 3 for Toys, Yelp, and LastFM.For ML-1M, the best accuracy favors a larger c; the results indicate dataset-dependent balances and frequency-component selections.
Model Complexity and Runtime Analyses: BSARec slightly increases total parameters, can add only 258 parameters with scalar β compared with SASRec, DuoRec, and FEARec, and runs faster per epoch than FEARec and DuoRec.The reported comparisons evaluate parameter counts and runtime across all datasets.
Performance under Different Setting: Under evaluation with 99 randomly sampled negative items, BSARec performs much better than SASRec and FMLP-Rec in most cases across Beauty, Sports, Toys, Yelp, LastFM, and ML-1M.This alternative strategy follows the evaluation approach used by SASRec and FMLP-Rec and produces tendencies similar to the main comparison.