PaperlayerView exact revision on arXiv

Source-linked AI summary

Filter-enhanced MLP is All You Need for Sequential Recommendation

Kun Zhou, Hui Yu, Wayne Xin Zhao, Ji-Rong Wen

arXiv:2202.13556v1cs.IRcs.AI

TL;DR

Sequential recommendation models must learn evolving preferences from logged behavior, although those logs contain noise and deep models can overfit them. The paper studies frequency-domain filtering and proposes FMLP-Rec, an all-MLP model with learnable filters. Filtering improves representative models, and FMLP-Rec outperforms RNN, CNN, GNN, and Transformer baselines on eight real-world datasets.

  • Problem

    Noisy logged behavior can cause deep sequential recommendation models, especially self-attention models, to overfit while modeling evolving user preferences.

  • Method

    FMLP-Rec removes self-attention, stacks MLP blocks, and uses learnable FFT-domain filters with inverse FFT reconstruction to attenuate noise.

  • Results

    FMLP-Rec outperforms state-of-the-art RNN, CNN, GNN, and Transformer-based baselines on eight real-world datasets.

  • Takeaways & Limitations

    Filtering algorithms can improve deep sequential recommendation models, while an all-MLP architecture with learnable filters is effective and efficient.

  • Takeaways & Limitations

    FMLP-Rec reduces potential noise influence in item representations rather than removing noisy items, whose ground-truth labels are usually unavailable.

Abstract

from arXiv · show

Recently, deep neural networks such as RNN, CNN and Transformer have been applied in the task of sequential recommendation, which aims to capture the dynamic preference characteristics from logged user behavior data for accurate recommendation. However, in online platforms, logged user behavior data is inevitable to contain noise, and deep recommendation models are easy to overfit on these logged data. To tackle this problem, we borrow the idea of filtering algorithms from signal processing that attenuates the noise in the frequency domain. In our empirical experiments, we find that filtering algorithms can substantially improve representative sequential recommendation models, and integrating simple filtering algorithms (eg Band-Stop Filter) with an all-MLP architecture can even outperform competitive Transformer-based models. Motivated by it, we propose \textbf{FMLP-Rec}, an all-MLP model with learnable filters for sequential recommendation task. The all-MLP architecture endows our model with lower time complexity, and the learnable filters can adaptively attenuate the noise information in the frequency domain. Extensive experiments conducted on eight real-world datasets demonstrate the superiority of our proposed method over competitive RNN, CNN, GNN and Transformer-based methods. Our code and data are publicly available at the link: \textcolor{blue}{\url{https://github.com/RUCAIBox/FMLP-Rec}}.

1 INTRODUCTION

Sequential recommendation models aim to capture evolving user behavior, but noisy logged data and over-parameterized self-attention can promote overfitting. FMLP-Rec addresses these issues with learnable frequency-domain filters in an all-MLP architecture and outperforms diverse baseline families.

  • Sequential recommendation captures dynamic user behavior to predict appropriate items over large item pools.
  • Transformer-based recommenders can overfit noisy logged behavior because self-attention attends to all sequence items.Stacked self-attention layers also involve many parameters, potentially producing over-parameterized architectures.
  • Filtering algorithms substantially improve deep sequential recommendation models, including Transformer-based SASRec.The experiments denoise item embeddings with three classical filtering algorithms.
  • FMLP-Rec removes self-attention and uses MLP blocks with FFT-based learnable filters to attenuate noise in item representations.The filters are optimized from raw data and can extract features across frequencies, including long- and short-term interactions.
  • Experiments on eight real-world datasets show FMLP-Rec outperforming state-of-the-art RNN, CNN, GNN, and Transformer baselines.

2 PRELIMINARIES

Sequential recommendation predicts a user's next interaction from an ordered item sequence. The paper introduces Fourier-domain filtering as a way to reduce noisy features, using FFT/IFFT operations whose efficient computation scales as O(Nlog N).

  • 2.1 Problem Statement: Sequential recommendation predicts the next item a user is likely to interact with from a chronological interaction sequence.
  • 2.1 Problem Statement: The method reduces noisy item features rather than learning to remove noisy items, because noisy-item labels are usually unavailable.
  • 2.2 Fourier Transform: The discrete Fourier transform converts a sequence into frequency-domain representations, with each frequency component combining the original input sequence.
  • 2.2 Fourier Transform: Fast Fourier Transform computes the discrete Fourier transform in O(Nlog N), while inverse FFT efficiently reconstructs the sequence.
  • 2.2 Fourier Transform: FFT exposes periodic characteristics in the frequency domain, enabling filtering algorithms such as low-pass filters to attenuate noisy signals.

3 EMPIRICAL ANALYSIS WITH FILTERING ALGORITHMS FOR RECOMMENDATION

The empirical analysis tests classical frequency-domain filters on representative sequential recommenders and on all-MLP variants derived from SASRec. Filtering generally improves these models, while suitable filters can let lightweight all-MLP variants outperform the Transformer-based SASRec.

  • Analysis Setup: The study evaluates filtering algorithms on GRU4Rec and SASRec using Amazon Beauty and Sports datasets.Filters are inserted between the embedding and sequence encoder layers without changing other components.
  • Filtering Algorithms: Each filter processes item-embedding features as FFT, filtering, then inverse FFT before sequence encoding.The evaluated filters are HPF, LPF, and BSF.
  • Results and Findings: LPF and BSF consistently improve representative models, with LPF best for GRU4Rec and BSF best for SASRec; HPF usually degrades performance.These results indicate that the most suitable filter depends on the sequential recommendation model.
  • Results and Findings: The item embedding matrix likely contains noise that affects sequential recommendation performance, and appropriate filtering can alleviate this problem.The analysis also reports that low-frequency information seems more important for sequential recommendation.
  • All-MLP Models: All-MLP SASRec variants remove multi-head self-attention and retain a filter layer, so they model item sequences using only MLPs.Most variants remain effective after self-attention removal.
  • All-MLP Models: The LPF all-MLP variant outperforms SASRec by a large margin, suggesting proper filtering can enable simple all-MLP models to surpass complex Transformer-based models.Removing noise information and self-attention also makes the model more lightweight and reduces overfitting risk, according to the analysis.

4 METHOD

FMLP-Rec replaces Transformer self-attention with stacked MLP blocks containing learnable frequency-domain filters. These filters attenuate noise while retaining sequence-level modeling through an equivalent circular convolution, with O(n log n) time complexity and Transformer-sized receptive field.

  • FMLP-Rec architecture: FMLP-Rec replaces Transformer multi-head self-attention with stacked MLP blocks and learnable filter structures for sequential recommendation.The architecture is designed to automatically learn suitable filters across recommendation scenarios.
  • Learnable filter-enhanced blocks: Each filter block applies FFT along the item dimension, modulates the complex spectrum with a learnable filter, and uses inverse FFT to recover sequence representations.Skip connections, layer normalization, and dropout follow the filtering operation.
  • Learnable filter-enhanced blocks: The point-wise feed-forward sublayer uses MLP and ReLU functions to capture nonlinear characteristics before producing the layer output.Trainable weights and biases parameterize the feed-forward transformation.
  • Theoretical analysis: The frequency-domain filter is equivalent to circular convolution in the time domain, allowing each feature dimension to use a learnable convolution kernel.Circular convolution provides a whole-sequence receptive field and can capture periodic patterns.
  • Complexity and receptive field: FMLP-Rec has O(n log n) time complexity and the same receptive field as Transformer self-attention.FFT and inverse FFT cost O(n log n), point-wise feed-forward networks cost O(n), and the circular convolution spans the whole sequence.

5 EXPERIMENT

Experiments cover eight real-world datasets spanning transaction records and click sessions, using ranking metrics with sampled negatives. FMLP-Rec performs consistently better than the baselines by a large margin on most datasets, while sequential methods generally outperform non-sequential methods.

  • Datasets: Eight datasets cover Amazon Beauty, Sports, and Toys, Yelp, Nowplaying, RetailRocket, Tmall, and Yoochoose across transaction and session-based recommendation.Transaction records are used for four datasets, while the remaining four contain click sessions.
  • Datasets: The preprocessing groups interactions by users or sessions, sorts them by ascending timestamps, and removes unpopular items and users with fewer than five interactions.This procedure is applied across the datasets, with transaction records distinguished from click sessions.
  • Evaluation: Evaluation uses HR@{1, 5, 10}, NDCG@{5, 10}, and MRR, pairing each ground-truth item with 99 randomly sampled unobserved items.HR@1 equals NDCG@1, so it is not reported separately as NDCG@1.
  • Experimental results: Sequential recommendation methods outperform non-sequential PopRec, FM, and AutoInt, indicating that sequential patterns matter for this task.Transformer-based methods mostly outperform the listed RNN-, CNN-, and gate-based methods, while GNN-based models are also comparable in some datasets.
  • Experimental results: FMLP-Rec performs consistently better than the baselines by a large margin on most datasets.The reported comparison includes RNN, CNN, GNN, and Transformer-based baselines.

6 FURTHER ANALYSIS

The analysis examines FMLP-Rec’s components and the broader usefulness of its learnable filters. Ablation results support the value of filter layers, while cross-model experiments show improvements across architectures.

  • Ablation Study: Removing any FMLP-Rec component degrades performance, with the largest degradation associated with removing the filter layer.The ablation study reports NDCG@10 on Beauty and Sports.
  • Ablation Study: FMLP-Rec outperforms variants that replace learnable filters with high-pass, low-pass, or band-stop filters.The authors attribute this to learnable filters adapting to the data distribution through SGD.
  • Applying Learnable Filters to Other Models: Learnable filters improve GRU4Rec, Caser, SASRec, and GCSAN after being inserted between their embedding and sequence-encoder layers.The tested base models represent RNN, CNN, Transformer, and GNN architectures.
  • Applying Learnable Filters to Other Models: FMLP-Rec outperforms the tested baselines and their learnable-filter variants.The authors connect this result to FMLP-Rec using only MLP layers and having fewer parameters than self-attention-based designs.

7 RELATED WORK

Related work traces sequential recommendation from Markov-chain and neural architectures to newer all-MLP models. The paper identifies noise sensitivity and designs learnable filters to help all-MLP models capture sequential characteristics.

  • Sequential Recommendation: Early sequential recommendation methods modeled item transitions with Markov chains, while later neural approaches used GRUs and other architectures to capture sequential patterns.The literature also introduced contextual information through memory networks, hierarchical structures, data augmentation, and pre-training.
  • Sequential Recommendation: Deep sequential recommendation models can be affected by noise in users’ historical behaviors, motivating the use of learnable filters.The supplied passage states that the paper adopts learnable filters to address this problem.
  • All-MLP Models: All-MLP models replace CNN or self-attention components with MLP-based mixing architectures and have performed well in image classification and semantic segmentation.GFNet additionally uses a 2D Fourier transform and global filter layer for frequency-domain spatial dependencies.
  • All-MLP Models: Because all-MLP models are hard to use for sequential characteristics, this approach incorporates filter layers into an all-MLP sequential recommender.The passage introduces this design as the paper’s response to the limitation of prior all-MLP models.

8 CONCLUSION

The paper concludes that noisy logged interactions motivate frequency-domain filtering for sequential recommendation. FMLP-Rec combines learnable filters with an all-MLP architecture and outperforms several neural baselines.

  • Conclusion: Filtering algorithms from digital signal processing alleviate the influence of noise in deep sequential recommendation models.The conclusion reports this finding from the paper’s empirical study.
  • Conclusion: FMLP-Rec is an all-MLP sequential recommendation model with learnable filters optimized by SGD to attenuate noise in the frequency domain.The all-MLP architecture is described as providing lower time complexity.
  • Conclusion: The learnable filters are equivalent to circular convolution in the time domain, providing a larger receptive field and better capture of periodic characteristics.
  • Conclusion: FMLP-Rec outperforms competitive RNN, CNN, GNN, and Transformer-based baselines.

A IMPLEMENTATION DETAILS

The implementation uses released or reimplemented baselines and fixes the main FMLP-Rec settings. These settings include embedding dimension 64, maximum sequence length 50, and two learnable filter blocks.

  • Implementation Details: Caser, HGN, and BERT4Rec use author-provided source code, while CLEA and other methods are implemented separately.Other methods are implemented based on RecBole, and hyperparameters follow the original papers’ suggestions.

B.1 Training Curves Analysis

FMLP-Rec achieves lower training loss and better testing accuracy than SASRec during training on Beauty and Sports. Its performance improves with deeper architectures, while gains become small beyond 16 layers and hidden size performs best around 64.

  • Training Curves Analysis: FMLP-Rec consistently achieves lower training loss and better testing accuracy than SASRec on Beauty and Sports under optimal hyper-parameter settings.The passages attribute this pattern to stronger fitting and generalization by FMLP-Rec.
  • Training Curves Analysis: Learnable filters attenuate noise in item representation matrices, helping capture useful characteristics and alleviate overfitting on noisy information.SASRec instead shows higher training loss and lower testing accuracy, reflecting difficulty training the over-parameterized Transformer model.
  • Hyper-parameter Analysis: Increasing the layer number improves FMLP-Rec performance, but gains become trivial at 16 layers.The most suitable deeper architecture remains an open direction for future investigation.
  • Hyper-parameter Analysis: A hidden size of about 64 gives the best FMLP-Rec performance on the investigated datasets.The passage identifies hidden size as one of the most important hyper-parameters to tune for a new dataset.

B.3 Visualization of Learned Filters on Other Datasets

The learned filters assign larger positive weights to low-frequency signals than to high-frequency signals across four datasets. Under full-ranking evaluation, FMLP-Rec generally outperforms representative RNN-, CNN-, and Transformer-based baselines.

  • Filter Visualization: Learnable filters show clear frequency-domain patterns, assigning larger positive weights to low-frequency than high-frequency signals across four datasets.The filters are visualized by averaging first-layer values across feature dimensions; Figure 7 encodes filter amplitudes as frequency weights.
  • Filter Interpretation: The learned filters function as special low-pass filters that attenuate high-frequency noise in item embeddings.This finding is consistent with the paper’s empirical observation that high-frequency information in item embedding matrices is usually noise.
  • Full-ranking Evaluation: Under full-sort evaluation on Beauty, Sports, Toys, and Yelp, FMLP-Rec generally outperforms SASRec and both perform much better than the other three baselines.The compared methods are GRU4Rec, Caser, SASRec, and FMLP-Rec; Table 7 reports the full-sort results.
Loading 2202.13556v1…