Source-linked AI summary
Denoising Self-attentive Sequential Recommendation
Huiyuan Chen, Yusan Lin, Menghai Pan, Lan Wang, Chin-Chia Michael Yeh, Xiaoting Li, Yan Zheng, Fei Wang, Hao Yang
TL;DR
Noisy user histories can contain irrelevant interactions, while dense self-attention gives all items some influence. Rec-Denoiser uses differentiable masks to prune irrelevant attention and applies Jacobian regularization; experiments report gains for self-attentive recommenders and robustness to perturbations.
Problem
Noisy sequential histories and dense self-attention can give irrelevant items influence, while existing self-attentive recommenders are insufficient for handling noisy items.
Source: §1 · p. 2Method
Rec-Denoiser learns differentiable masks that dynamically prune irrelevant self-attention connections and applies Jacobian regularization to Transformer blocks.
Source: §1 · p. 3, §6 · p. 14Results
Rec-Denoiser reports 5.05% ∼19.55% performance gains for self-attentive recommenders and effectiveness on multiple real-world tasks, with robustness to input perturbations.
Source: §1 · p. 4, §6 · p. 14Takeaways & Limitations
The approach preserves Transformer architectures while making attention sparse and enabling irrelevant dependencies to receive exactly zero attention scores.
Source: §1 · p. 3, §4.1.1 · p. 7The setup represents each user's items as a sequence in chronological order.
Source: §3.1 · p. 5
Abstract
from arXiv · showhide
Transformer-based sequential recommenders are very powerful for capturing both short-term and long-term sequential item dependencies. This is mainly attributed to their unique self-attention networks to exploit pairwise item-item interactions within the sequence. However, real-world item sequences are often noisy, which is particularly true for implicit feedback. For example, a large portion of clicks do not align well with user preferences, and many products end up with negative reviews or being returned. As such, the current user action only depends on a subset of items, not on the entire sequences. Many existing Transformer-based models use full attention distributions, which inevitably assign certain credits to irrelevant items. This may lead to sub-optimal performance if Transformers are not regularized properly.
1 INTRODUCTION
Self-attentive recommenders can model long- and short-term item dependencies, but noisy sequences and dense attention can give irrelevant items influence. Rec-Denoiser uses differentiable masks to prune irrelevant attention and Jacobian regularization to improve robustness.
1 INTRODUCTION
- Noisy clicks and other irrelevant interactions can make next-item predictions untrustworthy, while dense self-attention assigns credit to every item.The motivating example includes unrelated purchases for different family members in one sequence.Source: §1 · pp. 2–3
- Rec-Denoiser uses differentiable masks to dynamically prune task-irrelevant attention, producing sparse attention distributions with exactly zero scores for noisy items.The method is presented as a denoising strategy for self-attentive sequential recommenders.Source: §1 · pp. 3–4
- The approach changes attention distributions rather than Transformer architectures, supporting compatibility with existing Transformers and improved interpretability.The introduction describes this as an implementation advantage of Rec-Denoiser.Source: §1 · p. 3
- Probabilistic reparameterization makes binary masks trainable end to end, while Jacobian regularization on Transformer blocks is applied to improve robustness to noisy sequences.The contributions also describe an unbiased gradient estimator for optimizing binary masks.Source: §1 · p. 4
- 5.05% ∼19.55% performance gains are reported for Rec-Denoiser on self-attentive recommenders, alongside robustness against input perturbations.The introduction reports these gains without specifying metric labels or baselines in the supplied passage.Source: §1 · p. 4
2 RELATED WORK
Related work spans sequential recommendation models and sparse Transformers. The cited work motivates Rec-Denoiser as a data-driven alternative to fixed attention patterns and sparse-normalization approaches.
2.1 Sequential Recommendation
- Sequential recommendation research includes Markov, recurrent, convolutional, memory, and graph models, followed by Transformers that capture arbitrary dependencies but have received little robustness study.Transformer variants add bidirectional attention, time intervals, personalization, importance sampling, or sequence augmentation.Source: §2.1 · p. 4
2.2 Sparse Transformer
- Sparse Transformer methods use fixed patterns or sparse attention distributions, whereas Rec-Denoiser learns binary masks to decide which self-attention connections to keep or delete.The related-work discussion contrasts fixed schemas with trainable, data-driven pruning.Source: §2.2 · p. 5
3 PROBLEM AND BACKGROUND
Section 3 defines sequential recommendation as predicting a user's next item from interaction history and reviews self-attentive recommenders. It identifies dense, fixed, or random attention pruning as inadequate for handling irrelevant items in sequences.
3.1 Problem Setup
- Given a user's chronological interaction history, sequential recommendation predicts the next item; training uses the input sequence and its one-step-shifted targets.The sequence is converted to a fixed-length form by truncating or padding items.Source: §3.1 · p. 5
3.2 Self-attentive Recommenders
- Self-attentive recommenders use Transformer blocks to capture sequential dependencies, with SASRec, BERT4Rec, and TiSASRec among the cited models.A block combines self-attention and a point-wise feed-forward layer; stacking blocks can model hierarchical item dependencies.Source: §3.2 · p. 5, §3.2.2 · p. 6
3.2.1 Embedding Layer.
- The embedding layer represents a fixed-length item sequence and adds learnable positional embeddings to encode item positions.The resulting order-aware embeddings can be passed to Transformer-based models.Source: §3.2.1 · p. 5
3.2.2 Transformer Block.
- Self-attention computes dependencies among sequence items, while parallel attention heads enhance expressiveness and feed-forward layers add nonlinearity.Sequential recommendation can use unidirectional or bidirectional attention.Source: §3.2.2 · p. 6
3.2.3 Learning Objective.
- The learning objective predicts item relevance with an inner product and applies binary cross-entropy to the input sequence and its shifted targets.The passage identifies the item embedding and sigmoid function used in this objective.Source: §3.2.3 · p. 6
3.3 The Noisy Attentions Problem
- Dense attention credits every item, including irrelevant ones, while predefined sparse schemas cannot adapt to input data and random dropout can prune too aggressively.These limitations motivate the proposed data-driven differentiable masks for removing irrelevant attentions.Source: §3.3 · pp. 6–7
4 REC-DENOISER
Rec-Denoiser combines differentiable attention masks with Jacobian regularization. The section explains how it learns sparse attentions, estimates mask gradients, regularizes Transformer sensitivity, and trains with computational costs comparable in order to the original models.
4.1 Differentiable Masks
- Full attention may create irrelevant dependencies, unnecessary computation, and unexpected explanations, motivating differentiable masks.Rec-Denoiser pairs masks for self-attention layers with Jacobian regularization for Transformer blocks.Source: §4.1 · p. 7, §4 · p. 7
4.1.1 Learnable Sparse Attentions.
- Trainable binary masks prune task-irrelevant attention scores, and an L0 penalty encourages sparsity while allowing irrelevant scores to become exactly zero.The discrete masks make optimization challenging because the L0 objective is discontinuous and the binary state space is large.Source: §4.1.1 · p. 7
4.1.2 Efficient Gradient Computation.
- ARM provides an unbiased, low-variance estimator for optimizing the discrete mask variables, unlike gradient estimators that can be biased or high-variance.The authors also consider AR variants and describe a trade-off between estimator variance and computational complexity.Source: §4.1.2 · p. 8
4.2 Jacobian Regularization
- Frobenius-norm Jacobian regularization locally enforces a Lipschitz constraint and strictly bounds a Transformer block's residual error.Random projections compute the Jacobian norm and its gradient while reducing running time in practice.Source: §4.2 · p. 9
4.3.1 Joint Training.
- The joint objective controls attention sparsity and network robustness, and Rec-Denoiser remains compatible with Transformer recommenders without changing their main architectures.Setting masks to all ones and both regularizers to zero recovers the original design.Source: §4.3.1 · pp. 9–10
4.3.2 Model Complexity.
- Training retains the original Transformers' complexity order, while sparse attention maps enable faster feed-forward computation at inference.ARM uses two forward passes for low variance; the authors note that sequential recommenders often have few Transformer blocks.Source: §4.3.2 · p. 10
5 EXPERIMENTS
The experiments ask whether Rec-Denoiser improves on existing sequential recommenders, reduces the effects of noisy items, and benefits from its proposed components.
5 EXPERIMENTS
- RQ1 compares Rec-Denoiser with state-of-the-art sequential recommenders.The experiments also ask how it reduces the negative impacts of noisy items (RQ2) and how its components affect performance (RQ3).Source: §5 · p. 10
5.1 Experimental Setting
The evaluation uses five benchmark datasets, compares sequential and sparse-Transformer baselines, and measures ranking performance under a held-out-item protocol.
5.1.1 Dataset.
- Five benchmarks use last-item testing, second-to-last validation, and earlier sequence items for training.MovieLens is the densest dataset, while Beauty has the fewest actions per user.Source: §5.1.1 · p. 11
5.1.2 Baselines.
- Baselines include general sequential models and sparse Transformers, with Rec-Denoiser applied as a plugin to self-attentive backbones.The sparse comparisons are Sparse Transformer and α-entmax; several long-sequence language-model sparse methods are left for future work.Source: §5.1.2 · p. 11
5.1.3 Evaluation metrics.
- Hit@10 and NDCG@10 rank each ground-truth item against 100 randomly sampled negatives.Hit@10 counts inclusion in the top 10; NDCG@10 gives greater weight to higher-ranked positions.Source: §5.1.3 · pp. 11–12
5.1.4 Parameter settings.
- The models use tuned validation-set hyperparameters and test-set results, with sequence lengths set to 50 for dense datasets and 25 for sparse ones.Self-attentive models use two blocks and two heads; Rec-Denoiser’s β and γ are searched from 10^-1 to 10^-5.Source: §5.1.4 · p. 12
5.2 Overall Performance (RQ1)
Rec-Denoisers achieve the best overall performance across all five datasets. Their improvements over self-attentive backbones are significant, with reported gains for SASRec and BERT4Rec.
5.2 Overall Performance (RQ1)
- Rec-Denoisers obtain the best overall recommendation performance on all five datasets.Table 2 reports the overall performance of the compared methods and defines RI as relative improvement over each backbone.Source: §5.2 · p. 12
- Self-attentive sequential models generally outperform FPMC, GRU4Rec, and Caser, consistent with their ability to capture long-range item dependencies.This observation concerns the compared models’ performance in sequential recommendation.Source: §5.2 · p. 12
- Self-attentive variants benefit from added bidirectional attention, time intervals, and user personalization.The authors identify these auxiliary information sources as important for interpreting dynamic user behaviors.Source: §5.2 · p. 12
- 8.04% average Hit@10 and over 12.42% NDCG@10 improvement: SASRec+Denoiser versus SASRec.BERT4Rec+Denoiser improves average Hit@10 by 7.47% and NDCG@10 by 11.64% over vanilla BERT4Rec; all reported backbone comparisons have p-values < 1e-6.Source: §5.2 · pp. 12–13
- The authors attribute Rec-Denoiser’s improvements to retaining self-attention strengths, removing irrelevant dependencies, and smoothing gradients through Jacobian regularization.They state that gradient smoothness limits rapid output changes under input perturbations and generally improves recommendation generalization.Source: §5.2 · p. 13
5.3 Robustness to Noises (RQ2)
Under synthetic training-data corruption, Rec-Denoiser remains stronger than two sparse-attention baselines across noise ratios and datasets. Both ARM and AR variants reduce the negative effects of noisy sequences.
5.3 Robustness to Noises (RQ2)
- Rec-Denoiser with ARM or AR consistently outperforms α-entmax and Sparse Transformer across noise ratios on all datasets.The experiments randomly replace training items with sampled items excluded from validation and test sets, varying corruption from 0% to 25%; all models degrade as noise increases.Source: §5.3 · p. 13
- Both ARM and AR estimators reduce the negative impacts of noisy sequences, while AR performs better than α-entmax but worse than ARM.The authors attribute ARM’s stronger result relative to AR to its lower variance.Source: §5.3 · p. 13
5.4 Study of Rec-Denoiser (RQ3)
The study examines how Rec-Denoiser responds to sequence length and regularizer settings. Its performance improves with longer inputs and remains relatively stable across regularizer settings.
5.4 Study of Rec-Denoiser (RQ3)
- The analysis varies maximum sequence length n and sparsity and smoothness regularizers β and γ, using SASRec and SASRec-Denoiser.The study also notes that self-attentive models typically benefit from small block and head counts, such as H, L≤4; only two models are studied due to page limitations.Source: §5.4 · p. 14
- Rec-Denoiser improves Hit@10 with longer sequences on the densest and sparsest datasets, MovieLeans and Beauty.Figure 3 varies maximum length n from 20 to 80 while other optimal hyper-parameters remain unchanged.Source: §5.4 · p. 14
5.4.2 The regularizers β and
- On MovieLens, performance is relatively stable across regularizer settings, with the best result at β=0.01 and γ=0.001.Figure 4 changes one parameter while fixing the other at 0.01.Source: §5.4.2 · p. 14
6 CONCLUSION AND FUTURE WORK
Rec-Denoiser uses differentiable masks and Jacobian regularization to address noisy items and self-attention's vulnerability to small perturbations. Experiments across real-world sequential recommendation tasks illustrate the design's effectiveness, and the authors propose applying it to Transformer models in other tasks.
6 CONCLUSION AND FUTURE WORK
- Rec-Denoiser uses differentiable masks to prune irrelevant information and Jacobian regularization to improve Transformer-block robustness to small perturbations.The design adaptively targets the negative impacts of noisy items in self-attentive recommender systems.Source: §6 · p. 14
- Experiments on multiple real-world sequential recommendation tasks illustrate the effectiveness of the design.Source: §6 · p. 14
- The authors propose applying differentiable masks and Jacobian regularization to Transformer-based models in tasks beyond sequential recommendation.Demonstrating these contributions in real-world applications is left to future work.Source: §6 · p. 15