PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Attentive Sequential Recommendation

Wang-Cheng Kang, Julian McAuley

arXiv:1808.09781v1cs.IRcs.LG

TL;DR

Sequential recommendation must capture high-order user dynamics despite exponentially growing context, while Markov chains and RNNs each have limitations across data regimes. SASRec uses self-attention to adaptively weight past items, outperforming state-of-the-art baselines on sparse and dense datasets while running an order of magnitude faster than CNN/RNN alternatives.

  • Problem

    Sequential recommendation needs to capture high-order user dynamics succinctly as the contextual input space grows exponentially with past actions.

  • Method

    SASRec applies self-attention to adaptively weight previous items, combining long-range sequence context with predictions framed around a small number of actions.

  • Results

    SASRec outperforms state-of-the-art MC/CNN/RNN-based methods on sparse and dense benchmarks and is an order of magnitude faster than CNN/RNN alternatives.

  • Takeaways & Limitations

    SASRec adaptively emphasizes long-range dependencies on dense datasets and recent activities on sparse datasets, supporting recommendation across varying data density.

Abstract

from arXiv · show

Sequential dynamics are a key feature of many modern recommender systems, which seek to capture the `context' of users' activities on the basis of actions they have performed recently. To capture such patterns, two approaches have proliferated: Markov Chains (MCs) and Recurrent Neural Networks (RNNs). Markov Chains assume that a user's next action can be predicted on the basis of just their last (or last few) actions, while RNNs in principle allow for longer-term semantics to be uncovered. Generally speaking, MC-based methods perform best in extremely sparse datasets, where model parsimony is critical, while RNNs perform better in denser datasets where higher model complexity is affordable. The goal of our work is to balance these two goals, by proposing a self-attention based sequential model (SASRec) that allows us to capture long-term semantics (like an RNN), but, using an attention mechanism, makes its predictions based on relatively few actions (like an MC). At each time step, SASRec seeks to identify which items are `relevant' from a user's action history, and use them to predict the next item. Extensive empirical studies show that our method outperforms various state-of-the-art sequential models (including MC/CNN/RNN-based approaches) on both sparse and dense datasets. Moreover, the model is an order of magnitude more efficient than comparable CNN/RNN-based models. Visualizations on attention weights also show how our model adaptively handles datasets with various density, and uncovers meaningful patterns in activity sequences.

I. INTRODUCTION · II. RELATED WORK · A. General Recommendation

The paper frames sequential recommendation as balancing long-range behavioral modeling with robustness to sparse data, and introduces SASRec as a self-attention solution. It situates this approach among general recommendation methods and prior MC-, RNN-, and attention-based models.

  • I. INTRODUCTION: Sequential recommendation must capture useful high-order dynamics while avoiding an input space that grows exponentially with the number of historical actions used as context.The central challenge is combining personalized behavior models with contextual information from recent actions.
  • I. INTRODUCTION: MCs model short-range transitions from recent actions, whereas RNNs summarize all previous actions through a hidden state to predict the next action.MCs condition on one or a few previous actions; RNNs use a representation of the full history.
  • I. INTRODUCTION: MC methods perform well on high-sparsity data, while expressive RNNs require large, particularly dense datasets to outperform simpler baselines.The paper presents this as a trade-off between parsimonious modeling and the data requirements of higher-capacity models.
  • I. INTRODUCTION: SASRec applies self-attention to use the full action history while framing predictions around a small number of relevant actions.Its design aims to combine RNN-like long-term context with MC-like selective prediction.
  • I. INTRODUCTION: SASRec significantly outperforms state-of-the-art MC/CNN/RNN-based sequential methods across benchmark datasets and adapts its attention from long-range dependencies on dense data to recent activities on sparse data.The paper evaluates performance as a function of dataset sparsity.
  • I. INTRODUCTION: SASRec’s self-attention block enables parallel acceleration, making the model an order of magnitude faster than CNN/RNN-based alternatives.The paper also analyzes complexity and scalability, performs ablations, and visualizes attention weights to examine model behavior.
  • A. General Recommendation: General recommendation methods model user–item compatibility from historical feedback, including explicit ratings and implicit clicks, purchases, or comments.The related work notes that non-observed implicit-feedback data are ambiguous and motivates point-wise and pairwise methods.
  • A. General Recommendation: Related recommendation methods include Matrix Factorization and Item Similarity Models, alongside deep models such as NeuMF and AutoRec.MF uses latent user and item factors, ISM learns item-to-item similarities, NeuMF uses MLPs, and AutoRec uses autoencoders.

B. Temporal Recommendation … A. Embedding Layer

The paper situates SASRec among temporal, Markov-chain, recurrent, and attention-based recommendation methods, then defines sequential recommendation through embeddings, self-attention blocks, and prediction. Its embedding layer truncates or pads histories to a fixed length and combines item embeddings with positional information.

  • B. Temporal Recommendation: Temporal recommendation explicitly models activity timestamps to capture short- or long-term temporal drift, with TimeSVD++ splitting time into segments and modeling users and items separately.Examples include changing movie preferences over a decade or evolving business preferences.
  • C. Sequential Recommendation: Sequential recommendation methods model item transitions: FPMC combines matrix factorization for long-term preferences with first-order Markov transitions for short-term behavior.Higher-order Markov chains relate the next action to several previous actions.
  • C. Sequential Recommendation: RNN-based recommenders, including GRU4Rec, model click sequences through recurrent states, but dependencies between successive steps reduce efficiency.Sessionparallelism has been proposed to address this efficiency limitation.
  • D. Attention Mechanisms: Attention mechanisms focus outputs on relevant input parts and can improve interpretability; the Transformer showed that self-attention can deliver strong performance and efficiency without RNNs or CNNs.Earlier recommender applications generally added attention to models such as RNNs or factorization machines.
  • D. Attention Mechanisms: The paper summarizes its notation in Table I.This table is labeled “Notation.”
  • III. METHODOLOGY: SASRec formulates sequential recommendation as predicting the next item from a user’s preceding actions, using an embedding layer, self-attention blocks, and a prediction layer.Training uses the input sequence shifted by one position as the expected output.
  • A. Embedding Layer: The embedding layer converts histories into fixed-length sequences, retaining the most recent n actions or left-padding shorter sequences with a padding item.It retrieves E ∈ R^n×d from an item matrix M ∈ R^|I|×d, where d is the latent dimensionality.
  • A. Embedding Layer: Each input embedding combines an item embedding with a positional embedding, while experiments found fixed position embeddings performed worse in this setting.The paper analyzes positional-embedding effects quantitatively and qualitatively in its experiments.

B. Self-Attention Block

SASRec’s self-attention block transforms item embeddings into query, key, and value representations to adaptively aggregate relevant history. Causal masking prevents future-item leakage, while a shared point-wise feed-forward network adds nonlinear interactions across latent dimensions.

  • Self-Attention layer: Attention computes weighted sums of values, with weights determined by interactions between each query and key.The scale factor d helps avoid overly large inner products, especially at high dimensionality.
  • Self-Attention layer: Self-attention converts the input embedding bE into query, key, and value matrices through learned linear projections.The projection matrices WQ, WK, and WV are in R^d×d and make the model more flexible.
  • Causality: Causal masking forbids links from query Qi to key Kj when j > i, ensuring predictions use only preceding items.Without this restriction, the output at time t would contain embeddings of subsequent items.
  • Point-Wise Feed-Forward Network: A shared two-layer point-wise feed-forward network adds nonlinearity and interactions between latent dimensions after self-attention.The network is applied identically to every Si, with no interaction between Si and Sj, preserving the prevention of information leakage.

C. Stacking Self-Attention Blocks

SASRec stacks self-attention blocks to learn more complex item transitions from the first block’s representations. To support deeper networks, it uses residual connections, layer normalization, and dropout to address overfitting, instability, and training cost.

  • Block Stacking: SASRec stacks self-attention layers and feed-forward networks, using the first block’s outputs as the basis for subsequent blocks.The first block is defined as S(1) = S and F(1) = F.
  • Deep-Network Challenges: Deeper networks increase model capacity and can exacerbate overfitting, unstable training from vanishing gradients, and training time.
  • Stabilization Operations: Each layer applies normalization before g, dropout to g’s output, and a residual addition of the input.Here, g denotes either the self-attention layer or the feed-forward network.
  • Normalization and Regularization: Layer normalization normalizes features to zero mean and unit variance, while dropout randomly turns off neurons during training with probability p.Layer-normalization statistics are independent of other samples in the batch, and all neurons are used during testing.

D. Prediction Layer

SASRec predicts the next item by applying an MF relevance layer to the representation extracted by its self-attention blocks, then ranking item scores. A shared item embedding reduces model size and overfitting while preserving asymmetric transitions through nonlinear transformation, whereas adding explicit user embeddings does not improve performance.

  • Prediction mechanism: After b self-attention blocks extract information hierarchically from previously consumed items, SASRec predicts the next item using the resulting representation F(b)_t.The prediction is conditioned on the first t items.
  • Prediction mechanism: The MF prediction layer assigns each candidate item i a relevance score r_i,t, with higher scores indicating greater likelihood as the next item.Recommendations are generated by ranking these interaction scores.
  • Shared Item Embedding: A shared item embedding M reduces model size and alleviates overfitting, while the model’s nonlinear transformation can represent asymmetric item transitions.The feed forward network can achieve asymmetry using the same item embedding, unlike homogeneous inner products alone.
  • Shared Item Embedding: Empirically, using a shared item embedding significantly improves the model’s performance.
  • Explicit User Modeling: SASRec induces an implicit user embedding from all of a user’s actions, and adding an explicit user embedding does not improve performance.The paper attributes this result presumably to the model already considering all user actions.

E. Network Training · F. Complexity Analysis

SASRec trains on fixed-length, truncated-or-padded user sequences with binary cross entropy, ignoring padded targets and using Adam with per-step negative sampling. Its parameter count is independent of the number of users, while self-attention costs O(n^2d + nd^2), is parallelizable, and may require special handling for very long sequences.

  • E. Network Training: Network Training: User histories excluding the final action are converted into fixed-length sequences by truncating or padding items.The model uses the resulting sequence as input and defines corresponding expected outputs at each time step.
  • E. Network Training: Network Training: Adam optimizes the network, and each epoch randomly generates one negative item for every time step in every sequence.Adam is described as a stochastic-gradient-descent variant with adaptive moment estimation.
  • F. Complexity Analysis: Complexity Analysis: O(|I|d + nd + d^2) parameters are learned, avoiding growth with the number of users and remaining moderate when d is small.The parameters come from embeddings, self-attention, feed-forward, and layer-normalization components.
  • F. Complexity Analysis: Complexity Analysis: O(n^2d + nd^2) computation is required, with self-attention’s O(n^2d) term typically dominant.The feed-forward network and self-attention layer are the main computational sources.
  • F. Complexity Analysis: Complexity Analysis: Computation within each self-attention layer is fully parallelizable and therefore amenable to GPU acceleration.This contrasts with RNN methods such as GRU4Rec, whose computations depend on preceding time steps.
  • F. Complexity Analysis: Complexity Analysis: The model cannot ultimately scale to very long sequences, motivating restricted self-attention or splitting long sequences into short segments.Restricted attention can focus on recent actions while higher layers consider distant actions.

G. Discussion

SASRec generalizes several classic collaborative-filtering and item-similarity models through specific architectural reductions. Its self-attention formulation also provides an adaptive sequential item-similarity perspective alongside the local-pattern focus of Markov Chains and the sequence modeling of RNNs.

  • Reduction to Existing Models: SASRec reduces to FMC when self-attention is removed, item embeddings are unshared, and position embeddings are omitted.These operations eliminate the self-attention and positional components while retaining the corresponding factorized Markov structure.
  • Reduction to Existing Models: SASRec becomes equivalent to FPMC by applying the FMC reduction and adding an explicit user embedding through concatenation.FPMC combines matrix factorization for user preferences with factorized Markov Chains for short-term dynamics.
  • Reduction to Existing Models: SASRec reduces to FISM with one self-attention layer, uniform attention weights, unshared item embeddings, and no position embedding.Under this configuration, the model is described as an adaptive, hierarchical, sequential item-similarity model for next-item recommendation.
  • MC-based Recommendation: Markov-Chain recommenders capture local patterns by assuming the next item depends only on the previous L items, with first-order methods often strongest on sparse datasets.Existing approaches include first-order methods such as FPMC, HRM, and TransRec, and higher-order methods such as Fossil, Vista, and Caser.
  • RNN-based Recommendation: RNNs model user action sequences, while CNNs and self-attention have shown greater strength in some sequential settings and offer alternatives for recommendation sequence modeling.The passage also notes that RNNs are inefficient for parallel computation.

IV. EXPERIMENTS · A. Datasets

The experiments evaluate SASRec across four real-world datasets and address performance, architecture, efficiency, scalability, and attention-pattern questions. The datasets span different domains, platforms, and sparsity levels, using timestamp-ordered implicit-feedback sequences with temporal train/validation/test partitions.

  • IV. EXPERIMENTS: The experiments ask whether SASRec outperforms state-of-the-art CNN/RNN-based methods.They also examine the influence of SASRec components, training efficiency and scalability regarding n, and whether attention weights learn meaningful positional or attribute-related patterns.
  • A. Datasets: The evaluation uses four datasets from three real-world applications that vary substantially in domains, platforms, and sparsity.This variation supports comparisons across different data conditions.
  • A. Datasets: Amazon contributes separate Beauty and Games datasets derived from large product-review corpora and characterized by high sparsity and variability.Top-level Amazon product categories are treated as separate datasets.
  • A. Datasets: Steam contains 2,567,538 users, 15,474 games, and 7,793,069 English reviews spanning October 2010 to January 2018.It also includes play hours, pricing, media score, category, and developer information.
  • A. Datasets: MovieLens-1M supplies 1 million user ratings as a widely used collaborative-filtering benchmark.The study follows prior preprocessing and treats reviews or ratings as implicit feedback, with timestamps determining action order.
  • A. Datasets: For each user, sequences are split into the most recent action for testing, the second most recent for validation, and earlier actions for training.Testing inputs contain the training actions and validation action.
  • A. Datasets: The two Amazon datasets have the fewest average actions per user and item, Steam has many actions per item, and MovieLens-1M is densest.These statistics are reported in Table II.

B. Comparison Methods

The evaluation compares SASRec with three baseline groups: order-agnostic recommenders, first-order Markov-chain methods, and deep sequential models that use multiple prior actions. These baselines span popularity ranking, personalized matrix factorization, transition modeling, RNNs, and CNNs.

  • General recommendation methods: The first group comprises general recommenders that use user feedback without modeling action order.It includes PopRec, which ranks items by popularity, and BPR, which learns personalized rankings from implicit feedback using biased matrix factorization.
  • Sequential recommendation methods: The second group comprises first-order Markov-chain methods that condition recommendations on the last visited item.FMC factorizes item transitions, FPMC combines matrix factorization with first-order transitions, and TransRec models transitions with user-specific translation vectors.
  • Deep-learning sequential recommenders: The third group comprises deep-learning sequential recommenders that consider several or all previously visited items.GRU4Rec and GRU4Rec+ use RNNs, while Caser uses convolutions over embeddings of the L most recent items to capture high-order Markov chains.
  • Comparison scope: The study omits methods such as PRME, HRM, and Fossil because baselines above had outperformed them on similar datasets.The comparison also excludes temporal recommendation methods.

C. implementation Details … F. Ablation Study

SASRec uses a compact self-attention implementation and Top-N evaluation protocol, outperforming baselines across sparse and dense datasets. Ablations show that positional embeddings, shared item embeddings, residual connections, dropout, and hierarchical attention materially affect performance.

  • C. implementation Details: SASRec uses two self-attention blocks, learned positional embeddings, shared item embeddings, Adam optimization, and dataset-dependent dropout and sequence lengths.The learning rate is 0.001, batch size is 128, dropout is 0.2 or 0.5, and maximum sequence length is 200 or 50.
  • D. Evaluation Metrics: Hit Rate@10 measures whether the ground-truth next item appears among the top 10, while NDCG@10 additionally weights higher-ranked positions more heavily.With one test item per user, Hit@10 equals Recall@10 and is proportional to Precision@10.
  • D. Evaluation Metrics: Evaluation ranks each ground-truth item against 100 randomly sampled negative items, avoiding computation over all user-item pairs.The metrics are computed from rankings of 101 items per user.
  • E. Recommendation Performance: SASRec outperforms all baselines on sparse and dense datasets, improving average Hit Rate by 6.9% and NDCG by 9.6% over the strongest baseline.Non-neural methods perform better on sparse datasets, whereas neural approaches perform better on denser datasets.
  • E. Recommendation Performance: For every dataset, SASRec achieves satisfactory NDCG@10 performance with latent dimensionality d ≥40 and generally benefits from larger d.The examined range is d=10 to 50.
  • F. Ablation Study: Removing positional embeddings helps on the sparsest Beauty dataset but hurts on denser datasets, showing that action order matters more as sequences become richer.Without positional embeddings, attention depends only on item embeddings and ignores action order.
  • F. Ablation Study: Unsharing item embeddings, removing residual connections, or removing dropout impairs performance, with residual connections and dropout especially useful for sparse datasets.Unshared embeddings consistently hurt, while dropout regularizes test performance and residual connections propagate lower-layer information.
  • F. Ablation Study: Zero attention blocks performs poorly, two blocks improve dense-dataset results over one, three blocks are similar to two, and two attention heads are slightly worse than one.Two blocks help learn more complex item transitions, while multi-head decomposition may be unsuitable for the model’s small latent dimensionality.

G. Training Efficiency & Scalability

SASRec is evaluated for training speed, convergence time, and scalability in maximum sequence length using a single GTX-1080 Ti GPU. On ML-1M, it updates models far faster per epoch than Caser and GRU4Rec+ while scaling linearly with users, items, and actions.

  • Evaluation setup: The evaluation measures one-epoch training speed, convergence time, and scalability with respect to maximum sequence length n on a single GTX-1080 Ti GPU.These experiments address training efficiency and scalability together.
  • Training efficiency: 1.7 seconds per epoch makes SASRec over 11 times faster than Caser (19.1s/epoch) and 18 times faster than GRU4Rec+ (30.7s).GRU4Rec is omitted because of inferior performance; Caser and GRU4Rec+ use either complete data or the most recent 200 actions for fair comparison.
  • Training efficiency: SASRec is an order of magnitude faster than CNN/RNN-based recommendation methods in both training time per epoch and total training time.This comparison is summarized for ML-1M.
  • Scalability: SASRec scales linearly with the total number of users, items, and actions, while GPU parallelization addresses computation for maximum length n.The study measures training time and performance across different values of n.

H. Visualizing Attention Weights · V. CONCLUSION

SASRec’s attention visualizations show adaptive, position-aware, hierarchical behavior across datasets and identify relationships between similar items. The conclusion presents SASRec as an effective and efficient next-item recommendation model across sparse and dense datasets.

  • H. Visualizing Attention Weights: SASRec adaptively assigns attention weights to prior items using their position and item embeddings, revealing patterns through average position- and item-level visualizations.The analysis examines all training sequences rather than isolated examples.
  • H. Visualizing Attention Weights: Average positional attention weights are computed using only valid weights, preventing padding items in short sequences from affecting the averages.The visualization covers the last 15 positions at the last 15 time steps.
  • H. Visualizing Attention Weights: On sparse Beauty, SASRec attends more to recent items, whereas on dense ML-1M it attends to less recent items.This dataset-dependent behavior supports adaptation across different density regimes.
  • H. Visualizing Attention Weights: Without positional embeddings, attention is nearly uniform across previous items; with them, SASRec becomes more position-sensitive and favors recent items.The comparison isolates the effect of positional embeddings.
  • H. Visualizing Attention Weights: Higher hierarchical layers focus more on recent positions because earlier blocks already consider all previous items, making the mechanism adaptive, position-aware, and hierarchical.The proposed explanation concerns how attention varies across blocks.
  • H. Visualizing Attention Weights: In MovieLens-1M, attention between sampled movies forms an approximately block-diagonal heatmap, assigning larger weights to similar items sharing categories.The model identifies these relationships without category information in advance.
  • V. CONCLUSION: SASRec is a self-attention sequential model for next-item recommendation that models entire user sequences without recurrent or convolutional operations.It adaptively considers consumed items for prediction.
  • V. CONCLUSION: SASRec outperforms state-of-the-art baselines on sparse and dense datasets and is an order of magnitude faster than CNN/RNN-based approaches.These are the conclusion’s summarized empirical findings.
Loading 1808.09781v1…