PaperlayerView exact revision on arXiv

Source-linked AI summary

Sequential Recommendation via Stochastic Self-Attention

Ziwei Fan, Zhiwei Liu, Alice Wang, Zahra Nazari, Lei Zheng, Hao Peng, Philip S. Yu

arXiv:2201.06035v2cs.IRcs.AIcs.LG

TL;DR

Sequential recommendation must handle uncertain user behavior, missing collaborative transitivity in dot-product attention, and unconstrained positive-negative distances in BPR. STOSA uses Gaussian stochastic embeddings, Wasserstein self-attention, and BPR regularization, outperforming state-of-the-art baselines across five real-world datasets, especially for cold-start items.

  • Problem

    Existing sequential recommendation methods use deterministic representations and dot-product attention that do not incorporate dynamic uncertainty or collaborative transitivity, while BPR does not guarantee positive-negative dissimilarity.

  • Method

    STOSA represents items as Gaussian distributions, measures transitions with Wasserstein self-attention, and adds a BPR regularization term for positive-negative distances.

  • Results

    STOSA outperforms state-of-the-art recommendation methods across five real-world datasets, with relative improvements from 3.16% to 35.11% across all metrics and particular effectiveness on cold-start items.

  • Takeaways & Limitations

    The results support stochastic representations, collaborative transitivity, and distance regularization as effective components for sequential and cold-start recommendation.

  • Takeaways & Limitations

    The paper focuses on BPR, although the authors note that the same issue also occurs with other losses such as hinge loss.

Abstract

from arXiv · show

Sequential recommendation models the dynamics of a user's previous behaviors in order to forecast the next item, and has drawn a lot of attention. Transformer-based approaches, which embed items as vectors and use dot-product self-attention to measure the relationship between items, demonstrate superior capabilities among existing sequential methods. However, users' real-world sequential behaviors are \textit{\textbf{uncertain}} rather than deterministic, posing a significant challenge to present techniques. We further suggest that dot-product-based approaches cannot fully capture \textit{\textbf{collaborative transitivity}}, which can be derived in item-item transitions inside sequences and is beneficial for cold start items. We further argue that BPR loss has no constraint on positive and sampled negative items, which misleads the optimization. We propose a novel \textbf{STO}chastic \textbf{S}elf-\textbf{A}ttention~(STOSA) to overcome these issues. STOSA, in particular, embeds each item as a stochastic Gaussian distribution, the covariance of which encodes the uncertainty. We devise a novel Wasserstein Self-Attention module to characterize item-item position-wise relationships in sequences, which effectively incorporates uncertainty into model training. Wasserstein attentions also enlighten the collaborative transitivity learning as it satisfies triangle inequality. Moreover, we introduce a novel regularization term to the ranking loss, which assures the dissimilarity between positive and the negative items. Extensive experiments on five real-world benchmark datasets demonstrate the superiority of the proposed model over state-of-the-art baselines, especially on cold start items. The code is available in \url{https://github.com/zfan20/STOSA}.

1 INTRODUCTION

Sequential recommendation must model evolving interests despite uncertainty in user transitions, while existing dot-product attention misses uncertainty and collaborative transitivity. STOSA addresses these gaps with stochastic embeddings, Wasserstein self-attention, and a BPR regularization term, and reports stronger overall and cold-start performance.

  • Motivation: Sequential recommendation predicts a user’s next preferred item from chronologically ordered historical interactions.
  • Limitations: Existing dot-product self-attention methods fail to incorporate dynamic uncertainty and collaborative transitivity in sequential recommendation.
  • Dynamic uncertainty: Deterministic item vectors inadequately represent multifarious and fluctuating interests, making uncertainty crucial for modeling sequential behavior.
  • Collaborative transitivity: Collaborative transitivity derives additional similarities from connected item transitions and can alleviate cold-start issues through inducted collaborative neighbors.
  • Proposed framework: STOSA represents items as Gaussian distributions, using means for base interests and covariances for interest variability.
  • Proposed framework: STOSA combines Wasserstein self-attention with a BPR regularization term that constrains distances between positive and sampled negative items.
  • Results: STOSA outperforms state-of-the-art methods, with effectiveness reported especially for cold-start items.

2 RELATED WORK

Related work spans sequential recommendation, metric learning, and distributional representations. These lines of work motivate combining distance-based transitivity with uncertainty-aware item representations.

  • Ranking loss: The paper discusses BPR because it is widely used for top-N recommendation, while noting that the same issue also affects other losses such as hinge loss.
  • Sequential recommendation: Sequential recommendation research includes Markov-chain and matrix-factorization models, recurrent networks, and Transformer-based self-attention methods.
  • Metric learning: Metric learning uses distances such as Euclidean, Mahalanobis, and graph distance to measure dissimilarity between objects.
  • Metric learning: Distances usually satisfy the triangle inequality, providing an inductive bias that is useful under data sparsity.
  • Distributional representations: Distributional representations encode uncertainty and provide more flexibility than a single fixed embedding.
  • Distributional representations: Prior recommendation work represents users or items with Gaussian distributions, including DT4SR’s separate mean and covariance Transformers.

3 PRELIMINARIES AND DISCUSSIONS

Sequential recommendation organizes each user’s chronological interactions into a sequence and predicts potential next items. The preliminary self-attention formulation uses position-aware item embeddings and scaled dot products to learn item correlations.

  • Sequential recommendation: Sequential recommendation sorts each user’s interacted items chronologically and aims to recommend a top-N list of potential next items.
  • Self-attention: Self-attention treats sequence items as correlated but assigns them distinct importance at different positions.
  • Self-attention: The sequence is truncated or padded to a fixed maximum length before encoding.
  • Self-attention: Position-aware item embeddings combine item embeddings with positional embeddings before attention processing.
  • Self-attention: Scaled dot-product self-attention uses Q, K, and V projections of the same encoded sequence to learn latent item correlations.
  • Self-attention: Transformer sequence encoders also use point-wise feed-forward networks, residual connections, and layer normalization.

4 PROPOSED MODEL

STOSA represents items as Gaussian distributions and uses Wasserstein self-attention to model uncertain sequential relationships and collaborative transitivity. It further regularizes BPR loss by constraining distances between positive and negative items.

  • 4.1 Stochastic Embedding Layers: STOSA represents each item with an elliptical Gaussian distribution whose mean models base interests and covariance models interest variability.Separate mean and covariance embedding tables, along with positional embeddings, represent these two signals.
  • 4.2 Wasserstein Self-Attention Layer: Wasserstein self-attention measures position-wise item relationships using distances between stochastic embeddings while preserving triangle inequality.The attention uses negative 2-Wasserstein distance, and Gaussian sequence representations aggregate mean and covariance signals separately.
  • 4.2.1 Wasserstein Attention.: The Wasserstein distance supports uncertainty-aware dissimilarity, collaborative transitivity, and smoother measurements for non-overlapping distributions.The method also reports that Wasserstein attention can be computed with batch matrix multiplications without sacrificing efficiency relative to traditional self-attention.
  • 4.3 Feed-Forward Network and Layer Outputs: Mean and covariance outputs are passed through point-wise fully connected networks with ELU activation to learn nonlinear stochastic representations.ELU is adopted for numerical stability, while ones addition to covariance embeddings helps guarantee positive definiteness.
  • 4.4 Prediction: STOSA predicts the next item by comparing inferred sequence representations with candidate-item distributions using 2-Wasserstein distance.At evaluation, candidates are ranked by ascending distance because smaller scores indicate higher next-item probability.
  • 4.5 BPR Loss with Positive v.s. Negative: A hinge-based regularizer augments BPR loss so the positive-negative distance exceeds the prediction distance from the sequence representation to the positive item.In the ideal case, the sequence and positive item are close while both remain far from the sampled negative item.

5 EXPERIMENTS

Experiments evaluate STOSA against sequential, metric-learning, and static recommendation baselines on five Amazon benchmarks using standard top-N ranking metrics. STOSA achieves the best overall performance, with gains concentrated among users with longer histories and cold-start items, while its regularization weight requires careful tuning.

  • Overall Comparison: STOSA is compared with static, metric-learning, and sequential recommendation baselines on five public Amazon datasets using Recall@N, NDCG@N, and MRR.The evaluation ranks all items rather than using biased sampling and uses chronological train, validation, and test interactions with 5-core filtering.
  • Overall Comparison: STOSA obtains the best performance against all baselines in all metrics, with relative improvements ranging from 3.16% to 35.11%.MRR improvements against the second-best baseline range from 5.68% to 11.54%.
  • Parameters Sensitivity: With a proper selection of λ, STOSA outperforms SASRec and DT4SR, while MRR first improves and then drops as λ increases.The results indicate that the additional distance constraint matters, but excessive weighting can hurt because sampled negatives are not strictly negative.
  • Parameters Sensitivity: When λ = 0, STOSA still outperforms SASRec on most datasets except Tools, supporting contributions from Wasserstein self-attention and stochastic modeling.This setting also serves as an ablation of the additional regularization term.
  • Performances w.r.t Sequence Lengths: For the longest user sequence interval, relative NDCG@5 improvements range from 9.70% to 54.45% across datasets.Users with more interactions are described as having more diverse and uncertain interests, while STOSA remains competitive or better across most intervals.
  • Performances w.r.t Item Popularity: The largest improvements occur for items with no more than three training interactions, although performance worsens for popular items in Beauty and Toys.The paper attributes the cold-item gains to Wasserstein self-attention, regularization, and collaborative transitivity, while noisy neighbors may affect popular-item representations.
  • Wasserstein Self-Attentions Visualization: STOSA assigns more uniform attention weights than SASRec while both emphasize recent behaviors, consistent with broader collaborative-neighbor connections.SASRec highlights a smaller set of sequence items, whereas STOSA’s distribution is more uniform.

6 CONCLUSION

STOSA models dynamic uncertainty and collaborative transitivity in sequential recommendation, while regularizing BPR to separate positive and sampled negative items. Results on five real-world datasets support its effectiveness, including for cold-start recommendation.

  • STOSA models dynamic uncertainty and captures collaborative transitivity in sequential recommendation.
  • A regularization term added to BPR guarantees a large distance between positive items and sampled negative items.
  • Experiments and qualitative analyses on five real-world datasets demonstrate STOSA's effectiveness and superiority for alleviating cold-start item recommendation issues.

A COMPLEXITY ANALYSIS

STOSA has asymptotic space and time complexity similar to SASRec, while its parameter complexity is slightly smaller under the stated dimension allocation. Wasserstein self-attention and feed-forward networks dominate computation, with both models typically governed by O(n^2d).

  • STOSA uses the same latent size as SASRec by splitting dimensions equally between mean and covariance embeddings.For SASRec dimension d = 128, STOSA uses d_μ = 64 and d_Σ = 64.
  • STOSA has parameter complexity O(|V|d + nd + d^2/2), slightly smaller than SASRec's O(|V|d + nd + d^2).
  • Wasserstein self-attention has time complexity O(nd + n^2d + 4n^2) after matrix-multiplication reductions for diagonal covariance matrices.The Frobenius norm and mean-embedding distance can both be computed through matrix multiplications.
  • Including feed-forward networks, STOSA's complexity is O(nd + n^2d + 4n^2 + nd^2), compared with traditional self-attention's O(n^2d + nd^2).

B DATASETS AND PREPROCESSING

Dataset statistics are provided in Table 2.

  • The paper presents dataset statistics in Table 2.
  • Table 2 contains the dataset statistics referenced in the preprocessing discussion.
  • The dataset-statistics details are deferred to Table 2.

C IMPLEMENTATION DETAILS AND BASELINES GRID SEARCH

STOSA is implemented and tuned under a search space designed to make comparisons with baselines fair. BERT4Rec receives an additional search over its masking probability.

  • Implementation: STOSA is implemented in PyTorch on an Nvidia 3090 GPU with 64GB system memory.
  • Grid Search: Baseline embedding dimensions are searched over {64, 128}, while STOSA uses {32, 64} because it has separate mean and covariance embeddings.The paper describes this as a fair comparison.
  • Baselines: BERT4Rec's mask probability is searched over {0.1, 0.2, 0.3, 0.5, 0.7}.BERT4Rec is described as extending SASRec with bidirectional item transitions and a standard Cloze objective.

D.1 Predictions Comparison

Table 3 compares STOSA and SASRec prediction lists for user A278LEQK1TEPVB, while the accompanying analysis highlights STOSA’s more relevant cold-start recommendation.

  • Table 3 compares the prediction lists for user A278LEQK1TEPVB, identifying each item by ID and popularity and marking the ground-truth item in red.
  • STOSA ranked WF-7620 Printer first, a better version of the user’s last-interacted Inkjet Printer, whereas SASRec ranked Wraparound Labels first.
  • STOSA prioritized relevant items even when they were cold start, while SASRec favored the more popular Wraparound Labels.
Loading 2201.06035v2…