PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling Sequences as Distributions with Uncertainty for Sequential Recommendation

Ziwei Fan, Zhiwei Liu, Lei Zheng, Shen Wang, Philip S. Yu

arXiv:2106.06165v1cs.IRcs.AIcs.LG

TL;DR

Existing sequential methods assume deterministic users, although item transitions and user interests can be stochastic, and representing that uncertainty is challenging. DT4SR models items and sequences as Elliptical Gaussian distributions with separate mean and covariance Transformers, using Wasserstein distance. It outperforms baselines across three datasets and is reported to alleviate cold-start issues.

  • Problem

    Existing sequential methods assume deterministic users despite fluctuating item transitions and stochastic interests, while dynamically learning distributions and measuring sequence-distribution distances remain challenging.

    Source: §1 · pp. 1–2

  • Method

    DT4SR represents items as Elliptical Gaussian distributions and uses mean and covariance Transformers with Wasserstein distance for next-item recommendation.

    Source: §1 · p. 2, §3.1 · p. 2

  • Results

    DT4SR significantly outperforms all baselines across all three datasets and evaluation metrics, with average relative gains over the second-best baseline of 19.9% on Recall@1.

    Source: §4.3 · p. 4

  • Takeaways & Limitations

    The authors report that uncertainty-aware distributions alleviate cold-start recommendation, including for users with only one interaction.

    Source: §1 · p. 2, §4.4 · p. 4

Abstract

from arXiv · show

The sequential patterns within the user interactions are pivotal for representing the user's preference and capturing latent relationships among items. The recent advancements of sequence modeling by Transformers advocate the community to devise more effective encoders for the sequential recommendation. Most existing sequential methods assume users are deterministic. However, item-item transitions might fluctuate significantly in several item aspects and exhibit randomness of user interests. This \textit{stochastic characteristics} brings up a solid demand to include uncertainties in representing sequences and items. Additionally, modeling sequences and items with uncertainties expands users' and items' interaction spaces, thus further alleviating cold-start problems. In this work, we propose a Distribution-based Transformer for Sequential Recommendation (DT4SR), which injects uncertainties into sequential modeling. We use Elliptical Gaussian distributions to describe items and sequences with uncertainty. We describe the uncertainty in items and sequences as Elliptical Gaussian distribution. And we adopt Wasserstein distance to measure the similarity between distributions. We devise two novel Trans-formers for modeling mean and covariance, which guarantees the positive-definite property of distributions. The proposed method significantly outperforms the state-of-the-art methods. The experiments on three benchmark datasets also demonstrate its effectiveness in alleviating cold-start issues. The code is available inhttps://github.com/DyGRec/DT4SR.

1 INTRODUCTION

Transformer-based sequential recommendation models user histories to predict next items, but stochastic item transitions make deterministic representations incomplete. DT4SR represents items and sequences as distributions to capture uncertainty and address cold-start, reporting gains over state-of-the-art methods.

1 INTRODUCTION

  • Transformer encoders build on SASRec [11], which uses self-attention to measure correlations among item transitions.Later works [18] [22] add components to SASRec, demonstrating the Transformer backbone’s effectiveness for sequence modeling.Source: §1 · p. 1
  • Item transitions fluctuate across aspects such as genre and release year, challenging existing models’ ability to represent sequential correlations.The introduction illustrates transitions between items with distinct genres and years.Source: §1 · p. 1
  • Distribution-based sequence representations cover more data space, expanding users’ interaction spaces and alleviating cold-start for users with few interactions.The paper frames user interactions as stochastic events controlled by distributions of dynamic interests.Source: §1 · p. 2
  • DT4SR represents items with Elliptical Gaussian distributions and models sequence means and covariances with separate Transformers, using Wasserstein distance for recommendation.The method addresses the challenge of dynamically learning distribution parameters and measuring distances between sequence distributions.Source: §1 · p. 2
  • DT4SR significantly improves overall and cold-start recommendation performance over state-of-the-art methods, supporting uncertainty-aware distribution representations for sequential modeling.The authors identify modeling items and sequences as Elliptical Gaussian distributions as a first for sequential recommendation.Source: §1 · p. 2

2 PROBLEM DEFINITION

Sequential recommendation models a user’s chronological interaction history to predict the next item. The paper’s framework illustrates how mean and covariance sequences are used to produce a next-step item distribution.

2 PROBLEM DEFINITION

  • Sequential recommendation models a user’s chronological interaction sequence to predict the next item.The sequence is denoted S_u and consists of the user’s interacted items.Source: §2 · p. 2
  • The objective measures the probability that an item is next, given the user’s sequence.The displayed formulation expresses next-item prediction conditioned on S_u.Source: §2 · p. 2
  • Mean and Covariance Transformers take their respective embedding sequences as inputs and output the next-step item’s distribution.Figure 2 illustrates the proposed DT4SR framework.Source: Figure 2 · p. 2

3 PROPOSED MODEL

DT4SR represents items and sequences as distributions and uses separate mean and covariance Transformer pathways, with distribution-aware attention, feed-forward layers, and a covariance output layer that preserves positive definiteness.

3.1 Embedding Layers

  • Items use separate mean and covariance embeddings with positional embeddings to form sequence representations, retaining the most recent n interactions or padding shorter sequences.The mean and covariance embeddings together describe item distributions.Source: §3.1 · p. 2

3.2 Mean and Covariance Self-Attentions

  • Distribution Self-Attention models sequential correlations in mean and covariance representations, applying ELU after linear transformations for numerical stability.The module is applied separately in the mean and covariance Transformers.Source: §3.2 · p. 3

3.3 Mean and Covariance Feed-Forward Layers

  • The mean and covariance feed-forward layers add non-linearity for distribution representations by replacing ReLU with ELU and retaining standard Transformer techniques.The model also supports stacking layers and using multi-head attention.Source: §3.3 · p. 3

3.4 Layer Outputs

  • Because feed-forward outputs do not guarantee positive-definite covariance, the covariance output layer adds an all-ones vector after ELU.The resulting output is the item covariance embedding after the covariance Transformer layers.Source: §3.4 · p. 3

3.5 Loss and Optimization

  • Training uses the next item in each user sequence as the ground-truth prediction target at each position, including padding items as targets for padding positions.The setup predicts the next-step item's mean and covariance embeddings.Source: §3.5 · p. 3

3.5.1 Wasserstein Distance and Loss.

  • The model uses 2-Wasserstein distance to compare inferred and ground-truth Gaussian distributions, and a Wasserstein distance-based BPR loss to assess next-item prediction.The passage notes Wasserstein distance can compare distributions without overlap, unlike KL divergence.Source: §3.5.1 · p. 3, §3.5.2 · p. 3

3.5.2 Loss.

  • For evaluation, DT4SR scores all candidate items using 2-Wasserstein distance from the inferred next-item distribution and ranks items in ascending order.The inferred distribution is taken from the sequence's last position.Source: §3.5.2 · p. 3

4 EXPERIMENTS

The experiments assess DT4SR against recommendation baselines, compare performance across user sequence lengths, and examine cold-start items. They report overall gains and improvements in cold-start settings.

4.1 Datasets

  • The study evaluates DT4SR on three public Amazon review datasets, treating ratings as implicit positive feedback and sorting interactions by timestamp; the last interaction is test data.The second-to-last interaction is used for validation.Source: §4.1 · pp. 3–4

4.3 Overall Comparison (RQ1 and RQ2)

  • DT4SR significantly outperforms all baselines across three datasets and all metrics; average gains over the second-best baseline are 19.9% Recall@1, 6.7% Recall@5, 10.8% NDCG@5, and 10.4% MRR.Table 2 reports the overall comparison.Source: Table 2 · p. 4, §4.3 · p. 4

4.2 Experimental Settings

  • Evaluation used Recall@1 and Recall@5, NDCG@1 and NDCG@5, and MRR, with 1,000 sampled negatives per user; models were selected by validation MRR.The test-set results use each model’s best validation-MRR setting.Source: §4.2 · p. 4

4.4 Performance w.r.t Sequence length (RQ3)

  • DT4SR leads across sequence-length intervals; for one-interaction users it gains 9% Recall@5 over SASRec, while users with more than 20 interactions show considerable gains, especially on Toys.Figure 3 covers Amazon Toys and Amazon Games.Source: Figure 3 · p. 4, §4.4 · p. 4

4.5 Performance on Cold Start Items (RQ3)

  • For extremely cold-start items with one training interaction, DT4SR gains 100% on Toys and more than 50% on Games, while retaining comparative performance on frequent items.Figure 4 shows Recall@5 performance on cold-start items across Amazon Toys and Amazon Games.Source: §4.5 · p. 4, Figure 4 · p. 5

5 CONCLUSION

DT4SR introduces uncertainty into sequential recommendation by representing items and sequences with distributions. Experiments report improvements in overall performance and cold-start user and item recommendation.

5 CONCLUSION

  • DT4SR uses Transformer-based distribution representations for sequential recommendation, introducing uncertainty into item and sequence representations.The paper represents items with Elliptical Gaussian distributions and uses mean and covariance Transformers, with covariance constrained to be positive definite.Source: §5 · p. 5
  • The proposed mean and covariance Transformers learn distribution parameters while guaranteeing positive definiteness of the covariance.Source: §5 · p. 5
  • Experiments report that DT4SR significantly outperforms existing methods overall and in cold-start user and item recommendation.Source: §5 · p. 5

Related papers

Loading 2106.06165v1…