Source-linked AI summary
S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization
Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, Ji-Rong Wen
TL;DR
Existing neural sequential recommenders rely mainly on item prediction and can struggle with data sparsity and weak fusion of context with sequence information. S3-Rec uses MIM-based self-supervised pre-training with four correlation objectives, and experiments report superior performance across six datasets, especially with limited data, while also improving other architectures.
Problem
Existing neural sequential recommenders rely on item prediction loss, which is prone to data sparsity and does not adequately capture associations between context and sequence data.
Method
S3-Rec pre-trains a self-attentive recommender using four MIM objectives for correlations among attributes, items, subsequences, and sequences before fine-tuning for recommendation.
Results
Experiments on six real-world datasets show state-of-the-art performance, especially with limited training data, and improvements when the method is adapted to GRU and CNN models.
Takeaways & Limitations
MIM-based self-supervision improves sequential recommendation representations and can extend beyond the self-attentive architecture to other neural recommendation models.
Takeaways & Limitations
Future work is needed to design other self-supervised objectives and apply the approach to more complex tasks such as conversational and multimedia recommendation.
Abstract
from arXiv · showhide
Recently, significant progress has been made in sequential recommendation with deep learning. Existing neural sequential recommendation models usually rely on the item prediction loss to learn model parameters or data representations. However, the model trained with this loss is prone to suffer from data sparsity problem. Since it overemphasizes the final performance, the association or fusion between context data and sequence data has not been well captured and utilized for sequential recommendation. To tackle this problem, we propose the model S^3-Rec, which stands for Self-Supervised learning for Sequential Recommendation, based on the self-attentive neural architecture. The main idea of our approach is to utilize the intrinsic data correlation to derive self-supervision signals and enhance the data representations via pre-training methods for improving sequential recommendation. For our task, we devise four auxiliary self-supervised objectives to learn the correlations among attribute, item, subsequence, and sequence by utilizing the mutual information maximization (MIM) principle. MIM provides a unified way to characterize the correlation between different types of data, which is particularly suitable in our scenario. Extensive experiments conducted on six real-world datasets demonstrate the superiority of our proposed method over existing state-of-the-art methods, especially when only limited training data is available. Besides, we extend our self-supervised learning method to other recommendation models, which also improve their performance.
1 INTRODUCTION
Sequential recommendation must model evolving user behavior, but existing neural methods rely heavily on item prediction and do not adequately integrate contextual and sequence information. S3-Rec addresses these issues with MIM-based self-supervised pre-training, achieving strong performance across six datasets, particularly with limited training data.
- Sequential recommendation models user-item interactions using RNNs, CNNs, and self-attention to capture sequential patterns.
- Contextual information such as item attributes is important for improving neural sequential recommender performance.
- Reliance on a single item prediction loss can make neural sequential methods vulnerable to data sparsity, including when context parameters are learned through that objective alone.
- Existing methods may underrepresent the association between context data and sequence data, motivating more effective representation learning.
- S3-Rec pre-trains a self-attentive recommender with four MIM objectives covering item-attribute, sequence-item, sequence-attribute, and sequence-subsequence correlations.The pre-trained representations are then fine-tuned for recommendation.
- Experiments on six real-world datasets report state-of-the-art performance, especially when training data is limited, and the method also adapts to GRU and CNN architectures.
2 RELATED WORK
Related work covers sequential recommendation, general self-supervised learning, and MIM. S3-Rec differs by using correlations among contextual views at multiple granularities as self-supervised signals for sequential recommendation.
- Sequential recommendation research has progressed from Markov-chain methods to neural models including GRUs and their variants.
- Self-supervised learning constructs auxiliary training signals automatically from raw-data correlations and can transfer learned representations to downstream tasks.
- MIM maximizes mutual information between representations of multiple, potentially overlapping views, using views from other inputs as negative samples.
- S3-Rec applies MIM to correlations among attribute, item, and sequence views at different contextual granularities for sequential recommendation.The enhanced representations are intended to improve recommendation performance.
3 PRELIMINARIES
The paper formalizes sequential recommendation as next-item prediction from historical interactions and item attributes, then introduces MIM and its InfoNCE lower-bound implementation.
- 3.1 Problem Statement: Each user has an interaction sequence, each item has associated attributes, and contiguous interactions define subsequences.
- 3.1 Problem Statement: Sequential recommendation predicts a user’s next item from chronologically ordered historical interactions and the attributes associated with each item.
- 3.2 Mutual Information Maximization: Mutual information measures dependency between random variables by quantifying how knowing one reduces uncertainty about the other.
- 3.2 Mutual Information Maximization: Because direct MIM is usually intractable, the method uses InfoNCE as a practical lower bound based on noise contrastive estimation.
- 3.2 Mutual Information Maximization: InfoNCE represents an input through different views and uses a positive sample together with negative samples drawn from a proposal distribution.
- 3.2 Mutual Information Maximization: When all possible values are uniformly included, maximizing InfoNCE is analogous to maximizing standard cross-entropy, with negative sampling approximating the partition function.
4.1 Overview
S3-Rec enhances neural sequential recommenders by maximizing mutual information among raw-data views at multiple granularities. Its overview combines a Transformer-based base model with InfoNCE objectives for attributes, items, segments, and sequences.
- 4.1 Overview: S3-Rec responds to item-level optimization by maximizing mutual information among different views of raw sequential-recommendation data.
- 4.1 Overview: The approach treats attributes, items, segments, and sequences as input views and designs self-supervised objectives from their intrinsic correlations.
- 4.1 Overview: The model uses a Transformer-based recommender, enhances representations with InfoNCE objectives, and then discusses the resulting approach.
4.2 Base Model
The base model combines item and attribute embeddings with positional encoding, stacked self-attention blocks, and a prediction layer. Bidirectional attention is used during pre-training, while recommendation uses only information available before the current step.
- The base sequential recommender stacks embedding, self-attention, and prediction layers.
- Embedding Layer: Item and attribute one-hot representations are projected into dense embeddings, while positional encoding is added to item-sequence embeddings.
- Self-Attention Block: Each self-attention block contains multi-head self-attention and a point-wise feed-forward network for selectively extracting information from representation subspaces.
- Self-Attention Block: The attention mechanism forms query, key, and value projections and scales inner products by d/h to avoid excessively large values.
- Self-Attention Block: Pre-training removes attention masks to learn bidirectional context, whereas sequential recommendation masks future connections and uses only preceding information.
- The final layer computes a preference score for the next item from the user’s historical context.
4.3 Self-supervised Learning with MIM
S^3-Rec pre-trains representations by maximizing mutual information across item, attribute, sequence, and subsequence views. Four objectives recover associated attributes, masked items, masked attributes, and masked segments from contextual information.
- S^3-Rec constructs self-supervised losses from multi-view correlations and optimizes them during pre-training.
- Modeling Item-Attribute Correlation: AAP maximizes item-attribute mutual information by contrasting each item with its associated attributes and sampled negative attributes.
- Modeling Sequence-Item Correlation: MIP uses a Cloze task to recover a masked item from bidirectional surrounding sequence context.
- Modeling Sequence-Attribute Correlation: MAP recovers the attributes of a masked item from its surrounding context, explicitly modeling sequence-attribute correlation.
- Modeling Sequence-Segment Correlation: SP extends Cloze prediction from one item to a missing subsequence, using its surrounding sequence context to recover the segment.
4.4 Learning and Discussion
S^3-Rec separates learning into self-supervised pre-training and recommendation fine-tuning, then positions its objectives as a general framework for existing recommendation models. The framework connects attribute, item, sequence, and segment representations through additional correlation signals.
- S^3-Rec has pre-training and fine-tuning stages, using bidirectional and unidirectional Transformer architectures respectively.
- Pre-training optimizes four objectives modeling item-attribute, sequence-item, sequence-attribute, and sequence-segment correlations.
- The self-supervised framework captures intrinsic input correlations as additional signals and can incorporate existing recommendation methods.
- Discussion: AAP and MAP provide feature-interaction effects, while MIP captures sequential dependencies with bidirectional context in the pre-training stage.
- Discussion: For attribute-aware sequential models, AAP and MAP fuse attributes with items or sequential contexts, and pre-trained representations can be transferred to existing models.
5 EXPERIMENT
Experiments evaluate S^3-Rec on six datasets from four real-world platforms using standard top-k ranking metrics and leave-one-out evaluation. S^3-Rec consistently outperforms the compared baselines by a large margin across all six datasets.
- Datasets: The evaluation uses six datasets from four real-world platforms spanning different domains and sparsity levels.
- Datasets: The datasets include Meituan, three Amazon categories, Yelp, and LastFM, with item attributes drawn from categories, locations, keywords, brands, or tags.
- Experimental Setup: Performance is measured with HR@{1, 5, 10}, NDCG@{5, 10}, and MRR under a leave-one-out evaluation strategy.
- Baselines: The comparison includes non-sequential, sequential, and attribute-aware baselines, including PopRec, FM, AutoInt, GRU4Rec, Caser, SASRec, BERT4Rec, HGN, GRU4RecF, SASRecF, and FDSA.
- Experimental Results: SASRec and BERT4Rec outperform GRU4Rec and Caser, but directly concatenating attributes in GRU4RecF and SASRecF yields inconsistent improvements across datasets.
- Experimental Results: S^3-Rec performs consistently better than all baselines by a large margin on six datasets.
5.3 Further Analysis
Further analyses show that the self-supervised objectives contribute individually, transfer across recommendation models, help under sparse data, and improve training efficiency.
- Ablation Study: Removing any self-supervised objective decreases NDCG@10, confirming that all four objectives contribute to recommendation performance.The ablation evaluates Meituan, Beauty, Sports, and Toys, with FDSA included as a baseline.
- Ablation Study: AAP and MAP are more important than the other objectives across datasets, with larger performance drops after their removal.The authors suggest these objectives enhance item and sequence representations with attribute information.
- Applying Self-Supervised Learning to Other Models: All model variants enhanced by the self-supervised approach outperform the best baseline FDSA trained only with next-item prediction loss.The approach is applied to GRU4Rec, SASRec, AutoInt, and Caser, with architecture-specific adaptations.
- Applying Self-Supervised Learning to Other Models: After pre-training, all evaluated baselines improve on Beauty and Toys, while S3-Rec achieves the best performance among the compared models.The authors attribute S3-Rec's advantage partly to its bidirectional Transformer encoder during pre-training.
- Performance Comparison w.r.t. the Amount of Training Data: S3-Rec remains consistently better than baselines at every tested training-data proportion, especially with 20% of the data on Sports and Yelp.Performance declines as less training data is used, while the method is reported to alleviate data sparsity to some extent.
- Training and Convergence Analysis: The method benefits mostly from the first 20 pre-training epochs, then improves only slightly, and reaches its best performance after about 40 fine-tuning epochs.Pre-training captures correlations among attributes, items, subsequences, and sequences, while initialization helps faster convergence.
6 CONCLUSION
The paper concludes that S3-Rec uses mutual information maximization and four self-supervised objectives to enhance representations for sequential recommendation. Experiments show improvements over competitive baselines, while future work will explore additional objectives and more complex recommendation tasks.
- Conclusion: S3-Rec applies mutual information maximization with four objectives to learn correlations among attributes, items, segments, and sequences.The self-attentive recommender uses these learned correlations to enhance data representations.
- Future Work: Future work will investigate other self-supervised objectives and apply the approach to conversational and multimedia recommendation.