Intent Contrastive Learning for Sequential Recommendation

Yongjun Chen; Zhiwei Liu; Jia Li; Julian McAuley; Caiming Xiong

Source-linked AI summary

Intent Contrastive Learning for Sequential Recommendation

Yongjun Chen, Zhiwei Liu, Jia Li, Julian McAuley, Caiming Xiong

arXiv:2202.02519v1Feb 5, 2022cs.AI

TL;DR

Sequential recommendation models often miss latent user intents because intents lack labels and can differ from observed behavior patterns. ICL learns intent distributions by clustering and integrates them into sequential recommendation through contrastive self-supervised learning within a generalized EM framework. Across four datasets, ICLRec consistently improves recommendation performance and robustness to sparse or noisy interactions.

Problem
Latent user intents are unobserved, lack labels, may underlie distinct behavior sequences, and are difficult to integrate without redundant sequence information.
Method
ICL clusters user behavior sequences to learn intent distributions and alternates intent representation learning with sequential-recommendation optimization using contrastive self-supervised learning within an EM framework.
Results
ICLRec consistently outperforms existing methods across four datasets, with average gains over the best baseline ranging from 7.47% to 33.33% in HR and NDCG.
Takeaways & Limitations
ICL improves sequential recommendation and remains more robust under limited historical interactions and noisy individual sequences.
Takeaways & Limitations
The method has an overall training complexity dominated by 3 times the Transformer-based sequential-recommendation cost, while effective parallelization is left for future work.

Abstract

from arXiv · show

Users' interactions with items are driven by various intents (e.g., preparing for holiday gifts, shopping for fishing equipment, etc.).However, users' underlying intents are often unobserved/latent, making it challenging to leverage such latent intents forSequentialrecommendation(SR). To investigate the benefits of latent intents and leverage them effectively for recommendation, we proposeIntentContrastiveLearning(ICL), a general learning paradigm that leverages a latent intent variable into SR. The core idea is to learn users' intent distribution functions from unlabeled user behavior sequences and optimize SR models with contrastive self-supervised learning (SSL) by considering the learned intents to improve recommendation. Specifically, we introduce a latent variable to represent users' intents and learn the distribution function of the latent variable via clustering. We propose to leverage the learned intents into SR models via contrastive SSL, which maximizes the agreement between a view of sequence and its corresponding intent. The training is alternated between intent representation learning and the SR model optimization steps within the generalized expectation-maximization (EM) framework. Fusing user intent information into SR also improves model robustness. Experiments conducted on four real-world datasets demonstrate the superiority of the proposed learning paradigm, which improves performance, and robustness against data sparsity and noisy interaction issues.

1 INTRODUCTION

Users’ historical behaviors can reflect shared, unobserved intents, but existing intent modeling often depends on unavailable side information. ICL learns latent intent distributions from behavior sequences and integrates them into sequential recommendation through contrastive self-supervision.

Users’ purchasing behaviors may be driven by underlying intents that are not observed.
Shared intents can make users with different historical purchases seek the same future item, such as fishing swivels.
Existing intent-modeling methods often require side information such as user action types or item categories.
Latent-intent modeling is difficult because intents lack labels and distinct behaviors may reflect the same intent.
ICL learns intent distributions by clustering behavior sequences and fuses learned intents into SR with contrastive SSL.

2 RELATED WORK

Sequential recommendation models user behavior with methods ranging from Markov and factorization approaches to deep neural and Transformer architectures. Related recommendation research also explores intent modeling and contrastive self-supervised learning, including multi-task alternatives to two-stage training.

Early sequential recommendation methods model item-to-item patterns with Markov Chains, while FPMC combines sequential patterns with general interest.
Deep sequential recommenders include CNN-, RNN-, and Transformer-based models such as SASRec and BERT4Rec.
Recent intent-aware methods learn purchase purposes, multiple intents, or intent-guided neighbors for recommendation.
Contrastive SSL maximizes mutual information among positive data transformations while discriminating against negative examples.
S3-Rec uses two-stage pre-training and fine-tuning, whereas CL4SRec and CoSeRec use multi-task training to share information with SR.

3 PRELIMINARIES

Sequential recommendation encodes chronological user-item sequences to predict the next item, while contrastive SSL learns from transformed views of sequences. Latent-factor modeling introduces intents as unobserved variables and uses EM-style alternating optimization to address their missing observations.

3.1 Problem definition: Each user has a chronological interaction sequence, and next-item prediction selects the most likely subsequent item from the item set.
3.1 Problem definition: Sequences longer than a maximum length are truncated, while shorter sequences are left-padded with items.
3.2 Deep SR Models for Next Item Prediction: A deep sequence encoder maps historical behavior sequences to user-interest representations across position steps.
3.2 Deep SR Models for Next Item Prediction: Next-item prediction is optimized through likelihood or an adapted binary crossentropy objective, using sampled negatives when the item vocabulary is large.
3.3 Contrastive SSL in SR: Contrastive SSL creates two transformed views of one sequence as a positive pair and views from different sequences as negatives.
3.3 Contrastive SSL in SR: The transformed views are encoded, aggregated, and compared with InfoNCE to optimize the sequence encoder.
3.4 Latent Factor Modeling in SR: Latent intents are represented by a variable c, but estimating intents and model parameters creates a missing-variable chicken-and-egg problem.
3.4 Latent Factor Modeling in SR: Within generalized EM, intent representation learning estimates intent distributions in the E-step and contrastive SSL optimizes model parameters in the M-step.

4 METHOD

ICL alternates clustering-based intent representation learning with intent-aware contrastive optimization of a sequential recommendation model. It integrates intent contrastive learning, false-negative mitigation, and auxiliary sequence objectives within a generalized EM and multi-task training framework.

Intent Contrastive Learning: ICL estimates the intent distribution Q(c) by clustering sequence representations, then alternates intent estimation with model-parameter optimization in generalized EM.The E-step performs clustering, while the M-step updates model parameters using mini-batch gradient descent.
Intent Contrastive SSL: The method maximizes agreement between augmented sequence views and their corresponding intent prototypes, contrasting sequence representations with intents rather than only other sequence views.Sequence augmentation is optional for creating the intent view, although the paper applies augmentations to enlarge the training set.
False-Negative Mitigation: False-Negative Mitigation excludes minibatch users sharing the target user’s intent from negative comparisons.This addresses false negatives that arise when users in the same batch have the same intent.
Multi-Task Learning: The SR model jointly optimizes ICL, next-item prediction, and sequence-level self-supervised learning, with λ and β controlling the auxiliary-task strengths.The paradigm is implemented on a Transformer encoder as ICLRec and can also be applied to S3-Rec.
Complexity and Convergence: Training complexity is dominated by O(3 · (|U|^2d)) for three network objectives, while testing omits ICL and sequence-level SSL objectives.The authors note that Transformer-based computation can be parallelized and that generalized EM guarantees convergence.

5 EXPERIMENTS

Experiments on four public datasets evaluate ICLRec against sequential, self-supervised, and latent-factor baselines, including performance, efficiency, robustness, ablations, and sensitivity analyses. ICLRec consistently improves recommendation quality and shows benefits under sparse or noisy interactions.

Experimental Setup: Experiments use Sports, Beauty, Toys, and Yelp2 datasets, retaining users and items with at least five interactions.Predictions are ranked over the whole item set and evaluated with HR@5, HR@20, NDCG@5, and NDCG@20.
Experimental Setup: Baselines cover non-sequential matrix factorization, CNN-, RNN-, and Transformer-based sequential models, SSL-enhanced models, and latent-factor methods.The comparison includes BPR-MF, Caser, GRU4Rec, SASRec, BERT4Rec, S3-Rec_ISP, CL4SRec, and DSSRec.
Performance Comparison: ICLRec consistently outperforms existing methods on all datasets, with average improvements over the best baseline ranging from 7.47% to 33.33% in HR and NDCG.The authors attribute the gains to estimating intent distributions and fusing them into sequential recommendation through contrastive SSL.
Efficiency: 11.75 s/epoch is ICLRec’s update time on Sports, compared with 6.52 s/epoch for CL4SRec and 3.59 s/epoch for SASRec.ICLRec spends 3.21 seconds on intent representation learning and 8.54 seconds on multi-task learning; test-set evaluation times are about 12.72 seconds for all three models.
Robustness Analysis: ICLRec consistently outperforms SASRec across user groups, while its advantage over CL4SRec is mainly concentrated among users with low interaction frequency.CL4SRec fails to outperform SASRec for short sequences in Beauty and Yelp, whereas ICLRec remains beneficial for users with limited histories.
Robustness and Ablation Analysis: ICLRec’s performance drop ratio under noisy interactions is consistently lower than CL4SRec’s on Sports and Yelp, and its 15% noisy Sports result exceeds CL4SRec without noise.The study also reports a 41.11% average NDCG@20 improvement when adding ICL to S3-Rec_ISP across four datasets.

6 CONCLUSION

ICL models latent intent factors from user interactions and fuses them into sequential recommendation through a contrastive self-supervised objective. Experiments demonstrate its effectiveness on four datasets.

ICL models latent intent factors from user interactions and integrates them into sequential recommendation through a contrastive SSL objective.

A PSEUDO-CODE OF ICL FOR SR

The ICL training procedure alternates intent representation learning with multi-task optimization. It constructs two augmented sequence views, encodes them, and updates the intent distribution during training.

The algorithm alternates E-step intent representation learning with M-step multi-task learning until the maximum training epochs.
It updates the intent distribution function Q(c_i) using P_θ(c_i|S_u) during the M-step.
The procedure constructs two views of each user sequence using independently sampled augmentations.
The sequence encoder f_θ maps both augmented views to their corresponding representations.

B PROOF OF CONVERGENCE

The convergence proof shows that successive generalized-EM training iterations do not decrease the objective. Therefore, the ICL algorithm converges.

The proof establishes P_θ^(m+1)(S) ≥ P_θ^m(S) across successive training iterations.
Taking expectations conditioned on c and applying Johnson’s inequality yields the required monotonicity relation.
Because the objective is non-decreasing across iterations, the algorithm converges.

C DATASET INFORMATION

The paper reports dataset information in Table 3.

Table 3 presents the dataset information.

D IMPACT OF BATCH SIZE AND THE STRENGTH OF SEQCL TASK 𝛽

ICLRec remains stable across Yelp batch sizes and outperforms CL4SRec, while SeqCL helps only at small β and harms performance when over-weighted.

Impact of batch size: ICLRec outperforms CL4SRec across all tested Yelp batch sizes and remains relatively stable as batch size increases.The paper attributes this stability to learned intents helping identify true positives and reduce false-negative effects in contrastive learning.
Impact of SeqCL task strength β: Small SeqCL task strengths, such as β≤0.1, improve ICLRec performance.The reported improvement occurs when β is small.
Impact of SeqCL task strength β: Increasing β beyond the small-strength range causes ICLRec performance to drop significantly.The paper identifies this behavior as a limitation of SeqCL.
Impact of SeqCL task strength β: The SeqCL limitation is that maximizing mutual information between individual sequence pairs may break global relationships among users.This explains why excessive SeqCL weighting can reduce model performance.

E CASE STUDY

The case study examines category-based intent representations and visualizes user embeddings, finding that ICLRec captures stronger semantic structure than CL4SRec.

Quantitative analysis: ICLRec-A replaces learned intent representation with the mean of trainable item-category embeddings as each sequence’s intent prototype.The comparison is reported in the quantitative analysis using NDCG@20.
Qualitative analysis: ICLRec pulls users who interacted with the same item category closer and pushes users from different categories farther apart than CL4SRec.This comparison uses t-SNE visualizations of representations from 100 sampled users.
Qualitative analysis: The learned ICLRec representations capture more semantic structures, which the paper associates with improved performance.The case study compares ICLRec and CL4SRec representations using t-SNE.