PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Self-supervised Sequential Recommendation with Robust Augmentation

Zhiwei Liu, Yongjun Chen, Jia Li, Philip S. Yu, Julian McAuley, Caiming Xiong

arXiv:2108.06479v1cs.IRcs.AI

TL;DR

Sparse and noisy interactions, together with discrete correlated sequences and skewed lengths, make contrastive SSL challenging for sequential recommendation. CoSeRec addresses these issues with correlation-aware, length-aware augmentations jointly trained with next-item prediction, and experiments report improved performance and robustness on three datasets.

  • Problem

    Sparse and noisy interactions impair sequential recommendation, while discrete items, item correlations, and skewed sequence lengths complicate contrastive augmentation.

  • Method

    CoSeRec jointly optimizes next-item prediction and contrastive SSL using substitute and insert augmentations that leverage item correlations and sequence length.

  • Results

    Experiments on three benchmark datasets demonstrate CoSeRec’s effectiveness and robustness against sparse and noisy data.

  • Takeaways & Limitations

    Informative, length-aware augmentations provide a basis for applying contrastive SSL to sequential recommendation while preserving sequential correlations.

  • Takeaways & Limitations

    The paper identifies future work in more advanced model-based item-correlation functions and finer-grained augmentation–sequence-length analysis.

Abstract

from arXiv · show

Sequential Recommendationdescribes a set of techniques to model dynamic user behavior in order to predict future interactions in sequential user data. At their core, such approaches model transition probabilities between items in a sequence, whether through Markov chains, recurrent networks, or more recently, Transformers. However both old and new issues remain, including data-sparsity and noisy data; such issues can impair the performance, especially in complex, parameter-hungry models. In this paper, we investigate the application of contrastive Self-Supervised Learning (SSL) to the sequential recommendation, as a way to alleviate some of these issues. Contrastive SSL constructs augmentations from unlabelled instances, where agreements among positive pairs are maximized. It is challenging to devise a contrastive SSL framework for a sequential recommendation, due to its discrete nature, correlations among items, and skewness of length distributions. To this end, we propose a novel framework, Contrastive Self-supervised Learning for sequential Recommendation (CoSeRec). We introduce two informative augmentation operators leveraging item correlations to create high-quality views for contrastive learning. Experimental results on three real-world datasets demonstrate the effectiveness of the proposed method on improving model performance and the robustness against sparse and noisy data. Our implementation is available online at \url{https://github.com/YChen1993/CoSeRec}

1 INTRODUCTION

Sequential recommendation models next-item transitions from ordered user interactions, but sparse and noisy data challenge this goal. CoSeRec applies contrastive SSL with augmentations designed for discrete correlated sequences and skewed lengths.

  • Motivation: Sequential recommendation predicts a user’s next interaction from recent, temporally ordered item observations.Existing approaches model item relationships with Markov chains, RNNs, or Transformers.
  • Challenges: Sparse datasets undermine Transformer-based modeling, particularly for short sequences.Complex models are motivated by large NLP corpora, whereas sequential recommendation commonly has limited interaction data.
  • Challenges: Noisy interactions can misrepresent true item correlations and produce less satisfying recommendations.Promotional consumption is given as an example of an interaction that may yield negative feedback.
  • Contrastive SSL: Contrastive SSL maximizes agreement between augmented views of the same unlabeled instance.The approach uses positive pairs formed from different augmentations of one sequence.
  • Challenges: Sequential recommendation makes contrastive augmentation difficult because items are discrete, correlated, and distributed across skewed sequence lengths.Random perturbations may destroy item relationships and produce less confident positive pairs, especially for short sequences.
  • Contribution: CoSeRec combines correlation-aware augmentation, a contrastive objective, and multi-task optimization of recommendation and SSL objectives.Its substitute and insert operators leverage item correlations rather than relying only on random crop, reorder, or mask operations.

2 RELATED WORK

Sequential recommendation has progressed from transition-based models to RNNs and Transformers, while contrastive SSL has expanded across modalities and recommendation. Prior sequence augmentations nevertheless overlook item correlations and length skewness.

  • Sequential Recommendation: Sequential recommendation methods evolved from Markov chains to RNNs and then Transformer-based sequence encoders.These approaches progressively model pairwise, sequence-level, and complex item-transition correlations.
  • Sequential Recommendation: Limited receptive fields restrict some transition functions to shallow item correlations.This limits their ability to encode sequences fully.
  • Sequential Recommendation: Transformer-based SR models use self-attention to characterize complex item transition correlations.SASRec learns item importance, while BERT4Rec applies bidirectional Transformer encoding.
  • Contrastive SSL: Contrastive SSL trains encoders by maximizing agreement between two augmented views of one instance.The paradigm has been applied in vision, language, graphs, and recommender systems.
  • Contrastive SSL: Recommendation methods such as S3-Rec and related approaches use contrastive SSL with masks, feature dropping, or sequence augmentations.These methods extend pairwise contrastive learning into self-supervised recommendation training.
  • Research Gap: Existing sequence augmentation methods ignore crucial item correlations and sequence-length skewness.This limitation motivates more informative and length-aware augmentations for sequential recommendation.

3 PRELIMINARIES

Sequential recommendation represents each user’s chronological item history and predicts the next item. A Transformer encoder models sequence correlations and is trained with a next-item prediction loss.

  • Problem Formulation: Each user is associated with a chronological item sequence, and the task is to predict the next item.The sequence length is denoted by |s_u|, with the target item v_|s_u|+1.
  • Problem Formulation: The recommendation formulation evaluates candidate-item probabilities and selects the highest-probability item.This describes the operational prediction step after sequence encoding.
  • Transformer for SR: A Transformer encoder models item correlations using multi-head self-attention and produces position-wise embeddings.The position-wise Feed-Forward Network outputs embeddings used for next-item prediction.
  • Transformer for SR: The encoder is optimized with a log-likelihood next-item prediction loss.Each position’s embedding predicts the corresponding next item in the sequence.
  • Transformer for SR: The recommendation loss compares the next-item embedding with sampled negative items through a nonlinear activation.Item embeddings are retrieved from the encoder’s embedding table and jointly optimized with the Transformer.

4 METHODOLOGY

CoSeRec combines random and informative sequence augmentations with contrastive self-supervised learning for sequential recommendation. It uses item correlations and sequence length to construct more reliable views and jointly trains recommendation and contrastive objectives.

  • Data Augmentation: CoSeRec first reviews crop, mask, and reorder operators, then introduces informative substitute and insert augmentations for sequence views.The methodology organizes augmentations into random and correlation-aware operators before describing contrastive SSL and training.
  • Random Augmentation: Random crop, mask, and reorder can break sequential item correlations, with the effect amplified for short sequences.Reordering changes correlations directly, while perturbations are more damaging when sequences contain fewer items.
  • Informative Augmentation: Substitute replaces selected items with correlated items, while insert adds the most correlated item around selected positions to complete sequences.Substitution uses a substitution ratio; insertion produces an augmented sequence of length k + n.
  • Informative Augmentation: These informative augmentations perturb sequences with correlated items, increasing positive-pair confidence and extending interactions to alleviate cold-start issues.The operators are designed to preserve more sequential information while adding correlated interactions.
  • Item Correlations: CoSeRec fuses memory-based and model-based item correlations by taking the highest normalized value, initially relying on memory-based scores during training.After E epochs, the method switches to the hybrid correlation because early item representations are not informative.
  • Length-Aware Augmentation: For sequences of length at most K, CoSeRec selects from substitute, insert, and mask; longer sequences may also use crop and reorder.Masking remains available for short sequences because it models item relationships and encourages nearby items to become closer.
  • Contrastive Self-Supervised Learning: Contrastive SSL samples two augmentation operators per sequence, encodes the resulting views with Transformer outputs, and maximizes agreement between positive views.Sequence representations concatenate position-wise Transformer embeddings before contrastive optimization.

5 EXPERIMENTS

The experiments evaluate CoSeRec on three public datasets and address performance, augmentation choices, robustness to sparsity and noise, and sensitivity to settings.

  • Experiments use three public datasets to evaluate CoSeRec and investigate four research questions.The questions cover overall sequential-recommendation performance, optimal augmentations, robustness to data sparsity and noisy interactions, and the influence of settings.

5.1 Experimental Setting

The study evaluates next-item prediction on preprocessed 5-core datasets using whole-item-set ranking and standard top-k metrics, against sequential and non-sequential baselines.

  • Datasets: Beauty and Sports are Amazon review subcategories, while Yelp1 is a business-recommendation dataset.
  • Datasets: The preprocessing retains 5-core datasets, requiring every user and item to have at least five interactions.Ratings or reviews are treated as positive instances, and other interactions as negative instances.
  • Evaluation: Next-item prediction ranks candidates over the whole item set without negative sampling.Performance uses HR@k and NDCG@k for k in {5, 10, 20}.
  • Baselines: Baselines include popularity and matrix-factorization models, recurrent and convolutional sequential models, Transformer-based models, and self-supervised methods.The Transformer-based group includes SASRec, Bert4Rec, and S3-Rec.
  • Implementation: CoSeRec uses two self-attention blocks, two attention heads, 64-dimensional embeddings, and a maximum sequence length of 50.The model is optimized with Adam at learning rate 0.001 and uses early stopping after 40 unimproved validation epochs.

5.2 Overall Performance Comparison (RQ1)

CoSeRec consistently outperforms the compared models across all three datasets and evaluation metrics, while sequential and Transformer-based methods generally outperform simpler alternatives.

  • Overall Comparison: CoSeRec consistently outperforms other models on all datasets and evaluation metrics.Its improvements over the best baseline range from 3.54% to 30.88% in HR and NDCG.
  • Overall Comparison: Transformer-based methods perform better than other sequential-recommendation model types.The authors associate this with self-attention being more effective at capturing sequential patterns than CNNs and RNNs.
  • Overall Comparison: Non-sequential models perform worse than sequential-recommendation methods for next-item prediction.
  • Overall Comparison: CL4SRec performs better than the other baselines but worse than CoSeRec.The paper attributes CoSeRec’s advantage to its informative augmentations.

5.3 Augmentation Analysis (RQ2)

Augmentation analyses favor CoSeRec’s correlation-informed operators, especially for short sequences, and support using different augmentation sets according to sequence length.

  • Leave-One-Out Comparison: Removing Insert or Substitute consistently decreases performance on Sports and Beauty.Removing Crop or Reorder can significantly increase performance on Beauty.
  • Pair-Wise Comparison: 0.0307 NDCG@5 is achieved by the (I, S) pair on Beauty, versus 0.0255 for (I, I) and 0.0262 for (S, S).
  • Pair-Wise Comparison: Different augmentation operators generally outperform using two copies of the same operator.The analysis interprets multiple views as beneficial for capturing mutual information from different views.
  • Short Sequences: For short sequences, every combination containing informative augmentations outperforms CL4SRec.Adding M produces the largest improvement among the tested combinations.
  • Short Sequences: {S, I, M} outperforms {S, I, M, R, C} for short sequences.The result supports applying different augmentation operators to short and long sequences because short sequences are more sensitive to randomness.

5.4 Robustness Analysis (RQ3)

CoSeRec is evaluated under reduced training data and noisy test interactions to assess robustness. Its informative augmentations preserve performance better than CL4SRec as sparsity increases and support robustness to noise.

  • Sparsity and noise settings: The robustness analysis varies training-data availability and compares CoSeRec with CL4SRec on Sports and Beauty.Training data is reduced to 25%, 50%, 75%, and 100% while test data remains unchanged; noisy interactions are separately added to test sequences.
  • Sparsity robustness: At 50% training data, CL4SRec loses 76.19% of its original performance, whereas CoSeRec loses 34% on Sports.On Beauty, CoSeRec’s performance drops 52.14% under the same 50% training-data condition.
  • Noise robustness: Additional Insert and Substitute augmentations create higher-confidence positive views that improve encoder robustness to noisy interactions during inference.

5.5 Study of CoSeRec (RQ4)

The study examines CoSeRec’s augmentation, correlation, and training choices. Performance peaks at dataset-dependent augmentation settings, benefits from item correlations and contrastive learning, and favors multi-task over two-stage training.

  • Study design: The study varies Substitute and Insert ratios, the short-sequence threshold K, contrastive intensity λ, item-correlation types, and training strategy.
  • Hyperparameter study: Contrastive learning improves NDCG@5 by 32.88% on Sports and 12.82% on Beauty, but performance deteriorates when λ exceeds 0.1.The best K values are 4 and 12 on Sports and Beauty, respectively; β=0.4 and α=0.1 are reported as best ratios for the respective datasets.
  • Item correlations: Any item-correlation type improves performance over CL4SRec, while model-based correlation performs worse than memory-based and hybrid correlations.
  • Training strategy: Multi-task training outperforms the alternative two-stage strategy that pre-trains with contrastive SSL and then fine-tunes for next-item prediction.

6 CONCLUSION AND FUTURE WORK

The paper concludes that CoSeRec combines contrastive SSL, informative correlation-based augmentations, and sequence-length awareness to address sparsity and noisy interactions. Future work targets more advanced correlation functions, finer augmentation–length analysis, and alternative contrastive objectives.

  • Conclusion: CoSeRec jointly optimizes contrastive SSL and next-item prediction while using Substitute and Insert augmentations based on item correlations.
  • Conclusion: The framework adds sequence-length awareness to address length skewness and reports effectiveness and robustness across three benchmark datasets.
  • Future work: Future work will investigate reinforcement-learning-based correlation search, finer relations between augmentations and sequence length, and different contrastive learning functions.
Loading 2108.06479v1…