PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Learning for Recommender Systems: A Survey

Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Jundong Li, Zi Huang

arXiv:2203.15876v2cs.IR

TL;DR

Deep recommender systems achieve strong performance but depend on costly user-generated data, especially under sparse feedback. The survey systematically reviews self-supervised recommendation by defining SSR, organizing methods into four categories, and providing SELFRec for empirical comparison; it reports findings on self-supervised signals while identifying current limitations and future directions.

  • Problem

    Deep recommendation models require abundant training data, while recommendation-specific SSL lacked a systematic survey despite growing research activity.

  • Method

    The paper defines SSR, develops a four-category taxonomy, reviews methods and trade-offs, and releases SELFRec with datasets, metrics, and SSR implementations.

  • Results

    Experiments report that feature-level noise augmentations perform best on average, while contrastive methods generally outperform predictive methods in graph recommendation and BERT4Rec underperforms SASRec in sequential recommendation.

  • Takeaways & Limitations

    The survey provides a structured basis for comparing SSR approaches and selecting self-supervised signals for recommendation.

  • Takeaways & Limitations

    Existing SSR performance gains are often theoretically unjustified and poorly interpretable, while on-device SSR remains relatively unexplored.

Abstract

from arXiv · show

In recent years, neural architecture-based recommender systems have achieved tremendous success, but they still fall short of expectation when dealing with highly sparse data. Self-supervised learning (SSL), as an emerging technique for learning from unlabeled data, has attracted considerable attention as a potential solution to this issue. This survey paper presents a systematic and timely review of research efforts on self-supervised recommendation (SSR). Specifically, we propose an exclusive definition of SSR, on top of which we develop a comprehensive taxonomy to divide existing SSR methods into four categories: contrastive, generative, predictive, and hybrid. For each category, we elucidate its concept and formulation, the involved methods, as well as its pros and cons. Furthermore, to facilitate empirical comparison, we release an open-source library SELFRec (https://github.com/Coder-Yu/SELFRec), which incorporates a wide range of SSR models and benchmark datasets. Through rigorous experiments using this library, we derive and report some significant findings regarding the selection of self-supervised signals for enhancing recommendation. Finally, we shed light on the limitations in the current research and outline the future research directions.

1 INTRODUCTION

Recommendation models have achieved strong results but require abundant user data, while recommendation-specific SSL has grown without a systematic survey. This paper addresses that gap by defining SSR, organizing its methods, providing SELFRec, and identifying limitations and future directions.

  • Deep recommendation models are data-hungry, while user-generated recommendation data is costly to acquire.
  • Despite growing SSR research, prior SSL surveys did not systematically investigate recommendation-specific methods and challenges.Recommendation spans varied scenarios, objectives, and data types, alongside skewed distributions, biases, and large-vocabulary categorical features.
  • The survey reviews SSR research and introduces a precise definition plus a taxonomy of contrastive, generative, predictive, and hybrid methods.It discusses each category’s concepts, formulations, methods, strengths, and limitations.
  • SELFRec supports SSR implementation and evaluation with benchmark datasets, evaluation metrics, and more than 20 state-of-the-art methods.The library is introduced to facilitate empirical comparison and derive findings about effective self-supervised recommendation.
  • The paper identifies limitations in existing SSR research and outlines remaining challenges and future directions.

2 DEFINITION AND TAXONOMY

The survey defines self-supervised recommendation (SSR) around extracting supervision from raw data, training with augmented data, and using self-supervision to enhance recommendation. It organizes SSR into contrastive, generative, predictive, and hybrid paradigms.

  • Definition: SSR incorporates self-supervised tasks and augmented data to pre-train or train recommendation models.Augmented data is generated by transforming original data, such as dropping graph edges.
  • Definition: The self-supervised task is auxiliary: it is designed to enhance recommendation performance rather than serve as the end goal.
  • Definition: SSR extracts additional supervision signals from raw data to complement sparse explicit feedback.
  • Taxonomy: The taxonomy divides SSR models into contrastive, predictive, generative, and hybrid categories according to their self-supervised tasks.
  • Contrastive Methods: Contrastive methods pull augmented variants of the same instance together and push variants of different instances apart.Their self-supervised loss estimates mutual information through representations learned by a shared encoder.
  • Generative, Predictive, and Hybrid Methods: Generative methods reconstruct corrupted user or item profiles, whereas predictive methods generate samples or labels to guide the pretext task.Hybrid methods combine multiple self-supervised tasks, often using multiple encoders and projection heads.

3 DATA AUGMENTATION

SSR uses task-independent, model-agnostic augmentations across sequences, graphs, and features to create varied self-supervised views. These transformations include corruption, sampling, diffusion, perturbation, and mixing operations.

  • Overview: SSR augmentation methods are classified as sequence-based, graph-based, or feature-based and are mostly task-independent and model-agnostic.
  • Sequence-Based Augmentation: Sequence augmentations include item masking, cropping, reordering, insertion, and substitution.Masking retains primary intent information, cropping provides a local sequence view, and correlated-item operations address short sequences.
  • Graph-Based Augmentation: Graph augmentations use edge or node dropout, graph diffusion, and subgraph sampling to construct altered graph views.Diffusion can add top-K similarity edges to represent possible missing preferences, while subgraph sampling focuses on local structures.
  • Feature-Based Augmentation: Feature augmentations operate in attribute or embedding space through dropout, shuffling, perturbation, clustering, and mixing.Feature perturbation preserves most original information while introducing small differences, and feature mixing synthesizes examples.

4 CONTRASTIVE METHODS

Contrastive SSR creates multiple views of recommendation data and learns representations by contrasting related and unrelated structures, features, models, or contexts. The survey covers local, global, and local-global contrast across graph and sequence settings.

  • Contrastive Task Taxonomy: Contrastive pretext tasks are grouped into structure-level, feature-level, and model-level contrast according to their self-supervision signals.
  • Structure-Level Contrast: Structure-level contrast exploits similar semantics under perturbed graph or sequence structures through same-scale and cross-scale comparisons.
  • Local-Level Graph Contrast: SGL contrasts two augmented user-item graphs with shared LightGCN encodings using InfoNCE and jointly optimizes the contrastive and BPR losses.
  • Global-Level Contrast: Global-level sequence contrast compares two augmented sequences after aggregating their item representations into sequence representations.CL4SRec uses masking, cropping, and reordering to generate augmented sequence pairs and in-batch negatives.
  • Local-Global Contrast: Local-global contrast aligns local graph structures with global graph representations generated by a readout function.EGLN additionally uses graph diffusion, retaining top-K user-item similarities and iteratively updating the graph and representations.
  • Local-Context Contrast: Local-context contrast compares graph or sequence elements with contexts formed through ego-network sampling or clustering.NCL uses K-means prototypes and recursively adjusts them with expectation-maximization.

4.2 Feature-Level Contrast

Feature-level contrast creates augmented views by modifying input features or learned representations, but remains less explored because academic recommendation datasets often lack rich feature information. Model-level contrast instead perturbs the architecture or hidden representations while keeping the input unchanged.

  • Feature-Level Contrast: Feature-level contrast modifies input features or learned representations to create augmented views for recommendation.Its use is limited in academic datasets because feature and attribute information is often scarce.
  • Feature-Level Contrast: SL4Rec masks correlated item features using mutual information and applies dropout, making the contrastive task more challenging.Retained features may not recover the semantics of jointly masked correlated features.
  • Model-Level Contrast: Model-level contrast dynamically perturbs the model architecture or hidden representations to generate view pairs from unchanged inputs.Perturbed encoders are treated as a special case of augmenting intermediate hidden representations.
  • Model-Level Contrast: DuoRec uses different Transformer dropout masks, while SimGCL and XSimGCL add uniform noise to hidden representations.SimGCL and XSimGCL target more uniform node representations, and XSimGCL additionally introduces cross-layer contrast.
  • Contrastive Objective: Contrastive loss commonly maximizes mutual information through lower-bound estimators such as Jensen-Shannon divergence and InfoNCE.InfoNCE identifies positive samples among negatives and its NT-Xent form uses temperature τ.
  • Contrastive Objective: InfoNCE promotes representation alignment and uniformity, but similar users or items can become false negatives that impair recommendation performance.Multiple-positive variants address this issue by adding semantically positive samples to the loss.

4.5 Pros and Cons

Contrastive self-supervised recommendation is flexible and effective, but its success depends on choosing informative augmentations. The field lacks rigorous principles for selecting such augmentations, and some commonly used choices can hurt performance.

  • Pros: Contrastive methods cover many recommendation topics and can improve recommendation with lightweight architectures.Their flexibility comes from varying data augmentations and pretext tasks.
  • Cons: High-quality recommendation augmentations lack clear criteria, so existing methods often select them through trial and error.The paper notes that the field has neither rigorous understanding nor clear guidelines for augmentation quality.
  • Cons: Some augmentations previously considered useful have recently been shown to negatively affect recommendation performance.When augmentations are uninformative, the resulting contrastive task may fail.

5 GENERATIVE METHODS

Generative SSR reconstructs corrupted recommendation data and is organized around structure generation and feature generation. BERT-like masked prediction dominates recent work, but scaling Transformer-based pre-training can be computationally demanding.

  • Overview: Generative SSR reconstructs original inputs from corrupted versions and comprises structure-generation and feature-generation methods.Structure generation reconstructs corrupted sequences or graphs, whereas feature generation reconstructs attributes or representations.
  • Structure Generation: BERT4Rec randomly masks sequence items and predicts them from surrounding items using bidirectional representations.Its masked-item objective became widely used in BERT-like sequential recommendation models.
  • General-Purpose Pre-training: Generative pre-training can learn general-purpose representations for multiple downstream recommendation tasks.PeterRec and ShopperBERT exemplify this direction, beyond models pre-trained for one specific task.
  • Structure Generation: G-BERT combines graph neural networks with BERT and pre-trains using self-prediction and dual-prediction tasks.The tasks reconstruct masked codes from the same graph type and from the other graph type, respectively.
  • Feature Generation: Feature generation reconstructs user attributes, item textual features, or learned user/item representations with regression objectives.PMGT masks sampled nodes and recovers their features from remaining nodes, while another approach predicts future-sequence representations from past behaviors.
  • Pros and Cons: Transformer-based generative SSR methods may require intensive computation, especially when trained on large datasets or used to build general-purpose models.Current methods are often trained on small datasets with only one or two Transformer blocks, while larger-scale pre-training is substantially more demanding.

6 PREDICTIVE METHODS

Predictive SSR generates supervisory signals from complete original data by predicting informative samples or pseudo-labels. Its dynamic signals can align with evolving optimization objectives, but heuristic pseudo-labels require caution.

  • Overview: Predictive SSR uses self-generated signals from complete data and divides methods into sample prediction and pseudo-label prediction.Pseudo-label prediction includes relation classification and similarity regression.
  • Sample Prediction: Sample-prediction methods pre-train on original data, predict informative samples, and use them to enhance recommendation or generate better samples recursively.This connects SSR with self-training while retaining recommendation-specific sample generation.
  • Sample Prediction: ASReP augments short sequences with pseudo-prior items predicted by a Transformer pre-trained from right to left.The method targets poor performance on short sequences caused by limited user behaviors.
  • Sample Prediction: BiCAT jointly trains left-to-right and right-to-left directions to reduce inconsistency between reverse augmentation and forward recommendation.Predictive graph methods can likewise use node-feature or semantic similarities to generate samples.
  • Pseudo-Labels Prediction: Pseudo-label methods predict predefined relations with classification or continuous targets with regression.Continuous targets may represent attributes, probability distributions, or feature vectors generated by encoders or pre-computed actions.
  • Pseudo-Labels Prediction: BUIR uses asymmetric online and target graph encoders that bootstrap each other without negative sampling.The online network updates end-to-end, while the target encoder is updated through momentum-based updates.
  • Pros and Cons: Predictive methods generate more dynamic and flexible signals, but heuristic pseudo-labels can introduce risks that require caution.Their signals evolve with model parameters and may better align with the recommendation objective.

7 HYBRID METHODS

Hybrid self-supervised recommendation methods combine multiple pretext tasks, either collaboratively or in parallel, to provide more comprehensive supervision. Their flexibility can improve training effectiveness, but coordinating tasks increases complexity and cost.

  • 7 HYBRID METHODS: Hybrid methods combine multiple pretext tasks and are categorized as Collaborative SSR or Parallel SSR.Collaborative methods let generative or predictive tasks support contrastive learning, whereas parallel methods operate without correlations between tasks.
  • 7 HYBRID METHODS: CCL links masked-item prediction with contrastive learning, using predicted probabilities to augment sequences and curriculum learning to order contrastive difficulty.
  • 7 HYBRID METHODS: SEPT uses three graph views whose encoders predict semantically similar samples for one another as positive signals in a recursively improved contrastive task.
  • 7 HYBRID METHODS: Hybrid methods can improve training effectiveness when generative or predictive tasks dynamically generate samples for contrastive learning.
  • 7 HYBRID METHODS: Hybrid models require balancing interacting tasks, often through manual hyperparameter search or more complex architectures, and therefore incur higher training costs.

8 SELFREC: A LIBRARY FOR SELF-SUPERVISED RECOMMENDATION

SELFRec is an SSR-specific library created to address unreliable comparisons caused by inconsistent evaluation settings and limited suitability of general-purpose repositories. It provides an architecture and modular implementation environment for evaluating more than 20 SSR methods across benchmark datasets and metrics.

  • 8 SELFREC: A LIBRARY FOR SELF-SUPERVISED RECOMMENDATION: SELFRec addresses invalid SSR comparisons caused by inconsistent experimental settings, random hyperparameters, and modified baseline results.
  • 8 SELFREC: A LIBRARY FOR SELF-SUPERVISED RECOMMENDATION: SELFRec provides an architecture specialized for implementing self-supervised recommendation models.
  • 8 SELFREC: A LIBRARY FOR SELF-SUPERVISED RECOMMENDATION: The library includes benchmark datasets for general and sequential recommendation, more than 10 evaluation metrics, and more than 20 SSR methods.
  • 8 SELFREC: A LIBRARY FOR SELF-SUPERVISED RECOMMENDATION: SELFRec supports fast GPU execution, plug-and-play model expansion, modular development, and dedicated modules for data augmentations and self-supervised tasks.

9 EXPERIMENTAL FINDINGS

The experiments compare augmentation strategies and representative SSR models under controlled LightGCN and Transformer backbones. Feature-level augmentation, especially feature noise, performs strongly across scenarios, while model and structure augmentations depend more on dataset or scenario; contrastive methods lead in graphs, whereas sequential methods remain less effective overall.

  • 9 EXPERIMENTAL FINDINGS: The experiments control backbone effects by using LightGCN for graph models and Transformer for sequential models.
  • 9.1 Comparison of Data Augmentations in CL: Feature-level augmentations are highly effective in both graph and sequential scenarios, with feature noise producing the highest average improvement.
  • 9.1 Comparison of Data Augmentations in CL: Structure-level augmentations are less effective on sparse datasets, may degrade sequential performance, and can help on denser datasets.
  • 9.1 Comparison of Data Augmentations in CL: Model-level augmentation effectiveness varies across datasets, ranging from considerable improvement to minimal improvement.
  • 9.1 Comparison of Data Augmentations in CL: Sequential contrastive augmentations are less effective than graph counterparts, possibly because item transitions lack clear semantics for Transformer structures.
  • 9.2 Comparison of SSR Models: In graph recommendation, contrastive SSR performs best, generative SSR provides decent improvement, and predictive SSR performs poorly on datasets without attribute information.
  • 9.2 Comparison of SSR Models: In sequential recommendation, contrastive and predictive methods show similar improvements, while BERT4Rec performs much worse than SASRec without fine-tuning.
  • 9.2 Comparison of SSR Models: Sequential SSR methods still have room for improvement, and many are less effective than reported in their original papers.

10 DISCUSSION

The discussion identifies unresolved limitations in self-supervised recommendation, including weak theoretical grounding, uncertain robustness, and underexplored deployment and pre-training settings. It proposes recommendation-specific theory, explainability, security, on-device methods, and more efficient general-purpose pre-training as future directions.

  • 10 DISCUSSION: Existing SSR research lacks a systematic investigation of its limitations and future directions.The discussion section frames these limitations as areas requiring further study.
  • 10.1 Theory for Augmentation Selection: Current augmentation methods often rely on heuristics borrowed from other fields and require cumbersome trial-and-error selection.Recommendation behavior is tightly coupled to scenarios and includes noise and randomness, so transferred heuristics do not seamlessly apply.
  • 10.2 Explainable Self-Supervised Recommendation: Most SSR performance gains lack theoretical justification, while augmentations and self-supervised objectives remain difficult to interpret.Some graph augmentations previously considered informative may impair performance, and possible trade-offs with robustness remain unclear.
  • 10.3 Robust Self-Supervised Recommendation: SSR robustness to data-poisoning attacks remains unknown despite established attacks and defenses for supervised recommender systems.The discussion calls for new attack strategies and corresponding defenses for SSR models.
  • 10.4 On-Device Self-Supervised Recommendation: On-device SSR is less explored even though decentralized recommendation faces compressed models and limited labeled data.Knowledge distillation is identified as a possible complement to SSL for compensating accuracy degradation.
  • 10.5 Towards General-Purpose Pre-Training: General-purpose multimodal pre-training could support multiple recommendation tasks, but existing efforts are mostly BERT-like and need more efficient strategies and architectures.The proposed setting is especially relevant to scenarios with sparse training data and cheap downstream fine-tuning.

11 CONCLUSION

The survey reviews the state of self-supervised recommendation and organizes existing methods into a taxonomy. It also provides an empirical-comparison library, reports findings on self-supervised signals, and outlines directions addressing current limitations.

  • 11 CONCLUSION: The survey provides a comprehensive review of the current state of the art in self-supervised recommendation.
  • 11 CONCLUSION: It categorizes existing SSR methods and releases an open-source library for empirical comparison.The library supports evaluation of SSR models using benchmark datasets and metrics.
  • 11 CONCLUSION: The survey reports findings about selecting self-supervised signals and outlines future research directions addressing current limitations.
Loading 2203.15876v2…