PaperlayerView exact revision on arXiv

Source-linked AI summary

SelfCF: A Simple Framework for Self-supervised Collaborative Filtering

Xin Zhou, Aixin Sun, Yong Liu, Jie Zhang, Chunyan Miao

arXiv:2107.03019v3cs.IRcs.AIcs.LG

TL;DR

Negative sampling in implicit-feedback CF is costly and can treat future positives as negatives. SelfCF creates contrastive views by perturbing encoded user and item embeddings, without using negative samples. Across four datasets, it is competitive with or better than supervised counterparts and outperforms a self-supervised framework by up to 17.79% on average.

  • Problem

    Negative sampling for implicit-feedback CF is computationally costly and may select unobserved pairs that are future positives as negatives.

    Source: §1 · p. 2

  • Method

    SelfCF perturbs backbone output embeddings to create contrastive views and learns user and item representations from positive interactions without negative samples.

    Source: §1 · p. 4, §3 · p. 6

  • Results

    17.79% on average: SelfCF outperforms existing self-supervised frameworks and is competitive with or better than supervised counterparts in the reported experiments.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Posterior embedding perturbation lets SelfCF treat backbone implementations as black boxes while learning informative representations from positive pairs.

    Source: §1 · p. 4

    SelfCF uses only positive user–item interactions; combining it with supervised signals is left as a future direction.

    Source: §6 · p. 20

Abstract

from arXiv · show

Collaborative filtering (CF) is widely used to learn informative latent representations of users and items from observed interactions. Existing CF-based methods commonly adopt negative sampling to discriminate different items. Training with negative sampling on large datasets is computationally expensive. Further, negative items should be carefully sampled under the defined distribution, in order to avoid selecting an observed positive item in the training dataset. Unavoidably, some negative items sampled from the training dataset could be positive in the test set. In this paper, we propose a self-supervised collaborative filtering framework (SelfCF), that is specially designed for recommender scenario with implicit feedback. The proposed SelfCF framework simplifies the Siamese networks and can be easily applied to existing deep-learning based CF models, which we refer to as backbone networks. The main idea of SelfCF is to augment the output embeddings generated by backbone networks, because it is infeasible to augment raw input of user/item ids. We propose and study three output perturbation techniques that can be applied to different types of backbone networks including both traditional CF models and graph-based models. The framework enables learning informative representations of users and items without negative samples, and is agnostic to the encapsulated backbones. We conduct comprehensive experiments on four datasets to show that our framework may achieve even better recommendation accuracy than the encapsulated supervised counterpart with a 2$\times$--4$\times$ faster training speed. We also show that SelfCF can boost up the accuracy by up to 17.79% on average, compared with a self-supervised framework BUIR.

1 INTRODUCTION

Implicit-feedback CF must learn from positive interactions while unobserved pairs may be either negative or missing positives. SelfCF uses posterior embedding perturbations to learn without negative samples, addressing sampling costs and false-negative risks.

1 INTRODUCTION

  • Negative sampling adds computation and memory costs, requires checking sampled items against observed positives, and can label future positive interactions as negatives.Even methods treating all unobserved interactions as negative may misclassify future positives.Source: §1 · p. 2
  • Over 50% of DNS-sampled pairs are false negatives on MOOC at LightGCN early stopping, while Games remains below 10%.DNS sampling overhead is 2–3 times that of uniform sampling; Table 1 defines overhead relative to training time per epoch.Source: §1 · pp. 2–3, Figure 2 · p. 3, Table 1 · p. 3
  • Positive-only self-supervised learning uses Siamese networks to bring representations of two augmented views together, while negative samples can prevent collapse through repulsion.BYOL and SimSiam use architectural asymmetry to address collapse in positive-only learning.Source: §1 · pp. 3–4
  • SelfCF perturbs user and item latent embeddings after encoding to create contrastive views, using one encoder rather than two.The paper presents this posterior perturbation as a way to avoid augmenting user/item IDs directly.Source: §1 · p. 4
  • 17.79% on average: SelfCF outperforms existing self-supervised frameworks and is competitive with or better than supervised counterparts across the reported experiments.The contribution summary reports the comparison with existing SSL frameworks as up to 17.79% on average.Source: §1 · p. 4

2 RELATED WORK

The related-work section reviews collaborative filtering and self-supervised learning, then positions SelfCF as an output-augmentation alternative to prior positive-only recommendation methods.

2.1 Collaborative Filtering

  • Collaborative filtering predicts preferences from similar users or items; matrix factorization and deep models address sparsity, while graph-based CF aggregates higher-order interaction-graph information.LightGCN simplifies graph convolution by removing feature transformations and nonlinear activations that can harm recommendation performance.Source: §2.1 · pp. 4–5

2.2 Self-supervised Learning

  • Contrastive SSL attracts positive pairs and repels negative pairs, often storing negatives in a memory bank or queue; recommendation work also uses supplemental or graph-augmentation-free signals.Uniform user and item representations are described as a way to mitigate popularity bias and improve accuracy.Source: §2.2 · p. 5
  • BYOL and SimSiam learn from positive samples with Siamese networks, using momentum-updated target parameters or stop-gradient to address collapse.BUIR adapts BYOL to positive-only user–item interactions, whereas SelfCF shares an encoder and augments its output embeddings.Source: §2.2 · p. 5

3 THE SELFCF FRAMEWORK

SelfCF learns user and item representations from positive interactions using a shared-backbone Siamese design, then creates contrastive views by perturbing output embeddings. Its symmetric cross-view prediction loss is optimized through the online network while gradients are stopped on the target view.

3 THE SELFCF FRAMEWORK

  • SelfCF uses the same backbone parameters in its online and target networks, learns from positive user-item interactions, and omits BYOL/BUIR’s momentum encoder.Perturbing target-network output embeddings supplies different views instead of perturbing raw inputs.Source: §3 · pp. 5–6, §3.1 · p. 7

3.1 Data Augmentation via Output Perturbation

  • SelfCF creates output views with historical embeddings, embedding dropout, or graph-specific edge pruning.Historical embeddings combine prior and current outputs; dropout perturbs current embeddings, while edge pruning changes neighbor aggregation and costs more computation than dropout.Source: §3.1 · p. 7

3.2 The Loss Function

  • The loss symmetrically matches each predicted user or item embedding to its perturbed counterpart, with gradients stopped on the target network.The objective uses negative cosine similarity, regularizes online embeddings and the predictor, and predicts each positive interaction in both user-to-item and item-to-user directions.Source: §3.2 · pp. 8–9

4 EXPERIMENTS

The experiments assess whether positive-interaction-only self-supervised models can outperform supervised counterparts, how performance varies by user interaction history, and which SelfCF components matter. Evaluation spans four datasets from education and e-commerce.

4 EXPERIMENTS

  • The evaluation compares SelfCF with BUIR and eight baselines on four datasets, using BPR and LightGCN as backbones.All methods are trained on a single GeForce RTX 2080 Ti with 11 GB of memory.Source: §4 · p. 9
  • The experiments ask whether self-supervised models using only positive interactions outperform supervised counterparts, how results vary for cold-start and loyal users, and which components prevent collapse.The study compares user groups by interaction count and ablates components including the predictor and loss function.Source: §4 · p. 9

4.1 Dataset Description.

  • The four datasets are Amazon Video Games, Amazon Arts, Crafts and Sewing, Amazon Grocery and Gourmet Food, and COCO, with data filtered using a 5-core setting for users and items.They span e-commerce and education; the selection also considers release date and interaction-graph size.Source: §4.1 · pp. 9–10

4.2 Encapsulated Baselines and Framework BUIR.

The comparison includes supervised CF baselines and self-supervised frameworks, with BPR and LightGCN encapsulated as backbones for selected comparisons. Recommendation accuracy is measured with all-ranking Recall@K and NDCG@K.

4.2 Encapsulated Baselines and Framework BUIR.

  • The study includes supervised baselines and self-supervised frameworks, chiefly comparing SelfCF with BUIR and encapsulating BPR and LightGCN as backbones.Other frameworks use LightGCN because it performs better than BPR in the authors’ comparisons.Source: §4.2 · pp. 10–11
  • BUIR is the principal negative-sample-free comparator, while the baseline set also includes methods such as BPR, LightGCN, and SimGCL.The listed supervised baselines include models that use negative sampling, as well as approaches such as EHCF that treat unobserved pairs as negatives.Source: §4.2 · p. 10

4.3 Evaluation Metrics.

  • Recall@K and NDCG@K use all-ranking evaluation, treating every item a user has not interacted with as a candidate rather than sampling candidates.The compared cutoffs are K = 10, 20, and 50; NDCG@K is normalized to [0, 1].Source: §4.3 · p. 11

4.4 Hyper-parameters Settings.

The paper evaluates SelfCF against supervised and self-supervised baselines using chronological train, validation, and test splits. It reports that both classical and graph-based CF can benefit from SelfCF, with gains depending on the backbone and perturbation.

4.4 Hyper-parameters Settings.

  • Models use 64-dimensional user and item embeddings, Xavier initialization, Adam, and grid-searched SelfCF settings.The authors use L1 regularization for the predictor when BPR is encapsulated and L2 otherwise.Source: §4.4 · pp. 11–12

4.5 Overall Comparison

  • Evaluation uses chronological splits in a 7:1:2 ratio for training, validation, and testing.The authors adopt this offline setting to avoid data leakage, noting that online evaluation is not feasible in their academic environment.Source: §4.5 · p. 12
  • SelfCFed improves BUIR by up to 17.79% on average across four datasets, and by 21.19% when both use BPR.The authors report that embedding dropout is preferable among the three output perturbations across datasets.Source: §4.5 · p. 12
  • SelfCF improves BPR and LightGCN by up to 7.36% and 6.55%, respectively, across four datasets.The authors conclude that both classical and graph-based CF models can benefit from SelfCF.Source: §4.5 · p. 12

4.6 Efficiency of SelfCF

SelfCFed is reported to match or outperform four-layer LightGCN while using substantially less training time. The paper also links SelfCF's stop-gradient design to reduced representation covariance and improved recommendation performance.

4.6 Efficiency of SelfCF

  • SelfCFed is on par or better than four-layer LightGCN while requiring only one-half to one-quarter of its training time.The efficiency comparison is reported in Table 5.Source: §4.6 · p. 12, Table 5 · p. 15

4.7 Understanding the Learning of SelfCF

  • SelfCF with stop-gradient significantly lowers learned-representation covariance compared with the version without stop-gradient.The paper defines covariance regularization as the sum of squared off-diagonal coefficients of the representation covariance matrix; lower covariance indicates better de-correlation.Source: §4.7 · p. 15
  • With stop-gradient, Recall@20 consistently improves during training, whereas removing stop-gradient causes performance to collapse at a fixed level.The comparison is conducted on the Food dataset.Source: §4.7 · p. 15

4.8 Hyper-parameter Sensitivity

The sensitivity study evaluates SelfCF variants with LightGCN on Food, examining layer count and several hyperparameters. The reported variants show similar behavior, with SelfCFed comparatively stable across layer counts.

4.8 Hyper-parameter Sensitivity

  • The study uses Food with LightGCN and examines momentum, layer count, embedding dropout, and edge pruning; Games results are reported to show similar patterns.The Games results are placed in the appendix for reference.Source: §4.8 · p. 15
  • SelfCFed performance is not very affected by LightGCN layer count, while SelfCFep is unstable as layers increase.SelfCFhe and SelfCFed show relatively slow performance degradation as layers increase; the study varies layers from 1 to 8.Source: §4.8 · p. 16
  • Performance is less sensitive to momentum and dropout than to the regularization factor; weak regularization is preferred in practice.Recall@20 and NDCG@20 show consistent patterns in the hyperparameter study.Source: §4.8 · p. 16
  • The three SelfCF variants exhibit similar hyperparameter-study behavior, so subsequent recommendation-result and ablation analyses focus on SelfCFed..Source: §4.8 · p. 16

4.9 Diving into the Recommendation Results

On Food, SelfCF outperforms LightGCN across user-degree categories and reports its largest improvement for cold-start users. The representation analysis contrasts the mixed LightGCN embeddings with the more separated user and item representations learned by SelfCFed.

4.9 Diving into the Recommendation Results

  • 14.4% improvement in Recall@50 is reported for cold-start users, the largest gain; loyal users have the second-highest improvement at 12.80%.SelfCF outperforms LightGCN on Recall@50 and NDCG@50 in every user-degree category.Source: §4.9 · p. 17, Figure 10 · p. 18

4.10 Representation of nodes

  • LightGCN user and item representations are highly melded, whereas SelfCFed representations are more separated, with similar users and items clustering together.The authors say this pattern is in line with Equation 6.Source: §4.10 · p. 18

5 ABLATION STUDY

The ablations examine the predictor, loss function, and stop-gradient operator in SelfCF. They find that predictor design and loss choice affect performance, while SelfCF can work without stop-gradient but performs worse without it.

5 ABLATION STUDY

  • The ablation study uses SelfCFed on Food with a two-layer LightGCN backbone and embedding dropout set to 0.5.The predictor variants are compared in Table 6.Source: §5 · p. 18, §5.1 · p. 18, Table 6 · p. 19

5.1 Predictor

  • Removing the predictor leaves SelfCF working, but its performance falls to the level of the Popularity algorithm.A fixed random initialization for the predictor makes it difficult for the framework to learn good user/item representations; a 2-layer MLP is competitive with a 1-layer version.Source: §5.1 · pp. 18–19

5.2 Loss function

  • Cosine similarity performs better than cross-entropy similarity, although cross-entropy can prevent solution collapse to some extent.The authors attribute cosine similarity’s advantage to its direct capture of user-item interaction preference.Source: §5.2 · p. 19

5.3 Stop-gradient

  • SelfCF works even with a completely symmetric setup, but removing stop-gradient greatly reduces performance.The authors speculate that gradients flowing in both directions prevent the framework from learning the baseline’s optimal parameters.Source: §5.3 · pp. 19–20

6 CONCLUSION AND FUTURE DIRECTIONS

SelfCF learns user and item representations without negative samples or labels by perturbing backbone output embeddings. Across four datasets, it is competitive with supervised methods and compares favorably with BUIR; the paper also outlines several directions for extending the framework.

6 CONCLUSION AND FUTURE DIRECTIONS

  • SelfCF perturbs backbone output embeddings rather than user/item ID inputs, which the authors say cannot feasibly be augmented.The framework learns user and item representations without negative samples or labels.Source: §6 · p. 20
  • Across four datasets, SelfCF is on par with or better than BUIR and is competitive with supervised counterparts, with a reported 6.55% gain over LightGCN.The authors identify further study of predictor design, supervised-signal integration, embedding augmentation, and multimodal feature fusion as potential directions.Source: §6 · p. 20
  • The framework uses positive user-item interaction pairs, and the authors identify integrating supervised signals as future work.They also propose exploring other embedding augmentation methods and multimodal features, including item text and images.Source: §6 · p. 20
  • Predictor design is a future research direction because SelfCF learns predictor parameters as well as user and item representations.The paper notes that both 1-layer and 2-layer MLP predictors show promising performance.Source: §6 · p. 20

A HYPER-PARAMETER STUDY ON GAMES

The appendix reports Games hyperparameter trends for SelfCF and a separate MOOC comparison of LightGCN sampling methods. On MOOC, DNS ranks hard negatives more effectively at low K, while uniform sampling has better recall at K=20 and K=50.

A HYPER-PARAMETER STUDY ON GAMES

  • In Games, Recall@20 and NDCG@20 vary with momentum, embedding dropout, edge dropout, and regularization coefficient; the patterns align with Food.Fig. 13 plots NDCG@20 for three SelfCF variations across those hyperparameters.Source: A HYPER-PARAMETER STUDY ON GAMES · p. 23, Figure 13 · p. 24
  • On MOOC, DNS yields higher NDCG@10 when K is low, but uniform sampling has better recall at K=20 and K=50.The authors attribute DNS’s weaker recall at larger K to difficulty retrieving related, low-ranked items and to sampled negatives that may come from the test set.Source: A HYPER-PARAMETER STUDY ON GAMES · p. 24, Table 9 · p. 24

Related papers

Loading 2107.03019v3…