PaperlayerView exact revision on arXiv

Source-linked AI summary

SimpleX: A Simple and Strong Baseline for Collaborative Filtering

Kelong Mao, Jieming Zhu, Jinpeng Wang, Quanyu Dai, Zhenhua Dong, Xi Xiao, Xiuqiang He

arXiv:2109.12613v3cs.IRcs.AI

TL;DR

CF research has emphasized interaction encoders while giving less systematic attention to loss functions and negative sampling ratios. The paper proposes CCL within the unified SimpleX model and evaluates it across broad benchmarks, finding strong performance against sophisticated CF methods. It positions SimpleX as a simple, strong baseline for future CF evaluation.

  • Problem

    CF studies have not systematically evaluated the effects of loss functions and negative sampling ratios, despite their importance alongside interaction encoders.

  • Method

    The paper proposes cosine contrastive loss and integrates it into SimpleX, which combines matrix factorization with user-behavior aggregation.

  • Results

    Across 11 benchmark datasets and 29 CF models, SimpleX surpasses most sophisticated state-of-the-art methods, with up to 48.5% higher NDCG@20 than LightGCN.

  • Takeaways & Limitations

    SimpleX provides a simple, strong baseline and supports greater attention to loss functions and negative sampling in CF research.

Abstract

from arXiv · show

Collaborative filtering (CF) is a widely studied research topic in recommender systems. The learning of a CF model generally depends on three major components, namely interaction encoder, loss function, and negative sampling. While many existing studies focus on the design of more powerful interaction encoders, the impacts of loss functions and negative sampling ratios have not yet been well explored. In this work, we show that the choice of loss function as well as negative sampling ratio is equivalently important. More specifically, we propose the cosine contrastive loss (CCL) and further incorporate it to a simple unified CF model, dubbed SimpleX. Extensive experiments have been conducted on 11 benchmark datasets and compared with 29 existing CF models in total. Surprisingly, the results show that, under our CCL loss and a large negative sampling ratio, SimpleX can surpass most sophisticated state-of-the-art models by a large margin (e.g., max 48.5% improvement in NDCG@20 over LightGCN). We believe that SimpleX could not only serve as a simple strong baseline to foster future research on CF, but also shed light on the potential research direction towards improving loss function and negative sampling. Our source code will be available at https://reczoo.github.io/SimpleX.

1 INTRODUCTION

The paper argues that CF performance depends not only on interaction encoders but also on loss functions and negative sampling. It introduces CCL and SimpleX, then evaluates the approach broadly against existing CF models.

  • Research gap: CF learning comprises interaction encoding, loss optimization, and negative sampling, but prior work has focused mainly on more powerful encoders.Existing encoder designs include MLPs, auto-encoders, attention networks, transformers, and GNNs.
  • Research gap: Systematic comparisons of loss functions and negative sampling ratios are limited, despite empirical evidence that BPR with few negatives can underperform.The paper argues these choices can be as important as, or more important than, the interaction encoder.
  • Method: CCL maximizes cosine similarity for positive user-item pairs while minimizing negative-pair similarity toward a margin.The loss is tailored for collaborative filtering and is designed to improve embedding optimization.
  • Motivation: The work aims to provide a simple, strong baseline because weak baselines may cause researchers to overestimate improvements from new CF models.The stated goal is to foster more robust future CF research rather than criticize existing work.
  • Method: SimpleX unifies matrix factorization with user-behavior modeling by aggregating historically interacted items and combining the result with a user embedding.The model uses CCL together with a large negative sampling ratio.
  • Evaluation: SimpleX is evaluated on 11 benchmark datasets against 29 CF models and reports large effectiveness and efficiency gains.The experiments compare loss functions, negative sampling ratios, and runtime efficiency.

2 BACKGROUND AND RELATED WORK

The background formalizes implicit CF around interaction data and identifies interaction encoders, loss functions, and negative sampling as key modeling components. It also organizes representative methods into MF-based, autoencoder-based, GNN-based, and other categories.

  • CF formulation: This work focuses on implicit CF, which learns from clicks, visits, or purchases rather than explicit ratings.Explicit feedback can be transformed into implicit feedback through binarization.
  • CF formulation: In implicit CF, the interaction matrix records y_ui = 1 for observed user-item interactions and y_ui = 0 otherwise.The matrix represents the feedback used by the recommendation model.
  • CF components: Interaction encoders learn user and item embeddings that capture collaborative signals and behavioral similarity.The paper identifies interaction encoding as a core and extensively studied CF component.
  • CF components: Pointwise losses frame CF as classification or regression, whereas pairwise BPR makes positive-pair similarity exceed negative-pair similarity.BCE and MSE are pointwise examples; BPR is a pairwise example.
  • CF components: Negative sampling improves training efficiency when many interaction entries are unobserved, and this work concentrates on the negative sampling ratio.Other sampling research addresses informative negatives and selection bias.
  • Method categories: Representative CF methods are grouped into MF-based, autoencoder-based, GNN-based, and other approaches.The categories reflect different modeling strategies for user-item interactions and representations.

3 SIMPLEX

SimpleX combines a matrix-factorization-style model with behavior aggregation, cosine contrastive loss, and design choices intended to make large-scale negative sampling effective. Its architecture supports several aggregation variants and relates to established CF models while remaining a simple unified baseline.

  • Cosine Contrastive Loss: CCL maximizes cosine similarity for positive user-item pairs while pushing negative-pair similarity below a margin.The margin filters uninformative negatives, while w balances positive and negative loss terms.
  • Model Architecture: SimpleX combines matrix factorization with user-behavior modeling by aggregating interacted items and fusing the resulting vector with a user embedding.The final user representation is compared with item embeddings using cosine similarity and supplied to CCL.
  • Behavior Aggregation: The behavior aggregation layer supports average pooling, self-attention, and user-attention, with attention weights computed using global or user-specific query vectors.Average pooling treats interacted items uniformly, whereas attention mechanisms account for their relative importance or user preferences.
  • Behavior Aggregation: Average pooling is a robust first-choice aggregation method, while self-attention and user-attention can require more tuning and sometimes provide only marginal improvements.The comparison is reported on Amazon-Books and Yelp2018.
  • Relationships to Existing Models: SimpleX unifies components related to MF, YouTubeNet, and GNN-based models through its aggregation and fusion choices.With g = 1 it reduces to MF trained with CCL; other aggregation choices resemble simplified YouTubeNet or graph-attention mechanisms.
  • Relationships to Existing Models: Despite limited novelty in its individual design elements, SimpleX is presented as a simple and strong baseline that surpasses most state-of-the-art CF models by a large margin.The paper positions this result as evidence that unified design choices can produce a strong baseline for future CF research.

4 EXPERIMENTS

The experiments study loss functions, negative sampling ratios, model comparisons, CCL integration, efficiency, and generalization across 11 benchmark datasets. SimpleX is implemented with standard optimization and tuned hyperparameters for fair comparisons.

  • The evaluation covers loss functions, negative sampling ratios, comparisons with existing models, CCL integration, parameter and efficiency analyses, and validation on additional datasets.
  • Datasets: 11 benchmark datasets are used, including Amazon-Books, Yelp2018, Gowalla, six additional datasets, Movielens-20M, and MillionSongData.The main experiments focus on Amazon-Books, Yelp2018, and Gowalla; additional datasets support broader validation and comparisons with models from their original studies.
  • Baselines: The study compares SimpleX with 29 existing CF models spanning matrix-factorization, autoencoder, GNN-based, and other model types.
  • Implementation: SimpleX uses PyTorch, Adam optimization, L2 embedding regularization, and hyperparameter searches for learning rate, regularization weight, negative samples, and margin.The number of negative samples is searched from 1 to 2000, while embedding sizes match those of compared models for fairness.

4.2 Impact of Different Loss Functions

This section systematically compares representative loss functions using MF and examines why CCL performs well. CCL is consistently strongest across the three evaluated datasets and can filter hard negatives through its margin.

  • Setup: MF is used as a simple backbone to systematically compare CCL with representative losses including BPR, PHL, BCE, SCE, and MSE.
  • Results: CCL achieves the best performance on Amazon-Books, Yelp2018, and Gowalla, exceeding other losses by at least 16.7%, 9.2%, and 13.7% in Recall@20, respectively.
  • Results: BPR is strong on Gowalla but performs poorly on Amazon-Books and Yelp2018, making BPR-based training potentially sub-optimal on those datasets.
  • Mechanism: With margin m = 0.8, CCL makes only negative pairs with cosine similarity above 0.8 contribute to the loss, filtering harder-to-distinguish negatives.

4.3 Impact of Negative Sampling Ratio

The negative sampling ratio materially affects CF training, and its effect depends on the loss function. CCL remains superior across ratios and continues benefiting from many negative samples.

  • Increasing the number of negative samples within a certain range generally improves CF model performance.
  • MF trained with CCL consistently outperforms MF trained with other loss functions across different negative sampling ratios.
  • PHL, MSE, and BPR stabilize near 50 negative samples, whereas CCL, BCE, and SCE continue gaining performance up to 1000 samples.
  • Combining CCL with a large negative sampling ratio appears to be a promising setting for improving CF performance.

4.4 Performance Comparison to SOTA Models

SimpleX achieves the best overall performance on the three main datasets, while MF-CCL also performs exceptionally well. The comparisons further indicate that concise models can outperform more complex architectures and that existing baselines may be weak.

  • Main comparison: SimpleX achieves the best overall performance on Amazon-Books, Yelp2018, and Gowalla, improving Recall@20 over LightGCN by 41.9%, 8.0%, and 2.3%, respectively.
  • MF-CCL: MF-CCL surpasses previously reported MF-BPR results, reaches new state-of-the-art performance excluding SimpleX on Amazon-Books and Yelp2018, and is comparable to DGCF on Gowalla.
  • Baseline strength: YouTubeNet, CML, and SLIM exceed NGCF by more than 24% in Recall@20 and 28% in NDCG@20 on average.
  • Model complexity: Concise models including SLIM, YouTubeNet, CML, MF-CCL, and SimpleX can outperform more complex autoencoder-based and GNN-based models.

4.5 Incorporating CCL to Other Models

Replacing original losses with CCL improves YouTubeNet and LightGCN, although gains are smaller than for MF. The experiments identify the loss function as a major bottleneck in CF models.

  • CCL across models: CCL improves YouTubeNet and LightGCN over their original SCE and BPR losses on Amazon-Books and Yelp2018.The reported gains are smaller than those observed for MF.
  • CCL across models: CCL’s improvements are less significant for sophisticated YouTubeNet and LightGCN encoders than for MF.The paper attributes this contrast to stronger encoders learning biased collaborative signals, reducing the apparent impact of the loss function.
  • Research implications: The paper’s focus is to provide a simple strong baseline rather than exhaustively improve existing state-of-the-art models with different losses.It encourages co-design of interaction encoders, loss functions, and negative sampling.
  • Parameter analysis: Average pooling, self-attention, and user-attention obtain very similar results on Amazon-Books and Yelp2018.This supports the practical robustness of average pooling for behavior aggregation.
  • Parameter analysis: The negative loss weight w is vital, because both w=1 and w=1000 can reduce performance.The weight adjusts the ratio of positive and negative losses.

4.7 Efficiency Comparison

SimpleX is substantially more efficient than ENMF and LightGCN in the reported Amazon-Books comparison. It converges in about 30 epochs and achieves large total-training-time speedups.

  • Evaluation protocol: The efficiency comparison measures average training time per epoch, epochs to converge, and total training time on Amazon-Books.Test time is excluded from the total training-time comparison.
  • Convergence: SimpleX needs around 30 epochs to converge in training.The authors describe this convergence behavior as convenient for real application.
  • Training time: SimpleX’s total training time with a 1000:1 negative sampling ratio is approximately 2x faster than ENMF and 10x faster than LightGCN.The comparison uses the same implementation framework and hardware setting.
  • Training time: Reducing the negative sampling ratio to 100:1 can reduce SimpleX’s per-epoch training time to 40s.The passage states that this setting results in an optimized training time per epoch.

4.8 Evaluating SimpleX on More Datasets

Across eight additional datasets, SimpleX consistently outperforms the compared 2020 state-of-the-art models and remains competitive under inductive autoencoder comparisons. Its results support using SimpleX as a simple, strong baseline across varied settings.

  • Additional datasets: SimpleX consistently outperforms all compared 2020 state-of-the-art CF models across the additional datasets.The comparisons reuse corresponding papers’ preprocessing, experimental settings, and reported results for fairness.
  • Additional datasets: 12.8% to 33.6% improvement in NDCG@20 is reported for SimpleX over BGCF on Amazon-CDs, Amazon-Movies, and Amazon-Beauty.BGCF is identified as a recent GNN-based model.
  • Inductive evaluation: Under the strong generalization protocol, training, validation, and test users are disjoint, requiring inductive inference.Only item embeddings can be learned during training and transferred to validation and test settings.
  • Inductive evaluation: SimpleX outperforms SLIM but does not surpass Mult-VAE, EASER, and RecVAE in their complete forms on Movielens-20M and MillionSongData.The compared autoencoder models use substantially more parameters than SimpleX.
  • Inductive evaluation: With reduced embedding dimensions and low-rank EASER approximation, SimpleX clearly outperforms the autoencoder-based comparison models.The adjusted setting targets a more parameter-balanced comparison.
  • Implications: The authors conclude that SimpleX can serve as a simple and strong baseline for fairer CF comparisons and future research.The broader evaluation is presented as evidence for its effectiveness and robustness.

5 CONCLUSION

The paper argues that CF research should examine loss functions and negative sampling alongside interaction encoders. It proposes CCL and SimpleX, whose broad experiments show strong performance and efficiency while motivating more robust evaluation.

  • Conclusion: Loss functions and negative sampling have been largely neglected relative to interaction encoders in CF modeling.The conclusion identifies these as three key aspects of CF modeling.
  • Conclusion: The paper proposes cosine contrastive loss together with SimpleX, a simple and strong CF baseline.The authors report that SimpleX outperforms most state-of-the-art CF models by a large margin.
  • Conclusion: The released baseline and benchmarking results are intended to foster future CF research and more robust empirical evaluation.The conclusion encourages attention to key components beyond interaction encoders.
Loading 2109.12613v3…