PaperlayerView exact revision on arXiv

Source-linked AI summary

BARS: Towards Open Benchmarking for Recommender Systems

Jieming Zhu, Quanyu Dai, Liangcai Su, Rong Ma, Jinyang Liu, Guohao Cai, Xi Xiao, Rui Zhang

arXiv:2205.09626v5cs.IR

TL;DR

Recommender-system research lacks a widely recognized standard for reproducible evaluation, with studies often using inconsistent splits and experimental settings. BARS addresses this gap with an open benchmarking pipeline and leaderboard covering matching and ranking, while documenting reusable artifacts and encouraging community contributions.

  • Problem

    Recommender-system studies often use ad-hoc data splits, experimental settings, baselines, and incomplete implementation details, limiting reproducibility and consistency.

  • Method

    BARS builds a standardized, GLUE-like open benchmarking pipeline that records datasets, splits, protocols, models, configurations, logs, results, and reproducing steps.

  • Results

    BARS provides benchmark coverage for candidate item matching and CTR prediction, with reusable results and artifacts for researchers, practitioners, competitors, and educators.

  • Takeaways & Limitations

    An open leaderboard and freely available benchmarking artifacts can support more consistent model comparison, reuse, and reproducible recommendation research.

  • Takeaways & Limitations

    The benchmark remains limited by the time and labor required to implement and tune models, with broader tasks, scenarios, and metrics left for future development.

Abstract

from arXiv · show

The past two decades have witnessed the rapid development of personalized recommendation techniques. Despite significant progress made in both research and practice of recommender systems, to date, there is a lack of a widely-recognized benchmarking standard in this field. Many existing studies perform model evaluations and comparisons in an ad-hoc manner, for example, by employing their own private data splits or using different experimental settings. Such conventions not only increase the difficulty in reproducing existing studies, but also lead to inconsistent experimental results among them. This largely limits the credibility and practical value of research results in this field. To tackle these issues, we present an initiative project (namely BARS) aiming for open benchmarking for recommender systems. In comparison to some earlier attempts towards this goal, we take a further step by setting up a standardized benchmarking pipeline for reproducible research, which integrates all the details about datasets, source code, hyper-parameter settings, running logs, and evaluation results. The benchmark is designed with comprehensiveness and sustainability in mind. It covers both matching and ranking tasks, and also enables researchers to easily follow and contribute to the research in this field. This project will not only reduce the redundant efforts of researchers to re-implement or re-run existing baselines, but also drive more solid and reproducible research on recommender systems. We would like to call upon everyone to use the BARS benchmark for future evaluation, and contribute to the project through the portal at: https://openbenchmark.github.io/BARS.

1 INTRODUCTION

BARS addresses the lack of standardized, reproducible recommender-system evaluation by defining an open benchmarking pipeline and applying it across matching and ranking tasks. It combines benchmark artifacts, results, and community resources to support more reliable comparisons and continued development.

  • Motivation: Recommender-system studies often use private data splits, differing settings, weak baselines, or incomplete implementation details, making results inconsistent and difficult to reproduce.These practices also make fair comparison of model effectiveness harder.
  • Approach: BARS defines a standardized pipeline that records datasets, splits, evaluation protocols, models, hyper-parameters, running logs, evaluation results, and reproducing steps.The pipeline treats these components as required artifacts for reproducible benchmarking.
  • Approach: The paper distinguishes a GLUE-like leaderboard benchmark from the benchmarking pipeline used to produce it.This definition separates the public benchmark outcome from the process and artifacts used to generate it.
  • Scope and findings: The initial BARS release covers candidate item matching and CTR prediction, spanning more than 70 models on six datasets through over 8,000 experiments and 15,000 GPU hours.The experiments used grid search for hyper-parameter tuning, and the authors report that sufficient tuning may exaggerate the effectiveness of many recent methods.
  • Impact: BARS offers researchers, practitioners, competitors, and educators reusable benchmark results, source code, datasets, configurations, and benchmarking scripts.The project is intended to support model comparison, baseline implementation, and teaching.
  • Limitations: The benchmark remains limited by the time and labor required to implement and tune models, motivating continued community contributions.The authors present BARS as an initiative whose scope should expand through contributions.

2 BACKGROUND AND RELATED WORK

Recommender systems typically divide work between efficient candidate matching and feature-rich ranking, with research spanning multiple model families and persistent reproducibility concerns. BARS responds by comparing existing benchmarking efforts and extending coverage across matching and ranking.

  • 2.1 Overview of Recommender Systems: Modern recommender systems retrieve candidates from millions of items during matching, then rank a smaller set using personalized CTR prediction and other scores.Matching emphasizes efficiency, whereas ranking uses abundant user, item, and context features.
  • 2.2 Candidate Item Matching: Matching methods include collaborative filtering, two-tower networks, graph-based models, and autoencoder-based models.These approaches learn user-item representations or collaborative signals for candidate retrieval.
  • 2.2 Candidate Item Matching: Approximate nearest-neighbor tools such as Faiss, Milvus, and Scann support efficient top-k retrieval from large item pools.These tools address retrieval efficiency after representation learning.
  • 2.3 Ranking: Ranking research includes logistic regression, factorization machines, and deep models that use product, convolutional, or attention-based feature interactions.Feature interaction modeling is identified as essential for accurate CTR prediction.
  • 2.3 Ranking: User-behavior modeling in CTR prediction uses attention, memory, retrieval, multi-task, and multi-domain approaches to capture interests and address data sparsity.These methods mine patterns from historical behavior and extend ranking models beyond feature interactions.
  • 2.4 Benchmarking and Reproducibility: Existing recommendation studies raise concerns about unreproducible and unfair comparisons, while prior tools and frameworks do not fully resolve the lack of standard benchmarking results.BARS extends earlier efforts toward a standardized pipeline covering tasks in both matching and ranking.

3 OPEN BENCHMARKING PIPLELINE

BARS establishes a standardized open benchmarking pipeline for reproducible recommender-systems research. It records dataset splits, evaluation protocols, model and training details, metrics, results, and reproducing steps to support consistent comparisons and reuse.

  • Pipeline motivation: BARS defines a standardized open benchmarking pipeline to address unreproducible and incomparable recommender-system evaluations.The pipeline responds to arbitrary dataset use, private splits, inconsistent protocols, and incomplete implementation details.
  • Data splitting: Each specific data split receives a unique dataset ID, enabling reuse and direct comparison of results using the same split.For example, Criteo_x4 denotes a specific split, while other splits receive different IDs.
  • Evaluation protocol: Each data split is paired with a recorded evaluation protocol so models can be compared under consistent settings.Protocols specify choices such as rare-feature filtering and embedding dimension, with different settings receiving new IDs.
  • Benchmark artifacts: The benchmark records open-source models, hyperparameter configurations, running logs, and running environments as reproducibility artifacts.These records address missing or incomplete training details that otherwise hinder reproduction.
  • Evaluation metrics: BARS recommends Recall@K, HitRate@K, and NDCG@K for matching, and logloss and AUC for CTR prediction.The standardized metrics are intended to make benchmarking results easier to reuse and compare.
  • Reproduction and scope: The pipeline documents detailed reproducing steps and defines a rigorous standard intended to enhance research rigor.The authors state that missing pipeline components can produce incomplete or broken research artifacts.

4 BENCHMARKING FOR CANDIDATE ITEM MATCHING

BARS benchmarks candidate item matching on public datasets with standardized metrics, full-corpus evaluation, and 33 models. Results show strong GNN performance, comparable alternatives, and weaknesses in prior baseline comparisons.

  • Benchmarking Settings: Evaluations run on the entire item corpus rather than sampled metrics, reflecting practical top-K retrieval settings.The paper connects this choice to approximate nearest-neighbor tools such as Faiss.
  • Benchmarking Settings: BARS evaluates 33 candidate item matching models across collaborative filtering, autoencoder, two-tower, graph embedding, and GNN categories.The model set includes 13 GNN-based models and spans five model categories.
  • Benchmarking Results and Analysis: GNN-based models outperform many traditional collaborative filtering models, with progress from GC-MC through NGCF and LightGCN.The reported pattern associates increasingly appropriate graph modeling with improved recommendation accuracy.
  • Benchmarking Results and Analysis: Except for graph embedding models, the best models across four categories are generally comparable, so no method type dominates item matching.SLIM leads Recall@20 on AmazonBooks, while SimpleX performs best on Yelp2018; each category has models in the top five.
  • Benchmarking Results and Analysis: ItemKNN and SLIM outperform other models on AmazonBooks, while YoutubeDNN beats many newer models on two other benchmark datasets.These comparisons show why rigorous benchmarking must include older and less frequently compared baselines.
  • Benchmarking Results and Analysis: The absence of a unified benchmark can propagate omitted comparisons across subsequent studies and contribute to exaggerated improvement claims.BARS is intended to address this recurring omission through open, standardized evaluation.

5 BENCHMARKING FOR CTR PREDICTION

BARS benchmarks CTR prediction on three open datasets using standardized data handling, common metrics, and 32 models. Results show substantial historical gains from deep models, but recent models often differ little and rankings vary across datasets.

  • Evaluation Metrics: Logloss measures binary-classification loss with lower values better, whereas AUC measures positive-versus-negative ranking with higher values better.
  • Experimental Setup: BARS evaluates 32 CTR prediction models on Criteo, Avazu, and KKBox using logloss and AUC.The benchmark follows a shared random train/validation/test splitting protocol rather than private splits.
  • Benchmarking Results and Analysis: Deep models such as DNN, Wide&Deep, DeepFM, and DCN delivered significant improvements over earlier logistic-regression and factorization-machine approaches.
  • Benchmarking Results and Analysis: Top-performing models differ widely across datasets, although DeepFM, DCN, xDeepFM, ONN, and DCN-V2 perform robustly across datasets.
  • Benchmarking Results and Analysis: Recent deep-model gains are often less than 0.1% after sufficient tuning, making strong improvements in large-scale CTR prediction difficult.The paper connects this pattern with research on behavior sequences, multitask modeling, and cross-domain modeling.
  • Benchmarking Results and Analysis: BARS finds inconsistent results relative to some existing papers, including similar accuracies among several models and worse results for some recent proposals.The authors state that these findings raise questions about evaluation rigor and reproducibility and motivate more carefully tuned baselines.

6 CALL FOR CONTRIBUTIONS

BARS invites community contributions to expand, verify, tune, and apply its open benchmarking pipeline. Future versions could broaden task, domain, and metric coverage beyond the current benchmark.

  • Contribution Opportunities: The community can contribute datasets, repeatable splits, models, results, reproducibility checks, tuning, feedback, and adherence to the open benchmarking pipeline.
  • Future Directions: Future benchmark versions could add re-ranking and sequential recommendation tasks.
  • Benchmarking Resources: Table 5 includes substantially more new-model and new-dataset results than the authors’ previous work, with additional results available on the website leaderboard.
  • Future Directions: Specialized leaderboards could cover verticals such as news, music, and micro-video recommendation.
  • Future Directions: The current benchmarking results focus on accuracy, while future versions could add diversity, coverage, and fairness metrics.

7 CONCLUSION

BARS addresses reproducibility in recommender-systems research by covering candidate-item matching and CTR prediction through a standardized pipeline and benchmark website. The project is intended to support researchers, practitioners, and educators.

  • Conclusion: BARS targets reproducible research for candidate-item matching and CTR prediction in recommender systems.
  • Conclusion: The project provides a standardized benchmarking pipeline, a benchmark website, and comprehensive benchmarking results.
  • Conclusion: BARS is intended to benefit researchers, practitioners, and educators in the recommender-systems community.
Loading 2205.09626v5…