PaperlayerView exact revision on arXiv

Source-linked AI summary

RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms

Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, Yingqian Min, Zhichao Feng, Xinyan Fan, Xu Chen, Pengfei Wang, Wendi Ji, Yaliang Li, Xiaoling Wang, Ji-Rong Wen

arXiv:2011.01731v3cs.IR

TL;DR

Recommendation algorithms are difficult to implement and compare consistently because implementations and evaluation details vary across platforms and developers. RecBole provides a unified framework with reusable structures, broad benchmarks, GPU acceleration, and standardized evaluation, supporting reproducible development and comparison. The library currently covers 73 algorithms and 28 datasets, while its evaluation scope deliberately includes widely adopted protocols rather than selecting one best protocol.

  • Problem

    Recommendation algorithms are implemented inconsistently across platforms, and many lack public implementations or use differing loss functions and optimization strategies, complicating reproducibility.

  • Method

    RecBole unifies data, model, evaluation, configuration, execution, and utility modules with extensible representations and protocols for developing, running, and comparing recommendation algorithms.

  • Results

    73 recommendation algorithms are implemented on 28 datasets spanning general, sequential, context-aware, and knowledge-based recommendation.

  • Takeaways & Limitations

    RecBole is intended to improve reproducibility, ease development of new algorithms, and provide a benchmark framework for recommender systems.

  • Takeaways & Limitations

    RecBole provides widely adopted evaluation protocols but does not aim to identify the most suitable protocol for recommender-system evaluation.

Abstract

from arXiv · show

In recent years, there are a large number of recommendation algorithms proposed in the literature, from traditional collaborative filtering to deep learning algorithms. However, the concerns about how to standardize open source implementation of recommendation algorithms continually increase in the research community. In the light of this challenge, we propose a unified, comprehensive and efficient recommender system library called RecBole, which provides a unified framework to develop and reproduce recommendation algorithms for research purpose. In this library, we implement 73 recommendation models on 28 benchmark datasets, covering the categories of general recommendation, sequential recommendation, context-aware recommendation and knowledge-based recommendation. We implement the RecBole library based on PyTorch, which is one of the most popular deep learning frameworks. Our library is featured in many aspects, including general and extensible data structures, comprehensive benchmark models and datasets, efficient GPU-accelerated execution, and extensive and standard evaluation protocols. We provide a series of auxiliary functions, tools, and scripts to facilitate the use of this library, such as automatic parameter tuning and break-point resume. Such a framework is useful to standardize the implementation and evaluation of recommender systems. The project and documents are released at https://recbole.io/.

1 INTRODUCTION

RecBole addresses inconsistent implementations and evaluation practices by providing a unified, reusable framework for developing, reproducing, and comparing recommendation algorithms. It combines extensible data structures, broad benchmarks, GPU acceleration, and standardized evaluation protocols.

  • Unified recommendation framework: RecBole uses shared data, model, and evaluation modules so researchers can reuse common procedures and focus on model-specific interfaces.The library is built with PyTorch and supports comparison through configurable evaluation protocols.
  • General and extensible data structure: Its atomic files, Interaction representation, Dataset, and DataLoader unify inputs and automate data flow across recommendation tasks.These structures support flexible inputs and reduce development effort for new models.
  • Comprehensive benchmark models and datasets: 73 recommendation algorithms and 28 commonly used datasets provide a broad benchmark spanning general, sequential, context-aware, and knowledge-based recommendation.Configuration files, command-line support, automatic parameter tuning, and breakpoint resume facilitate model comparison and use.
  • Efficient GPU-accelerated execution: GPU-oriented training and testing accelerate execution, including top-K evaluation through a unified matrix computation and PyTorch topk().The acceleration strategy targets the time costs of model training, testing, and full-ranking recommendation evaluation.
  • Extensive and standard evaluation protocols: RecBole supports combinations of item ordering, data splitting, and full- or sample-based ranking through extensible evaluation interfaces.The framework provides widely adopted settings without claiming to identify the single most suitable protocol.

2 THE LIBRARY — RECBOLE

RecBole organizes experimentation around configuration, data, model, evaluation, execution, and utility modules. This architecture supports setup, model running and evaluation, and auxiliary functions such as tuning and logging.

  • Configuration and execution: The configuration module sets the experimental environment, including hyperparameters and running details.
  • Configuration and execution: Data, model, and evaluation modules form the library’s core, while the execution module runs and evaluates models under specified settings.
  • Supporting utilities: The utility module collects auxiliary functions such as automatic parameter tuning, logging, and evaluation metrics.

2.1 Data Module

RecBole’s data module standardizes recommendation inputs through atomic files and a unified Interaction representation. Its data flow transforms raw data into model-ready structures while supporting multiple recommendation tasks and extensible preprocessing.

  • Data flow: The data module transforms raw input through atomic files, Dataset, Dataloader, and Interaction before reaching recommendation algorithms.The stated flow is raw input → atomic files → Dataset → Dataloader → Interaction → algorithms.
  • Data preprocessing: Dataset provides preprocessing functions including k-core filtering and missing-value imputation during conversion from atomic files.The library also provides dataset conversion tools and supports filtering and splitting operations.
  • Atomic files: Six atomic file types characterize inputs for mainstream recommendation tasks, with flexible extension for new tasks.Atomic files are feature-based data frames that cover most mainstream task inputs and can be incrementally extended.
  • Atomic files: Atomic files support general, context-aware, knowledge-based, and sequential recommendation through task-specific file combinations, while social recommendation support remains under development.The same .inter file can be reused across tasks, and additional files such as .user and .item can extend a task’s input.
  • Extensibility: The atomic-file design separates task-independent input data from configuration-driven processing, enabling reuse of data files across recommendation tasks.Users can switch tasks without modifying the shared .inter file, while configuration determines how the files are processed.
  • Internal representation: Interaction is a flexible key-value structure whose keys reference input features and whose tensor values store batch or mini-batch data for algorithms.It is implemented as an abstract data type based on python.dict and torch.Tensor, supporting batch-based recommendation algorithm development.

2.2 Model Module

RecBole separates recommendation algorithms into a model module with reusable interfaces and components, supporting implementations across four recommendation tasks. It also provides tested models and auxiliary facilities for tuning and training management.

  • Model interfaces: The model module decouples algorithm implementations from other components through required interfaces for training and evaluation.New models connect to input and evaluation modules while ignoring unrelated implementation details.
  • Reusable components: The library organizes reusable loss functions, neural components, and initialization methods as individual building blocks for complex models.Examples include BPR loss, multilayer perceptrons, multi-head attention, graph neural networks, and Xavier initialization.
  • Model coverage: RecBole implements 73 recommendation models spanning general, sequential, context-aware, and knowledge-based recommendation.The collection includes recent neural methods alongside classic methods such as ItemKNN and FM.
  • Validation: All implemented models are evaluated on selected datasets, and a code reviewer examines their implementation correctness.Models are tested on two or four datasets, depending on the model.
  • Auxiliary facilities: Automatic parameter tuning, model saving and loading, breakpoint resume, loss monitoring, and early stopping support model development and training workflows.Tuning methods include grid search, random search, TPE, and adaptive TPE.

2.3 Evaluation Module

RecBole provides a common evaluation module with value-based and ranking-based metrics, multiple dataset protocols, and GPU-accelerated top-K evaluation. Its design supports comparisons across models while addressing computational costs in full ranking.

  • Evaluation protocols: The evaluation module standardizes comparisons by applying commonly used protocols across different recommendation models.It supports both value-based metrics for rating prediction and ranking-based metrics for top-K recommendation.
  • Evaluation protocols: Supported ranking protocols vary item sorting, data splitting, and whether evaluation uses full ranking or sampled candidates.These settings reflect ongoing disagreement about appropriate top-K evaluation procedures.
  • Evaluation architecture: The evaluation architecture encapsulates protocol functions into Group, Split, Order, and NegSample components.These components support different evaluation settings in a unified implementation.
  • Top-K acceleration: Top-K evaluation is accelerated by converting users’ item scores into a unified matrix and applying PyTorch’s GPU topk() operation.The strategy focuses on selecting and generating the highest-scoring K items after score estimation.
  • Top-K acceleration: Equal-sized user-by-item matrices make full-ranking and sample-based evaluation compatible with unified parallel computation.The approach includes all items, including training items, so subsequent evaluation matrices have consistent dimensions.
  • Efficiency analysis: The acceleration strategy is empirically analyzed with BPR by comparing time costs across Movielens datasets with and without acceleration.Table 8 reports the comparison in seconds for different dataset sizes.

3 USAGE EXAMPLES OF THE LIBRARY

RecBole’s usage workflow covers dataset preparation, configuration, filtering and splitting, model loading, training, and evaluation. Users can also tune hyperparameters automatically or implement new models through defined interfaces.

  • Parameter tuning: Contained models support either fixed parameters or automatically tuned parameters.The tuning workflow searches user-specified parameter ranges and saves optimal parameters with corresponding model performance.
  • Running existing models: Running an existing model proceeds from dataset formatting and configuration through data splitting, model loading, training, and evaluation.Configurations can come from files, command-line arguments, or parameter dictionaries.
  • Running existing models: Users can filter datasets by interaction requirements or time periods and split them by ratios or leave-one-out procedures.The resulting data are used to create training, validation, and testing sets.
  • Parameter tuning: The library offers four tuning methods: Grid Search, Random Search, Tree of Parzen Estimators, and Adaptive TPE.The tuning function is implemented using hyperopt.
  • Implementing new models: New models are implemented through __init__(), calculate_loss(), and predict() functions within task-specific abstract model classes.These functions define initialization, the optimization loss, and score prediction for training or testing.

4 COMPARISON WITH EXISTING LIBRARIES

RecBole is positioned as a comprehensive research library relative to existing recommender-system libraries, combining broad model and dataset coverage with extensible structures and research-oriented tools. The authors also emphasize that continued maintenance is needed after release.

  • Framework choice: The library uses PyTorch and reflects the broader shift of recommender-system libraries toward Python, TensorFlow, and PyTorch.The authors cite PyTorch’s easy debugging and NumPy compatibility as reasons for selecting it.
  • Coverage and comparison: RecBole provides 73 models across four recommendation categories and is described as having the most comprehensive models and benchmark datasets among existing libraries.The coverage includes general, sequential, context-aware, and knowledge-based recommendation.
  • Research utility: Its comprehensive implementations aim to reduce model re-programming while its extensible data structures and tools support new algorithm development.Automatic parameter tuning is given as an example of a research-oriented utility.
  • Maintenance: The authors state that open-source implementation requires ongoing maintenance and report responding to GitHub issues and fixing bugs after release.They report that 134 issues had been solved by August 21, 2021.

5 CONCLUSION

RecBole releases a unified recommender-system library spanning 73 algorithms and 28 datasets. It combines extensible data structures, standardized evaluation, and GPU-accelerated execution to support reproducible development and benchmarking.

  • 73 recommendation algorithms are implemented on 28 commonly used datasets.
  • General and extensible data structures provide a unified framework for developing new recommendation algorithms.The library uses flexible atomic files and an Interaction structure for GPU-based internal representations.
  • Extensive and standard evaluation protocols support comparison and testing across recommendation algorithms.
  • GPU-accelerated implementation uses optimization techniques to achieve efficient execution.
  • RecBole is intended to improve reproducibility, ease algorithm development, and establish a recommender-systems benchmark framework.Future work includes adding datasets, models, result visualization, and algorithm-debugging utilities.
Loading 2011.01731v3…