Source-linked AI summary
Elliot: a Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation
Vito Walter Anelli, Alejandro Bellogín, Antonio Ferrara, Daniele Malitesta, Felice Antonio Merra, Claudio Pomo, Francesco Maria Donini, Tommaso Di Noia
TL;DR
Recommender-system research faces difficult and often irreproducible evaluation choices across algorithms, data preparation, tuning, metrics, and statistical procedures. Elliot addresses this gap with a configuration-driven framework covering the experimental pipeline from data loading to results collection. Its broad support for models, preprocessing, metrics, and statistical tests is intended to make recommender evaluation more comprehensive and reproducible.
Problem
The abundance of algorithms, evaluation choices, and procedures makes rigorous, reproducible, and fair recommender-system evaluation difficult.
Method
Elliot runs a full recommendation-experiment pipeline from a simple configuration file, integrating data preparation, models, hyperparameter optimization, metrics, and statistical tests.
Results
Elliot provides broad support for 50 recommendation models, 36 evaluation metrics, multiple preprocessing and splitting strategies, and paired t-test and Wilcoxon analyses.
Takeaways & Limitations
Elliot provides a single configurable framework for conducting and reproducing extensive recommender-system evaluation experiments.
Abstract
from arXiv · showhide
Recommender Systems have shown to be an effective way to alleviate the over-choice problem and provide accurate and tailored recommendations. However, the impressive number of proposed recommendation algorithms, splitting strategies, evaluation protocols, metrics, and tasks, has made rigorous experimental evaluation particularly challenging. Puzzled and frustrated by the continuous recreation of appropriate evaluation benchmarks, experimental pipelines, hyperparameter optimization, and evaluation procedures, we have developed an exhaustive framework to address such needs. Elliot is a comprehensive recommendation framework that aims to run and reproduce an entire experimental pipeline by processing a simple configuration file. The framework loads, filters, and splits the data considering a vast set of strategies (13 splitting methods and 8 filtering approaches, from temporal training-test splitting to nested K-folds Cross-Validation). Elliot optimizes hyperparameters (51 strategies) for several recommendation algorithms (50), selects the best models, compares them with the baselines providing intra-model statistics, computes metrics (36) spanning from accuracy to beyond-accuracy, bias, and fairness, and conducts statistical analysis (Wilcoxon and Paired t-test). The aim is to provide the researchers with a tool to ease (and make them reproducible) all the experimental evaluation phases, from data reading to results collection. Elliot is available on GitHub (https://github.com/sisinflab/elliot).
1 INTRODUCTION
Recommendation-system evaluation has become difficult to conduct rigorously because researchers must choose among many algorithms, preprocessing and splitting strategies, tuning procedures, metrics, and reporting practices. Elliot addresses these obstacles by driving a complete, reproducible experiment from a simple configuration file.
- Only one-third of published recommendation-system experimental results were reproducible, while suboptimal baselines also produced unfair comparisons.
- Elliot runs the full experimental process from dataset loading through results gathering using a simple configuration file.
- Elliot supports 50 recommendation systems, 51 hyperparameter-tuning combinations, 13 splitting strategies, and 8 prefiltering policies.
- Its 36 metrics span accuracy, error, coverage, novelty, diversity, bias, and fairness, alongside paired t-test and Wilcoxon analyses.
2 PRIOR WORK
Prior work shows that reproducible recommender evaluation depends on coordinated choices across data preparation, candidate generation, measurement, and statistical testing. Existing frameworks broadened model and tuning support, but Elliot targets remaining gaps in preprocessing breadth and bias, fairness, and significance evaluation.
- Background: Reproducible evaluation protocols identify data splitting, item recommendation, candidate generation, and performance measurement, with dataset collection and statistical testing added in later work.
- Background: Splitting and candidate-generation strategies can substantially alter recommendation results, yet no standard splitting strategy exists and some candidate strategies yield inconsistent measurements.
- Background: Recommendation evaluation spans sometimes-contradictory dimensions including accuracy, coverage, novelty, diversity, bias, and fairness, without an established implementation for the newer bias and fairness measures.
- Related Frameworks: Existing frameworks added models, filtering, splitting, and hyperparameter tuning, but Elliot addresses gaps in preprocessing breadth and support for bias, fairness, and statistical significance tests.
3 ELLIOT
Elliot is an extensible, modular recommendation framework whose configurable pipeline makes experimental flow transparent while connecting inputs, processing modules, evaluation, and outputs.
- The framework is organized around eight functional modules, including configuration, data, run, evaluation, output, and optional modules.
- Figure 1 presents the framework as a pipeline that connects configuration and data modules to run, evaluation, and output modules.
- A customizable configuration file lets researchers specify experiment-flow details while Elliot builds the overall pipeline transparently.
3.1 Data Preparation
Elliot prepares experimental data through configurable loading, filtering, and splitting modules that support side information, reproducible preprocessing, and temporal or random evaluation designs.
- 3.1 Data Preparation: The Data modules load user-item inputs and side information, then pass them to Prefiltering and Splitting modules.
- 3.1 Data Preparation: Loading supports item features, visual embeddings, images, and knowledge-graph semantic features, while required side information can filter out unsuitable items.
- 3.1.2 Prefiltering.: Prefiltering offers threshold-based Filter-by-rating, k-core filtering, iterative k-core with bounded rounds, and retention of cold users.
- 3.1.2 Prefiltering.: Splitting supports Temporal, Random, and Fix strategies, including timestamp-based splits, hold-outs, repeated hold-outs, cross-validation, and precomputed partitions.
3.2 Recommendation Models
Elliot integrates a broad collection of recommendation models and supports configurable hyperparameter optimization across their search spaces.
- 3.2 Recommendation Models: The Recommendation module trains or restores Elliot models and supports integrating new models contributed by users.
- 3.2 Recommendation Models: Elliot integrates 50 recommendation models, including popular, state-of-the-art, graph-learning, visual-based, adversarial-robust, GAN-based, and content-aware recommenders.Thirty of the 38 popular models identified in reviewed frameworks are included.
- 3.2.1 Implemented Models.: Hyperparameter tuning supports Grid Search, Simulated Annealing, Bayesian Optimization, and Random Search, with four strategies for traversing each model’s search space.If users provide hyperparameter lists without a search strategy, Elliot automatically performs exhaustive Grid Search.
- 3.2.2 Hyper-parameter Tuning.: Table 1 summarizes Elliot’s data-elaboration and model-optimization functionalities alongside related frameworks.
3.3 Performance Evaluation
Elliot evaluates recommendations through configurable metrics and statistical tests, covering multiple performance dimensions and enabling hypothesis testing across recommenders.
- 3.3 Performance Evaluation: The evaluation phase contains distinct Metrics and Statistical Tests modules after recommendation training.
- 3.3 Performance Evaluation: Elliot provides 36 metrics across accuracy, error, coverage, novelty, diversity, bias, and fairness families, and any metric can drive model selection and tuning.
- 3.3.2 Statistical Tests.: Elliot supports Wilcoxon and paired t-test hypothesis testing through a configuration-file flag, retaining fine-grained results for multi-recommender evaluation.
3.4 Framework Outcomes
Elliot collects experiment results in configurable reports covering performance values, model parameters, weights, statistical tests, and recommendation lists.
- 3.4 Framework Outcomes: Performance Tables store test-set metric values for every configured model, including cut-off-specific and model-specific results.Tables can also use a <Model, Metric, Value> format and include cut-off-specific statistical tests plus a JSON summary of the best parameters.
- 3.4 Framework Outcomes: Elliot can save model weights to avoid retraining and export top-k recommendation lists as tab-separated User, Item, Predicted Score triples.
3.5 Preparation of the Experiment
Elliot runs extensive experiments from a single YAML configuration file that specifies data preparation, models, optimization, evaluation, and outputs.
- 3.5 Preparation of the Experiment: A YAML configuration file defines data loading, filtering, splitting, models, hyperparameter optimization, evaluation metrics, and output settings.The example uses Item-kNN, grid search, nDCG, and top_k settings.
- 3.5 Preparation of the Experiment: The framework is presented as able to run an extensive set of reproducible experiments by preparing one configuration file.
- 3.5 Preparation of the Experiment: The example configuration loads MovieLens 1M data and applies random subsampling with a 0.2 test ratio.
- 3.5 Preparation of the Experiment: The example configures grid search with cosine similarity and neighbor values of 50 and 100, evaluates nDCG at top-k 10, and enables recommendation saving.
4 EXPERIMENTAL SCENARIOS
Elliot demonstrates how configuration files can drive both basic and advanced reproducible recommendation experiments. The scenarios cover automated preprocessing, hyperparameter tuning, multiple metrics, and increasingly elaborate splitting strategies.
- 4 EXPERIMENTAL SCENARIOS: Elliot’s experimental scenarios show how configuration files drive reproducible recommendation experiments from data preparation through evaluation.The framework is illustrated with both a basic and a more advanced scenario.
- 4.1 Basic Configuration: The basic scenario automates filtering and splitting, grid-search hyperparameter optimization, accuracy and beyond-accuracy metrics at cutoffs 5 and 10, and model selection by nDCG@10.Attribute-I-kNN is compared with Random, ItemKNN, and an external user-implemented model.
- 4.2 Advanced Configuration: The advanced scenario uses random_subsampling for test splitting and random_cross_validation for model selection, with a default top-k cutoff of 50 when none is specified.Its configuration also includes an elaborate data-splitting strategy specified through a small number of configuration fields.
<...> prefiltering:
The configuration specifies preprocessing, splitting, model-search, and evaluation choices, while the resulting report organizes metrics by model and cutoff. The basic results compare selected models across accuracy and coverage measures.
- <...> prefiltering:: Configuration 2 combines user-average filtering, temporal hold-out splitting, external models, candidate hyperparameters, and evaluation at cutoffs 5 and 10.The listed metrics include nDCG, Precision, ItemCoverage, EPC, and Gini.
- <...> prefiltering:: Table 2 presents experimental results for Configuration 2, listing model values for nDCG and item coverage (ICov) at cutoffs 5 and 10.The columns are Model, nDCG@5, ICov@5, nDCG@10, and ICov@10.
5 CONCLUSION
Elliot provides a configuration-driven framework for rigorous, reproducible, multi-recommender evaluation. The conclusion positions its full pipeline as a distinctive contribution while identifying several planned extensions.
- 5 CONCLUSION: The framework supports configurable datasets, iterative k-core prefiltering, random subsampling, five-fold random cross-validation, bounded hyperparameter searches, and multiple accuracy and beyond-accuracy metrics.The advanced configuration enables Wilcoxon testing and includes UserMADrating among its complex metrics.
- 5 CONCLUSION: Elliot provides a full multi-recommender experimental pipeline controlled by a simple configuration file, spanning loading, preprocessing, splitting, optimization, evaluation, and statistical testing.Its reports can be directly analyzed and inserted into research papers.
- 5 CONCLUSION: The authors describe Elliot as the first recommendation framework providing a full multi-recommender pipeline from a simple configuration file, based on their review of existing frameworks.They also discuss the framework’s advantages and limitations in relation to the recommendation-system evaluation literature.
- 5 CONCLUSION: Planned extensions include sequential recommendation, adversarial attacks, reinforcement learning, differential privacy, sampled evaluation, and distributed recommendation.These directions are presented as future additions to Elliot.