Source-linked AI summary
Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches
Maurizio Ferrari Dacrema, Paolo Cremonesi, Dietmar Jannach
TL;DR
It is difficult to determine progress in neural recommendation because published comparisons can rely on weak baselines and be hard to reproduce. This paper systematically assesses recent approaches and finds that most reproduced methods do not consistently outperform simpler alternatives, motivating more rigorous evaluation practices.
Problem
Weak or incomplete baselines and limited reproducibility make it difficult to determine which recommendation method performs best across application scenarios.
Method
The study systematically scans 2015–2018 papers from four conferences, reproduces available results, and reruns experiments with additional nearest-neighbor and graph-based baselines.
Results
6 of 7 investigated deep learning techniques did not consistently outperform simple, fine-tuned baseline methods across the evaluated cases.
Takeaways & Limitations
The findings call for more rigorous evaluation and better research practices when assessing algorithmic contributions in neural recommendation.
Takeaways & Limitations
Hyper-parameter optimization for recommendation methods can require days or weeks even on small datasets, complicating reproducibility and scalability.
Abstract
from arXiv · showhide
Deep learning techniques have become the method of choice for researchers working on algorithmic aspects of recommender systems. With the strongly increased interest in machine learning in general, it has, as a result, become difficult to keep track of what represents the state-of-the-art at the moment, e.g., for top-n recommendation tasks. At the same time, several recent publications point out problems in today's research practice in applied machine learning, e.g., in terms of the reproducibility of the results or the choice of the baselines when proposing new models. In this work, we report the results of a systematic analysis of algorithmic proposals for top-n recommendation tasks. Specifically, we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort. For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques. The remaining one clearly outperformed the baselines but did not consistently outperform a well-tuned non-neural linear ranking method. Overall, our work sheds light on a number of potential problems in today's machine learning scholarship and calls for improved scientific practices in this area. Source code of our experiments and full results are available at: https://github.com/MaurizioFD/RecSys2019_DeepLearning_Evaluation.
1 INTRODUCTION
The study examines whether recent deep learning recommendation research is reproducible and genuinely improves top-n recommendation over simple, well-tuned baselines. Across 18 papers, only 7 were reproducible with reasonable effort, and 6 of those 7 did not consistently outperform simple heuristic methods.
- Motivation: The introduction identifies weak or poorly tuned baselines, inconsistent evaluation practices, and incomplete code or data sharing as barriers to reliable comparison.Differences in datasets, protocols, metrics, preprocessing, and missing preprocessing or tuning code can make cross-paper conclusions difficult.
- Research questions: The study targets two questions: how reproducible recent recommendation research is and whether its algorithms improve performance over relatively simple, well-tuned baselines.These questions motivate a systematic analysis of deep learning methods for top-n recommendation tasks.
- Comparative evaluation: In 6 of 7 reproducible cases, the proposed deep learning techniques did not consistently outperform simple heuristic baselines.The additional baselines included user- and item-based nearest neighbors and two simple graph-based variants.
2 RESEARCH METHOD
The study systematically screened 2015–2018 long papers on deep-learning top-n recommendation from KDD, SIGIR, WWW, and RecSys, then attempted reproduction under strict artifact and data criteria. Reproduced methods were evaluated using original procedures alongside simple baselines.
- Corpus selection: The analysis covered long papers from 2015–2018 in KDD, SIGIR, WWW, and RecSys that proposed deep-learning techniques for top-n recommendation.Studies of other recommendation tasks were excluded, and evaluation had to use classification or ranking metrics such as Precision, Recall, or MAP.
- Reproducibility: Reproduction relied primarily on author-provided artifacts, supplemented by contacting authors for missing code or data and waiting 30 days for responses.Skeleton code, exclusively private company data, and unshared web-collected data were classified as non-reproducible.
- Reproducibility: Only about one third of the analyzed works were reproducible, although the rate increased over the years; the sample was too small for reliable conference comparisons.Reproducibility required working or minimally modifiable code, at least one available dataset, and accessible or reconstructable train-test splits.
- Measurement Method: The authors refactored original implementations to separate training, hyper-parameter optimization, and prediction from a shared evaluation procedure also applied to baselines.This preserved each paper’s original evaluation procedure while enabling consistent baseline comparisons.
- Baselines: The baseline suite comprised conceptually simple methods including TopPopular, ItemKNN, UserKNN, content-based and hybrid ItemKNN variants, and the graph-based P3α algorithm.These methods use popularity, neighborhood similarities, item features, combined ratings and features, or three-step user-item random walks.
3 VALIDATION AGAINST BASELINES
Across reproducible recommendation methods, simple personalized, nearest-neighbor, graph-based, and linear baselines frequently matched or outperformed neural approaches. Mult-VAE was the clearest exception, consistently outperforming the tested baselines by 10–20% across configurations.
- CMN: CMN was never the best method on any dataset; personalized baselines often surpassed it, while unpersonalized TopPopular won on Epinions by a large margin.CMN was nevertheless much better than the other algorithms on Epinions apart from TopPopular, according to the supplied passage.
- MCRec: Correctly configured ItemKNN outperformed the complete MCRec model on every reported performance measure for MovieLens100k.The evaluation procedure was reproduced, but detailed results were limited to MovieLens because meta-path construction code was unavailable for the other datasets.
- CVAE: CVAE was favorable over baselines only at longer, relatively uncommon recommendation cutoffs, whereas pure collaborative-filtering methods often won at shorter list lengths.At longer list lengths, hybrid ItemKNN-CFCBF achieved the best results on dense CiteULike-a, with similar findings on sparse CiteULike-t.
- CDL: CDL showed the same short-list pattern as CVAE: pure collaborative-filtering baselines beat the hybrid method at short list lengths, while CDL achieved higher Recall beyond 100 in two of four dataset configurations.The comparison was conducted on dense CiteULike-a using reproduced experiments.
- NeuMF and SLIM: NeuMF was beaten by two personalized baselines on every Pinterest metric, but it clearly outperformed the simple baselines on MovieLens; SLIM also surpassed NeuMF on its evaluated dataset.These comparisons show that neural-method performance depended strongly on the dataset and baseline configuration.
- Mult-VAE: Mult-VAE consistently outperformed the simple baselines by 10%–20% across all tested configurations, providing the clearest case where a complex method won by a large margin.Its improvements over SLIM were small for NDCG but solid for Recall; additional cutoff measurements were reported because the original cutoff choices were inconsistent.
4 DISCUSSION
The discussion concludes that progress in neural recommendation is unclear because several neural methods do not outperform simpler algorithms, while weak baselines and inconsistent research practices obscure comparisons. Reproducibility is also limited by missing experimental code and substantial computational demands.
- 4 DISCUSSION: Reproducibility remains low because researchers often omit code for hyperparameter optimization, evaluation, preprocessing, and baseline implementations.Core algorithm code is shared more often than before, but missing surrounding experimental code makes reported findings difficult to validate.
- 4 DISCUSSION: Hyperparameter optimization can take days or weeks even on small datasets with GPU access, making computational complexity another barrier to reproducibility.The discussion contrasts current datasets containing a few hundred thousand ratings with the much larger Netflix Prize dataset.
- 4 DISCUSSION: Several recent neural recommendation methods do not outperform conceptually or computationally simpler algorithms, making the field’s achieved progress unclear.This conclusion holds despite the computational complexity of many proposed methods.
- 4 DISCUSSION: Poorly optimized or insufficiently documented baselines, data-splitting mistakes, and evaluation-protocol errors can create claims of phantom progress.The discussion notes that complex neural models may be compared against weak neural baselines; NCF [14] is outperformed by simple baselines on one dataset and offers little improvement on another, where linear regression also outperforms it.
- 4 DISCUSSION: Progress is difficult to assess because studies vary widely in datasets, evaluation protocols, metrics, and baselines, with over 20 public datasets used.Many datasets appear in only one or two papers, while metrics include Precision, Recall, Mean Average Precision, NDCG, and MRR.
- 4 DISCUSSION: The discussion attributes these problems to accuracy-centered research that often lacks a concrete hypothesis, including evaluating implicit feedback with MovieLens rating datasets.It argues that the search for marginally higher accuracy can dominate research without clear evidence that such gains add recommendation value.
5 SUMMARY
The analysis finds that reproducing recent neural top-n recommendation research remains challenging and that simpler algorithms often outperform most reviewed methods on some datasets. It therefore calls for more rigorous evaluation practices, while noting that the study is limited in scope and should be extended to other outlets, recommendation problems, and traditional baselines.
- 5 SUMMARY: Reproducing published neural recommendation research remains challenging, and most reviewed methods can be outperformed on some datasets by simpler algorithms.The simpler alternatives are described as conceptually and computationally less complex.
- 5 SUMMARY: The findings call for greater rigor and improved research practices when evaluating algorithmic contributions to recommendation.
- 5 SUMMARY: The analysis is limited to selected conference series, with planned extensions to other outlets, recommendation problems, and traditional baselines such as matrix factorization.