PaperlayerView exact revision on arXiv

Source-linked AI summary

A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research

Maurizio Ferrari Dacrema, Simone Boglio, Paolo Cremonesi, Dietmar Jannach

arXiv:1911.07698v3cs.IRcs.LGcs.NE

TL;DR

The paper examines whether claimed advances from increasingly deep neural methods reflect actual progress in top-n recommendation, given concerns about baseline choice, optimization, and reproducibility. It reproduces recent neural collaborative-filtering results and finds limited progress relative to simple and established non-neural methods, while identifying methodological issues and publication bias as relevant constraints.

  • Problem

    Evidence suggests increasingly deep learning methods are not consistently better than long-existing non-neural methods, while baseline choice, optimization, and standardized benchmarks complicate claims about progress.

  • Method

    The paper reproduces recent neural recommendation approaches based on collaborative filtering and analyzes their comparisons with tuned established baselines.

  • Results

    In 8 reproducible cases, simple nearest-neighbor approaches outperformed recent neural methods; across 12 methods, only 1 was clearly better than non-neural baselines on one dataset.

  • Takeaways & Limitations

    The analyzed evidence indicates limited progress and apparent stagnation in top-n recommendation, with baseline selection and optimization representing key methodological issues.

  • Takeaways & Limitations

    The conclusions are limited to the papers analyzed, while publication bias makes reproducibility studies harder to publish.

Abstract

from arXiv · show

The design of algorithms that generate personalized ranked item lists is a central topic of research in the field of recommender systems. In the past few years, in particular, approaches based on deep learning (neural) techniques have become dominant in the literature. For all of them, substantial progress over the state-of-the-art is claimed. However, indications exist of certain problems in today's research practice, e.g., with respect to the choice and optimization of the baselines used for comparison, raising questions about the published claims. In order to obtain a better understanding of the actual progress, we have tried to reproduce recent results in the area of neural recommendation approaches based on collaborative filtering. The worrying outcome of the analysis of these recent works-all were published at prestigious scientific conferences between 2015 and 2018-is that 11 out of the 12 reproducible neural approaches can be outperformed by conceptually simple methods, e.g., based on the nearest-neighbor heuristics. None of the computationally complex neural methods was actually consistently better than already existing learning-based techniques, e.g., using matrix factorization or linear models. In our analysis, we discuss common issues in today's research practice, which, despite the many papers that are published on the topic, have apparently led the field to a certain level of stagnation.

1 INTRODUCTION

Deep learning methods dominate recommender-systems research, but methodological choices complicate claims of progress. Reproducing recent neural collaborative-filtering results suggests that simple and established non-neural methods often perform as well or better.

  • Motivation: Deep learning techniques have become particularly popular and now dominate recommender-systems research.Their adoption was encouraged by success in language and image processing, despite their computational complexity.
  • Motivation: Recent evidence from information retrieval, time-series prediction, and session-based recommendation questions whether increasingly deep learning methods are consistently beneficial.Reported comparisons found long-existing non-neural or statistical methods sometimes outperforming recent neural approaches.
  • Methodological concern: Researchers can choose metrics, protocols, datasets, and baselines, while published claims are often broader than the specific experimental conditions supporting them.A comparison is informative only for a particular combination of performance measure, evaluation procedure, and dataset when both algorithms are properly optimized.
  • Study aim: The study reproduced results from recent top-tier conference papers to assess whether deep learning advances top-n recommendation from user-item rating matrices.The authors identified 26 relevant papers and reproduced 12 of them with reasonable effort.
  • Main finding: 8 reproducible cases favored simple long-known approaches, while only 1 of 12 neural methods was clearly better than non-neural baselines on one dataset.The non-neural comparisons included nearest-neighbor methods, matrix factorization, and linear models.

2 IDENTIFYING RELEVANT AND REPRODUCIBLE WORKS

The study systematically selected recent neural collaborative-filtering papers from major conferences and applied a conservative reproducibility definition based on runnable author-provided implementations and accessible evaluation data. Of 26 relevant papers, 12 met the reproducibility criteria.

  • Selection criteria: The sample targeted new neural collaborative-filtering methods for traditional top-n item-ranking tasks, including eligible hybrid methods.Session-based and group-recommendation papers were excluded.
  • Selection criteria: Eligible papers were published between 2015 and 2018 in SIGIR, KDD, TheWebConf, IJCAI, WSDM, or ACM RecSys.These venues were selected as major recommender-systems research outlets.
  • Reproducibility criteria: Reproducibility required runnable author-provided source code containing the model, data structures, training loop, and recommendation computation.The authors contacted researchers when code was unavailable or incomplete and treated no positive response after 30 days as unavailable.
  • Reproducibility criteria: The evaluation also required at least one publicly available dataset, with shared train/test splits used when available.Without shared splits, reproducibility depended on sufficient reporting of the experimental setup.
  • Outcome: The authors observed an increase over time in researchers sharing source code, possibly reflecting the growing importance of reproducibility in peer review.This was identified as a possible explanation rather than a demonstrated cause.

3 EVALUATION METHODOLOGY

The evaluation reran original experiments while adding systematically tuned baselines from multiple algorithm families. It preserved original datasets and evaluation settings, then optimized baseline hyperparameters for fairer comparisons.

  • Evaluation setup: The authors used each original method’s datasets, protocol, and performance metrics, while adding additional baselines.This design aimed to make comparisons fair within the experimental settings of the original papers.
  • Evaluation setup: The study refactored original code into a common evaluation framework while keeping core algorithm implementations unaltered.Datasets reported in the original papers and shared train/test splits were used when available.
  • Optimization: All examined baselines underwent extensive hyperparameter optimization, whereas neural methods generally used optimal values reported in their original papers.SpectralCF was the exception because its dataset splits required new optimization.
  • Optimization: Bayesian search evaluated 50 cases per algorithm, including the number of training epochs, before fitting the final model on combined training and validation data.The first 15 search cases were random initial points.
  • Baseline suite: The baseline suite covered popularity ranking, nearest-neighbor, graph-based, content-based, matrix-factorization, and linear approaches.The study selected algorithms from different families to obtain broad comparisons.
  • Graph-based methods: P3α and RP3β model graph-based item relations through two-step random walks, with RP3β additionally correcting similarities by item popularity.Both methods use neighborhood size and graph-related parameters as hyperparameters.
  • Optimization: Training epochs and stopping criteria were often omitted in the analyzed papers, although epoch selection can materially affect learning-model performance.The study therefore treated the number of epochs as a tunable parameter selected using validation data.

4 RESULTS – ANALYSIS OF REPRODUCIBILITY AND PROGRESS

Across the reproduced neural recommendation methods, simple nearest-neighbor and established machine-learning baselines frequently matched or surpassed the proposed approaches. The comparisons also exposed evaluation and reproducibility issues that affected several reported results.

  • CDL and CVAE: Simple hybrid baselines outperformed CDL and CVAE on three of four datasets, while each neural method was superior on only one very sparse dataset.The sparse datasets had only one interaction per user in training.
  • Neural collaborative filtering: Nearest-neighbor methods outperformed NCF on one of two datasets, while non-neural machine-learning methods remained better on the other.
  • Deep matrix factorization: Three of four datasets favored long-known baselines over the proposed method, with superiority appearing only on one very sparse dataset.The reported Hit Rate advantage occurred on that sparse dataset.
  • Mult-VAE: Mult-VAE outperformed neighborhood baselines across all metrics on two reproduced datasets, but SLIM was better on the second dataset when optimizing for NDCG.
  • Reproducibility and evaluation: Several evaluations were undermined by unreproduced results, test-data epoch selection, unusual cold-item splits, or fixed embedding sizes rather than dataset-specific tuning.For DELF, removing cold items caused performance to fall below machine-learning baselines; MCRec was reproducible only on MovieLens, where item-based nearest neighbors won on every measure.
  • Dataset and split effects: CMN was outperformed by simpler baselines on two of three datasets, while its remaining advantage was attributed to popularity bias in a highly skewed dataset.SpectralCF was competitive only on a shared split and did not work as expected after independently recreating the splits.

5 DISCUSSION

The analysis finds limited progress from deep learning in the examined top-n recommendation papers and identifies reproducibility and methodological practices that can create misleading accuracy gains. It therefore proposes guidelines and best practices aimed at improving reproducibility and progress.

  • Reproducibility: 12 of 26 papers could be reproduced using provided code and publicly available datasets.More than half lacked code or important details needed for reliable reproduction.
  • Methodological Issues: Baseline selection and tuning can distort state-of-the-art comparisons, especially when simple methods are omitted or baselines are not fully optimized.The paper states that optimization should include all relevant parameters and all baselines, including shrinkage for neighborhood methods.
  • Methodological Issues: Evaluation choices varied across 18 datasets, splitting procedures, negative-sampling counts, and metrics, making comparisons difficult.The reproducible papers used different data splits, 50, 100, or 1000 negative samples per positive sample, and metrics including Hit Rate, Precision, Recall, MAP, and NDCG.
  • Practical Considerations: Neural methods can require substantially more training time than simpler alternatives, although scalability is not presented as a methodological issue by itself.NeuMF requires four hours on MovieLens1M, compared with four minutes for SLIM and about 12 seconds for EASE_R.
  • Fundamental Issues: Flexible choices of datasets, metrics, protocols, and baselines increase opportunities to report apparent accuracy improvements.Examining many combinations can increase the probability that some combination makes a novel algorithm appear superior.
  • Fundamental Issues: Offline accuracy gains do not necessarily produce better perceived recommendation quality or business value.The paper distinguishes recommender-systems evaluation from domains where accuracy improvements more directly translate into better systems.
  • Guidelines: The paper recommends guidelines for methodological and publication practices, including publicly reporting information and materials needed for reproduction.It also recommends actions for journal editors and conference chairs.

6 CONCLUSION

The study finds limited progress in top-n recommendation despite many published methods, with methodological issues surrounding baseline choice and optimization. It calls for greater reproducibility and broader evaluation approaches that include humans and theoretical considerations.

  • Simple methods sometimes matched or outperformed recent complex neural methods, while neural methods often failed to outperform matrix factorization or linear models.The authors identify baseline choice and inadequate baseline optimization as key methodological issues.
  • Increasing reproducibility is presented as one strategy for mitigating observed methodological problems in recommender-systems research.The authors also recommend greater community awareness and corresponding initiatives.
  • The authors argue that methodological problems contribute to stagnation and that algorithm rankings depend on multiple factors rather than a single best model.They question how much offline experiments and computational accuracy measures alone can reveal.
  • Future evaluation should use multi-method approaches that consider the human in the loop and research grounded more in theory than network-architecture engineering.

A DETAILED ANALYSIS FOR REPRODUCIBLE ARTICLES

The appendix section introduces a detailed discussion of the analyzed papers and selected numerical results.

  • The appendix discusses the analyzed papers in greater detail and provides selected numerical results.

A.1 Collaborative Deep Learning for Recommender Systems (CDL)

The CDL and CVAE evaluations reproduce the reported setups but show that tuned simple baselines frequently outperform the neural methods. CVAE improves over CDL, yet CDL is characterized as a weak baseline.

  • Datasets: The evaluation used two publicly available CiteULike datasets because content information paired with the Netflix dataset was unavailable.
  • Evaluation: The original evaluation sampled P interactions per user for training and reported Recall at list lengths 50–300 plus Mean Average Precision at length 300.
  • CDL results: In three of four configurations, tuned ItemKNN CFCBF consistently outperformed CDL, often joined by ItemKNN CBF; CDL was substantially better only when P=1 on CiteULike-a.
  • CDL results: For CiteULike-a with P=10, even simple collaborative-filtering approaches outperformed CDL, while iALS was better in every reported measurement and cutoff length.
  • Reproducibility: The original evaluation contained an error that inflated absolute metric values, although the relative performance ordering remained unchanged after correction.
  • CDL results: The best results were achieved by ItemKNN CBF, which uses only item features to recommend similar items.
  • CVAE evaluation: CVAE was evaluated on the same CiteULike datasets and sparsity configurations as CDL, using Recall at different long list lengths.
  • CVAE results: On CiteULike-a with P=10, simple hybrid baselines outperformed CVAE on all measures, although CVAE consistently outperformed CDL.

A.3 Neural Collaborative Filtering (NCF)

The NCF evaluation compares neural collaborative filtering with non-neural methods on MovieLens and Pinterest under a leave-last-out protocol. NCF is competitive on MovieLens but generally does not outperform tuned non-neural methods, especially on Pinterest.

  • Datasets: The evaluation used MovieLens1M ratings converted to implicit feedback and a preprocessed Pinterest dataset containing 1.5 million interactions.
  • Evaluation: The authors used leave-last-out evaluation, hiding each user’s final timestamped interaction and ranking it against other recommendable items.
  • Baselines: The original ItemKNN tuning varied neighbor count but omitted other configurations, including shrinkage and normalization.
  • Evaluation: Training epochs were selected using test-set Hit Rate in the provided code, so the reproduction instead selected epochs using validation-set performance.
  • Results: NeuMF was competitive on MovieLens but was outperformed by all but one non-neural machine-learning method.
  • Results: On Pinterest, NeuMF outperformed only PureSVD, while most non-neural machine-learning techniques were similar or better.
  • Interpretation: The analysis notes that neural models can impose impractical computation costs when efficient dot-product algorithms are available.
  • Results: A well-tuned ItemKNN method achieved similar and sometimes better performance than machine-learning methods on Pinterest, unlike the clearer machine-learning advantage on MovieLens.

A.4 Deep Matrix Factorization (DMF)

DMF was evaluated on four filtered public rating datasets using leave-last-out ranking with Hit Rate and NDCG@10. It did not consistently outperform simple or established learning-based baselines, except notably on the extremely sparse Amazon Movies dataset.

  • Results: For three of four datasets, a simple baseline outperformed DMF on both evaluation measures.The reproduced experiments also did not confirm that normalized loss nce improved accuracy over binary loss bc.
  • Results: On both MovieLens datasets, iALS and SLIM outperformed DMF on both measures, while DMF exceeded traditional nearest-neighbor baselines on MovieLens1M.The smaller MovieLens100k showed a generally similar ranking.
  • Results: Amazon Music favored simple UserKNN and RP3β over DMF, whereas DMF was much better than other methods on Amazon Movies.EASER results for Amazon Movies were unavailable because its implementation required too much memory.
  • Interpretation: DMF’s stronger Amazon Movies result coincided with extreme sparsity after preprocessing removed 80% of interactions.The paper relates this pattern to better results previously observed for other methods on very sparse datasets.

A.5 Variational Autoencoders for Collaborative Filtering (Mult-VAE)

Mult-VAE was evaluated on explicit-feedback movie datasets and a music dataset under a user-holdout protocol. Its results were positive on MovieLens but mixed on Netflix, while EASER improved over it in most cases.

  • Datasets: Mult-VAE was evaluated on binarized MovieLens20M, Netflix, and music datasets, with users having insufficient interactions removed.Movie ratings above three were treated as positive signals; music users needed more than 20 interactions.
  • Evaluation: The evaluation held out users for validation and testing and compared Mult-VAE with iALS, SLIM, NCF, and CDAE.This protocol means UserKNN could not be applied to the MovieLens20M evaluation.
  • Results: On MovieLens, Mult-VAE outperformed all baselines on every reported and additional measurement, with SLIM about 1% to 2% lower in NDCG.The reproduction confirmed the authors’ positive claim for this dataset.
  • Results: On Netflix, SLIM outperformed Mult-VAE at every list length for NDCG, while Mult-VAE performed better for Recall.NDCG was the optimization criterion in the original paper.
  • Conclusion: EASER improved over Mult-VAE in most cases, supporting shallow models as a competitive solution.Mult-VAE was easy to reproduce because the authors publicly provided the required materials and consistently outperformed previous methods on at least one large, well-known dataset.

A.6 NeuRec: On Nonlinear Transformation for Personalized Ranking

NeuRec was assessed across four public datasets using multiple random splits and ranking metrics, but its reported results could not be reconstructed. In the reproduction, baselines generally outperformed it.

  • Datasets and evaluation: NeuRec was evaluated on MovieLens1M, HetRec, FilmTrust, and Frappe using five random 80%/20% training-test splits.Metrics included Precision, Recall, MAP, MRR, and NDCG at specified list lengths.
  • Baselines: The comparison included SLIM, BPR matrix factorization, NeuMF, and GMF, but baseline hyperparameter tuning was largely undocumented.NeuRec’s hyperparameters were selected by grid search and reported in detail.
  • Reproducibility: The reproduction could not obtain results comparable to those reported in the original NeuRec paper despite a runnable implementation and detailed hyperparameters.The discrepancy remained unexplained after contacting the authors.
  • Results: NeuRec was outperformed on every dataset and almost every measure by at least one baseline, usually several.On MovieLens, even the simplest baselines were better, with the best baselines ahead by a large margin.
  • Results: NeuRec led only on Precision@5 for the small Frappe dataset and was outperformed by RP3β on all other measures.The paper notes that Frappe results were unstable across random splits.

A.7 CoupledCF: Learning Explicit and Implicit User-item Couplings

CoupledCF combines user-item information on MovieLens1M and Tafeng, but its reproduced performance was not consistently superior to simpler methods. Non-neural baselines often matched or exceeded it, and the simpler DeepCF variant sometimes performed better.

  • Datasets and evaluation: CoupledCF was evaluated on MovieLens1M and the sparser Tafeng transaction dataset using leave-one-out ranking with 99 sampled negatives.Hit Rate and NDCG were measured at list lengths from 1 to 10.
  • Methodological assessment: The proposed model was systematically fine-tuned, whereas baseline hyperparameter tuning was not reported and test-set epoch selection was observed in the source code.The reproduction instead selected the optimal epoch with early stopping.
  • Reproducibility: The reproduced CoupledCF results were incomplete because the authors’ results could not be fully reproduced and apparent sampling issues affected the experiments.The experiments therefore used CoupledCF and its simplest DeepCF variant.
  • MovieLens results: On MovieLens, CoupledCF nearly always beat neighborhood and hybrid methods except at very short list lengths, but iALS and EASER were consistently better.The differences between CoupledCF and DeepCF were tiny, unlike the large differences reported originally.
  • Tafeng results: On Tafeng, nearest-neighbor methods outperformed CoupledCF by far, while CoupledCF was roughly at the level of TopPopular.DeepCF also produced better accuracy than CoupledCF; only pure content-based baselines failed to reach CoupledCF’s performance.

A.8 DELF: A Dual-Embedding based Deep Latent Factor Model for Recommendation

DELF was evaluated on MovieLens1M and Amazon Music under a leave-last-out protocol using Hit Rate and NDCG at 10. Its apparent Amazon Music advantage depended on cold-item recommendations; without them, established baselines consistently performed better.

  • Datasets: MovieLens1M and Amazon Music were binarized and filtered to users with more than 20 interactions.The Amazon Music dataset was reduced to less than one tenth of its original interaction count after preprocessing.
  • Evaluation: Leave-last-out evaluation ranked each hidden item against 99 sampled negatives using Hit Rate and NDCG@10.Baselines included BPR matrix factorization, iALS, DMF, and two NCF variants.
  • Results: On MovieLens1M, every machine-learning baseline outperformed DELF on all measures, while UserKNN was also competitive.NDCG@10 was used as the optimization criterion, and baseline hyperparameters were optimized extensively.
  • Results: On Amazon Music with cold items allowed, DELF EF achieved substantially better accuracy than all baselines.The test data contained an anomalously high number of cold items.
  • Results: When cold items were excluded, DELF was never best-performing: iALS, EASE_R, SLIM, and UserKNN outperformed it across the reported comparisons.On Amazon Music, iALS was better than all other methods, while DELF consistently beat only PureSVD among the compared methods.
  • Methodological issues: The evaluation used test data to select the number of epochs, causing information leakage from the test set.This methodological issue limits the validity of the reported comparison.

A.9 Outer Product-based Neural Collaborative Filtering (ConvNCF)

ConvNCF was evaluated on Yelp and Gowalla with leave-last-out ranking against sampled negatives. Traditional neighborhood methods generally outperformed it, while the evaluation also contained several methodological issues affecting the comparison.

  • Datasets: ConvNCF used filtered implicit-feedback versions of Gowalla and Yelp, retaining one earliest interaction per user-item pair.Items and users below dataset-specific interaction thresholds were removed, and author-provided train/test splits were used.
  • Evaluation: Each dataset used leave-last-out evaluation with 999 sampled negatives, measuring Hit Rate and NDCG at multiple list lengths.Embedding size was not tuned and was kept constant for all baselines.
  • Results: On Yelp, ConvNCF was consistently outperformed by RP3β, SLIM, and traditional nearest-neighbor methods.Other baselines also outperformed ConvNCF in most cases.
  • Results: On Gowalla, UserKNN outperformed ConvNCF in all but one measurement.ConvNCF was more competitive there, but simple machine-learning methods did not beat the simple baselines.
  • Methodological issues: The provided evaluation had test-set epoch selection, a fixed embedding size of 64, duplicate or overlapping negative samples, and incomplete sets of unique negatives.These issues compromise the comparability and validity of the reported baseline results.
  • Related comparison: The paper reports that CMN was slightly outperformed on all measures by algorithms from all method families, including traditional nearest-neighbor methods.On the smaller CiteULike dataset, CMN was often much worse than nearest-neighbor and machine-learning methods; TopPopular was best across all Epinions measurements.

A.12 Spectral Collaborative Filtering (SpectralCF)

SpectralCF was evaluated in regular and cold-start settings on three public datasets, but its reported strength depended on an anomalous MovieLens split. After replacing that split, all comparison baselines outperformed it, often matching TopPopular.

  • Datasets: SpectralCF was evaluated on binarized MovieLens1M, HetRec, and InstantVideo datasets after interaction-based filtering.MovieLens1M was reduced to 226k interactions, while the other datasets contained 71k and 22k interactions.
  • Evaluation: The regular evaluation randomly assigned 80% of each user’s interactions to training and used Recall and MAP at multiple list lengths.A cold-start setting varied each user’s training interactions from P=1 to P=5 and used Recall@20 and MAP@20.
  • Initial results: SpectralCF initially performed very well on MovieLens1M with the provided split but was relatively weak on HetRec and Amazon Video.The authors’ baselines included ItemKNN, BPR matrix factorization, iALS, NCF, GNMF, and GCMC.
  • Revised results: After new splits were created following the described procedure, SpectralCF was outperformed by all comparison baselines on MovieLens1M.It often exhibited recommendation quality equal to the TopPopular method, with similar observations on the other datasets.
  • Split analysis: The provided MovieLens split had a test Gini index of 0.92 versus 0.78 for the unsplit data, indicating unusually high popularity bias.Its test entropy was 8.5, compared with approximately 10 for the original data and a random split, making the test set easier to predict.
  • Benchmarking: The study optimized baseline hyperparameters individually and documented their ranges and distributions.The KNN normalization settings controlled similarity computation and row-wise L1 regularization.
Loading 1911.07698v3…