PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Difficulty of Evaluating Baselines: A Study on Recommender Systems

Steffen Rendle, Li Zhang, Yehuda Koren

arXiv:1905.01395v1cs.IRcs.LG

TL;DR

Recommender-systems research relies on baseline comparisons, but running baselines properly is difficult. The paper examines ML10M and the Netflix Prize, finding that careful baseline setup changes the reported ML10M comparison and that extensive community tuning is important for reliable empirical conclusions.

  • Problem

    Baseline comparisons are central to recommender-systems evaluation, but even hyperparameter searches may not ensure that methods are run properly.

    Source: §1 · p. 1, §3.3 · p. 10

  • Method

    The authors examine reported ML10M results, carefully rerun established baselines, and compare the findings with the Netflix Prize’s history of method improvement.

    Source: §1 · p. 2, §5 · p. 12

  • Results

    On ML10M, a carefully configured vanilla matrix factorization baseline outperformed reported baseline and newly proposed method results.

    Source: §1 · p. 2, §5 · p. 12

  • Takeaways & Limitations

    The authors conclude that empirical results are more reliable on standardized benchmarks where baselines have been tuned extensively by the research community.

    Source: §4.1 · p. 11

    Recently proposed methods might also improve with more careful tuning, which would further demonstrate the effort required for reliable experiments.

    Source: §2.3 · p. 9

Abstract

from arXiv · show

Numerical evaluations with comparisons to baselines play a central role when judging research in recommender systems. In this paper, we show that running baselines properly is difficult. We demonstrate this issue on two extensively studied datasets. First, we show that results for baselines that have been used in numerous publications over the past five years for the Movielens 10M benchmark are suboptimal. With a careful setup of a vanilla matrix factorization baseline, we are not only able to improve upon the reported results for this baseline but even outperform the reported results of any newly proposed method. Secondly, we recap the tremendous effort that was required by the community to obtain high quality results for simple methods on the Netflix Prize. Our results indicate that empirical findings in research papers are questionable unless they were obtained on standardized benchmarks where baselines have been tuned extensively by the research community.

1 Introduction

Recommender-systems research depends on numerical comparisons with baselines, but running those baselines properly is difficult. On ML10M, carefully configured matrix factorization surpassed reported baseline and newly proposed method results, while the paper argues for extensively tuned standardized benchmarks.

1 Introduction

  • Baseline comparisons are central to judging recommender-systems research, yet running baseline methods properly is difficult.The authors note that reliable experiments require reproducible protocols, tuned baselines, and statistical-significance testing.Source: §1 · p. 1
  • A carefully configured vanilla matrix factorization baseline on ML10M surpassed reported baseline and newly proposed method results, challenging conclusions from five years of benchmark research.The authors also report that well-known Netflix Prize methods performed best on ML10M.Source: §1 · p. 2
  • The authors recommend standardized benchmarks with baselines tuned extensively by the research community rather than one-off evaluations across a few datasets.They also note that suboptimal baselines do not erase the usefulness of techniques such as local low-rank structures, mixtures of matrix approximation, and autoencoders.Source: §1 · p. 2

2 Observations

The paper contrasts reported ML10M progress with results obtained by carefully rerunning established methods. It also connects the benchmark findings to the Netflix Prize, where repeated improvement of methods led to well-calibrated results.

2 Observations

  • Bayesian MF and SVD++ or timeSVD++ enhancements can outperform all previously reported ML10M results, including results for recently proposed algorithms.Bayesian MF had previously been reported among the poorest-performing methods.Source: §2 · p. 2
  • Figure 1 distinguishes inventors’ reported ML10M results from baselines run by authors of newly proposed methods.The cited works are [17, 31, 5, 36, 32, 3, 21, 4, 18, 19].Source: Figure 1 · p. 3
  • Matrix-factorization results varied across Netflix Prize publications, but competition-driven reruns eventually produced well-calibrated results.The authors describe the community effort to improve runs of the same method over time.Source: §2 · pp. 2–3

2.1 Movielens

The section examines reported rating-prediction results on the global-random-split ML10M benchmark, then reruns established matrix-factorization baselines. The reruns and known model enhancements substantially exceed previously reported results.

2.1 Movielens

  • Reported ML10M results portrayed SGD and Bayesian matrix factorization as poor performers, despite their use as common baselines in rating prediction.The benchmark uses RMSE on a global random 90:10 split.Source: §2.1 · p. 3

2.1.1 A Closer Look at Baselines

  • Biased MF and RSVD are essentially the same L2-regularized matrix-factorization method learned with SGD, while BPMF uses Gibbs sampling for the shared matrix-factorization model.ALS-WR is also described as the same model learned with a different algorithm; these distinctions make setup relevant to comparisons.Source: §2.1.1 · p. 4

2.1.2 Rerunning Baselines

  • SGD-MF achieved RMSE 0.7720 at 512 dimensions and RMSE 0.7756 at 64 dimensions, versus RSVD (0.8256) and Biased MF (0.803).The 512-dimensional SGD-MF result also outperformed LLORMA (0.7815), Autorec (0.782), Wemarec (0.7769), and I-CFN++ (0.7754).Source: §2.1.2 · p. 5
  • Bayesian MF achieved RMSE 0.7633 at 512 dimensions, below the previously reported BPMF value (0.8197) and MRMA (0.7634).The previously reported BPMF number is cited to [32].Source: §2.1.2 · p. 5
  • RMSE fell from 0.7587 with time effects to 0.7563 with implicit feedback, 0.7523 jointly, and 0.7485 with flipped implicit usage.The joint model resembles timeSVD++; the implicit model is equivalent to SVD++ [12, 27].Source: §2.1.2 · p. 5
  • A carefully set-up Bayesian matrix factorization baseline improved on the best reported ML10M result, MRMA, by 0.0144; recently proposed methods underperformed well-known baselines.The authors report that Bayesian MF alone outperformed any previously reported result on this benchmark.Source: §2.1.2 · p. 5

2.2 Netflix Prize

The Netflix Prize provides a history of the effort needed to calibrate results for vanilla matrix factorization. Its time-based split and repeated community improvements offer context for the paper’s comparison of benchmark practices.

2.2.1 Background

  • The Netflix Prize took about three years and hundreds of ensembled models to beat Cinematch’s RMSE of 0.9514 by the 10% target.The Grand Prize was awarded in 2009 for RMSE 0.8554.Source: §2.2.1 · p. 7
  • The competition withheld each user’s last six ratings, splitting them between probe and qualifying sets to create a time-based forecasting task.Unlike a global random split, this overrepresents users with little training data and requires extrapolating future ratings.Source: §2.2.1 · p. 7

2.2.2 Matrix Factorization

  • The original Netflix split’s calibrated results are not comparable to results from a global random 90:10 split of the Netflix data.The paper notes that global-random-split evaluations have become common in recent rating-prediction work.Source: §2.2.2 · p. 7
  • FunkSVD achieved probe RMSE 0.93, and later vanilla matrix-factorization reports reached 0.8985 for ALS and 0.8998 for SGD.The reported progression includes values on different sets, including probe and quiz, as specified in the passage.Source: §2.2.2 · p. 8

2.2.3 Refinements and Winning Algorithms

  • After matrix-factorization results converged, top teams added implicit feedback and time effects; winning models also ensembled diverse methods, often dozens per team.The most sophisticated timeSVD++ models achieved RMSEs as low as 0.8762.Source: §2.2.3 · p. 8

2.3 Discussion

The discussion contrasts Netflix Prize results, which became well calibrated through repeated reruns, with ML10M results that missed important method comparisons. It cautions that reported methods may also improve with careful tuning and argues that one-off evaluations are especially questionable.

2.3 Discussion

  • Netflix Prize reruns helped the community converge on which methods work well, while ML10M lacked comparable encouragement to rerun simple baselines.The paper links the difference partly to contest rewards for low RMSE and the limited publication value of strong results from old approaches.Source: §2.3 · p. 9
  • ML10M experiments show that methods reported as poor can perform very well, and that the best Netflix Prize methods also perform best on ML10M.The authors say prior ML10M results failed to reveal these patterns and trade-offs.Source: §2.3 · p. 9
  • Recently proposed methods could also improve with more careful tuning, so their reported results are not necessarily their best results.The authors frame this possibility as further evidence that reliable experimentation requires substantial tuning effort.Source: §2.3 · p. 9
  • One-off evaluations on a few datasets are prone to suboptimal results, and findings from non-standardized benchmarks are more questionable.The paper recommends standardized benchmarks with baselines tuned extensively by the research community for higher-confidence experiments.Source: §2.3 · p. 9

3 Insufficient Indicators for Experimental Reliability

The section argues that familiar safeguards do not establish that methods were set up well. Statistical significance, reproducibility, and hyperparameter search each leave important sources of experimental error unresolved.

3 Insufficient Indicators for Experimental Reliability

  • Statistical significance, reproducibility, and hyperparameter search are necessary but insufficient to ensure reliable results.The authors argue that proper setup matters more than these indicators.Source: §3 · p. 9

3.1 Statistical Significance

  • Statistical significance measures variance within one setup, not whether method A is better than B or whether either method is properly set up.The authors argue that setup errors are a larger source of error and that significance can create false confidence.Source: §3.1 · p. 10

3.2 Reproducibility

  • Reproducing authors’ code and numbers does not establish that the reproduced method was properly set up, even on public datasets with documented protocols.The paper says conclusions can remain questionable despite easy reproducibility on ML10M and the Netflix Prize.Source: §3.2 · p. 10

3.3 Tuned Hyperparameters

  • Hyperparameter search can miss important settings and does not replace method experience; unsearched choices such as shuffling, recentering, early stopping, or initialization also affect setup.The authors describe interpreting search results and transferring settings across model sizes as non-trivial.Source: §3.3 · p. 10

4 Improving Experimental Quality

The section proposes community-level improvements to experimental quality: standardized benchmarks and incentives to tune and improve baselines. It argues that benchmarks need well-defined protocols and that repeated research can still yield well-calibrated results.

4 Improving Experimental Quality

  • Reliable experiments are difficult for a single paper’s authors and require community effort, standardized benchmarks, and incentives to improve baseline results.The authors identify these as two key requirements for improving experimental quality.Source: §4 · p. 11

4.1 Standardized Benchmarks

  • Standardized benchmarks provide well-defined train–test splits and evaluation protocols, but many recurring recommender tasks still lack them.ML10M 10-fold CV and the Netflix Prize split are examples for rating prediction; the authors say communities should converge on benchmarks for recurring problems.Source: §4.1 · p. 11
  • The Netflix Prize showed little overfitting after more than 10 years, and its best-performing methods and patterns also held on ML10M.The authors acknowledge that overfitting may appear in the long run, while arguing that well-calibrated results outweigh risks from improper baselines.Source: §4.1 · p. 11

4.2 Incentives for Running Baselines

  • Novelty-based publication incentives discourage effort on well-known baselines, whereas the Netflix Prize rewarded experimentation with existing methods and better leaderboard results.The authors argue that incentives for tuning known methods on benchmarks are crucial and that this work requires expertise and time.Source: §4.2 · pp. 11–12
  • Machine-learning competitions can provide standardized benchmarks with well-calibrated results.The paper gives Kaggle and conference KDDCup competitions as examples.Source: §4.2 · p. 12

5 Conclusion

The conclusion reports that careful baseline setup overturned prior ML10M comparisons, including results from newly proposed methods. It argues that familiar experimental safeguards do not guarantee reliable results and calls for standardized benchmarks with substantial community tuning.

5 Conclusion

  • Carefully configured vanilla matrix factorization beat reported ML10M baseline results and every reported result from newly proposed methods.The authors also report that other well-known models such as SVD++ provide an even higher gain.Source: §5 · p. 12
  • The ML10M results remained unreliable despite reasonable hyperparameter searches, statistical significance reporting, and reproducibility.The authors say their findings question the practice of running baselines and experimenting on multiple datasets in research papers.Source: §5 · p. 12
  • The paper concludes that trustworthy baselines require standardized benchmarks and considerable community tuning effort.Source: §5 · p. 12

Appendix

Across the reported models, quality improves as embedding dimension grows; Bayesian prediction quality also improves with more sampling steps. The experiments use Movielens 10M and specify distinct setups for Bayesian learning and SGD.

Appendix

  • The Movielens 10M experiments use ten non-overlapping random splits, each with 90% training data and 10% test data.This protocol allows comparison with prior results using a random 90:10 split.Source: Appendix · p. 16

5.1 Factorization Models

  • The Factorization Machine experiments use user, item, time, implicit-user, and implicit-item features.Implicit user information is the set of movie IDs a user watched; implicit item information is the set of user IDs who watched a movie.Source: §5.1 · pp. 16–17

5.2 Bayesian Learning

  • Bayesian models use Gibbs sampling with settings for sampling steps, embedding dimension, and parameter initialization.The authors used Gaussian initialization with standard deviation 0.1 and tested up to 512 sampling steps.Source: §5.2 · p. 17
  • Bayesian models achieve better quality with larger embeddings and more sampling steps.Figure 3 varies embedding dimension with 512 sampling steps; Figure 4 varies sampling steps with 128-dimensional embeddings.Source: §5.2 · p. 17, Figure 3 · p. 17, Figure 4 · p. 18

5.3 Stochastic Gradient Descent

  • SGD tuning searched learning rate and regularization on a validation split, but richer hyperparameter selection may further improve performance.The authors used 5% of training data for validation and selected the same best values across all folds.Source: §5.3 · pp. 18–19
  • Larger embedding dimensions improve quality for both SGD and Gibbs-sampled matrix factorization across the compared range.Figure 5 compares dimensions from 16 to 512; the SGD hyperparameters were selected before final training.Source: §5.3 · p. 19, Figure 5 · p. 19
Loading 1905.01395v1…