Source-linked AI summary
Training seeds and model-selection stability in recommender-system evaluation
Juan Manuel Rodriguez, Oleg Lesota, Antonela Tommasel
TL;DR
The paper asks whether single-seed recommender experiments provide stable evaluation conclusions. It fixes the data partition and varies training seeds across configurations, finding detectable effects on metrics, model selection, validation–test transfer, and recommendation lists. The results support treating training seeds as part of the evaluation protocol.
Problem
Recommender results may depend on training seeds even when other experimental choices are fixed, limiting evidence about evaluation and model-selection stability.
Method
The study fixes the data partition and varies a single training seed across hyperparameter configurations, evaluating metric sensitivity, model selection, validation–test consistency, and list agreement.
Results
Changing seeds produces detectable user-level score differences, can alter validation-based choices and their test transfer, and can change top-k recommendation lists.
Takeaways & Limitations
Training seeds should be treated as part of the recommender-system evaluation protocol rather than as incidental implementation noise.
Takeaways & Limitations
The study captures full-run variability without disentangling initialization, batching, dropout, latent sampling, or training-time negative sampling.
Abstract
from arXiv · showhide
Recommender-system experiments often rely on a single random training seed, assuming that run-to-run stochasticity has limited impact on evaluation conclusions. This assumption is risky, as a training seed may influence several algorithm-dependent mechanisms, including parameter initialization, mini-batch ordering, dropout, masking, latent sampling, and training-time negative sampling. We examine this assumption by fixing the data partition and varying the training seed across hyperparameter configurations. We analyze seed effects at three levels: user-level metric sensitivity, validation-based model selection and recommendation-list agreement. Results show that seed variation is often detectable. Its impact depends on whether configurations are clearly separated, whether validation results transfer to test, and whether similar scores lead to similar top-$k$ lists. Findings suggest that reporting single-seed results can overstate the stability of recommender system evaluation, and that training seeds should be treated as part of the evaluation protocol rather than as incidental implementation noise.
1 Introduction
The paper investigates whether changing only the training seed can alter recommender-system evaluation and the conclusions drawn from fixed experimental setups. It treats the seed as controlling stochasticity throughout the full training run and examines effects on metrics and model selection.
- Changing the training seed may affect reported metrics and model-selection conclusions even when the data partition, hyperparameters, and evaluation protocol are fixed.
- The seed is treated as controlling stochastic components of the full training run rather than only parameter initialization.
- The study examines whether repeated seed changes alter user-level results, validation-to-test comparisons, configuration rankings, and validation-based selection losses.
- Detectable metric variation can leave the selected configuration unchanged, whereas near-tied validation results may change model selection.
2 Methodology
The methodology isolates training-seed variability by fixing the data partition and evaluating repeated seeds across datasets, recommender models, and hyperparameter configurations. It measures effects on performance estimates, model selection, validation–test transfer, and recommendation-list agreement.
- The evaluation fixes the data partition and measures training-seed effects on performance estimates, model-selection decisions, and recommendation lists.
- Three datasets use temporal, user-based train, validation, and test splits with an 80%/10%/10% allocation.
- Four models span matrix-factorization and sequential neural approaches: BPR, NeuMF, BERT4Rec, and SASRec.
- For each model–configuration pair, training is repeated with seeds {1, . . . , 10}, retaining the checkpoint with the best validation performance.
- Using nDCG@10 as the primary ranking metric, the study tests user-level metric distributions across seeds with the Friedman test.
- Model-selection analysis compares validation–validation and validation–test performance gaps and tests whether validation configuration wins transfer to test.
3 Experimental results
Training-seed effects are often statistically detectable, but their practical consequences vary with configuration separation, validation–test transfer, and dataset-dependent recommendation-list agreement. Metric sensitivity, model-selection instability, and list instability therefore represent related but distinct effects.
- Metric-level sensitivity: Steam shows the strongest metric-level signal, with Friedman tests rejecting the null for all configurations of all models on validation and test.
- Metric-level sensitivity: Seed sensitivity may remain within clearly separated configuration regions, whereas overlapping configurations such as NeuMF produce flatter comparisons.
- Model-selection stability: Model-selection effects range from small losses among similarly performing configurations to large validation-selection gaps and weak transfer to test.
- Validation–test consistency: Validation–test transfer depends on both dataset and model: it is generally informative on ML-1M, weaker on Amazon, and usually strong on Steam.
- Validation–test consistency: Validation selection can appear stable while the chosen configuration transfers unreliably to test, as illustrated by BPR on Amazon.
- Recommendation-list agreement: BPR lists are moderately similar on ML-1M, almost disjoint on Amazon All Beauty, and highly consistent on Steam.
- Recommendation-list agreement: Average Overlap@k exceeds Jaccard@k across datasets, yet agreement remains below 1, so repeated runs are not list-invariant.
4 Conclusions
Training-seed changes can affect recommender evaluation through user-level scores, validation-based configuration choices, and top-k recommendation lists. These effects vary in practical meaning, motivating treatment of training seeds as part of the evaluation protocol and highlighting limitations from fixed data partitions and un disentangled stochastic mechanisms.
- Training seeds can produce detectable differences in user-level scores, influence validation-based configuration choices, and change top-k recommendation lists.
- Some seed effects are statistically detectable without substantially changing model-selection outcomes, while others reveal weaker validation–test transfer or different lists under similar aggregate performance.
- Training seeds should be treated as part of the evaluation protocol rather than as incidental implementation noise.
- The analysis captures run-to-run variability but does not attribute it to specific stochastic mechanisms such as initialization, dropout, or training-time negative sampling.
- The study fixes the data partition, leaving future work to jointly analyze data-split, training, and evaluation-sampling seeds across additional models and datasets.