Source-linked AI summary
Exploring Data Splitting Strategies for the Evaluation of Recommendation Models
Zaiqiao Meng, Richard McCreadie, Craig Macdonald, Iadh Ounis
TL;DR
Data splitting is often overlooked in recommender-system evaluation, and prior work may not describe it sufficiently for reproducibility. The paper analyzes splitting strategies and model performance, finding that the strategy can markedly alter state-of-the-art system rankings.
Problem
Data splitting is often overlooked and insufficiently detailed for reproducibility, while evaluation methods face greater difficulty distinguishing models as performance gains become incremental.
Source: §1 · pp. 1–2Method
The paper analyzes recent recommendation literature and compares seven state-of-the-art models across three splitting strategies.
Source: §1 · p. 2Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Effective methodologies for evaluating recommender systems are critical, so that such systems can be compared in a sound manner. A commonly overlooked aspect of recommender system evaluation is the selection of the data splitting strategy. In this paper, we both show that there is no standard splitting strategy and that the selection of splitting strategy can have a strong impact on the ranking of recommender systems. In particular, we perform experiments comparing three common splitting strategies, examining their impact over seven state-of-the-art recommendation models for two datasets. Our results demonstrate that the splitting strategy employed is an important confounding variable that can markedly alter the ranking of state-of-the-art systems, making much of the currently published literature non-comparable, even when the same dataset and metrics are used.
1 INTRODUCTION
The introduction identifies data splitting as an underreported evaluation choice without common guidance in recommender systems. It examines how splitting strategies affect recommendation-model performance and reports that they can alter system rankings even under the same dataset and metrics.
1 INTRODUCTION
- As performance gains become more incremental, evaluation must distinguish between systems with similar performance.Source: §1 · p. 1
- Recommender-systems papers vary in datasets, metrics, baselines, and splitting strategies, making progress difficult to measure [3] [15].Source: §1 · p. 1
- Data splitting is often insufficiently reported, and recommender systems lack equivalent guidance for dividing datasets into training, validation, and testing sets.Source: §1 · p. 2
- The literature analysis finds substantial variation in data-splitting strategies currently employed.Source: §1 · p. 2
- The paper compares model performance across three splitting strategies to assess their impact.The analysis covers recent recommendation approaches and models from the state of the art.Source: §1 · p. 2
- Different splitting strategies strongly alter state-of-the-art model rankings even with the same dataset and metrics, making splitting a confounding variable.Source: §1 · p. 2
2 DATA SPLITTING STRATEGIES IN RECOMMENDATION MODELS
Recommendation studies use several data-splitting strategies, with little consistency even when they use the same dataset. The section describes their procedures and practical limitations.
Leave One Last
- Leave One Last holds out each user's final transaction for testing, normally uses the second-last transaction for validation, and leaves earlier transactions available for training; transactions may be items or baskets/sessions.Source: §2 · pp. 2–3
- Leave One Last maximizes training transactions, but testing only the final transaction may not reflect recommendation effectiveness over time.A small validation sample may also be insufficient for consistent convergence to an effective, generalizable model.Source: §2 · p. 3
Limitations
- Leave-one-last splits may leak future popularity or interaction signals into training, including learning an item's popularity before it occurs.The section also notes this future-data issue for temporal user-based splitting.Source: §2 · p. 3
Temporal splits
- Temporal user splitting reserves a percentage of each user's latest interactions, whereas temporal global splitting uses one shared cutoff for all users.The user-based boundary can vary across users and allow evidence about future interactions to leak into training; global splitting retains fewer users and items after intersection.Source: §2 · p. 3
Other strategies
- Random splitting selects a per-user train/test boundary randomly and is not reproducible unless authors release the splits they used.User splitting instead assigns users and their transactions to separate training and test sets, requiring support for recommending to cold-start users.Source: §2 · p. 4
Prior-work survey
- Only 2 of 17 reviewed models used temporal global splitting, and studies using the same dataset still differed in their reported strategies.VAECF and TiSASRec both used Movielens-1M but used different splits.Source: §2 · p. 4
3 EVALUATION METHODOLOGY
The study compares three splitting strategies on two grocery transaction datasets to test whether splitting affects recommendation evaluation. It evaluates seven recommenders using ranking metrics and Kendall’s τ.
Data split
- The experiments compare Leave One Last Item, Leave One Last Basket, and global temporal splitting; temporal user splitting is omitted as near-identical to Leave One Last Item.User splitting is omitted because it is rarely used and requires a different evaluation pipeline.Source: §3 · p. 4
Datasets
- The evaluation uses the Tafeng and Dunnhumby grocery transaction datasets, which contain interactions, baskets, and timestamps.Dataset statistics report training, validation, and test interactions after sampling.Source: §3 · p. 5, Table 2 · p. 5
Split construction
- For Leave One Last, the most recent item or basket is held out for testing, the second most recent for validation, and the remainder for training.Items purchased fewer than 10 times are filtered; global temporal splitting uses an 80/20 train-test split and the last 20% of the training subset for validation.Source: §3 · p. 5
Testing models
- Seven recommenders include classical NMF and BPR, neural item models NeuMF, VAECF, and NGCF, and grocery models Triple2vec and VBCAR.The first experiment uses recommended hyperparameters; the second tunes NeuMF, Triple2vec, and VBCAR.Source: §3 · p. 5
Evaluation metrics
- Model performance is measured with NDCG@10 and Recall@10, while Kendall’s τ quantifies ranking correlation between splitting strategies.Source: §3 · p. 5
4 RESULTS
Across the evaluated datasets and metrics, recommendation-model rankings change with the data splitting strategy. A larger comparison also finds only moderate rank correlations between strategies, indicating that they evaluate different aspects of recommendation.
4 RESULTS
- Across all four dataset–metric scenarios, models swap ranks between splitting strategies.The comparison covers seven models, two datasets, and three splitting strategies.Source: §4 · pp. 5–6
- Three places higher: under leave one last basket, Dunnhumby Recall@10 ranks NMF above BPR, VAECF, and NGCF after NMF ranked worst under leave one last item.Source: §4 · p. 6
- Triple2vec appears favored under leave one last item, while VBCAR ranks much higher under temporal evaluation.Swaps cluster around the most effective models in each scenario.Source: §4 · p. 6
- Rank differences reflect both which instances are selected, including future evidence availability during training, and differing train, validation, and test distributions.The authors note that seven systems alone could leave open whether observed swaps occurred by chance, then investigate with a larger sample.Source: §4 · p. 6
- Leave one last item produces a wider score distribution, while temporal and leave one last basket splits group systems more stratifiably.The authors caution that this does not show one strategy is better; the strategies evaluate different aspects of recommendation.Source: §4 · pp. 6–7
- For recommendation NDCG@10, 230 models generated by varying VBCAR and Triple2Vec hyperparameters yielded moderate Kendall’s τ correlations across splitting-strategy pairs, ranging from 0.5284 to 0.7630, with many rank swaps.Source: §4 · p. 6, Figure 2 · p. 7
5 CONCLUSIONS
Experiments across three splitting strategies, seven recommendation models, and two datasets show that splitting strategy can markedly alter state-of-the-art model rankings. The authors recommend reporting split details, including filtering, using temporal global splitting as the default, and releasing reusable data splits.
5 CONCLUSIONS
- Across three splitting strategies, seven models, and two datasets, splitting strategy can markedly alter state-of-the-art model rankings.Source: §5 · p. 7
- Report the splitting strategy, train/validation/test statistics, and any user or item filtering performed.The authors note that these choices can strongly impact performance.Source: §5 · p. 7
- Use temporal global splitting as the default because it is generally seen as the most realistic setting.Source: §5 · p. 7
- Release data splits so other researchers can reuse them.The paper's splits are available from the stated project URL.Source: §5 · p. 7