A Critical Study on Data Leakage in Recommender System Offline Evaluation

• Item popularity and availability change over time, as illustrated by sampled items and iPad activity across years.Source: §1 · pp. 2–3, Figure 1 · p. 3, Figure 2 · p. 3
• The experiments use BPR, NeuMF, SASRec, and LightGCN on four widely used datasets with leave-one-out splitting.Source: §1 · p. 4
• All four baseline models recommend future items to past test instances, revealing an unrealistic offline evaluation setting.Source: §1 · p. 4
• Data leakage affects recommendation accuracy unpredictably: more future training data may improve or deteriorate results.Source: §1 · p. 4
• Baseline model ranking orders vary with the amount of future data, making comparisons under leakage impractical and meaningless.Source: §1 · p. 5
• The proposed timeline scheme requires models to learn only from historical interactions to support more realistic offline evaluation.Source: §1 · p. 5

2 DATA LEAKAGE IN OFFLINE EVALUATION

• Offline evaluation uses static datasets because academic researchers often have limited access to online platforms, while online deployment naturally follows the global timeline.The paper formulates interactions as implicit feedback and focuses on predicting positive entries in the user-item matrix.Source: §2 · p. 5

2.2 Data Leakage in Recommender System

• Only split-by-timepoint observes the global timeline; static training sets can still leak future interactions even when models use timestamps as features.Random splits also ignore local timelines and may train on a user’s future interactions to predict that user’s current interactions.Source: §2 · p. 5, §2.2 · p. 8

2.1 Global Timeline and Local Timeline

• Across four datasets, item mean and median active periods exceed users’; user mean and median periods are under four years, and MovieLens users are typically active under one year.The authors caution that MovieLens timestamps are unreliable because many users have ratings recorded on a single day.Source: Figure 3 · p. 6, §2.1 · p. 6
• New items become available and users’ last interactions occur throughout the ten-year timeline in all four datasets.The figures track item releases and users’ last interactions week by week.Source: §2.1 · p. 6, Figure 4 · p. 7

2.2 Data Leakage in Recommender System

• Collaborative filtering can connect users through shared items and recommend items rated by another user only after the test user’s last interaction.Those later interactions are unavailable at prediction time, so training on them introduces future data into the prediction.Source: §2.2 · p. 8
• Among 85 recommender-system papers published in top-tier venues in 2017–2019, 53.7% used random split, 28% leave-one-out, and 12.3% split-by-timepoint.The authors report that most results from offline evaluations therefore do not well reflect model performance in production.Source: §2.2 · pp. 8–9

3 EXPERIMENT DESIGN AND SETUP

• Yearly training windows add future interactions stepwise for test instances in Y5, with experiments also evaluated on Y7 to reduce bias from a specific test year.For Y5, training begins with Y1–Y4 history and non-test interactions from Y5, then adds later years through Y10; this is a coarse-grained simulation.Source: §3 · pp. 9–10, §3.1 · p. 10

3.1 Dataset

• Experiments use ten years of timestamped implicit interactions from MovieLens-25M, Yelp, Amazon-music, and Amazon-electronic, with dataset-specific k-core filtering.The datasets were selected in part because timestamps allow reconstruction of a global timeline.Source: §3.1 · p. 10

3.2 Models

• The four widely used baselines are BPR, NeuMF, SASRec, and LightGCN, representing matrix-factorization, deep-learning, sequence-aware, and graph-neural-network approaches.Each model’s hyperparameters are tuned on a validation set comprising the test users’ second-last interactions.Source: §3.2 · pp. 10–11

3.3 Evaluation with Non-sampled Metrics

• Accuracy is measured with non-sampled rankings over all items observed in training and HR@20 and NDCG@20; the study examines recommendation lists and metric changes as future data increase.The authors note that these non-sampled metrics yield lower absolute values than sampled metrics because relevant items are ranked against all irrelevant items.Source: §3.3 · pp. 11–12

4.1 Impact on Top-N Recommendation List

• All four evaluated models recommended future items, and their counts increased as more future data entered training across the datasets.The reported counts cover top-20 recommendations for test years Y5 and Y7.Source: §4.1 · p. 12, Table 4 · p. 13
• Filtering future items before scoring is not an option, and recommending unavailable items can underestimate offline accuracy because they cannot produce a hit.The authors treat future-item recommendations as evidence that the evaluation setting is invalid.Source: §4.1 · pp. 12–13
• As more future data entered training, top-20 lists became less similar to no-leakage lists, beyond differences attributed to random seeds.The study compares extrinsic similarity against intrinsic similarity as a reference for training randomness.Source: §4.1 · pp. 14–15, Figure 7 · p. 15

4.2 Impact on Recommendation Accuracy

• More future training data can improve or deteriorate recommendation accuracy, making its impact unpredictable.Similar unpredictable performance patterns were reported for test year Y7 as for Y5.Source: §4.2 · pp. 15, 17
• 89.5%: reported performance-change magnitudes reached this level, with changes in both signs and no particular pattern as future data were added.Table 5 compares changes against results obtained without future training data.Source: §4.2 · p. 17, Table 5 · p. 17
• Across four datasets, baseline HR@20 ranking orders changed as future data increased, hindering reliable model comparisons.The ranking changes also impede reproducibility of recommendation performance.Source: §4.2 · pp. 17–18, Table 6 · p. 18, §4.3 · p. 19

4.3 Summary on Impact of Data Leakage

• The proposed timeline scheme processes interactions chronologically and predicts at test instances using only historical interactions, motivating a revisit of offline evaluation.It allows incremental updates or retraining a batch model when a test instance is encountered.Source: Figure 9 · p. 19, §4.3 · p. 19

5 TIMELINE SCHEME: TOWARD A MORE REALISTIC EVALUATION

• Ignoring the global timeline can prevent models from capturing temporal context and can make reported performance comparisons difficult to reproduce.The analysis reports that this issue applies to leave-one-out and other splits that ignore the timeline.Source: §5 · p. 19
• Split-by-timepoint avoids data leakage but may still be unrealistic when its test period is too short or too long.A short period may not provide enough test instances; a long period can obscure changing interaction context, making its duration difficult to choose.Source: §5 · pp. 19–20
• The timeline scheme sorts interactions chronologically and predicts each test instance using the model’s history and items available at that time.Test instances are sampled along the timeline, allowing models to learn temporal context while preventing leakage from later interactions.Source: §5 · p. 20
• Sliding-window evaluation can use past interactions to predict a following period, but window length, hyperparameter tuning, and accuracy interpretation over time require careful choices.Long windows can intensify cold-start issues; short windows can be computationally expensive, and validation windows must avoid leakage.Source: §5 · pp. 20–21
• Recommendation accuracy can vary across time windows, so the best-performing model at one point may become less competitive later.Possible reporting strategies include averaging over multiple windows or using the latest window.Source: §5 · p. 21
• The timeline scheme requires incremental models or batch retraining on historical interactions, whose amount is difficult to choose and whose growing volume can make retraining unscalable.Using only a subset may forget long-term preferences, while preferences can also change over time.Source: §5 · p. 21

6 RELATED WORK

• Prior studies report that recommendation results vary across data splits, while this work attributes such non-comparability in part to leakage and uncaptured dynamic context.The discussion also challenges comparisons between random and time-aware splits because random splitting leaks future information.Source: §6 · pp. 21–22
• Unlike work comparing splitting strategies, this study examines whether and how leakage arises and how it affects recommendation comparisons.It also argues that split-by-timepoint may not be a good choice based on user and item activity periods and interaction dynamics.Source: §6 · p. 22
• Temporal leave-one-out evaluates interactions along each user’s local timeline, whereas this paper’s scheme samples test interactions across the dataset’s global timeline.Related work also reports that model performance can vary over time.Source: §6 · p. 22
• Earlier research raised future-information leakage, but the authors describe systematic study of its impact as lacking.They characterize their contribution as a comprehensive critical study from the global-timeline perspective.Source: §6 · p. 22
• The paper distinguishes its global-timeline focus from local-timeline work and applies that focus to recommender problems including time-aware and sequential recommendation.The authors contrast their concern with ignoring global time in offline evaluation against studies focused on user- or item-specific timelines.Source: §6 · p. 23
• Because many published models are not incremental, this study uses experiments to examine global-timeline problems in batch-learning models.Related stream-based work commonly uses prequential evaluation or variants that test new interactions before training on them.Source: §6 · p. 23

7 CONCLUSION

• Data leakage can make models recommend future items, alter recommendation lists, and unpredictably affect accuracy and relative model rankings.The authors conclude that published results may not reflect online performance and do not reliably identify which models will perform better there.Source: §7 · p. 23
• The proposed timeline scheme evaluates test instances across time using only interactions available before each instance, to capture changing temporal context.The model must learn incrementally, or batch-learning models must be retrained using historical interactions.Source: §7 · p. 23
• For batch-learning models, retraining may be infeasible because choosing how much historical interaction data to use is difficult.The authors note that many proposed models are not capable of incremental learning.Source: §7 · pp. 23–24

A.1 Impact of Data Leakage on Recommendation Performance of Test Year Y7

• Figure 10 reports HR@20 and NDCG@20 for BPR, NeuMF, SASRec, and LightGCN on test year Y7.The caption identifies the metrics, models, and test-year scope but does not state comparative outcomes.Source: Figure 10 · p. 27
• The figure’s model set comprises BPR, NeuMF, SASRec, and LightGCN.Source: Figure 10 · p. 27
• The figure covers two ranking metrics, HR@20 and NDCG@20.Source: Figure 10 · p. 27