Source-linked AI summary
Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1M
Dario Di Palma, Felice Antonio Merra, Maurizio Sfilio, Vito Walter Anelli, Fedelucio Narducci, Tommaso Di Noia
TL;DR
The paper examines whether LLMs memorize recommendation datasets, an underexplored question relevant to understanding evaluation and bias. It probes MovieLens-1M through prompted retrieval across GPT and Llama models. The authors report that models retrieve dataset content, and that memorization is related to recommendation performance and popularity bias.
Problem
Limited evidence addresses whether LLMs have memorized public recommendation datasets, a gap relevant because memorization may affect evaluation generalizability and popularity bias.
Source: §1 · p. 1Method
The study defines memorization as prompted recovery of item attributes, user profiles, or interactions, then probes MovieLens-1M across GPT and Llama models.
Source: §1 · pp. 1–2, §2.1 · p. 2, §2.2 · p. 2Results
All examined models retrieve substantial MovieLens-1M content, and memorization is tied to recommendation performance and reflects the dataset’s popularity bias.
Source: §4 · p. 4, §3.2 · p. 4, §3.4 · p. 4Takeaways & Limitations
The findings raise concerns that evaluating LLM recommenders on memorized datasets may not establish generalization to unseen data.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Large Language Models (LLMs) have become increasingly central to recommendation scenarios due to their remarkable natural language understanding and generation capabilities. Although significant research has explored the use of LLMs for various recommendation tasks, little effort has been dedicated to verifying whether they have memorized public recommendation dataset as part of their training data. This is undesirable because memorization reduces the generalizability of research findings, as benchmarking on memorized datasets does not guarantee generalization to unseen datasets. Furthermore, memorization can amplify biases, for example, some popular items may be recommended more frequently than others. In this work, we investigate whether LLMs have memorized public recommendation datasets. Specifically, we examine two model families (GPT and Llama) across multiple sizes, focusing on one of the most widely used dataset in recommender systems: MovieLens-1M. First, we define dataset memorization as the extent to which item attributes, user profiles, and user-item interactions can be retrieved by prompting the LLMs. Second, we analyze the impact of memorization on recommendation performance. Lastly, we examine whether memorization varies across model families and model sizes. Our results reveal that all models exhibit some degree of memorization of MovieLens-1M, and that recommendation performance is related to the extent of memorization. We have made all the code publicly available at: https://github.com/sisinflab/LLM-MemoryInspector
1 Introduction
The paper asks whether LLMs memorize recommendation datasets, a question that has received limited attention despite its relevance to understanding model effectiveness. It investigates MovieLens-1M and frames memorization as a potential source of evaluation, bias, and comparison concerns.
1 Introduction
- Limited attention has been paid to whether LLM effectiveness reflects memorization of the datasets used during training.The introduction identifies this as an open question about why LLM-based recommendation methods perform as they do.Source: §1 · p. 1
- Prior work in other fields quantified memorization in language-model training data, including the Pile and coding samples.The cited examples report at least 1% of the Pile memorized by GPT-J-6B and extraction of 56% of samples used to train GPT-Neo.Source: §1 · p. 1
- Memorized test data may make performance results unreliable, amplify popularity bias, and complicate comparisons with standard recommenders.The paper notes that standard recommenders do not inherently possess cross-domain knowledge.Source: §1 · p. 1
- The study provides a preliminary examination of MovieLens-1M, defining recommendation-dataset memorization and probing Llama and GPT models.It also investigates recommendation performance, dataset characteristics including bias, and whether memorization varies with model size.Source: §1 · pp. 1–2
2 Methodology
The methodology defines recommendation-dataset memorization through recoverable item attributes, user profiles, and user-item interactions. It evaluates prompting-based extraction and coverage across MovieLens-1M and multiple GPT and Llama models.
2.1 Definitions of Memorization
- Memorization is operationalized as recovering item metadata or user attributes from an identifier, or predicting a user’s interaction from prior interactions.The dataset is represented through item metadata, user metadata, and user-item interaction records.Source: §2.1 · p. 2
2.2 Evaluation Protocol
- MovieLens-1M was selected for analysis after examining papers accepted at ACM RecSys 2024 and ACM SIGIR 2024; it was most frequently used, appearing in 17.2% and 22% of them.Source: §2.2 · p. 2
- Few-shot prompting was selected over tested zero-shot and Chain-of-Thought approaches, and the study defines three coverage-based memorization metrics.The authors leave exploration of methods such as automatic prompt engineering to future work.Source: §2.2 · p. 2
- The evaluation covers six Llama models and three GPT models, spanning multiple model sizes to examine size-related differences in memorization.The tested models include Llama variants from 1B to 405B and GPT-4o, GPT-4o mini, and GPT-3.5 turbo.Source: §2.2 · p. 3
- Generation settings were fixed for deterministic behavior, with temperature 0, top_p 1, both penalties 0, and random seed 42.Source: §2.2 · p. 3
3 Results and Discussion
Across the four analyses, the models retrieved MovieLens-1M items, user attributes, and interactions; recommendation performance tracked memorization, and both memorization and item retrieval varied with model scale and popularity.
3.1 Analysis of Memorization
- GPT-4o recovered 80.76% of items, while GPT-3.5 turbo recovered 60.47%; Llama-3.2 1B had the lowest item coverage at 1.93%.The analysis queried exact MovieID::Title matches.Source: §3.1 · p. 3
- GPT-3.5 turbo recovered attributes for 17.38% of users; GPT-4o recovered 9.37% of interactions, while Llama-3.1 405B recovered 8.32%.None of the examined models were free of MovieLens-1M knowledge.Source: §3.1 · p. 3
3.2 Impact on Recommendation
- GPT-4o achieved HR@1 of 0.2796 versus 0.0316 for GPT-4o mini; higher-memorization models within groups also showed superior recommendation performance.The comparison used MovieLens-1M without filtering, with an 80%/20% train-test split and leave-n-out evaluation.Source: §3.2 · pp. 3–4, Table 2 · p. 4
3.3 Impact of Model Scale
- Llama-3.1 405B had mean memorization of 12.9%, versus 5.82% for Llama-3.1 8B; scaling up was associated with greater memorization and better recommendation performance.From 405B to 8B, average nDCG decreased 54.23% and HR decreased 47.36%.Source: §3.3 · p. 4
3.4 Popularity Memorization
- Figure 4 shows greater item coverage for highly popular than rarely interacted items; GPT-4o retrieved 89.06% and 63.97%, respectively.The authors attribute disproportionate memorization of popular movies to their overrepresentation in training data.Source: Figure 4 · p. 4, §3.4 · p. 4
4 Conclusion and open directions
GPT and Llama models can retrieve substantial portions of MovieLens-1M’s item catalog, user metadata, and interactions. They also reflect the dataset’s popularity bias, and recommendation performance is closely tied to memorization; future work will develop extraction and mitigation approaches.
- Substantial portions of MovieLens-1M’s item catalog, user metadata, and user interactions can be accurately retrieved from GPT and Llama models.Source: §4 · p. 4
- LLMs reflect MovieLens-1M’s popularity bias, and their recommendation performance is closely tied to memorization.The authors raise potential concerns about the validity of current evaluation practices for LLM-based recommenders.Source: §4 · p. 4
- Future work will develop ML-optimized memorization-extraction techniques and mitigation approaches to improve the reliability of LLM evaluation and use in recommendation.Source: §4 · p. 4