Source-linked AI summary
How Well Does Generative Recommendation Generalize?
Yijie Ding, Zitian Guo, Jiacheng Li, Letian Peng, Shuai Shao, Wei Shao, Xiaoqiang Luo, Luke Simon, Jingbo Shang, Julian McAuley, Yupeng Hou
TL;DR
The paper asks whether generative recommendation models genuinely generalize better than item ID-based models and develops an instance-level analysis to test that question. It categorizes predictions by memorization or generalization, analyzes the apparent advantage at token level, and adaptively ensembles both paradigms. GR models excel on generalization-related instances, item ID-based models perform better on memorization-related instances, and adaptive weighting improves overall performance.
Problem
The paper addresses the limited systematic evidence for whether generative recommendation models generalize better than conventional item ID-based models.
Method
The paper categorizes instances by item-transition memorization or generalization, analyzes GR behavior at the token level, and adaptively ensembles GR with item ID-based models.
Results
GR models excel on generalization-related instances, item ID-based models perform better on memorization-related instances, and adaptive ensembling consistently improves overall performance over individual and fixed-weight models.
Takeaways & Limitations
The two recommendation paradigms are complementary, while apparent item-level generalization in GR models often reduces to token-level memorization.
Takeaways & Limitations
The experiments aggregate token memorizations with hop ≤4 and use a 256 × 3 semantic ID quantization followed by one identifier token.
Abstract
from arXiv · showhide
A widely held hypothesis for why generative recommendation (GR) models outperform conventional item ID-based models is that they generalize better. However, there is few systematic way to verify this hypothesis beyond a superficial comparison of overall performance. To address this gap, we categorize each data instance based on the specific capability required for a correct prediction: either memorization (reusing item transition patterns observed during training) or generalization (composing known patterns to predict unseen item transitions). Extensive experiments show that GR models perform better on instances that require generalization, whereas item ID-based models perform better when memorization is more important. To explain this divergence, we shift the analysis from the item level to the token level and show that what appears to be item-level generalization often reduces to token-level memorization for GR models. Finally, we show that the two paradigms are complementary. We propose a simple memorization-aware indicator that adaptively combines them on a per-instance basis, leading to improved overall recommendation performance.
1 Introduction
The paper addresses the lack of systematic evidence about whether generative recommendation generalizes better by separating memorization- and generalization-dependent instances. Experiments compare these capabilities across models and motivate an adaptive combination of complementary paradigms.
- Motivation: GR models tokenize each item into shared sequences of sub-item tokens, unlike conventional models that use one unique item ID.Semantic IDs are one example of this tokenization approach.
- Analytical framework: Each instance is categorized by whether correct prediction requires reusing observed transition patterns or composing known patterns to infer unseen transitions.The framework focuses on the interaction between history and target rather than target-item frequency alone.
- Empirical findings: Across seven real-world datasets, GR models excel on generalization-related subsets but generally underperform item ID-based models on memorization-related subsets.The comparison benchmarks TIGER as a semantic ID-based GR model against SASRec as an item ID-based conventional model.
- Mechanism: Token-level analysis finds that many transitions appearing to require item-level generalization for GR models reduce to memorization of sub-item token transition patterns.This analysis is introduced to investigate the mechanism behind GR models’ generalization capability.
- Complementarity: An adaptive ensemble combines GR and item ID-based models using instance-specific weights predicted from whether memorization or generalization is primarily required.The method consistently improves overall performance over both individual models and naive fixed-weight ensembles.
2 Defining Memorization and Generalization
The framework defines memorization and generalization using item transitions within each user history, then extends generalization to multi-hop compositions and labels remaining cases as uncategorized.
- Task definition: An item transition is a directed pair [i_s → i_t] drawn from a user history, with hop count equal to the distance between the two items.Each instance is categorized using the set of transitions from historical items to the target.
- Memorization: An instance is memorization-related when its 1-hop transition [i_t−1 → i_t] appeared in training, regardless of the user history containing it.Under this condition, correct prediction is possible by memorizing training data.
- Generalization: An instance is generalization-related when it is not memorization-related and contains an item transition inferable or composable from observed training transitions.A single instance may satisfy multiple generalization types.
- Generalization types: The framework includes transitivity, symmetry, and second-order symmetry, which infer target transitions through intermediate or reversed observed relationships.Transitivity bridges two transitions via an intermediate item, while symmetry uses the reverse transition.
- Multi-hop generalization: The same criteria extend to multi-hop transitions, using the minimum hop count when multiple applicable transitions have different distances.Multi-hop patterns can bypass unnecessary intermediate items during inference.
- Substitutability and uncategorized cases: Substitutability is defined as a generalization type involving only multi-hop transitions, because bypassing intermediate items requires generalization.Instances satisfying none of the specified criteria within the maximum hop count are labeled uncategorized.
3 Performance Breakdown: Item IDs vs. Semantic IDs
Experiments on seven public datasets compare TIGER and SASRec across memorization and generalization categories. TIGER is stronger on generalization, while SASRec is stronger on memorization, with difficulty increasing for more complex or longer-hop generalization.
- Experiment setup: The study evaluates TIGER and SASRec on seven public datasets using memorization, generalization, and uncategorized test subsets.The datasets include Amazon Reviews, Amazon 2023, Steam, and Yelp; testing uses a leave-last-out split.
- Memorization versus generalization: +58.8% on Office, +56.7% on Beauty, and +39.8% on Sports: TIGER consistently outperforms SASRec on generalization subsets.Both models perform substantially better on memorization than on generalization overall.
- Generalization categories: Both models perform better on Substitutability and Symmetry than on Transitivity and 2nd-Symmetry.The latter categories require composing knowledge from multiple examples, whereas the former require induction from a single training example.
- Generalization hops: Performance worsens monotonically as hop distance increases, with SASRec declining faster and TIGER remaining more robust for longer-hop generalization.SASRec can outperform TIGER in low-hop settings, but its performance drops faster as hop distance grows.
- Data ratios: Memorization cases form a much smaller portion than generalization cases in every dataset, and most generalization instances combine information from multiple training examples.Only a small fraction can be inferred from a single training instance.
4 Mechanism Analysis: A Token-Level Lens
The paper explains GR’s generalization–memorization trade-off through token-level prefix transitions: item-level generalization often relies on memorized token patterns, while shared prefixes can dilute item-specific memorization.
- 4.1 Prefix N-Gram Memorization: GR models often interpret item-level generalization as token-level prefix memorization within the semantic ID space.Semantic ID prefixes expose shared transition patterns across items, allowing unseen item transitions to be inferred from observed prefix transitions.
- 4.1 Prefix N-Gram Memorization: A test transition is n-gram prefix-memorizable when both items’ first n semantic-ID tokens occur in training, even if the exact items differ.The analysis aggregates prefix memorization across multiple transition hops, using semantic-ID prefixes as the token-level pattern.
- 4.2 From Item Generalization to Token Memorization: More than 5% of symmetry, transitivity, and 2nd-symmetry generalization transitions are explainable as 3-hop prefix memorization, while over 99% admit at least 1-gram prefix memorization.This shows that unseen item-level transitions frequently retain matching prefix transitions in the training data.
- 4.3.1 Token Memorization Enables Better Generalization: Token memorization support correlates with better generalization, and TIGER’s advantage over SASRec grows with support count and prefix length.Supported instances achieve substantially higher NDCG@10 for both models, with a larger TIGER gain on prefix-supported instances.
- 4.3.2 Token Memorization Dilutes Item Memorization: Token memorization can dilute item memorization when item transition probability is high but prefix transition probability is low.TIGER spreads probability mass across items sharing a prefix transition, producing significant NDCG loss and often causing underperformance on memorization tasks.
- 4.4 Mechanism Validation: Smaller codebooks improve generalization by +10.24% relative but reduce memorization by −7.62% relative, while denser tokenization regularizes training toward shared prefix structure.Across tested codebook sizes, higher token memorization ratios consistently improve generalization while degrading memorization.
5 Memorization-Aware Adaptive Ensemble
The paper uses a confidence-based indicator to estimate whether each instance favors memorization, then adaptively weights SASRec and TIGER. The adaptive ensemble generally outperforms individual models and fixed-weight combinations, especially where their performance crossover is stronger.
- Adaptive ensemble strategy: The framework dynamically weights SASRec and TIGER according to each input’s estimated need for memorization or generalization.The approach uses a training-free indicator followed by empirical validation against ground-truth categories.
- Adaptive ensemble indicators: The maximum softmax probability of the item ID-based model serves as a proxy for memorization likelihood.The rationale is that memorization-related instances are closer to the training distribution and yield more confident ID-based predictions.
- Indicator validation: As the indicator value increases, the proportion of memorization-related instances rises monotonically.Instances with higher values therefore increasingly favor the model specialized for memorized patterns.
- Indicator validation: At lower indicator values TIGER performs better, whereas SASRec becomes increasingly competitive and may outperform TIGER at higher values.This crossover matches the interpretation that lower values correspond more closely to generalization and higher values to memorization.
- Recommendation results: The adaptive ensemble generally outperforms both individual base models and the fixed-weight ensemble baseline.Its gains are more prominent on datasets with a stronger model crossover effect, where the indicator can exploit each model’s specialization.
6 Related Works
The related-work discussion situates generative recommendation against conventional sequential recommendation and reviews prior approaches to studying memorization and generalization. It emphasizes the challenge of characterizing how training data influences model behavior.
- Generative recommendation: Generative recommendation tokenizes each item as a sequence of sub-item tokens, unlike conventional models that typically use unique item IDs.This distinction motivates treating generative recommendation as a separate paradigm in the paper’s comparison.
- Memorization and generalization: Prior memorization and generalization research spans classical machine learning models and modern large language models.A central challenge is characterizing the influence of training data on model behavior.
7 Conclusion
The paper systematically compares generative and item ID-based recommendation through memorization and generalization. It finds that their strengths differ, that generative models’ item-level generalization often reduces to token-level memorization, and that adaptive combination improves performance.
- Conclusion: The proposed framework categorizes instances by the item transition patterns required for correct prediction.This enables separate evaluation of memorization-related and generalization-related instances.
- Conclusion: Generative recommendation models excel on generalization-related instances, while item ID-based models perform better on memorization-related instances.The conclusion reports this pattern across the paper’s experiments.
- Conclusion: Item-level generalization in generative models often reduces to token-level memorization.The paper uses token-level analysis to explain the divergence between the two paradigms.
- Conclusion: Adaptively adjusting ensemble weights with a memorization-aware indicator improves recommendation performance.The result supports treating the two paradigms as complementary rather than mutually exclusive.