Source-linked AI summary
Self-Evolving Memory for Generative Recommendation
Xinyu Lin, Zhuosong Jiang, Zixiao Suo, Siqin Wang, Hanqing Zeng, Hanchao Yu, Yinglong Xia, Jiang Zhang, Aashu Singh, Fei Liu, Wenjie Wang, Fuli Feng, Yang Song, Qifan Wang, Tat-Seng Chua
TL;DR
Shared autoregressive adaptation can let dominant patterns overwhelm underrepresented preferences as user behavior evolves. LION uses sparse Key-Value memory and consolidation supervision to structure that evolution, and experiments on real-world datasets validate its effectiveness. The authors identify fixed Top-K routing and the T5 backbone as boundaries for future exploration.
Problem
In generative recommendation, heterogeneous preference shifts are adapted within a shared autoregressive parameter space, where dominant patterns can overwhelm underrepresented ones.
Source: §1 · p. 2Method
LION uses sparsely activated Key-Value memory pathways to isolate behavioral patterns and a consolidation loss to reinforce underrepresented patterns.
Source: §1 · p. 3Results
Experiments on three real-world datasets validate LION’s effectiveness in improving performance across heterogeneous user populations during continual evolution.
Source: §1 · p. 3, §7 · p. 15Takeaways & Limitations
LION provides a memory-based approach for structured continual evolution across heterogeneous behavioral patterns.
Source: §7 · p. 15The current routing uses fixed Top-K activation, and extending LION beyond T5 to LLM-based architectures remains future work.
Source: §7 · p. 15
Abstract
from arXiv · showhide
Generative recommendation has emerged as a promising end-to-end paradigm for personalized recommendation. However, user preferences continuously evolve over time, making self-evolving an essential capability for generative recommender systems. Existing evolving strategies, such as continual retraining and distillation-based adaptation, directly update the shared model parameters using streaming interactions. Nevertheless, we find that directly applying such strategies to generative recommendation introduces a critical issue, termed evolution conflict. Specifically, heterogeneous preference shifts from different users are optimized within a fully shared autoregressive parameter space, causing dominant behavioral patterns to progressively dominate the model evolution process while underrepresented patterns become increasingly overlooked. To address this issue, we propose a self-evolving memory paradigm for generative recommendation, aiming to enable effective evolution across heterogeneous behavioral patterns. We further identify three key principles for effective self-evolving recommendation systems, including isolated memorization, reinforced evolution, and scalable application. Guided by these principles, we develop LION, a simple yet effective framework centered on a sparse Key-Value memory layer. Specifically, LION introduces sparse memory activation to isolate the evolution of different behavioral patterns, while a consolidation loss is designed to reinforce the learning of underrepresented preference dynamics during continual adaptation. Extensive experiments on diverse real-world datasets demonstrate the effectiveness of LION under various continual evolution settings (e.g., per-period evaluation, user/item group evaluation, and evolution convergence analysis). The codes are released at https://github.com/JazyJiang/Self-Evolving-Memory-for-Generative-Recommendation.
1 Introduction
Generative recommenders must adapt to changing preferences, but shared-parameter evolution can let dominant behavior patterns overwhelm underrepresented ones. The paper proposes LION, a sparse-memory framework guided by three principles, and reports effectiveness across real-world continual-evolution settings.
1 Introduction
- Generative recommendation models generate item identifiers autoregressively, while user preferences evolve over time and require continual adaptation.Prior evolving approaches include continual retraining and regularization-based adaptation, including replay and distillation techniques.Source: §1 · pp. 1–2
- Continual adaptation in a shared autoregressive parameter space creates evolution conflict: dominant patterns can overwhelm underrepresented preference dynamics.The paper describes continual retraining and distillation-based adaptation as strategies that update shared model parameters using streaming interactions.Source: §1 · pp. 2–3
- LION combines sparse Key-Value memory, pattern-specific activation, and consolidation supervision to support isolated memorization, reinforce underrepresented patterns, and scale continual adaptation.The paper reports effectiveness on three real-world datasets across continual-evolution settings.Source: §1 · p. 3
- The proposed memory paradigm is guided by isolated memorization, reinforced evolution, and scalable application for heterogeneous behavioral patterns.These principles call for differentiated evolution pathways, stronger supervision for underrepresented patterns, and efficient scaling as users and interactions grow.Source: §1 · pp. 2–3
2 Task Formulation
The task formulation treats recommendation as autoregressive next-item identifier generation from a user’s interaction history. Self-evolving recommendation adapts the model at successive periods using streaming data, with existing strategies differing in which interactions they retrain on.
2 Task Formulation
- Given a user’s interaction history, a generative recommender predicts the next item’s identifier token by token, conditioned on prior generated tokens.The identifier is a sequence of tokens, and the model’s shared parameters capture user preference transitions.Source: §2 · pp. 3–4
- Self-evolving recommendation represents streaming interactions as successive periods and updates the recommender after each period for deployment in the next.The parameters inherited from the previous period are adapted using retraining data for the current period.Source: §2 · p. 4
- Existing adaptation uses either latest-period-only continual fine-tuning or distillation-based fine-tuning on all interactions observed through the current period.The formulation denotes these retraining sets as the current period alone or the cumulative history, respectively.Source: §2 · p. 4
3 Identification of Evolution Conflict
The paper attributes evolution conflict to competing gradients in a shared parameter space, compounded by more frequent common patterns and stronger gradient magnitudes. It argues for differentiated pathways, stronger supervision of underrepresented patterns, and scalable memory.
3 Identification of Evolution Conflict
- Common- and uncommon-pattern objectives can have conflicting optimization directions within the shared parameter space.The corresponding gradients are defined for the two objectives, and their inner product may be negative.Source: §3 · p. 4
- Common patterns occur more frequently in streaming interactions, producing asymmetric gradient magnitudes that favor common-pattern updates.The paper states that the common-pattern gradient magnitude is much greater than the uncommon-pattern gradient magnitude.Source: §3 · p. 4
- This combination leaves common-pattern users with partially canceled updates while uncommon-pattern users face progressively worsening performance as weaker gradients are overwhelmed.The update becomes progressively dominated by the common-pattern gradient.Source: §3 · p. 4
- The analysis motivates differentiated optimization pathways, stronger supervision for underrepresented patterns, and scalable evolution without prohibitive overhead.These are the three stated design principles for self-evolving memory.Source: §3 · p. 4
4 LION
LION uses a sparse Key-Value memory layer to support isolated memorization and reinforced evolution in generative recommendation. Its design combines memory routing, two complementary training losses, and continual deployment across streaming periods.
4 LION
- LION routes different behavioral patterns to sparse memory parameters, avoiding the substantial parameter overhead of dense user-specific personalization.The framework is centered on a sparse Key-Value memory layer for continual evolution.Source: §4 · p. 4, §4.1.1 · p. 5
4.1.1 Sparse Memory Evolution
- The memory layer selects Top-K slots by similarity to the user representation and aggregates their values into a representation integrated with Transformer hidden states.Each of the N trainable Key-Value pairs contains a key and corresponding value.Source: §4.1.1 · p. 5
- A full-history query activates memory, while Transformer layers before and after the memory layer process only the recent interaction subsequence.This asymmetric design grounds memory activation in full behavioral context while keeping backbone computation efficient.Source: §4.1.1 · p. 6
4.1.2 Reinforced Evolution
- LION combines recommendation loss on autoregressive outputs with consolidation loss on the memory-enhanced representation to reinforce preference modeling during evolution.The consolidation supervision is intended to give underrepresented patterns sufficient signals throughout continual evolution.Source: §4.1.2 · p. 6
4.1.3 Instantiation
- At each evolving period, LION fine-tunes the backbone and memory on new streaming interactions, then deploys the updated model for next-item generation in the following period.The memory layer is inserted into intermediate Transformer layers to activate differentiated slots for heterogeneous behavioral patterns.Source: §4.1.3 · p. 6
4.2 Theoretical Analysis
LION’s theoretical analysis addresses gradient conflict, per-pattern convergence, and parameter efficiency. It relates sparse activation to conflict limited to overlapping memory slots, faster convergence, and scalability with growing user populations.
4.2.1 Gradient Conflict Reduction
- Under sparse activation, gradient conflict is confined to the overlap between common- and uncommon-pattern activated memory slots.The analysis bounds conflict to the overlapping directions rather than all activated slots; routing specialization is an observed trend, not a guarantee, and may reduce overlap.Source: §4.2.1 · pp. 6–7
4.2.2 Convergence Analysis
- Under the stated smoothness and learning-rate conditions, LION reaches ε-stationarity in fewer steps than the baseline, with larger convergence speedup for uncommon patterns.The paper attributes the larger speedup to uncommon patterns experiencing lower gradient pressure from common patterns and smaller overlap.Source: §4.2.2 · p. 7
4.2.3 Parameter Efficiency and Scalability
- LION’s sparse memory uses 2Nd parameters regardless of user population, versus |U| · d for dense per-user personalization, and is strictly cheaper when |U| > 2N.Adding new users incurs zero additional parameters, supporting scalable evolution without sacrificing representational expressiveness.Source: §4.2.3 · p. 8
5 Experiments
Across three Amazon Review datasets, LION is evaluated under chronological continual adaptation and generally outperforms competing methods. The experiments examine subgroup performance, component effects, gradient alignment, convergence, and memory compactness.
5.1.1 Datasets; 5.1.2 Evaluation Metrics
- The evaluation uses three Amazon Review datasets, chronologically split into five equal-interaction periods, with users grouped by activity level.Each next-item sample uses up to 10 previous interactions; evaluation uses Recall@K and NDCG@K for K ∈ {10, 20}.Source: §5.1.1 · p. 8, §5.1.2 · p. 8
5.2 Overall Performance (RQ1)
- LION consistently outperforms TIGER and surpasses all baselines in most cases across the three datasets.Table 1 reports sample-weighted averages over four evaluation periods and five user groups; LSAT and SAIL-PIW are the strongest continual-learning baselines.Source: Table 1 · p. 9, §5.2 · p. 9
- Across datasets and periods, LION gains more on inactive than active users and maintains improvements in both groups throughout adaptation.The group comparison uses active G1–G2 and inactive G3–G5 users.Source: §5.2 · p. 10, Table 2 · p. 10
5.3.1 Ablation Study (RQ2)
- Adding consolidation without the sparse memory layer can hurt performance, while the sparse memory layer alone yields only marginal overall improvement.Full-sequence routing and consolidation on top of the memory layer both contribute positively, with consolidation the dominant driver.Source: §5.3.1 · pp. 10–11, Table 3 · p. 11
5.3.2 Effect of λ (RQ2)
- Increasing λ from zero brings a clear performance gain, and performance remains strong for λ values from 0.3 to 1.0 on both datasets.The per-dataset optimum is λ=1.0 for Games and λ=0.5 for CDs.Source: §5.3.2 · pp. 11–12, Table 4 · p. 12
5.3.3 Gradient Conflict Reduction (RQ3); 5.3.4 Item-Side Gradient Alignment (RQ3)
- LION has uniformly positive active–inactive gradient cosine on the sparse memory layer, while sparse memory also mitigates gradient conflict between popular and long-tailed items.The item comparison defines popular items as the top 60% and long-tailed items as the tail 40%; conflict is relatively weaker on Games.Source: §5.3.3 · p. 12, Figure 5 · p. 12, §5.3.4 · p. 12
5.3.5 Empirical Convergence Analysis (RQ4)
- On Games and CDs, LION converges faster and reaches lower evaluation loss than TIGER at every period.Loss rises at the start of each new period as incoming interactions reflect shifted user behaviors, and LION recovers faster after each jump.Source: §5.3.5 · pp. 12–13, Figure 7 · p. 13
5.3.6 Memory Compactness (RQ4)
- Across a 16× memory-vocabulary range and a 4× range in K, Recall@10 stays within ±0.4% across all datasets.The memory footprint is fixed and does not grow with user population.Source: §5.3.6 · p. 13
6 Related Works
Prior recommendation work addresses continual learning, generative recommendation, and memory through distinct approaches. This work frames memory instead as a means to evolve pattern-specific representations, and identifies isolated parameter evolution as its core contribution.
Continual Learning in Recommendation
- Traditional continual recommendation uses retraining or regularization, while generative continual methods often separate new from old knowledge or short- from long-term interests.Fine-tuning can overwrite prior knowledge, whereas full retraining retains history at substantial computational cost.Source: §6 · p. 13
- The paper identifies isolated parameter evolution as a paradigm for generative recommendation, with sparse KV memory as one scalable instantiation.Combining it with complementary isolation mechanisms remains a direction for future work.Source: §6 · p. 14
Generative Recommendation
- Generative recommenders autoregressively produce item identifiers, but heterogeneous user gradients can conflict in their shared parameters and frequent patterns can dominate evolution.The paper presents this coupling as motivation for self-evolving memory in generative recommendation.Source: §6 · p. 14
Memory for Recommendation
- Prior recommendation memories store interaction histories, model dynamic preferences, or persist KV states for computation reuse.This work instead uses memory to structure distinct behavioral patterns and evolve their representations as new data arrives.Source: §6 · p. 14
7 Conclusion and Future Work
The conclusion frames evolution conflict as competition between behavioral patterns in shared autoregressive parameters and presents LION’s sparse memory as a way to support their structured evolution. Experiments on three real-world datasets report improvements for common and uncommon patterns, while future work points to broader backbones and more adaptive routing.
7 Conclusion and Future Work
- LION uses sparsely activated Key-Value memory pathways and a consolidation loss to support continual evolution across heterogeneous user populations.The loss reinforces underrepresented patterns; experiments on three real-world datasets report improved performance for both common and uncommon patterns across continual-evolution settings.Source: §7 · p. 15
- Evolution conflict arises when heterogeneous behavioral patterns share autoregressive parameters, allowing dominant patterns to overwhelm underrepresented ones.Source: §7 · p. 15
- Future directions include extending beyond T5, replacing fixed Top-K routing with adaptive activation budgets, and combining memory with lightweight user-specific adapters.Source: §7 · p. 15
A.1 Hyper-parameter Settings of Baselines
The appendix specifies tuned configurations for baseline methods and LION, then states shared LION defaults selected through cross-dataset analysis. It also identifies a dataset-statistics table.
A.1 Hyper-parameter Settings of Baselines
- Baseline tuning varies by method, including RecICL example counts and epochs, LSAT fusion weight, SAIL-PIW stability coefficient, PISA plasticity and loss weights, and PESO KL strength.For LoRA-based baselines, the T5 backbone and LoRA parameters are jointly trained.Source: §A.1 · p. 15
- LION searches memory position, recent-history length, consolidation weight, and memory-key/Top-K configurations; shared defaults are layer 5, history length 2, nk = 64, and Top-K = 32.The consolidation weight λ is selected per dataset by validation.Source: §A.1 · p. 15
- Table 6 presents statistics for the datasets used in the study.Source: Table 6 · p. 15
A.2 Hyperparameter Analysis
The hyperparameter analysis reports Recall@10 results for recent-history length and memory-layer position. It favors a short history window and finds layer 5 tied-or-best across datasets.
A.2 Hyperparameter Analysis
- The reported routing rationale favors very recent interactions; ℓ=0 performs poorly because input token embeddings lack the contextual structure needed for key routing.Source: §A.2 · p. 15
- Recall@10: hrec=2 performs best or tied-best across all datasets, while longer windows mildly reduce performance on Toys and CDs.Table 7 reports the four-period sample-weighted average.Source: §A.2 · p. 15, Table 7 · p. 16
- Recall@10: layer 5 is tied-or-best on every dataset, while positions from ℓ=1 onward are viable and deeper placement has a small marginal effect.Table 8 reports the four-period sample-weighted average.Source: §A.2 · p. 15, Table 8 · p. 16