Source-linked AI summary
When Memory Takes Gradients: Collaborative Vector Memory for Agentic Recommender Systems
Hanchong Chen, Xing Tang, Lingjie Li, Xiongfeng Shan, Xiuqiang He
TL;DR
Text memories are costly to maintain and cannot preserve catalog-wide collaborative structure or directly absorb ranking gradients. CoVeMem stores graph-trained states, retrieves candidate-relevant history as soft tokens, and trains the LLM to read and rank through them, matching or exceeding text-memory agents while avoiding additional maintenance calls.
Problem
Text memory requires serial LLM rewrites and loses catalog-wide collaborative structure, while ranking gradients cannot directly update stored text.
Method
CoVeMem uses LightGCN user and item states as a memory bank, retrieves candidate-relevant history, maps states into soft tokens, and trains reading through alignment and masked listwise co-training.
Results
CoVeMem exceeds the strongest text-memory baseline in nearly all evaluated settings across four benchmarks, with especially clear gains in interaction-dense domains.
Takeaways & Limitations
Across four benchmarks, CoVeMem matches or exceeds the strongest text-memory agent while requiring zero additional memory-maintenance calls beyond the shared static profile.
Takeaways & Limitations
Candidate-conditioned retrieval uses the user’s causal history and cannot surface the target item being scored.
Abstract
from arXiv · showhide
Agentic recommender systems ground each decision of a large language model (LLM) in a persistent memory of the user, and in existing agents that memory is text: a narrative written and maintained by further LLM calls. Text limits this memory in two ways. It is updated one rewrite at a time, so exploiting the full interaction history is prohibitively expensive; and collaborative evidence, graded similarity over an entire catalog, does not survive translation into sentences. We propose CoVeMem (Collaborative Vector Memory), which vectorizes the collaborative core of the agent's memory. Frozen LightGCN user and item states form the memory bank; at each decision, the candidate set itself retrieves the most relevant historical states, which enter the LLM's context as soft tokens alongside a light textual profile. Contrastive alignment to item-semantic anchors, followed by listwise co-training with masked candidates, teaches the model to read these states and to rank through them; a pointwise yes/no readout scores each candidate. Across four instruction-grounded recommendation benchmarks, CoVeMem matches or exceeds the strongest collaborative text-memory agent on 19 of 20 metric cells while requiring zero additional LLM calls for memory maintenance beyond the shared static profile, against per-interaction calls for text memory. The memory now takes gradients: the full interaction history, out of reach for text, becomes available as training data for what the agent remembers and for how it reads what it remembers.
1 Introduction
Existing agentic recommender memories are textual narratives maintained through costly serial rewrites, limiting their ability to retain collaborative structure. CoVeMem replaces the collaborative component with retrievable vector states that the LLM learns to read through ranking supervision, improving performance while avoiding recurring maintenance calls.
- Motivation: Text memories serialize interaction history into evolving narratives through serial LLM rewrites, making full-history exploitation expensive and slow.They also lose collaborative structure when graded catalog-wide evidence is translated into sentences.
- Approach: CoVeMem uses graph-trained LightGCN user and item states as a persistent collaborative memory bank alongside a short textual profile.The candidate set retrieves relevant historical states for each decision.
- Approach: A projector and LoRA adapter translate retrieved history, user, and candidate states into soft tokens that the LLM can interpret.This trainable interface is called the parametric memory reader.
- Learning: Masked listwise co-training removes title-only shortcuts, teaching the model to compare candidates through collaborative tokens.The method trains memory reading from the ranking task rather than relying only on textual rewrites.
- Results: CoVeMem exceeds the strongest text-memory baseline in nearly all evaluated settings across four instruction-grounded benchmarks, with especially clear gains in interaction-dense domains.It also avoids recurring LLM calls for memory maintenance beyond the one-time textual profile.
2 Related Work
Prior work studies textual memory for LLM recommendation and separately injects collaborative representations into LLM recommenders. CoVeMem combines these directions by making collaborative representations the persistent collaborative component of a complete agent memory.
- LLM agents for recommendation: Agentic recommendation research uses persistent memory to carry past preferences and interactions into later decisions.Examples include static profiles, dynamic feedback-updated memories, and text memories adjusted for collaborative filtering.
- Integrating collaborative information into LLMs: Collaborative-LLM recommenders project user, item, behavioral, or historical-item representations into LLM token space and often adapt the model with LoRA.Related systems differ in whether embeddings are frozen, how histories are represented, and how candidates are scored.
- Positioning: CoVeMem is presented as the first agentic recommender to use collaborative representations as persistent memory with candidate-conditioned retrieval and a ranking-trained memory reader.This distinguishes it from systems that inject collaborative representations only for conventional recommendation.
3 Method
CoVeMem combines a frozen graph-trained collaborative memory with a trainable LLM reader, retrieving candidate-relevant history and injecting it as soft tokens alongside a textual profile. Alignment and masked listwise co-training teach the reader to use these states, while pointwise yes/no scoring ranks candidates.
- Collaborative Vector Memory: CoVeMem combines a LightGCN memory bank of frozen user and item states with a short textual profile.The bank is trained once on the training-interaction graph and then remains fixed across decisions.
- Collaborative Vector Memory: Candidate-aware retrieval selects K=5 historical states by similarity to the candidate centroid rather than by recency.Retrieval uses the available causal history and requires only dot products, without an LLM call.
- Learning to Read the Memory: A gated projector maps retrieved history, user, and candidate states into soft tokens, while a LoRA adapter teaches the LLM how to read them.Together, the projector and adapter form the trainable parametric memory reader; the states themselves remain frozen.
- Learning to Read the Memory: Semantic-anchor alignment initializes projected item states in the LLM language space using a symmetric in-batch contrastive loss.Each item state is aligned with an anchor built from its title and available category or description tokens.
- Learning to Read the Memory: Listwise co-training uses prefix-causal ranking events and masks candidate titles so masked comparisons must rely on collaborative tokens.When the positive is masked, the margin loss compares only masked candidates; otherwise it uses the full candidate set.
- Memory-Grounded Adjudication: At inference, each candidate receives a compact prompt and a parallel yes/no logit score, avoiding generated ranked-list parsing.The retrieved history slots are computed once from the candidate set and shared across the candidate prompts.
4 Experiments
Across four instruction-grounded recommendation domains, CoVeMem is evaluated against non-agentic and text-memory baselines, through ablations, backbone substitutions, and memory-efficiency measurements. It matches or exceeds MemRec on 19 of 20 metric cells, while its injected states, LoRA reader, and zero-maintenance design determine its gains and cost profile.
- Experimental Setup: The evaluation uses four domains, leave-one-out splits, ten-candidate test lists, and Hit@K and NDCG@K metrics.The domains are Amazon Book, Goodreads, Amazon MovieTV, and Yelp; each test list contains one held-out target and nine negatives.
- Overall Performance: CoVeMem matches or exceeds MemRec on 19 of 20 metric cells, and all remaining text-memory agents score below it on every metric.The largest advantage is on Goodreads, while MovieTV shows the same direction across all five metrics at smaller margins.
- Overall Performance: On Yelp and Books, where item states have less direct collaborative evidence, CoVeMem leads on Yelp and stays on par on Books while retaining textual evidence.The results describe collaborative information as a complement to textual semantics in this lower-interaction regime.
- Memory Ablation Study: Most of CoVeMem’s Hit@1 advantage persists under target-aware and recency history selection, indicating that injected states provide most of the gain.The text-only variants fall into the text-memory-agent range of Table 2, although target-aware selection generally outperforms recency.
- Memory Ablation Study: Removing LoRA reduces performance to Vanilla LLM level on Yelp and below it on Goodreads, because unchanged attention cannot effectively read the injected states.The injected states instead interfere with textual evidence when the attention mechanism is not adapted to them.
- Collaborative Backbone Study: All five alternative collaborative backbones train successfully, and the three strongest outperform the text-only variant on every metric.The strongest backbone surpasses default LightGCN on every metric, whereas the weakest shows no clear advantage over text-only memory.
- Memory Maintenance Efficiency: CoVeMem incurs zero LLM tokens for memory maintenance and has the lowest total per-user token cost on every dataset.Its persistent components are built once offline, and its nongenerative yes/no readout leaves only decision-time inputs; it occupies the high-accuracy end of the Pareto frontier.
5 Conclusion
CoVeMem combines target-aware retrieval of collaborative states with contrastive alignment and masked co-training so an LLM can read vector memory for ranking. Across four benchmarks, it matches or exceeds the strongest text-memory agent on 19 of 20 metric cells while requiring zero additional memory-maintenance calls.
- Conclusion: CoVeMem retrieves frozen collaborative states into the LLM context and trains a reader to use them for ranking.Its collaborative vector memory is paired with a light textual profile, and contrastive alignment plus masked co-training teaches the read interface.
- Conclusion: CoVeMem preserves evidence in collaborative form rather than serializing it into sentences, making the full interaction history available as event-level supervision.The conclusion presents the memory carrier as a separate design axis and notes that conventional recommenders can supply the collaborative memory.
A.1 Instruction Positions
The instruction-position rule card uses train-only interaction density and popularity concentration to select early, middle, or late instruction placement. The generated thresholds assign each benchmark to a position without validation or test results.
- Rule Inputs: The rule uses interaction density d and concentration ratio q computed from training statistics.The statistics count training interactions, items, and interactions involving the most popular 1% of items.
- Placement Rule: Early applies when d ≥5 or q ≥25%; middle applies below both early thresholds when d ≥2.5 or q ≥12%; late applies otherwise.The rule was generated without access to validation or test results.
- Dataset Assignments: The rule assigns Goodreads to early, MovieTV to middle, and Yelp and Books to late.The assignments follow each dataset’s reported density and concentration values.
A.2 Textual Profile Construction
Profiles are generated offline from training-only user evidence and constrained to bounded item, metadata, review, and output lengths. The normalized profile is stored once and rendered with a uniform character limit.
- Profile Inputs: Each profile uses only train-split interactions, reviews, and item metadata.The input includes at most 12 recent training items, with titles and categories truncated to 60 characters and reviews to 300 characters.
- Generation: Qwen2.5-7B-Instruct generates each profile at temperature 0 with a maximum of 160 tokens.The Yelp generation template is reproduced in Listing 3.
- Storage and Rendering: Profiles are normalized and stored once offline, then rendered on one line and uniformly truncated to 220 characters.The code supplement includes the generation script and domain-matched templates.
A.3 Training Configuration
Training combines a lightweight LoRA adaptation and gated state projector with a train-only LightGCN bank, alignment, and masked-candidate co-training. Experiments use a fixed hardware and software environment.
- Adaptation: The LoRA adapter uses rank r=4, α=8, dropout 0.05, and all four attention projections.The projections are Wq, Wk, Wv, and Wo.
- State Projector: The gated projector maps d=64 states through hidden widths 64, 256, 1024, and 3584 before LayerNorm output.Token-history slots are zero-padded when fewer than K items are available.
- Collaborative Training: The LightGCN bank uses three propagation layers and 50 BPR epochs on the train-only graph, with alignment temperature τ=0.07.The training graph uses seed 42.
- Execution Environment: All experiments ran on Ubuntu 22.04.5 LTS with an Intel Xeon Gold 6348 host and one NVIDIA A800 80 GB GPU.The software stack is specified in the main paper.
A.4 Checkpoint Selection
Checkpoint selection retains the epoch achieving the best ranking on a fixed 400-user validation subsample using the test-time pointwise yes/no readout.
- Selection Rule: The best-ranking checkpoint is selected after each co-training epoch on a fixed 400-user validation subsample.The subsample is drawn once with seed 4242, and ranking uses the same pointwise yes/no readout as testing.
B Prompt Artifacts
The appendix specifies prompt artifacts, offline profile construction, train-only rule generation, and two complementary prompts for training and evaluation. It also describes how domain statistics guide instruction placement relative to collaborative memory tokens.
- B.3 Training and Evaluation Prompts: The appendix renders a listwise co-training event with shuffled candidates, independently masked candidates, and loss restricted to the masked subset.In the shown Goodreads draw, the positive candidate is masked and retains only its item tokens.
- B.3 Training and Evaluation Prompts: The per-candidate evaluation prompt asks for a yes/no judgment on whether one candidate is the best held-out match among the current candidate set.The rendered example includes five history slots, a collaborative user token, and the candidate’s item token.
- B.3 Training and Evaluation Prompts: The two prompt forms preserve one memory schema while changing user-token position and selecting history according to each decision’s candidate set.Lookup by token ID injects each vector at its corresponding placeholder despite prompt-order differences.
- B.3 Training and Evaluation Prompts: The rule generator receives only train-split domain statistics and a position mechanism, then outputs an interpretable per-domain configuration rule without evaluation results.The output assigns each domain to early, middle, or late instruction placement.
- B.3 Training and Evaluation Prompts: The system’s pointwise prompt combines the user, optional instruction position, textual profile, historical titles, collaborative user memory, candidate item memory, and a scoring task.Position A places the instruction before the profile and memories; position B places it between user memory and the candidate; position C places it after the candidate.
- B.3 Training and Evaluation Prompts: Recency makes late instructions more influential but moves candidate and collaborative tokens away from the decision, whereas early placement prioritizes collaborative evidence.Middle placement increases instruction recency over early placement while preserving candidate-token adjacency to the decision point.
- B.3 Training and Evaluation Prompts: The rule-generation statistics cover Yelp, Goodreads, MovieTV, and Books with domain-specific interaction counts, users-per-item density, and popularity concentration.The listed domains range from 1.74 to 10.54 users per item and report top-1% item-share statistics where available.
- B.2 Textual Profile Generator: The offline Yelp profile generator summarizes visited places and reviews into concise, objective preference descriptions covering cuisines, atmosphere, price sensitivity, and recurring themes.It uses up to 12 train-only evidence lines, optionally including the user’s review for each title or place.