PaperlayerView exact revision on arXiv

Source-linked AI summary

Evoking User Memory: Personalizing LLM via Recollection-Familiarity Adaptive Retrieval

Yingyi Zhang, Junyi Li, Wenlin Zhang, Penyue Jia, Xianneng Li, Yichao Wang, Derong Xu, Yi Wen, Huifeng Guo, Yong Liu, Xiangyu Zhao

arXiv:2603.09250v1cs.IR

TL;DR

Personalized LLMs need memory retrieval that is both efficient and capable of recovering context beyond surface matches. RF-Mem introduces an uncertainty-guided dual-path retriever that switches between Familiarity and iterative Recollection, and experiments report robust gains across tasks and corpus scales. The study evaluates simulated and public benchmarks rather than private user data, with deployment risks in sensitive applications requiring safeguards.

  • Problem

    Existing personalized memory retrieval is limited by one-shot similarity search and lacks adaptive coordination between fast Familiarity and deeper Recollection.

  • Method

    RF-Mem estimates familiarity using mean similarity and entropy, then selects direct top-K Familiarity retrieval or clustered, query-mixed iterative Recollection.

  • Results

    RF-Mem achieves robust gains across generation and retrieval tasks and scales reliably to million-entry corpora, outperforming one-shot and full-context alternatives.

  • Takeaways & Limitations

    Adaptive integration of deliberate Recollection can improve personalized memory retrieval while preserving efficiency across changing familiarity conditions.

  • Takeaways & Limitations

    Evaluation relies on simulated and publicly available benchmarks without private user data, and sensitive deployment requires data governance, privacy, and fairness safeguards.

Abstract

from arXiv · show

Personalized large language models (LLMs) rely on memory retrieval to incorporate user-specific histories, preferences, and contexts. Existing approaches either overload the LLM by feeding all the user's past memory into the prompt, which is costly and unscalable, or simplify retrieval into a one-shot similarity search, which captures only surface matches. Cognitive science, however, shows that human memory operates through a dual process: Familiarity, offering fast but coarse recognition, and Recollection, enabling deliberate, chain-like reconstruction for deeply recovering episodic content. Current systems lack both the ability to perform recollection retrieval and mechanisms to adaptively switch between the dual retrieval paths, leading to either insufficient recall or the inclusion of noise. To address this, we propose RF-Mem (Recollection-Familiarity Memory Retrieval), a familiarity uncertainty-guided dual-path memory retriever. RF-Mem measures the familiarity signal through the mean score and entropy. High familiarity leads to the direct top-K Familiarity retrieval path, while low familiarity activates the Recollection path. In the Recollection path, the system clusters candidate memories and applies alpha-mix with the query to iteratively expand evidence in embedding space, simulating deliberate contextual reconstruction. This design embeds human-like dual-process recognition into the retriever, avoiding full-context overhead and enabling scalable, adaptive personalization. Experiments across three benchmarks and corpus scales demonstrate that RF-Mem consistently outperforms both one-shot retrieval and full-context reasoning under fixed budget and latency constraints. Our code can be found in the Reproducibility Statement.

1 INTRODUCTION

Personalized memory retrieval must surface user-specific histories and contexts without relying solely on costly full-context prompting or shallow one-shot similarity. RF-Mem addresses this gap by combining Familiarity and deliberate Recollection through uncertainty-guided adaptive retrieval.

  • Personalized memory retrieval targets user-specific histories, preferences, and contextualized interactions rather than objective open-domain facts.
  • Human memory combines rapid, coarse Familiarity with slower Recollection that reconstructs contextual details when familiarity is insufficient.
  • Existing memory retrieval methods largely reduce retrieval to one-shot similarity search, overlooking deliberate Recollection and adaptive switching between retrieval paths.
  • RF-Mem estimates familiarity uncertainty from probe-retrieval scores and adaptively selects either direct Familiarity retrieval or structured Recollection.
  • RF-Mem combines clustering and query–centroid mixing to reconstruct evidence chains in embedding space while remaining lightweight and near one-shot in latency.

2 METHOD: RECOLLECTION–FAMILIARITY MEMORY RETRIEVAL

RF-Mem uses familiarity uncertainty to choose between efficient one-shot retrieval and deeper recollection. Its Recollection path iteratively clusters and mixes retrieved evidence under explicit computation budgets.

  • RF-Mem processes a user query, selects a retrieval path using familiarity uncertainty, extracts memory text, and generates an answer with an LLM.
  • Familiarity uncertainty-driven retrieval selection: The selection policy uses mean similarity and entropy: high familiarity or low uncertainty selects Familiarity, while weak familiarity or high uncertainty selects Recollection.
  • Familiarity retrieval: The Familiarity path directly returns top-K memory fragments from similarity search, providing rapid recognition with minimal additional computation.
  • Recollection retrieval: The Recollection path performs multi-round evidence expansion when the probe signal is unfamiliar, progressively reconstructing context beyond surface matches.
  • Recollection retrieval: Each Recollection round retrieves candidates, clusters them with KMeans, and uses cluster centroids as branching anchors for further query expansion.
  • Recollection retrieval: Centroids are blended with the current query through alpha-mixing, while beam width, fanout, round limits, and recursion caps keep computation tractable.

3 EXPERIMENTS

Experiments evaluate RF-Mem across personalized generation and retrieval benchmarks, memory scales, retrievers, indexing methods, and query-expansion settings. RF-Mem generally provides strong accuracy or recall while controlling input size and retrieval latency.

  • Overall performance in personalized generation: RF-Mem reaches the highest PersonaMem overall accuracy at 32K, 128K, and 1M tokens: 0.6350, 0.5394, and 0.4589.At 32K, it exceeds Full Context by +0.0221 using 3.6k average input tokens versus 24.7k.
  • Overall performance in personalized generation: RF-Mem leads or remains competitive on hybrid and transfer-style PersonaMem tasks across corpus scales, including Aligned Recommendations, New Scenarios, and Shared Facts.At 1M, it is strongest on Revisit Reasons, Aligned Recommendations, Shared Facts, and New Ideas, while Full Context is infeasible.
  • Overall performance in personalized generation: RF-Mem reduces retrieval latency relative to always-on Recollection: 5.09ms versus 7.09ms at 32K, 4.27ms versus 7.86ms at 128K, and 6.28ms versus 7.12ms at 1M.Its entropy-based controller preserves similar token budgets to Dense Retrieval while regulating when deeper retrieval is used.
  • Overall performance in personalized retrieval: Across PersonaBench and LongMemEval, RF-Mem matches or surpasses strong baselines in recall while maintaining robust behavior across retriever backbones.Under MiniLM, its overall Recall@10 is 0.6071, compared with 0.5964 for Familiarity and 0.6062 for Recollection.
  • Overall performance in personalized retrieval: Familiarity performs best on fact-centric queries, whereas Recollection is stronger on context-heavy tasks and improves Recall@10 to 0.6267 versus 0.5561 on PersonaBench Preference Hard under MiniLM.On LongMemEval, Recollection also lifts Recall@5 by more than 0.02 across multiple retrievers, such as 0.7351 versus 0.7136 under MiniLM.
  • Adaptability: RF-Mem integrates with summary-based indexing, nearline query expansion, and iterative retrieval pipelines as an online retrieval layer.It achieves the highest overall score with Search-o1 on PersonaMem and consistently matches or surpasses Familiarity baselines with HyDE on PersonaBench.

4 RELATED WORKS

Personalized memory retrieval systems organize user histories through query reformulation, index construction, and retrieval strategies. However, most existing methods still treat retrieval as one-shot similarity recognition, overlooking deliberate recollection and adaptive switching between the two paths.

  • Personal memory retrieval targets user histories, preferences, and contextual interactions rather than the objective facts typically targeted by standard RAG.
  • Query reformulation: Query reformulation methods expand or refine retrieval queries using keyphrases, memory-inspired cues, task-oriented queries, or contextual attributes.
  • Index construction: Index-construction methods organize memories through summaries, clusters, reflective or hierarchical representations, and relational graphs.
  • Retrieval frameworks: Retrieval frameworks rank memory items by semantic similarity, progressing from keyword search to dense retrievers and stronger encoders.
  • Most prior systems implicitly reduce memory retrieval to one-shot Familiarity, neglecting deliberate Recollection and adaptive switching between the two paths.

5 CONCLUSION

The paper reframes personalized memory retrieval through dual-process theory, adding deliberate Recollection to similarity-based Familiarity and adaptively switching between them. Experiments report robust gains across generation and retrieval tasks while scaling to million-entry corpora.

  • RF-Mem adaptively switches between one-shot Familiarity retrieval and deliberate, stepwise Recollection using familiarity uncertainty.
  • RF-Mem achieves robust gains across both generation and retrieval tasks and scales reliably to million-entry corpora.
  • The conclusion identifies deliberate recollection as important for personalizing LLM memory retrieval.

ETHICS STATEMENT

The study uses simulated and public benchmarks without collecting private user data, but real-world deployment raises risks involving privacy, profiling, fairness, and behavioral bias. The authors call for governance safeguards when extending the framework to sensitive applications and real user data.

  • The evaluation relies solely on simulated and publicly available PersonaMem, PersonaBench, and LongMemEval datasets without personally identifiable or sensitive data.
  • Deploying preference-based personalized retrieval on real user data may expose sensitive histories, create unintended profiling, or reinforce behavioral biases.
  • The authors recommend strict data governance, privacy preservation, and fairness guidelines for sensitive applications such as healthcare and personal decision-making.

REPRODUCIBILITY STATEMENT

The study supports reproducibility through public datasets, documented model and hyperparameter choices, sensitivity analyses, and released code and scripts. These materials are intended to facilitate replication under different configurations.

  • All datasets used are publicly available, with detailed source references provided in the main text.
  • Model architectures and hyperparameters including B, F, α, τ, θhigh, and θlow are documented in Appendix B.
  • Appendix D reports hyperparameter sensitivity analyses to assess robustness across configurations.
  • The authors release code and scripts to further facilitate replication.

A DATASET DETAILS

The study evaluates personalized memory retrieval across PersonaMem, PersonaBench, and LongMemEval, varying corpus scale, user-history sources, and task distributions.

  • PersonaMem: PersonaMem evaluates retrieval and generation from user-specific histories and preferences in long-term dialogue traces across increasing memory-corpus sizes.The study constructs three corpora of increasing size following the benchmark setup.
  • Dataset organization: The dataset tables report corpus statistics and task distributions for the evaluated memory benchmarks.These include LongMemEval corpus sizes and PersonaMem task distributions.
  • PersonaBench: PersonaBench aggregates heterogeneous histories from six users, including social conversations, user-AI interactions, and e-commerce purchases.Each user contributes about 44 queries grounded in roughly 88 memory items.
  • PersonaBench: PersonaBench complements PersonaMem by testing retrieval across casual dialogue and structured purchase records.This setting broadens personalization scenarios beyond long user-AI dialogues.
  • LongMemEval: LongMemEval tests retrieval of task-relevant information from synthetic, extended user-specific corpora in small and medium memory settings.LongMemEval-S associates each question with approximately 50 memories, while LongMemEval-M uses larger associated corpora.

B IMPLEMENTATION DETAILS

Implementation uses benchmark-specific memory construction, retrievers, thresholds, and controlled prompting, with empirical score distributions examined across datasets.

  • Runtime setup: Experiments run on a single exclusively allocated NVIDIA A100 GPU under Ubuntu while measuring runtime.The GPU is dedicated to the process during runtime evaluation.
  • PersonaMem: PersonaMem uses dialogue-turn memory chunks, advanced LLM generation, and multi-qa-MiniLM-L6-cos-v1 retrieval with fixed routing hyperparameters.The reported thresholds are θhigh = 0.6 and θlow = 0.3, with λ = 20, B = 3, and F = 2.
  • Score analysis: Empirical mean-score distributions are examined for PersonaMem, PersonaBench, and LongMemEval datasets.The associated figures report mean score ¯s and entropy H(p), while LongMemEval uses separate small and medium settings.
  • PersonaBench: PersonaBench evaluates multiple retrievers and uses session-level memory construction for Recall-based retrieval evaluation.The setup includes multi-qa-MiniLM-L6-cos-v14, all-MiniLM-L6-v25, and bge-base-en-v1.56.
  • LongMemEval: LongMemEval uses session-level memories, Recall evaluation, multiple retrievers, and thresholds θhigh = 0.6 and θlow = 0.0.The reported settings include B = 4, F = 1, and λ = 20 for both LongMemEval variants.
  • Prompting: The multiple-choice evaluation constrains the generator to return exactly one option from (a), (b), (c), or (d).Retrieved memory is prepended as dialogue history alongside the question, instruction, and candidate options.

D.1 RESULT ACROSS CATEGORY IN PERSONAMEM

Category-level PersonaMem results show complementary strengths: Familiarity suits direct factual recall, Recollection helps contextual reasoning, and RF-Mem combines both through adaptive switching.

  • Familiarity: Familiarity performs strongly on direct factual categories but degrades as tasks require contextual integration, especially at larger memory scales.It reaches 1.0000 on food recommendation at 32k and 0.5759 on movie recommendation at 128k.
  • Recollection: Recollection improves reasoning-intensive retrieval by reconstructing dispersed cues across sessions.It reaches 0.5938 versus 0.5625 in family relations at 32k and 0.5280 versus 0.4534 in therapy at 1M.
  • RF-Mem: RF-Mem combines Familiarity and Recollection through adaptive switching and achieves the best overall results at all corpus scales.It improves legal consultation at 32k to 0.94 versus 0.78/0.88 and family relations at 1M to 0.60 versus 0.57/0.58.
  • Hyperparameter sensitivity: Moderate query–centroid mixing generally supports recollection expansion, whereas extreme α values reduce stability.For Recall@5, α around 0.3–0.6 yields the best results, with MiniLM reaching approximately 0.47 overall recall.
  • Hyperparameter sensitivity: Moderate entropy gating preserves adaptivity by routing complex cases toward Recollection without defaulting prematurely to Familiarity.MiniLM Recall@10 peaks at approximately 0.60 when τ is around 0.2–0.25, while τ ≥0.35 reduces adaptivity.

D.3 SENSITIVITY ANALYSIS OF B AND F

Sensitivity analyses show that RF-Mem benefits from controlled expansion: moderate beam width and fanout balance early precision with broader coverage, while α should be calibrated to retrieval depth. Case studies and alternative clustering experiments further show both the value and limits of recollection.

  • Effect of B and F: Recall@5 decreases from 0.72 at F = 1 to below 0.60 at F = 4 in LongMemEval-S, showing that excessive fanout dilutes top-ranked precision.Recall@10 peaks around F = 1 or F = 2, including 0.83 in LongMemEval-S and 0.56 in LongMemEval-M, before declining at larger fanout.
  • Effect of B and F: Recall@50 remains above 0.98 across LongMemEval-S settings and reaches 0.76 in LongMemEval-M at B = 3, F = 2, indicating stronger long-depth coverage with expansion.The results support broad exploration at longer retrieval depths, but not unrestricted expansion.
  • Effect of B and F: B = 2 or 3 with F = 1 or 2 offers the best balance between early precision and broad coverage.These settings align with controlled, stepwise recollection rather than unrestricted search.
  • Effect of α: Smaller α favors short-path precision, whereas larger α improves long-path coverage; Recall@10 has an intermediate optimum.No single α setting dominates across retrieval depths, so α remains application-dependent.
  • Effect of K: RF-Mem achieves higher accuracy than either baseline across corpus scales and avoids Recollection’s over-expansion by invoking it selectively.Performance improvements saturate beyond moderate probe sizes, with K = 10–20 generally sufficient for familiarity estimation and switching.
  • Case studies: The recollection case study shows iterative clustering and query–centroid mixing progressively uncover complementary evidence across multiple rounds.The winning Familiarity case also shows that surface cues can sometimes cover intent better than multi-round recollection, which may drift toward dominant but misaligned clusters.
  • Clustering alternatives: RF-Mem (DBSCAN) improves over its recollection baseline, while RF-Mem (Spectral) raises R@5 Overall from 0.4522 to 0.4651 over the Spectral baseline.These findings indicate that α-mixing is not limited to KMeans and generalizes across cluster geometries.
  • Mixing alternatives: RF-Mem (Gate) obtains 0.4602 R@5 Overall versus 0.4701 for the KMeans-based implementation, with larger gaps in Social Info and Pref-Hard.The comparison supports α-mixing over the evaluated nonlinear gated alternative.

F.4 COMPLEXITY–COVERAGE TRADE-OFF

RF-Mem’s complexity is bounded by beam width, fanout, and rounds rather than the full memory corpus. Its formal analysis connects this bounded overhead to the method’s reported accuracy–latency improvements.

  • Complexity bounds: Familiarity has time TFam = O(K csim), while Recollection uses similarity search and clustering costs controlled by B, F, and R.The stated cost components are similarity evaluations and B-way k-means updates on retrieved batches.
  • Complexity bounds: Recollection costs O((BF + RF) csim + R cclust(B)), which is polynomial in B, F, and R.The bound captures both candidate expansion across rounds and repeated clustering updates.
  • Complexity–coverage trade-off: When BF + RF ≪ M, Recollection is strictly cheaper than full-context processing O(M).This establishes a formal cost advantage when the controlled retrieval parameters remain small relative to total memory size.
  • Formal guarantees: The formal results bound mis-selection and computation while supporting the empirical accuracy–latency improvements reported for RF-Mem.Lemmas justify the control signals, theorems establish optimality within a monotone policy class, and propositions provide bounds.

H LIMITATION AND FUTURE WORK

RF-Mem’s current scope is limited to dialogue-style personalized text memory and a static embedding index, while future work targets broader modalities, richer uncertainty signals, temporal dynamics, and tighter retrieval-generation integration.

  • The evaluation covers dialogue-style personalized memory but not cross-modal histories or other modalities.
  • List entropy is an effective lightweight uncertainty proxy, but may not fully capture task difficulty, user intent, or semantic ambiguity.
  • The static embedding index does not model temporal memory updates or conflicts across long-term sessions.
  • Deployments should safeguard against unintended resurfacing of sensitive long-term user information.
  • Future work proposes multimodal and cross-domain evaluation to test RF-Mem beyond personalized text-based memory.
  • Future work includes richer uncertainty cues, temporal indexing, and tighter integration between retrieval and generation.
Loading 2603.09250v1…