Source-linked AI summary
PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations
Hyojeong Yu, Hyukhun Koh, Minsung Kim, Yunah Jang, Kyomin Jung
TL;DR
Long-term personalized guidance requires more than factual memory retrieval: models must integrate temporally distributed evidence, evolving user trajectories, and potentially incorrect assumptions. PRAGMA provides a controlled benchmark for this setting and finds that current retrieval, memory, and long-context systems struggle both to recover relevant evidence and to use it effectively. The benchmark therefore exposes limitations in memory-grounded personalized reasoning, especially for corrective guidance.
Problem
Existing conversational-memory evaluations mainly focus on retrieval and factual recall, while open-ended personalized guidance requires reasoning over evolving user trajectories and potentially incorrect assumptions.
Method
PRAGMA uses controlled, human-validated longitudinal conversations with annotated evidence and four event- or trajectory-grounded query types covering aligned and corrective guidance.
Results
Across retrieval systems, memory systems, and long-context models, current systems struggle both to recover appropriate conversational evidence and to use it effectively for personalized guidance.
Takeaways & Limitations
Personalized guidance evaluation must assess robust conversational retrieval together with downstream memory-grounded reasoning beyond factual evidence recall.
Takeaways & Limitations
The benchmark uses human-validated histories generated through a controllable synthetic pipeline and evaluates guidance in a single-turn setting.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly deployed as personalized assistants that interact with users over extended periods of time. As conversations grow longer, relying on full interaction histories becomes increasingly inefficient and unreliable: long contexts introduce substantial computational overhead, making it difficult for models to consistently identify and utilize the most relevant information for the current request. These challenges have motivated memory systems that structure and retrieve user-specific information. In realistic interactions, users often seek practical guidance such as recommendations, planning, and decision support. Unlike factual recall tasks, personalized guidance requires models to integrate information across multiple past conversations and reason about changing user preferences and experiences. However, existing conversational memory evaluations mainly focus on retrieval and factual recall. To study this challenge, we introduce PRAGMA, a benchmark for evaluating personalized guidance in long-term conversations. PRGAMA contains curated longitudinal conversation histories, evidence annotations, and guidance scenarios grounded in evolving user contexts and incorrect user assumptions. Experiments across retrieval systems, memory systems, and long-context models reveal that current systems struggle both to recover the appropriate conversational evidence and to effectively use it for personalized guidance. Our results highlight the need for memory architectures that support robust conversational retrieval and memory-grounded reasoning beyond evidence recall.
1 Introduction
PRAGMA addresses the gap between factual conversational-memory evaluation and personalized guidance that requires reasoning over evolving user contexts and potentially incorrect assumptions. It introduces a controlled benchmark and finds that current systems struggle to retrieve and use relevant conversational evidence.
- Personalized guidance requires both conversational retrieval and coherent memory-grounded reasoning because relevant evidence can be temporally distributed and implicitly connected to the request.
- Existing conversational-memory work primarily evaluates retrieval and factual recall, while practical guidance requires reasoning over evolving experiences and assumptions.
- PRAGMA evaluates personalized guidance grounded in long-term conversations, including event-level memories, evolving user states, and conflicting user assumptions.
- The benchmark uses a controlled, human-validated construction pipeline with longitudinal conversations, evolving memory dependencies, and fine-grained evidence annotations.
- Experiments across retrieval, memory, and long-context systems show difficulty recovering relevant evidence and using it effectively for personalized guidance.
2 Related Work
Existing conversational-memory benchmarks mainly test whether models recover or reproduce past information, leaving open-ended personalized guidance grounded in evolving trajectories less thoroughly evaluated.
- LongMemEval, LoCoMo, and ConvoMem evaluate factual question answering, dialogue understanding, and memory-grounded responses over long conversational histories.
- These benchmarks primarily assess recovering or reproducing past information rather than using memory for open-ended practical guidance.
- Personalized-generation benchmarks study implicit reasoning, preference evolution, recommendation, or profile conditioning but use narrower settings or multiple-choice evaluation.
- Overall, existing benchmarks provide limited evaluation of open-ended personalized guidance grounded in evolving user trajectories.
3 PRAGMA
PRAGMA constructs long-term conversational contexts and query types that test event- and trajectory-grounded guidance under aligned or incorrect assumptions. It combines controlled generation, evidence annotation, human validation, and retrieval- and response-level evaluation.
- Query Design: PRAGMA organizes queries along memory dynamics and query alignment, producing Event-Align, Event-Correct, Traj-Align, and Traj-Correct categories.
- Query Design: Corrective queries require identifying inconsistencies between a user’s assumptions and prior event evidence or longer-term trajectory before giving guidance.
- Query Design: Trajectory queries use multi-axis longitudinal states, and aligned queries hide those axes behind abstract descriptors to reduce lexical shortcuts.
- Benchmark Construction: The construction pipeline expands evidence and filler topics into timestamped sessions, interleaves filler sessions, and prevents trivial boundary-position or recency heuristics.
- Benchmark Construction: The final benchmark contains 100 users and 400 queries, with approximately 160K conversational tokens per user history.
- Annotations and Evaluation Protocols: Retrieval is evaluated with session-level evidence coverage, while responses are judged for alignment with user history and grounding in annotated evidence.
4 Experimental Setup
PRAGMA evaluates RAG and memory systems alongside long-context baselines using two generation models, reference conditions, and retrieval-focused metrics. Responses are assessed for alignment and grounding with query-type-specific evaluation, while retrieval is measured by recall and exact recall.
- Models and Baselines: Experiments use gpt-5-mini and qwen3-30b-a3b-instruct, with bge-base-en-v1.5 embeddings for retrieval.
- Models and Baselines: The experiments compare RAG systems, A-MEM, Mem0, SimpleMem, and full-context, no-context, and oracle reference conditions.Two generation models are used, with memory systems providing top-k retrieved records as response context.
- Reference Conditions: No-context, oracle-session, oracle-summary, and full-context conditions isolate the effects of conversational evidence access and long-context conditioning.Oracle-session supplies gold evidence sessions, whereas oracle-summary supplies summarized evidence from annotated metadata.
- Evaluation: Retrieval evaluation reports Recall for retrieved evidence-session coverage and Exact Recall for recovering all required evidence sessions.
- Evaluation: Response evaluation uses alignment for consistency with user history and grounding for explicit support from annotated evidence.The benchmark applies query-type-specific rubrics and reports robustness checks with additional judging models.
5 Experimental Results
Across retrieval, memory, and long-context systems, PRAGMA exposes failures in both recovering relevant evidence and using it for personalized guidance. Retrieval granularity and evidence presentation matter, but even strong retrieval does not ensure grounded responses, especially for corrective and trajectory-grounded queries.
- Main Results: Full-Context achieves only 8.00 grounding on Trajectory-Align under gpt-5-mini, below practical retrieval and memory systems.Complete conversational history alone is therefore insufficient for robust personalized guidance in this evaluation.
- Query-Type Differences: Event-Correct alignment remains between 2.00 and 8.00 across practical systems, while Trajectory-Correct alignment remains below 43.00 under gpt-5-mini.Corrective guidance is particularly difficult when user assumptions conflict with prior memory.
- Query-Type Differences: Dense-session retrieval improves Trajectory-Align alignment from 53.00 to 73.00 over turn-level retrieval under gpt-5-mini.Trajectory-grounded queries benefit from broader conversational context, although grounding remains limited.
- Main Results: A-MEM reaches 81.00 alignment but only 16.00 grounding on Trajectory-Align under gpt-5-mini.Alignment scores are generally higher than grounding scores, indicating that plausible guidance may remain weakly supported by conversational evidence.
- Gold Retrieval Is Not Enough: Oracle-Summary reaches 99.00 alignment and 86.00 grounding on Trajectory-Align, compared with 78.00 and 17.00 for Oracle-Session.The result indicates that concise evidence summaries can be more useful than simply exposing the relevant sessions.
- Retrieval Results: Dense retrieval achieves 62.48 Recall but 0.00 Exact Recall on Trajectory-Align, while BM25 achieves 51.95 Recall with similarly low complete recovery.
- Retrieval Results: Query Rewriting reaches 95.95 Recall and 80.00 Exact Recall on Trajectory-Align but only 25.00 Alignment and 20.00 Grounding.Strong evidence recovery therefore does not necessarily produce strong downstream responses.
- Retrieval Granularity: Session-level retrieval outperforms turn-level retrieval for trajectory-grounded queries, whereas Event-Correct queries often favor turn-level retrieval.Broader session context can obscure the fine-grained evidence needed for correction.
6 Analysis
The analysis separates memory preservation, retrieval accessibility, and response utilization, revealing tradeoffs between retaining conversational detail and making evidence usable for generation. Corrective failures persist even with inconsistency identification, while summarized representations can improve downstream evidence use.
- Corrective Guidance: Corrective grounding remains substantially lower despite strong retrieval performance and explicit inconsistency identification.The findings indicate that corrective failures cannot be explained solely by failure to detect inconsistencies.
- Corrective Guidance: Extremely low corrective alignment is consistent with over-accommodation to user assumptions, making effective assistance require challenging incorrect beliefs.
- Memory Pipeline: The memory pipeline is decomposed into preservation, retrieval accessibility, and response utilization.These dimensions distinguish whether evidence is stored, retrieved, and reflected in the final response.
- Memory Tradeoffs: A-MEM preserves 99.1% of evidence on Traj-Correct but retrieves only 48.9%.Nearly perfect preservation through raw-content storage does not guarantee downstream accessibility.
- Memory Abstraction: Summarized memory systems often achieve stronger response-stage utilization than more detailed memory representations and standard RAG.Concise structured memories can expose evidence in a form generation models incorporate more reliably.
7 Conclusion
PRAGMA evaluates personalized guidance beyond factual recall under evolving preferences and potentially incorrect assumptions. Results across RAG, memory, and long-context models show failures in both retrieval and utilization, motivating memory-grounded longitudinal reasoning.
- Contribution: PRAGMA evaluates whether models provide grounded guidance under evolving user preferences and potentially incorrect assumptions.
- Findings: Experiments reveal substantial failures in both memory retrieval and utilization, including cases where retrieved evidence does not yield grounded personalized guidance.
Limitations
PRAGMA’s controlled synthetic construction enables annotation and trajectory control but may not fully represent natural conversations. Its single-turn and session-level evidence evaluations also leave interactive recovery and fine-grained reasoning underrepresented.
- Synthetic histories enable evidence annotation and trajectory control but may not capture the ambiguity and variability of natural long-term conversations.
- Single-turn guidance evaluation excludes recovery through iterative interaction or follow-up dialogue.
- Session-level evidence annotations improve reliability and scalability but may miss multiple valid reasoning paths or partially implicit evidence.
Ethical Considerations
PRAGMA uses controlled generation and human validation to construct personalized guidance scenarios while reducing shortcut solutions. Its validation process targets realism, evidence quality, and the intended reasoning demands across four query types.
- Benchmark construction: PRAGMA uses a controlled pipeline to generate personas, trajectories, events, filler sessions, and four query types within timestamped long-term histories.
- Human validation: Six annotators reviewed conversations, queries, and evidence annotations for realism, completeness, leakage, supported inconsistencies, and resistance to superficial heuristics.
- Query-type validation: Validation criteria cover event-level alignment and correction plus trajectory-level reasoning over evolving preferences, circumstances, and conflicting decisions.
- Validation caveat: Annotators could rewrite unnatural queries while preserving their intended type and evidence dependency, but examples were not exhaustively double-annotated and agreement statistics were not reported.
- Longitudinal design: The running example spans 354 days, with five relevant evidence sessions distributed over 333 days among unrelated sessions.
B.1 Implicitness Analysis
PRAGMA quantifies how much queries leave relevant evidence and intended reasoning implicit. Corrective queries are more implicit than aligned queries, and higher implicitness corresponds to weaker retrieval and alignment outcomes.
- Measures: Retrieval Implicitness measures evidence recoverability from the query surface, while Instructional Implicitness measures whether the intended personalized reasoning behavior is explicit.
- Query-type patterns: 0.6817 is the average implicitness score for Event-Correct, compared with 0.5227 for Event-Align; Trajectory-Correct reaches 0.6656 versus 0.4902 for Trajectory-Align.
- Outcome associations: Higher retrieval implicitness tends to accompany lower retrieval F1, while higher overall implicitness is associated with lower downstream alignment performance.
- Benchmark statistics: PRAGMA contains 100 users and 400 queries across four query types, with 72.0% of its 4,167 sessions serving as irrelevant filler.
- Evaluation validation: Gold reference responses score near ceiling, whereas no-context responses score substantially lower under the rubric-based evaluation.
E.1 Additional Baseline Results
Additional baselines show that memory organization and evidence synthesis affect personalized guidance differently across query types and models. No single representation consistently optimizes alignment and grounding.
- Baseline patterns: RAPTOR performs strongly on aligned queries but remains limited on corrective reasoning, while LightMem is generally comparable to existing memory systems.
- Evidence synthesis: Synth improves corrective-query performance over turn-level dense retrieval under the same retrieval granularity, highlighting explicit evidence synthesis.
- Model generalization: Across three additional models, summarized oracle evidence substantially improves performance over original evidence sessions.
- Corrective reasoning: Claude reaches 65.00 alignment on Event-Correct with Oracle-Summary, versus 100.00 on Event-Align and Trajectory-Align and 98.00 on Trajectory-Correct.
- Representation trade-offs: No single memory representation is consistently optimal across models and query types; structured or summarized memories benefit some settings, while session retrieval benefits others.
- Retrieval versus utilization: Standard RAG can retrieve all annotated evidence yet fail to incorporate it fully, whereas structured memories can produce more complete responses with fewer retrieved items.
F.3 Grounding Failures in Follow-up Interactions
The case study shows that alignment alone can miss grounding failures in follow-up personalized guidance. Retaining the user’s specific prior constraints produces advice that differs substantially from generic recommendations.
- The case contrasts an aligned, grounded oracle-summary condition with an aligned, poorly grounded SimpleMem condition.
- The grounded condition retains the user’s low-salt meal plan, nutritionist consultation, and lower-sugar choices when advising about sports hydration drinks.
- The poor-grounding condition gives generic advice endorsing sports drinks for hard, sweaty runs without connecting it to the user’s low-salt nutrition plan.
- The example motivates evaluating alignment and grounding separately, especially when earlier responses become context for later reasoning.