Source-linked AI summary
RealMem: Benchmarking LLMs in Real-World Memory-Driven Interaction
Haonan Bian, Zhiyuan Yao, Sen Hu, Zishan Xu, Shaolei Zhang, Yifu Guo, Ziliang Yang, Xueran Han, Huacan Wang, Ronghao Chen
TL;DR
Existing memory benchmarks underrepresent long-term project interactions, where agents must track evolving goals and context across sessions. RealMem addresses this gap with a benchmark and dynamic-memory synthesis pipeline, finding that current systems struggle to maintain coherent project threads and long-term dependencies.
Problem
Existing memory benchmarks focus on casual or task-oriented dialogue and do not adequately evaluate long-term project interactions with evolving goals and interleaved context.
Method
RealMem combines project foundation construction, multi-agent dialogue generation, and memory and schedule management to synthesize evolving project interactions.
Results
Current memory systems struggle to maintain coherent project threads and proactively align with evolving contexts in long-term interactions.
Takeaways & Limitations
The substantial gap between evaluated methods and the Oracle upper bound identifies long-term memory integration as a core bottleneck for project-oriented agents.
Takeaways & Limitations
RealMem currently focuses on memory-centric challenges in long-term project interactions and does not yet assess tool-use capabilities.
Abstract
from arXiv · showhide
As Large Language Models (LLMs) evolve from static dialogue interfaces to autonomous general agents, effective memory is paramount to ensuring long-term consistency. However, existing benchmarks primarily focus on casual conversation or task-oriented dialogue, failing to capture **"long-term project-oriented"** interactions where agents must track evolving goals. To bridge this gap, we introduce **RealMem**, the first benchmark grounded in realistic project scenarios. RealMem comprises over 2,000 cross-session dialogues across eleven scenarios, utilizing natural user queries for evaluation. We propose a synthesis pipeline that integrates Project Foundation Construction, Multi-Agent Dialogue Generation, and Memory and Schedule Management to simulate the dynamic evolution of memory. Experiments reveal that current memory systems face significant challenges in managing the long-term project states and dynamic context dependencies inherent in real-world projects. Our code and datasets are available at [https://github.com/AvatarMemory/RealMemBench](https://github.com/AvatarMemory/RealMemBench).
1 Introduction
RealMem addresses the gap between conventional dialogue-memory benchmarks and long-term project-oriented interaction, where agents must maintain evolving context across interleaved sessions. It introduces a benchmark and synthesis pipeline designed to evaluate proactive, coherent memory use, while evaluations show current systems struggle with coherent project threads.
- Memory is foundational for agents that must sustain context-aware collaboration and support applications ranging from personalized chatbots to AGI.
- Long-term project interactions require organic queries, interleaved sessions, evolving states, and proactive alignment with ambiguous user intents.
- Existing benchmarks emphasize casual conversation or artificial task dialogues, leaving real-world workflow coherence insufficiently represented.
- RealMem contains over 2,000 cross-session dialogues across eleven project scenarios and evaluates natural queries that use accumulated memory to maintain project continuity.
- Its three-stage synthesis pipeline constructs project foundations, generates multi-agent dialogues, and manages memory and schedules as interactions evolve.
- Extensive evaluations find that state-of-the-art memory systems struggle to maintain coherent project threads and proactively align with evolving contexts.
2 Related Work
Prior benchmarks mainly test retention, retrieval, or updating in episodic dialogue settings. RealMem instead evaluates memory for evolving project states, interleaved queries, and proactive alignment.
- Existing dialogue benchmarks evaluate memory largely through static recall or controlled updating over long contexts.
- RealMem targets long-term project-oriented interactions rather than isolated fact retrieval.
- RealMem’s query design includes temporal reasoning, static retrieval, dynamic updating, and proactive alignment.
3 Method
RealMem synthesizes realistic long-term project interactions through coupled project planning, multi-agent dialogue generation, and closed-loop memory and schedule management. The design preserves global coherence while allowing memory and project states to evolve across sessions.
- The framework combines static project foundation construction, dynamic interaction generation, and closed-loop memory feedback.
- Project Foundation Construction: Project foundations encode personas, goals, and dynamic project attributes that track temporal changes in user progress and context.
- Project Foundation Construction: Blueprints, event lists, and session summaries hierarchically align local dialogues with global project narratives, while interleaving sessions across projects.
- Multi-Agent Dialogue Generation: User and Assistant Agents generate sessions from structured project background, historical memory points, and established schedules.
- Multi-Agent Dialogue Generation: The User Agent receives session-relevant context, whereas the Assistant Agent receives memory relevant to the current session.
- Memory and Schedule Management: Memory Extraction, Schedule, and Deduplication Agents convert dialogues into updated memory points and schedules while reducing redundant entries.
- Memory and Schedule Management: Retrieved memory is fed into later dialogue cycles, enabling continuous refinement and coherent interaction across sessions.
4 Evaluation
RealMem evaluates retrieval and generation quality using quantitative metrics and semantic judgments. Its measures distinguish ranking accuracy, semantic usefulness, and whether responses incorporate users’ evolving states.
- The evaluation combines standard quantitative retrieval metrics with LLM-based semantic judgments.
- Retrieval Evaluation: Recall@k and NDCG@k at k ∈{10, 20} measure strict retrieval ranking accuracy.
- Retrieval Evaluation: Mem Recall measures semantic coverage of relevant information, while Mem Helpful measures the practical usefulness of retrieved context.
- Generation Evaluation: QA Score evaluates whether responses consistently incorporate the user’s dynamic state rather than merely producing fluent, context-agnostic text.
5 Experiment
RealMem evaluates memory systems in long-term, project-oriented interactions using multiple context settings and fine-grained task categories. Results show complementary architectural strengths, substantial context-management challenges, domain variation, and efficiency trade-offs.
- Experiment Setup: Representative systems are evaluated with GPT-4o-mini and GPT-4o across memory-only and session-based context settings.Memory-only context uses retrieved memory entries, whereas session-based context supplies the corresponding original dialogue sessions; MemoryOS is unavailable in the latter setting.
- Overall Evaluation: MemoryOS leads in memory-only generation, while Graph Memory achieves the highest scores when session context is available.The results associate MemoryOS’s memory-only performance with hierarchical memory and Graph Memory’s session-context performance with modeling intricate entity relationships.
- Overall Evaluation: 0.7235 Recall@20 is achieved by A-mem, whereas Graph Memory leads NDCG@10 at 0.5654, aligning with its stronger generation performance.The comparison indicates that A-mem retrieves broader but noisier information, while Graph Memory prioritizes more relevant context.
- Overall Evaluation: MemoryOS leads dynamic retrieval categories and proactive alignment, while Graph Memory dominates temporal reasoning; performance generally declines from single-session to multi-session settings.MemoryOS records QA 0.521 for Dynamic Updating, QA 0.571 for Proactive Alignment, and QA 0.375 for Graph Memory in Temporal Reasoning; its session QA scores are 0.505 single-session and 0.468 multi-session.
- Scenario-Based Performance Analysis: MemoryOS performs strongest in consultative and creative domains, but Code Architecture scores below 0.4 across QA Score, Recall, and Helpful Score.The scenario analysis reports Helpful Score values approaching or exceeding 0.6 to 0.8 in several open-ended domains, while technical dependency tracking exposes limitations under hard constraints.
- Efficiency and Cost Analysis: Memory incorporation latency exceeds retrieval speed across systems, while MemoryOS combines 0.151 s retrieval latency with the highest token cost of 9,201.A-mem uses fewer resources but trades efficiency for retrieval accuracy; system-level rankings also align with human preferences, with MemoryOS highest and Mem0 lowest.
6 Conclusion
RealMem benchmarks long-term memory in realistic, project-oriented interactions, requiring agents to track evolving goals and maintain coherent project states across sessions. Its results identify persistent limitations in handling long-term project dependencies and a substantial gap from the Oracle upper bound.
- RealMem evaluates long-term memory in realistic, project-oriented interactions where agents track evolving goals across sessions.
- The benchmark emphasizes sustained multi-session projects and controlled simulation of dynamic memory evolution.
- Existing memory systems remain fundamentally limited in handling long-term project dependencies.
- The substantial gap between evaluated methods and the Oracle upper bound highlights effective long-term memory integration as a core bottleneck.
Limitations
RealMem relies substantially on Gemini 2.5 models for data collection and currently evaluates memory-centric challenges without tool use assessment.
- Data construction relies significantly on Gemini 2.5 models, with human annotation used for label verification.
- The benchmark’s evaluation scope is limited to memory-centric challenges in long-term project-oriented interactions.
- RealMem does not yet assess tool use capabilities, which the authors plan to incorporate in future work.
A.1 Supplementary Details of RealMem
The supplementary section provides statistical information and definitions to clarify RealMem’s dataset composition and task taxonomy, including an overview of its evaluation scenarios.
- The appendix supplies additional statistical information and key definitions for understanding RealMem’s data composition and task taxonomy.
- Table 7 presents a statistical overview of the RealMem dataset.
A.1.1 Definition of Question Types
RealMem classifies user queries by retrieval requirements and interaction logic into static retrieval, dynamic updating, proactive alignment, and temporal reasoning.
- The taxonomy is presented as four query types categorized by retrieval requirements and interaction logic.
- Static Retrieval queries advance the project state from confirmed context, requiring retrieval of the latest stable state without changing prior decisions.
- Dynamic Updating queries modify, delete, or resolve conflicts in existing plans by retrieving constraints and revising the project state.
- Proactive Alignment queries provide vague or emotional feedback, requiring retrieval of long-term priorities or preferences to propose the next action.
- Temporal-reasoning queries process time-sensitive information by validating schedules or sequencing events chronologically.
- Table 8 summarizes the eleven evaluation scenarios and their core task attributes.
A.1.2 Definition of Memory Types
The paper organizes memory into persona, project-state, and schedule categories, then illustrates how these memories support diverse project interactions through incremental progression, proactive alignment, and dynamic updating. Its evaluation also reports strong domain sensitivity, with Code Architecture Design performing worst across methods.
- Memory Categories: Persona memory stores user attributes, goals, and preferences to maintain personalization across sessions.It includes profiles, long-term goals, dietary restrictions, and preferred writing styles.
- Memory Categories: Project-state memory records evolving task content, such as itinerary details, architecture decisions, and plot outlines, to preserve project continuity.These memories function as the project knowledge base and reduce redundant inquiries.
- Memory Categories: Schedule memory stores appointments, deadlines, and routines to support temporal reasoning and detect conflicts with proposed plans.The category represents structured, time-sensitive information on the user’s global timeline.
- Scenario Coverage: The benchmark covers eleven scenarios spanning life planning, professional and skill development, and personal well-being, with explicit attributes defining evaluation boundaries.Examples include travel, financial and project planning, academic writing, code architecture, fitness, mental health, and health consultation.
- Evaluation Pattern: 0.363 is the MemoryOS QA score for Code Architecture Design, the lowest-scoring topic across evaluated methods.The paper attributes this pattern to difficulties with strict logical consistency and precise dependency tracking in current retrieval-based memory systems.
- Case Studies: Case studies show memory supporting incremental progression, proactive alignment, and dynamic updating as users refine itineraries and introduce constraints.The examples include using prior food preferences to advance itinerary planning, retrieving booking priorities after affirmative feedback, and reallocating itinerary time while preserving a 12-day limit.