PaperlayerView exact revision on arXiv

Source-linked AI summary

From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents

Md Nayem Uddin, Kumar Shubham, Eduardo Blanco, Chitta Baral, Gengyu Wang

arXiv:2604.20006v1cs.CL

TL;DR

Long-term-memory benchmarks have mostly tested retrieval, leaving consolidation and frequent knowledge updates underexamined. Memora evaluates these challenges across long-horizon conversations with three memory-grounded tasks and introduces FAMA to penalize obsolete-memory use. Results expose frequent invalid-memory reuse, failures to maintain consistent memory states, and only limited improvements from memory agents.

  • Problem

    Existing benchmarks emphasize shallow retrieval, providing limited evidence about consolidating information over extended interactions and handling frequent memory mutation.

  • Method

    Memora simulates weeks-to-months conversations, evaluates remembering, reasoning, and recommending, and uses FAMA to reward valid memory while penalizing obsolete memory.

  • Results

    Evaluations reveal frequent obsolete-memory reuse and persistent failures to maintain consistent belief states, while memory agents provide only limited improvements.

  • Takeaways & Limitations

    Long-term conversational memory requires explicit integration of forgetting, consolidation, and mutation as first-class design principles.

  • Takeaways & Limitations

    Memora relies on simulated long-horizon conversations, which cannot fully capture the ambiguity and unpredictability of real user interactions.

Abstract

from arXiv · show

Personalized agents that interact with users over long periods must maintain persistent memory across sessions and update it as circumstances change. However, existing benchmarks predominantly frame long-term memory evaluation as fact retrieval from past conversations, providing limited insight into agents' ability to consolidate memory over time or handle frequent knowledge updates. We introduce Memora, a long-term memory benchmark spanning weeks to months long user conversations. The benchmark evaluates three memory-grounded tasks: remembering, reasoning, and recommending. To ensure data quality, we employ automated memory-grounding checks and human evaluation. We further introduce Forgetting-Aware Memory Accuracy (FAMA), a metric that penalizes reliance on obsolete or invalidated memory when evaluating long-term memory. Evaluations of four LLMs and six memory agents reveal frequent reuse of invalid memories and failures to reconcile evolving memories. Memory agents offer marginal improvements, exposing shortcomings in long-term memory for personalized agents.

1 Introduction

Existing long-term-memory benchmarks largely test shallow retrieval, while persistent assistants must consolidate information and update or discard memories as users’ circumstances change. Memora addresses this gap with temporally extended tasks and FAMA, exposing frequent reuse of obsolete memories and limited gains from memory agents.

  • LLMs are stateless across interactions, so preferences, corrections, and goals are not retained unless users explicitly reintroduce them.
  • Long-term memory requires accumulating experiences, reconciling changes, and discarding outdated knowledge rather than merely recalling past information.
  • 94% of LoCoMo questions and 85% of LongMemEval questions require evidence from no more than two previous sessions, emphasizing shallow cross-session retrieval.
  • Memora requires memory consolidation and mutation across weekly, monthly, and quarterly sessions, covering remembering, reasoning, and recommending tasks.
  • FAMA rewards valid memory use while penalizing obsolete or deleted memory, enabling evaluation of mutation over long interaction histories.
  • Memora evaluations reveal frequent obsolete-memory reuse, inconsistent belief states, and only limited improvements from long-term memory agents.

2 Related Works

Prior work has expanded long-term-memory evaluation through longer conversations and personalization, but it still predominantly treats memory as fact retrieval. Consequently, consolidation across extended timelines and frequent mutation remain limited testing targets.

  • Long-context modeling differs from long-term memory because realistic agents need persistent, updatable memory rather than simply larger prompts.
  • Existing benchmarks have increased conversation length and tested personal-fact recall, question answering, summarization, narrative recall, and million-token dialogues.
  • Personalization benchmarks evaluate persona-consistent dialogue, evolving user-state inference, and user-specific facts or events over extended interactions.
  • Across both research lines, long-term memory remains predominantly fact retrieval, with limited emphasis on consolidation and frequent mutation.

3 Memora

Memora constructs long-horizon, memory-grounded conversations from simulated personas and explicit memory traces. Its questions span remembering, reasoning, and recommending, with criteria that require valid information and exclude outdated information.

  • Memora’s simulation-driven pipeline generates long-term conversations and evaluation tasks from persona-level seed data spanning weeks to months.
  • The benchmark models preference, activity, and goal memories across ten professional persona profiles.
  • A session simulator introduces, updates, or invalidates memory entities under temporal constraints and maintains a persistent state after every session.
  • Automated grounding checks and human verification filter generated conversations for alignment with their underlying memory traces.
  • Memora’s conversations show low lexical overlap and low semantic similarity, indicating linguistic diversity without formulaic reuse.
  • Evaluation questions cover remembering, reasoning, and recommending, pairing valid-memory inclusion criteria with outdated-memory exclusion criteria.

4 Experiments

The experiments evaluate both language models operating over conversation histories and memory agents that store and retrieve user information. Evaluation uses memory-grounded criteria and FAMA to distinguish valid memory use from obsolete-memory reuse.

  • Memora compares direct LLM processing of multi-session histories with long-term memory agents that explicitly store and retrieve user information.
  • Four LLMs are evaluated under standard and reasoning-enabled inference to test consolidation of long interaction histories.
  • Each question uses memory-presence criteria for valid information and forgetting-absence criteria for invalidated or deleted information.
  • Three LLM judges independently score each criterion, and majority voting determines the final evaluation outcome.
  • 88.3% average agreement with human annotations supports the reliability of the LLM-based judgments, with Cohen’s κ ranging from 0.86 to 0.90.
  • FAMA aggregates criterion-level judgments, rewarding valid-memory use and penalizing reliance on obsolete memory.

5 Results

FAMA results show that long-term memory performance declines with longer, mutation-heavy histories, varies sharply by task, and is overstated by evaluations that ignore obsolete memories. Explicit memory agents help factual remembering but remain brittle in reasoning and temporal updating.

  • Performance Across Temporal Durations: Performance generally declines from weekly to quarterly settings, with the largest deterioration often appearing in remembering and reasoning.For example, MemoBase declines from 43.6 to 15.18 on remembering and from 18.00 to 1.00 on reasoning.
  • Performance Across Tasks: Memory agents achieve 119.45 average aggregated FAMA on remembering, versus 65.60–65.80 for language models.On recommending, language models score 144.72 without reasoning and 153.34 with reasoning, compared with 138.37 for agents; reasoning remains low overall.
  • Performance Across Tasks: Reasoning is the weakest task overall, with best average performance of 27.55 for memory agents versus 12.37 and 13.92 for language models.The results indicate that strong retrieval does not guarantee effective reasoning over temporally distributed memory.
  • The Effect of Forgetting-Aware Evaluation: Standard memory-presence accuracy overestimates performance because it does not penalize reliance on obsolete or invalidated memories.Applying FAMA produces large score reductions across language models and memory agents.
  • The Effect of Forgetting-Aware Evaluation: FAMA reductions diverge by system type: language-model reductions fall from 32.6 weekly to 17.8 quarterly, while memory-agent reductions rise from 18.2 to 29.5.The paper attributes the language-model pattern to omitted information beyond the context window and the memory-agent pattern to reliance on memories that should have been revised or discarded.
  • The Effect of Forgetting-Aware Evaluation: Forgetting-aware penalties can change system rankings, as Nemori surpasses MemoryOS and A-Mem despite lower memory-presence accuracy.In the monthly setting, Nemori’s reduction is 15.4, compared with 28.4 for MemoryOS and 29.5 for A-Mem.
  • Forgetting-Aware Performance Variability: FAMA performance is highly variable across temporal durations and tasks, indicating that current systems are brittle rather than merely inaccurate.Figures 3 and 4 visualize variability across tasks and durations using mean per-question FAMA scores and error bars.

6 Error Analysis

Error patterns differ by task: recommendation failures involve outdated or partial preferences, remembering failures involve incomplete retrieval, and reasoning failures involve missing elements that prevent consolidation.

  • Recommendation errors: 64% of recommendation errors were caused by outdated memory not being forgotten, while 28% involved partial preference retrieval.Agents often retrieved historical preferences without applying recent updates.
  • Remembering errors: 72% of remembering errors involved partial memory retrieval, often producing incomplete structured outputs.Agents retrieved some but not all required memory items.
  • Reasoning errors: 100% of reasoning errors involved incomplete retrieval of relevant memory elements, preventing correct consolidation.Missing elements impaired synthesis across accumulated information.
  • Cross-task pattern: Task-specific errors collectively reveal failures in forgetting outdated information, retrieving complete memories, and consolidating them across interactions.The analysis covered 75 incorrect predictions, with 25 samples per task.

7 Conclusion

Memora is a controlled stress test for temporally consistent long-term memory, using explicit memory traces and FAMA to expose failures that standard metrics miss. The findings support treating forgetting, consolidation, and mutation as first-class design principles.

  • Benchmark contribution: Memora provides a controlled stress test that isolates long-term memory challenges and enables diagnostic evaluation.Its interactions use explicit memory traces to assess temporally consistent memory states rather than isolated recall.
  • Metric: FAMA penalizes reliance on invalidated memory and exposes substantial performance gaps across LLMs and memory agents.The metric is designed to capture failures that standard metrics do not.
  • Implication: Long-term conversational memory requires explicit integration of forgetting, consolidation, and mutation as first-class design principles.This conclusion follows from the benchmark’s diagnostic findings.

Limitations

Memora uses simulated long-horizon conversations and a constrained set of personas and memory categories, trading realism and breadth for scalable, standardized evaluation. The benchmark should therefore be viewed as a lower bound on real-world long-term memory difficulty.

  • Simulation scope: Memora relies on simulated long-horizon conversations with explicit memory creation, mutation, and deletion.Simulation supports scalable, standardized evaluation but cannot fully capture ambiguity and unpredictability in real interactions.
  • Data trade-off: Real-world memory logs would require user consent, privacy handling, and manual annotation, making them costly and difficult to scale or standardize.These constraints motivate the use of simulation.
  • Interpretation: Memora should be viewed as a lower bound because real deployments add implicit updates and contradictory signals.Systems that fail in simulation are unlikely to generalize to more complex real-world settings.
  • Coverage: The benchmark centers on a constrained set of personas and memory categories, limiting the breadth of evaluated user states.The supplied passage identifies this as a further design trade-off.

A Seed Data Details

Memora constructs evolving user states from professional personas and three memory types, regulates updates across temporal horizons, and generates controlled multi-turn dialogues for evaluation.

  • Personas: Memora uses ten professional personas as stable semantic anchors for simulating diverse long-term user behavior.Each persona represents a distinct professional role and has assigned preference archetypes.
  • Memory types: The benchmark models preference, activity, and goal memory to capture evolving behaviors and long-term objectives.These memory types support analysis of consolidation, mutation, and forgetting.
  • Preference memory: Preference memory evolves through additions, updates, and deletions across movies, books, music, and travel.It provides the primary signal for personalized recommendation tasks.
  • Activity memory: Activity memory captures dynamic personal and work behaviors, including recurring time-indexed activities such as expense tracking and task management.It is the most dynamic and frequently updated memory type.
  • Goal memory: Goal memory represents relatively stable long-term objectives that anchor reasoning over activity history.Examples include financial budgets and fitness targets.
  • Update rules: Operational constraints determine whether memory entries can be added, updated, or deleted according to their category.Append-only records allow additions, while mutable artifacts can support updates and deletions.
  • Temporal design: Temporal constraints interleave memory-grounded and memory-neutral sessions across weekly, monthly, and quarterly configurations.Longer durations increase memory operations and pressure for consolidation and mutation.
  • Generation pipeline: The Session Manager tracks persona, memory and operation types, relevant fields, and pre-session and post-session memory states.This structured representation drives conversation generation.

B.5 Auto-Evaluation and Grounding Verification

Memora uses automated grounding checks and regeneration, supplemented by human review, to ensure generated conversations faithfully express valid memory operations without reintroducing invalidated information.

  • Generated conversations are evaluated against session specifications and targeted memory-grounding checks, then regenerated until they achieve full alignment.The checks assess whether intended memory operations are expressed accurately and completely.
  • The evaluation–regeneration cycle provides targeted feedback about missing, incorrect, or inconsistent information while preserving the original conversational structure.
  • Most conversations converge within a small number of evaluation iterations, indicating stable and efficient grounding enforcement.
  • Human annotators review a stratified 5% sample across personas, memory types, and operation types.They verify required information, exclusion of invalidated or deleted information, and dialogue quality.
  • Memora's validation criteria require memory presence, forgetting absence, and natural, coherent, linguistically diverse conversations.

C.1 LLM Judge Details and Reliability

Memora evaluates semantic memory correctness with atomic presence and forgetting criteria judged by three deterministic LLMs using majority voting. The judges show high agreement across temporal settings, supporting the protocol's reliability.

  • Memora decomposes each evaluation question into atomic criteria that separately assess valid memory presence and exclusion of invalidated or deleted information.This separation enables fine-grained analysis of memory consolidation and mutation.
  • Three independent judges—GPT-4.1, Claude Haiku 4.5, and Gemini 2.5 Flash—evaluate semantic meaning and intent rather than exact wording.
  • Deterministic decoding and majority voting produce reproducible criterion-level decisions from structured binary judgments.Judges use temperature 0.0 and return yes/no labels with confidence scores and explanations.
  • Parsing retries and conservative handling of invalid outputs prevent evaluation failures from artificially inflating model performance.
  • Most evaluations yield unanimous agreement across temporal spans, while partial disagreements account for a smaller share and decline relatively as duration increases.
  • κ values consistently exceed 0.80 for all judge pairs across weekly, monthly, and quarterly evaluations, indicating near-perfect agreement.
  • Agreement remains stable under the benchmark's high consolidation and frequent memory-mutation conditions, supporting robust interpretation across judges.

C.2 Human Validation of LLM Judges

Memora calibrates its multi-judge evaluation against human annotation using stratified sampling and independent binary labels. Human agreement is high overall, with lower agreement concentrated in split decisions.

  • The human-validation study samples 100 criteria across unanimous and majority-consensus patterns to cover both high-confidence and disagreement cases.The sample contains 25 criteria in each of four consensus categories.
  • Three human annotators independently assign binary labels using the model response, evaluation criterion, and annotation instructions.
  • Agreement between LLM majority votes and humans is highest for unanimous cases and lower for split decisions, concentrating discrepancies in ambiguous instances.
  • Human annotator agreement ranges from 93% to 95%, with Cohen’s κ between 0.86 and 0.90.
  • The study concludes that strong LLM–human alignment supports the validity and reliability of Memora's majority-vote evaluation protocol.

Session 142 (Reasoning)

The reasoning example tests whether an agent integrates a persistent step goal with accumulated activity history. The response correctly reports performance against the goal and indicates that it was met.

  • The response reports a 7,500-step daily goal and an average of approximately 10,321 recent steps, exceeding the target.
  • The evaluation marks the goal as met because the reported average exceeds 7,500 steps.
  • The underlying activity series records that the user met or exceeded the daily goal in 100% of the sessions in this series.

E Additional Experimental Details

The appendix describes evaluation settings for LLMs and external-memory agents, and compares the backends and retrieval mechanisms used by the evaluated agents.

  • Agent comparison: The compared agents differ in storage backends, retrieval strategies, and embedding models, while sharing a common evaluation pipeline.The shared pipeline includes answer generation, retry handling, and progress checkpointing.
  • Agent comparison: Retrieval mechanisms range from vector similarity and embedding lookup to hybrid vector–keyword search and opaque provider-managed systems.Opaque mechanisms abstract away the underlying embedding model or retrieval logic from the agent implementation.
  • LLM-based evaluation: LLMs are evaluated by receiving full multi-session conversation histories directly in context, without external memory systems.This setting targets intrinsic long-context memory and consolidation capabilities across models with varying native context lengths.
  • Agent-based evaluation: External-memory agents incrementally ingest conversations, store user-specific information, retrieve relevant memories, and condition answers on retrieved content.All agents use identical conversation streams and question sets, with a unified user identifier format for persistence across sessions.
  • Agent-based evaluation: The evaluation covers six representative agents spanning local, cloud-based, and hybrid memory designs.The systems include A-Mem, LangMem, Mem-0, MemoBase, MemoryOS, and Nemori.
Loading 2604.20006v1…